[tex]f_x=4y^2-2xy^2-y^3=y^2(4-2x-y)=0\implies y=0\text{ or }2x+y=4[/tex]
[tex]f_y=8xy-2x^2y-3xy^2=xy(8-2x-3y)=0\implies x=0\text{ or }y=0\text{ or }2x+3y=8[/tex]
For now, ignore any critical points that occur on the boundary of [tex]\mathcal D[/tex].
Case 1: If [tex]y=0[/tex], then either [tex]x=0[/tex] or [tex]2x=8\implies x=4[/tex], so we have two critical points (0, 0) and (4, 0), both on the boundary.
Case 2: Suppose [tex]2x+y=4[/tex]. Then ...Case 2a: ... if [tex]x=0[/tex], then [tex]y=4[/tex] - critical point at (0, 4) on the boundary.
Case 2b: ... if [tex]y=0[/tex], then [tex]2x=4\implies x=2[/tex] - critical point at (2, 0) on the boundary.
Case 2c: ... if [tex]2x+3y=8[/tex], then
[tex]\begin{cases}2x+y=4\\2x+3y=8\end{cases}\implies x=1\text{ and }y=2[/tex]
- critical point at (1, 2) not on the boundary.
We compute the Hessian matrix of [tex]f(x,y)[/tex]:
[tex]\mathbf H(x,y)=\begin{bmatrix}f_{xx}&f_{xy}\\f_{yx}&f_{yy}\end{bmatrix}=\begin{bmatrix}-2y^2&4(2-x)y-3y^2\\4(2-x)y-3y^2&8x-2x^2-6xy\end{bmatrix}[/tex]
[tex]\det\mathbf H(1,2)=32>0\text{ and }f_{xx}(1,2)=-8<0[/tex]
which means (1, 2) is a local maximum, where we get [tex]f(1,2)=4[/tex].
Now we consider the boundary in three parts:
(1) Fix [tex]x=0[/tex]. Then [tex]f(0,y)=0[/tex]. It's possible that 0 is the smallest value of [tex]f(x,y)[/tex] whereever you go.
(2) Fix [tex]y=0[/tex]. Then [tex]f(x,0)=0[/tex], and we get the same result as in the previous case.
(3) Fix [tex]x+y=6[/tex]. Then [tex]f(x,6-x)=-2x(x-6)^2[/tex]. Denote this function by [tex]F(x)[/tex]. We compute the first derivative and find the critical points:
[tex]F'(x)=-2(x-6)^2-4x(x-6)=-2(x-6)(3x-6)=0\implies x=6\text{ or }x=2[/tex]
then compute the second derivative and determine its sign at these points:
[tex]F''(x)=-2(3x-6)-2(x-6)(3)=-12(x-4)\implies\begin{cases}F''(6)=-24<0\\F''(2)=24>0\end{cases}[/tex]
which indicates that a maximum occurs as [tex]x=6[/tex] and a minimum at [tex]x=2[/tex]. Respectively, we have [tex]6+y=6\implies y=0[/tex] and [tex]2+y=6\implies y=4[/tex], and [tex]f(6,0)=0[/tex] and [tex]f(2,4)=-64[/tex].
So the absolute maximum of [tex]f(x,y)[/tex] over [tex]\mathcal D[/tex] is 4 at (1, 2), and the absolute minimum is -64 at (2, 4).
