Value function

The value function of an

In a problem of

supremum

of the objective function taken over the set of admissible controls. Given ${\displaystyle (t_{0},x_{0})\in [0,t_{1}]\times \mathbb {R} ^{d}}$, a typical optimal control problem is to

${\displaystyle {\text{maximize}}\quad J(t_{0},x_{0};u)=\int _{t_{0}}^{t_{1}}I(t,x(t),u(t))\,\mathrm {d} t+\phi (x(t_{1}))}$

subject to

${\displaystyle {\frac {\mathrm {d} x(t)}{\mathrm {d} t}}=f(t,x(t),u(t))}$

with initial state variable ${\displaystyle x(t_{0})=x_{0}}$.[8] The objective function ${\displaystyle J(t_{0},x_{0};u)}$ is to be maximized over all admissible controls ${\displaystyle u\in U[t_{0},t_{1}]}$, where ${\displaystyle u}$ is a Lebesgue measurable function from ${\displaystyle [t_{0},t_{1}]}$ to some prescribed arbitrary set in ${\displaystyle \mathbb {R} ^{m}}$. The value function is then defined as

with ${\displaystyle V(t_{1},x(t_{1}))=\phi (x(t_{1}))}$, where ${\displaystyle \phi (x(t_{1}))}$ is the "scrap value". If the optimal pair of control and state trajectories is ${\displaystyle (x^{\ast },u^{\ast })}$, then ${\displaystyle V(t_{0},x_{0})=J(t_{0},x_{0};u^{\ast })}$. The function ${\displaystyle h}$ that gives the optimal control ${\displaystyle u^{\ast }}$ based on the current state ${\displaystyle x}$ is called a feedback control policy,^[4] or simply a policy function.^[9]

Bellman's principle of optimality roughly states that any optimal policy at time ${\displaystyle t}$, ${\displaystyle t_{0}\leq t\leq t_{1}}$ taking the current state ${\displaystyle x(t)}$ as "new" initial condition must be optimal for the remaining problem. If the value function happens to be continuously differentiable,^[10] this gives rise to an important partial differential equation known as Hamilton–Jacobi–Bellman equation,

${\displaystyle -{\frac {\partial V(t,x)}{\partial t}}=\max _{u}\left\{I(t,x,u)+{\frac {\partial V(t,x)}{\partial x}}f(t,x,u)\right\}}$

where the maximand on the right-hand side can also be re-written as the Hamiltonian, ${\displaystyle H\left(t,x,u,\lambda \right)=I(t,x,u)+\lambda (t)f(t,x,u)}$, as

${\displaystyle -{\frac {\partial V(t,x)}{\partial t}}=\max _{u}H(t,x,u,\lambda )}$

with ${\displaystyle \partial V(t,x)/\partial x=\lambda (t)}$ playing the role of the

Given this definition, we further have ${\displaystyle \mathrm {d} \lambda (t)/\mathrm {d} t=\partial ^{2}V(t,x)/\partial x\partial t+\partial ^{2}V(t,x)/\partial x^{2}\cdot f(x)}$, and after differentiating both sides of the HJB equation with respect to ${\displaystyle x}$,

${\displaystyle -{\frac {\partial ^{2}V(t,x)}{\partial t\partial x}}={\frac {\partial I}{\partial x}}+{\frac {\partial ^{2}V(t,x)}{\partial x^{2}}}f(x)+{\frac {\partial V(t,x)}{\partial x}}{\frac {\partial f(x)}{\partial x}}}$

which after replacing the appropriate terms recovers the costate equation

${\displaystyle -{\dot {\lambda }}(t)=\underbrace {{\frac {\partial I}{\partial x}}+\lambda (t){\frac {\partial f(x)}{\partial x}}} _{={\frac {\partial H}{\partial x}}}}$

where ${\displaystyle {\dot {\lambda }}(t)}$ is

The value function is the unique viscosity solution to the Hamilton–Jacobi–Bellman equation.^[13] In an online closed-loop approximate optimal control, the value function is also a Lyapunov function that establishes global asymptotic stability of the closed-loop system.^[14]

References