\(\def\Res{\mathtt{Res}}\def\Real{\mathbb{R}}\)

 Consider optimal control problem for \(x\in\Real, u\in\Real\):
\[
\dot{x}=u,\quad J_0^T=\int_0^T\frac{u^2}{2}dt +\cos(x(T))\to\min.
\] [20]
Find maximal \(T\) for which the cost function (referred to as \(S\) in the course notes) at time \(0\),
\[
V(x, 0)=inf_{\{u(t)\}_{0\leq t\leq T}, x(0)=x}\ J_0^T
\]
is smooth as a function of the initial point \(x\) (i.e. has continuous derivative).Solution. As we discussed, the cost function for the problem above is given by
\[
V(x,0)=\min_X C(x,X,T), \mathrm{where} C(x,X,T)=\frac{(xX)^2}{2T}+\cos(X)
\]
(here \(X=x(T)\) is the potential position of the end of the trajectory, and \((xX)^2/2T\) is the minimal cost of reaching it from \(x\)). The cost function loses smoothness when \(C\) acquires nonunique minimum for some \(x\). This can happen only if \(C\) becomes nonconvex (in \(X\), i.e. when the second derivative
\[
\frac{d^2C}{dX^2}=\frac1T\cos(X)
\]
vanishes somewhere. This happens at \(T=1\).  [15]
Describe \(V(0,0)\) as a function of \(T\).Solution. We are looking at
\[
V(0,0)=\min_X \frac{(X)^2}{2T}+\cos(X)
\]
For \(T\leq 1\) this is a convex function (of \(X\)), having vanishing derivative at \(0\), and the corresponding minimum attained at \(0\) is \(1\).
For \(T\gt 1\), there are two competing (global) minima, at point \(X(T)\) solving
\[
X(T)/T=\sin(X(T)), 0\lt X(T)\lt \pi
\]
(there is no closed form expression for it), and the resulting cost, as a function of \(T\), is shown below. Note that it converges to \(1\) as \(T\to\infty\).
 [20]
 Consider LQR problem with
\[
\dot{x}=Ax+bu, J=\int_0^T c x^2+ru^2 dt +mx(T)^2,
\]
where
\[
A=\left(
\begin{array}{cc}2&0\\0&1\end{array}
\right),
b=\left(\begin{array}{c}0\\1\end{array}\right),c=(1\quad 1).
\] [15] Solve corresponding Riccati differential equation for \(T=1, r=1, m=0\);
 [20]
Same for \(T=1, r=1, m=6\). Is the solution regular (bounded, smooth) for all \(t\)?  [30]
Find smallest \(m\) for which the RDE with \(P(T)=mE\) – with \(E\) the identity matrix, – is regular for any \(T\).
Solution:
One constructs, in the usual fashion, the Hamiltonian matrix
\[
H=\left(
\begin{array}{cccc}
2&0&0&0\\0&1&0&1\\1&1& 2& 0\\1& 1& 0& 1
\end{array}
\right)
\]Using computer algebra, one can find \(\exp(T H)\), and apply it to the \(2n\times n\) matrix \(E;mE\), corresponding to the terminal value of \(X,Y\). Then one can find \(P=YX^{1}\) – the solution of RDE at \(t=0\). The result is somewhat long:
To check, whether \P\) is bounded, it is enough to check that \(\det X\) does not vanish.
Again, using computer algebra we get
\[
\det X(t)=e^{(2 – \sqrt{2}) T} (2 + \sqrt{2} – \sqrt{2} m +
e^{2 \sqrt{2} T} (2 – \sqrt{2} + \sqrt{2} m))/4.
\]We see that for \(m=0\), the determinant is positive for all \(T\), while for \(m=6\) it is vanishing at
\[
Tt=\frac{1}{2\sqrt{2}}\log{\frac{2+7\sqrt{2}}{7\sqrt{2}2}}\approx .14485…
\]It is immediate that \(\det X\) vanishes for some \(Tt>0\) iff
\[
m=\frac{\sqrt{2}+1e^{2 \sqrt{2}(Tt)}(\sqrt{2}1)}{e^{2 \sqrt{2}(Tt)}1}.
\]
As \(e^{2 \sqrt{2}(Tt)}\) varies from \(1\) to \(\infty\), the values of \(m\) that could lead to vanishing of \(\det X\) lie in the interval \((\infty,\sqrt{2}+1)\). Thus the smallest \(m\) ensuring regularity of \(P\) is \(\sqrt{2}+1\).
 Consider optimal control problem for \(x\in\Real, u\in\Real\):
In the Problem 2 cost function, should c*x be cx so the dimensions match?
It should, and it is, – belatedly, – now. Thanks!
How would we solve x(T)^2 if x is a 2×1 column vector?
Think of x(T) as the norm, rather than the absolute value.
How are you defining “regular” in this context? I didn’t see anything in Basar’s notes, but it’s possible I missed something.
Not exploding…
Can you be more explicit about the arguments of V in problem 1? The notes use a number of different conventions, with variations on x, u, and t.
Updated.
I thought $$V(x,0)$$ is minimizing $$J_0^T$$ with respect to $$u(t)$$ and $$x(T)$$, and therefore, $$V(x,0)$$ will be a function of $$x(0)$$. But the definition of $$V(x,0)$$ stated in problem 1.a is minimizing $J_0^T$ with respect to $$u(t)$$ and $$x(0)$$, then $$V(x,0)$$ will not be a function of $$x(0)$$. Not sure at which part i am getting it wrong.
I thought \(V(x,0)\) is minimizing \(J_0^T\) with respect to $$u(t)$$ and \(x(T)\), and therefore, \(V(x,0)\) will be a function of \(x(0)\). But the definition of \(V(x,0)\) stated in problem 1.a is minimizing \(J_0^T\) with respect to \(u(t)\) and \(x(0)\), then \(V(x,0)\) will not be a function of \(x(0)\). Not sure at which part I am getting it wrong.
Isn’t the problem 2 too complicated to do hand calculation? It’s frustrating…… Using matlab doesn’t work either because the result is to complicated in variables. Is there a simpler way to deal with p2?
The result is moderately complicated, but does involve normal form of a 4×4 matrix, so that I expect some computer assistance.
Thanks
For Problem 1 (b), is there an analytical solution to V(0,0) or do we just need to plot it?
No close form solution is known; just describe the minimizers and sketch the value of the function.