What is the course about?

The aim of this course if to introduce you to continuous-time stochastic control. Continuous-time differential equation are a fundamental tool for modelling real-world phenomena, with major applications in physics, biology, finance and engineering. A necessary step towards this aim is to introduce you to the basics of Ito calculus. We will do so in two steps: first, we will discuss jump driven processes. This class of process can be used to model continuous-time Markov chain, among other things. We will spend some time exploring the subject and introduce the Ito rule, show how to reverse the time evolution in this stochastic setting and derive the density equation (Fokker-Planck equation). In a second part, we will introduce Brownian processes as limits of jump processes, discuss system theoretic issues and derive the (continuous-time) Kalman-Bucy filter.

Contents

  1. Review of Probability Sets and Probability Spaces, Probability Distributions on Vector Spaces, Independence and Conditional Probability, Moments and their Generating Function, Transformation of Densities, Empirical Determination of Densities
  2. Poisson Counters and Differential Equations Poisson Counters, Finite-State,Continuous-Time Jump Processes, The Itô Rule for Jump Processes, Computing Expectations, The Equation for the Density, The Backwards Evolution Equation, Computing Temporal Correlations, Linear Systems with Jump Process Coefficients
  3. Wiener Processes and Differential Equations Gaussian Distributions, Brownian Motion ,Stochastic Differential Equations, The Itô Rule, Expectations, Finite Difference Approximations, A Digression on Stratonovic Calculus, The Fokker-Planck Equation, Computing Temporal Correlations, Linear Equations, Asymptotic Behavior, Stochastic Approximation, Exit Times.
  4. Dynamics Programming and Optimal control. Dynamic programming, principle of optimality, HJB equation.
  5. Pseudo-Random processes Pseudorandom Number Generators, Uniform versus Gaussian Distributions, The Results of Jacobi and Bohl, Sierpinskii and Weyl, Other Difference Equations, Differential Equations.
  6. System Concepts Modeling, Deterministic Linear Systems, Covariance and the Power Spectrum, The Zero-Crossings Problem, Stochastic Realization Problem, Linear Stationary Realizations, Spectral Factorization, The Gauss-Markov Heat Bath, Reducibility, Covariance Generation with Finite State Processes
  7. Estimation Theory Continuous Time Formulation, More General State Spaces, An Exponential Representation, Extrapolation and Smoothing, The Orthogonality Principle, Linear Estimation Theory, Linear Smoothing, Identification of Linear Systems, Baum-Welch Identification
  8. Stochastic Control Stochastic Control with Perfect Observations, The Linear, Quadratic, Gaussian Problem, Multiplicative Noise, Markov Decision Problems, The Minimum Return Function, The Constant Coefficient Cases, Controllability, Stochastic Control with Noisy Observations, Stochastic Control with No Observations