brianceco/continuous_reinforcement_learning

Reinforcement learning (RL) is a machine learning paradigm which formalizes the trial-and-error approach to learning: an agent interacts with an environment via some policy they can control, so as to maximize some reward. It has been widely successful in achieving high level performance in a wide range of activities, including games like GO, chess, and Dota. It has also been leveraged towards more important tasks, among them, teaching LLMs how to add.

Even with all of the attention RL has received within the ML community, it is a surprising fact that the theory of RL in continuous time and continuous (state and action) space has remained largely underdeveloped. The following blog post aims to give a very brief introduction to a research program carried out by Professor Xunyu Zhou and his collaborators over the last ~5 years to develop a theory of continuous RL. I will focus in particular on two companion papers, (Jia & Zhou, 2022a) and (Jia & Zhou, 2022b) by Zhou and Professor Yanwei Jia, developing continuous-time-and-space analogues of policy evaluation and policy improvement methods within the theory of relaxed stochastic control.

1. What is (discrete) RL?

Let’s begin by quickly recalling RL theory in discrete time and space, fixing notation and outlining what results we actually want to extend to the continuous setting.

Let’s consider a (time-homogeneous) Markov Decision Process (MDP) \(\{X_t\} _{t =0}^{\infty}\) with finite state space $\mathcal{X}$ and finite action space $\mathcal{A}$. The dynamics of $X_t$ are governed by a transition kernel

\[p(x^\prime|x,a)\coloneqq \mathbf{P}(X_{t+1}^{a}=x^\prime|X_{t}=x,A_t=a)\]

from $\mathcal{X} \times \mathcal{A}$ to $\mathcal{X}$, where \(\{A_{t}\}_{t=0}^{\infty}\) is the action process following an admissible policy \(A_t \sim \pi(da\mid X^{A}_t) \in \euscr{P}(\mathcal{A})\). We are interested in the episodic setting, in which our environment evolves over some finite time horizon \(T \in \mathbf{N}\). The goal is to identify an admissible policy \(\pi^{\ast}=\pi^{\phi^{\ast}}\) from some parametric family \(\phi \in \Phi \subset \mathbf{R}^{N_\phi}\) which is optimal from the point of view that it maximizes the value function

\[J(x;\pi) \coloneqq \E_{x}\left[ \sum\limits_{t=0}^{T}\beta ^{t} r(X^{A}_t,A_t) \right]\]

defined as the cumulative discounted expected reward, with discount rate $\beta \in (0,1)$ and reward function $r\colon \mathcal{X} \times \mathcal{A}\to \euscr{P}(\mathbf{R})$.

We aim to identify an optimal policy via gradient ascent along the value function for the current policy $\pi^{\phi}$ (i.e., the actor). Since the true value function is unavailable to us, we must reformulate the policy gradient $\partial J / \partial\phi$ in terms of the value function and/or Q-function, which we substitute with an estimate $J^{\theta}(x)$ (i.e. the critic). Such an Actor-Critic (AC) algorithm thus consists of iteratively carrying the following two steps:

1. (Approximation in value space $\Theta$). Given the current policy $\phi$, we need to estimate the associated value function $J^{\theta}(x)\approx J(x;\phi)$ from some parametric class $\theta \in \Theta \subset \mathbf{R}^{N_{\theta }}$. This is the policy evaluation (PE) step.
2. (Approximation in policy space $\Phi$). We update our policy estimate via gradient ascent along the value function $J(x;\phi)$ at the current state $X^{\phi _t}_t$:

\[\phi \leftarrow \phi + \alpha \frac{\partial J}{\partial \phi}(X_{t}^{\phi};\phi ).\]

This is the policy improvement (or policy gradient (PG)) step.

Regarding estimation in value space, various gradient-based PE algorithms can be found within the RL literature, in both the offline and online setting, which all update the critic parameter $\theta $ via (stochastic) gradient descent along some target estimate $\widetilde{J}(x)$ for the value function $J(x)$:

\[\begin{align}\label{SGDPE} \theta &\leftarrow \theta -\frac{1}{2} \alpha \frac{\partial }{\partial \theta } (\widetilde{J}(X^{\phi }_t)-J^{\theta}(X^{\phi }_t))^2\\ & =\theta + \alpha \left( \widetilde{J}(X^{\phi }_t)-J^{\theta}(X^{\phi }_t) \right) \frac{\partial J}{\partial \theta } (X^{\phi }_t) .\end{align}\]

This defines gradient-based methods. If $\widetilde{J}(x)$ is an unbiased estimate of $J(x;\phi )$, then the above stochastic approximation algorithm has guaranteed convergence to a local optimum, provided the weights $\alpha$ satisfy some standard growth conditions. A popular choice of offline algorithm is taking \(\widetilde{J}(X_t^{\phi })=G_t \coloneqq \sum\limits_{k=t}^{T} r(X_t^{\phi }, A^{\phi }_t)\) the return from time $t$ onwards, which defines the (gradient) Monte Carlo algorithm. There are also popular choices for the target estimate which themselves depend on the current critic parameter $\widetilde{J}^{\theta}(x)$. For example, we can take the temporal difference (TD(0)) target \(\widetilde{J}(X_t^{\phi })=r(X_{t}^{\phi },A^{\phi }_t)+J^{\theta}(X_{t+1}^{\phi})\). If we use as our update step the true gradient of the of the squared error, we get the gradient TD(0) (GTD(0)) algorithm. Often times however it is useful to still use the second line in Equation \ref{SGDPE} to update $\theta $, ignoring the effect of changing $\theta $ on the target $\widetilde{J}^{\theta }(x)$, as this leads to faster learning. Since this corresponds to a stochastic approximation learning algorithm which is not minimization of the gradient of a loss function, they are consequently called semi-gradient methods of PE.

The PG step boils down to estimating the gradient $\frac{\partial J}{\partial \phi } (x;\phi )$ of the critic. In order to develop approximation algorithms, we must first characterize this gradient in a way that is amenable to estimation via observed data. Recalling the definition of the value function, taking the (iterated) conditional expectation with respect to the first action $A_0^{\phi}$, we have

\[J(x_0;\phi )= \sum\limits_{a_0 \in \mathcal{A}}^{} \pi^{\phi }(a_0|x_0)Q^{\phi }(x_0,a_0)\]

where \(Q(x,a;\phi )\coloneqq\E_{x,a}\left[ \sum\limits_{t=0}^{T} \beta ^{t}r(X^{\phi }_t,A^{\phi }_t) \right]\) is the $Q$-function associated to $\pi^{\phi}$. Taking the gradient with respect to $\phi $ yields

\[\frac{\partial J}{\partial \phi } (x_0;\phi ) = \sum\limits_{a_0 \in \mathcal{A}}^{} \frac{\partial \pi^{\phi }}{\partial \phi } (a_0|x_0) Q(x_0,a_0;\phi ) + \pi^{\phi }(a_0|x_0)\frac{\partial Q}{\partial \phi } (x_0,a_0;\phi ).\]

We consequently unroll the $Q$-function one step forward in time by conditioning on $X^{\phi }_1$, using the Markovian structure of $(X^{\phi }_t,A_t^{\phi })$:

\[Q(x_0,a_0;\phi )=\E[r(x_0,a_0)] + \beta \sum\limits_{x_1 \in \mathcal{X}}^{} p(x_1|x_0,a_0)J(x_1;\phi )\]

where implicitly we note $J(x_1;\phi)$ is a summation up to $T-1$. Substituting this into the previous equation yields

\[\begin{align*} \frac{\partial J}{\partial \phi } (x_0;\phi ) = \sum\limits_{a_0 \in \mathcal{A}}^{} \frac{\partial \pi^{\phi }}{\partial \phi } (a_0|x_0) Q(x_0,a_0;\phi ) + \beta\pi^{\phi }(a_0|x_0) \sum\limits_{x_1 \in \mathcal{X}}^{}p(x_1|x_0,a_0) \frac{\partial J}{\partial \phi } (x_1;\phi ) .\end{align*}\]

Iterating this procedure, we obtain

\[\frac{\partial J}{\partial \phi } (x_0;\phi ) = \sum\limits_{x \in \mathcal{X}}^{} \rho ^{\phi }_{x_0}(x) \underbrace{\sum\limits_{a \in \mathcal{A}}^{} \frac{\partial \pi^{\phi }}{\partial \phi } (a|x)Q(x,a;\phi )}_{\eqqcolon \ell (x)}\]

where \(\rho^{\phi }_{x_0}(x)\coloneqq\sum\limits_{t=0}^{T} \beta ^{t}\mathbf{P}_{x_0}(X^{\phi }_t=x)\) is the (discounted) occupation time at state $x$. The current formulation of the gradient cannot be efficiently estimated via a single sample $(X^{\phi }_t,A^{\phi }_t)$, as it requires estimating the gradient of the policy and associated $Q$-function over all state-action pairs. To overcome this, we rewrite the gradient as

\[\begin{align*} \frac{\partial J}{\partial \phi } (x_0;\phi ) &= \sum\limits_{x \in \mathcal{X}}^{} \E_{x_0}\left[ \sum\limits_{t=0}^{T} \beta ^{t}\bm{1}_{\{X^{\phi }_t=x\} } \right] \ell (x)\\ &= \E_{x_0} \left[\sum\limits_{t=0}^{T} \beta ^{t} \sum\limits_{x \in \mathcal{X}}^{}\bm{1}_{\{X^{\phi }_t=x\} }\ell (X^{\phi }_t) \right] \\ &= \E_{x_0} \left[ \sum\limits_{t=0}^{T} \beta ^{t} \ell (X^{\phi }_t) \right] \\ &= \E_{x_0} \left[ \sum\limits_{t=0}^{T} \beta ^{t} \sum\limits_{a \in \mathcal{A}}^{} \pi^{\phi }(a|X^{\phi }_t)\frac{\partial \log \pi^{\phi }}{\partial \phi } (a|X^{\phi }_t) Q(X^{\phi }_t,a;\phi ) \right] \\ &= \E_{x_0}\left[ \sum\limits_{t=0}^{T} \beta ^{t} \E\left[ \frac{\partial \log \pi^{\phi }}{\partial \phi } (A^{\phi }_t|X^{\phi }_t)Q(X^{\phi}_t,A^{\phi }_t;\phi )|X^{\phi }_t \right] \right] \\ &= \E_{x_0}\left[ \sum\limits_{t=0}^{T} \beta ^{t} \frac{\partial \log \pi^{\phi }}{\partial \phi } (A^{\phi }_t|X^{\phi }_t)Q(X^{\phi}_t,A^{\phi }_t;\phi ) \right] \\ &= \E_{x_0}\left[ \sum\limits_{t=0}^{T} \beta ^{t} \frac{\partial \log \pi^{\phi }}{\partial \phi } (A^{\phi }_t|X^{\phi }_t)\left(Q(X^{\phi}_t,A^{\phi }_t;\phi ) -B(X^{\phi })\right)\right] \end{align*}\]

where in the last line $B(x)$ is an arbitrary function of the state, called the baseline, and we have made use of the fact that \(\sum\limits_{i=1}^{N_\phi } \partial \log \pi^{\phi } / \partial \phi =0\). Thus, we have represented the gradient of the critic as an expectation with respect to a given sample \(\{(X^{\phi }_t,A^{\phi }_t)\}_{t=0}^{T}\). This formulation of the policy gradient lends itself to both offline and online algorithms, where in the latter case at time $t$, we update $\phi $ using only the state-action pair \((X_t^{\phi },A^{\phi }_t)\), rather than the entire trajectory.

Of course, in place of the true $Q$-function, we use the estimate $Q^{\theta }(x,a)$ from our PE step. If in place of $Q(X^{\phi }_t,A^{\phi }_t;\phi )$, we instead had the return $G^{\phi }(X^{\phi }_t,A^{\phi }_t)$ from time $t$ onwards, we would get the actor-only REINFORCE with baseline algorithm.

The benefit of including a baseline is that we get an estimator for the gradient of the critic which remains unbiased but with potentially smaller variance. The most common choice of baseline is $B(x)=J^{\theta }(x;\phi _t)$, the estimate of the value function for the current policy, yielding the advantage AC (A2C) algorithm. In this setting, the policy parameter intuitively updates $\phi$ in the direction which maximizes the log-likelihood of the current policy with respect to the current action $A^{\phi}_t$, with the step size scaled by the advantage of that action, the latter defined by

\[\mathit{Ad}^{\theta }(X^{\phi }_t,A^{\phi }_t;\phi)\coloneqq Q^{\theta }(X^{\phi}_t,A^{\phi };\phi) - J^{\theta }(X^{\phi }_t;\phi ) .\]

2. Stochastic control meets RL

2.1. Exploration: relaxed stochastic control

Stochastic control (SC) and reinforcement learning (RL) are both methods for optimizing sequential decision making under uncertainty. The idea is that an agent is able to interact with their environment (state) via a series of actions (controls). Where SC and RL diverge is that the former is model-based, i.e. it presumes one has perfect knowledge of the dynamics by which the environment (stochastically) evolves, and consequently seeks to obtain existence/uniqueness results for an optimal feedback control defined in terms of a deterministic policy of the state. In contrast, RL begins from a starting point of incomplete information, where an agent only learns about the (unknown) dynamics of their environment via their interaction with it, generating realization(s) from which the agent can learn an (approximately) optimal policy. In order to achieve this, the agent chooses actions both to optimize their reward (as in SC) and to explore their environment. It is this latter condition which necessitates actions being sampled randomly according to a policy (distribution), independently of the randomness of the environment. The correct way to include randomness in control into the SC setting is via the theory of relaxed stochastic control.

Let’s begin with classical the classical setup. We fix the canonical path space $\Omega=C([0,T], \mathbf{R})$ equipped with the Borel $\sigma$-algebra $\mathcal{B}$ induced by the topology of uniform convergence, and the Wiener measure $\mathbf{P}_W$. Let $\mathcal{F}^{W}_t$ denote the associated filtration generated by $W _t$. Consider a state process $X_t^{a}$ which evolves via an Ito diffusion

\[dX^{A}_t = b(t,X_t,A_t)dt + \sigma(t,X_t,A_t)dW_t.\]

For ease of notation, let’s assume every process in sight is one-dimensional for simplicity. Here $A_t$ is our control process. As in the discrete RL setting, we seek to maximize the value function

\[J^{A}(t,x)\coloneqq \E_{t,x}^{\mathbf{P}^{W}}\left[ \int_{t}^{T} e^{-\beta (s-t)}r(s,X^{a}_s,a_s)\,ds +e^{-\beta (T-t)} h(X^{a}_T) \right]\]

That is, $J^A$ is given by the expected discounted running and terminal rewards $r(t,x,a)$ and $h(x)$ over a finite time horizon $T>0$. We seek a function $a(t,x)$ so that the value function $J^{\ast}(t,x)$ of the feedback policy $A^{\ast}_t = a(t,X^{a^{\ast}}_t)$ satisfies

\[J^{\ast}(t,x)\coloneqq J^{A^{\ast}}(t,x) = \sup _{A_t \in \mathcal{A}}J^{A}(t,x)\]

To ensure the classical stochastic control problem is well-posed, we assume the following standard conditions.

Assumptions 1.

1. The functions $b,\sigma, r,h$ are continuous in their respective arguments.
2. The coefficients $b,\sigma$ are uniformly Lipschitz continuous with linear growth in $x$.
3. $r$ and $h$ have polynomial growth in $(x,a)$ and $x$ respectively.

The first assumption is a convenience so we remain on the canonical path space of continuous functions indexed by $[0,T]$. The second assumption guarantees strong existence and uniqueness of $X^{A}$, together with a bound \(\E\sup_{t \in [0,T]}\lvert X_t\rvert^{\mu } \leq C\E\lvert X_0 \rvert^\mu\) for any $\mu \geq 2$. The third assumption ensures the value function is finite at all times.

Translating this into the RL paradigm amounts to injecting randomness into our action process. To do so, we extend our probability space via $\mathbf{P}=\mathbf{P}_W\times \mathbf{P}_Z$, where $Z\sim \mathrm{Unif}([0,1])$ governs the exploration of our actions. Namely, we now consider (absolutely-continuous) distribution-valued controls $\pi_t \in \euscr{P}(A)$, aka policies, which are moreover Markovian in the sense that they have the feedback form $\pi_t(da)=\pi(da\mid t,x)$. We then consider a randomized action process $A^{\pi}_t \sim \pi_t$ sampled according to our policy. We can formalize this for example by defining

\[A^{\pi}_t \coloneqq Q_{\pi(dA\mid t,X^{\pi}_t)}(Z)\]

where $Q$ denotes the quantile function of $\pi(da \mid t,X^{\pi}_t)$, and $X^{\pi}(t)$ is the state process

\[dX^{\pi}_t = b(t,X^{\pi}_t,A^{\pi}_t)dt + \sigma(t,X^{\pi}_t,A^{\pi}_t)dW_t .\]

Of course, $a^{\pi}$ is defined recursively in terms of the state $X^{\pi}$, so we must guarantee that this SDE (with randomized coefficients given the dependence on $Z$) exists, which requires some reasonable assumptions on the policy $\pi$. Now that we have randomized actions and state dynamics, we can formulate the objective of our continuous RL problem, which is simply to optimize the exploration-regularized value function

\[J^{\pi}(t,x) \coloneqq \E_{t,x}^{\mathbf{P}}\left[ \int_{t}^{T} e^{-\beta (s-t)}(r(s,X^{\pi}_s,A^{\pi}_s) +\gamma H(\pi(da|t,X^{\pi}_s))) \,ds +e^{-\beta (T-t)} h(X^{\pi}_T) \right]\]

where we have introduced the differential entropy regularization

\[H(\pi)\coloneqq -\int_{a \in \mathcal{A}} \pi(a)\log(\pi(a)) \,da\]

with temperature parameter $\gamma \geq 0$ to encourage exploration. Because we will use it repeatedly, for convenience we abbreviate the non-discounted integrand

\[F(t,x,a)\coloneqq r(t,x,a) + \gamma H(\pi(a|t,x)).\]

While we have setup a reasonable model for continuous RL, from a theoretical standpoint, it is kind of annoying that we are working with controls $A_t^{\pi}$ randomized exogenously to our state noise $W_t$. This introduces questions about exactly which tools from classical stochastic control we can apply to our setting. In order to resolve such technical points, we would like to integrate out the policy randomization from our state process. The key observation is that, conditioned on the state $X_t^{\pi}$, the drift and variance processes satisfy

\[\overline{b}(t,X^{\pi}_t)\coloneqq \E[b(t,X^{\pi}_t,A^{\pi}_t) |X_t^{\pi}]= \int_{\mathcal{A}}^{} b(t,X^{\pi}_t,a)\pi(a|t,X^{\pi}_t)\,da\]

and

\[\overline{\sigma}^2(t,X^{\pi}) = \int_{\mathcal{A}}^{} \sigma^2(t,X^{\pi}_t,a)\pi(a|t,X^{\pi}_t)\,da .\]

This motivates the relaxed control state process $\overline{X}_t^{\pi}$ characterized by the SDE

\[d \overline{X}_t^{\pi} = \overline{b}(t,\overline{X}_t^{\pi})dt + \overline{\sigma}(t,\overline{X}_t)dW_t\]

where

\[\overline{b}(t,x) \coloneqq \int_{A}^{} b(t,x,a)\pi(da \mid t,x) \hspace{10pt} \text{and}\hspace{10pt} \overline{\sigma}(t,x)\coloneqq\sqrt{\int_{A}^{} \sigma^2(t,x,a)\pi(da \mid t,x)} .\]

Thus, \(\overline{X}_t^{\pi}\) represents the environment dynamics where we have averaged out the policy exploration \(\pi_{t}\). While \(X^{\pi}_t\) can be observed by sampling the policy \(A^{\pi}_t \sim \pi(da\mid t,X^{\pi}_t)\), \(\overline{X}^{\pi}_t\) is unobservable. The authors motivate the definition of \(\overline{X}_t^{\pi}\) via a law of large numbers argument which was formalized in (Jia et al., 2025). In fact it is somewhat nontrivial to show that the two processes agree in law. That the distribution of the 1-dimensional marginals of \(X_s^{\pi}\) and \(\overline{X}_s^{\pi}\) coincide is an immediate consequence of Corollary 4.7 of the ``Markovian projection formula” of (Brunick & Shreve, 2013). In particular, this implies by Fubini’s formula that the value function of the exploratory control coincides with the value function of the relaxed control

\[J^{\pi}(t,x) = \E^{\mathbf{P}^{W}}_{t,x}\left[ \int_{t}^{T} e^{-\beta (s-t)}\int_{A}^{} F(t,\overline{X}^{\pi}_{t},a,\pi)\pi(a|t,\overline{X}^{\pi}_t)\,dadt + e^{-\beta (T-t)}h(\overline{X}^{\pi}_T) \right] .\]

Again, let’s denote by $\overline{F}(t,x) \coloneqq \int_{A}^{} F(t,x,a,\pi)\pi(a \mid t,x)\,da$ the associated non-discounted integrand.

To finish the setup, we restrict the policies we consider to some admissible space $\pi \in \mathbf{A}\subset \euscr{P}(\mathcal{A})^{[0,T] \times \mathbf{R}}$ to ensure the relaxed control problem is well-posed. Namely, under some reasonable Lipschitz continuity and polynomial growth conditions similar to Assumption 1, we can guarantee the relaxed control state SDE admits a unique strong solution initialized at $x\in \mathbf{R}$ which also satisfies a polynomial growth condition:

\[\begin{equation}\label{eqn:polygrowth} \E^{\mathbf{P}^{W}}[\sup _{0 \leq t \leq T}|\overline{X}^{\pi}_t|^{\mu }] \leq C(1+|x| ^{\mu }). \end{equation}\]

2.2. Evaluation: martingality conditions

The first stage in the actor-critic algorithm is policy evaluation. Namely, suppose we have an admissible policy $\pi^{\phi}$ whose value function $J(t,x)\coloneqq J(t,x;\phi )$ we would like to approximate $J^{\theta}(t,x)$. In order to devise algorithms for estimating the critic $J^{\theta }(t,x)$, we need useful characterizations of the value function. We have two, both of which follow from the observation that the process defined at times $[t,T]$ by

\[\begin{align} \widetilde{M}_s &\coloneqq e^{-\beta s}J(s,x) + \int_{t}^{s} e^{-\beta u} \widetilde{F}(u,\widetilde{X}_u)\,du\label{eqn:martingale}\\ &= \widetilde{\mathbf{E}}_{s,x} \left[ \int_{t}^T e^{-\beta u} \widetilde{F}(u,\widetilde{X}_u)du + e^{-\beta T}h(\widetilde{X}_T) \right] \end{align}\]

is an $(\mathcal{F}^{\overline{X}}_s, \mathbf{P}^{W})$-martingale in the case $\widetilde{X}=\overline{X}$, $\widetilde{F}=\overline{F}$, and $\widetilde{\mathbf{E}}=\mathbf{E}^{\mathbf{P}^W}$, and a $(\mathcal{F}^{X^{\pi}}_s, \mathbf{P})$-martingale in the case $\widetilde{X}=X^{\pi}$, $\widetilde{F}=F$, and $\widetilde{\mathbf{E}}=\mathbf{E}^\mathbf{P}$.

Theorem 1 (Martingality characterization of the value function).

Let $J^{\theta }\colon [0,T] \times \mathbf{R} \to \mathbf{R}$ be a $C^{1,2}$ function such that $J^{\theta }(T,x)=h(x)$. Define $M^{\theta }_t$ as in Equation \ref{eqn:martingale} with $J$ replaced by $J^{\theta}$. The following are equivalent:

1. $J^{\theta }=J$ is the value function.

2. For any initialization $(t,x)\in [0,T) \times \mathbf{R}^{d}$, the process

   \[\overline{M}^{\theta }_s \coloneqq e^{-\beta s}J^{\theta }(s,\overline{X}^{\pi}_s) + \int_{t}^{s} e^{-\beta s}\overline{F}(u,\overline{X}^{\pi}_u)\,du\]

   is a $(\mathcal{F}_{t}^{\overline{X}^{\pi}},\mathbf{P}^{W})$-martingale on $[t,T]$.

3. (Martingale orthogonality). For every $(\mathcal{F}_t^{X^{\pi}}, \mathbf{P})$-progressively measurable bounded process $\xi_t$,

   \[\begin{equation}\label{eqn:mo}\E^{\mathbf{P}}\int_{0}^{T} \xi_t\,dM^{\theta }_t=0.\end{equation}\]

Proof.

That the true value function satisfies both (2) and (3) is immediate from Ito's formula. To see (2) implies (1), we note that $$\begin{align*} e^{-\beta t}J^\theta(t,x)&= \overline{M}_t^\theta \\ &= \E^{\mathbf{P}^{W}}[\overline{M}_T^\theta|\mathcal{F}^{\overline{X}^{\pi}}_t]\\ &= \E^{\mathbf{P}^{W}}[\overline{M}_T|\overline{X}^{\pi}_t=x] \\ &= e^{-\beta t}J(t,x) \end{align*}$$ where we used the Markov property for $\overline{X}^{\pi}_t$ in line 3 together with the Martingale property for $\overline{M}_s$. We will now show (3) is equivalent to (2). We first show that (2) is equivalent to (3'), where we replace $M^\theta_t$ in the martingale orthogonality condition with $\overline{M}^\theta_t$. Firstly note that $$ dM^{\theta }_t = dJ^{\theta }(t,\overline{X}^{\pi}_t)+\overline{F}(t,\overline{X}^{\pi}_t)dt-\beta J^{\theta}(t,\overline{X}^{\pi}_t)dt. $$ It is clear that if $\overline{M}^{\theta }_t$ is a martingale then integrating any bounded test process $\xi_t^\prime=\xi_t e^{\beta t}$ against $\overline{M}^{\theta }_t$ yields a mean zero martingale. For the converse, by It\^o's lemma, $\overline{M}^{\theta }_t$ is a diffusion, hence $\overline{M}^{\theta }_t=\int_{0}^{t} \widetilde{b}_s\,ds + \int_{0}^{t} \widetilde{\sigma}_s\,dW_s$ for some $\euscr{F}^{X^{\pi}}$-adapted processes $\widetilde{b}_t, \widetilde{\sigma}_t$. It follows that $$ \E \int_{0}^{T} \xi_t \widetilde{b}_t\,dt=0 $$ for an arbitrary $\euscr{F}_t^{\overline{X}^{\pi}}$-adapted finite variation process $\xi_t$, thus by denseness we conclude $\widetilde{b}_t=0$ and thus $M^{\theta }_t$ is a $(\euscr{F}^{X^{\pi}}_t,\mathbf{P})$-martingale. Now to show that this completes the proof, note that any $(\euscr{F}^{X^{\pi}}_t,\mathbf{P})$-progressively measurable process $\xi_t$ takes the form $\xi_t \coloneqq \xi(t,X_{t \wedge \bullet }^{\pi})$ for some measurable map $\xi\colon [0,T] \times C([0,T], \mathbf{R})\to \mathbf{R}$. Defining $\overline{\xi}_t\coloneqq \xi(t,\overline{X}^{\pi}_{t \wedge \bullet } )$, we observe $$ \E^{\mathbf{P}}\left[ \int_{0}^{T} \xi_t\,dM^{\theta }_t \right] = \E^{\mathbf{P}^{W}}\left[ \int_{0}^{T} \overline{\xi}_te^{\beta t}\,d\overline{M}^{\theta }_t \right] $$ from which the equivalence of (3) and (3') is immediate.

This theorem allows us to generalize various PE algorithms from discrete to continuous time.

1. Using the 1st martingality characterization of the value function, we can choose $\theta \in \Theta $ via minimization of the mean-square error

   \[\begin{align*} \mathrm{ML}(\theta )&\coloneqq \|\overline{M}^{\theta }_{T}-\overline{M}^{\theta}_\bullet\|_{\mathbf{L}^2(\mathcal{F}^{\mathbf{X}^{\pi}}_t)}^2\\ & = \E^{\mathbf{P}}\left[ \int_{0}^{T}\left( e^{-\beta T}h(X_T^{\pi})-e^{-\beta t}J^{\theta }(t,X^{\pi}_{t}) + \int_{t}^{T} e^{-\beta s}F(s,X^{\pi}_s,a^{\pi}_s)\,ds\right)^2\,dt \right] \end{align*}\]

   via SGD. The function $\mathrm{ML}(\theta )$ is known as the martingale loss function and can be seen as a continuous-time analogue of gradient Monte Carlo.

2. Using the martingale orthogonality condition with test process $\xi_t = \frac{\partial J^{\theta }}{\partial \theta }(t,X^{\pi}_t)$, via stochastic approximation the corresponding update step is

   \[\begin{align*} \frac{\partial J^{\theta }}{\partial \theta}(t,X^{\pi}_t) dM^{\theta }_t &\approx ( J^{\theta }(t+\Delta t,X^{\pi}_{t+\Delta t})- J^{\theta }(t,X^{\pi}_{t})\\ & + (r(t,X^{\pi}_{t},A^{\pi}_{t}) + \gamma H(\pi(A^{\pi}_t|t,X^{\pi}_t)) - \beta J^\theta(t, X^\pi_t)) \Delta t) \frac{\partial J^{\theta }}{\partial \theta} (t,X^{\pi}_t) \end{align*}\]

   hence we obtain an analogue of the semi-gradient TD(0) algorithm.

3. Again using the martingale orthogonality condition with test process $\xi_t=\frac{\partial J^{\theta }}{\partial \theta } (t,X^{\pi}_t)$, but now choose $\theta \in \Theta $ via minimization of the generalized methods of moments (GMM) loss functional

   \[\frac{1}{2} \left( \frac{\partial J^{\theta }}{\partial \theta }(t,X^{\theta }_t) \Delta M^{\theta }_t\right)^2\]

   via SGD, which corresponds to GTD(0).

2.3. Improvement: PG (PE in disguise)

Having carried out the policy evaluation step to obtain the critic $J^{\theta }(t,x;\phi)$ estimating the true value function $J(t,x;\phi )$, we now look towards improving our estimation of the current policy via the policy gradient step. Let $g(t,x;\phi )\coloneqq \frac{\partial }{\partial \phi } J(t,x;\phi)$ denote the policy gradient. Provided $J$ is sufficiently regular (i.e. $C^{1,2}$), it is characterized as the unique solution of the Feynman-Kac formula

\[\begin{cases} \int_{A}^{} (\mathcal{L}^{a}J(t,x;\phi ) + r(t,x,a)+ \gamma H(\pi^{\phi }(a|t,x)) - \beta J) \pi(a|t,x)\,da =0\\ J(T,x;\pi^{\phi })=h(x) \end{cases}\]

and moreover, our assumptions on $b, \sigma, \pi$ imply $J(t,x;\phi )$ has polynomial growth in $x$. Differentiating in $\phi$ gives a new system of $N_{\phi }$ equations characterizing the policy gradient

\[\begin{cases} \int_{A}^{}(\mathcal{L}^{a}g(t,x;\phi ) - \beta g(t,x;\phi ) + \widetilde{r}(t,x,a;\phi )) \pi^{\phi }(a|t,x) da=0\\ g(T,x;\pi^{\phi })=0 \end{cases}\]

where

\[\widetilde{r}(t,x,a;\phi )\coloneqq \gamma q(t,x,a) + (\mathcal{L}^{a}J(t,x;\phi ) + r(t,x,a)+ \gamma H(\pi^\phi(a|t,x)) - \beta J(t,x;\phi))\frac{\partial }{\partial \phi }\log \pi(a|t,x).\]

and $q(t,x,a)=\frac{\partial H \circ \pi}{\partial \phi}(t,x,a) $. Thus, Feynman-Kac implies that $g^{\phi}$ is uniquely characterized as the conditional expectation

\[g(t,x;\phi )=\E_{t,x}^{\mathbf{P}}\left[ \int_{t}^{T}e^{-\beta (s-t)} \widetilde{r}(s,X^{\pi}_s,A^{\pi}_s;\phi )\,ds \right]\]

i.e., $g$ is a value function with respect to a new reward $\widetilde{r}$, and hence estimating the policy gradient reduces to policy evaluation. In fact, this is easier then general PE as we only need to estimate the ``value function” $g(t,x;\phi )$ at the current state $X^{\phi }_t$, rather than estimating the entire function.

We are not done however, as our reward $\widetilde{r}$ is defined in terms of the generator $\mathcal{L}^{a}J$, which is not observable and so we must re-characterize the policy gradient in a way which is amenable to learning from the environment. As always, the approach is use Ito’s lemma together with martingality. Namely, we have

\[d J(t,X^{a}_t;\phi ) = \mathcal{L}^{a}J(t,X^{a}_t) dt +d N_t^{a}\]

for some $(\mathcal{F}_t^{W},\mathbf{P}^{W})$-local martingale $N^{a}_t$. Thus, we can use a standard localization argument to get a true martingale which vanishes upon taking expectations provided we have sufficient regularity to pass to the limit. The following assumptions guarantee that such a technique will be possible.

Assumptions 2.

1. For all $\phi \in \Phi$, $\int_{\mathcal{A}}^{} \mid\widetilde{r}(t,x,a;\phi )\mid \pi^{\phi }(a\mid t,x)da$ is of polynomial growth in $x$ uniformly in $t$.
2. For all $\phi \in \Phi$, $\int_{\mathcal{A}}^{} \left\lvert\frac{\partial }{\partial \phi } \log \pi^{\phi }(a\mid t,x)\right\rvert^2 \pi^{\phi }(a \mid t,x) \,da$ is continuous in $(t,x)$.

The above was intuition for the argument we will use, replacing $X^{a}_t$ with the true exploratory state process $X^{\phi }_t$.

Theorem 2 (Representation of policy gradient).

The policy gradient is given by

\[\begin{align}\label{eqn:pg} g(t,x;\phi ) = \E^{\mathbf{P}}_{t,x}\left[ \int_{t}^{T} e^{-\beta (s-t)} \left(\frac{\partial }{\partial \phi } \log \pi^{\phi }(A^{\pi}_s|s,X^{\phi }_s)e^{\beta s}dM_s^{\phi} + \gamma q(s,X^{\phi }_s,A^{\phi}_s)ds\right) \right] \end{align}\]

where $M_{t}^{\phi}$ is defined as in Equation \ref{eqn:martingale} with $\pi=\pi^{\phi}.$

Proof.

Define the exit times $\tau _{n}\coloneqq \inf \{t:|X^{\phi }_{t}| \geq n \} .$ Applying Ito's lemma to $J(s,X^{\phi }_s)$, we have $$ \begin{align*} & \int_{t}^{T \wedge \tau _n} e^{-\beta (s-t)} \left(\frac{\partial }{\partial \phi } \log \pi^{\phi }(A^{\pi}_s|s,X^{\phi }_s)e^{\beta s}dM_s^{\phi} + \gamma q(s,X^{\phi }_s,A^{\phi}_s)ds\right) \\ &= \int_{t}^{T \wedge \tau _n} e^{-\beta (s-t)} \left\{ \frac{\partial }{\partial \phi } \log \pi^{\phi }(A^{\phi }_s | s, X^{\phi }_s) \left[\left(\mathcal{L}^{A^{\pi}_s}J(s,X^{\phi }_s) - \beta J(s,X^{\phi }_s) - J(s,X^{\phi }_s) + F(t,X^{\phi }_t,A^{\phi }_t)\right)ds\right.\right.\\ &\left.\left. + \frac{\partial J}{\partial x} (s,X^{\phi }_s) \sigma(s,X^{\phi }_s,A^{\phi }_s) dW_s\right] + \gamma q(s,X^{\phi }_s,A^{\phi }_s)ds \right\} .\end{align*} $$ We claim the Ito integral is a true $(\mathcal{F}_t, \mathbf{P})$-martingale and hence its expectation vanishes. For $s \in [t, T \wedge \tau _n]$, we have the bound $$ \begin{align*} &\left| \int_{\mathcal{A}}\left(\frac{\partial }{\partial \phi } \log \pi^{\phi }(a|s,X^{\phi })_s \frac{\partial J}{\partial x} (s,X^{\phi }_s) \sigma(s,X^{\phi }_s,a)\right)\pi(a|s,X^{\phi }_s) da\right|^2 \\ & \leq \int_{\mathcal{A}}\left|\frac{\partial }{\partial \phi } \log \pi^{\phi }(a|s,X^{\phi }_s) \frac{\partial J}{\partial x} (s,X^{\phi }_s) \sigma(s,X^{\phi }_s,a)\right|^2\pi(a|s,X^{\phi }_s) da \tag{Jensen} \\ & \leq C\int_{\mathcal{A}}\left|\frac{\partial }{\partial \phi } \log \pi^{\phi }(a|s,X^{\phi }_s) \frac{\partial J}{\partial x} (s,X^{\phi }_s)\right|^2\pi(a|s,X^{\phi }_s) da \tag{linear growth in $\sigma$} \\ & \leq C \int_{\mathcal{A}}\left|\frac{\partial }{\partial \phi } \log \pi^{\phi }(a|s,X^{\phi }_s)\right|^2\pi(a|s,X^{\phi }_s) da \tag{continuity of $\frac{\partial J}{\partial \theta}$} \\ & \leq C \tag{Assumption 3} \end{align*} $$ where $C=C(n,\phi,T)$. Thus, taking expectations above and comparing terms, we see $$ \begin{align*} &\E^{\mathbf{P}}_{t,x}\left[\int_{t}^{T \wedge \tau _n} e^{-\beta (s-t)} \left(\frac{\partial }{\partial \phi } \log \pi^{\phi }(A^{\pi}_s|s,X^{\phi }_s)e^{\beta s}dM_s^{\phi} + \gamma q(s,X^{\phi }_s,A^{\phi}_s)ds\right)\right]\\ &= \E^{\mathbf{P}}_{t,x} \left[ \int_{t}^{T \wedge \tau _n} e^{-\beta (s-t)}\widetilde{r}(s,X^{\phi }_s,A^{\phi }_s;\phi )\,ds \right] .\end{align*} $$ Finally, it remains justifying applying dominated convergence to take the limit $n \to \infty$. We have $$ \begin{align*} &\E^{\mathbf{P}^{W}}_{t,x}\left[\int_{t}^{T} e^{-\beta (s-t)}\int_{\mathcal{A}}^{} |\widetilde{r}(s,X^{\phi }_s,a;\phi )|\pi(a|s,X^{\phi }_s)\,dadt\right] \\ & \leq C\E^{\mathbf{P}^{W}}\left[1+\sup _{t \leq s \leq T}|X^{\phi }_s|^{\mu }|\right] \tag{Assumption 2}\\ & \leq C (1+|x|^{\mu }) \tag{Equation \ref{eqn:polygrowth}} \end{align*} $$

There are two takeaways from Theorem 2. Firstly, the first integrand in the policy gradient is precisely the log-likelihood of the current estimated policy scaled by the (continuous) TD error $dM^{\phi }_s$ of the true value function $J(t,x;\phi)$. Secondly, in order to estimate the policy gradient at time $t$ via Theorem 2, we require the entire sample trajectory \((X_s^{\phi }, A^{\phi }_s)_{s \in [0,T]}\), hence we can only use this to produce offline algorithms. To derive online PG algorithms, we proceed analogously as in the case of PE. Namely, we would like to a martingale orthogonality condition to serve as a strong necessary condition for the policy gradient. We can achieve this for the policy gradient at an optimal policy \(\pi^{\ast}\coloneqq \pi^{\phi ^{\ast}}\) occurring at an interior point \(\phi ^{\ast}\in \Phi \setminus \partial\Phi\). That is, assume $J(t,x;\phi^{\ast})$ is maximized at $\phi ^{\ast}$ for any $(x,t)$. Then the first-order-condition $g(0,x;\phi ^{\ast})=0$ combined with Feynman-Kac implies

\[\begin{align*} 0&= \int_{\mathcal{A}}^{} \widetilde{r}(t,X^{\ast}_s,a;\phi ^{\ast})\pi^{\phi ^{\ast}}(a|t,x)da .\end{align*}\]

From this, together with the fact that we showed the local-martingale $N^{a}_t$ is a true martingale, we obtain the following in exactly the same way as Theorem 1.

Theorem 3 (Martingale orthogonality for policy gradient).

For every $(\mathcal{F}_t^{X^{\ast}}, \mathbf{P})$-progressively measurable bounded process $\eta$ and $x \in \mathbf{R}$, we have

\[\begin{equation}\label{eqn:mo2} 0 = \E^{\mathbf{P}}_{x}\left[ \int_{0}^{T} \eta _s \left( \frac{\partial }{\partial \phi } \log \pi^{\ast}(A^{\ast}_s|s,X^{\ast}_s)d M^{\ast}_s + \gamma q(s,X^{\ast}_s,A^{\ast}_s)ds \right) \right] .\end{equation}\]

We use Theorems 2 and 3 in conjunction with Theorem 1 to design offline and online AC algorithms respectively.

3. AC algorithms

As in the discrete-time MDP setting, our AC algorithms consist of successive applications of policy evaluation and policy gradient. Having carried out the theoretical analysis of PE and PG in continuous time, we now discretize over a finite-sample grid $t_k=k\Delta t$ for $0 \leq k \leq \left\lfloor{T / \Delta t }\right\rfloor \eqqcolon K$.

We begin with an example of an offline algorithm. This is inherently episodic, with critic-actor parameters $(\theta , \phi )$ updated after a full trajectory (i.e. episode) \((X^{\pi}_{t_k}, A^{\pi}_{t_k})\) is sampled. We use the martingale orthogonality condition from Theorem 1 to carry out policy evaluation using a general test process $\xi_t$ which corresponds to different choices of algorithms. The policy improvement step is carried out using a discrete-time estimate of the policy gradient given by Theorem 2.

For the online learning algorithm, we now update the critic using the martingale orthogonality condition after each time step $t_k$, rather than after sampling an entire state-action trajectory. We estimate the policy gradient using Theorem 3, taking a test function $\eta_{t_k}$ which satisfies $\eta_{t_k}=0$ for $t_k > t$ at time $t=t_{l}$, and again updating incrementally after each time step.

3.1 Application to MVO

Finally, as an application of the continuous RL we have developed, let’s apply the AC algorithm to a toy mean-variance portfolio optimization considered in (Wang & Zhou, 2020). Consider a stock price $S_t$ which follows a GBM

\[\frac{dS_t}{S_t} = \mu dt + \sigma dW_t .\]

We assume a risk-free rate $r>0$, a nondegenerate volatility $\sigma>0$, and an initial portfolio balance $x_0$. Letting $A_t$ denote the discounted dollar holdings in the stock at time $t$, the associated discounted PnL of the portfolio evolves via the dynamics

\[d X^{A}_t = A_t[(\mu - r )dt + \sigma dW_t] = a_t \frac{d(e^{-rt}S_t)}{e^{-rt}S_t} .\]

Our goal is to optimize the terminal mean-variance criterion

\[J^{A}(x_0)=\inf_{A} \E[(X^{A}_T-w)^2] - (w-z)^2\]

which minimizes the terminal variance of wealth subject to the constraint $\E[X^{A}_T]=z$, with the latter encoded by the Lagrange multiplier $w$.

We turn this into a relaxed stochastic control problem following the procedure outlined in Section 2.1, obtaining a state $X^{\pi}$ driven by an exploratory control $A_t\sim \pi(da

t,x)$.

We consider the parametric family of value functions of the form

\[J^{\theta }(t,x;w) = (x-w)^2 e^{-\theta _3(T-t)}+\theta _2(t^2-T^2) + \theta _1 (t-T)-(w-z)^2\]

and Gaussian policies

\[\pi^{\phi }(da|t,x;w)= \mathcal{N}(da|-\phi _1(x-w),e^{\phi _2+\phi _3(T-t)}). .\]

As we are now considering a minimization problem, we encourage exploration by penalizing by the differential entropy

\[-H(\pi^{\phi }(da|t,x;w)) = -\frac{1}{2}\left(\log (2\pi e) + \phi _2 + \phi _3(T-t)\right)\]

We set the discount rate $\beta =0$, and the temperature parameter $\gamma=0.1$. For PE, we use the TD(0) algorithm, which requires the critic gradients

\[\frac{\partial J^{\theta }}{\partial \theta _1} (t,x;w)=t-T, \hspace{10pt} \frac{\partial J^{\theta }}{\partial \theta _2} (t,x;w)= t^2-T^2, \hspace{10pt} \frac{\partial J^{\theta }}{\partial \theta _3} (t,x;w) = (x-w)^2 e^{-\theta _3(T-t)}(t-T).\]

For PG, the gradient of the log-likelihood is given by

\[\begin{align*} \frac{\partial \log \pi^{\phi }}{\partial \phi _1} &= - (a+\phi _1(x-w))(x-w)e^{-\phi _2 -\phi _3 (T-t)}\\ \frac{\partial \log \pi^{\phi }}{\partial \phi _2} &= - \frac{1}{2} (1 - (a+\phi _1(x-w))^2 e^{-\phi _2 -\phi _3 (T-t)})\\ \frac{\partial \log \pi^{\phi }}{\partial \phi _3} &= - \frac{T-t}{2}\left(1 - (a+\phi _1(x-w))^2 e^{-\phi _2 -\phi _3 (T-t)}\right) .\end{align*}\]

In addition to the critic and actor parameters, we need to learn the Lagrange multiplier $w$ associated to our target discounted mean wealth $z$. We do this via stochastic approximation

\[w \leftarrow w - \alpha_w \frac{1}{m} \sum\limits_{i=j-m+1}^{j} X^{(i)}_T.\]

We consider $\Delta t = \frac{1}{252}$ daily return data. For the offline algorithm, we simulate 20 years of training data, which we randomly sub-sample in batches of 128 1-year blocks, for a total of $N=2000$ episodes. We set the batch size for updating the Lagrange multiplier to $m=10$ episodes. For the online algorithm, we consider 100 years of training data, and we update our actor and critic after each time step, and our Lagrange multiplier after each episode/year ($m=1$). We repeat the experiment with 100 samples, and report the mean and standard errors of the mean, variance, and Sharpe ratio of terminal discounted wealth $X_T$.

Unsurprisingly, looking at the out-of-sample results in Figure 5, we see the offline algorithm achieves better performance as it is able to learn continuously from the entirety of the training data. Of course, a policy trained via online learning is generally preferable as it is more robust to regime shifts, heteroskedasticity, and other forms of nonstationarity present in financial data. The standard approach to combine the best of both offline and online learning is offline-to-online (O2O) learning (Nair et al., 2020), in which part of the backtesting data is used for offline learning to burn in the model parameters, before transitioning to online learning.

References