Motivated by recent advances of reinforcement learning and direct data-driven control, we propose policy gradient adaptive control (PGAC) for the linear quadratic regulator (LQR), which uses online closed-loop data to improve the control policy while maintaining stability. Our method adaptively updates the policy in feedback by descending the gradient of the LQR cost and is categorized as indirect, when gradients are computed via an estimated model, versus direct, when gradients are derived from data using sample covariance parameterization. Beyond the vanilla gradient, we also showcase the merits of the natural gradient and Gauss-Newton methods for the policy update. Notably, natural gradient descent bridges the indirect and direct PGAC, and the Gauss-Newton method of the indirect PGAC leads to an adaptive version of the celebrated Hewer’s algorithm. To account for the uncertainty from noise, we propose a regularization method for both indirect and direct PGAC. For all the considered PGAC approaches, we show closed-loop stability and convergence of the policy to the optimal LQR gain. Simulations validate our theoretical findings and demonstrate the robustness and computational efficiency of PGAC.

Consider a discrete-time linear system

where \( t\in \mathbb{N}\) , \( x_t\in\mathbb{R}^{n}\) is the state, \( u_t\in\mathbb{R}^{m}\) is the control input, \( w_t \in \mathbb{R}^n\) is the noise, and \( z_t\) is the performance signal of interest. We assume that \( (A,B)\) are controllable and the weighting matrices \( (Q, R)\) are positive definite.

The LQR problem is phrased as finding a state-feedback gain \( K\in \mathbb{R}^{m\times n}\) that minimizes the \( \mathcal{H}_2\) -norm of the transfer function \( \mathscr{T}(K):w \rightarrow z\) of the closed-loop system

When \( A+BK\) is stable, it holds that [36]

where \( \Sigma\) is the closed-loop state covariance matrix obtained as the positive definite solution to the Lyapunov equation

We refer to \( C(K)\) as the LQR cost and to (3)-(4) as a policy parameterization of the LQR.

For known \( (A,B)\) , there are alternative formulations to find the optimal LQR gain \( K^*:=\arg\min_{K}C(K)\) of (3)-(4), e.g., via the celebrated Riccati equation [36]. We recall a well-known iterative algorithm to solve the Riccati equation, called Hewer’s algorithm [33] (also known as policy iteration in reinforcement learning (RL) [7]), which starts from an initial stabilizing policy \( K_0\) and alternates between

where (5.a) is the policy evaluation, and (5.b) is the policy improvement. The Hewer’s algorithm has asymptotic convergence guarantees to the solution of the Riccati equation and the optimal policy \( K^*\) , which are fixed points of (5) [33].

For unknown \( (A,B)\) , there is a plethora of data-driven methods to find \( K^*\) , some of which we recapitulate below.

The conventional approach to data-driven LQR design follows the certainty-equivalence principle: it first identifies nominal system matrices \( (A,B)\) from data, and then solves the LQR problem regarding the identified model as the ground-truth. The identification step is based on subspace relations among the state-space data. Consider a \( t\) -long time series of states, inputs, unknown noises, and successor states

which satisfy the system dynamics

We assume that the data is persistently exciting (PE) [37], i.e., the block matrix of input and state data

has full row rank

which is necessary for data-driven LQR design [38].

Based on the subspace relations (7) and the rank condition (8), an estimated model \( (\hat{A},\hat{B})\) can be obtained as the unique solution to the ordinary least-squares problem

Following the certainty-equivalence principle [26], the system \( (A,B)\) is replaced with its estimate \( (\hat{A},\hat{B})\) in (3)-(4), and the LQR problem can be reformulated as

where \( \Sigma\) is the positive definite solution to

The problem formulation (10)-(11) is termed indirect certainty-equivalence LQR design [27].

Instead of estimating a dynamical model (9), the{ direct data-driven} LQR design aims to find \( K^*\) directly from a batch of PE data [25, 26, 21]. For this purpose, our previous work [21] proposes a policy parameterization based on the sample covariance of input-state data

Under the PE rank condition (8), the sample covariance \( \Phi\) is positive definite, and there exists a unique solution \( V\in \mathbb{R}^{(n+m)\times n}\) to

for any given \( K\) . We refer to (13) as the covariance parameterization of the policy. Note that the dimension of the parameterized policy \( V\) does not scale with data length \( t\) .

For brevity, define the data matrices \( \overline{W}_0= W_0D_0^{\top}/t, \overline{X}_1= X_1D_0^{\top}/t, \) {whose dimensions do not depend on \( t\) .} Then, the closed-loop matrix can be written as

Following the certainty-equivalence principle, we disregard the uncertainty \( \overline{W}_0\) in (14). Substituting \( A+BK\) with \( \overline{X}_1V\) in (3)-(4) and together with (13), the LQR problem becomes

with the gain matrix \( K = \overline{U}_0V\) , where \( \Sigma\) is the positive definite solution to the Lyapunov equation

The problem formulation (15)-(16) is termed direct certainty-equivalence LQR design, and its solution coincides with that of the indirect design (10)-(11) [21].

We focus on the following adaptive control problem.

Problem 1 concerns both stability and optimality, which is in line with the definition of adaptive control by Zames [6], i.e., improve over the best with prior information. Throughout the article, we make the following assumptions From Section 3, we use \( X_{0,t}, U_{0,t}, W_{0,t}, X_{1,t}\) to denote the data series of length \( t\) in (6). We also add a subscript \( t\) to other notations to highlight the time dependence. .

Assumption 1 is standard in data-driven control [25, 26, 38]. Note that the noise does not necessarily follow any particular statistics and can even be adversarial and correlated. Assumption 2 is a quantitative condition of persistency of excitation and implies the rank condition (8). It can be satisfied by adding probing noise to the control input and holds under another PE definition [39, Definition 2], which can be verified by examining solely the inputs. Notice again that the PE assumption is universal in adaptive control [8, 7].

By Zames [6], adaptive control involves the acquisition of information about the plant. Define the signal-to-noise ratio (SNR) of data at time \( t\) as

which describes the ratio between the useful and useless information \( D_{0,t}\) and \( W_{0,t}\) [26, 27]. We adopt the SNR as an information metric, as the model uncertainty scales inversely with \( \text{SNR}_t\) .

This information metric satisfies Zames’s first monotonicity principle for adaptation and learning [6], as \( \text{SNR}_t\) is usually a monotone increasing function of time.

To solve Problem 1, we propose a policy gradient adaptive control (PGAC) framework alternating between control and gradient-based policy update. Specifically, the control input is in the form of \( u_t = K_t x_t + e_t\) , where the state-feedback gain \( K_t\) is the parameterized policy at time \( t\) , and \( e_t\) is a probing noise ensuring the PE condition. We assume that the initial gain \( K_{t_0}\) is stabilizing, which is common in adaptive control [9, 10, 11, 12, 13, 14]. Define \( \mathcal{S}:=\{K\in \mathbb{R}^{m\times n}|\rho(A+BK)<1\}\) as the set of stabilizing gains.

We let Assumption 3 hold in the rest of the article. The policy \( K_t\) is updated over time with gradient methods of the LQR cost. When \( (A,B)\) are known, the policy iterates as

where \( \nabla C(K_t)\) is the exact policy gradient. Then, the closed-form expression of \( \nabla C(K)\) is given below.

By Lemma 1, given knowledge of \( (A, B)\) , the policy gradient of the LQR cost can be computed by solving two Lyapunov equations in (4) and (19). While the cost \( C(K)\) is non-convex, it satisfies favorable properties such as gradient dominance and local smoothness, leading to global linear convergence of the gradient descent (17) [18, 40].

Even when all gains are stabilizing \( K_t \in \mathcal{S}, \forall t \geq t_0\) , due to switches of the policies, the stability of the closed-loop system is unclear. Moreover, since \( (A, B)\) are unknown in the adaptive control setting, the policy gradient can only be approximated using online closed-loop data, and the corresponding convergence remains unclear.

Depending on how the policy gradient is approximated, we categorize PGAC as indirect, when the gradient is computed with an estimated model (9), versus direct, when the gradient is directly computed from data with the covariance parameterization (13)-(15). In the sequel, we propose indirect and direct PGAC methods with convergence and stability guarantees.

Indirect PGAC uses the least-squares estimates \( (\hat{A},\hat{B})\) in (9) to approximately compute the policy gradient. The details are presented in Algorithm 1. We require an initial stabilizing gain and a batch of offline PE data. In the online stage, the control input contains a probing noise \( e_t\) to ensure Assumption 2. After observing the new state, we use recursive least-squares to identify an updated model estimate. Let \( \phi_t = [u_t^{\top},x_t^{\top}]^{\top}\) . Then, the recursive least-squares iteration is

which is an efficient rank-one update. Then, we perform one-step policy gradient descent with stepsize \( \eta\) , where the gradient is computed with the estimated model.

Lemma 1 will play an important role in quantifying the effects of replacing the exact policy gradient with the approximated policy gradient in (20). Together with Lemmas 3 and 4, we can show the convergence of \( K_t\) in Algorithm 1.

Apart from the convergence, we need to show stability of the closed-loop system. Since \( K_t\) is updated over time, we resort to stability analysis of switching or time-varying systems [12].

It is known that a policy \( K \in \mathcal{S}\) if and only if \( K\) is \( (\kappa, \alpha)\) -strongly stable for some \( \kappa\) and \( \alpha\)  [12]. Definition 1 is quantitative and will lead to explicit bounds on the state. Since a sequence of policies \( \{K_t\}\) is applied to the system, we require a stronger notion of stability under switching policies [12].

With sequential stability, we can show boundedness of the state [12]. By Definition 2, sequential stability holds if \( \{K_t\}\) is strongly stable uniformly in time, and the change in two consecutive policies is sufficiently small. In PGAC, this can be achieved by carefully controlling the stepsize \( \eta\) . In the following theorem, we show that Algorithm 1 achieves both stability and optimality.

We make several remarks on Theorem 1. First, under a sufficiently large SNR and a small stepsize, the system is sequentially stable under switches of \( K_t\) . The state is explicitly bounded by two terms, i.e., an exponential decrease in the initial state plus an upper bound on the probing and process noise. In comparison, stability of the adaptive dynamical programming (ADP) approach [7, 8] has not been certified and is unclear. While the one-shot-based adaptive control methods ensure stability, they require a dwell time to mitigate the burn-in effect of switching policies and a safety mechanism in case of divergence [13, 12, 10].

Second, the optimality gap of the LQR gain is upper bounded by two terms signifying an exponential decrease in the initial optimality gap plus a bias scaling inversely with the SNR. This aligns with Zames’s second monotonicity principle [6], i.e., the performance monotonously increases with the information metric. Our results capture the convergence rate of the policy for different SNR functions in Remark 1. For example, the optimality gap decreases as \( \mathcal{O}(1/\sqrt{t})\) for \( \text{SNR}_t \sim \mathcal{O}(\sqrt{t})\) , which corresponds to the case of Gaussian noise and a constant excitation level, and \( \mathcal{O}(t^{-\frac{1}{4}})\) for the diminishing excitation case with \( \text{SNR}_t \sim \mathcal{O}(t^{\frac{1}{4}})\) . Notice that the corresponding known optimal convergence rates are \( \mathcal{O}({1}/{t})\) and \( \mathcal{O}({1}/\sqrt{t})\) , respectively, obtained by one-shot-based adaptive control [11]. The latter requires a more conservative SNR condition such that the perturbation of the Riccati equation has a quadratic local approximation [9]. Nevertheless, we observe that Algorithm 1 achieves the optimal convergence rate in simulations.

Instead of computing the policy gradient with an estimated model, direct PGAC updates the policy based on the sample covariance parameterization (13)-(15). The details are presented in Algorithm 2, which is also known as data-enabled policy optimization (DeePO) [21]. In the online stage, we use sample covariance to parameterize the policy (27) and perform one-step projected gradient descent on the parameterized policy (28). Define \( \mathcal{V}: = \{V\in \mathbb{R}^{(n+m)\times n} |\rho(\overline{X}_1V)<1 \}\) as the feasible set of the covariance-parameterized LQR (15). The gradient and projection expressions are as below [21].

By Lemma 5, given data \( (X_0, U_0, X_1)\) , the policy gradient of the LQR cost can be computed by solving two Lyapunov equations in (16) and (25). The projection is adopted to ensure the subspace constraint in (15). The stepsize \( \eta_t\) is a variable to be specified later. As in indirect PGAC, Algorithm 1 also has an efficient recursive implementation [21].

As a novel result beyond [21], we show that the direct policy update (27)-(29) via projected gradient and the indirect counterpart (20) are equivalent up to a scaling matrix.

Lemma 6 enables us to leverage the previous results of indirect PGAC for the analysis of Algorithm 2.

Theorem 2 has three differences with Theorem 1 due to the relation between indirect and direct policy update in Lemma 6. First, the allowed SNR depends on the condition number of the scaling matrix \( M_t\) . Second, the stepsize \( \eta_t\) scales with the inverse of \( \|M_t\|\) . Third, the convergence rate of the policy depends on the minimal singular value of \( M_t\) , which by Lemma 6 scales with the excitation level \( \gamma_t\) . For the case of constant excitation, \( \underline{\sigma}(M_t)\) is lower bounded by a constant, and the convergence of the policy reduces to that of indirect PGAC (23). The diminishing excitation case does not lead to an explicit rate as \( \underline{\sigma}(M_t)\) is time-varying. Note that in adaptive control it is essential to maintain sufficient excitation to prevent the bursting phenomenon [16]. We refer to more comparisons between direct and indirect PGAC to simulations in Section 6.

We discuss the special case of uniformly bounded noise and constant excitation considered in our previous work [21]. In this case, the minimal singular value of \( M_t\) is lower bounded by a constant, and both \( \|M_t\|\) and the condition number of \( M_t\) converge asymptotically. Thus, the effect of \( M_t\) can be neglected, and we can take a constant stepsize \( \eta\) . Then, the condition in Theorem 2 reduces to that of [21, Theorem 2]. Theorem 2 reveals a linear convergence of the policy with respect to the initial optimality gap, which improves over the sublinear rate previously shown in [21, Theorem 2].

Let us vectorize \( K\) as \( \theta = \text{vec}(K^{\top})\) . Then, the natural policy gradient update of the LQR cost \( C(\theta)\) is given by With a slight abuse of notation, we use \( C(\theta)\) and \( C(K)\) interchangeably.

where \( \nabla C(\theta)\) is the vanilla gradient, and \( F_{\theta}\succ 0\) is the Fisher information matrix (FIM) that captures the curvature of the parameter space of \( \theta\) , which biases the gradient descent in the direction of large uncertainties. The corresponding update for \( K\) takes the form [18]

where \( {\Sigma}\) satisfies (4), \( P\) satisfies (19), and the last equality follows from Lemma 2. We briefly highlight the merits of natural gradient.

Motivated by (31), for indirect PGAC with natural gradient, we substitute the policy update (20) in Algorithm 1 with

where \( \hat{P}_{t+1}\) is the positive definite solution to Lyapunov equation (19) with the estimated model \( (\hat{A}_{t+1}, \hat{B}_{t+1})\) .

For direct PGAC with natural gradient, we notice that the covariance parameterization (13) defines a smooth bijection between the spaces of \( K\) and \( V\) . By leveraging the fact that FIM captures the geometry of the paramterization space, we show that indirect and direct PGAC are bridged by adopting the natural gradient descent for the policy update.

We provide convergence and stability guarantees for PGAC with natural gradient.

The theoretical guarantees in Theorem 3 are similar to those of PGAC with the vanilla gradient. Nevertheless, we note that performing natural gradient descent is computationally more efficient and leads to faster convergence rate; see Remark 2.

The Gauss-Newton method is a quasi-Newton method under a particular Riemannian metric [40]. It iterates as

with the constant stepsize \( 0<\eta <1\) . The stepsize of the Gauss-Newton method can be chosen more aggressively than the vanilla and natural gradient, leading to a faster convergence rate. With a special choice of stepsize \( \eta = 1/2\) , the Gauss-Newton update (33) is equivalent to

This coincides with the Hewer’s algorithm (5) and enjoys local quadratic convergence. However, the Gauss-Newton update is computationally less efficient than natural gradient, as computing (33) involves the inverse of a matrix.

Inspired by (33), for indirect PGAC we substitute the policy update (20) in Algorithm 1 with

Our theoretical guarantees of indirect PGAC with the Gauss-Newton update (34) are as below.

Since the stepsize can be chosen aggressively, it usually leads to faster convergence than PGAC with the vanilla and natural gradient. With the stepsize \( \eta = 1/2\) , our indirect PGAC with Gauss-Newton update is equivalent to an adaptive version of the Hewer’s algorithm (also referred to as online identification-based policy iteration in [15]); see Algorithm 3. However, Algorithm 3 generally does not have sequential stability guarantees, as the choice of \( \eta = 1/2\) may not ensure a sufficiently small policy change. An exception is when the initial policy is sufficiently close to the optimal LQR gain.

In Theorem 5, the condition on the initial policy \( K_{t_0}\) can be replaced with a condition on the SNR, given that \( K_{t_0}\) is obtained as the solution of the indirect certainty-equivalence LQR (10) with offline data. This aligns with the convergence and stability results in [15]that hold under sufficiently small identification error. The local quadratic convergence in (36) inherits from properties of the quasi-Newton method [40]. Note that the dependence of the convergence rate on the SNR is the same as that of Theorem 1.

For the indirect certainty-equivalence LQR (10)-(11), the closed-loop covariance matrix \( \Sigma\) satisfies the Lyapunov equation (11). However, given \( (A, B)\) , the Lyapunov equation that should be met is (4). For brevity, define

Then, the difference between the right-hand sides of (11) and (4) is

For well-behaved noise statistics, \( \|\overline{W}_0\|\) usually vanishes quickly with time [34, Lemma 1], and the first two terms dominate the difference. To reduce the difference, we introduce the regularizer \( \text{Tr}(\Phi^{-1}\Xi)\) to the certainty-equivalence cost (10), leading to the regularized cost

where \( \Sigma\) satisfies (11), and \( \lambda>0\) is the regularization coefficient. The regularizer is a correction to the penalty matrices of the LQR problem. In line with [28], the regularization also compensates the uncertainty in the cost function and promotes exploitation [34]. We demonstrate these merits via simulation in Section 6.3.

Next, we derive the gradient expression of the regularized cost (37). Consider the following partition of \( \Phi^{-1}\)

where \( (\Phi^{-1})_{uu}\in \mathbb{R}^{m\times m}\) and \( (\Phi^{-1})_{xx}\in \mathbb{R}^{n\times n}\) . Define \( Q_{\lambda} = Q + \lambda (\Phi^{-1})_{xx}\) and \( R_{\lambda} = R + \lambda (\Phi^{-1})_{uu}\) .

The proof of Lemma 8 follows the same vein as that of [18, Lemma 1] and is omitted due to space limitation.

To apply regularization to the indirect PGAC, we replace the gradient in Algorithm 1 with that of the regularized cost in Lemma 8. To ensure convergence of \( {K_t}\) to \( K^*\) , the effect of the regularizer must vanish over time. Since \( \text{Tr}(\Phi_t^{-1}\Xi)\) scales with \( \|\Phi_t^{-1}\| \leq \mathcal{O}(1/\gamma_t^2)\) , setting \( \lambda_t \leq \mathcal{O}(\delta_t\gamma_t)\) ensures that the regularization decays as \( \mathcal{O}(1/\text{SNR}_t)\) . We show this decay rate is sufficient for convergence and stability of the regularized indirect PGAC.

Under the condition \( \lambda_t \leq \mathcal{O}(\delta_t\gamma_t)\) , the theoretical guarantees can be extended to PGAC with natural gradient and Gauss-Newton method, whose formal proofs are omitted.

Leveraging the covariance parameterization in (13), the regularizer in (37) can be reformulated as \( \text{Tr}(V\Sigma V^{\top}\Phi)\) . Then, the direct LQR cost with regularization is

We derive the gradient expression of \( J(V;\lambda)\) , the proof of which follows the same vein as that of [21, Lemma 2] and is omitted due to space limitation.

As in regularized indirect PGAC, the regularization coefficient should scale as \( \lambda_t \leq \mathcal{O}(\delta_t\gamma_t)\) . Then, by leveraging the relation between indirect and direct PGAC in Lemma 6, we have the convergence and stability results.

We discuss the selection of the regularization coefficient \( \lambda_t = \lambda_0 \delta_t\gamma_t\) under special cases of Remark 1. For the bounded noise case, a constant \( \lambda_t\) is sufficient. For the Gaussian noise case, we let \( \lambda_t \sim \mathcal{O}(1/\sqrt{t})\) for constant excitation and \( \lambda_t \sim \mathcal{O}(t^{-\frac{3}{4}})\) for diminishing excitation. Our results align with [41], which shows that a constant \( \lambda_t\) over time is not favorable and may lead to a trivial solution in direct parameterization [25]. Note that we still need to tune the coefficient \( \lambda_0\) for optimal performance and stability.

We compare the convergence and stability of the one-shot-based adaptive control, direct and indirect PGAC with the vanilla gradient, natural gradient, and Gauss-Newton methods. We set the number of offline data to \( t_0=20\) and let \( x_0 = 0\) , \( u_t\sim \mathcal{N}(0,I_3)\) for \( t<t_0\) . For \( t \geq t_0\) , we set \( u_t =K_tx_t + e_t\) with \( e_t \sim \mathcal{N}(0,I_3)\) to ensure persistency of excitation. The noise is drawn from \( w_t\sim \mathcal{N}(0, I_3)\) , which corresponds to a poor \( \text{SNR}_t \in [-5,0]\) dB. We set the stepsize to \( \eta = 0.02\) for indirect PGAC with the vanilla gradient, \( \eta = 0.2\) for natural gradient, \( \eta = 0.5\) for the Gauss-Newton method (Algorithm 3), and \( \eta_t = 0.2/\|M_t\|\) for direct PGAC. The one-shot-based method obtains \( K_t\) from the Riccati equation with an estimated model (21) every time step. For all the methods, we set \( K_{t_0}\) as the certainty-equivalence LQR (10) with offline data.

Fig. 2 shows the optimality gap of the policy sequence for the one-shot-based method and direct and indirect PGAC with the vanilla gradient under the same randomness realization. For clarity of presentation, we omit the curves of indirect PGAC with natural gradient and Gauss-Newton, which lie between those of the one-shot-based method and indirect PGAC with the vanilla gradient. We make several key observations. First, all the methods improve the performance over the initial policy using online closed-loop data, and the optimality gap converges asymptotically at the rate \( \mathcal{O}(1/t)\) , which implies that our theoretically certified rate \( \mathcal{O}(1/\sqrt{t})\) is conservative. Second, the one-shot-based method diverges at the early stage due to unexpected large noise. In contrast, PGAC methods exhibit more stable convergence. This is expected in presence of noise, as the gradient is more robust than the minimizer.

Fig. 3 shows their average finite-horizon cost \( \sum_{i=0}^{t-1} \|z_i\|^2/t\) as a function of time, where \( z_t\) is the performance output in (2). We observe that all methods exhibit highly similar average costs, which converge rapidly to a positive constant. This indicates that the closed-loop systems resulting from both PGAC and the one-shot-based method are stable. Furthermore, the convergence behavior aligns with our theoretical results, i.e., the state is bounded by the sum of an exponentially decaying term and a steady-state bias determined by the probing and process noise. We note that the divergence at the early stage is due to abrupt large process noise.

We compare their efficiency in terms of the computation time. For indirect PGAC, we apply recursive least squares (21) for identification. For direct PGAC, we apply the rank-one update in [21, Section V-B]. For the one-shot-based method, the certainty-equivalence LQR problem (10) is solved using the dlqr function in MATLAB. We perform \( 20\) independent trials and record the mean of the computation time for running \( t=500\) time steps. The results are summarized in Table I. We observe that compared with the one-shot-based method, the PGAC approaches require significantly less computational time. This gap is due to that PGAC performs only one or two Lyapunov equation solves per iteration for gradient descent, whereas the one-shot-based method involves solving a Riccati equation at each step. Among all the PGAC approaches, direct PGAC consumes highest computation time as the dimension of the optimization matrix \( V\in \mathbb{R}^{(n+m)\times n}\) is higher than that of \( K\in \mathbb{R}^{m\times n}\) in the indirect case. For indirect PGAC, the vanilla gradient method is slowest as it requires solving two Lyapunov equations per time step instead of one for natural gradient and Gauss-Newton methods. It is worth noting that while we demonstrate the results using a simple third-order system (39), the computational gap between PGAC and the one-shot-based method grows substantially with increasing system dimension; we refer the reader to our previous work [21, Section V] for more comprehensive simulation results.

We study the impact of regularization for both indirect and direct PGAC with the vanilla gradient. We set the stepsize to \( \eta = 0.2\) for indirect PGAC and \( \eta_t = 0.2/\|M_t\|\) for direct PGAC. Consider \( 100\) independent trials. We denote by \( \mathcal{P}\) the percentage the algorithm converges without any instability issue and by \( \mathcal{M}\) the median of the optimality gap \( {(C(K_T) - C^*)}/{C^*}\) with \( T=1000\) through all trails where convergence is observed. The results are reported in Table II, where the one-shot-based method without regularization is for comparison. With regularization, both indirect and direct PGAC significantly outperform the one-shot-based method in terms of algorithm stability, i.e., the robustness of policy updates against noise. Furthermore, direct PGAC achieves a smaller optimality gap compared to the one-shot-based method. This is because our regularizer well compensates the uncertainty induced by noise in both the closed-loop covariance matrix and the cost function [34].

It follows from Assumptions 1 and 2 that \( \|[\hat{B}_t, \hat{A}_t] - [B, A]\| = \|W_{0,t} D_{0,t}^{\dagger}\| \leq \|W_{0,t}\|\|D_{0,t}^{\dagger}\| \leq {\delta_t}/{ \gamma_t}. \)

We first provide some useful bounds.

The proof frequently invokes the following perturbation analysis result for Lyapunov equations [21, Lemma 15].

Consider the policy gradient update with estimated \( (\hat{A}, \hat{B})\)

and the update with the ground-truth \( (A,B)\)

The convergence of the update (41) can be shown by leveraging Lemmas 3 and 4. If \( \eta \leq 1/l\) , then

where \( l\) is the smoothness parameter and \( \mu\) is the gradient dominance constant. To show the convergence of (40), we first quantify the distance between the exact gradient and the approximated gradient. Let \( p_1\) be a scalar function

and \( p_2 = {C^2(K) }/(\underline{{\sigma}(Q)p_1})\) . For the ease of notation, we omit the subscript in \( \gamma_t\) and \( \delta_t\) when it is clear from the context.

Next, we quantify how the difference in the policy gradient in Lemma 12 affects the difference in the cost \( |C(K'')-C(K')|\) . We leverage the following preliminary lemma from [18], which follows directly from Lemma 11.

Now, we bound the difference in the cost functions.

With Lemma 14 and (42), we show the convergence of (40)

To show the convergence of Algorithm 1, we need to first prove that \( C(K_t)\) is uniformly upper-bounded, such that the polynomials \( p_i, i\in\{1,2,\dots, 7\}\) have uniformly bounds. Let

where \( \underline{l} := l(C^*)\) , \( \mu = {\|\Sigma^*\|}/\underline{{\sigma}(R)}\) . Since the quantities \( l\) and \( p_i\) are function of \( C(K)\) , let \( \bar{l}, \bar{p}_1, \underline{p}_2, \bar{p}_3, \bar{p}_5, \bar{p}_6, \underline{p}_7\) be the associated quantities at \( \overline{C}\) , and \( \underline{p}_4\) be the quantity at \( C^*\) .

Then, under the condition (48), it follows that the convergence certificate (23) in Theorem 1 holds with \( \nu_5 = \bar{p}_3\bar{p}_6\) .

Next, we show the sequential stability of the closed-loop system of Algorithm 1. To this end, we first find proper system-theoretic matrices for \( (\kappa, \alpha)\) -strong stability.

We prove that the policy sequence of Algorithm 1 is sequentially strong stable.

With sequential strong stability, we establish the boundedness of the state under Algorithm 1. The proof follows the argument in [12, Lemma 3] and is included here for completeness.

Finally, we notice that the upper bound of \( \eta\) in (49) depends on \( \gamma_t/\delta_t\) . By using the bound \( \gamma_t/\delta_t \geq 1/\underline{p}_2\) , it follows that \( \eta\) has a constant upper bound, which completes the proof.

By the covariance parameterization (13) and the chain rule, the gradient satisfies \( \nabla J(V) = \overline{U}_{0}^{\top}\nabla \hat{C}(K)\) . Substituting it into (28) and using the parameterization (13) yield (26).

To show that \( \underline{\sigma}(M_t) \geq \gamma_t^4\) , note that

Then, it holds that \( \underline{\sigma}(M_t) = \sigma_m (\Phi_t \Pi_{\overline{X}_{0,t}} \Phi_t^{\top})\) . By the inequalities of singular value of matrix products, it follows that \( \underline{\sigma}(M_t) \geq \underline{\sigma}(\Phi_t)\sigma_m(\Pi_{\overline{X}_{0,t}}\Phi_t^{\top}) \geq \underline{\sigma}(\Phi_t) \sigma_m(\Pi_{\overline{X}_{0,t}}) \underline{\sigma}(\Phi_t) = (\underline{\sigma}(\Phi_t))^2\geq \gamma_t^4. \)

We omit the subscript of \( M_t\) when it is clear from the context. By Lemma 6, it suffices to consider the policy gradient update with estimated \( (\hat{A}, \hat{B})\)

and the gradient update with the ground-truth \( (A,B)\)

We have the following convergence result for (52).

Next, we show that the cost of Algorithm 2 has an upper bound, i.e., \( C(K_t) \leq \overline{C}\) , where \( \overline{C}\) is given by (47).

Furthermore, we show the convergence of Algorithm 2. Under the condition (53), we have

where the last inequality follows from \( \eta_t \leq 1/(\bar{l}\|M_{t}\|)\) . Then, the convergence certificate in Theorem 2 holds with \( \nu_6 = {\bar{p}_6\bar{p}_3}/{\bar{l}}\) . Finally, we prove that the policy sequence \( \{K_t\}\) of Algorithm 2 is sequential strong stable.

Noting that the condition on stepsize in (17) can be further simplified by using \( \gamma_t/\delta_t \geq 1/\underline{p}_2\) , the proof is completed.

First, we derive the natural policy gradient descent over \( V\) . Let \( \psi = \text{vec}(V^{\top})\) . Then, the covariance parameterization between \( K\) and \( V\) (13) can be expressed in terms of \( \theta\) and \( \psi\)

with a subspace constraint \( (\overline{X}_0 \otimes I_n)\psi = \text{vec}(I_n). \) Thus, the FIM with respect to \( \psi\) follows from the transformation rule

Since \( \psi\) lies in an affine space, we should restrict the Fisher inverse to its tangent space \( \mathcal{N}(\overline{X}_0\otimes I_n)\) . Denote \( O\) as a unit orthogonal basis of \( \mathcal{N}(\overline{X}_0)\) . Then, the corresponding orthogonal basis of \( \mathcal{N}(\overline{X}_0\otimes I_n)\) is \( (O\otimes I_n)\) , and the Fisher inverse along the tangent space is given by \( ((O\otimes I_n)^{\top}F_\psi (O\otimes I_n))^{-1}.\) Finally, the natural gradient method updates \( \psi\) in the feasible affine space as \( \psi^+ = \psi - \eta \Delta \psi\) , where \( \Delta \psi= (O\otimes I_n) ((O\otimes I_n)^{\top}F_\psi (O\otimes I_n))^{-1} (O\otimes I_n)^{\top} \nabla J(\psi). \)

Next, we show that the change in \( \psi\) leads to the change of \( \theta\) in (30). By the parameterization (54), \( \theta^+ =(\overline{U}_0\otimes I_n)\psi^+ = \theta - \eta (\overline{U}_0\otimes I_n) \Delta \psi\) . By the chain rule, it holds that \( \nabla J(\psi) = (\overline{U}_0\otimes I_n)^{\top} \nabla C(\theta)\) . Note that by Lemma 6, the matrix \( (\overline{U}_0\otimes I_n)(O\otimes I_n) = (\overline{U}_0O)\otimes I_n \) is invertible. Then, combining (55) and the expression of \( \Delta \psi\) , it follows that the update of \( \theta\) is \( \theta^+ = \theta - \eta F_{\theta}^{-1}\nabla C(\theta)\) , which is exactly (30).

Finally, noting that the above derivations hold under the estimated model \( (\hat{A}_{t+1}, \hat{B}_{t+1})\) , the proof is completed.

The proof follows the same vein as that of Theorem 1. Consider the policy gradient update with estimated \( (\hat{A}, \hat{B})\)

and the update with the ground-truth \( (A,B)\)

where \( E\) is defined in (43). We first quantify the difference between the exact and approximated natural gradient \( \hat{E}-E\) , and then quantify \( C(K'')-C(K')\) .

The proof follows from that of Lemma 12 and is omitted.

By [18, Lemma 15], if \( \eta \leq p_{10}\) , then the progress of the exact natural gradient descent (57) satisfies

Together with Lemma 24, the progress of (56) is

Define \( \overline{C} = C^* + C(K_0) + 1 + \frac{1}{2\|R\|\mu}\) . Then, we show that the cost is uniformly upper bounded by \( \overline{C}\) .

Then, under the condition (59), it holds that

Let \( \eta = 1/2\) . Consider the Gauss-Newton update with estimated \( (\hat{A}, \hat{B})\)

and the update with the ground-truth \( (A,B)\)

where \( E\) is defined in (43). We first quantify the difference between the exact and approximated Gauss-Newton gradient, and then quantify \( C(K'')-C(K')\) .

The proof follows from that of Lemma 12 and is omitted.

By [18, Lemma 8], the Gauss-Newton method (62) yields

Together with Lemma 28, the progress of (61) is

Let \( \overline{C} = C^* + c + \frac{c}{2\|\Sigma^*\|}\) for any constant \( c>0\) . Then, we have the following result.

The rest of the proof follows the same vein as that of Theorem 1 and is omitted.

[1] Anuradha M Annaswamy and Alexander L Fradkov A historical perspective of adaptive control and learning Annual Reviews in Control 2021 52 18–41

[2] Anuradha M Annaswamy Adaptive control and intersections with reinforcement learning Annual Review of Control, Robotics, and Autonomous Systems 2023 6 65–93

[3] RF Drenick and RA Shahbender Adaptive servomechanisms Transactions of the American Institute of Electrical Engineers, Part II: Applications and Industry 1957 76 5 286–292

[4] Petros A Ioannou and Jing Sun Robust adaptive control PTR Prentice-Hall Upper Saddle River, NJ 1996 1

[5] Zhongsheng Hou and Shuangshuang Xiong On Model-Free Adaptive Control and Its Stability Analysis IEEE Transactions on Automatic Control 2019 64 11 4555-4569 10.1109/TAC.2019.2894586

[6] George Zames Adaptive control: Towards a complexity-based general theory Automatica 1998 34 10 1161–1167

[7] Frank L Lewis and Draguna Vrabie Reinforcement learning and adaptive dynamic programming for feedback control IEEE Circuits and Systems Magazine 2009 9 3 32–50

[8] Tao Bian and Zhong-Ping Jiang Value iteration and adaptive dynamic programming for data-driven adaptive optimal control design Automatica 2016 71 348–360

[9] Horia Mania and Stephen Tu and Benjamin Recht Certainty Equivalence is Efficient for Linear Quadratic Control Advances in Neural Information Processing Systems 2019 32 Curran Associates, Inc.

[10] Yiwen Lu and Yilin Mo Almost Surely \( \sqrt{T}\) Regret Bound for Adaptive LQR IEEE Transactions on Automatic Control (early access) 2025

[11] Max Simchowitz and Dylan Foster Naive exploration is optimal for online LQR International Conference on Machine Learning 2020 8937–8948 PMLR

[12] Alon Cohen and Tomer Koren and Yishay Mansour Learning Linear-Quadratic Regulators Efficiently with only \( \sqrt{T}\) Regret International Conference on Machine Learning 2019 1300–1309 PMLR

[13] Jafar Abbaszadeh Chekan and Cedric Langbort Fully Adaptive Regret-Guaranteed Algorithm for Control of Linear Quadratic Systems arXiv preprint arXiv:2406.07746 2024

[14] Iasson Karafyllis and Maria Kontorinaki and Miroslav Krstic Adaptive control by regulation-triggered batch least squares IEEE Transactions on Automatic Control 2019 65 7 2842–2855

[15] Bowen Song and Andrea Iannelli Robustness of Online Identification-based Policy Iteration to Noisy Data arXiv preprint arXiv:2504.07627 2025

[16] Brian D.O. Anderson Failures of adaptive control theory and their resolution Commun. Inf. Syst. 2005 5 1 1-20

[17] Bin Hu and Kaiqing Zhang and Na Li and Mehran Mesbahi and Maryam Fazel and Tamer Başar Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies Annual Review of Control, Robotics, and Autonomous Systems 2023 6 123–158

[18] Maryam Fazel and Rong Ge and Sham Kakade and Mehran Mesbahi Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator International Conference on Machine Learning 2018 1467–1476

[19] Hesameddin Mohammadi and Armin Zare and Mahdi Soltanolkotabi and Mihailo R. Jovanović Convergence and Sample Complexity of Gradient Methods for the Model-Free Linear Quadratic Regulator Problem IEEE Transactions on Automatic Control 2022 67 5 2435-2450 10.1109/TAC.2021.3087455

[20] Feiran Zhao and Keyou You and Tamer Başar Global Convergence of Policy Gradient Primal-Dual Methods for Risk-Constrained LQRs IEEE Transactions on Automatic Control 2023 68 5 2934-2949 10.1109/TAC.2023.3234176

[21] Feiran Zhao and Florian Dörfler and Alessandro Chiuso and Keyou You Data-Enabled Policy Optimization for Direct Adaptive Learning of the LQR arXiv preprint arXiv:2401.14871 2024

[22] Anuradha M Annaswamy and Anubhav Guha and Yingnan Cui and Sunbochen Tang and Peter A Fisher and Joseph E Gaudio Integration of adaptive control and reinforcement learning for real-time control and learning IEEE Transactions on Automatic Control 2023 68 12 7740–7755

[23] Ivan Markovsky and Florian Dörfler Behavioral systems theory in data-driven analysis, signal processing, and control Annual Reviews in Control 2021 52 42–64

[24] Jeremy Coulson and John Lygeros and Florian Dörfler Data-enabled predictive control: In the shallows of the DeePC 18th European Control Conference (ECC) 2019 307–312

[25] Claudio De Persis and Pietro Tesi Formulas for data-driven control: Stabilization, optimality, and robustness IEEE Transactions on Automatic Control 2019 65 3 909–924

[26] Florian Dörfler and Pietro Tesi and Claudio De Persis On the Certainty-Equivalence Approach to Direct Data-Driven LQR Design IEEE Transactions on Automatic Control 2023 68 12 7989-7996 10.1109/TAC.2023.3253787

[27] Florian Dörfler and Pietro Tesi and Claudio De Persis On the Role of Regularization in Direct Data-Driven LQR Control 61st IEEE Conference on Decision and Control (CDC) 2022 1091-1098 10.1109/CDC51059.2022.9992770

[28] Alessandro Chiuso and Marco Fabris and Valentina Breschi and Simone Formentin Harnessing uncertainty for a separation principle in direct data-driven predictive control Automatica 2025 173 112070

[29] Feiran Zhao and Florian Dörfler and Keyou You Data-Enabled Policy Optimization for the Linear Quadratic Regulator 62nd IEEE Conference on Decision and Control (CDC) 2023 6160-6165 10.1109/CDC49753.2023.10383470

[30] Feiran Zhao and Ruohan Leng and Linbin Huang and Huanhai Xin and Keyou You and Florian Dörfler Direct Adaptive Control of Grid-Connected Power Converters via Output-Feedback Data-Enabled Policy Optimization arXiv preprint arXiv:2411.03909 2024

[31] Xuerui Wang and Feiran Zhao and Andres Jurisson and Florian Dörfler and Roy S. Smith Unified Aeroelastic Flutter and Loads Control via Data-Enabled Policy Optimization IEEE Transactions on Aerospace and Electronic Systems 2025 (early access) 1-12 10.1109/TAES.2025.3566351

[32] Niklas Persson and Feiran Zhao and Mojtaba Kaheni and Florian Dörfler and Alessandro V Papadopoulos An Adaptive Data-Enabled Policy Optimization Approach for Autonomous Bicycle Control arXiv preprint arXiv:2502.13676 2025

[33] Gary Hewer An iterative technique for the computation of the steady state gains for the discrete optimal regulator IEEE Transactions on Automatic Control 1971 16 4 382–384

[34] Feiran Zhao and Alessandro Chiuso and Florian Dörfler Regularization for Covariance Parameterization of Direct Data-Driven LQR Control arXiv preprint arXiv:2503.02985 2025

[35] Sarah Dean and Horia Mania and Nikolai Matni and Benjamin Recht and Stephen Tu On the sample complexity of the linear quadratic regulator Foundations of Computational Mathematics 2020 20 4 633–679

[36] Brian DO Anderson and John B Moore Optimal control: linear quadratic methods Courier Corporation 2007

[37] Jan C Willems and Paolo Rapisarda and Ivan Markovsky and Bart LM De Moor A note on persistency of excitation Systems & Control Letters 2005 54 4 325–329

[38] Henk J Van Waarde and Jaap Eising and M Kanat Camlibel and Harry L Trentelman The informativity approach: To data-driven analysis and control IEEE Control Systems Magazine 2023 43 6 32–66

[39] Jeremy Coulson and Henk J Van Waarde and John Lygeros and Florian Dörfler A quantitative notion of persistency of excitation and the robust fundamental lemma IEEE Control Systems Letters 2022 7 1243–1248

[40] Jingjing Bu and Afshin Mesbahi and Maryam Fazel and Mehran Mesbahi LQR arXiv preprint arXiv:1907.08921 2019

[41] Xiong Zeng and Laurent Bako and Necmiye Ozay Noise Sensitivity of the Semidefinite Programs for Direct Data-Driven LQR arXiv preprint arXiv:2412.19705 2024