Stochastic control

Stochastic control is a subfield of control theory which deals with the existence of uncertainty in the data. The designer assumes, in a Bayesian probability-driven fashion, that a random noise with known probability distribution affects the state evolution and the observation of the controllers. Stochastic control aims to design the optimal controller that performs the desired control task with minimum average cost despite the presence of these noises.

An extremely well studied formulation in stochastic control is that of linear-quadratic-Gaussian problem. Here the model is linear, and the objective function is the expected value of a quadratic form, and the additive disturbances are distributed in a Gaussian manner. A basic result for discrete time centralized systems is the certainty equivalence property: that the optimal control solution in this case is the same as would be obtained in the absence of the additive disturbances. This property is applicable to all systems that are merely linear and quadratic (LQ), and the Gaussian assumption allows for the optimal control laws, that are based on the certainty-equivalence property, to be linear functions of the observations of the controllers.

This property fails to hold for decentralized control, as was demonstrated by Witsenhausen in the celebrated Witsenhausen's counterexample.

Any deviation from the above assumptions—a nonlinear state equation, a non-quadratic objective function, or noise in the multiplicative parameters of the model—would cause the certainty equivalence property not to hold. In the discrete-time case with uncertainty about the parameter values in the transition matrix and/or the control response matrix of the state equation, but still with a linear state equation and quadratic objective function, a matrix Riccati equation can still be obtained for iterating to each period's solution.^ch.13 The discrete-time case of a non-quadratic loss function but only additive disturbances can also be handled, albeit with more complications.

Jerome L. Stein (Division Applied Mathematics, Brown University) Stochastic Optimal Control and US Financial Crisis (Springer-Science, 2012) is an interdisciplinary book uses stochastic optimal control to explain the financial/debt crisis of 2008. Page proofs and reviews may be downloaded from <http://www.dam.brown.edu/people/Stein.htm>. A sketch of the main point is as follows.

5.3. The Stochastic Optimal Control Analysis

Abstract: As explained in the earlier chapters, the financial structure rested upon the ability of the mortgagors to service their debts. That is where “systemic risk” was to be found. When the households could not service their debts, the highly leveraged financial structure collapsed. For this reason the focus is upon the optimal debt ratio of households. The stochastic optimal control approach was discussed and the basic propositions were derived in chapter four. In the present chapter I draw upon the results of that chapter. The hypothetical/idealized optimizer is the investor in real estate who finances his purchase with a mortgage debt. The optimum debt ratio is selected to maximize the expectation of a concave function of net worth at some terminal date, subject to stochastic processes on the house price and interest rate. The optimum debt/net worth ratio depends most crucially upon the assumed stochastic process concerning the price or capital gains variable. I explained in chapter four that the expected growth of net worth is maximal when the debt ratio is optimal. As the debt ratio exceeds the optimal, the expected growth declines and the variance of growth increases. The excess debt is the actual less the optimal. As the excess debt rises, the probability that the household cannot pay its debt increases. Thereby we have an early warning signal. Since no one knows what is the “true” stochastic process there is uncertainty concerning the true optimal debt ratio. In chapter four, I showed that the difference between the actual debt ratio and the true optimal is directly related to the difference between the actual growth of net worth and the optimal growth. This way one can have bounds for the effect of misspecification of stochastic process upon growth of net worth. Several stochastic processes in chapter four were considered and the optimal debt ratio was derived for each case. In the present chapter I simply state the results: the optimal debt ratio that is implied by the alternative stochastic processes, Model I and Model II. On the basis of models I/II, I derive an equation for an empirical estimate of the optimal debt ratio. That is the benchmark optimal ratio. I then derive empirical estimates of the excess debt and show that it is a warning signal of a crisis. In this manner, I reject the Greenspan et al view that the crisis was unpredictable.

A review by Peter Clark, Cato Journal Spring/summer 2012 is:

Review of Jerome Stein’s Stochastic Optimal Control and the U.S. Financial Debt Crisis

Kredit & Kapital, No. 2 2012 CATO Journal,Spring/Summer 2012.

                                           Peter Clark, Chevy Chase, Maryland

At one point during the recent financial crisis the Queen of England reportedly asked economists at the London School of Economics a seemingly straightforward question: Why did academic economists fail to foresee the crisis? This question can be broadened to include central banks, the IMF, and technical specialists on Wall Street (“Quants”). Professor Stein’s timely book provides a cogent and convincing answer to this question. He devotes one chapter to why the Fed and the Fund failed to anticipate the crisis, and another chapter to the failure of the Quants and their mathematical models to properly measure the risks associated with the new financial instruments which they had invented. What sets the book apart from the burgeoning literature on the crisis (which Stein draws on) is that it develops a theoretically well-based measure of optimal debt which provides a yardstick against which to compare the actual debt level. As the latter rises significantly above the benchmark, there is a growing risk that the debt has become unsustainable, i.e., that a financial bubble has been generated which is increasingly likely to collapse. Stein applies this approach to show that asset values became vastly out of line with the fundamentals of the U.S housing market and with the balance sheet of the insurance and financial firm, AIG. Stein’s approach generates an early warning signal of impending financial collapse which he also applies to the bubble in agricultural land prices in the 1980s. Stein in addition provides a timely analysis of the (still ongoing in 2012) financial crisis in Europe and makes a strong case that it reflected just as much excessive private debt as an over-indebted government sector.

In Chapter 2 Stein discusses why the Fed and the Fund were oblivious to the signs of an emerging bubble in the U.S. housing market and the risk this posed to the wider financial system. He states that one reason for this failure was the lack of adequate economic models, but he does not describe the models then in use or explain their inability to capture financial sector vulnerability and systemic risk. The core of Stein’s argument is that the Fed was not predisposed to attempt to identify financial bubbles because it believed there was no reliable way to do so; pricking such a bubble prematurely could involve unnecessary costs; and the preferred strategy was to use monetary policy to “mop up” after a bubble had burst of its own accord to deal with any adverse economy-wide macroeconomic fallout. This hands-off approach for dealing with financial bubbles became known as the Jackson Hole Consensus, named after Jackson Hole, Wyoming, where this view was first enunciated at one of the annual summer conferences on monetary economics and policy sponsored by the Kansas City Federal Reserve. Stein notes that the Fed was lulled into a false sense of security because this strategy was viewed as successful in dealing with previous bubbles; the macroeconomic situation at the time of low inflation and sustained growth – dubbed the Great Moderation - was quite benign; and because it was felt that a downturn in the U.S. housing sector would not seriously affect the broader economy. This sanguine view was clearly revealed when the Federal Reserve released on January 12, 2012, the transcripts of the meetings of the Federal Open Market Committee in 2006. At the June meeting that year, a Fed economist reported to the Committee that “we have not seen – and don’t expect – a broad deterioration in mortgage credit quality.” And at the December meeting Geithner, then president of the Federal Reserve Bank of New York, said “The softer-than expected recent numbers don’t argue, in our view, for a substantial reassessment of the risks to the outlook.” Stein notes that there were some people in the financial industry, state bank regulators, and the FDIC who warned of the dangers in the housing market, and he cites Robert Schiller who argued in 2007 that it was not possible to explain the boom in terms of such fundamentals as rent and construction costs. Stein himself presents a revealing chart which shows that by 2006 the ratios of debt service and housing prices relative to disposable income rose to two and three times, respectively, their long-run averages. This real-time data could have set off alarm bells that there was a housing bubble, but the Fed was not looking for signs of a bubble. Stein also cites the report of the Fund’s own Independent Evaluation Office (IEO) which concluded that the institution had missed key elements that generated the financial meltdown, including in particular the deteriorating lending standards in mortgage financing. The IEO attributed the failure of the Fund to give clear warnings to groupthink: “a tendency among homogeneous groups to consider issues only within a certain paradigm and not to challenge its basic premises.” With a few exceptions, there was also groupthink among the computer scientists, mathematicians, and physicists who were attracted to Wall Street and who are collectively referred to as “Quants.”

Stein shows in Chapter 3 that the finance models used by the Quants were inadequate and consequently they grossly underestimated risk. The Quants used the basic building blocks of modern finance – the Capital Asset Pricing Model, the Black-Scholes-Merton Options Pricing Model, and the Efficient Market Hypothesis – to construct new financial instruments which were thought to enable investors to reduce risk. These included credit default swaps (CDS), essentially insurance contracts on bonds and other financial instruments, and collateralized debt obligations (CDO) which are bundles of securities, such as credit-card debt and mortgages. In the case of CDOs, large numbers of mortgages were securitized into bundles by financial intermediaries who then sold the packages to large investors, mutual funds, pension funds, insurance companies and hedge funds. The Quants worked out a method of prioritization in which tranches in the pool of mortgages were sold with different risk ratings. Losses or defaults in the bundle would be borne first by investors who held the lowest (equity) tranche; when losses in the underlying mortgages exhausted this tranche, the holders of the medium (mezzanine) tranche would be affected; but the top or senior AAA tranche holders would lose money only if losses exceeded the assets in all the tranches. The attribution of different levels of risk – from AAA to BBB or lower – to specific tranches enabled the sellers of the bundles of mortgages to satisfy the specific risk preferences of investors. 

Stein makes a major contribution in this chapter by explaining the many flaws in this strategy. First, the financial intermediaries that bundled the mortgages did not undertake due diligence to assess the risk associated with the individual components of the bundle. Second, the strategy assumed that the tranches were statistically independent in the population (bundle), but no analysis was undertaken to support this dubious assumption by analyzing conditional probabilities of default across tranches. Third, the method used by the Quants to measure the risk of a tranche assumed a stable distribution of house price increases which was patently unrealistic, as it was based on the continuation of past price increases and thus made no allowance for the possibility of price declines. Stein’s key point is that this flawed methodology gave the illusion that one could create AAA securities out of risky mortgages, and this led to very high leverage. When the housing bubble burst, the sale of supposedly AAA assets at fire sale prices had a cascading effect on the rest of the financial system as the house of cards created by the Quants collapsed. In light of this devastating critique of Quant methodology, one wonders whether the defects identified by Stein have been corrected, or at least ameliorated. It would have added to the book’s relevance if he had provided some answers to this question.

In Chapter 4 Stein explains the rationale for his approach for identifying an asset bubble – Stochastic Optimal Control (SOC) – and then applies this technique to three areas of the U.S. economy in Chapters 5 – 7. He is motivated by the keen desire to use a well-founded analytical framework to establish a benchmark against which to assess the actual level of debt and to evaluate whether it is excessively risky. This benchmark is the optimal level which balances risk against the expected growth in the value of the asset acquired by taking on the debt. As actual debt rises above the optimal level and becomes excessive, expected growth declines, risk increases and the probability of a debt crisis rises. The basic equation relates the change in net worth of the mortgagor positively to the capital gain and the productivity of the real estate purchased with the bank loan, and negatively to the interest rate paid on this loan and the consumption of the mortgagor. Both the productivity of the asset and consumption are assumed to be known, whereas the capital gain on the asset and the interest are stochastic variables that are negatively correlated. In his model, Stein rules out what he calls a “free lunch” and constrains the drift in the capital gain not to exceed the mean rate of interest, although he emphasizes that such a free lunch helped generate the housing bubble. Optimal debt is generated by the maximization of a concave function of net worth at a terminal date T subject to the dynamic stochastic processes. On the assumption that the stochastic variables follow Brownian Motion, the maximization results in an intuitively satisfying result where the optimum net worth is determined by the maximum distance between the growth of net worth (a linear function of the level of debt) and the mean risk (a convex function of the level of debt), which is shown graphically in Figure 4.1 in the book. The corresponding optimum debt ratio is positively related to the difference between the return on the asset and the interest rate, and negatively related to the variances of these two variables and the negative covariance between them. Thus the SOC approach used by Stein yields a simple expression for the optimal debt ratio which rigorously incorporates the key elements of risk and return: the optimal debt ratio is a function of (net return – risk elements)/risk.

Prof. Stein provides an empirical implementation of this theoretical approach in Chapter 5. He makes the strong assumption that the mean rate of appreciation of housing prices is equal to the mean interest rate on mortgages, and therefore these two key variables drop out of the empirical part of hid analysis. The other component of the return on housing, namely, the productivity of the stock of residential capital, is approximated by the ratio of rental income to an index of housing prices (RENTPRICE). Stein gets around the problem that these two variables are in different units by normalizing the ratio, i.e., subtracting the mean of RENTPRICE and dividing by its standard deviation. Stein argues that this procedure removes the risk elements in the basic equation for the optimal debt ratio, even though the variances of the capital gain and the interest rate, as well as their covariance, are integral to the determination of the optimal debt. The level of the debt ratio is approximated by the ratio of household debt service payments to disposable income, and this variable, DEBTSERVICE, is normalized in the same way as RENTPRICE. Finally, Stein measures excess debt as RENTPRICE – DEBTSERVICE, and when he plots this difference, he finds that it is on a sharply rising trend from the mid-1990s, and that by 2005 it was about two standard deviations above the mean. As this year was roughly coincided with the peak of the housing price bubble, Stein is quite justified in claiming that it is a useful warning signal of an overheated and unsustainable boom in the U.S. housing market. The subsequent default of vast quantities subprime mortgages then led to a financial meltdown on account of excess leverage and the close web of connections linking financial intermediaries which Stein describes in the last section of Chapter 5.

It was this web of financial connections which led to the fall of the American International Group (AIG), the subject of Chapter 6. This insurance behemoth once had an AAA rating and over a trillion dollars in assets. Through its London AIG Financial Products (AIGFP) subsidiary, it sold huge numbers of CDS contracts to banks and hedge funds who sought to hedge the risk of a decline in the CDOs they owned in the face of the collapsing U.S. housing market. Stein describes how AIG became enmeshed in the financial crisis of 2007-8 because it seriously underestimated the risk it had incurred by failing to take account of the negative correlation between the CDOs it insured and its CDS liabilities: the CDS claims on it increased when the value of the insured CDOs declined. AIGFP did not monitor the market value of the CDOs that were guaranteed by the CDS contracts it had written. Stein evaluates the actuarial literature on optimal risk and capital requirements for insurers and develops a version of the SOC approach that combines the economics and actuarial literature. The chapter concludes with an assessment of the bailout of AIG by the government. Chapter 7 moves back in time to the 1980s and examines the crisis in the agricultural sector and that involving savings and loan associations. Stein describes the common features of these two crises and explains why they remained localized and unlike the mortgage bubble which came later, did not infect the rest of the financial system, He shows how the same SOC techniques described in earlier chapters could have generated useful early warning signals of the agricultural crisis.

Stein concludes his book with a very timely and perceptive analysis of the ongoing debt crisis in Europe. He argues convincingly that the origin of this crisis lies in the external debts of the European countries which reflected not only excessive government budget deficits but also an excess of private investment relative to saving. Thus the exclusive focus of the European Union and European policymakers on the fiscal origin of the crisis is misplaced because it ignores the problem of the excessive external debt of the entire economy. He brings to bear an array of charts and tables to support his case that the origin of the crisis differs widely across eurozone members who were by no means all fiscal profligates. Stein also provides an insightful theoretical analysis of the origins of the crisis based on his Natural Real Exchange Rate Model (NATREX) which can explain the simultaneous evolution of two endogenous variables: the real exchange rate and external debt. His NATREX model has been widely used to assess whether observed exchange rates are out of line with their fundamental determinants.

Professor Stein has written a timely book on the financial crisis emanating from the collapse of the U.S. mortgage market, as well as on the European financial crisis. It synthesizes the large economics literature on the former with the mathematics literature on stochastic optimal control (SOC). It should appeal both to economists and mathematicians interested in how SOC techniques could have been used to provide early warning signals of the recent crises, as well as to those interested in risk management. Because it is a readable account of financial crises that have generated incredibly high economic costs, the book should also be read by policy makers. It conveys the very important lessons that early warning signals can be used to detect the emergence of financial bubbles, and that it is incumbent on policy makers to act on this information. Thanks to Professor Stein, the once common view that financial bubbles should be left to work themselves out – as embodied in the Jackson Hole Consensus – is no longer tenable.

References:

Jerome L. Stein, Stochastic Optimal Control and US financial Crisis, Springer-Science, 2012 Peter Clark, review of Stochastic Optimal Control and US financial Crisis, Cato Journal, Spring/summer 2012