There has been rapid growth in the volume of trading on futures exchanges in recent years. For example, Irwin and Sanders (2012) document that trading volumes in agricultural futures markets have increased by a factor of 3 since 2000. Futures contracts allow market participants to lock in today the price of future transactions covering a wide range of commodities and financial assets. The price of such futures contracts is a potentially valuable source of information about market expectations.

## The role of time-varying risk premia

Exploiting this information has proved difficult in practice, however, because the presence of a time-varying risk premium may drive a wedge between the current futures price and the expected spot price of the underlying asset (e.g. Fama and French 1987). In other words, the futures price is an adequate measure of the market expectation only in the unlikely case of a zero risk premium.

Even though the market expectation may in principle be recovered by adjusting the observed futures price by an estimate of the time-varying risk premium, a common problem in applied work is that there are as many measures of market expectations as there are estimates of the risk premium, and these risk premium estimates may differ substantially. Thus, attempts to pin down the market expectation have often proved elusive. In a recent study we propose a general solution to this problem that allows one to identify the best possible estimate of the market expectation for any set of risk premium estimates (see Baumeister and Kilian 2014).

The central idea is that – in the presence of a risk premium – the risk-adjusted futures price is the conditional expectation of the price and hence the minimum mean-squared prediction error (MSPE) predictor by construction (see Granger 1969). This fact allows one to rank alternative model specifications based on their MSPEs and to identify the most accurate measure of the market expectation. We illustrate this approach by solving the long-standing problem of how to recover the market expectation of the price of crude oil.

## Empirical illustration: Oil futures market

The price of oil is one of the key economic variables for the assessment of macroeconomic performance and risks at central banks and international organisations. It plays an important role in designing environmental policies, and it has an immediate impact on a wide range of industries such as the automobile industry, airlines, and utility companies. It also has implications for the economic viability of the production of crude oil from Canadian oil sands and the viability of US shale oil production, which directly affects the energy security of the US.

The evolution of the price of oil is highly uncertain and difficult to predict with a reasonable degree of accuracy. For many years, the standard practice among policymakers and central bankers, in the business community, in the financial press, and in the academic literature, has been to interpret the price of crude oil futures as the market expectation of the spot price of crude oil. The use of oil futures prices as out-of-sample oil price forecasts relies on this interpretation, as does the use of oil futures prices as a measure of oil price expectations of firms and consumers in microeconomic models.

The popularity of this approach has several explanations. First, futures prices are simple to use and readily available in real time. Second, there is a reluctance to depart from what is viewed as the collective wisdom of the financial market, which presumably knows better than any individual oil price forecaster. Relying on what is perceived to be the market expectation also absolves the forecaster from any culpability for forecast errors, because no one can reasonably be expected to beat the market. Third, there is evidence that futures prices have some forecasting power at longer horizons, although their forecast accuracy has varied substantially over time. Moreover, until recently there were few alternatives available to oil price forecasters. Fourth, while it is well understood that time-varying risk premia would invalidate the use of oil futures prices as oil price forecasts, it has proved difficult to reject the absence of a time-varying risk premium based on the traditional statistical tests of forecast efficiency proposed by Fama and French (1987).

This practice has been challenged in recent years by a large number of empirical studies documenting the existence of time-varying risk premia in the oil futures market. These studies move beyond the statistical framework proposed by Fama and French and provide direct evidence that returns in oil futures markets can be predicted using a range of aggregate and commodity market-specific financial and macroeconomic variables. A new consensus has been emerging in the academic literature that time-varying risk premia are an important feature of the crude oil market. For example, Singleton (2014) concludes that “the evidence for time-varying risk premiums in oil markets … seems compelling”.

Although the evidence for time-varying risk premia may seem overwhelming at first sight, closer inspection reveals that it is difficult to draw general conclusions from this literature because the studies in question differ along many dimensions including the estimation period, forecast horizon, and evaluation period. We therefore provide for the first time a systematic comparison of the predictive power of a wide range of risk premium models proposed in the literature. We quantify the estimated risk premia in dollar terms and investigate their sign, their magnitude, and their variability across alternative model specifications. We show that there is tremendous variability in the risk premium estimates across model specifications, creating uncertainty about the magnitude of this risk premium as well as the implied market expectation of the price of oil. For example, alternative estimates of the risk premium for the same month may differ by as much as $56.

This model uncertainty can be resolved based on the observation that the risk-adjusted futures price is the conditional expectation of the price of oil and hence the minimum MSPE predictor by construction. We therefore select among the candidate risk premium models the model that implies the expectations measure for the dollar price of oil with the smallest MSPE. Our analysis reveals little empirical support for estimates of the risk premium based on return regressions of the type popular in recent applied work on oil markets. Our preferred estimate of the risk premium is instead based on an updated version of the term structure model of the oil futures market developed by Hamilton and Wu (2014). Risk-adjusted futures prices based on this model reduce the MSPE by between 20% at the three-month horizon and 34% at the 12-month horizon compared with the unadjusted oil futures price. Their directional accuracy ranges from 61% to 68% and is highly statistically significant.

**Figure 1**. Oil-price expectations based on the Hamilton–Wu term structure model

Based on this model, we provide monthly time series estimates of the market expectation of the price of oil for 1992–2014. These expectations may differ substantially from the observed futures price. Figure 1 illustrates, for example, that the 12-month-ahead market expectation of the price of oil rose from $30 initially to a peak of $100 in 2008. There is no evidence that the market anticipated the collapse of the price of oil in late 2008. In fact, even when the spot price reached $134 in June 2008, market participants did not expect the price to remain at this level. After 2009, the one-year-ahead market expectation of the price of oil stabilised near $90.

Our analysis also helps explain the apparent failure of the oil futures price as a predictor of the spot price of oil during the surge in the price of oil between 2003 and mid-2008. A long-standing puzzle is why during 2003–2008 oil futures prices remained largely unchanged amidst rising spot prices. Figure 2 illustrates that the discrepancy between futures prices and realised spot prices is explained in part by a positive risk premium.1

**Figure 2**. Selected trajectories of the futures price, the realised spot price, and the risk-adjusted futures price implied by the Hamilton–Wu model

In extracting the market expectation of the price of oil from the futures price, it is essential to estimate the risk premium based on the full sample. This approach provides the most efficient estimate of the oil price expected by the market at each point in time in the past, which is the relevant expectations measure, for example, in estimating economic models of automobile purchases, investment decisions under uncertainty, environmental policies, and regulatory reforms. In contrast, if the objective is to improve the accuracy of out-of-sample forecasts of the price of oil by risk-adjusting the oil futures price, real-time estimates of the risk premium are required. Such estimates may be constructed based on recursive or rolling regressions possibly subject to delays in the availability of the data and revisions of preliminary data.

Not surprisingly, estimating the risk premium in real time is more challenging than estimating it using the full-sample information. We found that even the risk-adjusted forecast based on the Hamilton and Wu (2014) term structure model is unable to improve on the accuracy of the unadjusted oil futures price. Similar results hold for all other model specifications in a real-time setting. We concluded that the accuracy of forecasts based on the oil futures price cannot be improved by adjusting the futures price by real-time estimates of the risk premium. There are, of course, other approaches to forecasting oil prices that have been shown to have superior real-time forecast accuracy and can be implemented by central banks and other forecasters.

## Concluding remarks

Expectations play a key role in a wide range of forward-looking economic models. Although we chose to illustrate our procedure for recovering the market expectation in the context of the oil futures market, the underlying methodology is general, and can be applied to futures prices for foreign exchange, interest rates, and many other commodities when there is disagreement between alternative models of the time-varying risk premium.

*Disclaimer: The views expressed in this column are those of the authors and should not be attributed to the Bank of Canada.*

## References

Baumeister, C and L Kilian (2014), “A General Approach to Recovering Market Expectations from Futures Prices with an Application to Crude Oil”, CEPR Discussion Paper 10162.

Fama, E F and K R French (1987), “Commodity Futures Prices: Some Evidence on Forecast Power, Premiums, and the Theory of Storage”, *Journal of Business* 60: 55–73.

Granger, C W J (1969), “Prediction with a Generalized Cost of Error Function”, *Operations Research Quarterly* 20: 199–207.

Hamilton, J D and C J Wu (2014), “Risk Premia in Oil Futures Prices”, *Journal of International Money and Finance* 42: 9–37.

Irwin, S H and D R Sanders (2012), “Financialization and Structural Change in Commodity Futures Markets”, *Journal of Agricultural and Applied Economics* 44: 371–396.

Singleton, K J (2014), “Investor Flows and the 2008 Boom/Bust in Oil Prices”, *Management Science* 60: 300–318.

## Footnote

1 It may seem that the problem of identifying the market expectation could alternatively have been solved by searching for the model with the most predictive power for the return on oil futures contracts. Indeed, this is one metric by which return regressions in the literature have often been evaluated. There is no reason, however, for the model that minimises the MSPE for the rate of return also to minimise the MSPE for the spot price of oil expressed in dollars, because the loss functions differ. In fact, it can be shown that minimising the MSPE of the rate of return produces inaccurate measures of oil price expectations. In addition, evaluating the risk premium models under a different loss function than the loss function used in their estimation also helps deal with the problem of data mining in fitting return regressions.