Model Evaluation
Whim and Rigour

History warns us that the selective use of evidence to support theories can yield short-term confirmation, but that this may represent nothing more than the corroboration of prejudice. Simple positive correlations between the money supply and inflation, or between unemployment and the level of real wages appeal to intuition but do not alone provide a solid foundation on which to build policy.

In Discussion Paper No. 68, Research Fellows David Hendry and Grayham Mizon discuss the problem of model evaluation, emphasising the distinction between the process of 'discovery' and the process of 'justification'. They argue that serendipity in the creation of theories and models is desirable and even essential if models are ever to escape the confines of accepted theory. Edward Leamer, in an influential article in the American Economic Review, has criticized existing econometric models as 'whimsical' in the sense of being arbitrary reflections of the beliefs of the data analyst. Hendry and Mizon disagree, arguing that 'whimsy' plays an important role in the generation of new and genuinely innovative theories. There is, however, no role for whimsy in assessing a theory; Hendry and Mizon argue that this should be done systematically and thoroughly.

The paper describes criteria designed to promote such a systematic and thorough assessment of economic models. There are four sources of information relevant to model evaluation, according to Hendry and Mizon: a priori theory, sample data, properties of the measurement system used to collect the sample data, and the performance of alternative models. Hendry and Mizon describe a model as congruent when it is coherent with all four types of information. Part of the reason for ensuring that a model is consistent with all these forms of evidence is to persuade others of its value, by demonstrating that it is robust to minor changes in specification, and that it performs at least as well as any other available model. This latter property, that of requiring a model to encompass its rivals, imposes high demands on a model, but when satisfied provides impressive evidence in its favour.

The model design criteria discussed by Hendry and Mizon do not constitute a unique modelling strategy, but rather provide a set of necessary conditions to be satisfied if a model is to be worthy of serious consideration by people other than its originator. The authors argue that whilst there is no known set of sufficient conditions for good model design, the ones they describe have proved to be valuable in practice. The procedures they advocate improve research efficiency, because they do not require the estimation of a large number of models before finding a reasonable one. They also lead to the selection of more 'desirable' models, which are less likely to predict badly when new information becomes available.
Hendry and Mizon observe that one of the difficulties in arguing the case for econometrics is the fact that it is essentially destructive; the weeding out of inadequate models and economic theories has been one of its notable achievements. This destructive role of econometrics does serve a constructive end, however: the identification of the best available model for a particular purpose at any point in time.


Procrustean Econometrics: or Stretching and Squeezing Data
David F Hendry and Grayham E Mizon

Discussion Paper No. 68, October 1985 (ATE)