How well does a particular model work?
One form of model validation is to blindly test how well a fitted
model predicts future, or otherwise unknown, outcomes.
Once we have those future, or previously unknown, data at hand
we can retrospectively judge a model's ability to predict those
data.
Essentially we are challenging if our 'best' model is a good-enough
, or even useful, model.
Such verification presumes that the statistical error characteristics
of the future or unknown data are the same as for the data used
to estimate the model parameters.
Weather forecasters use retrospection to evaluate the accuracy
of their predictions, then improve them.
Biologists use retrospection to evaluate the accuracy of fishing
or hunting harvest forecasts or the accuracy of trends in population
abundance, particularly for exploited or endangered species.
Retrospection is a powerful and conceptually simple diagnostic
tool that relates directly to the purpose of the model.
It should be used whenever it is feasible.
|