Application of information theory to measure and increase the skill of long-term forecasting

The research program proposed here aims to systematically apply information theory metrics to post-processing and evaluation of long-range forecasts, with the goal of quantifying and increasing their usefulness to end users. To support climate adaptation, NOAA and other national and international forecast centers are providing long-range forecasts based on increasingly sophisticated ensembles and super-ensembles of dynamical model runs. For such forecasts to be useful to end users, post-processing must be applied to move from often biased model outputs to calibrated probability distributions for quantities of interest that calibrate model output based on the observational record and past forecasts or hindcasts. The concept of information gain (IG) from a baseline probability distribution function (PDF) for the quantity of interest to a refined PDF that incorporates model predictions offers an intuitive measure of the skill of models at long-range forecasting that has a solid theoretical basis and provides an objective function for optimally combining multiple dynamical forecasts with climatology and statistical patterns.

The main components of the proposed work are (1) evaluate IG of current and archived forecast products compared to suitable baseline PDFs based on simple statistical models that incorporate persistence and trends; (2) develop and test generally useful methods for constructing maximally informative PDFs from available single-model or multimodel ensemble forecasts; (3) evaluate the statistical uncertainty of expected IG computed from finite available samples; (4) compare IG to other widely used metrics for the ranking of forecast models and post-processing methods in order to understand the behavior and respective advantages of different methods. We will test and demonstrate our methods with existing sets of archived model forecasts and hindcasts, including NOAA’s Climate Prediction Center seasonal forecasting product and the new NCEP Climate Forecast System Version 2, focusing on the seasonal (<1 year) prediction timeframe for which more independent calibration data are available. We will deliver not only publications describing our results but an open-source software tool to apply information metrics for the post-processing and evaluation for any given forecast problem that has available historical calibration data, facilitating the adoption and further development of information metrics by diverse research and applications communities.

Our project goals are closely aligned with the “intra-seasonal to decadal climate prediction” MAPP competition’s priority area of achieving “an objective comparative evaluation of climate prediction skill . . . to assess optimal prediction methodologies for specific applications”. We believe that the research and software tool proposed here will tangibly advance the MAPP Program objective of “developing integrated assessment and prediction capabilities relevant to decision makers”, and through it NOAA’s goal of a “Weather-Ready Nation” achieved through delivering relevant environmental information.