Model Diagnostics button_banner.fw

Mission

The Model Diagnostics Task Force is constituted to develop, coordinate, and implement process-based model evaluation metrics and a metrics framework in National modeling center metrics packages, leveraging ongoing efforts at the modeling centers toward advancing model evaluation and development capabilities.

This Task Force is constituted of researchers whose projects were successfully evaluated as part of the FY18 competition held by NOAA’s Modeling, Analysis, Predictions, and Projections program on the topic of Process Oriented Metrics. This competition was motivated by a number of factors, including community interest in moving beyond performance oriented metrics toward process oriented metrics, ongoing efforts to develop next-generation climate and Earth system models, evolving plans for CMIP, and a need to link model development and evaluation efforts across modeling centers. Nine projects were selected as a result of the competitive process, including eight focused on developing metrics and one that will develop metrics as well as define a direction and framework for the overall effort.

Task Force Function and Implementation

MAPP Task Force Concept and Terms of Reference

Leadership

Overall lead: David Neelin, University of California, Los Angeles
NCAR lead: Andrew Gettelman, National Center for Atmospheric Research
GFDL lead: John Krasting, NOAA Geophysical Fluid Dynamics Laboratory
Co-lead: Allison Wing, Florida State University
Co-lead: Yi Ming, NOAA Geophysical Fluid Dynamics Laboratory
Co-lead: Eric Maloney, Colorado State University


Participants

Relevant MAPP Program PIs and selected additional invitees.
To view the full Participants list, please visit the Participants page.

Projects

To view the full list of Projects, please click here.

News & Events

Powerful New Software Helps Expedite Weather and Climate Forecasting Improvements

Monsoon shower in New Mexico. Credit: John Folwer. Used under a Creative Commons Licesnse 

Much like a doctor tries to diagnose the cause of a patient’s symptoms, a team of NOAA-funded researchers has been helping to diagnose the causes behind poor weather and climate model forecasts. Now, thanks to the team’s new model evaluation software, scientists can more quickly and easily identify the source of model errors to accelerate improvements and help Americans better plan and respond.

Developed by the NOAA Climate Program Office’s Model Diagnostics Task Force (MDTF), the software package was recently transferred to NOAA’s Geophysical Fluid Dynamics Laboratory (GFDL) where it has already led to significant advancements in model performance, including forecasts of regional precipitation and extreme events like monsoons. 

“Many long-standing model biases have been cut roughly by half,” said David Neelin, UCLA Professor of Atmospheric and Oceanic Sciences and MDTF Lead, about the new software. “These improvements are critical for advancing a number of NOAA priorities.”

Diving deeper into weather and climate model errors

Americans are increasingly seeking accurate weather and climate forecasts from NOAA. But to improve model errors and provide more reliable forecasts, scientists can’t just fix temperature or precipitation—there’s no single knob or equation in the model controlling each variable. Model developers need to go deeper and address issues with underlying physical and chemical processes in the models that drive these errors.

With this in mind, the NOAA Climate Program Office’s Modeling, Analysis, Predictions, and Projections (MAPP) program, in partnership with GFDL, created the Model Diagnostic Task Force (MDTF). The Task Force brings together researchers and scientists from universities, national modeling centers, and research laboratories to focus on “diagnostics” that evaluate or diagnose the error source in different atmospheric, land, and ocean processes in models.

Creating the MDTF was motivated by the 2012 Working Group on Numerical Experimentation (WGNE) Task Force, which advocated for a diagnostic framework focused on the Madden Julian Oscillation—an eastward moving disturbance of clouds and rain that circles the tropics every 30-60 days. Since 2015, the MDTF has expanded on that concept and has been developing software that collects diagnostics for many key weather and climate model processes—called Process Oriented Diagnostics (PODs)—produced by researchers at universities and modeling centers, into a central framework that can be readily used as a model evaluation tool. 

The image illustrates multiple weather climate and model process areas included in the new Model Diagnostic Task Force software package.

“The idea is to work closely with modeling centers to come up with a way to enable the modeling community outside of the centers [...] to easily contribute their diagnostics,” said Eric Maloney, Colorado State University Professor and MDTF Co-lead, during a 2017 interview with NOAA. “Then, modeling centers will be able to apply the diagnostics to their simulations to hopefully lead to fantastic modeling improvements in an accelerated fashion.”

Now at GFDL, the package is also moving to an open development framework on GitHub to encourage diagnostic contributions from the public, facilitate the exchange of ideas between modeling centers and the scientific community, and make it easier for researchers to integrate their work. By engaging the modeling community through a collaborative approach, the package could help quicken the transition of research innovations into advances in life-saving forecasts and projections. 

“Ultimately, 10 years down the line, I think that model diagnosis work [...] may be some of my most important contributions,” Maloney said.

An evolving evaluation software for more accurate forecasts and projections

The MDTF’s evolving software package has eight different diagnostics implemented, 11 under development to be included by 2022, and 28 proposed which may be implemented by 2024. Current diagnostics help tackle things like atmospheric convection, clouds, and radiative processes, while near-term diagnostics will cover tropical and extratropical cyclones and phenomena like the El Niño-Southern Oscillation. In the longer-term, future diagnostics will address key processes that drive our climate’s sensitivity to greenhouse gas emissions and other processes critical to modeling marine ecosystem variability and change.

“A lot of the diagnostics are aligned with key NOAA priorities—high impact water events, extreme weather events, weather and climate events that have high societal and economic impacts,” said John Krasting, who now oversees the development of the package at GFDL. “This package allows users of our models and model developers to explore NOAA mission-relevant processes and identify how we’re doing and ways we can improve.”

Development of the software framework for process-oriented diagnostics was most recently supported by the National Oceanic and Atmospheric Administration (NOAA) Climate Program Office Modeling, Analysis, Predictions and Projections (MAPP) Program (grant # NA18OAR4310280). Additional support was provided by University of California Los Angeles, the Geophysical Fluid Dynamics Laboratory, the National Center for Atmospheric Research, Colorado State University, Lawrence Livermore National Laboratory and the US Department of Energy.

All of the process-oriented diagnostics modules (PODs) were contributed by members of the NOAA Model Diagnostics Task Force with MAPP support.

 

 

Print

x


MAPP


FY20-Footprint

Contact

Dr. Daniel Barrie
Acting MAPP Program Director
P: 301-734-1256
E: daniel.barrie@noaa.gov

Courtney Byrd
MAPP Program Specialist
P: 301-734-1257
E: courtney.byrd@noaa.gov

Wenfei Ni
MAPP Program Specialist
P: 
E: wenfei.ni@noaa.gov

Dr. Annarita Mariotti
MAPP Program Director, on detail to EOP/OSTP
P: 301-734-1237
E: annarita.mariotti@noaa.gov

«May 2021»
MonTueWedThuFriSatSun
262728293012
3456789
10111213141516
17181920212223
24252627282930
31123456

CPO HEADQUARTERS

1315 East-West Highway Suite 100
Silver Spring, MD 20910

ABOUT US

Americans’ health, security and economic wellbeing are tied to climate and weather. Every day, we see communities grappling with environmental challenges due to unusual or extreme events related to climate and weather.