Since the 1990s, the Earth System Grid Federation (ESGF) and its precursor, the Earth System Grid, have stored, indexed, managed, and delivered output from general circulation models, global climate system models, and Earth system models (ESMs) in support of the World Climate Research Programme’s evolving series of Coupled Model Intercomparison Projects (CMIPs). CMIP data underpins the United Nations Intergovernmental Panel on Climate Change’s (IPCC’s) comprehensive climate assessments released every six to seven years and also informs IPCC special reports focused on climate vulnerabilities, adaptation scenarios, and mitigation strategies.
ESGF standardizes and organizes observational and simulation data, allowing scientists to compare models against actual observations and reanalysis. More than 40,000 users—including scientists and policymakers—from 2,700 sites on six continents are sharing data through ESGF. More than 45 petabytes of data have been downloaded to the climate community through the federation, making it one of the most complex, successful big data systems in existence.
ESGF combines grid-based computing with a distributed architecture, keeping participating members sovereign while simultaneously linking them together. To achieve this, ESGF developers created a unique system of nodes that requires very little explicit coordination while still providing a robust “data space” for storage and computation. A key to ESGF’s success is its ability to effectively produce, validate, and analyze research results collaboratively, so that, for example, new results generated by one team member are immediately accessible to the rest of the team, who can annotate, comment on, and otherwise interact with those results.
While ESGF began at Livermore, it is now a proudly multi-national effort. The Department of Energy’s (DOE’s) current contribution to ESGF on behalf of the United States—known as ESGF2—is a collaboration among Oak Ridge, Lawrence Livermore, and Argonne national laboratories. LLNL computational scientists contribute in three key areas: 1) modernization of ESGF’s web search and other back-end applications, 2) development of data hosting and data provision tools, and 3) management of data preparation and publishing workflows.
What’s next
As computational and storage technologies rapidly increase in capability and capacity, enabling ESM simulations of increased complexity and fidelity at higher resolutions, the demands on the ESGF global peer-to-peer network of model output data servers have grown significantly. ESGF2 computational scientists are working with international federation partners on a series of upgrades that will make using the data easier and faster while improving how the information is curated. Their work will leverage the latest software tools, cloud computing resources, the world’s most powerful supercomputers, and DOE’s Energy Sciences Network for high-speed data transfer.
Learn more about ESGF
- ESGF: Supporting Climate Research Collaboration (LLNL web article)
- ESGF website
- ESGF launches effort to upgrade climate projection data system (LLNL news article, 2022)
- Interconnecting a World of Petabytes (LLNL magazine article, 2018)
