OBSERVER: Putting the ‘quality’ in air quality measurements
OBSERVER: Putting the ‘quality’ in air quality measurements
evan
Thu, 11/07/2024 – 01:07
The information provided by the Copernicus Atmosphere Monitoring Service (CAMS) is critically important to a wide range of users across Europe and the rest of the world. CAMS data on air pollution and health, solar energy, greenhouse gases, and climate forcing are made even more reliable through a range of evaluation processes. These are based on comparisons of CAMS data with a wide variety of atmospheric measurements, recorded by instruments on satellites, aircraft, balloons, and on the ground, and conclude that the CAMS data are of consistently high quality. This week’s Observer delves into the complexities of forecasting air pollutants and allergens, the importance of quality control in CAMS’s operations, and the independent evaluations that increase user confidence in its products and information.
Forecasting air pollutants and allergens is a challenging task
The forecasts and analyses from CAMS include a wide range of harmful pollutants associated with human activities, such as transport and industry, and from natural sources such as wildfires and volcanoes.
Air pollution is one of the biggest risks to human health. According to the World Health Organization, air pollution is associated with 6.7 million premature deaths each year.
In Europe, around one quarter of the people suffer from hay fever. As well as being debilitating for individuals, the burden on the health care system stemming from this condition can be significant. Estimates suggest that the costs of treatment and missed workdays can be as much as around €150 billion each year.
Some air pollutants can be transported by the wind for large distances, across and between countries and continents. For example, smoke from wildfires in North America can reach into the Arctic and as far as Europe, and dust from the Sahara Desert is frequently carried across the Mediterranean to Europe and can be transported across the Atlantic as far as the Caribbean.
Chart showing daily maximum dust particulate matter concentration at ground level, originating from the Sahara Desert and being transported across continental Europe. Credit: European Union, Copernicus Atmosphere Monitoring Service
Airborne pollutants have a complex, non-uniform spatial distribution which is influenced by geography, weather patterns, and human activity, so their concentration in the atmosphere varies over time. One example is diurnal variation such as those caused by morning and evening ‘rush hour’ traffic peaks. There is also the so-called ‘weekend effect’, particularly seen in Europe, where there is less pollution on Sundays.
As pollution levels are heavily influenced by weather, there are also seasonal changes. For example, in winter months, cooler temperatures and stable atmospheric conditions result in less mixing within the atmosphere, while emissions can be affected by the increased use of heating and power generation. Pollution accumulates and concentrations of particulate matter can increase close to the surface. During the summer, heatwaves and other periods of warm weather can lead to the production of ground-level ozone.
CAMS aims to accurately model the start and end date of pollen seasons, long-range transport of airborne particulate matter, how air quality will change with the weather, and if pollution levels will be under or over set reporting thresholds, in addition to other valuable air quality metrics.
Given the inherent complexity and dynamic nature of atmospheric conditions, as well as the diverse user base of CAMS products, maintaining the high quality of forecasts is paramount, as is ensuring that users can trust the accuracy and consistency of the information they are using. This is why evaluation and quality control (EQC) throughout all levels of the CAMS service is crucial.
Plot for 4-10 May 2024 showing what was modelled for diurnal variation in ground-level ozone against what was observed. Credit: European Union, Copernicus Atmosphere Monitoring Service
Quality evaluation has been a priority right from the start
From the very first days of the European precursor projects that ultimately led to the creation of CAMS, quality evaluation activities have been considered a key element of the service. Initially, the focus was on development work—for example, the work done for the individual models used to predict the movement in the atmosphere of reactive gases such as ozone and carbon monoxide. Now, evaluation activities are carried out on all parts of the service by specialist teams working autonomously, independent of the CAMS product developers.
‘The evaluation process has evolved over many years in parallel with the work that CAMS undertakes. Giving users an independent view of the quality of data products has been a valuable step in building confidence,’ explains Henk Eskes, Senior scientist at the Royal Netherlands Meteorological Institute (KNMI), who coordinates the evaluation of global data on behalf of CAMS.
Evaluation at all levels
The CAMS evaluation process is based on comparing the CAMS products to independent observational data that have not been used to produce the CAMS forecasts and looking at how closely the CAMS air pollution analysis reproduces those measurements.
The observational data used to create and evaluate these forecasts are collected from numerous sources, including satellites, devices fitted to commercial aircraft, radiosondes carried by weather balloons, and ground-based measurement stations.
Though most of the data gathered are used to produce the forecasts and analyses, a proportion is used solely for evaluation purposes, to independently verify that the results are accurate. For example, the daily pollen forecasts are compared with independent observational data from around 100 ground stations run by the European Aeroallergen Network.
The teams carrying out evaluations look at the big picture and at each metric in detail, making multiple comparisons. These include average pollution levels for regions or countries, year-to-year and day-to-day variations, extreme events such as major wildfires or dust storms, and instances where air quality thresholds are exceeded. The aim is to use the highest-quality measurements available, including existing and emerging data from ground-based stations in South America, Africa, and Asia. Special effort is put into extending the database of independent observations and acquiring new datasets for the evaluation activities.
Specialist teamwork
As part of its routine EQC process, CAMS reviews both global and regional analyses and forecasts to ensure that the output meets expectations. Reporting is carried out on each individual modelling system as well as the regional ‘ENSEMBLE’, which takes all 11 separate European air quality models used to produce the CAMS European near-real time air quality forecasts together to generate a single combined forecast.
In addition, the CAMS global and regional forecasting systems are continually developed and upgraded, with a new version released each year. Before an upgrade is implemented, extensive testing is carried out. The new version of the model is run in parallel with the operational service for six months in the case of the global system and for one month in the case of the regional system. Both systems are evaluated against independent observations to verify that the new versions show improvement in their ability to forecast air quality.
The regional evaluation is performed by six partner organisations, coordinated by The Norwegian Meteorological Institute (MET Norway). These forecasts deliver information on air quality, with hourly analyses and one four-day forecast every day. Due to the much more diverse set of observations involved, there are 12 partners who evaluate the global models. Around 65 different datasets are used, supplied by measurement networks worldwide. These include observations of pollutants, aerosols, reactive gases, and greenhouse gases. Partner organisations are selected based on various factors including experience in specific types of observation.
Regular reporting of quality assessments
For users and data providers, regular evaluation of CAMS data quality is crucial. Regional and global reports are created using a variety of quantitative metrics, visual graphs, and data tables. These reports evaluate the individual models and the overall forecasts on a daily, weekly, seasonal, and annual basis against observational data from surface and airborne measurements, as well as remote sensing data. All this information is shared on the CAMS website and on the dedicated CAMS regional and global evaluation servers.
The daily evaluation is published online, allowing for a rapid understanding of how the models are performing. A more in-depth evaluation is undertaken quarterly, using all available datasets. The datasets which are used for evaluation are submitted for their own quality assurance process, but this sometimes takes over a year to conduct. CAMS has therefore set up special agreements with measurement providers to improve and automate this process whenever possible. This ensures a steady supply of good quality measurements, enabling close-to real time assessment of the CAMS products.
In the longer-term, annual EQC reports are made for the CAMS reanalysis products. The reanalysis products use a combination of models and historic observations to recreate past concentrations of polluting gases, aerosols, and greenhouse gases. The evaluation teams look at the stability of these reanalyses, checking for drifts against observations and their overall quality.
The quarterly and annual CAMS evaluations are shared via written reports, and the teams producing regional evaluations have also developed specific plots and charts of how observations compare to the models to help users understand the data.
Screenshot of the CAMS global evaluation server, showing an evaluation of the ozone forecast for the month of April. Credit: European Union, Copernicus Atmosphere Monitoring Service
What quality evaluation means for users
Just like weather forecasts, air quality forecasts can be inaccurate, but the invaluable work of the EQC teams ensure that the quality of forecasts and the observational data feeding into them are constantly being monitored and evaluated.
Daily evaluation updates mean those responsible for managing and developing the models can receive near real-time feedback on performance, enabling a rapid response to any concerns, such as problems with input data resulting in an abrupt degradation of the quality of the outputs of a particular model. The models are extremely complex, so being alerted to any problems in the input data and being able to take corrective measures is critical for ensuring that they maintain a high-level of performance.
‘For our end users, what is most important is that CAMS and its partner organisations are prioritising the quality of the products. Knowing that a forecast is highly likely to be correct is invaluable. By bringing together experts from across Europe, we can consistently deliver quality information,’ concludes Richard Engelen, Deputy Director of CAMS.
Thu, 11/07/2024 – 12:00
