OBSERVER: Data cubes: Enabling and facilitating Earth Observation applications

OBSERVER: Data cubes: Enabling and facilitating Earth Observation applications

OBSERVER: Data cubes: Enabling and facilitating Earth Observation applicationsAnnekatrien Debien
Thu, 14/10/2021 – 13:10

Earth in cubes. (Source: Daniel Diaz at Pixabay)

There is no doubt that we live in an era of data. Data are all around us — we download them on our smartphones and use them for various purposes. They help us forecast weather and plan crop treatment, cities use data to better plan their infrastructure and transport or improve living conditions. The Copernicus programme, with its full, free and open data policy has been a game changer in terms of data availability, unleashing new opportunities for small and medium-sized enterprises (SMEs). Thanks to the Copernicus programme’s data and products, SMEs around the world can make commercial use of satellite data and derived products for diverse services and applications.

However, for inexperienced users such as SMEs or start-ups, handling large volumes of data and integrating data streams from different sources into their own internal environment can be a significant challenge. They often lack the technical or scientific skills needed to process and integrate Copernicus data and products and thus do not make the most out of the economic opportunities the Copernicus programme offers.

How can data cubes help?

Data cubes can be a real game-changer in the use of Earth Observation (EO) data. The volumes of data provided by EO are significant. Transferring and storing large quantities of data is never easy, and that is where data cubes are extremely helpful. Generally, data cubes are multidimensional matrices in which data are grouped or combined. They not only allow to view data in a multidimensional way, but also allow pre-processing and fast access to summarised data.

Data from different sources are converted into a uniform resolution and bound together by certain selected characteristics. Once in the cube, the different layers of EO data are stored and processed in a common gridded multidimensional database, which makes further use of the data easier. The data don’t have to be transformed again and can directly be used for computation. This is especially handy for machine learning, but also for integration and assessment of data, or for any other occasion when fast access to analysis-ready data is required.

When it comes to Copernicus, data cubes can be a real asset in boosting user uptake of its data and products, making them available in a more user-friendly and ready-to-use way.

Data cubes and Copernicus

Using data from Copernicus can be challenging for several reasons: because of the large amounts of data provided (as mentioned above) and because the data have to be processed to higher-level products and/or combined with products from Copernicus services to fully unlock their potential. The usability of the data expands with every transformation. However, those transformations require highly developed technical skills, such as knowing where to find the data, as well as programming experience to perform the transformations themselves. Furthermore, there are important software and hardware requirements to data cube operations, since large disk space and ample processing power are necessary. Uptake of satellite data and Copernicus products by the general public is thus limited by these factors.

Storing and processing data from the Copernicus Sentinel satellites and information products from the Copernicus Services in data cubes lifts many barriers for less experienced users. After the corrections and processing actions in the cube, users are provided with highly standardised and harmonised time series of satellite data available through a user-friendly interface. Moreover, the cloud-based storage and processing allow the download of ready-to-use data for a specific region without the need for the demanding pre-processing steps or the download of all data files.

Several data cube projects are underway. They are dedicated to increasing the availability of Copernicus data and encouraging a broader group of users to take advantage of the many opportunities it offers.

For example, the Euro Data Cube platform was created to lower the technical barriers that are often present when extracting useful information from Earth Observation (EO) data and preparing them for analysis. It combines EO data from various sources in one place, including data from the Sentinel satellites, data from other commercial very-high-resolution satellite missions, as well as products and data from Copernicus services and other EO initiatives. 

Visualisation of an Earth system data cube by ESA. (Source: Euro Data Cube Facility service: the ultimate EO resource for researchers and value-adders)

Several data sets from the Copernicus services are available in the Euro Data Cube through xcube-gen, a service/software that converts data from various sources into a data-cube-friendly format. Others can be generated upon request by xcube-gen and made available as persistent, user-defined data cubes in the Euro Data Cube (EDC). Find out more about Euro Data Cube and its functionalities on their website.

Another data cube project, the H2020 project Data Cube Service for Copernicus (DCS4COP), is more specialised in Copernicus data. It is designed to address the challenges of developing downstream services using Copernicus big data by integrating data from the Copernicus programme in a data cube system.

A market-ready service was created in 2018 by the H2020 project DCS4COP — the EODataBee. This service integrates Sentinel data, Copernicus service products and user supplied data in a data cube system. It offers environmental information, particularly on water quality and focussing on the coastal and inland water markets. Find out more about DCS4COP and its functionalities here.

EODataBee’s water quality product visualised in the XCUBE viewer, showing the image and time-series view. (Source: DCS4COP)

EODataBee’s water quality product visualised in the XCUBE viewer, showing the image and time-series view. (Source: DCS4COP)

Finally, another data cube project which includes Copernicus data is the Earth Server Data Cube — a global federation providing a uniform data space for spatio-temporal Earth data. The federation presents itself as a location-transparent, open community of large-scale Earth data providers, including research and supercomputing centres, companies and agencies. Its goal is to create a “single point of entry” for Earth data and services for the entire planet. Copernicus Sentinel archives are available in the Earth Server Data Cube to mix and match with many other data assets. As a consequence, the Earth Server offers petabytes of data of a great variety through its growing network , such as radar and optical satellite data, atmospheric data, elevation data and thematic cubes like that for global sea ice.

Assessing the impact of the COVID-19 pandemic with data cubes

The COVID-19 pandemic has dramatically affected and changed the patterns of many aspects of human life on Earth. Mobility, business, production and many more activities came to a halt or were drastically reduced because of the restrictions that were put in place to stop the spread of the virus. Data helped decision makers understand what effect these measures had.

The RACE (Rapid Action on Coronavirus and EO) dashboard is an online platform developed by the European Commission and the European Space Agency that demonstrates how the use of Earth observation data can help shed new light on societal and economic changes currently taking place due to the coronavirus pandemic. The dashboard showcases examples of how different analyses of Copernicus Earth Observation data, ground-based observations and advanced numerical models can illustrate these socio-economic and environmental changes across all EU and ESA Member States. It not only captures the effects of the lockdown, but also shows how Europe is on the road to recovery and relaunching activities.

In this context, data cubes can be very handy. The applications on the platform use various data sources, including the previously mentioned Euro Data Cube. Using the Euro Data Cube, a team of EO experts analysed thirty of the busiest European airports and prepared a tool that monitors change in airplane traffic compared to pre-pandemic levels using satellite data. The airplane traffic indicator incorporated in the RACE platform is based on the detection of airplanes in images captured by the Copernicus Sentinel-2 satellites. This information on the online platform can be accessed by anyone. The observed changes have interestingly reflected a shift in human and economic activities due to COVID-19 and are of particular interest when assessing the impact of COVID-19 on our society.

Air traffic indicator visualised on the RACE platform. (Source: Twitter – ESA Earth Observation)

The Euro Data Cube also provided a specially configured environment for the COVID-19 Custom Script Challenge by ESA, the European Commission and the Sentinel Hub. The contest ran in April and May 2020 and aimed to uncover ideas for how satellite data could help monitor and mitigate the situation linked to the pandemic and contribute to finding solutions to facilitate recovery and adaptation to the crisis in three main areas: activity of economic operators, human activity distribution and agricultural activity. In this context, the Euro Data Cube team provided a dedicated environment that included a large array of EO and other relevant data, as well as data viewing and processing tools.

More information about the contest and the winners can be found here.

Powering future EO applications with data cubes

Fast analysis and seamless integration of data is becoming increasingly necessary in these rapidly changing times. Data cubes help make data processing easier and lift the burden of harmonising data from different sources by placing them in a common “cube” framework, ready to be analysed.

With continuously increasing volumes of big data and products, data cubes can help increase the market uptake of Copernicus data. Projects like DCS4COP or Euro Data Cube help users process, combine and analyse data from multiple sources. This broadens the range of information that can be processed and makes the analysis of the results more powerful. It also makes data analysis accessible to SMEs, NGOs and other types of organisations that can benefit from the potential offered by Copernicus data but that lack the technical skills or facilities to fully exploit it.

The high volumes of data that can be analysed in a common grid enable users to quickly react to ongoing events such as the COVID-19 crisis and assist policymakers and governmental authorities to monitor the situation and find solutions.

As the Copernicus programme continues to provide ever more reliable and accurate data, having become the world-leading EO data provider, data cubes are likely to become an important tool in enabling and facilitating downstream EO applications that provide solutions to current and future societal challenges.

Thu, 14/10/2021 – 12:00

Read More