Data Cube System Requirements

The focus of this document is to provide potential users with a reference for the type of systems that the Data Cube is targeting and what sort of performance they can expect for various investment levels. Covered topics include basic requirements that are common to all Data Cube systems, methods for estimating hardware requirements, example systems, and benchmarks of our current systems. Using this document, a user should be able to create an estimate of their hardware needs based on their requirements and get an idea of the performance that they will be aiming for.

Basic System Requirements

While the Data Cube is flexible regarding basic requirements, all scalable Data Cube systems will have some of the same properties:

Storage

Storage space scales with amount of data that the system is responsible for managing. For a general estimate on the storage space required, we use the following general formula for Landsat data:

*Estimate = (number of path/rows) * (365/revisit time in days) * (years of storage) * (data_size) * 1.5 (margin_of_safelty_multiplier) *

Using a baseline of 22 acquisitions per year (16 day revisit) and a scene size of 1.2Gb (unzipped, uncompressed) with a multiplier of 1.5 to account for the ingested dataset, this yields roughly 40Gb per path-row per year. That means for 15 years of historical data over a single path row, around 600Gb of storage would be required.

For example, we have 170 Landsat 7 scenes stored along with the ingested data. This amounts to 202Gb for the scenes and 36Gb for the ingested data. This is slightly below the threshold described above, leaving some room for error. This figure can be reduce by storing archived and compressed original datasets rather than uncompressed data. This is a figure to be used for general estimation and should be padded by a factor of 3 - 4 to account for general storage, the storage of analysis output products, and to prevent bumping into storage limits in the future.

Memory and CPU

The ability to run concurrent processes is based on the number of cores, and the size of each concurrent process depends on the amount of available memory. Analyses on large datasets often require splitting a large geospatial and temporal region into smaller ‘chunks’ for more efficient processing. A ‘chunk’ in this context refers to a smaller portion of a larger dataset; large analysis regions are broken into many smaller subsets and processed independently. For example, a region made up of four square degrees may be split into four one square degree regions and processed concurrently to decrease analysis time. The chunk size can be manipulated by area or time slices: For Landsat data, pulling in one square degree for one acquisition is around 400Mb of memory. In a low memory setup, it is possible to chunk analyses into smaller geographic chunks, e.g. splitting a one square degree analysis into ten 0.1 square degree analyses and combining the end result.

The general guidelines for baseline system requirements are below:

System summaries

As summary of CEOS Data Cube systems can be found below:

Location CPU Memory Storage Uses and Capabilities
AMA Offices 1x 8 core processor 64Gb 8Tb Development server for 1-4 users, including UI hosting and Jupyter notebooks
AMA AWS Instances 2x 36 core processors 128Gb 16Tb Server capabilities (UI hosting) and processing capabilities for ~20-30 users (estimated)
Colombian IDEAM Facilities 32x 8 core processors, two per blade 1536Gb 24Tb Data access/analysis for ~100 or more users at environmental agencies

This includes a local development server, our demonstration server hosted on AWS, and the Colombian IDEAM system. We have benchmarked our own systems and provide some data points below. The estimated number of concurrent users for our home office system is based on the number of active Data Cube developers while the AWS instance capabilities are estimated using the numbers provided by our Colombian colleagues. The number of supported users could be stretched in all of the systems listed above by smaller chunking and longer processing times.

Primitive Benchmarking

We have performed some sample tasks on the systems we have available in order to provide some reference for how various systems perform with various tasks. These are all based on UI task execution times, so they are fully parallelized and chunked into 10-200 chunks for multicore processing. Larger systems are capable of higher concurrency with the same number of active users. While execution times may be similar between systems, multiple concurrent jobs could be executed at once on the higher number of core systems. This difference can be seen with the tasks that require a larger number of chunked tasks like the median pixel custom mosaic; the median pixel mosaic splits into 200 tasks, while the most recent pixel splits into 10. Execution time is referring to a full (2000-2016) analysis over a 1 degree square in Colombia. TSM was done on a 1 degree square on Lake Chad. There are no differing application settings between machines to make use of more cores, so the execution times of our AWS servers could be optimized at the expense of the number of supported concurrent users if desired.

These analysis cases involve the processing of 91 scenes for mosaicking, fractional cover, and water detection and 314 scenes for the TSM analysis.

System Custom Mosaic (Median) Custom Mosaic (Most Recent) Water Detection TSM Fractional Cover
AMA Offices[1] 767s 177s 175s 597s 361s
AMA AWS Instance (1 Node) 298s 67s 127s 357s 113s
AMA AWS Instance (2 nodes) 189s 62s 106s 318s 99s

[1] AMA Office Setup: Linux server with 64GB RAM, 64-bit 1.2 GHz Intel latest generation Xeon, 8 cores with hyperthreading processor, 8TB Hard Drive.

It can be seen above that having more cores decreases the processing time significantly due to the larger machines (AWS) having a sufficient number of cores to concurrently process all chunks at once. There is a clear drop off when comparing 36 and 72 cores, as most of the analyses were only split into 10-20 chunks. The main distinguishing point between the two AWS benchmarks is the ability to concurrently process more tasks at once. This is seen with the median pixel custom mosaics; the performance gain is still significant between the two AWS benchmarks, signifying the ability for the AWS instances to process more concurrent users.

Appendix

Additional Examples:

Included below is a table of raw data-volume for satellite imagery based on varying spatial and temporal requirements.

Landsat 7 SR product

1 path/row 5 path/rows 10 path/rows 15 path/rows 20 path/rows 30 path/rows
1 yr 39.6 gb 198.0 gb 396.0 gb 594.0 gb 792.0 gb 990.0 gb
5 yr 198.0 gb 990.0 gb 1.98 tb 2.97 tb 3.96 tb 4.95 tb
10 yr 396.0 gb 1.98 tb 3.96 tb 5.94 tb 7.92 tb 9.9 tb
15 yr 594.0 gb 2.97 tb 5.94 tb 8.91 tb 11.88 tb 14.85 tb
20 yr 792.0 gb 3.96 tb 7.92 tb 11.88 tb 15.84 tb 19.8 tb

Sentinel 2 Level-1C

1 path/row 10 path/rows 20 path/rows 30 path/rows 40 path/rows 50 path/rows
1 yr 36.5 GB 365.0 TB 730.0 TB 1.095 TB 1.46 TB 1.825 TB
5 yr 182.5 GB 1.825 TB 3.65 TB 5.475 TB 7.3 TB 9.125 TB
10 yr 365.0 GB 3.65 TB 7.3 TB 10.95 TB 14.6 TB 18.25 TB
15 yr 547.5 GB 5.475 TB 10.95 TB 16.425 TB 21.9 TB 27.375 TB
20 yr 730.0 GB 7.3 TB 14.6 TB 21.9 TB 29.2 TB 36.5 TB

MODIS Level 1B

1 path/row 10 path/rows 20 path/rows 30 path/rows 40 path/rows 50 path/rows
1 yr 120.45 GB 1.2045 TB 2.409 TB 3.6135 TB 4.818 TB 6.0225 TB
5 yr 602.25 GB 6.0225 TB 12.045 TB 18.0675 TB 24.09 TB 30.1125 TB
10 yr 1.2045 TB 12.045 TB 24.09 TB 36.135 TB 48.18 TB 60.225 TB
15 yr 1.80675 TB 18.0675 TB 36.135 TB 54.2025 TB 72.27 TB 90.3375 TB
20 yr 2.409 TB 24.09 TB 48.18 TB 72.27 TB 96.36 TB 120.45 TB