blog

Summary NASA has been migrating over 100 petabytes of data from on prem systems to the cloud for the last several years and is now able to focus on measures of efficiency that were not possible before. Earthmover and Development Seed worked with NASA to pilot Earthmover’s open-source Icechunk tensor storage engine to optimize cloud-native data access for level 3 datasets on top of existing archival files without a costly and complex data migration. When applied to the popular GPM IMERG precipitation dataset, this technology enabled a 100x speedup for extracting time series data compared to existing approaches for the same task, while also simplifying the end-user experience. Opportunity: Existing archival data formats are not designed for high-performance access in cloud object storage …