How DTN scaled weather forecast data to petabytes per day

by | Jun 28, 2022 | Technology

We are excited to bring Transform 2022 back in-person July 19 and virtually July 20 – 28. Join AI and data leaders for insightful talks and exciting networking opportunities. Register today!

Weather forecasting is one of the oldest data challenges. Data scientists are constantly exploring how new modeling techniques or better data architectures could enable more acute and timely forecasts. Consumers might only be interested in knowing whether to bring an umbrella or sunscreen, but better weather forecasts can help businesses of all types improve operations and reduce the impact of severe weather events. 

DTN, the largest private weather service, has recently taken advantage of new Amazon Web Services (AWS) high-performance compute services to dramatically scale its weather forecasting pipeline size, accuracy and timeliness. These improved forecasts are already showing dividends for DTN’s operational intelligence services for agriculture, shipping, utilities and other industries. 

“Our ability to utilize the cloud and high computing scalable infrastructure is improving our ability to leverage weather forecasts by the day,” DTN’s vice president of weather operations, Renny Vandewege, told VentureBeat. 

Vandewege said that weather data currently powers a $2 billion — $3 billion market for operational intelligence services and is growing quickly

A combination of several new AWS services have helped DTN to scale the amount of data it processes from terabytes to petabytes per day; increase resolution from 10 km down to finer 1 km “pixels;” and increase the number of forecast from two per day to four per day. DTN plans to deliver hourly updates in the near future. 

Moving to the cloud

DTN started moving more of its data infrastructure to the AWS cloud to help scale its data and operational intelligence offerings. However, it still relied on its own managed supercomputers to power all of its forecasts. Over the last few years, Vandewege’s team started working with Amazon on a proof of concept to run high-resolution models using parallel clusters. 

The new DTN improvements take advantage of new AWS Hpc6a instances built for tightly coupled high-performance computing (HPC) workloads. This allows DTN to assemble a virtual supercomputer across virtual machine …

Article Attribution | Read More at Article Source

Share This