DeepZoom

A Microsoft Azure Function which gathers wind and temperature predictions from NOAA, then collates them for use by navigational software.

General

• Azure Function
  • developed using an HTTP trigger
  • in production, runs on a hourly timer
  • run time is a couple of minutes
• Uses Python 3.8 and:
  • azure-functions
  • azure-storage-blob
  • numpy
  • pygrib

Workflow

1. read result information (such as time and date) from previous run (JSON)
2. Check NOAA for latest data
   • nomads.ncep.noaa.gov/cgi-bin/filter_gfs_0p25.pl
   • parse through the HTTP to find latest available data folder,
   • return date, time, and list of available files
3. see if date/hour has changed, and enough files available
   • sometimes the new folder is present but not all files are present (we ask for 48 hours)
   • exit if nothing new
4. Download files
   • create local directory for downloads
   • set levels and vars to only download the data we need
     • 100x savings in bandwidth and disk space
5. Create many individual grids, 10 degrees by 10 degrees (at .25 resolution = 40x40)
   • loop over latitude, longitude
   • create temp directory
   • for each file, get data for that grid
     • I developed custom pygrib/numpy code to quickly slice and dice
   • proprietary code to compress to 8 bits per measurement
     • convert u,v (wind) to norm, direction
     • temperature
   • append to outputfile
   • at end of loop, delete download files
6. "Layer" the grids
   • read all the little files
   • add to an io.BytesIO
   • upload to a blob using containerClient
   • delete temp directory
7. create new JSON status document