Unidata 2021 Equipment Award Report

Rick Cruse
Daryl Herzmann

Unidata is a community program out of UCAR that provides data and software tools to universities. The program is funded by the National Science Foundation.

Unidata offers a yearly competitive equipment grant award to encourge adoptation of their software tools and enhancements of data services offered to the community. The IEM submitted a successful proposal for the 2021 award. This page summarizes the status of deliverables outlined in the proposal.

The proposal funded the purchase of a Dell R7525 server modelled after the specification used by LetsEncrypt to increase database server capacity. The server was purchased in early July 2021 with the following specs:

The server was deployed by mid July with the intention of running ZFS and PostgreSQL over the NVMe drives. While the performance of this combination was an order of magnitude better than what the IEM had in place at the time, it was not as fast as using a more conventional approach of XFS over software RAID10. Not to delve too deeply into the technical reasons why, but the lack of ZFS Direct IO Support seemed to be the main culprit. Nevertheless, the server was put into production by the end of July 2021 and meet the proposal goals.

Goal: Increase the capacity for processing and delivering hydrological data

The IEM attempts a robust processing of SHEF formatted hydrological data provided by NOAA/NWS via Unidata's IDD data stream. Prior to this grant, the IEM download portal had some arcane throttles in place to prevent the server from being overwhelmed by requests. These throttles were removed and server successfully delivers thousands of requests per day.

Related to this work, but not directly funded by this proposal, were software enhancements made to the open source pyIEM library to directly process SHEF encoded data. Previously, a convoluted process of piping the data to an ancient FORTRAN library called SHEFIT was used. The new approach is faster, more robust, and includes a test suite to ensure the library is working as expected.

Goal: Increase capacity for serving data requested by Unidata's Siphon

Unidata's Siphon python library allows users to easily access various internet data sources. One of the supported data sources is IEM's Upper Air sounding archive. The server purchased allowed this server to perform much better and again, with out any throttles in place.

Goal: Increase web service capacity for serving data to the community

Unidata community members utilize a number of web services and tools provided by the IEM website. These include:

The purchased server partially supports all of these within the IEM webfarm and has improved the performance of each. Addtionally, besides running a PostgreSQL database server, this server also runs a python FastAPI server that provides the IEM API.