View larger image

IEM Daily Archiving

16 Apr 2025 05:30 AM
The IEM attempts to archive a lot of data each day. The goal being to make accessible long term archives available for both funded research projects that support the IEM and also for anybody else to freely use. This archiving activity takes on a number of different forms with a "flat file" directory tree being the subject of today's Daily Feature. This directory tree is organized by UTC date and subsequent sub-folders. The predictable and stable URLs allow for automated downloads and scraping. The featured chart presents the daily total of archived file size, the accumulated storage, and some annotated dates explaining why the archive sizes increased at that time. There are some practical constraints to archival capacity, so the scope of archival has not changed much over the past five years or so. There is an associated ISU project called MTArchive with a similar directory structure, but archives more meteorological specific datasets. MTArchive also has a different funding model, so thus why the service is separate.

Voting:
Good: 22
Bad: 0

Tags:   iem   archive  

View larger image

2024: Billions of Observations

31 Dec 2024 05:30 AM
To close out 2024, today's featured chart presents an estimate of the amount of new data added during 2024 and stored within the IEM PostgreSQL databases. Note that the numbers presented are not exactly point in time observations, but a measure of the number of database rows within various tables/relations. Please also note that the plot uses a logarithmic axis for readability. A small army of computer servers attempt to process numerous data streams into forms to be stored within a database. The featured chart totals an estimate of just under five billion rows of data added for the year! Most all of this data is made immediately available for download via numerous APIs and services. It is a considerable challenge to keep all of these data flows going, but the community feedback remains very positive with many usage cases at Iowa State University and beyond supported by this effort. Here's to another year of keeping the IEM chunking along!

Voting:
Good: 16
Bad: 0

Tags:   2024   iem   archive