70 - Getting ahead of the curve
The sudden change gave the virus an unfair advantage.
Germany in April 2021: in the midst of one of the strongest Covid-19 waves in Germany, the Robert Koch Institute decided to change the data format for the output of daily vaccination figures without prior notice. This made waves, especially among science and data journalists, as it meant that web scraping programs set up to use the old output format no longer worked overnight. As a result, the RKI data could no longer be crawled automatically and produced nothing but error messages. In this case, the sudden change in format and content led to considerable extra work for the journalists affected and to outages on websites and dashboards that millions of people were using to find out about the pandemic.
Data providers should be proactive and provide advance notice of upcoming format changes. For example, publications can be made available in the old and new formats for a transitional period so that data users can adapt. Good planning of the data schema and the data formats to be used in advance also helps to avoid unnecessary major changes in the course of the research process.
- Open Data Anti Patterns - Hase und Igel: https://github.com/transportkollektiv/opendata-antipatterns/blob/main/patterns/formataenderung.md
- https://x.com/datentaeterin/status/1380203124858699778