Converting data organised for visual perception into machine-readable formats

Abstract

Spreadsheets are used to store an extraordinary amount of important data. The fact that spreadsheets are both easy to use and allow users a great deal of flexibility in how they store their data is a significant reason why they are so popular. Users often use a variety of layout techniques to make the data easy for humans to understand. But this layout also creates problems for traditional Extract-Transform-Load (ETL) tools. We propose a program that allows users to easily extract data from Excel files by selecting the cells containing the data and metadata thereby determining the data hierarchy. We have used this program to extract data of the Agricultural Structure Survey on land use and livestock in Germany, which does not follow a nationwide standard, leading to large differences in the structuring of the data between the federal states, making it a good benchmark.