Research Output
A Comparison of the Efficiencies of various Structured and Semi- Structured Data Formats in Data Analysis and Big Data Analytic Development
  As data volumes grow, so too does our need and ability to analyse it. Cloud computing technologies offer a wide variety of options for analysing big data and make this ability available to anyone. However, the monetary implications for doing this in an inefficient fashion could surprise those who may be used to an on-premises solution to big data analysis, as they move from a model where storage is limited and processing power has little cost implications, to a model where storage is cheap but compute is expensive. This paper investigates the efficiencies gained or lost by using each of five data formats, CSV, JSON, Parquet, ORC and Avro, on Amazon Athena, which uses SQL as a query language over data at rest in Amazon S3, and on Amazon EMR, using the Pig language over a distributed Hadoop architecture. Experiment results suggest that ORC is the most efficient data format to use on the platforms tested against, based on time and monetary costs.

Citation

Peng, T., & Graham, H. (2024, July). A Comparison of the Efficiencies of various Structured and Semi- Structured Data Formats in Data Analysis and Big Data Analytic Development. Presented at DATA 2024: 13th International Conference on Data Science, Technology and Applications, Dijon, France

Authors

Keywords

Athena, Cloud, Big Data, Data Format, Semi-Structured, Efficiency, EMR

Monthly Views:

Available Documents