Data ingestion and storage strategies for data warehouses in the context of data streaming: an overview of recent advances

Abstract

In the current landscape of data-driven decision-making, data warehouses have proven to be highly valuable tools, especially with the emergence of big data characterized by its volume, velocity, and variety. This study provides a systematic review of data ingestion and storage strategies for data warehouses in the context of data streaming, focusing on the latest advancements and methodologies to address the challenges posed by continuous data streams. Modeling schemas of data warehouses are not the focus of recent studies, as existing schemas seem to satisfy current needs. Instead, the focus has shifted towards optimizing operational processes like ETL (Extract, Transform, Load) and join operations. The review highlights techniques such as parallel processing, in-memory computing, and distributed computing as critical to enhancing data ingestion and storage capabilities. This work synthesizes recent research, providing insights into how modern data warehouses can efficiently process and store streaming data to support real-time analytics and decision-making. The findings offer valuable guidance for developing scalable and efficient data warehousing solutions.