Real-time market data warehousing platform for data visualization

Abstract

Data warehousing is a data management system that supports business intelligence activities like analytics. It collects data from heterogenous data sources and store this data while providing capability to analyze users. The existing warehousing system in the LSEG surveillance system is not supported for market data analysis on real-time basis. It keeps separate component to analyze and commit that data to storage. This is an outdated model as existing warehousing model has limited features when compared to modern warehousing technologies. Through out critical analysis of utilization, limitations, benefits of modern technologies for warehousing system implementation, technical design has been finalized with two hypotheses. Experimental based analysis has been performed to satisfy two hypothesis – new system developing with Apache Impala and Kudu can perform faster query response than existing system, and the new system can perform analytical queries for generating data summarization graphs without help of third component for data summarizing. With set of proper experimental query execution for both existing and new system above two hypotheses have been satisfied successfully. But since the algorithm used for data summarization by existing summarization process is client legacy algorithm, the two summarization graphs don’t equal exactly point to point– but it has been able to provide visual summarization output like existing summarization graph. So, implementing existing summarization methodology can be model as UDF which is fallen into future works of the project.