Data retrieval and dynamic data organizing solution for a high volume, high performance Electronic Trading Systems.

Abstract

The Electronic Trading System that is being discussed here is owned by Millennium Information Technologies Limited. It has been developed to automate gigantic, time critical trading systems and bench marked as the fastest trading system in the world. Service Desk (SD) is a component process of this application. Querying is the primary functionality offered by SD. It provides querying facility on various market data such as orders and trades. The main objective of this project is to improve the query processing performance of the SD process for large volumes of data.

In the current system, SD process functions in such a way that, it uses a serial query processing mechanism. As the key methodology to improve the SD query processing time, a tremendously designed data parallelizing mechanism has been introduced. Parallel architecture has been implemented based on the popular Master Slave paradigm.

SD process is decomposed in to single Master process and multiple Slave processes at multiple data sources used for query processing. This architecture not only improves the query processing but also increases the scalability.

A prototype application was developed using GNU C++ language as a POC. Due to certain intellectual property rights, it is prohibited to expose client’s proprietary software to outside. Hence, simulators (ex: middleware used for messaging) were developed from the scratch and no third party software tools were utilized.

Various tests and evaluations that performed with this prototype revealed valuable as well as interesting observations. Test results compared with the existing SD process statistics and observed that the new architecture/solution shows a super linear speed up. These findings may assist in decreasing the query processing latencies. It helps Millennium IT to reach their one of the goals that, bench mark the product to the next level as the fastest query processing product on high volume data.