Search Optimization in Large Flat Files

Abstract

Handling flat files with large size, is time consuming to search using basic sequential search. This project proposes to use data structure for an optimized implementation of flat file content search. Searching in flat files is a wide area, where most of researches have worked on. This approach is different from indexing, which is the usual approach in flat file database systems. Search in flat files can be done usually going through the file and loading that to the memory fully. So our expectation was to implement a searching mechanism without loading the file in to the memory in an optimized way. This project presents a search mechanism designed, using tree approach to optimize search but with specific type of query. Many searching algorithms have been implemented and compared to find out the best data suitable data structure and it was proved that hash table implementation is the most efficient data structure ,but with few limitations. Basic sequential search was beatable up to a specific file size, but when the file size is growing largely, it was not beatable. So hash table is suitable only for a specific range of file size. It s proved that using fragmented files, other than a single file is more efficient. Hash table with fragmented multiple flat files is the most suitable implementation, if the content of the flat file is known. If the content is unknown, this implementation is not efficient. This report also describes an example system where this search mechanism can be applied to optimize its work.