Optimizing Content Searching over Archive Files for Multi-cores

Abstract

Capabilities of computers continue to increase in a rapid pace with new concepts and technologies. Computer processing power, memory capacity, storage capacity and data/memory access speed have been significantly improved. However most of the useful Linux commands do not gain these advantages due to their initial sequential design. If it can be modified or re-designed to be compliant with modern technologies, community can benefit.

The grep is a very effective pattern searching command used by Linux users. However it is used for general text formats and it cannot be directly applied on special file formats. Other utilities have been developed to address specific requirements by wrapping the grep command. There are several such utilities to search patterns in compressed files, and they decompress and feed into the grep. The zgrep (from ZUtils), zgrep (from gzip), bzgrep, xzgrep and zipgrep are some utilities and they can work on one or more compression formats. Those utilities can be time consuming and they can be improved by applying optimization techniques such as parallelism etc. This project is targeted to optimizing content searching on compressed file formats on multi-core systems.

Community has developed good compression/decompression technologies (pigz, lz4, plzip, lbzip2 etc), and command-line parallelism technologies (GNU make, GNU pexec, GNU parallel etc). We have utilized these innovations to provide a high level solution to optimizing content searching on compressed file formats.

Using high level solutions which are selected by the analysis, we provide a command based and GUI based utility targeting to give some relief to the users in harnessing the power of multi-cores for content searching on compressed, archived formats. As well as our work may also be helpful to motivate other innovative research for the problem.