Heartbeat alert system for web based systems based on server log analysis

Abstract

The quality of a web system relies on various factors. In the early stage of the system, developers have to focus on some of the quality factors in the source code level. After making the system public, system administrators are also responsible for maintaining the quality of the service offered. High availability of the system is the one of the main concern administrators have to fulfill. This project is a study of the server behavior to achieve the high availability of the system. This study is focused on analyzing the server logs to identify server status and forecast any issues. Using the gathered information, we will propose a methodology to monitor the web system. At the end of the project a working prototype of the proposed solution will be delivered. The historical log analyzing setup consists of Apache access log store which has approximately 18GB log files and an Apache Hadoop single node server which is connected with Apache Hive setup. Analysis is performed to extract the data from the logs to compare the server behavior when the server is unreachable or in a critical state and when the server is operating smoothly. The analysis will introduce a measurement – bot density, which is a combination of the all session count and the automatic web site crawlers count within a specific time interval. Using this value, the developed system will monitor the web system to maintain the high availability of the system. At the end of the research, it will propose how to improve the methodology to derive more accurate value for the bot density. And how the other server information can be used to improve this system.