Log management

Log management (LM) comprises an approach to dealing with large volumes of computer-generated log messages (also known as audit records, audit trails, event-logs, etc). LM covers log collection, centralized aggregation, long-term retention and log analysis (in real-time and in bulk after storage).

Systems administrators usually perform LM analysis for reasons of security, of operations (such as system or network administration) or of regulatory compliance.

Effectively analyzing large volumes of diverse logs can pose many challenges — such as huge log-volumes (reaching hundreds of gigabytes of data per day for a large organization), log-format diversity, undocumented proprietary log-formats (that resist analysis) as well as the presence of false log records in some types of logs (such as intrusion-detection logs).

Users and potential users of LM can build their own log management and intelligence tools, assemble the functionality from various open-source components, or acquire (sub-)systems from commercial vendors.

Log Management Key Features and Technology

In order to deploy a Log Management architecture that will meet your corporate Chad strategy, Your need may be Security, Operations or Application log analysis, The organization need first take the following steps:

• Step 1: Define the requirement and goals, motivation can be Security log analysis, application problem analysis, reports and more.

• Step 2: Define the logging framework, log types and system specification where logs are generated.

• Step 3: Determine according to your goals, are you going to collect the logs, or analyze, report and monitor the logs on remote machine. If you plan on collecting log data, How long should it be archived, encrypted or not.Regulatory compliance may provide specification for such needs.

• Step 4: What information and intelligence are you planning to extract out of your logs, end user patterns reports, application problems or more.

• Step 5: Evaluate technology and vendors solution to select a production the best fit your needs, you may also select to build a log management solution internally leveraging open source solutions. Add only reporting and analytics layer later on for intelligence.

Deployment life-cycle

One view of assessing the maturity of an organization in terms of the deployment of log-management tools might use successive categories such as:

• Level 1: in the initial stages, organizations use different log-analyzers for analyzing the logs in the devices on the security-perimeter. They aim to identify the patterns of attack on the perimeter infrastructure of the organization.

• Level 2: with increased use of integrated computing, organizations mandate logs to identify the access and usage of confidential data within the security-perimeter.

• Level 3: at the next level of maturity, the log analyzer can track and monitor the performance and availability of systems at the level of the enterprise — especially of those information-assets whose availability organizations regard as vital.

• Level 4: organizations integrate the logs of various business-applications into an enterprise log manager for better value proposition.

• Level 5: organizations merge the physical-access monitoring and the logical-access monitoring into a single view.

See also

• Log management knowledge base
• Server log
• Web log analysis software
• Web counter
• Data logging
• Common Log Format
• Syslog

References

• Chris MacKinnon: "LMI In The Enterprise". Processor November 18, 2005, Vol.27 Issue 46, page 33. Online at http://www.processor.com/editorial/article.asp?article=articles%2Fp2746%2F09p46%2F09p46.asp, retrieved 2007-09-10