Log Management Best Practices And Systems

Managing logs can be a complex endeavour. Not only does a typical organization generate a ton of them, but they do come from a variety of sources, each with a potentially different format and containing different information. To put a semblance of order into something that can quickly get chaotic, log management was invented. Today, we’re having a look at the log management best practices and systems. We hope that it will help you see clearly through this.

We’ll start off by a short description of log management. Then, we will dive right into the best practices of log management. We’ll explore whether you should use a ready-made system or do it yourself. We will also have a look at what—and what not—to monitor, followed by log security and retention as well as storage considerations. And before we review some of the best log management systems, we’ll have a look at the various management tasks, the review and maintenance of logs, the correlation of data sources, and some automation considerations.

About Log Management

Simply defined, a log is the automatically-produced and time-stamped documentation of an event relevant to a particular system. When an event takes place on a system, a log—or log entry—is generated. Different systems will generate logs for different events. As for log management, it generally refers to the processes and policies used to administer and facilitate the generation, transmission, analysis, and storage of log data. Log management typically implies a centralized system where logs from multiple sources are aggregated.

Log management is not just log collection, though. As the name implies, the management part is important. Once logs are received by the log management system, they are “translated” into a common format. It is necessary as different systems format logs differently and include different data in their logs. To make searching and event correlation easier, one of the purposes of log management systems is to ensure that all collected log entries are stored in a uniform format.

Talking about searching and even correlation, this is another major feature of most log management systems. The best log management systems feature a powerful search engine. It lets administrators zero-in on exactly what’s needed. Furthermore, event correlation will automatically group related events, even if they are from different sources.

Log Management Best Practices

Log management is a complex process, there’s not much we can do about it. With this complexity comes the risk of doing it wrong. To avoid that, we’ve compiled a list of some of the best practices of log management. Our goal is to give you as much information as possible in order to choose the best log management system for your needs but, more importantly, to get the most out of it.

Log Management System Or DIY?

For some reason, some people believe that they can manually implement a “log management system”. If you’re among these people, stop kidding yourself immediately. Although it is possible to implement some form of log management manually, the required efforts far outweigh what’s required to implement a true log management system. And with several free and open-source tools available, the argument of cost is not a valid one.

It almost always makes sense to use a managed logging solution that is built, supported, and scaled by a reputable vendor rather than building out a system on your own. With them, all you typically need to do is connect your sources and destinations and you’re ready to analyze system and application logs the easy way. You’ll be free to spend more time monitoring and logging rather than building out your logging infrastructure.

Knowing What To Monitor (And What Not)

Knowing what to log is important, but it is even more important to know what not to log. Just because you can log something doesn’t necessarily mean you should. Logging too much often does nothing more than making it harder to find data that actually matters. Furthermore, the extra volume of logs adds complexity and cost to your log storage and management processes. It is important to think ahead about what will and won’t be logged before starting to implement a log management platform. It will prevent costly mistakes and will allow you to better size your tool.

Consider carefully what you actually need to log. Production environments that are critical for compliance or for auditing purposes should most likely be logged. So should data that helps you troubleshoot performance problems, solve user-experience issues or monitor security-related events.

Conversely, there is stuff that you do not need to log like, for instance, test environments that are not an essential part of your business processes. There is also data that you will choose not log for compliance or security reasons. For example, if a user has enabled a do-not-track setting, you should not log data associated with that user.

Implementing A Log Security And Retention Policy

Logs may contain sensitive data. For that reason, you need to have a log security policy. It will be invaluable in, for instance, ensuring that sensitive data gets anonymized or encrypted. Also, the secure transport of log data to log management systems mandates the use of encrypted transport using TLS or HTTPS on the client and on the server side.

As for a retention policy, logs from different sources or systems might require different retention times. For instance, logs that are primarily used for troubleshooting may work with relatively short retention times such as a few days—or even a few hours. On the other hand, security-related logs or business transaction logs require longer retention times, often for regulatory compliance. Considering this, your retention policy should be flexible and adaptable, depending on the log source or type of log.

Log Storage Considerations

Keeping log data uses up valuable storage space. When planning the storage capacity for logs, you need to consider high load peaks. In most circumstance, the amount of log of data per day is relatively constant. It mainly depends on system utilization and/or the number of transactions per day. However, when something goes wrong, you can expect accelerated growth in the log volume. If your log storage has limits that you exceed, you could lose the latest logs. To mitigate this effect, the best log management systems use a cyclic buffer. It deletes the oldest data first before any storage limit is applied.

Also, log storage should have its own security policy. Most attackers will try to avoid or delete their traces in log files. To avoid that, you should ship logs in real-time to the central log storage—preferably off-site—and secure it. Thus, if an attacker has access to your infrastructure off-site logs will keep the evidence untampered.

Reviewing And Maintaining Logs

Log maintenance is an important part of log management, if not the most important part. Unmaintained logs can lead to longer troubleshooting, data exposure risks, and higher log storage costs. Review the logs generated by your systems and adjust the logging level to your needs. You should consider usability, operational and security aspects.

Make log level configurable

Some system logs are too verbose while others don’t provide enough information. Unfortunately, there isn’t always something you can do about it. Most systems provide adjustable log levels. They are the key to configure the verbosity of logs and ensuring that what has to be logged is and what is not important isn’t.

Inspect audit logs frequently

Acting on security issues is crucial. This is why one should always have an eye on logs. If your log management system doesn’t have that feature—many of them do, use external security tools such as auditd or OSSEC. They implement real-time log analysis and generate alert logs pointing to potential security issues. And in addition to that, you should define alerts on critical events in order to be notified quickly on any suspicious activity.

Correlate Data Sources

Logging is only one element of a global monitoring strategy. For truly effective monitoring, you need to complement log management with other types of monitoring like monitoring based on events, alerts and tracing. Doing that is the best way to get the whole picture of what’s going on at any point in time. While logs are good for providing high-definition detail on issues, this is most useful when you take some distance to look at the forest before zooming into the trees.

Log management doesn’t work well in a silo. Nothing does. You should most definitely complement it with other types of monitoring such as network monitoring, infrastructure monitoring, and more. And in an ideal world, your monitoring solution should be comprehensive enough to provide all your monitoring information in one place. Alternatively, it could integrate with other tools that provide this information. The goal here is to have, as much as possible, a single-pane view of the entire environment.

Log Management And Automation

Log management can help you catch issues early on thereby saving you and your team valuable time and energy. It can also help you find opportunities for automation. Most log management tools will let you set up custom alerts that trigger when something happens. Some will even let you set up automated actions to be initiated when these alerts are triggered. You should use as much automation as your management tool will allow. Despite the time you’ll spend setting up this automation, you’ll find that it was well worth it the first time you encounter an incident.

The Top 6 Log Management Tools

We’ve scoured the market trying to find the best log management tool. We’ve tried to put together a list which includes various types of tools. After all, everyone’s needs are different and the best tool for one is not necessarily the best for someone else.

1. SolarWinds Security Event Manager (FREE TRIAL)

SolarWinds is a common name in the field of network administration tools. It’s been around for about two decades and it has brought us some of the best bandwidth monitoring tools and NetFlow analyzers and collectors. The company is also well-known for publishing several free tools that address some specific needs of network administrators such as subnet calculator or a syslog server.

When it comes to log management, The company’s offering is now called the SolarWinds Security Event Manager. It was recently renamed from Log & Event Manager, probably to better reflect the fact that this is actually much more than just a log management system. Many of its advanced features put it in the Security Information and Event Management (SIEM) range. It has, for instance, real-time event correlation and real-time remediation, two SIEM-like features.

Let’s have a look at some of the SolarWinds Security Event Manager’s main features. The tool can eliminate threats quickly using instantaneous detection of suspicious activity and automated responses. It can also perform security event investigation and forensics for mitigation and compliance. And talking about compliance, the product will allow you to demonstrate it, thanks to its audit-proven reporting for HIPAA, PCI DSS, and SOX, among others. This tool also has file integrity monitoring and USB device monitoring, two features that are way above what we commonly see in log management systems.

Prices for the SolarWinds Security Event Manager start at $4,585 for up to 30 monitored nodes. Licenses for up to 2500 nodes can be purchased making the product highly scalable. And if you want to verify hands-on that the product is right for you, a free, full-featured 30-day trial is available.

2. SolarWinds Papertrail (FREE PLAN AVAILABLE)

In second place, we have another great product called Papertrail, a recent acquisition by SolarWinds. Papertrail is a popular cloud-based log management system. It aggregates log files from a wide variety of popular products like Apache or MySQL as well as Ruby on Rails apps, different cloud hosting services and other standard text log files. Papertrail users can then use the web-based search interface or the command-line tools to search through these files to help diagnose bugs and performance issues. The tool also integrates with other SolarWinds products such as Librato and Geckoboard for graphing results.

Papertrail is a cloud-based, software as a service (SaaS) offering from SolarWinds. It is easy to implement, use, and understand. And it will give you instant visibility across all systems in minutes. The tool has a very effective search engine that can search both stored and streaming logs. And it is lightning fast.

Papertrail is available under several plans including a free plan. It is somewhat limited, though, and only allows 100 MB of logs each month. It will, however, allow 16 GB of logs in the first month which is equivalent to giving you a free 30-day trial. Paid plans start at $7/month for 1GB/month of logs, 1 year of archive and 1 week of index. Noise filtering allows the tool to preserve data by not saving useless logs.

3. ManageEngine EventLog Analyzer

ManageEngine, another common name with network administrators, makes an excellent log management system called the ManageEngine EventLog Analyzer. The product will collect, manage, analyze, correlate, and search through the log data of over 700 sources using a combination of agentless and agent-based log collection as well as log import.

Speed is one of the ManageEngine EventLog Analyzer’s strength. It can processes log data at an impressive 25,000 logs/second and detect attacks in real-time. It can also perform fast forensic analysis to reduce the impact of a breach. The system’s auditing capabilities extend to the network perimeter devices’ logs, user activities, server account changes, user accesses, and more, helping you meet security auditing needs.

The ManageEngine EventLog Analyzer is available in a feature-reduced free edition which only supports 5 log sources or in a premium edition which starts at $595 and varies according to the number of devices and applications. A free, full-featured 30-day trial version is also available.

4. Ipswitch Log Management Suite

The Log Management Suite is a product from Ipswitch, the same company that brought us WhatsUp Gold, an immensely popular network monitoring tool. This is an automated tool which collects, stores, archives and saves system logs, Windows events, and W3C/IIC logs. Furthermore, its continuous log surveillance will alert you of any suspicious activity.

Frequently audited events such as access rights and file, folder and object privileges can be followed, generating alerts as needed and used to build compliance reports for HIPAA, SOX, FISMA, PCI, MiFID, or Basel II compliance. The tool can also help you transform your raw log data into meaningful data for managers or IT security teams, thanks to its automated filtering, correlating, reporting, and converting features.

Pricing information for the Log Management Suite is not readily available from Ipswitch. The product can be purchased either directly from the publisher or through Ipswitch’s reseller network. A free trial version is also available.

5. Alert Logic Log Manager

Alert Logic’s primary focus is on security and compliance. And since log management is closely related to both, it’s no surprise that the company offers the Alert Logic Log Manager. This cloud-based tool offers automated and unified log management across all your environments. It will collect, aggregate, and search log data from the cloud, server, application, security, and network assets.

The Alert Logic Log Manager includes log monitoring and analysis as well as log review which is done live by human analyzers. Alert Logic’s experts will alert you of possible threat activity 365 days a year. The service will also help meet the log review requirements of SOC 2, HIPAA, and SOX and offload the burden of reviewing logs and following up on events, to comply with PCI/DSS 10.6, 10.6.1, 10.6.3

Pricing information for the Alert Logic Log Manager is not readily available from the web and you’ll need to contact Alert Logic sales to get a formal quote. A free trial is also not available but a free demo can be arranged by contacting Alert Logic.

6. Nagios Log Server

You might already know Nagios as an excellent network monitoring package. Offered it a free and open-source as well as in a commercial version, the product has a solid reputation. For log management, Nagios‘ offering is called the Nagios Log Server. It is a complete package with centralized log management, monitoring, and analysis. This tool can simplify the process of searching your log data. It also lets you set alerts to be notified of potential threats Furthermore, the software has high availability and fail-over built right into it. Its easy source setup wizards can help you with configuring your servers and other devices to send their log data to the platform, allowing you to start monitoring your logs within minutes.

The Nagios Log Server provides easy correlation of log events across all logging sources in just a few clicks. The system will let you view log data in real-time, letting analyze and solve problems in real-time, as they occur. Another strength of the product is its impressive scalability. This tool keeps meeting your needs as your organization grows. If need be, additional Nagios Log Server instances can be added to a monitoring cluster, allowing you to quickly add more power, speed, storage, and reliability.

With all these features, one would expect a hefty price tag. It is not the case and the single-instance price for the Nagios Log Server is a very reasonable $3 995. Despite not offering a free trial, a free online demo is available, should you prefer to have a first-hand look at the product before making a purchase decision.