8 Best Log Management Software for Faster Troubleshooting
Today’s systems are generating a lot of logging data. On many platforms, every single event, important or not, is logged somewhere. Typically, logs are stored locally. This makes sense as logs are linked to their source. But when trying to troubleshoot issues and find their root cause, that often means we have to look at multiple log files on numerous devices. Wouldn’t it be nice if all the logs from all devices were stored in one place? Log management is that and a lot more, as you’re about to find out. And today, we’re reviewing the top log management systems.
We’ll start off by trying to explain what log management is. As you’ll see, it can be a lot more than just centralizing log storage. Next, we’ll talk about logging protocols. It is rather important as log management wouldn’t likely exist without them. We’ll then try to differentiate syslog servers from log management systems. Unfortunately, there is no clear demarcation between them. We’ll follow with a discussion on Security Information and Event Management systems because this is another type of system that is often confused with log management, thanks to the somewhat unclear definition of each. And finally, we’ll review the top eight log management systems we could find.
Log Management – What It Is
Before we can talk about log management, let’s see what a log is. Simply defined, a log is the automatically-produced and time-stamped documentation of events relevant to a particular system. Whenever an event takes place on a system, a log is generated. Different systems will generate logs for different events and many systems give administrators some degree of control over what generates a log and what doesn’t.
When we’re talking about log management, we’re referring to the processes and policies used to administer and facilitate the generation, transmission, analysis, storage, archiving and eventual disposal of large volumes of log data. Log management implies a centralized system where logs from multiple sources are collected.
But log management is not just log collection. The management part is the most important. Log management systems typically have multiple functionalities, collecting logs being just one of them.
Once logs are received by the log management system, they need to be “translated” into a common format. Different systems format logs differently and include different data in their logs. Some start a log with the date and time, some start it with an event number. Some only include a log ID while other include a full textual description of the event. One of the purposes of log management systems is to ensure that all collected log entries are stored in a uniform format. This will make searching and event correlation much easier down the line.
Talking about searching and even correlation, this is another important function of many log management systems. Some of them feature a powerful search engine that allows administrators to zero-in on precisely what they need. Correlation functions will automatically group related events, even if they are from different sources. How—and how successfully—different log management system accomplish that is a major differentiating factor.
Logging Protocols
Log management would be much more difficult, if at all possible, if it were not for logging protocols. A few of them exist that define what data is to be included in logs, how that should be formatted and how they should be transmitted between systems.
Syslog is arguably the most-used logging protocol. Invented in the early eighties, it has become the de-facto standard for Unix-like systems. One of the greatest assets of the syslog protocol is how it separates the software that generates logs, the system that stores them, and the software that reports and analyzes them. Using the Syslog protocol makes log management much easier. Many non-Unix devices such as switches routers and other networking equipment from many vendors use a variant of the syslog protocol.
Microsoft Windows, as you may have guessed, uses a different logging system. It might have to do with the fact that Windows operating systems and applications have logs that typically contain much more information than syslog permit. Fortunately, the Windows Event Collector functions provide a mean for log management systems can use to receive events from Windows hosts.
No matter what logging protocol is used, an important part of log management is configuring devices to send their logs to the management system. This is different from other tools such as network monitoring systems, where the tool fetches data from the hosts.
Log Servers Vs Log Management
Since it has been available on every Unix-like system for a quite a while, Syslog if often used as a log server with one computer receiving syslog data from several others. While this centralized storage of logs has definite advantages, it is not log management.
To deserve the Log Management System name, a product must include at least some of the more advanced functions. According to Wikipedia, log management is comprised of the following functions: log collection, centralized log aggregation, long-term log storage and retention, log rotation, log analysis, log search, and reporting. Log servers often only offer the log collection and storage and rarely more than that. Each of the log management systems on our top list offers at least some of the more advanced functions.
How About SIEM Systems?
Another popular technology that is often associated with logs and confused with log management systems is Security Information and Event Management, or SIEM. This is quite different from log management although it is closely related. In fact, some products advertised as log management systems are actually SIEM systems while some basic SIEM systems are nothing more than log management systems.
The main reason for that confusion is that log management—or at least, log analysis—is an important component of SIEM systems. In fact, SIEM systems typically take log management to the next level by adding some intelligence to the process. These systems perform log analysis with the ultimate goal of identifying security issues. They will, for instance, look for signs of unsuccessful logins which would indicate an unauthorized intrusion attempt. These systems will automatically scan log entries looking for anything unusual.
SIEM systems have more to do with IT security than IT management and while some do include extensive log management features, many can also use an external log management systems and it’s not uncommon to see both systems running side by side.
The Best Log Management Software
Now that we have a common understanding of what log management is and what it is not, let’s have a look at what’s available. We’ve searched the market for some of the best log management systems. Our initial finding is that there are a lot of them and many of them very good. But we only have so much space so we’re about to review the eight most interesting ones we could find.
1. SolarWinds Papertrail
SolarWinds is a common name in the field of network administration tools. It’s been around for almost 20 years and has brought us one of the best bandwidth monitoring tools and one of the best NetFlow analyzers and collectors. The company is also well-known for publishing several free tools that address some specific needs of network administrators such as subnet calculator or a syslog server.
A few years ago, SolarWinds acquired Papertrail, a popular log management system. It aggregates log files from a wide variety of popular products like Apache or MySQL as well as Ruby on Rails apps, different cloud hosting services and other standard text log files. Papertrail users can then use the web-based search interface or the command-line tools to search through these files to help diagnose bugs and performance issues. Papertrail also integrates with other SolarWinds products such as Librato and Geckoboard for graphing results.
Papertrail is a cloud-based, software as a service (SaaS) offering from SolarWinds. It is easy to implement, use, and understand. And it will give you instant visibility across all systems in minutes. The tool has a very effective search engine that can search both stored and streaming logs. And it is lightning fast.
Papertrail is available under several plans including a free plan. It is somewhat limited, though, and only allows 100 MB of logs each month. It will, however, allow 16 GB of logs in the first month which is equivalent to giving you a free 30-day trial. Paid plans start at $7/month for 1GB/month of logs, 1 year of archive and 1 week of index. Noise filtering allows the tool to preserve data by not saving useless logs.
2. SolarWinds Log & Event Manager (FREE TRIAL)
Our next entry is another product from SolarWinds called the SolarWinds Log & Event Manager. Contrary to our previous entry, this is a locally installed product. And it’s also much more than just a log management system. Many of the advanced features of this product put it in the SIEM range. It has real-time vent correlation and real-time remediation, for instance.
Here’s an overview of the SolarWinds Log & Event Manager’s main features. It eliminates threats quickly using instantaneous detection of suspicious activity and automated responses. It can also perform security event investigation and forensics for mitigation and compliance. And talking about compliance, the product will allow you to demonstrate it, thanks to its audit-proven reporting for HIPAA, PCI DSS, and SOX, among others. This tool also has file integrity monitoring and USB device monitoring, two features that are way above what we commonly see in log management systems.
Prices for the SolarWinds Log & Event Manager start at $4,585 for up to 30 monitored nodes. Licenses for up to 2500 nodes can be purchased making the product highly scalable. And if you want to verify hands-on that the product is right for you, a free, full-featured 30-day trial is available.
3. ipswitch Log Management Suite
The Log Management Suite is a tool from Ipswitch, the same company that brought us WhatsUp Gold, an immensely popular network monitoring tool. This is an automated tool which collects, stores, archives and saves system logs, Windows events, and W3C/IIC logs. Furthermore, its continuous log surveillance will alert you of any suspicious activity.
Frequently audited events such as access rights and file, folder and object privileges can be followed, generating alerts as needed and used to build compliance reports for HIPAA, SOX, FISMA, PCI, MiFID, or Basel II compliance. The tool can also help you transform your raw log data into meaningful data for managers or IT security teams, thanks to its automated filtering, correlating, reporting, and converting features.
Pricing information for the Log Management Suite is not readily available from Ipswitch. The product can be purchased either directly from the publisher or through Ipswitch’s reseller network. A free trial version is also available.
4. ManageEngine EventLog Analyzer
ManageEngine, another common name with network administrator, makes an excellent log management system called the ManageEngine EventLog Analyzer. The product will collect, manage, analyze, correlate, and search through the log data of over 700 sources using a combination or agentless and agent-based log collection as well as log import.
Speed is one of the ManageEngine EventLog Analyzer’s strength. It can processes log data at an impressive 25,000 logs/second and detect attacks in real time. It can also perform fast forensic analysis to reduce the impact of a breach. The system’s auditing capabilities extend to the network perimeter devices’ logs, user activities, server account changes, user accesses, and more, helping you meet security auditing needs.
The ManageEngine EventLog Analyzer is available in a feature-reduced free edition which only supports 5 log sources or in a premium edition which starts at $595 and varies according to the number of devices and applications. A free, full-featured 30-day trial version is also available.
5. Nagios Log Server
Nagios is best known for its excellent network monitoring software but its Log Server is possibly just as interesting. Aptly called the Nagios Log Server, it offers centralized log management, monitoring, and analysis. The Nagios Log Server simplifies the process of searching your log data. It also lets you set alerts to be notified of potential threats Furthermore, the software has high availability and fail-over built right in. Its easy source setup wizards will help you quickly configure servers to send all log data and start monitoring your logs in minutes.
The Nagios Log Server lets you easily correlate log events across all servers in just a few clicks. And it allows you to view log data in real-time, giving you the ability to analyze and solve problems as they occur. The product features impressive scalability and it will continue to meet your needs as your organization grows. Additional Nagios Log Server instances can be added to a monitoring cluster, allowing you to quickly add more power, speed, storage, and reliability.
The single-instance price for the Nagios Log Server is $3 995 and although a free trial doesn’t appear to be available, a free online demo is should you prefer to have a first-hand look at the product.
6. Alert Logic Log Manager
Alert Logic’s primary focus is security and compliance. And since log management is closely related to both, it’s no surprise that the company offer the Alert Logic Log Manager. This cloud-based tool offers automated and unified log management across all your environments. It will collect, aggregate, and search log data from the cloud, server, application, security, and network assets.
The Alert Logic Log Manager includes log monitoring and analysis as well as log review which is done live by human analyzers. Alert Logic’s experts will alert you of possible threat activity 365 days a year. The service will also help meet the log review requirements of SOC 2, HIPAA, and SOX and offload the burden of reviewing logs and following up on events, to comply with PCI/DSS 10.6, 10.6.1, 10.6.3
Pricing information for the Alert Logic Log Manager is not readily available from the web and you’ll need to contact Alert Logic sales to get a formal quote. A free trial is also not available but a free demo can be arranged by contacting Alert Logic.
7. LogDNA
Founded in 2015, LogDNA is the new kid on the block. The company claims that “LogDNA is the fastest, most intuitive, and cost-effective log management system”. It all starts with the installation which takes only a couple of minutes before you can start monitoring your logs. No matter how logs are generated and transmitted, hundreds of custom integration schemes are available to centralize logs into a single pane.
LogDNA can be cloud-based or self-hosted, depending on your preference. It is highly scalable and can handle hundreds of thousands of logs per second and dozens of terabytes per customer, per day in total security with real-time log analysis. The company and its products are SOC2, PCI, and HIPAA compliant as well as Privacy Shield certified.
With its simple, pay-per-GB pricing model which eliminates contracts and fixed data buckets, the company has one of the lowest total cost of ownership. Several subscription plans are available with increasing features. The bottom-tier plan is free and paid plans vary from $1.50/GB/month to $3/GB/month depending on the retention duration and the number of users. A free, full-featured 14-day trial is also available.
8. Graylog
Last on our list is a product called Graylog. The product offers many interesting features. The tool will parse and enrich logs and event data from any data source. Its processing pipelines allow for some flexibility in routing, blacklisting, modifying and enriching messages in real-time. Graylog will search through terabytes of log data to discover and analyze important information. The powerful search syntax lets you find exactly what you are looking for.
With Graylog, you can create dashboards to visualize metrics and observe trends in one central location. You can use field statistics, quick values, and charts from the search results page to dive in for deeper analysis of your data. The system also has the option to trigger actions or issue notifications on events such as such as failed login attempts, exceptions or performance degradation.
Graylog is available either as a free and open-source, feature-limited version which also has limited support or as an enterprise version with extended features and unlimited support. A trial license can also be obtained by contacting Graylog sales.