When operating a single server or a small number of servers it is, for an administrator, achievable to monitor all individual logfiles of the operating system. Administrators operating larger numbers of servers do however have a challenge. Regularity checking all logfiles to ensure all your servers behaving as expected and no suspicious activities are taking place in no longer an option. It is humanly not possible to monitor thousands of operating system deployments.
When operating a large footprint of, for example, Oracle Linux deployments there is the operational and security need to monitor logfiles. However, with only a certain number of administrators who maintain hundreds or sometimes thousands of servers you cannot depend on manual labor. Opening, reading and interpreting the content of multiple logfiles per server simply takes too much time, correlating different logfiles from different servers or searching on all servers for specific entries call for a more automated and centralized solution.
A number of solutions are available in both commercial and opensource solutions. Oracle Enterprise Manager provides a solution for example and is part of the wider Oracle Enterprise Manager framework for maintenance. Splunk provides a well know solution which enables operational teams a way to consolidate and mine all logging from operating systems and other components.
A new, and very interesting, solution is however provided by the people from ElasticSearch. ElasticSearch is currently rebranded to Elastic. As part of the software stack they provide a solution for the above mentioned challenge is provided in the form of LogStash. Logstash can be used to consolidate logfiles from all servers and place them in ElasticSearch. Another, freely available product named Kibana can be used to analyze the logging.
A simple deployment would look a bit like the high level representation below. All components used in this diagram are freely available.
Kibana is an open source (Apache Licensed), browser based analytics and search dashboard for Elasticsearch. Kibana is a snap to setup and start using. Kibana strives to be easy to get started with, while also being flexible and powerful, just like Elasticsearch. As soon as logfiles from your servers start being indexed by Elasticsearch you can start using Kibana to analyze them.
In the above example we stated that you can ship logfiles from the Linux operating system and consolidate / analyze them. However, the same applies for all logfiles on your system. This can include for example the logfiles your application server produces or the logfiles produced by your Oracle database.
Having a centralized logging solution in place which can consolidate all your logging and makes it easy to search and analyze can bring big benefits to your operational excellence and your ability to quickly manage large numbers of servers (and applications) with a limited number of staff. The beauty of the Logstash / Elasticsearch solution is that it is relatively cheap to implement and use.