Tenshi: System Log Analysis
Introduction and Installation
The analysis of log files is a good compromise to the regular absence of consideration in companies for these files that contain crucial information for protecting systems against intrusions performed by hackers within the networks themselves.
Note: this document was written in 2006; certain versions and configurations of the software used may be different from those mentioned; please refer to the official project sites in case of problems.
Dedicating a person to this analysis is not economically feasible in many companies, so implementing software that automates this process as much as possible is a lesser evil, provided that the number of reported alerts remains reasonable while maintaining the quality of the alerts that will be escalated; this requires fine-tuning and is therefore often complex to implement.
LogWatch, Logsentry, Logcheck, and Swatch are the most well-known of these solutions in the open source world; through this document we will introduce Tenshi, a log analyzer formerly known as Wasabi.
Tenshi offers multiple functionalities: email reports, high processing capacity, use of regular expressions (perl regexp), exception handling and filtering.
These features are combined with a very simple installation and configuration flexibility that will delight those most resistant to this type of application: a single file based on a concept of objects or groups of objects, intuitive syntax and internal “crontab”.
Also worth noting is the minimal resources needed for this analysis, enabling the processing of multiple files with a large number of entries.
Debian
With Debian, Tenshi installs as follows:
|
|
Sources
For installation from the sources:
|
|
Tenshi is a program based on the PERL language; for sending reports via email, it requires the installation of the additional “Net::SMTP” module.
This module provides an API for the SMTP protocol; its installation can be done through another module that allows access to the CPAN archive (Comprehensive PERL Archive Network) which groups all homologated PERL modules together with their various documentation.
To start the CPAN connection:
|
|
For newcomers to using the CPAN module, you can retrieve and consult information related to the “Net::SMTP” module with the following command:
|
|
The installation of the module is done as follows:
|
|
We can see that the installation was performed correctly; we can now disconnect from CPAN:
|
|
Let’s look at the Tenshi archive that we previously retrieved; first decompress and unarchive it:
|
|
Then access the directory that was recently created and contains the files needed to install the application:
|
|
List the directory to see its structure:
|
|
The next two operations consist of creating a “tenshi” user and a group of the same name that will govern the execution of Tenshi on your system:
|
|
Before proceeding with the installation itself, it’s necessary, depending on the operating system and distribution, to adjust the configuration of the “Makefile” file; for me it was necessary to change the installation path of the manual pages by modifying line 9 of the file on a “debian-like” system:
|
|
Replace “mandir = /usr/man” with “mandir = /usr/share/man/”.
You can now install Tenshi on the system:
|
|
Configuration
Let’s examine the configuration file part by part to understand how Tenshi works, which will also introduce us to the mechanisms that govern log analysis.
Once the installation is complete, we can move on to the next step, which is configuration: this is done through a single file called “tenshi.conf” located at /etc/tenshi/tenshi.conf
|
|
At the beginning of the file, we find the “generic” configurations, including the setting of the previously mentioned “tenshi” user and “tenshi” group created on the system that will own the processes related to the future launch of Tenshi:
|
|
The following configuration line requires some adjustments. By default, we find “set pidfile /var/run/tenshi.pid”, but the files governing PIDs are located in the /var/run/
directory, which by default belongs to the “root” user:
|
|
The following operations must be performed to solve this problem:
- Create a directory specific to the Tenshi PID
|
|
- Set the necessary permissions for the future creation of PID files in this directory when Tenshi launches
|
|
- Verify that the previous operation was successful:
|
|
Then all that remains is to replace the line “set pidfile /var/run/tenshi.pid” with “set pidfile /var/run/tenshi/tenshi.pid” in the configuration file to avoid warnings related to the mentioned problem during future Tenshi launches.
The rest of the configuration allows you to define the log files that will be monitored and whose alerts will be reported to you; by default, you’ll find the following two lines:
|
|
Ideally, and to adjust according to the context of your distribution regarding the logging management of various events by “syslogd” or others, you can add the following lines to make this list as exhaustive as possible and the analysis more effective:
|
|
Next come some optimizations that you can leave by default, unlike the definition of the SMTP server that will be used to send reports if you do not want to use the default one running on “localhost”:
|
|
Tenshi has the unique characteristic of not using the system’s “crontab” for its operation; it’s therefore possible to configure different automation elements directly in the “tenshi.conf” file according to several criteria.
Elements of type “queue” are to be set up via the following syntax: “set queue []”; the default subject is “tenshi report” if you don’t specify specific ones.
Below you can see 6 elements configuring the sending from “webmaster@xxx@mycompany.com” to “xxx@mycompany.com” according to different frequencies and periods throughout the day based on their criticality.
For the “mail” element, scheduled sending at 6:30 PM of a report every day:
|
|
For the “nf” element, scheduled sending every thirty minutes of a report every day:
|
|
For the “report” element, scheduled sendings every 2 hours in the interval between 9 AM and 5 PM of a report every day:
|
|
For the “misc” element, scheduled sendings under the same conditions as the previous element:
|
|
For the “critical” element, scheduled sending immediately for each identified occurrence:
|
|
For the “root” element, scheduled sending immediately for each identified occurrence:
|
|
Sending reports to multiple users is also possible to configure; simply replace “xxx@mycompany.com” with “xxx@mycompany.com, root@xxx@mycompany.com” for example.
Following in the file are the exceptions that will allow you not to report information that is not of interest; they are sent to a “trash” element that is not part of the report sending “crontab”:
|
|
Similarly, we remove the indications for repeated entries:
|
|
An example group for the SSHD service; definition of the group:
|
|
All occurrences of the string “sshd: fatal: Timeout before authentication for” regardless of the following characters indicating that an SSH session has been closed due to inactivity for too long; this will be recorded in the reports belonging to the “report” element:
|
|
All occurrences of the string “critical ^sshd: Illegal user” indicating that an username not conforming to the system has been used for SSH authentication; this will be recorded in the reports belonging to the “critical” element:
|
|
All occurrences of the string “sshd: Connection from” regardless of the following characters indicating an established SSH connection; this will be recorded in the reports belonging to the “report” element:
|
|
All occurrences of the string “sshd: Connection closed” regardless of the following characters indicating a closed SSH session; this will be recorded in the reports belonging to the “report” element:
|
|
All occurrences of the string “sshd: Closing connection” regardless of the following characters indicating the ongoing closure of an SSH connection; this will be reported in the reports belonging to the “report” element:
|
|
All occurrences of the string “sshd: Found matching (.+) key: (.+)” regardless of the following and intermediate characters indicating that the key proposed for SSH authentication has been found in the list of system keys; this will be recorded in the reports belonging to the “report” element:
|
|
All occurrences of the string “sshd: Accepted publickey” regardless of the following characters indicating that the public key used for opening an SSH session has been accepted; this will be recorded in the reports belonging to the “report” element:
|
|
All occurrences of the string “sshd: Accepted rsa for (.+) from (.+) port (.+)” regardless of the following and intermediate characters indicating that the proposed RSA key has been accepted; this will be recorded in the reports belonging to the “report” element:
|
|
All occurrences of the string “sshd(pam_unix): session opened for user root by root(uid=0)” indicating the opening of an SSH session with the root account from the root account; this will be recorded in the reports belonging to the “root” element:
|
|
All occurrences of the string “sshd(pam_unix): session opened for user root by (uid=0)” indicating the opening of an SSH session with the root account; this will be recorded in the reports belonging to the “root” element:
|
|
And finally closing the sshd group:
|
|
On the same principles, several other groups are to be defined according to your system and its applications/services.
The “Sendmail” service with:
|
|
The “Sm-mta” service:
|
|
The “Ipop3d” service with:
|
|
The “Imapd” service with:
|
|
The “login(pam_unix)” service with:
|
|
The “su(pam_unix)” service with:
|
|
Then all records of the string “netfilter” for filtering will be sent to the “nf” element:
|
|
All records of the string “sudo” for commands executed under an account other than the active SHELL account will be sent to the “critical” element:
|
|
All records of the string “init” for service launches will be sent to the “critical” element:
|
|
All records of the string “passwd(pam_unix):” for password changes will be sent to the “report” element:
|
|
All other records not matching the previous identification regular expressions will be sent to the “misc” element:
|
|
Usage
With configurations done, we move on to using Tenshi by putting ourselves in concrete alert cases to receive the corresponding alarms on a predefined email provided that the basic configurations have been properly performed beforehand.
Tenshi functions as a service, which notably implies that you can close the terminal where you launched it without stopping it (equivalent to adding the “&” character at the end of a SHELL command).
During a previous “ls”, you may have noticed the presence of 3 initialization scripts for Linux operating systems (Debian and Gentoo distributions) and Solaris:
|
|
We are currently using a “Debian-like” distribution (Ubuntu), so we need to copy the corresponding initialization script into the /etc/init.d/
directory which contains all the service initialization scripts of the system:
|
|
Test this script:
|
|
Add execution rights to the script:
|
|
Start the Tenshi daemon:
|
|
Test the effective launch of Tenshi:
|
|
Stop Tenshi:
|
|
Forced stop of Tenshi in case of problems with the scripts:
|
|
Restart Tenshi:
|
|
Tenshi can also be launched outside of initialization scripts to, for example, test a configuration file or launch the service with an alternative file. Debug mode is also possible among others, and will be very useful if you encounter a problem particularly with sending emails,
|
|
The last point to address, and not the least, are the reports with the alerts that will be sent to you by email. The first is a report from the “report” element for a closure due to inactivity for too long on an active session of the SSH service:
|
|
The second example is a report for the “misc” element with an SSH authentication error, as well as the execution of a command by the system “crontab” and the opening of a “pam_unix” session for the root account associated with this previous entry “_/1 _ * * * root /usr/local/rtm/bin/rtm 41 >/dev/null 2>/dev/null” of the “crontab”:
|
|
The log analysis system is functional; all that remains now is to adapt the configuration according to your needs and the time you want to allocate to reading the reports from it.
If you would like additional information, two mailing lists are available:
The first is for general discussions; to subscribe, simply send a message to tenshi-user+subscribe@lists.inversepath.com and messages for the list should be sent to tenshi-user@lists.inversepath.com
The second is dedicated to Tenshi developments; subscribe at tenshi-announce+subscribe@lists.inversepath.com and list messages at tenshi-announce@lists.inversepath.com
References
Last updated 09 May 2008, 17:36 CEST.