Adiscon is the company responsible for developping LogAnalyzer, a syslog (rsyslog, syslog-ng…) and/or flat file « analyzer ».
By analyzer, understand that it enables you to display the log in a meaningful way, splitting it depending on « views » and enabling real search filters.

If you don’t have the money for things liks Splunk and you are not convinces by other new projects using Rails, NoSQL and other tools that are a pain in the ass to install, you may fallback to LogAnalyzer.

Loganalyzer is a LAMP (Apache, PHP, Mysql) application. Installing it is as easy as putting the PHP files in you web Documentroot and browse to it. You will be asked a few questions and you can start importing your log files.

To do that you need to create « sources » of data. A source can now be a flat file on the local disk or a database (Mysql).
The choice for us was to use rSyslog on our servers. rSyslog is installed on every UNIX host. It is then configured to send the logs to a « loghost » server.
Additionaly, on some hosts, flat files like the Apache access_log or Error_log are also sent by rSyslog.
This lead to a client rsyslog.conf like :

  1. #rsyslog v3 config file
  2.  
  3. # if you experience problems, check
  4. # http://www.rsyslog.com/troubleshoot for assistance
  5.  
  6. #### MODULES ####
  7.  
  8. $ModLoad imuxsock.so    # provides support for local system logging (e.g. via logger command)
  9. $ModLoad imklog.so      # provides kernel logging support (previously done by rklogd)
  10. #$ModLoad immark.so     # provides –MARK– message capability
  11. $ModLoad omrelp.so      # load RELP output module to send logs using RELP instead of UDP-TCP
  12. $ModLoad imfile         # load the file input module to scan specific log files
  13.  
  14. #### GLOBAL DIRECTIVES ####
  15.  
  16. # Use default timestamp format
  17. $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
  18.  
  19. # File syncing capability is disabled by default. This feature is usually not required,
  20. # not useful and an extreme performance hit
  21. #$ActionFileEnableSync on
  22.  
  23. #### File Input module ####
  24.  
  25. $InputFileName /opt/app/log/mylog.xml
  26. $InputFileTag mylog.xml:
  27. $InputFileStateFile mylog.xml
  28. $InputFileFacility local5
  29. $InputFileSeverity notice
  30. $InputRunFileMonitor
  31.  
  32. #### RULES ####
  33. $template TraditionalFormatWithPRI,"%PRI-text%: %timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"
  34.  
  35. # Log all kernel messages to the console.
  36. # Logging much else clutters up the screen.
  37. #kern.* /dev/console
  38.  
  39. # Log anything (except mail) of level info or higher.
  40. # Don’t log private authentication messages!
  41. *.info;mail.none;authpriv.none;cron.none;local5.!notice /var/log/messages;TraditionalFormatWithPRI
  42.  
  43. # The authpriv file has restricted access.
  44. authpriv.* /var/log/secure
  45.  
  46. # Log all the mail messages in one place.
  47. mail.* -/var/log/maillog
  48.  
  49. # Log cron stuff
  50. cron.* /var/log/cron
  51.  
  52. # Everybody gets emergency messages
  53. *.emerg *
  54.  
  55. # Save news errors of level crit and higher in a special file.
  56. uucp,news.crit /var/log/spooler
  57.  
  58. # Save boot messages also to boot.log
  59. local7.* /var/log/boot.log
  60.  
  61. # ### begin forwarding rule ###
  62. # The statement between the begin … end define a SINGLE forwarding
  63. # rule. They belong together, do NOT split them. If you create multiple
  64. # forwarding rules, duplicate the whole block!
  65. # Remote Logging (we use TCP for reliable delivery)
  66. #
  67. # An on-disk queue is created for this action. If the remote host is
  68. # down, messages are spooled to disk and sent when it is up again.
  69. $WorkDirectory /var/spool/rsyslog # where to place spool files
  70. $ActionQueueFileName fwdRule1 # unique name prefix for spool files
  71. $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
  72. $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
  73. $ActionQueueType LinkedList # run asynchronously
  74. $ActionResumeRetryCount -1 # infinite retries if host is down
  75.  
  76. # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
  77. *.* @@loghost:514
  78. #*.* :omrelp:loghost:20514;RSYSLOG_ForwardFormat
  79.  
  80. # ### end of the forwarding rule ###

On the server (receiver) side, the rSyslog will put data into files, rotating every day, and inside the DB.
Actualy I don’t know yet if it is a good idea. For the moment I keep everything. I may consider archiving the flat files and deleting old logs from the DB.
Then the DB would act as a buffer to search and correlate.

This is the server rsyslog.conf :

  1. # if you experience problems, check
  2. # http://www.rsyslog.com/troubleshoot for assistance
  3.  
  4. # rsyslog v3: load input modules
  5. # If you do not load inputs, nothing happens!
  6. # You may need to set the module load path if modules are not found.
  7.  
  8. # stats module – experimental
  9. # supprimer pour la mise en prod
  10. $ModLoad impstats
  11. $PStatInterval 600
  12. $PStatSeverity 7
  13. syslog.debug /var/log/rsyslog-stats
  14.  
  15. $ModLoad immark # provides –MARK– message capability
  16. $ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
  17. $ModLoad imklog # kernel logging (formerly provided by rklogd)
  18. $ModLoad ommysql # locad Mysql backend module
  19.  
  20. $WorkDirectory /opt/rsyslog/work # default location for work (spool) files
  21.  
  22. # Templates
  23. $template RemoteHost,"/opt/rsyslog/data/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/%syslogfacility-text%.log"
  24. # used for Cisco, vanilla syslog when we can’t parse host name
  25. $template RemoteFromHost,"/opt/rsyslog/data/%FROMHOST%/%$YEAR%/%$MONTH%/%$DAY%/%syslogfacility-text%.log"
  26.  
  27. # ######### Receiving Messages from local host only ##########
  28. # Log all kernel messages to the console.
  29. # Logging much else clutters up the screen.
  30. #kern.* /dev/console
  31.  
  32. # Log anything (except mail) of level info or higher.
  33. # Don’t log private authentication messages!
  34. *.info;mail.none;authpriv.none;cron.none -/var/log/messages
  35.  
  36. # The authpriv file has restricted access.
  37. authpriv.* /var/log/secure
  38.  
  39. # Log all the mail messages in one place.
  40. mail.* -/var/log/maillog
  41.  
  42. # Log cron stuff
  43. cron.* -/var/log/cron
  44.  
  45. # Everybody gets emergency messages
  46. *.emerg *
  47.  
  48. # Save news errors of level crit and higher in a special file.
  49. uucp,news.crit -/var/log/spooler
  50.  
  51. # Save boot messages also to boot.log
  52. local7.* /var/log/boot.log
  53.  
  54. # ######### Receiving Messages from Remote Hosts ##########
  55. # at this point, we consider only remote messages
  56. $RuleSet remote
  57.  
  58. # TCP Syslog Server:
  59. # provides TCP syslog reception and GSS-API (if compiled to support it)
  60. $ModLoad imtcp.so # load module
  61. $InputTCPServerBindRuleset remote
  62. $InputTCPServerRun 514 # start up TCP listener at port 514
  63.  
  64. # TCP Syslog Server for Cisco – need port over 1024 :
  65. # provides TCP syslog reception and GSS-API (if compiled to support it)
  66. $ModLoad imtcp.so # load module
  67. $InputTCPServerBindRuleset remote
  68. $InputTCPServerRun 1470 # start up TCP listener at port 514
  69.  
  70. # UDP Syslog Server:
  71. $ModLoad imudp.so # provides UDP syslog reception
  72. $InputUDPServerBindRuleset remote
  73. $UDPServerRun 514 # start a UDP syslog server at standard port 514
  74.  
  75. # RELP Syslog (Reliable Event Logging Protocol)
  76. # WARNING : RELP messages are not passed through ruleset yet and will be logged localy
  77. $ModLoad imrelp # load RELP input module to send reliable messages – better than TCP and UDP
  78. $InputRELPServerRun 20514 # RELP port on 20514
  79. # #########
  80.  
  81. # log remote hosts to specific files
  82. $ActionQueueType LinkedList # use asynchronous processing
  83. $ActionQueueFileName dbq # set file name, also enables disk mode
  84. $ActionResumeRetryCount -1 # infinite retries on insert failure
  85. *.* -?RemoteHost
  86.  
  87. # log everything to mysql using default template
  88. #*.* :ommysql:database-server,database-name,database-userid,database-password
  89. #*.* :ommysql:localhost,loganalyzer,rsyslog,password
  90.  
  91. # syslog-ng style template used by centreon-syslog
  92. $template sysMysql,"INSERT INTO logs (host,facility, priority,level,tag,datetime,program,msg,ProcessID) VALUES (‘%HOSTNAME%’,’%syslogfacility%’,’%syslogpriority%’,’%syslogseverity%’,’%syslogtag%’, ‘%timereported:::date-mysql%’,'%programname%’, ‘%msg%’,'%procid:R,ERE,0,ZERO:[0-9]+–end%’)", SQL
  93. *.* :ommysql:localhost,loganalyzer,rsyslog,password;sysMysql
  94.  
  95. #
  96. # END remote logging
  97. #

On the mysql side you will need to install the syslog-ng database schema. Mine is changed to also support ProcessID. (I will join the schema soon).
On the Loganalyzer side you need to define sources and views.
Sources will define which data source to use… flat file, mysql…
To be able to use Mysql you will need to also create (or use the provided) DBMapping. This will match the DB fields to the Syslog Fields.
Finaly, the « view » will decide which fields to display. For example, you could decide to display only the date and message of the error logs in a view because the hostname is not usefull. In another view you could decide that the date is not important but the host is, with the HTTP status code…

OK. You are all set-up with Loganalyzer… but you can only log-in with users from the mysql database.
My users are in LDAP.
While it would be hard to change the whole application to authenticate and search the groups in LDAP, the authentication part is easy to do.
I just changed the functions_users.php so, instead of looking the user in the DB, do the auth on LDAP, create the user in the DB and continue with the normal behaviour…

Here is the code of the new function in include/functions_users.php :

  1. function CheckLDAPUserLogin( $username, $password )
  2. {
  3. global $content;
  4.  
  5. $ldap_filter=‘(&’.$content[‘LDAPSearchFilter’].‘(‘.$content[‘LDAPUidAttribute’].‘=’.$username.‘))’;
  6.  
  7. // Open LDAP connection
  8. if (!($ds=ldap_connect($content[‘LDAPServer’],$content[‘LDAPPort’]))) {
  9. return false;
  10. }
  11. ldap_set_option($ds, LDAP_OPT_PROTOCOL_VERSION, 3);
  12.  
  13. // Bind as the privilegied user
  14. if (!($r=ldap_bind($ds,$content[‘LDAPBindDN’],$content[‘LDAPBindPassword’]))) {
  15. return false;
  16. }
  17. // search for the user
  18. if (!($r=ldap_search( $ds, $content[‘LDAPBaseDN’], $ldap_filter, array("uid","cn","localentryid","userpassword") ))) {
  19. DieWithFriendlyErrorMsg( "Debug Error: Could not login user ‘" . $username . "’
  20.  
  21. <strong>Sessionarray</strong>
  22. <pre>" . var_export($_SESSION, true) . "</pre>
  23. <strong>Search Filter </strong>: " . $ldap_filter );
  24. return false;
  25. }
  26.  
  27. $info = ldap_get_entries($ds, $r);
  28. if (!$info || $info["count"] != 1) {
  29. DieWithFriendlyErrorMsg( "Debug Error: Could not login user ‘" . $username . "’
  30.  
  31. <strong>Sessionarray</strong>
  32. <pre>" . var_export($_SESSION, true) . "</pre>
  33. <strong>Search Filter </strong>: " . $ldap_filter );
  34. return false;
  35. }
  36.  
  37. // now we have the user data. Do a bind to check for his password
  38. if (!($r=ldap_bind( $ds, $info[0][‘dn’],$password))) {
  39. return false;
  40. }
  41.  
  42. // for the moment when a user logs in from LDAP, create it in the DB.
  43. // then the prefs and group management is done in the DB and we don’t rewrite the whole Loganalyzer code…
  44.  
  45. // check if the user already exist
  46. $sqlquery = "SELECT * FROM " . DB_USERS . " WHERE username = ‘" . $username . "’";
  47. $result = DB_Query($sqlquery);
  48. $myrow = DB_GetSingleRow($result, true);
  49. if (!isset($myrow[‘is_admin’]) ) {
  50. // Create User
  51. $result = DB_Query("INSERT INTO " . DB_USERS . " (id, username, password, is_admin, is_readonly) VALUES (".$info[0][‘localentryid’][0].", ‘$username‘, ‘rnd".md5(mt_rand()."rnd")."’, 0, 1)");
  52. DB_FreeQuery($result);
  53. $myrow[‘is_admin’] = 0;
  54. $myrow[‘last_login’] = 0;
  55. $myrow[‘is_readonly’] = 1;
  56. }
  57.  
  58. $myrowfinal[‘username’] = $info[0][$content[‘LDAPUidAttribute’]][0];
  59. $myrowfinal[‘password’] = "hidden";
  60. $myrowfinal[‘dn’] = $info[0][‘dn’];
  61. $myrowfinal[‘ID’] = $info[0][‘localentryid’][0];
  62. $myrowfinal[‘is_admin’] = $myrow[‘is_admin’];
  63. $myrowfinal[‘is_readonly’] = $myrow[‘is_readonly’];
  64. $myrowfinal[‘last_login’] = $myrow[‘last_login’];
  65.  
  66. return $myrowfinal;
  67. // Default return false
  68. return false;
  69. }
  70. function CheckUserLogin( $username, $password )
  71. {
  72. global $content;
  73.  
  74. // if auth LDAP is used, don’t check the DB yet
  75. if ( GetConfigSetting("LDAPUserLoginRequired", "") == "true") {
  76. $myrow = CheckLDAPUserLogin( $username, $password );
  77. }
  78. else {
  79. // TODO: SessionTime and AccessLevel check
  80. $md5pass = md5($password);
  81. $sqlquery = "SELECT * FROM " . DB_USERS . " WHERE username = ‘" . $username . "’ and password = ‘" . $md5pass . "’";
  82. $result = DB_Query($sqlquery);
  83. $myrow = DB_GetSingleRow($result, true);
  84. }

The rest of the code is the same.
Also I had to add some configuration entries in the config.php file

  1. // — LDAP auth options
  2. $CFG[‘LDAPUserLoginRequired’] = true; // activate LDAP auth
  3. $CFG[‘LDAPServer’] = "localhost"; // LDAP server hostname or IP
  4. $CFG[‘LDAPPort’] = 389; // LDAP port, 389 or 636 for SSL
  5. $CFG[‘LDAPBaseDN’] = "ou=my,o=ldap"; // Base DN for LDAP search
  6. $CFG[‘LDAPSearchFilter’] = "(objectclass=inetOrgPerson)"; // search filter
  7. $CFG[‘LDAPUidAttribute’] = "uid"; // the LDAP attribute used in the search to find the user. ex : uid, cn
  8. $CFG[‘LDAPBindDN’] = "cn=Manager,ou=my,o=ldap"; // DN of the privileged user for the search
  9. $CFG[‘LDAPBindPassword’] = ‘secret’; // Password of the privilegied user
  10. $CFG[‘LDAPGroupAttribute’] = ‘member’; // attribute used to search for groups
  11. // —

Here is the diff that you can put in a file and use the « patch » command to install.

  1. — include/functions_users_orig.php 2012-01-19 14:36:24.000000000 -0500
  2. +++ include/functions_users.php 2012-01-19 14:33:12.000000000 -0500
  3. @@ -157,16 +157,83 @@
  4. }
  5. }
  6.  
  7. -function CheckUserLogin( $username, $password )
  8. +function CheckLDAPUserLogin( $username, $password )
  9. {
  10. global $content;
  11.  
  12. - // TODO: SessionTime and AccessLevel check
  13. + $ldap_filter=’(&amp;’.$content[‘LDAPSearchFilter’].’(‘.$content[‘LDAPUidAttribute’].’=’.$username.’))‘;
  14.  
  15. - $md5pass = md5($password);
  16. - $sqlquery = "SELECT * FROM " . DB_USERS . " WHERE username = ‘" . $username . "’ and password = ‘" . $md5pass . "’";
  17. - $result = DB_Query($sqlquery);
  18. - $myrow = DB_GetSingleRow($result, true);
  19. + // Open LDAP connection
  20. + if (!($ds=ldap_connect($content[‘LDAPServer’],$content[‘LDAPPort’]))) {
  21. + return false;
  22. + }
  23. + ldap_set_option($ds, LDAP_OPT_PROTOCOL_VERSION, 3);
  24. +
  25. + // Bind as the privilegied user
  26. + if (!($r=ldap_bind($ds,$content[‘LDAPBindDN’],$content[‘LDAPBindPassword’]))) {
  27. + return false;
  28. + }
  29. + // search for the user
  30. + if (!($r=ldap_search( $ds, $content[‘LDAPBaseDN’], $ldap_filter, array("uid","cn","localentryid","userpassword") ))) {
  31. + DieWithFriendlyErrorMsg( "Debug Error: Could not login user ‘" . $username . "’
  32.  
  33. <strong>Sessionarray</strong>
  34. <pre>" . var_export($_SESSION, true) . "</pre>
  35. <strong>Search Filter </strong>: " . $ldap_filter );
  36. + return false;
  37. + }
  38. +
  39. + $info = ldap_get_entries($ds, $r);
  40. + if (!$info || $info["count"] != 1) {
  41. + DieWithFriendlyErrorMsg( "Debug Error: Could not login user ‘" . $username . "’
  42.  
  43. <strong>Sessionarray</strong>
  44. <pre>" . var_export($_SESSION, true) . "</pre>
  45. <strong>Search Filter </strong>: " . $ldap_filter );
  46. + return false;
  47. + }
  48. +
  49. + // now we have the user data. Do a bind to check for his password
  50. + if (!($r=ldap_bind( $ds, $info[0][‘dn’],$password))) {
  51. + return false;
  52. + }
  53. +
  54. + // for the moment when a user logs in from LDAP, create it in the DB.
  55. + // then the prefs and group management is done in the DB and we don’t rewrite the whole Loganalyzer code…
  56. +
  57. + // check if the user already exist
  58. + $sqlquery = "SELECT * FROM " . DB_USERS . " WHERE username = ‘" . $username . "’";
  59. + $result = DB_Query($sqlquery);
  60. + $myrow = DB_GetSingleRow($result, true);
  61. + if (!isset($myrow[‘is_admin’]) ) {
  62. + // Create User
  63. + $result = DB_Query("INSERT INTO " . DB_USERS . " (id, username, password, is_admin, is_readonly) VALUES (".$info[0][‘localentryid’][0].", ‘$username’, ‘rnd".md5(mt_rand()."rnd")."’, 0, 1)");
  64. + DB_FreeQuery($result);
  65. + $myrow[‘is_admin’] = 0;
  66. + $myrow[‘last_login’] = 0;
  67. + $myrow[‘is_readonly’] = 1;
  68. + }
  69. +
  70. + $myrowfinal[‘username’] = $info[0][$content[‘LDAPUidAttribute’]][0];
  71. + $myrowfinal[‘password’] = "hidden";
  72. + $myrowfinal[‘dn’] = $info[0][‘dn’];
  73. + $myrowfinal[‘ID’] = $info[0][‘localentryid’][0];
  74. + $myrowfinal[‘is_admin’] = $myrow[‘is_admin’];
  75. + $myrowfinal[‘is_readonly’] = $myrow[‘is_readonly’];
  76. + $myrowfinal[‘last_login’] = $myrow[‘last_login’];
  77. +
  78. + return $myrowfinal;
  79. + // Default return false
  80. + return false;
  81. +
  82. +}
  83. +function CheckUserLogin( $username, $password )
  84. +{
  85. + global $content;
  86. +
  87. + // if auth LDAP is used, don’t check the DB yet
  88. + if ( GetConfigSetting("LDAPUserLoginRequired", "") == "true") {
  89. + $myrow = CheckLDAPUserLogin( $username, $password );
  90. + }
  91. + else {
  92. + // TODO: SessionTime and AccessLevel check
  93. + $md5pass = md5($password);
  94. + $sqlquery = "SELECT * FROM " . DB_USERS . " WHERE username = ‘" . $username . "’ and password = ‘" . $md5pass . "’";
  95. + $result = DB_Query($sqlquery);
  96. + $myrow = DB_GetSingleRow($result, true);
  97. + }
  98.  
  99. // The admin field must be set!
  100. if ( isset($myrow[‘is_admin’]) )

For the SQL of the logs tables :

  1. CREATE TABLE `logs` (
  2. `host` varchar(128) collate utf8_unicode_ci DEFAULT NULL,
  3. `facility` varchar(10) collate utf8_unicode_ci DEFAULT NULL,
  4. `priority` varchar(10) collate utf8_unicode_ci DEFAULT NULL,
  5. `level` varchar(10) collate utf8_unicode_ci DEFAULT NULL,
  6. `tag` varchar(10) collate utf8_unicode_ci DEFAULT NULL,
  7. `datetime` datetime DEFAULT NULL,
  8. `program` varchar(256) collate utf8_unicode_ci DEFAULT NULL,
  9. `msg` text collate utf8_unicode_ci,
  10. `seq` bigint(20) UNSIGNED NOT NULL AUTO_INCREMENT,
  11. `counter` int(11) NOT NULL DEFAULT ’1′,
  12. `fo` datetime DEFAULT NULL,
  13. `lo` datetime DEFAULT NULL,
  14. `processid` char(8) collate utf8_unicode_ci DEFAULT NULL,
  15. PRIMARY KEY (`seq`),
  16. KEY `datetime` (`datetime`),
  17. KEY `priority` (`priority`),
  18. KEY `facility` (`facility`),
  19. KEY `program` (`program`),
  20. KEY `host` (`host`),
  21. KEY `host_datetime` (`host`,`datetime`)
  22. ) ENGINE=MyISAM AUTO_INCREMENT=1339013 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;