Adiscon is the company responsible for developping LogAnalyzer, a syslog (rsyslog, syslog-ng…) and/or flat file « analyzer ».
By analyzer, understand that it enables you to display the log in a meaningful way, splitting it depending on « views » and enabling real search filters.

If you don’t have the money for things liks Splunk and you are not convinces by other new projects using Rails, NoSQL and other tools that are a pain in the ass to install, you may fallback to LogAnalyzer.

Loganalyzer is a LAMP (Apache, PHP, Mysql) application. Installing it is as easy as putting the PHP files in you web Documentroot and browse to it. You will be asked a few questions and you can start importing your log files.

To do that you need to create « sources » of data. A source can now be a flat file on the local disk or a database (Mysql).
The choice for us was to use rSyslog on our servers. rSyslog is installed on every UNIX host. It is then configured to send the logs to a « loghost » server.
Additionaly, on some hosts, flat files like the Apache access_log or Error_log are also sent by rSyslog.
This lead to a client rsyslog.conf like :

On the server (receiver) side, the rSyslog will put data into files, rotating every day, and inside the DB.
Actualy I don’t know yet if it is a good idea. For the moment I keep everything. I may consider archiving the flat files and deleting old logs from the DB.
Then the DB would act as a buffer to search and correlate.

This is the server rsyslog.conf :

On the mysql side you will need to install the syslog-ng database schema. Mine is changed to also support ProcessID. (I will join the schema soon).
On the Loganalyzer side you need to define sources and views.
Sources will define which data source to use… flat file, mysql…
To be able to use Mysql you will need to also create (or use the provided) DBMapping. This will match the DB fields to the Syslog Fields.
Finaly, the « view » will decide which fields to display. For example, you could decide to display only the date and message of the error logs in a view because the hostname is not usefull. In another view you could decide that the date is not important but the host is, with the HTTP status code…

OK. You are all set-up with Loganalyzer… but you can only log-in with users from the mysql database.
My users are in LDAP.
While it would be hard to change the whole application to authenticate and search the groups in LDAP, the authentication part is easy to do.
I just changed the functions_users.php so, instead of looking the user in the DB, do the auth on LDAP, create the user in the DB and continue with the normal behaviour…

Here is the code of the new function in include/functions_users.php :

The rest of the code is the same.
Also I had to add some configuration entries in the config.php file

Here is the diff that you can put in a file and use the « patch » command to install.

For the SQL of the logs tables :

APC is one of PHP Opcode Cache on the « market ». It is free and should be bundled inside the next revision of PHP, version 6.

I won’t go deep into how OpCode caches work, you will find a lot of docs, just google for APC, Xcache, eAccelerator… What I can say is that APC (as other caches do) will « save » binary parts of your PHP code into memory and use it when you call for the same PHP function again. This way you save all the PHP file opening, parsing, etc.

Maybe you noticed I said « into memory ». Yes. When you start APC it take a small amount (64 or 32Mo standard) of memory and store the binary parts in it.
This is fine but what if you have a really big website with lots of functions, classes, includes… like when you’re using Typo3 and tons of extensions ?

One thing you won’t come accross often on internet is « How (the hell) do I know if APC is performing well ? »

First, most of the time, the PHP page generation time is halved when you activate APC. This is a good clue. Great.
But what about memory consumption ?

During my tests, I found that APC completly flush the memory if it cannot add a new object. I don’t know if it’s a normal behaviour but it’s what I noted. This is why you have to extensively do your testing and tune the memory size before going to production.

This can be done really easyly with the apc.php page provided in the  APC package. You may not have it if you used some Linux package installer like yum or aptitude. If so, you’ll have to download the APC source from their website and copy apc.php to your web DocumentRoot or wherever you want it. As this is giving sensible informations, I would recomment to put it in a secured place.
We are using Typo3 here so I put the page in the /typo3 folder, which is protected and only accessible by the backend users. This is secure enough for now. Else, use the default login process, user « apc » and password « password ». This can be changed in the page. Disable it if you put the file inside an already protected location.

Browse to your apc.php page and see what happen. First, you’ll get a nasty PHP error if you don’t have APC PHP module enabled. Ensure it is enables in /etc/php.d/apc.conf (on CentOS, can be somewhere else on other distro).

What you see here is, on the left, some metrics and informations of the versions, uptime…
On the right you have the memory utilisation and the cache hit/miss representation. Also, you can see the fragmentation of the memory. You don’t want that but actualy I’m not sure you have a way to reduce it…

The things you need to check are on the left side, in the panel « File Cache Information ».

Hits : how many objects were already in the cache and were used

Miss : how many objects were NOT in the cache, for whatever reason : first request for it, memory full, object can’t be cached…

Cache Full Count : how many times the cache (memory) was full and flushed

This last one is one of the most important and need to be checked first when you website start to slow.

Monitoring

Now you want to monitor this with your Nagios/Centreon or whatever… Well.
I found a « APC monitoring project » out there, APC-PHP-MONITOR on GitHub. This was my starting point as this one is really basic.

As they do, I made a PHP script you have to put in your PHP website somewhere. I used a basic check to ensure only the right IP can access this script. In my case, 10.1.1.88.

Then I modified the PHP script so I can use it from Nagios or whatever. I named the file check_php_apc_cache.php. It’s not a superbe PHP script. It’s just dirty working fine for now.

You can then create a Nagios check or Template. Mine looks like this :

Finaly, the result :

First, get VirtualBox
Then, get a Fedora 11 image
You may also need a « z7″ compressor to un-z7 the image. You can get 7za from the Macports

Then configure and start the VM. I had to add 3 NICs, so I have 4 network interfaces, enough to play. I also set the first one as Bridge instead of the default NAT, so my VM have a real IP.
I then have to log as root, chance the /etc/sudoers so Wheel users can sudo. Then I added fedora (default user) to Wheel group in /etc/groups.
Now I can sudo. We are close to be able to install Oracle database. While I’m at it, go to Oracle website and download the 2 install zip files. This is quite huge, around 2.1Gb. Be carefull when you unzip (not yet), as everything lives in the « database » folder…
You will also need the Grid Infrastructure Software.
Please note we are installong the 32bits versions, but the 64bits version is the same, only the packages to download are different. Click on the « view all » to get the Grid Infrastructure Software.

For Oracle 11G R2 to work on linux you need to fulfill some dependencies, starting with some RPM packages. Use ‘yum’ to search for them and install them. Here is a list according to Oracle Linux recommendations :

11G R2 now comes with a « bundeled NTP server », I mean, Oracle now can sync the time of every node in the cluster. No need of NTPD, and no evictions due to bad Solaris xntpd server. Just disable ntpd or ensure it’s not running before installing Oracle database.
Also, configure SSHD and kernel parameters, if needed, as Oracle prerequisite.
Now, let’s go with Oracle. Create an oracle user with :

As root, create a /opt/oracle folder and give RWX rights to oracle user.
Create a SSL key for user Oracle, add your personal public key to authorized_keys and log as oracle user. Copy the Oracle install files to the home dir of this user.

Log-in again with your oracle user, setting X11 forwarding (use -X -Y if you are using a mac) :

Don’t take account for X11 errors, as long as you have the install window.
First question is giving out your email address for security updates… As you ARE a good DBA/Sysadmin, you won’t need this. Click next

As I don’t have time and I KNOW I will not do better, check there for some more informations on installing Oracle 11G R2 RAC ASM.

Reading the newsgroups and forums, it seems nobody have a real tutorial on how to add it. You will also notice that Ubuntu builders have no plan to include it one day… (even if it is free, not under specific licence and need almost no dependency (Freetds is needed but I don’t know if it is mandatory to build).

Whatever, I need this module, and I came with a working solution :

apt-get source php5
cd /usr/lib/php5/php5-5.2.3/ext/mssql
phpize
./configure
make
make install

Just create a ini file in /etc/php5/conf.d/mssql.ini with :

# configuration for php MsSQL
moduleextension=mssql.so

And restart you Apache server.

!! Be warned that the module may be removed at the next package upgrade !! You will have to compile it again.

As usual in any new linux distro, Apache is not installed the same way as the previous. on ubuntu, you’ll find a bunch of files and directory in /etc/apache2.
I ended searching on « how can I add the LDAP authentication module, authzn_ldap. This module is in the mods-available directory.

One solution seems to link it to mods-enabled directory.
Or you can use the (new to me) utility ‘a2enmod’, which stand for Apache2 Enable Module. you also have a2dismod to remove a module or a2dissite to remove a site (if your site conf is in the /etc/apache2/site-available directory.

I haven’t been waiting for Ubuntu to offer that as I’m doing such a thing for almost 8 years now. Moreover, and this is something I would like to release one day, all my apache vhost conf is stored in Ldap, and managed through a set of PHP pages. I just have to change the conf from the web interface and clic « dump conf », and every modified entry is dumped to the right file, and the link is made or removed automaticaly if needed.
Wait for it…

The Jack plus name is made of two parts : the application name and the « channel » name. The application name must be unique globaly, and the channel must be unique in each application.
While some seems to be configurable in both parts, some are not :

Darkice, by defaults, create a « darkice-PID:left » and  « darkice-PID:right », where PID is the real process ID of Darkice. This ensure 2 darkice will not have the same name. The drowback is that it is very hard to find which darkice is which, moreover if you want to script that.
I made a patch for darkice so you can freely change that.

With Ecasound, a command line sound mixer and processing you can only change the channel name. If you start more Ecasound, you will have another process ecasound_2, then ecasound_3… which is not really better than the way Darkice is doing it natively.
Another issue is when you kill one Ecasound, they all die ! This is a huge issue I will have to work on quickly, but I’m pretty sure this is due to a jack naming issue.

Then what should the convention be ?

As the important part is the application name, not the channel name, it seems that this part should be configurable.  Then the channel name could be, by default, either a direction and a number, or a direction and a name :

- ecasound-test1:in_1
-  ecasound-test1:in_2
- ecasound-test1:out_1
- ecasound-test1:ou_2

- ecasound-filter2:in_left
- ecasound-filter2:in_right

Devs, please, think of it !

But how to do this in C++ ?

I had to change the source code of Rotter as the file mode mask was hard coded in the source file rotter.c
Rotter is a little and handy application that connect to Jack Audio and dump some inputs into a file (mp3 or wave depending on your compile settings). It have the ability to generate one file per hour, spread in multiple directories.

The hardcoded mask was 755, which is right for many applications, but not for me. I will have some users, with a different UID than rotter, writing some files within the same per day directory.
I so had to change the hardcoded mask to use the « umask » C function instead. Easy. But getting the file mode from the mask wan’t that easy

I finaly got something working, and here the result.
This is just the modified function in rotter.c :

static int mkdir_p( const char* dir )
{
        int result = 0;

        if (directory_exists( dir )) {
                return 0;
        }

        // added by prune
        // get the umask value (by stting it to 0) then set it back
        mode_t mask = umask (0);
        umask (mask);

        // Compute the right mode
        mode_t DIR_MODE = ((0777) ^ mask);

        if (mkdir(dir, DIR_MODE) < 0) {
                if (errno == ENOENT) {
                        // ENOENT (a parent directory doesn't exist)
                        char* parent = strdup( dir );
                        int i;

                        // Create parent directories recursively
                        for(i=strlen(parent); i>0; i--) {
                                if (parent[i]=='/') {
                                        parent[i]=0;
                                        result = mkdir_p( parent );
                                        break;
                                }
                        }

                        free(parent);

                        // Try again to create the directory
                        if (result==0) {
                                result = mkdir(dir, DIR_MODE);
                        }

                } else {
                        result = -1;
                }
        }

        return result;
}