Adiscon is the company responsible for developping LogAnalyzer, a syslog (rsyslog, syslog-ng…) and/or flat file « analyzer ».
By analyzer, understand that it enables you to display the log in a meaningful way, splitting it depending on « views » and enabling real search filters.

If you don’t have the money for things liks Splunk and you are not convinces by other new projects using Rails, NoSQL and other tools that are a pain in the ass to install, you may fallback to LogAnalyzer.

Loganalyzer is a LAMP (Apache, PHP, Mysql) application. Installing it is as easy as putting the PHP files in you web Documentroot and browse to it. You will be asked a few questions and you can start importing your log files.

To do that you need to create « sources » of data. A source can now be a flat file on the local disk or a database (Mysql).
The choice for us was to use rSyslog on our servers. rSyslog is installed on every UNIX host. It is then configured to send the logs to a « loghost » server.
Additionaly, on some hosts, flat files like the Apache access_log or Error_log are also sent by rSyslog.
This lead to a client rsyslog.conf like :

On the server (receiver) side, the rSyslog will put data into files, rotating every day, and inside the DB.
Actualy I don’t know yet if it is a good idea. For the moment I keep everything. I may consider archiving the flat files and deleting old logs from the DB.
Then the DB would act as a buffer to search and correlate.

This is the server rsyslog.conf :

On the mysql side you will need to install the syslog-ng database schema. Mine is changed to also support ProcessID. (I will join the schema soon).
On the Loganalyzer side you need to define sources and views.
Sources will define which data source to use… flat file, mysql…
To be able to use Mysql you will need to also create (or use the provided) DBMapping. This will match the DB fields to the Syslog Fields.
Finaly, the « view » will decide which fields to display. For example, you could decide to display only the date and message of the error logs in a view because the hostname is not usefull. In another view you could decide that the date is not important but the host is, with the HTTP status code…

OK. You are all set-up with Loganalyzer… but you can only log-in with users from the mysql database.
My users are in LDAP.
While it would be hard to change the whole application to authenticate and search the groups in LDAP, the authentication part is easy to do.
I just changed the functions_users.php so, instead of looking the user in the DB, do the auth on LDAP, create the user in the DB and continue with the normal behaviour…

Here is the code of the new function in include/functions_users.php :

The rest of the code is the same.
Also I had to add some configuration entries in the config.php file

Here is the diff that you can put in a file and use the « patch » command to install.

For the SQL of the logs tables :

APC is one of PHP Opcode Cache on the « market ». It is free and should be bundled inside the next revision of PHP, version 6.

I won’t go deep into how OpCode caches work, you will find a lot of docs, just google for APC, Xcache, eAccelerator… What I can say is that APC (as other caches do) will « save » binary parts of your PHP code into memory and use it when you call for the same PHP function again. This way you save all the PHP file opening, parsing, etc.

Maybe you noticed I said « into memory ». Yes. When you start APC it take a small amount (64 or 32Mo standard) of memory and store the binary parts in it.
This is fine but what if you have a really big website with lots of functions, classes, includes… like when you’re using Typo3 and tons of extensions ?

One thing you won’t come accross often on internet is « How (the hell) do I know if APC is performing well ? »

First, most of the time, the PHP page generation time is halved when you activate APC. This is a good clue. Great.
But what about memory consumption ?

During my tests, I found that APC completly flush the memory if it cannot add a new object. I don’t know if it’s a normal behaviour but it’s what I noted. This is why you have to extensively do your testing and tune the memory size before going to production.

This can be done really easyly with the apc.php page provided in the  APC package. You may not have it if you used some Linux package installer like yum or aptitude. If so, you’ll have to download the APC source from their website and copy apc.php to your web DocumentRoot or wherever you want it. As this is giving sensible informations, I would recomment to put it in a secured place.
We are using Typo3 here so I put the page in the /typo3 folder, which is protected and only accessible by the backend users. This is secure enough for now. Else, use the default login process, user « apc » and password « password ». This can be changed in the page. Disable it if you put the file inside an already protected location.

Browse to your apc.php page and see what happen. First, you’ll get a nasty PHP error if you don’t have APC PHP module enabled. Ensure it is enables in /etc/php.d/apc.conf (on CentOS, can be somewhere else on other distro).

What you see here is, on the left, some metrics and informations of the versions, uptime…
On the right you have the memory utilisation and the cache hit/miss representation. Also, you can see the fragmentation of the memory. You don’t want that but actualy I’m not sure you have a way to reduce it…

The things you need to check are on the left side, in the panel « File Cache Information ».

Hits : how many objects were already in the cache and were used

Miss : how many objects were NOT in the cache, for whatever reason : first request for it, memory full, object can’t be cached…

Cache Full Count : how many times the cache (memory) was full and flushed

This last one is one of the most important and need to be checked first when you website start to slow.

Monitoring

Now you want to monitor this with your Nagios/Centreon or whatever… Well.
I found a « APC monitoring project » out there, APC-PHP-MONITOR on GitHub. This was my starting point as this one is really basic.

As they do, I made a PHP script you have to put in your PHP website somewhere. I used a basic check to ensure only the right IP can access this script. In my case, 10.1.1.88.

Then I modified the PHP script so I can use it from Nagios or whatever. I named the file check_php_apc_cache.php. It’s not a superbe PHP script. It’s just dirty working fine for now.

You can then create a Nagios check or Template. Mine looks like this :

Finaly, the result :

Sometimes you need to be able to send SMS from a computer. This is obvious when you’re talking of monitoring your servers. Email alerts are good, but what if you’re sleeping ?
I used to set up a SMS modem, with it’s own SIM card and subscription. This is really easy, using smsd daemon, to send SMS. Just put a formated file in a directory and bam!, you SMS is fired.
But what when you can’t set a SMS Modem in a datacenter ?

Then you have to use an online SMS provider.

you just have to call an API to send a SMS

it’s not free (you pay per SMS when the modem can have an unlimited SMS sending subscription
it is network dependant ; if you have a network failure, you will never be informed

Once you know that, you can start using it.
I will set up two monitoring plateforms : one inside the datacenter, monitoring every services and local network and one outside, on a hosted server somewhere else, to check the network connecivity of the datacenter.

As our main monitoring is using Nagios, I had to write scripts for it. I now have 2 python scripts : one for sending SMS, one to check how many SMS I have on my account.

I first had a quick look on the french SMS providers. I chosed www.smsenvoi.com as they are one of the cheapest and the http and mail API are free (yes, some providers make you pay for that !!). They are also based in Marseilles. I’m happy thinking people there can do better than people from Paris

So, let’s dig the http API. You can find it at http://www.smsenvoi.com/pgs_api.php
It’s quite simple, even if ugly : you send data in POST request, like : userLogin=your.account@domain.com&userPassword=your_password_clear&…
And call a page, depending on your need :

• smsEnvoi.php : send a message

• smsNumber.php : see how many SMS you have availlable in your account

• smsStatus.php and smsMNC.php : check if a SMS have been sent (this is used when you do mass sending)

You can do pretty neat things. Just check on their website what the URL’s are.
To be honest, I only re-write, in a simple way, what the PHP code do but in Python.

You can find the two scripts attached.
sendSMS.py take a message from the line input and send it. You have to start it for each recipient. Even if it’s easy to add many recipient at the same time, Nagios can’t do that. I’m just keeping it simple.
check_sms_free is a script (nagios check formated) to check how many SMS you still have in your account. The Nagios check is defines to be at WARNING state when only 50 SMS are availlable and in CRITICAL state when only 10 are.

Here are the Nagios config (I do have it in Nconf, but here is what is generated in the statis Nagios file) :

Define the username and password for the smsenvoi account in resource.cfg as $USER8$ and $USER9$

Then, you can add the check to a host :

Don’t forget to define the cellphone number of each users in the « PAGER » field :

Note the format of the phone number, with international format, starting with +
And voila…
Find the two scripts here : sendSMS.py and check_sms_free

Todos :
- limit the message size to 160 caracters, which is the max for a single SMS. If you send messages longer, they will count as two SMS.
- catch every exception in the httplib Python framework (this is really basic for the moment)

As you can see, it is composed of XML files. We’ll go through the VLC.xml file and try to learn about it. Then you’ll be able to download it, or change or build your own in a XML editor.
When you’re done changing it, just quit the EUcontrol software and start it over.

Structure of the file :

Let’s go through the two main parts.

AppSetWide

First part define the application, the preset creation informations and a possible target host. Remember that the controler is networked, and could control any host on the network. I’ll have to try this with laptop soon…
Then you have the CustomWheelCommands.
If you read my second article about the MC Transport, you remember that you have to « activate » the wheel jog and the shutter with a soft key. In fact, the soft key only call a pre-defines command.
Here, we can learn that :
- NormalKeys section is used for… normal keys.
- ShiftKeys section is used for… keys when you press the shift key of the MC Transport (the lower right button)
- WheelSoftKey8 is the jog when turned right
- WheelSoftKey9 is the jog when turned left
- OneShot means no repeat of the command
- LWIN=ON is used to activate the command key of the mac keyboard. You also have LALT=ON (ALT or Option on Mac), LCTL=ON (Control), LSHFT=ON (Shift key)
- the keystroke is defines between the field

You can define multiple CustomWheelFunction, just change the NAME and the UID. About the UID, I actually don’t know how it is calculated. Maybe it just have to be unique in the XML file, or worse, in ALL XML files…

You use the exact same settings for the shuttle :

WheelSoftKey17 is the position 0. 18 to 24 is for the + (right) section. 16 to 10 for the – (minus) side. 25 and above is for the transitions. I haven’t tried it yet.

ControlBindings

ControlBindings are made of Sections.
I can actually see 14 Sections. I think some of then are used by the MC Transport, while others are used by MC Control or other Euphonix hardware.
Inside a Section is a Bank. For the moment, every Section seems to only have one Bank, with ID=0.
In Section=2, Bank is empty :

I really don’t want to mess things up. Let’s go to what is defined for standard control : Section 10

This is working exactly the same way as the section before. This section is defined for the so called « soft keys »

- SoftKey1 is the first on the left
- Command is linked to the UID of the CustomWheelFunction defined before

Here you define how the SoftKey2 behave.
- RepeatEntry= »True » is to activate the auto-repeat of the keystroke
- DELAY= »500″ is the time to wait between the first time and the second
- FREQ= »100″ is the repetition delay 10 times per second, so 100ms. Section The interface you can only choose from 30 to 2 times per second. Maybe changing it in the XML file will allow to set it to 1 times per second ?

Section ID=11 is made for the Numpad. Nothing to change here for now…
Section ID=12 is for the Transport Keys, the 7 keys around the shuttle.
Let’s take some examples :

Here, again, you have an auto-repeat. The delay have been set to « as fast as possible ». FREQ=16 means, theoricaly, every 16ms. Set to FREQ=1000 for once per second.
COLOR=ff00b70b set the color of the button. This is a dark green. I don’t really understand how the MC Transport handle color. Also I don’t understant how they code the color on 4 hexa blocs… If you use this, the switch will be light green. Else, it will be light yellow.
Everything else is already known.

Now you should have everything you need to build your own config file. Enjoy and share your own presets. You can get my VLC preset on the download page, or use the direct link here.

First thing to do is register your controler. Go the the Euphonix support’s website.

This is where you’ll find the software download, the email support link and the docs.

When everything is working, as explained in previous post, you should have the green « € » in the top right of the menu, and the controler should display the name of the software you are actualy using (safari ? firefox ?)

I had some controler « freeze » at some point. Then, the controler keeps displaying the same application name, whichever application is in focus. For the moment, the only solution I found is to power-off the controler, go in the EuControl settings (the only way to get acces to the menu) and quit.
Then, restart the EuControl application, wait until the menu icon stop rolling, then power-on the controler.
As I said, I emailed the support about that without any real answer. I openned a new case today… we’ll see…

If you are using a supported application the controler comes pre-configured. This is not the case for most basic applications like Safari, Firefox, iTunes or even the finder.
While the MC Transport is not made for that specificaly, configuring these applications will help us understand how the controler work.

Basic notions :

you have two ways to do that : clic on the green icon in the menu and go to the settings -> « soft keys » tab or, simply, clic on the « eye » key on the controler, at the lower left of the keypad.

Before beeing able to use the jog or shuttle you have to enable it. Most of the time the activation key is the first soft key for the jog and the second for the shuttle. When switching into an application, every key should be off (no light). clic again to activate

You can change the color of the transport keys (the one arount the shuttle). The 2 first and 2 last keys can be yellow or red. The 3 keys in the middle can be yellow or green. For the moment, I haven’t found a way to light a key or change its color depending on the action. I openned a case to the support to have more informations. For example, keeping the key green while pressed, or changing the color at every clic… Another request, which may be linked, is having two actions for the same key. This way, you can have a yellow key. When first clic, send the « play » shortcut of the application and turn to red. When pressed again, send the « stop » or « pause » keystroke and turn back to yellow… I hope the support will give a clue on how to do that, or add support in the next software release

The pdf manual will lead you through the configuration process, which is really easy for the basic stuffs. Let’s write down a walkthrough…
All this is done in the configuration software, not the controler.

- select the button to assign
- give a name in the « soft key label » box AND press enter to validate
- change the font or color if needed
- clic on « command… » or double-clic on the button
- clic on the « add » menu and select what to do. Most of the time it will be « key », and if you want to make the button an activation button for the jog or shuttle, use « wheel »
- enter a keystroke or a list of them. Doing this enable you to do scripts
- you can add more option, again, to script the button

If you’re programming the button to activate the wheel, select wheel and a custom function.
If you’re programming the wheel, you have to :
- clic on the « + » sign and give a name to the « command »
- as you do when programming a key, add a keystroke or another function. This will be used when turning the wheel right (clockwise) by default
- clic on the « turn jog right » menu and change to « turn jog left »
- add the keystroke used when turn anti-clockwise

If you’re programming the shuttle, this is a little bit different. You can add action for each of the 15s step of the wheel (-7 -> 0 -> +7) and for each transition (0 ->1, 1->2 …) and the other way around (2->1, 1->0…)

These are the basics. I’ll try to give examples in the next post

[EDIT]
Now that I have my disk array working, I had another try.
Once again, the MC Transport stayed locked on the « Mail » application. This is not the first time, and for the moment, the only answer from the support was « disable firewall in your mac » and « start the audio mixer of FCP ». These answers were done after 10 days, or more. Seriously, Euphonix, what are you doing here ?

If this happen to you, follow these steps :
- power off the MC Transport
- go in the green « € » icon -> « EU Control Settings »
- go to the menu and quit
- poweron the MC Transport and wait until the « Euphonix » logo appear
- start the EU Control application. The menu icon should turn to green.

For the moment, while the hardware is perfectly built, I can’t say the software is working fine… If you haven’t bought yet, WAIT !
Euphonix team, if you read this, I’m willing to spend some time debugging for you. Just provide me a beta software, or one with extra debug. But please, stop giving stupid answers.

For the moment I havn’t seen any manual firmware upgrade tool to have every drives in the same version. Actualy, this is even better : if you can, always choose drives with different firmware and series for RAID arrays : you will not have all your drives failing in case of a bug related to the production.

Now, let’s see the perfs…
I will not give every results here. Moreover, RAID initialization will probably take one or two days… So I will not wait for the array to be built before using it. I’m sure this will have an impact on the perfs.

Let’s start with the Host : This is a MacPro 2009 with dual CPU Nehalem 2.66GHz and 12Gb of RAM :

RAID card configuration :
The RAID card is a HighPoint 4322 with dual SAS external output. It has 512Mo of cache and an intel processor for the RAID : this not a « software » RAID card !
Main settings, device settings….

Nothing really special here.

What will be tested

When enabled, the data are not directly written to the disk. Instead they are kept in the 512Mo card cache. The good point is that it gives, as you will soon see, a really good write throughoutput. The bad side is, in case of power failure, data are lost… One solution is to get a APC which will handle in case of power failure. You can use a smaller APC to power the disk rack only but you will have to buy the battery pack for the RAID card. I think the battery pack is really (too much) expensive : Highpoint, stop making money on options !!!
The other option is Write Through.

The card can handle RAID 0 (stripe), 1 (mirror), 5, 6, JBOD (Just a Bunch O Disk : no RAID) and any kind of mix, with spare disks. We’ll test RAID 1 and RAID 5.

The card can handle 8 drives. If I have time, I’ll try to bench with 4 drives and 8 drives.

By default the block size is 64k. It can grow up to 1Gb. This is the the RAID card will use to write data. Some people are saying 128k should be fine with big drives and big files… We’ll test that

Test 1 : RAID 5, 8 drives, Write Back, 64k block size

This is a full default when creating an array.
Aja System Test :

BlackMagic Disk Speed Test :

HD Drive Speed Test :

First thing : every speed test software have it’s own way to report. What I can say is that the write speed is MUCH faster than the read speed. This is certainly due to the write cache. I’m sure this value will drop when the write-through option is selected.
Whatever, the RAID array can sustain around 7 streams of 1080p DVCpro. I’m shooting with a Sony EX1, around 35Mbps VBR : I’ll be good

Test 2 : RAID 5, 8 drives, Write Back, 128k block size

Here we see a little improvement. Video files are usualy big, and a bigger block size seem to be helpful.

When taking off Journalizing, we remove one layer on the OS side. This should give some improvements in throughoutput, but it has a price : if the host crash (or the array go offline) while working, we may lose some data. Concidering that 1) I don’t have any battery on the RAID card and 2) I don’t have an APC on the array, the lease we can do is have a journalized FS.
So, this test is only for benchmark resons :

BlackMagic does not show any improvement or loss.
Conclusion : Disabling journalization does not give a real improvement in throughoutput and make the data even more insecure. I’ll stick with a Jounalized FS.
Another conclusion is that using a 1Gb file for testing is not enough. Switch to 4Gb to have a more realistic test, where buffer size can be filled and used for a longer time.

Test 3 : RAID 5, 8 drives, Write Through, 128k block size

We know write through will give less throughoutput on writing. Let’s see how much.
Another thing to enlight : sometimes the Disk Utility seems to have problem creating a 12Tb FS, or mounting it. You have to wait for a timeout and start it over… I actualy have no clue on the why…

Conclusion : reading is REALLY efficient here. On the other hand, writing is now SLOW. I really don’t understand the results. Does « Write Back » means using the cache for writing and « Write Through » using the cache for reading ? Why not both ? I’ll have to ask Highpoint on that. Another solution is that the read cache would be less effitient than with no cache ?
HD Drive speed test, which is only doing a write test, shows us the same result for writing :

Aja test gives no improvement in reading. This test is seriously puzzling me… LEt me know if you have any clue on why the Black Magic test is giving so many reads/sec.
This is co-related with the « iostat » command I’m running alog with the tests :
During the Black Magic test :

Read :