One of my best friend always tell me : « buy, don’t wait ». But I chosed to wait a little bit for the new Macpro Nehalem. And it finaly happened : http://www.apple.com/macpro/

Building my config was simple : best possible but not crazily expensive. I finaly settled on a dual CPU, 2.66GHz (8 threads in total), the ATI 4870 video card, wifi, bluetooth and wireless keyboard, which is for sale at 50€ (pickup in Paris). The RAM amount was a real challenge… I wanted 8Gb.
As you may know, new Nehalem processor have a 3 lane memory controler. The Macpro have 4 slots of ram per CPU.
8 Gb = 2x2Go per cpu or 4x1Gb per cpu.
Problem is, to use the whole CPU ram throughoutput, which is the key point on the nehalem, you have to use 3 memory modules.
I decided to use 3x2Gb per cpu, having a final HUGE 12Go of memory
Hopefully, the RAM modules for the macpro are really « cheap » actually. I used the Apple ram extension for 180€. I could have chosed to go with the reseller’s ram for 130€… but I think it’s better to have 6 identical RAM modules instead of 4+2. Remember RAM modules work by 3 at a time.

So, you’re asking, where are the picture… There you go :

The HD card
Things never change alone. Just one month ago, BlackMagic Design released it’s new Decklink card, the Decklink HD Extreme 3.


I just installed it in the mac and installed the driver. I will try it today.
First visible change : you have a new monitor in the monitor preference pane. While this seems to be cool, it is a serious problem when using a Wacom intuos or the special corner feature to get exposé or dashboard.
More on this later

The Raid card
As for the Decklink card, the Raid card is a new one too : Highpoint Rocketraid 4322.
Most of people used the 3522, which is almost the same, with less RAM and a slower CPU (800MHz instead of 1200MHz). While it seems no one sell the 4322 card in France, I had to buy it from Germany. I think I had a real good price, far lower than other cards : 530€.

See both the Decklink and the raid card in the mac :

Here is the back of the computer. See the HD-SDI IN/OUT of the Decklink on top, then the IN/OUT HDMI. Then the two Mini-SAS input of the RAID card, then the video card, with one DVI and one mini-displayport.

The RAID Case
You may ask what can I plug in the mini-SAS raid card… In fact, many things
I chosed to use a 8 disk enclosure with mini-SAS inputs from Enhance Technologies.
I chosed the E8MS as it have mini-SAS input and accept S-ATA drives. Note that you can have the SAME box, in light grey instead of dark grey, from ProAVIo.
The proAvIo box is not sold in France. I had to call the English reseller… which wasn’t OK to sell the box alone, without drives. Of course, their drives were far more expensive (special RAID ones, you can also find it everywhere).
I finaly found a company selling the Enhance Tech model in France, for around 500€, and empty !
I went to the « rue Montgallet » where all the computer parts resellers are, and bought 8×1.5To Seagate drives. This will give me a huge 10.5 To of storage !!
Moreover, it seems the 1.5To disks have a better throughoutput than the 1To ones. And they have the 5 years waranty intead of 3 years for the 1To.

See the pics :

I will try to do a step by step of the RAId configuration and add some RAID tests, but for now, I’m sad to say the RAID card is not working !
The RAID card is managed through a web interface. I can see the drives, create the RAID array and mount it. Then, the initialization fail. Then, the RAID card stop responding and the whole system hang.
I also had strange USB power loss some time. Don’t know if this is related to the RAID card.

Two last notes on what Apple can improve :
- the mini displayport of the video card : WHY ? and why not displayport instead of mini ?
- the RAID card, on slot 2, is just over the video card fan… There should be more space between the two PCI slots so the display card can breath.

One new API from Google. This one allow you to create charts (graphiques, pour les francophones).

Philippe Mougin is our guest and comes with a clean example with integration to Apple’s Cocoa programming. Check this here.

Voila le site d’une personne qui a une solution. Cela interessera particulierement mon ami Akhen, qui fete son anniversaire demain. Tiens, c’est pour toi : http://toxicsoftware.com/run-python-script/

This modified source-tree should be working with most Solaris 10 and maybe 8, 9, on Sparc or Intel. Just untar it, do a « ./buildtarball myversion » and use the tar.gz file for install. You may also need the Install script I modified to install in /opt/deflector/dss. I hope to find time to have the install dir on the command line so anybody can use it.

You will see a new parameter in streamingserver.xml where you define the IP you wish to subscribe Multicast streams on : <PREF NAME= »ip_listen_for_multicast » >0</PREF>
This parameter is set to 0 by default, which means a default behaviour, using INADDR_ANY as subscription IP. You can set it to an IP on your host to subscribe to a special interface like :
<PREF NAME= »ip_listen_for_multicast » >10.16.240.19</PREF>

It seems another solution would be to change the multicast lan route. Lets say your primary inteface have IP 10.16.248.19 and you have multicast comming to another interface whose IP is 10.16.240.19. You do it like this :

route -n delete -interface 224.0/4 -gateway 10.16.248.19
route -n add -interface 224.0/4 -gateway 10.16.240.19

My patch is not a end by itself as this IP should be, at least, be set per multicast stream. It’s just OK for my need, my lack of time and my small C++ experience.

Just come to me if you want to have more details about the config files or start/stop script.

By the way, the .tar.gz file with patched source is here and a patch file (diff file to apply using gpatch on Solaris) for the original source code is here

The best thing you can do with DSS, when you own the source part of the stream, is to use multicast. This way, you stream once from your encoder, and many DSS servers can access the stream and reflect it as unicast to your customers.

Once said, this sounds good and simple. In fact, not that simple if your DSS server is « multi homed ».

Multi homed is the way you say your server is using multiples IP on different networks, VLAN and/or NICs.

Our servers une MANY networks. One on the first NIC (nic 0) for admin tasks. NICs 1 and 2 are aggregated to have 2Gb bandwidth for data. On this Aggregated interface, we have many IP set on different networks and VLANs.

To cut the story short, the multicast stream comes on interface aggr543510 (vlan 543 on aggregated interface 510). Clients connect to admin NIC for test, and aggr536510 (vlan 536).

By default, DSS, when reflecting a multicast stream, bind any IP on the port supplied in the SDP file and subscribe to the multicast stream. This is pretty usual, and any other application would do the same.
The troubles comes when you are using multiple NICs. The bind is effectylevly done on * (any IP), but the multicast subscribe is done on ONLY ONE INTERFACE !

Let’s have a netstat -g (works on Solaris, on OS X, use netstat -a -i and try to find out the info on your own)

# netstat -g
Group Memberships: IPv4
Interface Group                RefCnt
--------- -------------------- ------
lo0       ALL-SYSTEMS.MCAST.NET      1
e1000g0   224.1.1.9                 1
e1000g0   ALL-SYSTEMS.MCAST.NET      1
e1000g34001:1 ALL-SYSTEMS.MCAST.NET      1
e1000g34001:2 ALL-SYSTEMS.MCAST.NET      1
e1000g36001:1 ALL-SYSTEMS.MCAST.NET      1
e1000g36001:2 ALL-SYSTEMS.MCAST.NET      1
e1000g244001 ALL-SYSTEMS.MCAST.NET      1
e1000g543001:1 ALL-SYSTEMS.MCAST.NET      1

The important line is the one with « e1000g0 224.1.1.9 « . There you can see that Multicast Network 224.1.1.9 is subscribed on interface e1000g0.
But e1000g0 is our admin interface !!!

So DSS will never see any data to reflect. And your client will be set to pause.

I Called Sun Support yesterday to have them comment about this. As I said, this is not really DSS related. It comed from the way the OS (every UNIX os ?) deals the multicast subscription. No news yet. I’m pretty I’ll be told that it is a normal behaviour.
…

Once that said, what will I do to have DSS work on my architecture ?

The key lies in CommonUtilitiesLib/UDPSocket.cpp
YES, we are back again to C++ programming !

The original source code says :

OS_Error UDPSocket::JoinMulticast(UInt32 inRemoteAddr)
{
    struct ip_mreq  theMulti;
        UInt32 localAddr = fLocalAddr.sin_addr.s_addr; // Already in network byte order

#if __solaris__
    if( localAddr == htonl(INADDR_ANY) )
         localAddr = htonl(SocketUtils::GetIPAddr(0));
#endif

    theMulti.imr_multiaddr.s_addr = htonl(inRemoteAddr);
    theMulti.imr_interface.s_addr = localAddr;

    int err = setsockopt(fFileDesc, IPPROTO_IP, IP_ADD_MEMBERSHIP, (char*)&theMulti, sizeof(theMulti));
    //AssertV(err == 0, OSThread::GetErrno());
    if (err == -1)
         return (OS_Error)OSThread::GetErrno();
    else
         return OS_NoErr;
}

There you can see 2 things :

As I already said I’m not C++ expert. I just changed the code so the IP to subscribe, defined in « localAddr », is set to the IP I need to subscribe on :

OS_Error UDPSocket::JoinMulticast(UInt32 inRemoteAddr)
{
    struct ip_mreq  theMulti;
        UInt32 localAddr = fLocalAddr.sin_addr.s_addr; // Already in network byte order

/*
#if __solaris__
    if( localAddr == htonl(INADDR_ANY) )
         localAddr = htonl(SocketUtils::GetIPAddr(0));
#endif
*/

// Set by Prune - 20070712
    localAddr  = htonl(0x0a10f07b);

    theMulti.imr_multiaddr.s_addr = htonl(inRemoteAddr);
    theMulti.imr_interface.s_addr = localAddr;

// Set by Prune - 20070712
qtss_printf("Multicast local %x\n", localAddr);
qtss_printf("Multicast remote %x\n", htonl(inRemoteAddr));

    int err = setsockopt(fFileDesc, IPPROTO_IP, IP_ADD_MEMBERSHIP, (char*)&theMulti, sizeof(theMulti));
    //AssertV(err == 0, OSThread::GetErrno());
    if (err == -1)
         return (OS_Error)OSThread::GetErrno();
    else
         return OS_NoErr;
}

So the local IP is set to « localAddr = htonl(0x0a10f07b); » where 0a10f07b is the hexa value of IP 10.16.240.123
I also add some debug that you can see if you start DSS with the -d option.

Then everything is OK and you have the right interface subscribed :

# netstat -g
Group Memberships: IPv4
Interface Group                RefCnt
--------- -------------------- ------
lo0       ALL-SYSTEMS.MCAST.NET      1
e1000g0   ALL-SYSTEMS.MCAST.NET      1
e1000g34001:1 ALL-SYSTEMS.MCAST.NET      1
e1000g34001:2 ALL-SYSTEMS.MCAST.NET      1
e1000g36001:1 ALL-SYSTEMS.MCAST.NET      1
e1000g36001:2 ALL-SYSTEMS.MCAST.NET      1
e1000g244001 ALL-SYSTEMS.MCAST.NET      1
e1000g543001:1 224.1.1.9                 1
e1000g543001:1 ALL-SYSTEMS.MCAST.NET      1

OK, I have to do it better, like getting the interface to bind from the config file. This was just a late night test

But what is the right way to do it ?
The server (DSS process) can’t know what interface to subscribe to. Does it have to subscribe it on all interfaces ? This does not seem possible other than opening many sockets. So the server will then have to decide which one to use. tricky isn’t it ?

That is where I am actualy. No real good solution but a hack to have it working.
Then, what will happen if one day I want another multicast stream coming from another network ? …
Keep reading here as I’ll post any information I have if Sun or DSS mailing list answer me.

When I have my whole architecture up and running I will go for data aggregation to get the full connections over all servers.

You will find 2 Cacti Graph Templates in the download section, with the 2 python scripts to collect data.
The python scripts have to be copied in the cacti/scripts directory, while the XML files have to be imported using the « Import Template » link in the Cacti Console.

Bandwidth template and script
Connection template an script

For the python scripts to work, you will have to enable the « remote admin » in DSS. Change this in your streamingserver.xml file :
- set LocalAccessOnly to false
- add your IP’s in the IPAccessList
- check enable_remote_admin is true, which is the default
- use your admin user or add anoter one in the admin group using qtpasswd

        
                10.16.248.10
                true
                false
                50
                true
                admin
        

In this case your remote IP is 10.16.248.10
You can define multiple IP’s like this :

        
                
                        127.0.0.*
                        10.16.248.*
                
                true
                false
                50
                true
                admin
        

I also provide a simple python script to get easily any DSS Admin URL : dss-query_url.py

The usage is simple (but you may have to add python before it or change the python link in the first line of the script). You just need to provide the hostname, the port (554), the login of an admin account (the account you use to log in the HTML interface – the one you define at the install stage) and its password.
The you can supply any URL. You will learn more about Admin URL in the « Documentation/admin-protocol-README.txt » file in the source tree.
You will also find some usefull informations on each fields in the file « APIStubLib/QTSS.h ».

Let’s have an example :

python dss-query_url.py myhost.mydomain.com  554 admin password /modules/admin/server/*?command=get
Container="/admin/server/"
qtssServerAPIVersion="262144"
qtssSvrDefaultDNSName="0.0.0.0"
qtssSvrDefaultIPAddr="0"
qtssSvrServerName="DSS"
qtssRTSPSvrServerVersion="5.5.5"
qtssRTSPSvrServerBuildDate="Jul  6 2007, 16:35:39"
qtssSvrRTSPServerHeader="Server: DSS/5.5.5 (Build/489.16; Platform/Solaris; Release/Darwin; state/beta; )"
qtssSvrState="1"
qtssSvrIsOutOfDescriptors="false"
qtssRTSPCurrentSessionCount="127"
qtssRTSPHTTPCurrentSessionCount="0"
qtssRTPSvrNumUDPSockets="8"
qtssRTPSvrCurConn="126"
qtssRTPSvrTotalConn="256"
qtssRTPSvrCurBandwidth="5027784"
qtssRTPSvrTotalBytes="3809807612"
qtssRTPSvrAvgBandwidth="3247539"
qtssRTPSvrCurPackets="776"
qtssRTPSvrTotalPackets="10865424"
qtssSvrStartupTime="1183999975017"
qtssSvrGMTOffsetInHrs="-20629"
qtssSvrDefaultIPAddrStr="0.0.0.0"
qtssSvrCurrentTimeMilliseconds="1184070706181"
qtssSvrCPULoadPercent="23.004150"
qtssSvrNumReliableUDPBuffers="0"
qtssSvrReliableUDPWastageInBytes="0"
qtssMP3SvrCurConn="0"
qtssMP3SvrTotalConn="0"
qtssMP3SvrCurBandwidth="0"
qtssMP3SvrTotalBytes="0"
qtssMP3SvrAvgBandwidth="0"
qtssSvrServerBuild="489.16"
qtssSvrServerPlatform="Solaris"
qtssSvrRTSPServerComment="Release/Darwin; state/beta; "
qtssSvrNumThinned="113"
qtssSvrRTSPPorts/
qtssSvrHandledMethods/
qtssSvrModuleObjects/
qtssSvrPreferences/
qtssSvrMessages/
qtssSvrClientSessions/
qtssSvrConnectedUsers/
error:(0)

Quoting the « APIStubLib/QTSS.h » file :

    // These parameters ARE pre-emptive safe.

    qtssServerAPIVersion            = 0,    //read  //UInt32            //The API version supported by this server (format 0xMMMMmmmm, where M=major version, m=minor version)
    qtssSvrDefaultDNSName           = 1,    //read  //char array        //The "default" DNS name of the server
    qtssSvrDefaultIPAddr            = 2,    //read  //UInt32            //The "default" IP address of the server
    qtssSvrServerName               = 3,    //read  //char array        //Name of the server
    qtssSvrServerVersion            = 4,    //read  //char array        //Version of the server
    qtssSvrServerBuildDate          = 5,    //read  //char array        //When was the server built?
    qtssSvrRTSPPorts                = 6,    //read  // NOT PREEMPTIVE SAFE!//UInt16         //Indexed parameter: all the ports the server is listening on
    qtssSvrRTSPServerHeader         = 7,    //read  //char array        //Server: header that the server uses to respond to RTSP clients

    // These parameters are NOT pre-emptive safe, they cannot be accessed
    // via. QTSS_GetValuePtr. Some exceptions noted below

    qtssSvrState                    = 8,    //r/w   //QTSS_ServerState  //The current state of the server. If a module sets the server state, the server will respond in the appropriate fashion. Setting to qtssRefusingCon
nectionsState causes the server to refuse connections, setting to qtssFatalErrorState or qtssShuttingDownState causes the server to quit.
    qtssSvrIsOutOfDescriptors       = 9,    //read  //Bool16            //true if the server has run out of file descriptors, false otherwise
    qtssRTSPCurrentSessionCount     = 10,   //read  //UInt32            //Current number of connected clients over standard RTSP
    qtssRTSPHTTPCurrentSessionCount = 11,   //read  //UInt32            //Current number of connected clients over RTSP / HTTP

    qtssRTPSvrNumUDPSockets         = 12,   //read      //UInt32    //Number of UDP sockets currently being used by the server
    qtssRTPSvrCurConn               = 13,   //read      //UInt32    //Number of clients currently connected to the server
    qtssRTPSvrTotalConn             = 14,   //read      //UInt32    //Total number of clients since startup
    qtssRTPSvrCurBandwidth          = 15,   //read      //UInt32    //Current bandwidth being output by the server in bits per second
    qtssRTPSvrTotalBytes            = 16,   //read      //UInt64    //Total number of bytes served since startup
    qtssRTPSvrAvgBandwidth          = 17,   //read      //UInt32    //Average bandwidth being output by the server in bits per second
    qtssRTPSvrCurPackets            = 18,   //read      //UInt32    //Current packets per second being output by the server
    qtssRTPSvrTotalPackets          = 19,   //read      //UInt64    //Total number of bytes served since startup

    qtssSvrHandledMethods           = 20,   //r/w       //QTSS_RTSPMethod   //The methods that the server supports. Modules should append the methods they support to this attribute in their QTSS_Initialize_Role.
    qtssSvrModuleObjects            = 21,   //read  // this IS PREMPTIVE SAFE!  //QTSS_ModuleObject // A module object representing each module
    qtssSvrStartupTime              = 22,   //read      //QTSS_TimeVal  //Time the server started up
    qtssSvrGMTOffsetInHrs           = 23,   //read      //SInt32        //Server time zone (offset from GMT in hours)
    qtssSvrDefaultIPAddrStr         = 24,   //read      //char array    //The "default" IP address of the server as a string

    qtssSvrPreferences              = 25,   //read      //QTSS_PrefsObject  // An object representing each the server's preferences
    qtssSvrMessages                 = 26,   //read      //QTSS_Object   // An object containing the server's error messages.
    qtssSvrClientSessions           = 27,   //read      //QTSS_Object // An object containing all client sessions stored as indexed QTSS_ClientSessionObject(s).
    qtssSvrCurrentTimeMilliseconds  = 28,   //read      //QTSS_TimeVal  //Server's current time in milliseconds. Retrieving this attribute is equivalent to calling QTSS_Milliseconds
    qtssSvrCPULoadPercent           = 29,   //read      //Float32       //Current % CPU being used by the server

    qtssSvrNumReliableUDPBuffers    = 30,   //read      //UInt32    //Number of buffers currently allocated for UDP retransmits
    qtssSvrReliableUDPWastageInBytes= 31,   //read      //UInt32    //Amount of data in the reliable UDP buffers being wasted
    qtssSvrConnectedUsers           = 32,   //r/w       //QTSS_Object   //List of connected user sessions (updated by modules for their sessions)

    qtssMP3SvrCurConn               = 33,   //r/w       //UInt32    //Number of MP3 client sessions connected
    qtssMP3SvrTotalConn             = 34,   //r/w       //UInt32    //Total number of MP3  clients since startup
    qtssMP3SvrCurBandwidth          = 35,   //r/w       //UInt32    //Current MP3 bandwidth being output by the server in bits per second
    qtssMP3SvrTotalBytes            = 36,   //r/w       //UInt64    //Total number of MP3 bytes served since startup
    qtssMP3SvrAvgBandwidth          = 37,   //r/w       //UInt32    //Average MP3 bandwidth being output by the server in bits per second

    qtssSvrServerBuild              = 38,   //read      //char array //build of the server
    qtssSvrServerPlatform           = 39,   //read      //char array //Platform (OS) of the server
    qtssSvrRTSPServerComment        = 40,   //read      //char array //RTSP comment for the server header
    qtssSvrNumThinned               = 41,    //r/w      //SInt32    //Number of thinned sessions
    qtssSvrNumParams                = 42
};

[UPDATE]

There is a problem inside my template, where the max value of the bandwidth is too low.
If you have already created some graphs, you can use the rrdtool to change this :

/opt/monitor/rrdtool-1.2.12/bin/rrdtool tune   /opt/monitor/apache/htdocs/cacti/rra/cygnusadm_qtssrtpsvrbandwidth_1187.rrd --maximum  "qtssRTPSvrBandwidth:1e+12"
/opt/monitor/rrdtool-1.2.12/bin/rrdtool tune   /opt/monitor/apache/htdocs/cacti/rra/cygnusadm_qtssrtpsvrbandwidth_1187.rrd --maximum  "qtssMP3SvrBandwidth:1e+12"

10e+12 may be a bit high, but you won’t get any problem even with a loaded server

If you haven’t created any graphs yet, change the Cacti Data Template for DSS – Current Bandwidth.
For each Data Source Item, change the Maximum Value to 100000000000 like this :

If you haven’t imported the template yet, download it again, I changed it today (20070711)

Please excuse me for the mistake.

In fact, it is not the call to OS::GetNumProcessors() in Server.tproj/RunServer.cpp which is causing the problem. This call is done only once.

It’s the one in Server.tproj/QTSServerInterface.cpp.
In fact this part of the code is run every second for server statistics, leading to a problem with the slow unix command ‘uname -X’ and parsing of the output.

The best solution would be that the number of CPU is cached somewhere, so the server does not have to get it each time it have to divide the CPU usage !

One solution, as I said earlier, is to change this function to use something better than a unix command.
The other solution is to have the number of CPU cached.
The last solution is to remove the call to this function in Server.tproj/QTSServerInterface.cpp

If you do that you may end with a CPU load over 100%. But your server will not crash at 200 streams

I filled a bug at Apple and I hope it will be taken into account :
5316778
DSS crash on Solaris 10 after 200 streams

Vu que le format HDV n’est pas fait pour l’édition, et qu’Apple nous donne leur nouveau codec prores 422 gratuitement avec la suite FCS, j’ai fait 2 tests de re-encodage.
En partant du fichier HDV, j’ai fait une recompression en Prores 140×1080 et une en 1920×1080 en 50i.
Et voilà le résultat (dans l’ordre : HDV, AIC, Prores 140×1080 et 1920×1080) :

Ce que l’on remarque :
- le fichier HDV de base est plus sombre que les autres. Peut-être un changement de gamma ou d’espace colorimétrique (passage de YPrPb en RVB ?)
- le fichier AIC semble un peu plus flou
- le prores 1920×1080 semble un peu flou lui aussi -> il subit un changement de résolution, qui peut être cause le problème.

La question : si j’envoie le projet dans Color, le clips ont tous un problème d’entrelacement, sauf le prores 1920×1080. Pourquoi ?
2eme question : Dans color, le clip prores 1440×1080 est « stretch ». Il est moins large ! Color aurait il un problème avec les fichiers dont la taille des pixels n’est pas carrée ?
….

Je rajouterais des images de Color et du problème d’entrelacement de Motion plus tard.