Apple

New MacPro Nehalem and 10To Raid5 Array

overview
I bought a dual 2GHz Powermac G5 when it first came out, almost 6 years ago. I never had a problem with it. It’s still working as a charme. But, when you’re working with HD footage, it gets a real pain !

One of my best friend always tell me : « buy, don’t wait ». But I chosed to wait a little bit for the new Macpro Nehalem. And it finaly happened : http://www.apple.com/macpro/
Lire la suite de l’article »

Google is everywhere

Google Chart

One new API from Google. This one allow you to create charts (graphiques, pour les francophones).

Philippe Mougin is our guest and comes with a clean example with integration to Apple’s Cocoa programming. Check this here.

Faire du python dans automator (Leopard)

http://toxicsoftware.com/run-python-script

Voila le site d’une personne qui a une solution. Cela interessera particulierement mon ami Akhen, qui fete son anniversaire demain. Tiens, c’est pour toi : http://toxicsoftware.com/run-python-script/

Having DSS compiles on Solaris 10 [part 7] – DSS and Multicast streams

I finaly got DSS working fine on Solaris, with Multicast source on non primary interface.

This modified source-tree should be working with most Solaris 10 and maybe 8, 9, on Sparc or Intel. Just untar it, do a « ./buildtarball myversion » and use the tar.gz file for install. You may also need the Install script I modified to install in /opt/deflector/dss. I hope to find time to have the install dir on the command line so anybody can use it.

You will see a new parameter in streamingserver.xml where you define the IP you wish to subscribe Multicast streams on : <PREF NAME= »ip_listen_for_multicast » >0</PREF>
This parameter is set to 0 by default, which means a default behaviour, using INADDR_ANY as subscription IP. You can set it to an IP on your host to subscribe to a special interface like :
<PREF NAME= »ip_listen_for_multicast » >10.16.240.19</PREF>

It seems another solution would be to change the multicast lan route. Lets say your primary inteface have IP 10.16.248.19 and you have multicast comming to another interface whose IP is 10.16.240.19. You do it like this :

route -n delete -interface 224.0/4 -gateway 10.16.248.19
route -n add -interface 224.0/4 -gateway 10.16.240.19

My patch is not a end by itself as this IP should be, at least, be set per multicast stream. It’s just OK for my need, my lack of time and my small C++ experience.

Just come to me if you want to have more details about the config files or start/stop script.

By the way, the .tar.gz file with patched source is here and a patch file (diff file to apply using gpatch on Solaris) for the original source code is here

Having DSS compiles on Solaris 10 [part 6] – DSS and Multicast streams

And here we go again with DSS !
This time it’s not a real compilation problem and is not only Solaris related.

The best thing you can do with DSS, when you own the source part of the stream, is to use multicast. This way, you stream once from your encoder, and many DSS servers can access the stream and reflect it as unicast to your customers.

Once said, this sounds good and simple. In fact, not that simple if your DSS server is « multi homed ».

Multi homed is the way you say your server is using multiples IP on different networks, VLAN and/or NICs.

Our servers une MANY networks. One on the first NIC (nic 0) for admin tasks. NICs 1 and 2 are aggregated to have 2Gb bandwidth for data. On this Aggregated interface, we have many IP set on different networks and VLANs.

To cut the story short, the multicast stream comes on interface aggr543510 (vlan 543 on aggregated interface 510). Clients connect to admin NIC for test, and aggr536510 (vlan 536).

By default, DSS, when reflecting a multicast stream, bind any IP on the port supplied in the SDP file and subscribe to the multicast stream. This is pretty usual, and any other application would do the same.
The troubles comes when you are using multiple NICs. The bind is effectylevly done on * (any IP), but the multicast subscribe is done on ONLY ONE INTERFACE !

Let’s have a netstat -g (works on Solaris, on OS X, use netstat -a -i and try to find out the info on your own)

# netstat -g
Group Memberships: IPv4
Interface Group                RefCnt
--------- -------------------- ------
lo0       ALL-SYSTEMS.MCAST.NET      1
e1000g0   224.1.1.9                 1
e1000g0   ALL-SYSTEMS.MCAST.NET      1
e1000g34001:1 ALL-SYSTEMS.MCAST.NET      1
e1000g34001:2 ALL-SYSTEMS.MCAST.NET      1
e1000g36001:1 ALL-SYSTEMS.MCAST.NET      1
e1000g36001:2 ALL-SYSTEMS.MCAST.NET      1
e1000g244001 ALL-SYSTEMS.MCAST.NET      1
e1000g543001:1 ALL-SYSTEMS.MCAST.NET      1

The important line is the one with « e1000g0 224.1.1.9 « . There you can see that Multicast Network 224.1.1.9 is subscribed on interface e1000g0.
But e1000g0 is our admin interface !!!

So DSS will never see any data to reflect. And your client will be set to pause.

I Called Sun Support yesterday to have them comment about this. As I said, this is not really DSS related. It comed from the way the OS (every UNIX os ?) deals the multicast subscription. No news yet. I’m pretty I’ll be told that it is a normal behaviour.

Once that said, what will I do to have DSS work on my architecture ?

The key lies in CommonUtilitiesLib/UDPSocket.cpp
YES, we are back again to C++ programming !

The original source code says :

OS_Error UDPSocket::JoinMulticast(UInt32 inRemoteAddr)
{
    struct ip_mreq  theMulti;
        UInt32 localAddr = fLocalAddr.sin_addr.s_addr; // Already in network byte order


#if __solaris__
    if( localAddr == htonl(INADDR_ANY) )
         localAddr = htonl(SocketUtils::GetIPAddr(0));
#endif

    theMulti.imr_multiaddr.s_addr = htonl(inRemoteAddr);
    theMulti.imr_interface.s_addr = localAddr;


    int err = setsockopt(fFileDesc, IPPROTO_IP, IP_ADD_MEMBERSHIP, (char*)&theMulti, sizeof(theMulti));
    //AssertV(err == 0, OSThread::GetErrno());
    if (err == -1)
         return (OS_Error)OSThread::GetErrno();
    else
         return OS_NoErr;
}

There you can see 2 things :

  • There is a special part for Solaris
  • The interface (the IP) to bind to and to subscribe is set here
  • As I already said I’m not C++ expert. I just changed the code so the IP to subscribe, defined in « localAddr », is set to the IP I need to subscribe on :

    OS_Error UDPSocket::JoinMulticast(UInt32 inRemoteAddr)
    {
        struct ip_mreq  theMulti;
            UInt32 localAddr = fLocalAddr.sin_addr.s_addr; // Already in network byte order
    
    /*
    #if __solaris__
        if( localAddr == htonl(INADDR_ANY) )
             localAddr = htonl(SocketUtils::GetIPAddr(0));
    #endif
    */
    
    // Set by Prune - 20070712
        localAddr  = htonl(0x0a10f07b);
    
        theMulti.imr_multiaddr.s_addr = htonl(inRemoteAddr);
        theMulti.imr_interface.s_addr = localAddr;
    
    // Set by Prune - 20070712
    qtss_printf("Multicast local %x\n", localAddr);
    qtss_printf("Multicast remote %x\n", htonl(inRemoteAddr));
    
        int err = setsockopt(fFileDesc, IPPROTO_IP, IP_ADD_MEMBERSHIP, (char*)&theMulti, sizeof(theMulti));
        //AssertV(err == 0, OSThread::GetErrno());
        if (err == -1)
             return (OS_Error)OSThread::GetErrno();
        else
             return OS_NoErr;
    }
    

    So the local IP is set to « localAddr = htonl(0x0a10f07b); » where 0a10f07b is the hexa value of IP 10.16.240.123
    I also add some debug that you can see if you start DSS with the -d option.

    Then everything is OK and you have the right interface subscribed :

    # netstat -g
    Group Memberships: IPv4
    Interface Group                RefCnt
    --------- -------------------- ------
    lo0       ALL-SYSTEMS.MCAST.NET      1
    e1000g0   ALL-SYSTEMS.MCAST.NET      1
    e1000g34001:1 ALL-SYSTEMS.MCAST.NET      1
    e1000g34001:2 ALL-SYSTEMS.MCAST.NET      1
    e1000g36001:1 ALL-SYSTEMS.MCAST.NET      1
    e1000g36001:2 ALL-SYSTEMS.MCAST.NET      1
    e1000g244001 ALL-SYSTEMS.MCAST.NET      1
    e1000g543001:1 224.1.1.9                 1
    e1000g543001:1 ALL-SYSTEMS.MCAST.NET      1

    OK, I have to do it better, like getting the interface to bind from the config file. This was just a late night test :)

    But what is the right way to do it ?
    The server (DSS process) can’t know what interface to subscribe to. Does it have to subscribe it on all interfaces ? This does not seem possible other than opening many sockets. So the server will then have to decide which one to use. tricky isn’t it ?

    That is where I am actualy. No real good solution but a hack to have it working.
    Then, what will happen if one day I want another multicast stream coming from another network ? …
    Keep reading here as I’ll post any information I have if Sun or DSS mailing list answer me.

    Collecting infos of DSS in Cacti

    I had some time yesteday to add DSS monitoring into Cacti SNMP Monitoring tool.
    I decided to break the data collection into 2 separate graphs : one for current number of connections and one for current server bandwidth. Of course you will add other host graph, like NIC bandwidth and CPU to get a real view of your server’s health.
    So far, here is what you will have :

    Cacti DSS Connections
    Cacti DSS Bandwidth

    When I have my whole architecture up and running I will go for data aggregation to get the full connections over all servers.

    You will find 2 Cacti Graph Templates in the download section, with the 2 python scripts to collect data.
    The python scripts have to be copied in the cacti/scripts directory, while the XML files have to be imported using the « Import Template » link in the Cacti Console.

    Bandwidth template and script
    Connection template an script

    For the python scripts to work, you will have to enable the « remote admin » in DSS. Change this in your streamingserver.xml file :
    – set LocalAccessOnly to false
    – add your IP’s in the IPAccessList
    – check enable_remote_admin is true, which is the default
    – use your admin user or add anoter one in the admin group using qtpasswd

            
                    10.16.248.10
                    true
                    false
                    50
                    true
                    admin
            
    

    In this case your remote IP is 10.16.248.10
    You can define multiple IP’s like this :

            
                    
                            127.0.0.*
                            10.16.248.*
                    
                    true
                    false
                    50
                    true
                    admin
            
    

    I also provide a simple python script to get easily any DSS Admin URL : dss-query_url.py

    The usage is simple (but you may have to add python before it or change the python link in the first line of the script). You just need to provide the hostname, the port (554), the login of an admin account (the account you use to log in the HTML interface – the one you define at the install stage) and its password.
    The you can supply any URL. You will learn more about Admin URL in the « Documentation/admin-protocol-README.txt » file in the source tree.
    You will also find some usefull informations on each fields in the file « APIStubLib/QTSS.h ».

    Let’s have an example :

    python dss-query_url.py myhost.mydomain.com  554 admin password /modules/admin/server/*?command=get
    Container="/admin/server/"
    qtssServerAPIVersion="262144"
    qtssSvrDefaultDNSName="0.0.0.0"
    qtssSvrDefaultIPAddr="0"
    qtssSvrServerName="DSS"
    qtssRTSPSvrServerVersion="5.5.5"
    qtssRTSPSvrServerBuildDate="Jul  6 2007, 16:35:39"
    qtssSvrRTSPServerHeader="Server: DSS/5.5.5 (Build/489.16; Platform/Solaris; Release/Darwin; state/beta; )"
    qtssSvrState="1"
    qtssSvrIsOutOfDescriptors="false"
    qtssRTSPCurrentSessionCount="127"
    qtssRTSPHTTPCurrentSessionCount="0"
    qtssRTPSvrNumUDPSockets="8"
    qtssRTPSvrCurConn="126"
    qtssRTPSvrTotalConn="256"
    qtssRTPSvrCurBandwidth="5027784"
    qtssRTPSvrTotalBytes="3809807612"
    qtssRTPSvrAvgBandwidth="3247539"
    qtssRTPSvrCurPackets="776"
    qtssRTPSvrTotalPackets="10865424"
    qtssSvrStartupTime="1183999975017"
    qtssSvrGMTOffsetInHrs="-20629"
    qtssSvrDefaultIPAddrStr="0.0.0.0"
    qtssSvrCurrentTimeMilliseconds="1184070706181"
    qtssSvrCPULoadPercent="23.004150"
    qtssSvrNumReliableUDPBuffers="0"
    qtssSvrReliableUDPWastageInBytes="0"
    qtssMP3SvrCurConn="0"
    qtssMP3SvrTotalConn="0"
    qtssMP3SvrCurBandwidth="0"
    qtssMP3SvrTotalBytes="0"
    qtssMP3SvrAvgBandwidth="0"
    qtssSvrServerBuild="489.16"
    qtssSvrServerPlatform="Solaris"
    qtssSvrRTSPServerComment="Release/Darwin; state/beta; "
    qtssSvrNumThinned="113"
    qtssSvrRTSPPorts/
    qtssSvrHandledMethods/
    qtssSvrModuleObjects/
    qtssSvrPreferences/
    qtssSvrMessages/
    qtssSvrClientSessions/
    qtssSvrConnectedUsers/
    error:(0)

    Quoting the « APIStubLib/QTSS.h » file :

        // These parameters ARE pre-emptive safe.
        
        qtssServerAPIVersion            = 0,    //read  //UInt32            //The API version supported by this server (format 0xMMMMmmmm, where M=major version, m=minor version)
        qtssSvrDefaultDNSName           = 1,    //read  //char array        //The "default" DNS name of the server
        qtssSvrDefaultIPAddr            = 2,    //read  //UInt32            //The "default" IP address of the server
        qtssSvrServerName               = 3,    //read  //char array        //Name of the server
        qtssSvrServerVersion            = 4,    //read  //char array        //Version of the server
        qtssSvrServerBuildDate          = 5,    //read  //char array        //When was the server built?
        qtssSvrRTSPPorts                = 6,    //read  // NOT PREEMPTIVE SAFE!//UInt16         //Indexed parameter: all the ports the server is listening on
        qtssSvrRTSPServerHeader         = 7,    //read  //char array        //Server: header that the server uses to respond to RTSP clients
    
        // These parameters are NOT pre-emptive safe, they cannot be accessed
        // via. QTSS_GetValuePtr. Some exceptions noted below
        
        qtssSvrState                    = 8,    //r/w   //QTSS_ServerState  //The current state of the server. If a module sets the server state, the server will respond in the appropriate fashion. Setting to qtssRefusingCon
    nectionsState causes the server to refuse connections, setting to qtssFatalErrorState or qtssShuttingDownState causes the server to quit.
        qtssSvrIsOutOfDescriptors       = 9,    //read  //Bool16            //true if the server has run out of file descriptors, false otherwise
        qtssRTSPCurrentSessionCount     = 10,   //read  //UInt32            //Current number of connected clients over standard RTSP
        qtssRTSPHTTPCurrentSessionCount = 11,   //read  //UInt32            //Current number of connected clients over RTSP / HTTP
    
        qtssRTPSvrNumUDPSockets         = 12,   //read      //UInt32    //Number of UDP sockets currently being used by the server
        qtssRTPSvrCurConn               = 13,   //read      //UInt32    //Number of clients currently connected to the server
        qtssRTPSvrTotalConn             = 14,   //read      //UInt32    //Total number of clients since startup
        qtssRTPSvrCurBandwidth          = 15,   //read      //UInt32    //Current bandwidth being output by the server in bits per second
        qtssRTPSvrTotalBytes            = 16,   //read      //UInt64    //Total number of bytes served since startup
        qtssRTPSvrAvgBandwidth          = 17,   //read      //UInt32    //Average bandwidth being output by the server in bits per second
        qtssRTPSvrCurPackets            = 18,   //read      //UInt32    //Current packets per second being output by the server
        qtssRTPSvrTotalPackets          = 19,   //read      //UInt64    //Total number of bytes served since startup
        
        qtssSvrHandledMethods           = 20,   //r/w       //QTSS_RTSPMethod   //The methods that the server supports. Modules should append the methods they support to this attribute in their QTSS_Initialize_Role.
        qtssSvrModuleObjects            = 21,   //read  // this IS PREMPTIVE SAFE!  //QTSS_ModuleObject // A module object representing each module
        qtssSvrStartupTime              = 22,   //read      //QTSS_TimeVal  //Time the server started up
        qtssSvrGMTOffsetInHrs           = 23,   //read      //SInt32        //Server time zone (offset from GMT in hours)
        qtssSvrDefaultIPAddrStr         = 24,   //read      //char array    //The "default" IP address of the server as a string
    
        qtssSvrPreferences              = 25,   //read      //QTSS_PrefsObject  // An object representing each the server's preferences
        qtssSvrMessages                 = 26,   //read      //QTSS_Object   // An object containing the server's error messages.
        qtssSvrClientSessions           = 27,   //read      //QTSS_Object // An object containing all client sessions stored as indexed QTSS_ClientSessionObject(s).
        qtssSvrCurrentTimeMilliseconds  = 28,   //read      //QTSS_TimeVal  //Server's current time in milliseconds. Retrieving this attribute is equivalent to calling QTSS_Milliseconds
        qtssSvrCPULoadPercent           = 29,   //read      //Float32       //Current % CPU being used by the server
    
        qtssSvrNumReliableUDPBuffers    = 30,   //read      //UInt32    //Number of buffers currently allocated for UDP retransmits
        qtssSvrReliableUDPWastageInBytes= 31,   //read      //UInt32    //Amount of data in the reliable UDP buffers being wasted
        qtssSvrConnectedUsers           = 32,   //r/w       //QTSS_Object   //List of connected user sessions (updated by modules for their sessions)
        
        qtssMP3SvrCurConn               = 33,   //r/w       //UInt32    //Number of MP3 client sessions connected
        qtssMP3SvrTotalConn             = 34,   //r/w       //UInt32    //Total number of MP3  clients since startup
        qtssMP3SvrCurBandwidth          = 35,   //r/w       //UInt32    //Current MP3 bandwidth being output by the server in bits per second
        qtssMP3SvrTotalBytes            = 36,   //r/w       //UInt64    //Total number of MP3 bytes served since startup
        qtssMP3SvrAvgBandwidth          = 37,   //r/w       //UInt32    //Average MP3 bandwidth being output by the server in bits per second
    
        qtssSvrServerBuild              = 38,   //read      //char array //build of the server
        qtssSvrServerPlatform           = 39,   //read      //char array //Platform (OS) of the server
        qtssSvrRTSPServerComment        = 40,   //read      //char array //RTSP comment for the server header    
        qtssSvrNumThinned               = 41,    //r/w      //SInt32    //Number of thinned sessions
        qtssSvrNumParams                = 42
    };
    

    [UPDATE]

    There is a problem inside my template, where the max value of the bandwidth is too low.
    If you have already created some graphs, you can use the rrdtool to change this :

    /opt/monitor/rrdtool-1.2.12/bin/rrdtool tune   /opt/monitor/apache/htdocs/cacti/rra/cygnusadm_qtssrtpsvrbandwidth_1187.rrd --maximum  "qtssRTPSvrBandwidth:1e+12"
    /opt/monitor/rrdtool-1.2.12/bin/rrdtool tune   /opt/monitor/apache/htdocs/cacti/rra/cygnusadm_qtssrtpsvrbandwidth_1187.rrd --maximum  "qtssMP3SvrBandwidth:1e+12"
    

    10e+12 may be a bit high, but you won’t get any problem even with a loaded server :)

    If you haven’t created any graphs yet, change the Cacti Data Template for DSS – Current Bandwidth.
    For each Data Source Item, change the Maximum Value to 100000000000 like this :
    cacti data template

    If you haven’t imported the template yet, download it again, I changed it today (20070711)

    Please excuse me for the mistake.

    Having DSS compiles on Solaris 10 [part 5]

    This should be the last part :)

    In fact, it is not the call to OS::GetNumProcessors() in Server.tproj/RunServer.cpp which is causing the problem. This call is done only once.

    It’s the one in Server.tproj/QTSServerInterface.cpp.
    In fact this part of the code is run every second for server statistics, leading to a problem with the slow unix command ‘uname -X’ and parsing of the output.

    The best solution would be that the number of CPU is cached somewhere, so the server does not have to get it each time it have to divide the CPU usage !
    :)

    One solution, as I said earlier, is to change this function to use something better than a unix command.
    The other solution is to have the number of CPU cached.
    The last solution is to remove the call to this function in Server.tproj/QTSServerInterface.cpp

    If you do that you may end with a CPU load over 100%. But your server will not crash at 200 streams :)

    I filled a bug at Apple and I hope it will be taken into account :
    5316778
    DSS crash on Solaris 10 after 200 streams

    Un logiciel de son opensource pour OS X

    Ca s’appel Ardour et ca tourne sur osX, en universal binary. Pour le moment il faut installer Jack et l’interface est en X11. Ca semble puissant, je vais installer ca a la maison pour tester.

    Comparaison de codecs sur du HDV

    J’ai fait un petit test avec la camera HDV d’un ami, une Sony FX1.
    J’ai ensuite fait 2 imports dans FinalCut Pro 6 : un import en HDV et un import en Apple Intermediate Codec.

    Vu que le format HDV n’est pas fait pour l’édition, et qu’Apple nous donne leur nouveau codec prores 422 gratuitement avec la suite FCS, j’ai fait 2 tests de re-encodage.
    En partant du fichier HDV, j’ai fait une recompression en Prores 140×1080 et une en 1920×1080 en 50i.
    Et voilà le résultat (dans l’ordre : HDV, AIC, Prores 140×1080 et 1920×1080) :

    test codec 1
    test codec 2

    Ce que l’on remarque :
    – le fichier HDV de base est plus sombre que les autres. Peut-être un changement de gamma ou d’espace colorimétrique (passage de YPrPb en RVB ?)
    – le fichier AIC semble un peu plus flou
    – le prores 1920×1080 semble un peu flou lui aussi -> il subit un changement de résolution, qui peut être cause le problème.

    La question : si j’envoie le projet dans Color, le clips ont tous un problème d’entrelacement, sauf le prores 1920×1080. Pourquoi ?
    2eme question : Dans color, le clip prores 1440×1080 est « stretch ». Il est moins large ! Color aurait il un problème avec les fichiers dont la taille des pixels n’est pas carrée ?
    ….

    Je rajouterais des images de Color et du problème d’entrelacement de Motion plus tard.

    en attendant l’iphone…

    fabiquez le votre en carton ici grace a ce PDF.