Internet, UNIX, Video, Leisure…
Apple
New MacPro Nehalem and 10To Raid5 Array
29/03/09

I bought a dual 2GHz Powermac G5 when it first came out, almost 6 years ago. I never had a problem with it. It’s still working as a charme. But, when you’re working with HD footage, it gets a real pain !
One of my best friend always tell me : « buy, don’t wait ». But I chosed to wait a little bit for the new Macpro Nehalem. And it finaly happened : http://www.apple.com/macpro/
Lire la suite de l’article »
Google is everywhere
28/12/07

One new API from Google. This one allow you to create charts (graphiques, pour les francophones).
Philippe Mougin is our guest and comes with a clean example with integration to Apple’s Cocoa programming. Check this here.
Faire du python dans automator (Leopard)
28/12/07

Voila le site d’une personne qui a une solution. Cela interessera particulierement mon ami Akhen, qui fete son anniversaire demain. Tiens, c’est pour toi : http://toxicsoftware.com/run-python-script/
Having DSS compiles on Solaris 10 [part 7] – DSS and Multicast streams
16/07/07
I finaly got DSS working fine on Solaris, with Multicast source on non primary interface.
This modified source-tree should be working with most Solaris 10 and maybe 8, 9, on Sparc or Intel. Just untar it, do a « ./buildtarball myversion » and use the tar.gz file for install. You may also need the Install script I modified to install in /opt/deflector/dss. I hope to find time to have the install dir on the command line so anybody can use it.
You will see a new parameter in streamingserver.xml where you define the IP you wish to subscribe Multicast streams on : <PREF NAME= »ip_listen_for_multicast » >0</PREF>
This parameter is set to 0 by default, which means a default behaviour, using INADDR_ANY as subscription IP. You can set it to an IP on your host to subscribe to a special interface like :
<PREF NAME= »ip_listen_for_multicast » >10.16.240.19</PREF>
It seems another solution would be to change the multicast lan route. Lets say your primary inteface have IP 10.16.248.19 and you have multicast comming to another interface whose IP is 10.16.240.19. You do it like this :
route -n delete -interface 224.0/4 -gateway 10.16.248.19
route -n add -interface 224.0/4 -gateway 10.16.240.19
My patch is not a end by itself as this IP should be, at least, be set per multicast stream. It’s just OK for my need, my lack of time and my small C++ experience.
Just come to me if you want to have more details about the config files or start/stop script.
By the way, the .tar.gz file with patched source is here and a patch file (diff file to apply using gpatch on Solaris) for the original source code is here
Having DSS compiles on Solaris 10 [part 6] – DSS and Multicast streams
13/07/07
And here we go again with DSS !
This time it’s not a real compilation problem and is not only Solaris related.
The best thing you can do with DSS, when you own the source part of the stream, is to use multicast. This way, you stream once from your encoder, and many DSS servers can access the stream and reflect it as unicast to your customers.
Once said, this sounds good and simple. In fact, not that simple if your DSS server is « multi homed ».
Multi homed is the way you say your server is using multiples IP on different networks, VLAN and/or NICs.
Our servers une MANY networks. One on the first NIC (nic 0) for admin tasks. NICs 1 and 2 are aggregated to have 2Gb bandwidth for data. On this Aggregated interface, we have many IP set on different networks and VLANs.
To cut the story short, the multicast stream comes on interface aggr543510 (vlan 543 on aggregated interface 510). Clients connect to admin NIC for test, and aggr536510 (vlan 536).
By default, DSS, when reflecting a multicast stream, bind any IP on the port supplied in the SDP file and subscribe to the multicast stream. This is pretty usual, and any other application would do the same.
The troubles comes when you are using multiple NICs. The bind is effectylevly done on * (any IP), but the multicast subscribe is done on ONLY ONE INTERFACE !
Let’s have a netstat -g (works on Solaris, on OS X, use netstat -a -i and try to find out the info on your own)
# netstat -g
Group Memberships: IPv4
Interface Group RefCnt
--------- -------------------- ------
lo0 ALL-SYSTEMS.MCAST.NET 1
e1000g0 224.1.1.9 1
e1000g0 ALL-SYSTEMS.MCAST.NET 1
e1000g34001:1 ALL-SYSTEMS.MCAST.NET 1
e1000g34001:2 ALL-SYSTEMS.MCAST.NET 1
e1000g36001:1 ALL-SYSTEMS.MCAST.NET 1
e1000g36001:2 ALL-SYSTEMS.MCAST.NET 1
e1000g244001 ALL-SYSTEMS.MCAST.NET 1
e1000g543001:1 ALL-SYSTEMS.MCAST.NET 1
The important line is the one with « e1000g0 224.1.1.9 « . There you can see that Multicast Network 224.1.1.9 is subscribed on interface e1000g0.
But e1000g0 is our admin interface !!!
So DSS will never see any data to reflect. And your client will be set to pause.
I Called Sun Support yesterday to have them comment about this. As I said, this is not really DSS related. It comed from the way the OS (every UNIX os ?) deals the multicast subscription. No news yet. I’m pretty I’ll be told that it is a normal behaviour.
…
Once that said, what will I do to have DSS work on my architecture ?
The key lies in CommonUtilitiesLib/UDPSocket.cpp
YES, we are back again to C++ programming !
The original source code says :
OS_Error UDPSocket::JoinMulticast(UInt32 inRemoteAddr)
{
struct ip_mreq theMulti;
UInt32 localAddr = fLocalAddr.sin_addr.s_addr; // Already in network byte order
#if __solaris__
if( localAddr == htonl(INADDR_ANY) )
localAddr = htonl(SocketUtils::GetIPAddr(0));
#endif
theMulti.imr_multiaddr.s_addr = htonl(inRemoteAddr);
theMulti.imr_interface.s_addr = localAddr;
int err = setsockopt(fFileDesc, IPPROTO_IP, IP_ADD_MEMBERSHIP, (char*)&theMulti, sizeof(theMulti));
//AssertV(err == 0, OSThread::GetErrno());
if (err == -1)
return (OS_Error)OSThread::GetErrno();
else
return OS_NoErr;
}
There you can see 2 things :
As I already said I’m not C++ expert. I just changed the code so the IP to subscribe, defined in « localAddr », is set to the IP I need to subscribe on :
OS_Error UDPSocket::JoinMulticast(UInt32 inRemoteAddr)
{
struct ip_mreq theMulti;
UInt32 localAddr = fLocalAddr.sin_addr.s_addr; // Already in network byte order
/*
#if __solaris__
if( localAddr == htonl(INADDR_ANY) )
localAddr = htonl(SocketUtils::GetIPAddr(0));
#endif
*/
// Set by Prune - 20070712
localAddr = htonl(0x0a10f07b);
theMulti.imr_multiaddr.s_addr = htonl(inRemoteAddr);
theMulti.imr_interface.s_addr = localAddr;
// Set by Prune - 20070712
qtss_printf("Multicast local %x\n", localAddr);
qtss_printf("Multicast remote %x\n", htonl(inRemoteAddr));
int err = setsockopt(fFileDesc, IPPROTO_IP, IP_ADD_MEMBERSHIP, (char*)&theMulti, sizeof(theMulti));
//AssertV(err == 0, OSThread::GetErrno());
if (err == -1)
return (OS_Error)OSThread::GetErrno();
else
return OS_NoErr;
}
So the local IP is set to « localAddr = htonl(0x0a10f07b); » where 0a10f07b is the hexa value of IP 10.16.240.123
I also add some debug that you can see if you start DSS with the -d option.
Then everything is OK and you have the right interface subscribed :
# netstat -g
Group Memberships: IPv4
Interface Group RefCnt
--------- -------------------- ------
lo0 ALL-SYSTEMS.MCAST.NET 1
e1000g0 ALL-SYSTEMS.MCAST.NET 1
e1000g34001:1 ALL-SYSTEMS.MCAST.NET 1
e1000g34001:2 ALL-SYSTEMS.MCAST.NET 1
e1000g36001:1 ALL-SYSTEMS.MCAST.NET 1
e1000g36001:2 ALL-SYSTEMS.MCAST.NET 1
e1000g244001 ALL-SYSTEMS.MCAST.NET 1
e1000g543001:1 224.1.1.9 1
e1000g543001:1 ALL-SYSTEMS.MCAST.NET 1
OK, I have to do it better, like getting the interface to bind from the config file. This was just a late night test
But what is the right way to do it ?
The server (DSS process) can’t know what interface to subscribe to. Does it have to subscribe it on all interfaces ? This does not seem possible other than opening many sockets. So the server will then have to decide which one to use. tricky isn’t it ?
That is where I am actualy. No real good solution but a hack to have it working.
Then, what will happen if one day I want another multicast stream coming from another network ? …
Keep reading here as I’ll post any information I have if Sun or DSS mailing list answer me.
Collecting infos of DSS in Cacti
10/07/07
I had some time yesteday to add DSS monitoring into Cacti SNMP Monitoring tool.
I decided to break the data collection into 2 separate graphs : one for current number of connections and one for current server bandwidth. Of course you will add other host graph, like NIC bandwidth and CPU to get a real view of your server’s health.
So far, here is what you will have :


When I have my whole architecture up and running I will go for data aggregation to get the full connections over all servers.
You will find 2 Cacti Graph Templates in the download section, with the 2 python scripts to collect data.
The python scripts have to be copied in the cacti/scripts directory, while the XML files have to be imported using the « Import Template » link in the Cacti Console.
Bandwidth template and script
Connection template an script
For the python scripts to work, you will have to enable the « remote admin » in DSS. Change this in your streamingserver.xml file :
- set LocalAccessOnly to false
- add your IP’s in the IPAccessList
- check enable_remote_admin is true, which is the default
- use your admin user or add anoter one in the admin group using qtpasswd
10.16.248.10
true
false
50
true
admin
In this case your remote IP is 10.16.248.10
You can define multiple IP’s like this :
127.0.0.*
10.16.248.*
true
false
50
true
admin
I also provide a simple python script to get easily any DSS Admin URL : dss-query_url.py
The usage is simple (but you may have to add python before it or change the python link in the first line of the script). You just need to provide the hostname, the port (554), the login of an admin account (the account you use to log in the HTML interface – the one you define at the install stage) and its password.
The you can supply any URL. You will learn more about Admin URL in the « Documentation/admin-protocol-README.txt » file in the source tree.
You will also find some usefull informations on each fields in the file « APIStubLib/QTSS.h ».
Let’s have an example :
python dss-query_url.py myhost.mydomain.com 554 admin password /modules/admin/server/*?command=get
Container="/admin/server/"
qtssServerAPIVersion="262144"
qtssSvrDefaultDNSName="0.0.0.0"
qtssSvrDefaultIPAddr="0"
qtssSvrServerName="DSS"
qtssRTSPSvrServerVersion="5.5.5"
qtssRTSPSvrServerBuildDate="Jul 6 2007, 16:35:39"
qtssSvrRTSPServerHeader="Server: DSS/5.5.5 (Build/489.16; Platform/Solaris; Release/Darwin; state/beta; )"
qtssSvrState="1"
qtssSvrIsOutOfDescriptors="false"
qtssRTSPCurrentSessionCount="127"
qtssRTSPHTTPCurrentSessionCount="0"
qtssRTPSvrNumUDPSockets="8"
qtssRTPSvrCurConn="126"
qtssRTPSvrTotalConn="256"
qtssRTPSvrCurBandwidth="5027784"
qtssRTPSvrTotalBytes="3809807612"
qtssRTPSvrAvgBandwidth="3247539"
qtssRTPSvrCurPackets="776"
qtssRTPSvrTotalPackets="10865424"
qtssSvrStartupTime="1183999975017"
qtssSvrGMTOffsetInHrs="-20629"
qtssSvrDefaultIPAddrStr="0.0.0.0"
qtssSvrCurrentTimeMilliseconds="1184070706181"
qtssSvrCPULoadPercent="23.004150"
qtssSvrNumReliableUDPBuffers="0"
qtssSvrReliableUDPWastageInBytes="0"
qtssMP3SvrCurConn="0"
qtssMP3SvrTotalConn="0"
qtssMP3SvrCurBandwidth="0"
qtssMP3SvrTotalBytes="0"
qtssMP3SvrAvgBandwidth="0"
qtssSvrServerBuild="489.16"
qtssSvrServerPlatform="Solaris"
qtssSvrRTSPServerComment="Release/Darwin; state/beta; "
qtssSvrNumThinned="113"
qtssSvrRTSPPorts/
qtssSvrHandledMethods/
qtssSvrModuleObjects/
qtssSvrPreferences/
qtssSvrMessages/
qtssSvrClientSessions/
qtssSvrConnectedUsers/
error:(0)
Quoting the « APIStubLib/QTSS.h » file :
// These parameters ARE pre-emptive safe.
qtssServerAPIVersion = 0, //read //UInt32 //The API version supported by this server (format 0xMMMMmmmm, where M=major version, m=minor version)
qtssSvrDefaultDNSName = 1, //read //char array //The "default" DNS name of the server
qtssSvrDefaultIPAddr = 2, //read //UInt32 //The "default" IP address of the server
qtssSvrServerName = 3, //read //char array //Name of the server
qtssSvrServerVersion = 4, //read //char array //Version of the server
qtssSvrServerBuildDate = 5, //read //char array //When was the server built?
qtssSvrRTSPPorts = 6, //read // NOT PREEMPTIVE SAFE!//UInt16 //Indexed parameter: all the ports the server is listening on
qtssSvrRTSPServerHeader = 7, //read //char array //Server: header that the server uses to respond to RTSP clients
// These parameters are NOT pre-emptive safe, they cannot be accessed
// via. QTSS_GetValuePtr. Some exceptions noted below
qtssSvrState = 8, //r/w //QTSS_ServerState //The current state of the server. If a module sets the server state, the server will respond in the appropriate fashion. Setting to qtssRefusingCon
nectionsState causes the server to refuse connections, setting to qtssFatalErrorState or qtssShuttingDownState causes the server to quit.
qtssSvrIsOutOfDescriptors = 9, //read //Bool16 //true if the server has run out of file descriptors, false otherwise
qtssRTSPCurrentSessionCount = 10, //read //UInt32 //Current number of connected clients over standard RTSP
qtssRTSPHTTPCurrentSessionCount = 11, //read //UInt32 //Current number of connected clients over RTSP / HTTP
qtssRTPSvrNumUDPSockets = 12, //read //UInt32 //Number of UDP sockets currently being used by the server
qtssRTPSvrCurConn = 13, //read //UInt32 //Number of clients currently connected to the server
qtssRTPSvrTotalConn = 14, //read //UInt32 //Total number of clients since startup
qtssRTPSvrCurBandwidth = 15, //read //UInt32 //Current bandwidth being output by the server in bits per second
qtssRTPSvrTotalBytes = 16, //read //UInt64 //Total number of bytes served since startup
qtssRTPSvrAvgBandwidth = 17, //read //UInt32 //Average bandwidth being output by the server in bits per second
qtssRTPSvrCurPackets = 18, //read //UInt32 //Current packets per second being output by the server
qtssRTPSvrTotalPackets = 19, //read //UInt64 //Total number of bytes served since startup
qtssSvrHandledMethods = 20, //r/w //QTSS_RTSPMethod //The methods that the server supports. Modules should append the methods they support to this attribute in their QTSS_Initialize_Role.
qtssSvrModuleObjects = 21, //read // this IS PREMPTIVE SAFE! //QTSS_ModuleObject // A module object representing each module
qtssSvrStartupTime = 22, //read //QTSS_TimeVal //Time the server started up
qtssSvrGMTOffsetInHrs = 23, //read //SInt32 //Server time zone (offset from GMT in hours)
qtssSvrDefaultIPAddrStr = 24, //read //char array //The "default" IP address of the server as a string
qtssSvrPreferences = 25, //read //QTSS_PrefsObject // An object representing each the server's preferences
qtssSvrMessages = 26, //read //QTSS_Object // An object containing the server's error messages.
qtssSvrClientSessions = 27, //read //QTSS_Object // An object containing all client sessions stored as indexed QTSS_ClientSessionObject(s).
qtssSvrCurrentTimeMilliseconds = 28, //read //QTSS_TimeVal //Server's current time in milliseconds. Retrieving this attribute is equivalent to calling QTSS_Milliseconds
qtssSvrCPULoadPercent = 29, //read //Float32 //Current % CPU being used by the server
qtssSvrNumReliableUDPBuffers = 30, //read //UInt32 //Number of buffers currently allocated for UDP retransmits
qtssSvrReliableUDPWastageInBytes= 31, //read //UInt32 //Amount of data in the reliable UDP buffers being wasted
qtssSvrConnectedUsers = 32, //r/w //QTSS_Object //List of connected user sessions (updated by modules for their sessions)
qtssMP3SvrCurConn = 33, //r/w //UInt32 //Number of MP3 client sessions connected
qtssMP3SvrTotalConn = 34, //r/w //UInt32 //Total number of MP3 clients since startup
qtssMP3SvrCurBandwidth = 35, //r/w //UInt32 //Current MP3 bandwidth being output by the server in bits per second
qtssMP3SvrTotalBytes = 36, //r/w //UInt64 //Total number of MP3 bytes served since startup
qtssMP3SvrAvgBandwidth = 37, //r/w //UInt32 //Average MP3 bandwidth being output by the server in bits per second
qtssSvrServerBuild = 38, //read //char array //build of the server
qtssSvrServerPlatform = 39, //read //char array //Platform (OS) of the server
qtssSvrRTSPServerComment = 40, //read //char array //RTSP comment for the server header
qtssSvrNumThinned = 41, //r/w //SInt32 //Number of thinned sessions
qtssSvrNumParams = 42
};
[UPDATE]
There is a problem inside my template, where the max value of the bandwidth is too low.
If you have already created some graphs, you can use the rrdtool to change this :
/opt/monitor/rrdtool-1.2.12/bin/rrdtool tune /opt/monitor/apache/htdocs/cacti/rra/cygnusadm_qtssrtpsvrbandwidth_1187.rrd --maximum "qtssRTPSvrBandwidth:1e+12"
/opt/monitor/rrdtool-1.2.12/bin/rrdtool tune /opt/monitor/apache/htdocs/cacti/rra/cygnusadm_qtssrtpsvrbandwidth_1187.rrd --maximum "qtssMP3SvrBandwidth:1e+12"
10e+12 may be a bit high, but you won’t get any problem even with a loaded server
If you haven’t created any graphs yet, change the Cacti Data Template for DSS – Current Bandwidth.
For each Data Source Item, change the Maximum Value to 100000000000 like this :

If you haven’t imported the template yet, download it again, I changed it today (20070711)
Please excuse me for the mistake.
Having DSS compiles on Solaris 10 [part 5]
6/07/07
This should be the last part
In fact, it is not the call to OS::GetNumProcessors() in Server.tproj/RunServer.cpp which is causing the problem. This call is done only once.
It’s the one in Server.tproj/QTSServerInterface.cpp.
In fact this part of the code is run every second for server statistics, leading to a problem with the slow unix command ‘uname -X’ and parsing of the output.
The best solution would be that the number of CPU is cached somewhere, so the server does not have to get it each time it have to divide the CPU usage !
One solution, as I said earlier, is to change this function to use something better than a unix command.
The other solution is to have the number of CPU cached.
The last solution is to remove the call to this function in Server.tproj/QTSServerInterface.cpp
If you do that you may end with a CPU load over 100%. But your server will not crash at 200 streams
I filled a bug at Apple and I hope it will be taken into account :
5316778
DSS crash on Solaris 10 after 200 streams
Un logiciel de son opensource pour OS X
4/07/07
Ca s’appel Ardour et ca tourne sur osX, en universal binary. Pour le moment il faut installer Jack et l’interface est en X11. Ca semble puissant, je vais installer ca a la maison pour tester.
Comparaison de codecs sur du HDV
13/06/07
J’ai fait un petit test avec la camera HDV d’un ami, une Sony FX1.
J’ai ensuite fait 2 imports dans FinalCut Pro 6 : un import en HDV et un import en Apple Intermediate Codec.
Vu que le format HDV n’est pas fait pour l’édition, et qu’Apple nous donne leur nouveau codec prores 422 gratuitement avec la suite FCS, j’ai fait 2 tests de re-encodage.
En partant du fichier HDV, j’ai fait une recompression en Prores 140×1080 et une en 1920×1080 en 50i.
Et voilà le résultat (dans l’ordre : HDV, AIC, Prores 140×1080 et 1920×1080) :
Ce que l’on remarque :
- le fichier HDV de base est plus sombre que les autres. Peut-être un changement de gamma ou d’espace colorimétrique (passage de YPrPb en RVB ?)
- le fichier AIC semble un peu plus flou
- le prores 1920×1080 semble un peu flou lui aussi -> il subit un changement de résolution, qui peut être cause le problème.
La question : si j’envoie le projet dans Color, le clips ont tous un problème d’entrelacement, sauf le prores 1920×1080. Pourquoi ?
2eme question : Dans color, le clip prores 1440×1080 est « stretch ». Il est moins large ! Color aurait il un problème avec les fichiers dont la taille des pixels n’est pas carrée ?
….
Je rajouterais des images de Color et du problème d’entrelacement de Motion plus tard.