Internet, UNIX, Video, Leisure…
12Tb RAID for cheap with Hitachi 2Tb drives
It’s been some time now that i’m fighting with Seagate for they damn 1.5To drives not working as they should.
Hitachi finaly released 2Tb drives with good throughoutput and price.
While trying not to be screwed a second time, I started buying only 2 drives. Grosbill, a french reseller, have them for 164€. These two were labelled from August 2009, with firmware 10D.
As everything was working fine, with slightly better results than the 1.5Tb Seagate drives (without RAID, of course), I bought 6 more from Macway. They used to be specialised in Apple hardware but now have standard hardware with really good price : 157€. These drives were labeled September 2009 with firmware 20N. Hopefully, they also work as expected : they did not break the compatibility in firmware upgrade 
For the moment I havn’t seen any manual firmware upgrade tool to have every drives in the same version. Actualy, this is even better : if you can, always choose drives with different firmware and series for RAID arrays : you will not have all your drives failing in case of a bug related to the production.
Now, let’s see the perfs…
I will not give every results here. Moreover, RAID initialization will probably take one or two days… So I will not wait for the array to be built before using it. I’m sure this will have an impact on the perfs.
Let’s start with the Host : This is a MacPro 2009 with dual CPU Nehalem 2.66GHz and 12Gb of RAM :
RAID card configuration :
The RAID card is a HighPoint 4322 with dual SAS external output. It has 512Mo of cache and an intel processor for the RAID : this not a « software » RAID card !
Main settings, device settings….
Nothing really special here.
What will be tested
When enabled, the data are not directly written to the disk. Instead they are kept in the 512Mo card cache. The good point is that it gives, as you will soon see, a really good write throughoutput. The bad side is, in case of power failure, data are lost… One solution is to get a APC which will handle in case of power failure. You can use a smaller APC to power the disk rack only but you will have to buy the battery pack for the RAID card. I think the battery pack is really (too much) expensive : Highpoint, stop making money on options !!!
The other option is Write Through.
The card can handle RAID 0 (stripe), 1 (mirror), 5, 6, JBOD (Just a Bunch O Disk : no RAID) and any kind of mix, with spare disks. We’ll test RAID 1 and RAID 5.
The card can handle 8 drives. If I have time, I’ll try to bench with 4 drives and 8 drives.
By default the block size is 64k. It can grow up to 1Gb. This is the the RAID card will use to write data. Some people are saying 128k should be fine with big drives and big files… We’ll test that
Test 1 : RAID 5, 8 drives, Write Back, 64k block size
This is a full default when creating an array.
Aja System Test :
BlackMagic Disk Speed Test :
HD Drive Speed Test :
First thing : every speed test software have it’s own way to report. What I can say is that the write speed is MUCH faster than the read speed. This is certainly due to the write cache. I’m sure this value will drop when the write-through option is selected.
Whatever, the RAID array can sustain around 7 streams of 1080p DVCpro. I’m shooting with a Sony EX1, around 35Mbps VBR : I’ll be good
Test 2 : RAID 5, 8 drives, Write Back, 128k block size
Here we see a little improvement. Video files are usualy big, and a bigger block size seem to be helpful.
When taking off Journalizing, we remove one layer on the OS side. This should give some improvements in throughoutput, but it has a price : if the host crash (or the array go offline) while working, we may lose some data. Concidering that 1) I don’t have any battery on the RAID card and 2) I don’t have an APC on the array, the lease we can do is have a journalized FS.
So, this test is only for benchmark resons :
BlackMagic does not show any improvement or loss.
Conclusion : Disabling journalization does not give a real improvement in throughoutput and make the data even more insecure. I’ll stick with a Jounalized FS.
Another conclusion is that using a 1Gb file for testing is not enough. Switch to 4Gb to have a more realistic test, where buffer size can be filled and used for a longer time.
Test 3 : RAID 5, 8 drives, Write Through, 128k block size
We know write through will give less throughoutput on writing. Let’s see how much.
Another thing to enlight : sometimes the Disk Utility seems to have problem creating a 12Tb FS, or mounting it. You have to wait for a timeout and start it over… I actualy have no clue on the why…
Conclusion : reading is REALLY efficient here. On the other hand, writing is now SLOW. I really don’t understand the results. Does « Write Back » means using the cache for writing and « Write Through » using the cache for reading ? Why not both ? I’ll have to ask Highpoint on that. Another solution is that the read cache would be less effitient than with no cache ?
HD Drive speed test, which is only doing a write test, shows us the same result for writing :
Aja test gives no improvement in reading. This test is seriously puzzling me… LEt me know if you have any clue on why the Black Magic test is giving so many reads/sec.
This is co-related with the « iostat » command I’m running alog with the tests :
During the Black Magic test :
disk0 disk1 disk2 disk3 disk5 disk6 cpu load average
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m
0.00 0 0.00 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1 1 98 0.68 0.36 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.64 44 5.39 3 1 96 0.70 0.37 0.33
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.40 80 9.79 1 1 98 0.70 0.37 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 126.68 79 9.76 1 1 98 0.70 0.37 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 124.75 96 11.68 1 1 98 0.70 0.37 0.33
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.52 84 10.29 1 1 97 0.70 0.37 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.88 98 12.03 1 1 98 0.73 0.38 0.33
10.51 63 0.65 0.00 0 0.00 304.50 32 9.51 0.00 0 0.00 0.00 0 0.00 121.28 81 9.58 1 2 97 0.73 0.38 0.33
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 118.31 97 11.19 1 1 98 0.73 0.38 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 126.07 108 13.28 1 1 98 0.73 0.38 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.19 74 9.04 1 1 98 0.73 0.38 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.07 71 8.66 1 1 97 0.75 0.39 0.34
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2 1 98 0.75 0.39 0.34
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1 1 98 0.75 0.39 0.34
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1 1 97 0.75 0.39 0.34
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.58 2150 263.62 1 2 97 0.75 0.39 0.34
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.58 12366 1516.48 1 6 93 0.85 0.42 0.35
0.00 0 0.00 0.00 0 0.00 1024.00 8 7.99 0.00 0 0.00 0.00 0 0.00 125.58 12465 1528.69 2 5 93 0.85 0.42 0.35
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.58 12521 1535.48 1 5 94 0.85 0.42 0.35
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.59 12433 1524.88 1 6 93 0.85 0.42 0.35
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.58 12359 1515.71 1 6 93 0.85 0.42 0.35
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.59 12401 1520.94 2 6 92 0.94 0.44 0.36
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.57 12428 1523.99 1 5 93 0.94 0.44 0.36
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.59 12407 1521.71 1 6 93 0.94 0.44 0.36
68.00 1 0.07 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 125.58 12424 1523.73 2 6 93 0.94 0.44 0.36
6.00 2 0.01 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.58 8148 999.17 2 4 94 0.94 0.44 0.36</code>
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m
0.00 0 0.00 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1 1 98 0.68 0.36 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.64 44 5.39 3 1 96 0.70 0.37 0.33
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.40 80 9.79 1 1 98 0.70 0.37 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 126.68 79 9.76 1 1 98 0.70 0.37 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 124.75 96 11.68 1 1 98 0.70 0.37 0.33
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.52 84 10.29 1 1 97 0.70 0.37 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.88 98 12.03 1 1 98 0.73 0.38 0.33
10.51 63 0.65 0.00 0 0.00 304.50 32 9.51 0.00 0 0.00 0.00 0 0.00 121.28 81 9.58 1 2 97 0.73 0.38 0.33
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 118.31 97 11.19 1 1 98 0.73 0.38 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 126.07 108 13.28 1 1 98 0.73 0.38 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.19 74 9.04 1 1 98 0.73 0.38 0.33
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.07 71 8.66 1 1 97 0.75 0.39 0.34
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 2 1 98 0.75 0.39 0.34
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1 1 98 0.75 0.39 0.34
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1 1 97 0.75 0.39 0.34
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.58 2150 263.62 1 2 97 0.75 0.39 0.34
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.58 12366 1516.48 1 6 93 0.85 0.42 0.35
0.00 0 0.00 0.00 0 0.00 1024.00 8 7.99 0.00 0 0.00 0.00 0 0.00 125.58 12465 1528.69 2 5 93 0.85 0.42 0.35
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.58 12521 1535.48 1 5 94 0.85 0.42 0.35
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.59 12433 1524.88 1 6 93 0.85 0.42 0.35
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.58 12359 1515.71 1 6 93 0.85 0.42 0.35
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.59 12401 1520.94 2 6 92 0.94 0.44 0.36
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.57 12428 1523.99 1 5 93 0.94 0.44 0.36
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.59 12407 1521.71 1 6 93 0.94 0.44 0.36
68.00 1 0.07 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 125.58 12424 1523.73 2 6 93 0.94 0.44 0.36
6.00 2 0.01 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.58 8148 999.17 2 4 94 0.94 0.44 0.36</code>
We clearly see a 10MB/s write, then a 1.5MB/s read.
During the Aja test (on a 4GB file) :
[code lang="bash"]
disk0 disk1 disk2 disk3 disk5 disk6 cpu load average
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m
0.00 0 0.00 9.33 15 0.14 0.00 0 0.00 9.26 19 0.17 0.00 0 0.00 126.53 125 15.43 2 2 97 0.79 0.49 0.39
14.59 61 0.87 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 126.70 141 17.43 2 2 96 0.79 0.49 0.39
0.00 0 0.00 0.00 0 0.00 819.20 5 3.99 0.00 0 0.00 0.00 0 0.00 126.60 131 16.17 1 2 97 0.79 0.49 0.39
42.77 228 9.51 4.01 2318 9.07 238.71 28 6.52 4.02 2311 9.07 0.00 0 0.00 126.60 131 16.18 3 4 93 0.79 0.49 0.39
23.69 65 1.50 4.00 3202 12.51 0.00 0 0.00 4.00 3328 13.00 0.00 0 0.00 126.53 125 15.43 4 4 92 0.79 0.49 0.39
7.99 313 2.44 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 126.37 113 13.93 1 8 91 0.81 0.50 0.40
8.21 280 2.24 5.71 28 0.16 0.00 0 0.00 6.89 18 0.12 0.00 0 0.00 122.57 145 17.33 1 3 95 0.81 0.50 0.40
8.21 292 2.34 5.20 10 0.05 0.00 0 0.00 4.00 5 0.02 0.00 0 0.00 126.47 120 14.80 3 2 95 0.81 0.50 0.40
8.05 255 2.00 4.86 14 0.07 0.00 0 0.00 4.75 16 0.07 0.00 0 0.00 126.47 120 14.80 2 2 96 0.81 0.50 0.40
8.78 148 1.27 6.00 2 0.01 1024.00 4 4.00 4.62 13 0.06 0.00 0 0.00 126.44 118 14.55 2 2 96 0.81 0.50 0.40
8.00 101 0.79 4.86 14 0.07 0.00 0 0.00 4.79 61 0.28 0.00 0 0.00 126.47 120 14.80 2 3 96 0.74 0.49 0.39
7.48 31 0.23 4.96 25 0.12 0.00 0 0.00 6.50 24 0.15 0.00 0 0.00 126.54 126 15.55 2 1 97 0.74 0.49 0.39
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 126.53 2871 354.75 1 2 96 0.74 0.49 0.39
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 126.56 4274 528.24 3 3 95 0.74 0.49 0.39
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 126.56 4396 543.32 1 3 96 0.74 0.49 0.39
0.00 0 0.00 0.00 0 0.00 1024.00 4 4.00 0.00 0 0.00 0.00 0 0.00 126.56 4279 528.83 1 2 97 0.68 0.48 0.39
0.00 0 0.00 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 126.57 4164 514.63 3 2 95 0.68 0.48 0.39
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 126.56 4086 505.07 1 2 97 0.68 0.48 0.39
5.79 7 0.04 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 126.56 4089 505.38 2 3 95 0.68 0.48 0.39
11.04 111 1.20 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 126.56 4083 504.63 2 3 95 0.68 0.48 0.39</code>
Same write, but only 512 MBps read…
<h2>Test 4 : RAID 0, 8 drives, Write Back, 128k block size</h2>
This gives a 14.6Tb drive. This is the best you can do, but without ANY data security.
<img src="http://www.lecentre.net/blog/wp-content/uploads/2009/10/test4-aja.jpg" alt="test4-aja" title="test4-aja" width="445" height="534" class="alignnone size-full wp-image-920" />
<img src="http://www.lecentre.net/blog/wp-content/uploads/2009/10/test4-bm.jpg" alt="test4-bm" title="test4-bm" width="668" height="556" class="alignnone size-full wp-image-921" />
<img src="http://www.lecentre.net/blog/wp-content/uploads/2009/10/test4-hds.jpg" alt="test4-hds" title="test4-hds" width="383" height="473" class="alignnone size-full wp-image-922" />
<img src="http://www.lecentre.net/blog/wp-content/uploads/2009/10/test4-aja-swp.jpg" alt="test4-aja-swp" title="test4-aja-swp" width="448" height="536" class="alignnone size-full wp-image-923" />
<h2>Test 5 : RAID 5, 8 drives, Write Back, 128k block size with "dd"</h2>
This is just a quick read/write test, bypassing the FS Cache on the reads. See result of Disk6. This is done using 1MB block size.
Write :
[code lang="bash"]dd if=/dev/zero of=/Volumes/raid/toto bs=1024000 count=20480000
62208+0 records in
62208+0 records out
63700992000 bytes transferred in 91.218295 secs (698335702 bytes/sec)
disk0 disk1 disk2 disk3 disk5 disk6 cpu load average
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m
56.00 1 0.05 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.89 5404 674.87 3 6 91 0.98 0.51 0.45
0.00 0 0.00 0.00 0 0.00 819.20 5 3.99 0.00 0 0.00 0.00 0 0.00 127.86 5502 687.07 2 6 92 0.98 0.51 0.45
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.89 5476 683.85 1 6 92 0.98 0.51 0.45
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.87 5521 689.43 1 6 93 0.98 0.51 0.45
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.88 5381 671.96 1 6 92 0.98 0.51 0.45
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.88 5151 643.29 2 6 92 1.07 0.53 0.46
68.00 1 0.07 0.00 0 0.00 1024.00 4 3.99 0.00 0 0.00 0.00 0 0.00 127.88 5382 672.10 1 6 93 1.07 0.53 0.46
11.29 31 0.34 0.00 0 0.00 328.75 16 5.13 0.00 0 0.00 0.00 0 0.00 127.86 5384 672.27 1 7 92 1.07 0.53 0.46
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.96 3600 442.82 1 4 94 1.07 0.53 0.46
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.89 5394 673.63 1 6 93 1.07 0.53 0.46
56.00 1 0.05 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.88 5334 666.19 2 6 92 1.22 0.57 0.47
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.86 5253 655.89 1 6 92 1.22 0.57 0.47
135.20 5 0.66 0.00 0 0.00 1024.00 4 3.99 0.00 0 0.00 0.00 0 0.00 127.88 5374 671.14 2 6 92 1.22 0.57 0.47
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.89 5405 675.00 1 6 93 1.22 0.57 0.47
4.06 195 0.77 0.00 0 0.00 21.23 13 0.27 0.00 0 0.00 0.00 0 0.00 127.88 5335 666.21 2 7 91 1.22 0.57 0.47
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.86 5446 680.08 2 6 92 1.28 0.60 0.48
28.00 1 0.03 0.00 0 0.00 1024.00 4 4.00 0.00 0 0.00 0.00 0 0.00 127.88 5222 652.19 3 7 90 1.28 0.60 0.48
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.89 5431 678.27 3 7 91 1.28 0.60 0.48
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.88 5383 672.28 2 7 91 1.28 0.60 0.48
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.86 5446 680.02 1 6 93 1.26 0.60 0.48
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m
56.00 1 0.05 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.89 5404 674.87 3 6 91 0.98 0.51 0.45
0.00 0 0.00 0.00 0 0.00 819.20 5 3.99 0.00 0 0.00 0.00 0 0.00 127.86 5502 687.07 2 6 92 0.98 0.51 0.45
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.89 5476 683.85 1 6 92 0.98 0.51 0.45
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.87 5521 689.43 1 6 93 0.98 0.51 0.45
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.88 5381 671.96 1 6 92 0.98 0.51 0.45
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.88 5151 643.29 2 6 92 1.07 0.53 0.46
68.00 1 0.07 0.00 0 0.00 1024.00 4 3.99 0.00 0 0.00 0.00 0 0.00 127.88 5382 672.10 1 6 93 1.07 0.53 0.46
11.29 31 0.34 0.00 0 0.00 328.75 16 5.13 0.00 0 0.00 0.00 0 0.00 127.86 5384 672.27 1 7 92 1.07 0.53 0.46
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 125.96 3600 442.82 1 4 94 1.07 0.53 0.46
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.89 5394 673.63 1 6 93 1.07 0.53 0.46
56.00 1 0.05 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.88 5334 666.19 2 6 92 1.22 0.57 0.47
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.86 5253 655.89 1 6 92 1.22 0.57 0.47
135.20 5 0.66 0.00 0 0.00 1024.00 4 3.99 0.00 0 0.00 0.00 0 0.00 127.88 5374 671.14 2 6 92 1.22 0.57 0.47
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.89 5405 675.00 1 6 93 1.22 0.57 0.47
4.06 195 0.77 0.00 0 0.00 21.23 13 0.27 0.00 0 0.00 0.00 0 0.00 127.88 5335 666.21 2 7 91 1.22 0.57 0.47
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.86 5446 680.08 2 6 92 1.28 0.60 0.48
28.00 1 0.03 0.00 0 0.00 1024.00 4 4.00 0.00 0 0.00 0.00 0 0.00 127.88 5222 652.19 3 7 90 1.28 0.60 0.48
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.89 5431 678.27 3 7 91 1.28 0.60 0.48
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.88 5383 672.28 2 7 91 1.28 0.60 0.48
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 127.86 5446 680.02 1 6 93 1.26 0.60 0.48
Read :
dd if=/dev/rdisk6s2 of=/dev/null bs=1024000 count=20480000
23379+0 records in
23379+0 records out
23940096000 bytes transferred in 49.482015 secs (483814090 bytes/sec)
23379+0 records in
23379+0 records out
23940096000 bytes transferred in 49.482015 secs (483814090 bytes/sec)
disk0 disk1 disk2 disk3 disk5 disk6 cpu load average
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 460 449.69 1 2 98 0.87 0.68 0.52
0.00 0 0.00 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 1000.00 475 463.39 1 2 97 0.87 0.68 0.52
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 475 464.32 1 2 98 0.80 0.67 0.52
10.83 6 0.06 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 477 466.28 0 2 98 0.80 0.67 0.52
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 473 462.40 1 2 97 0.80 0.67 0.52
0.00 0 0.00 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 1000.00 472 461.42 2 2 96 0.80 0.67 0.52
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 468 456.55 1 2 97 0.80 0.67 0.52
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 460 449.69 1 2 97 0.90 0.69 0.53
0.00 0 0.00 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 1000.00 475 464.31 1 2 97 0.90 0.69 0.53
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 479 468.25 1 2 97 0.90 0.69 0.53
60.00 1 0.06 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 475 464.32 5 2 93 0.90 0.69 0.53
770.50 16 12.03 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 1000.00 465 454.57 6 3 92 0.90 0.69 0.53
857.66 29 24.26 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 480 469.20 1 2 97 0.83 0.68 0.52
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 471 460.44 1 2 97 0.83 0.68 0.52
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 476 465.33 0 2 98 0.83 0.68 0.52
68.00 1 0.07 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 1000.00 477 465.33 0 2 98 0.83 0.68 0.52
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 445 434.11 1 2 98 0.83 0.68 0.52
10.98 45 0.48 0.00 0 0.00 330.75 16 5.16 0.00 0 0.00 0.00 0 0.00 1000.00 454 442.89 0 3 97 0.76 0.67 0.52
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 459 448.71 1 2 98 0.76 0.67 0.52
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 462 450.69 1 2 97 0.76 0.67 0.52
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us sy id 1m 5m 15m
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 460 449.69 1 2 98 0.87 0.68 0.52
0.00 0 0.00 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 1000.00 475 463.39 1 2 97 0.87 0.68 0.52
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 475 464.32 1 2 98 0.80 0.67 0.52
10.83 6 0.06 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 477 466.28 0 2 98 0.80 0.67 0.52
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 473 462.40 1 2 97 0.80 0.67 0.52
0.00 0 0.00 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 1000.00 472 461.42 2 2 96 0.80 0.67 0.52
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 468 456.55 1 2 97 0.80 0.67 0.52
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 460 449.69 1 2 97 0.90 0.69 0.53
0.00 0 0.00 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 1000.00 475 464.31 1 2 97 0.90 0.69 0.53
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 479 468.25 1 2 97 0.90 0.69 0.53
60.00 1 0.06 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 475 464.32 5 2 93 0.90 0.69 0.53
770.50 16 12.03 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 1000.00 465 454.57 6 3 92 0.90 0.69 0.53
857.66 29 24.26 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 480 469.20 1 2 97 0.83 0.68 0.52
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 471 460.44 1 2 97 0.83 0.68 0.52
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 476 465.33 0 2 98 0.83 0.68 0.52
68.00 1 0.07 0.00 0 0.00 819.20 5 4.00 0.00 0 0.00 0.00 0 0.00 1000.00 477 465.33 0 2 98 0.83 0.68 0.52
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 445 434.11 1 2 98 0.83 0.68 0.52
10.98 45 0.48 0.00 0 0.00 330.75 16 5.16 0.00 0 0.00 0.00 0 0.00 1000.00 454 442.89 0 3 97 0.76 0.67 0.52
0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 459 448.71 1 2 98 0.76 0.67 0.52
68.00 1 0.07 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 0.00 0 0.00 1000.00 462 450.69 1 2 97 0.76 0.67 0.52
With 128k block size :
dd if=/dev/zero of=/Volumes/raid/toto bs=131072 count=20480000
disk6 cpu load average
KB/t tps MB/s us sy id 1m 5m 15m
127.87 4899 611.77 1 6 93 0.53 0.44 0.45
127.88 4963 619.80 1 6 93 0.53 0.44 0.45
127.90 4859 606.91 1 6 93 0.57 0.45 0.45
124.78 4559 555.53 1 6 93 0.57 0.45 0.45
127.87 4915 613.78 1 6 93 0.57 0.45 0.45
127.90 4994 623.79 1 6 93 0.57 0.45 0.45
127.88 4980 621.91 1 6 93 0.57 0.45 0.45
127.87 4946 617.65 3 7 91 0.68 0.47 0.46
127.11 4479 556.03 1 7 93 0.68 0.47 0.46
127.87 4867 607.77 1 6 93 0.68 0.47 0.46
127.87 4886 610.14 2 7 92 0.68 0.47 0.46
127.87 4932 615.91 1 7 92 0.68 0.47 0.46
127.87 4865 607.53 1 6 93 0.79 0.50 0.47
127.90 4949 618.15 1 6 93 0.79 0.50 0.47
127.87 4930 615.67 1 7 93 0.79 0.50 0.47
127.88 4991 623.29 1 6 92 0.79 0.50 0.47
127.88 4979 621.81 1 7 91 0.79 0.50 0.47
127.88 4978 621.65 1 6 93 0.88 0.52 0.48
127.87 4899 611.78 1 7 92 0.88 0.52 0.48
127.90 4962 619.79 1 5 94 0.88 0.52 0.48
KB/t tps MB/s us sy id 1m 5m 15m
127.87 4899 611.77 1 6 93 0.53 0.44 0.45
127.88 4963 619.80 1 6 93 0.53 0.44 0.45
127.90 4859 606.91 1 6 93 0.57 0.45 0.45
124.78 4559 555.53 1 6 93 0.57 0.45 0.45
127.87 4915 613.78 1 6 93 0.57 0.45 0.45
127.90 4994 623.79 1 6 93 0.57 0.45 0.45
127.88 4980 621.91 1 6 93 0.57 0.45 0.45
127.87 4946 617.65 3 7 91 0.68 0.47 0.46
127.11 4479 556.03 1 7 93 0.68 0.47 0.46
127.87 4867 607.77 1 6 93 0.68 0.47 0.46
127.87 4886 610.14 2 7 92 0.68 0.47 0.46
127.87 4932 615.91 1 7 92 0.68 0.47 0.46
127.87 4865 607.53 1 6 93 0.79 0.50 0.47
127.90 4949 618.15 1 6 93 0.79 0.50 0.47
127.87 4930 615.67 1 7 93 0.79 0.50 0.47
127.88 4991 623.29 1 6 92 0.79 0.50 0.47
127.88 4979 621.81 1 7 91 0.79 0.50 0.47
127.88 4978 621.65 1 6 93 0.88 0.52 0.48
127.87 4899 611.78 1 7 92 0.88 0.52 0.48
127.90 4962 619.79 1 5 94 0.88 0.52 0.48
dd if=/dev/rdisk6s2 of=/dev/null bs=131072 count=20480000
disk6 cpu load average
KB/t tps MB/s us sy id 1m 5m 15m
128.00 3687 460.83 1 2 97 0.38 0.51 0.48
128.00 3683 460.34 1 2 97 0.38 0.51 0.48
128.00 3807 475.84 0 2 98 0.38 0.51 0.48
128.00 3810 476.22 1 2 97 0.38 0.51 0.48
128.00 3720 464.97 1 2 97 0.38 0.51 0.48
128.00 3726 465.72 1 2 97 0.51 0.53 0.49
128.00 3615 451.85 1 2 97 0.51 0.53 0.49
128.00 3610 451.25 1 2 97 0.51 0.53 0.49
128.00 3589 448.60 1 2 97 0.51 0.53 0.49
128.00 3598 449.73 1 2 96 0.51 0.53 0.49
128.00 3549 443.60 0 2 98 0.47 0.52 0.49
128.00 3635 454.34 1 2 97 0.47 0.52 0.49
128.00 3635 454.35 0 2 98 0.47 0.52 0.49
128.00 3653 456.59 1 2 97 0.47 0.52 0.49
128.00 3639 454.85 1 2 97 0.47 0.52 0.49
128.00 3539 442.36 1 2 97 0.51 0.53 0.49
128.00 3673 459.10 1 2 97 0.51 0.53 0.49
128.00 3101 387.62 1 2 97 0.51 0.53 0.49
127.96 3495 436.73 1 2 97 0.51 0.53 0.49
128.04 3547 443.47 2 2 96 0.51 0.53 0.49
KB/t tps MB/s us sy id 1m 5m 15m
128.00 3687 460.83 1 2 97 0.38 0.51 0.48
128.00 3683 460.34 1 2 97 0.38 0.51 0.48
128.00 3807 475.84 0 2 98 0.38 0.51 0.48
128.00 3810 476.22 1 2 97 0.38 0.51 0.48
128.00 3720 464.97 1 2 97 0.38 0.51 0.48
128.00 3726 465.72 1 2 97 0.51 0.53 0.49
128.00 3615 451.85 1 2 97 0.51 0.53 0.49
128.00 3610 451.25 1 2 97 0.51 0.53 0.49
128.00 3589 448.60 1 2 97 0.51 0.53 0.49
128.00 3598 449.73 1 2 96 0.51 0.53 0.49
128.00 3549 443.60 0 2 98 0.47 0.52 0.49
128.00 3635 454.34 1 2 97 0.47 0.52 0.49
128.00 3635 454.35 0 2 98 0.47 0.52 0.49
128.00 3653 456.59 1 2 97 0.47 0.52 0.49
128.00 3639 454.85 1 2 97 0.47 0.52 0.49
128.00 3539 442.36 1 2 97 0.51 0.53 0.49
128.00 3673 459.10 1 2 97 0.51 0.53 0.49
128.00 3101 387.62 1 2 97 0.51 0.53 0.49
127.96 3495 436.73 1 2 97 0.51 0.53 0.49
128.04 3547 443.47 2 2 96 0.51 0.53 0.49
Using 64k block size does not change the results.
What we can see here is that the throughoutput is quite stable. Again, results in raw data are almost the same as with the benchmark tools.
Conclusion of static tests
This gives really good results, far more than I need, whatever the mode used. I will have to perform the tests again with a half-full and 80% full array, just to validate the throughoutput is still stable. Of course, it will be less than with an empty array.
Next test will be with FinalCut, and how it can handle big HD files from the EX1 (XDcam-EX 420 1080p @25fps VBR) and Proress 422HQ files.
For the moment :
1) buy a battery for the RAID card (between 100€ and 200€) and enable Write Back
2) use RAID5 for data security
3) user 128k Block Size on disk
The last comment will be about price :
- RAID card : under 600€
- RAID enclosure : 800€ (have to check, I really don’t remember)
- Drives : 157€x8 = 1256€
Total : 2656€ for an array usable for HD (and 4:4:4). Remember this is a 12Tb array with really good perfs.