Does anyone have any advice for doing I/O with small files ( say 30,000 files making up 290MB or an average of 9.8KB per file, dataset is tarred up on anon ftp at neuron.neurology.uiowa.edu ) over NFS with Yellow Dog Linux. One of our reasons for switching to YDL is filesystem performance. It performs quite a bit faster than OS X Server with HFS+CS filesystem when working with our types of small files (locally).
However, when doing NFS writes it is extremely slow (see benchmarks below). So the NFS + Filesystem interaction (or the low congestion 1 hop gige ethernet network, doubtful) is introducing some serious delays. Is there another RAID config that works better for NFS or is this just a problem with NFS? With NFS I've tried async, various rwsizes, noac, nocts, and even UDP (dangerous) to see if it was my configuration (on the server it's exported async). Perhaps it is yet the configuration that is causing the issue, but I doubt it. ------ LOCAL TO RAID TRANSFERS ------- --Transfer of 30,000 files (290MB) from YDL 4.1 XServe local disk to FC attached RAID 5 ext 2 [EMAIL PROTECTED] /]# rm -rf /mnt/right/ppg/test2/* [EMAIL PROTECTED] /]# date; cp -R /test/extracted_data.dir /mnt/right/ppg/test2/; date; Thu Aug 17 13:12:07 CDT 2006 Thu Aug 17 13:12:57 CDT 2006 [EMAIL PROTECTED] /]# rm -rf /mnt/right/ppg/test2/* [EMAIL PROTECTED] /]# date; cp -R /test /mnt/right/ppg/; date; Thu Aug 17 15:39:06 CDT 2006 Thu Aug 17 15:39:56 CDT 2006 --Transfer of 30,000 files (290MB) from OS X Server XServe local disk to FC attached RAID 5 HFS+CS tcsh% date ; sudo cp -R /test/extracted_data.dir /ppg/test2 ; date ; Thu Aug 17 13:16:44 CDT 2006 Thu Aug 17 13:28:26 CDT 2006 tcsh% sudo rm -rf /ppg/test2/* -- double check for sanity tcsh% date ; sudo cp -R /test/extracted_data.dir /ppg/test2 ; date ; Thu Aug 17 13:35:56 CDT 2006 Thu Aug 17 13:47:33 CDT 2006 So, we're talking about 50 seconds to write 290MB (5.8 MB/sec) under YDL local to raid. Not terrible I guess for RAID 5, or is it? Would RAID 3 be better here? (I imagine so, and am willing to sacrifice read rates, which would still be good) Does NFS play nicer with RAID 3? OK, now for NFS, for this I used a subset dataset so I'm not waiting for 20 minutes on the benchmark. The data set is roughly 7000 files (56MB) and is also on anon ftp at neuron.neurology.uiowa.edu. If you wanna see something interesting, check these out then check out the same transfers using ssh protocol (Chris, I may have told you this wasn't significant, but I must not have tested correctly, please trust these results however as I am being quite careful so as to give myself a very clear picture of what is going on). ------ LOCAL CLIENT TO NFS MOUNTED RAID TRANSFERS (Dell P 360 SUSE 10.1 client)----- --Transfer of 7000 files from SUSE Linux 10.1 local reiser3 to NFS mounted OS X Server HFS+CS RAID5 volume (osx-ppg is an nfs mount with options rwsize=8192, vers3) /# date; cp -r /test/extracted_data.dir /osx-ppg/test2/; date; Thu Aug 17 15:43:14 CDT 2006 Thu Aug 17 15:44:52 CDT 2006 --Transfer of 7000 files from SUSE Linux 10.1 local reiser3 to NFS mounted YDL 4.1 RAID5 ext2 volume. /# date; cp -r /test/extracted_data.dir /ppg/test2/; date; Thu Aug 17 15:53:49 CDT 2006 Thu Aug 17 16:00:01 CDT 2006 WHOA THERE!! Looking at the two above benchmarks using NFS, I can only scratch my head. Anybody got any ideas?? Why would the write to the YDL server take longer than the OS X server, if the same operation done locally is 12 times faster under YDL?? Heres the same thing from a Powermac G5 dual 2.7 Tiger client. ------ MORE LOCAL CLIENT TO NFS MOUNTED RAID TRANSFERS (PMac G5 10.4 Client) ----- --Transfer of 7000 files from OS X 10.4 local disk to NFS mounted OS X Server HFS+CS RAID5 tcsh% date; sudo cp -r /test/ /Volumes/yoda.gige.uiowa.edu/test2; date; Thu Aug 17 16:13:12 CDT 2006 Thu Aug 17 16:15:28 CDT 2006 --Transfer of 7000 files from OS X 10.4 local disk to NFS mounted YDL ext2 RAID5 tcsh% date ; sudo cp -r /test/ /ppg/test2 ; date ; Thu Aug 17 16:37:56 CDT 2006 Thu Aug 17 16:46:46 CDT 2006 Once again, writing to the YDL server takes way too long for some reason even from a Mac client, which removes the client as a cause. I'm using async in exports but....that should make moving small files faster. Here are some rsync transfers of the 7000 file dataset that show where the problem may lie, and that is with NFS' interaction with the ext2 filesystem or the Linux 2.6 kernel in general. I don't claim to know because the second I do the problem seems to change :-) Does anyone know why this is happening? ----- LOCAL CLIENT TO RAID OVER SSH TRANSFERS ------ --rsync transfer of 7000 files from SUSE 10.1 local reiser3 to OS X Server HFS+CS RAID5 /# date ; rsync -a /test/extracted_data.dir [EMAIL PROTECTED]:/Volumes/Left/ppg-backup/test2/ ; date ; Thu Aug 17 16:53:01 CDT 2006 Thu Aug 17 16:54:42 CDT 2006 --rsync transfer of 7000 files from SUSE 10.1 local reiser3 to YDL ext2 RAID 5 /# date ; rsync -a /test/extracted_data.dir [EMAIL PROTECTED]:/mnt/right/ppg/test2/ ; date ; Thu Aug 17 16:57:09 CDT 2006 Thu Aug 17 16:57:14 CDT 2006 I'm concerned about some ratios I'm seeing here, like the 12:1 ratio of OSXS:YDL where YDL is 12 times faster on the local to RAID transfer, and then the 1:4 ratio of OSXS:YDL where YDL is 4 times SLOWER over NFS from the same linux client (tests using a Mac are below and show a similar ratio). But using rsync the results show a whopping 16:1 ratio similar to the local transfers. So, what could cause this (very bad) problem with writing small files over NFS when using YDL 4.1 NFSD? Are there implications with data integrity? Thank you everyone for any comments or clues as to what's happening in the guts, or any suggestions on how to resolve this? What about rsyncd, can it be used to statically "mount" filesystems on a client? I'm reachin here. Not sure about SMB either (can't have ugo permissions with SMB I don't believe) FTP? Not transparantly available like NFS. Thanks again. Nick -- Nick Jones University of Iowa Dept of Neurology Systems Analyst 319-356-0451 _______________________________________________ yellowdog-general mailing list [email protected] http://lists.terrasoftsolutions.com/mailman/listinfo/yellowdog-general HINT: to Google archives, try '<keywords> site:terrasoftsolutions.com'
