At high load levels "async" is a dangerous option. What it means is that when
an NFS client has copied its data to the NFS server (i.e. memory, not disk) it
accepts the acknowledgment and carries on assuming the data have been
committed. The "sync" option means that the acknowledgment is not sent until
the server has received acknowledgment from the disks that the data are safely
committed.
In sort, with "async" you are playing russian roulette with your data if the
server dies unexpectedly or the cache gets full in a nasty way.
In practice neither usually makes much difference. The key thing is how much
data you transfer at once, because the NFS overhead of managing a transaction
is quite large.
In contrast, using "noatime" is probably what everyone wants, and leave the
client and server to negotiate the largest possible rsize and wsize (e.g. 1MB).
So, write 1 byte at a time and performance is sludge, write 1 megabyte and you
should get line speed (e.g. ~120MB/s for 1gig Ethernet). Some old CCP4 programs
(e.g. FFT, I believe) used disk based Unix sorts which approximated to the
first scenario and were absolutely dreadful over NFS. All these things should
be directed at local disks or even ramdisks if possible.
Hope this helps,
Robert
--
Dr. Robert Esnouf,
University Research Lecturer
and Head of Research Computing,
Wellcome Trust Centre for Human Genetics,
Old Road Campus, Roosevelt Drive,
Oxford OX3 7BN, UK
Emails: rob...@strubi.ox.ac.uk Tel: (+44) - 1865 - 287783
and rob...@well.ox.ac.uk Fax: (+44) - 1865 - 287547
---- Original message ----
>Date: Wed, 31 Jul 2013 12:36:59 +0100
>From: CCP4 bulletin board <CCP4BB@JISCMAIL.AC.UK> (on behalf of Kay Diederichs
><kay.diederi...@uni-konstanz.de>)
>Subject: Re: [ccp4bb] Advise on setting up/ maintaining a Ubuntu cluster
>To: CCP4BB@JISCMAIL.AC.UK
>
>I have a very different experience with NFS: we are using Gigabit Ethernet,
>and a 64bit RHEL6 clone with ECC memory as a file server; it has RAID1 ext4
>home directories and RAID6 ext4 for synchrotron data. We have had zero
>performance or reliability problems with this in a computer lab with ~ 10
>workstations, and I have seen 115 MB/sec file transfers via NFS, at peak
>times.
>Just make sure to export using the "async" option.
>
>HTH,
>
>Kay
>
>On Wed, 31 Jul 2013 09:21:48 +0900, Francois Berenger <beren...@riken.jp>
>wrote:
>
>>Be careful that running data intensive jobs over NFS
>>is super slow (at least an order of magnitude compared
>>to writing things on a local disk).
>>Not only the computation is slow, but you may be slowing down
>>all other users of the cluster too...
>>
>>F.
