Comments in line.
On 7 juil. 09, at 19:36, Dai Ngo wrote:
Without any tuning, the default TCP window size and send buffer size
for NFS
connections is around 48KB which is not very optimal for bulk
transfer. However
the 1.4MB/s write seems to indicate something else is seriously wrong.
My sentiment as well.
iSCSI performance was good, so the network connection seems to be OK
(assuming
it's 1GbE).
Yup - I'm running at wire speed on the iSCSI connections.
What is your mount options look like?
Unfortunately, ESX doesn't give any controls over mount options
I don't know what datastore browser does for copying file, but have
you tried
the vanilla 'cp' command?
The datastore browser copy command is just a wrapper for cp from what
I can gather. All types of copy operations to the NFS volume, even
from other machines top out at this speed. The NFS/iSCSI connections
are in a separate physical network so I can't easily plug anything
into it for testing other mount options from another machine or OS.
I'll try from another VM to see if I can't force a mount with the
async option to see if that helps any.
You can also try NFS performance using tmpfs, instead of ZFS, to
make sure
NIC, protocol stack, NFS are not the culprit.
From what I can observe, it appears that the sync commands issues
over the NFS stack are slowing down the process, even with a
reasonable number of disks in the pool.
What I was hoping for was the same behavior (albeit slightly risky) of
having writes cached to RAM and then dumped out in an optimal manner
to disk, as per the local behavior where you see the flush to disk
operations happening on a regular cycle. I think that this would be
doable with an async mount, but I can't set this on the server side
where it would be used by the servers automatically.
Erik
erik.ableson wrote:
OK - I'm at my wit's end here as I've looked everywhere to find
some means of tuning NFS performance with ESX into returning
something acceptable using osol 2008.11. I've eliminated
everything but the NFS portion of the equation and am looking for
some pointers in the right direction.
Configuration: PE2950 bi pro Xeon, 32Gb RAM with an MD1000 using a
zpool of 7 mirror vdevs. ESX 3.5 and 4.0. Pretty much a vanilla
install across the board, no additional software other than the
Adaptec StorMan to manage the disks.
local performance via dd - 463MB/s write, 1GB/s read (8Gb file)
iSCSI performance - 90MB/s write, 120MB/s read (800Mb file from a VM)
NFS performance - 1.4MB/s write, 20MB/s read (800Mb file from the
Service Console, transfer of a 8Gb file via the datastore browser)
I just found the tool latencytop which points the finger at the ZIL
(tip of the hat to Lejun Zhu). Ref: <http://www.infrageeks.com/zfs/nfsd.png
> & <http://www.infrageeks.com/zfs/fsflush.png>. Log file: <http://www.infrageeks.com/zfs/latencytop.log
>
Now I can understand that there is a performance hit associated
with this feature of ZFS for ensuring data integrity, but this
drastic a difference makes no sense whatsoever. The pool is capable
of handling natively (at worst) 120*7 IOPS and I'm not even seeing
enough to saturate a USB thumb drive. This still doesn't answer why
the read performance is so bad either. According to latencytop,
the culprit would be genunix`cv_timedwait_sig rpcmod`svc
From my searching it appears that there's no async setting for the
osol nfsd, and ESX does not offer any mount controls to force an
async connection. Other than putting in an SSD as a ZIL (which
still strikes me as overkill for basic NFS services) I'm looking
for any information that can bring me up to at least reasonable
throughput.
Would a dedicated 15K SAS drive help the situation by moving the
ZIL traffic off to a dedicated device? Significantly? This is the
sort of thing that I don't want to do without some reasonable
assurance that it will help since you can't remove a ZIL device
from a pool at the moment.
Hints and tips appreciated,
Erik
_______________________________________________
nfs-discuss mailing list
nfs-disc...@opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
