I'd be worried about it getting "fast" all of sudden. Test crash consistency. If you test something like file creation you should be able to estimate if it should be that fast. (So it should be some fraction of theoretical IOPS on the drives/backing rbd device...)
If it's too fast then maybe the "sync" isn't working properly... Jan > On 03 Jun 2016, at 16:26, David <dclistsli...@gmail.com> wrote: > > Zheng, thanks for looking into this, it makes sense although strangely I've > set up a new nfs server (different hardware, same OS, Kernel etc.) and I'm > unable to recreate the issue. I'm no longer getting the delay, the nfs export > is still using sync. I'm now comparing the servers to see what's different on > the original server. Apologies if I've wasted your time on this! > > Jan, I did some more testing with Fuse on the original server and I was > seeing the same issue, yes I was testing from the nfs client. As above I > think there was something weird with that original server. Noted on sync vs > async, I plan on sticking with sync. > > On Fri, Jun 3, 2016 at 5:03 AM, Yan, Zheng <uker...@gmail.com > <mailto:uker...@gmail.com>> wrote: > On Mon, May 30, 2016 at 10:29 PM, David <dclistsli...@gmail.com > <mailto:dclistsli...@gmail.com>> wrote: > > Hi All > > > > I'm having an issue with slow writes over NFS (v3) when cephfs is mounted > > with the kernel driver. Writing a single 4K file from the NFS client is > > taking 3 - 4 seconds, however a 4K write (with sync) into the same folder on > > the server is fast as you would expect. When mounted with ceph-fuse, I don't > > get this issue on the NFS client. > > > > Test environment is a small cluster with a single MON and single MDS, all > > running 10.2.1, CephFS metadata is an ssd pool, data is on spinners. The NFS > > server is CentOS 7, I've tested with the current shipped kernel (3.10), > > ELrepo 4.4 and ELrepo 4.6. > > > > More info: > > > > With the kernel driver, I mount the filesystem with "-o name=admin,secret" > > > > I've exported a folder with the following options: > > > > *(rw,root_squash,sync,wdelay,no_subtree_check,fsid=1244,sec=1) > > > > I then mount the folder on a CentOS 6 client with the following options (all > > default): > > > > rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.3.231,mountvers=3,mountport=597,mountproto=udp,local_lock=none > > > > A small 4k write is taking 3 - 4 secs: > > > > # time dd if=/dev/zero of=testfile bs=4k count=1 > > 1+0 records in > > 1+0 records out > > 4096 bytes (4.1 kB) copied, 3.59678 s, 1.1 kB/s > > > > real 0m3.624s > > user 0m0.000s > > sys 0m0.001s > > > > But a sync write on the sever directly into the same folder is fast (this is > > with the kernel driver): > > > > # time dd if=/dev/zero of=testfile2 bs=4k count=1 conv=fdatasync > > 1+0 records in > > 1+0 records out > > 4096 bytes (4.1 kB) copied, 0.0121925 s, 336 kB/s > > > Your nfs export has sync option. 'dd if=/dev/zero of=testfile bs=4k > count=1' on nfs client is equivalent to 'dd if=/dev/zero of=testfile > bs=4k count=1 conv=fsync' on cephfs. The reason that sync metadata > operation takes 3~4 seconds is that the MDS flushes its journal every > 5 seconds. Adding async option to nfs export can avoid this delay. > > > > > real 0m0.015s > > user 0m0.000s > > sys 0m0.002s > > > > If I mount cephfs with Fuse instead of the kernel, the NFS client write is > > fast: > > > > dd if=/dev/zero of=fuse01 bs=4k count=1 > > 1+0 records in > > 1+0 records out > > 4096 bytes (4.1 kB) copied, 0.026078 s, 157 kB/s > > > > In this case, ceph-fuse sends an extra request (getattr request on > directory) to MDS. The request causes MDS to flush its journal. > Whether or not client sends the extra request depends on what > capabilities it has. What capabilities client has, in turn, depend on > how many clients are accessing the directory. In my test, nfs on > ceph-fuse is not always fast. > > Yan, Zheng > > > > Does anyone know what's going on here? > > > > > > > Thanks > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
