Hi Greg, Yes, this still happens after the updatedb fix. [root@xxx dan]# mount ... zzz:6789:/ on /mnt/ceph type ceph (name=cephfs,key=client.cephfs)
[root@xxx dan]# pwd /mnt/ceph/dan [root@xxx dan]# dd if=/dev/zero of=yyy bs=4M count=2000 2000+0 records in 2000+0 records out 8388608000 bytes (8.4 GB) copied, 9.21217 s, 911 MB/s Then 30s later: 2014-02-21 16:16:11.315110 osd.326 x:6836/31929 683 : [WRN] 1 slow requests, 1 included below; oldest blocked for > 32.432401 secs 2014-02-21 16:16:11.315317 osd.326 x:6836/31929 684 : [WRN] slow request 32.432401 seconds old, received at 2014-02-21 16:15:38.882584: osd_op(client.16735018.1:22522476 100000352bf.000002a4 [write 0~4194304 [8@0],startsync 0~0] 0.5447d769 snapc 1=[] e42655) v4 currently waiting for subops from [357,191] And no slow requests for other active clients. Reminder, this is 1GigE client, 64GB RAM, kernel 3.13.0-1.el6.elrepo.x86_64, kernel mounted cephfs. I can't reproduce this on a 1GigE client with only 8GB ram, 3.11.0-15-generic and 3.13.4-031304-generic. (The smaller RAM client is writing at 110-120MB/s vs the 900MB/s writes seen on the big RAM machine -- obviously the writes are all buffered on the big ram machine). Maybe the RAM isn't related, though, as with fdatasync mode we still see the slow requests: [root@xxx dan]# dd if=/dev/zero of=yyy bs=4M count=2000 conv=fdatasync 2000+0 records in 2000+0 records out 8388608000 bytes (8.4 GB) copied, 78.26 s, 107 MB/s 2014-02-21 16:26:15.202047 osd.818 x:6803/128164 1219 : [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.446683 secs 2014-02-21 16:26:15.202194 osd.818 x:6803/128164 1220 : [WRN] slow request 30.446683 seconds old, received at 2014-02-21 16:25:44.754914: osd_op(client.16735018.1:22524842 100000352bf.00000355 [write 0~4194304 [12@0],startsync 0~0] 0.c36d4557 snapc 1=[] e42655) v4 currently waiting for subops from [558,827] Cheers, Dan -- Dan van der Ster || Data & Storage Services || CERN IT Department -- On Thu, Feb 20, 2014 at 4:02 PM, Gregory Farnum <g...@inktank.com> wrote: > Arne, > Sorry this got dropped -- I had it marked in my mail but didn't have > the chance to think about it seriously when you sent it. Does this > still happen after the updatedb config change you guys made recently? > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com > > > On Fri, Jan 31, 2014 at 5:52 AM, Arne Wiebalck <arne.wieba...@cern.ch> > wrote: > > Hi, > > > > We observe that we can easily create slow requests with a simple dd on > > CephFS: > > > > --> > > [root@p05153026953834 dd]# dd if=/dev/zero of=xxx bs=4M count=1000 > > 1000+0 records in > > 1000+0 records out > > 4194304000 bytes (4.2 GB) copied, 4.27824 s, 980 MB/s > > > > ceph -w: > > 2014-01-31 14:28:44.009543 osd.450 [WRN] 1 slow requests, 1 included > below; > > oldest blocked for > 31.088950 secs > > 2014-01-31 14:28:44.009676 osd.450 [WRN] slow request 31.088950 seconds > old, > > received at 2014-01-31 14:28:12.920423: osd_op(client.16735018.1:22493091 > > 100000352b3.000002e9 [write 0~4194304,startsync 0~0] 0.518f2eef snapc > 1=[] > > e32400) v4 currently waiting for subops from [87,1190] > > <--- > > > > From what we see, the OSDs are not busy, so we suspect that it is the > client > > starting all requests, > > but then the requests take longer than 30 secs to finish writing, i.e. > > flushing the client-side buffers. > > > > Is our understanding correct? > > Do these slow requests have an impact on requests from other clients, > i.e. > > some OSD resources > > consumed by these clients? > > > > The setup is: > > Client: kernel 3.13.0, 1GbE > > MDS Emperor 0.72.2 > > OSDs Dumpling 0.67.5 > > > > Thanks! > > Dan & Arne > > > > > > -- > > Arne Wiebalck > > CERN IT > > > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
