Re: [ceph-users] Extremely slow small files rewrite performance

2014-10-24 Thread Yan, Zheng
On Fri, Oct 24, 2014 at 8:47 AM, Sergey Nazarov wrote: > Any update? > The short answer is that when the command is executed for second time, the MDS needs to truncate the file zero length. The speed of truncate a file is limited by the OSD speed. (creating file and write data to the file are asy

Re: [ceph-users] Extremely slow small files rewrite performance

2014-10-24 Thread Sergey Nazarov
Any update? On Tue, Oct 21, 2014 at 3:32 PM, Sergey Nazarov wrote: > Ouch, I think client log is missing. > Here it goes: > https://www.dropbox.com/s/650mjim2ldusr66/ceph-client.admin.log.gz?dl=0 > > On Tue, Oct 21, 2014 at 3:22 PM, Sergey Nazarov wrote: >> I enabled logging and performed same t

Re: [ceph-users] Extremely slow small files rewrite performance

2014-10-21 Thread Sergey Nazarov
Ouch, I think client log is missing. Here it goes: https://www.dropbox.com/s/650mjim2ldusr66/ceph-client.admin.log.gz?dl=0 On Tue, Oct 21, 2014 at 3:22 PM, Sergey Nazarov wrote: > I enabled logging and performed same tests. > Here is the link on archive with logs, they are only from one node > (f

Re: [ceph-users] Extremely slow small files rewrite performance

2014-10-21 Thread Sergey Nazarov
I enabled logging and performed same tests. Here is the link on archive with logs, they are only from one node (from the node where active MDS was sitting): https://www.dropbox.com/s/80axovtoofesx5e/logs.tar.gz?dl=0 Rados bench results: # rados bench -p test 10 write Maintaining 16 concurrent wr

Re: [ceph-users] Extremely slow small files rewrite performance

2014-10-21 Thread Gregory Farnum
Can you enable debugging on the client ("debug ms = 1", "debug client = 20") and mds ("debug ms = 1", "debug mds = 20"), run this test again, and post them somewhere for me to look at? While you're at it, can you try rados bench and see what sort of results you get? -Greg Software Engineer #42 @ h

Re: [ceph-users] Extremely slow small files rewrite performance

2014-10-21 Thread Sergey Nazarov
It is CephFS mounted via ceph-fuse. I am getting the same results not depending on how many other clients are having this fs mounted and their activity. Cluster is working on Debian Wheezy, kernel 3.2.0-4-amd64. On Tue, Oct 21, 2014 at 1:44 PM, Gregory Farnum wrote: > Are these tests conducted us

Re: [ceph-users] Extremely slow small files rewrite performance

2014-10-21 Thread Gregory Farnum
Are these tests conducted using a local fs on RBD, or using CephFS? If CephFS, do you have multiple clients mounting the FS, and what are they doing? What client (kernel or ceph-fuse)? -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Tue, Oct 21, 2014 at 9:05 AM, Sergey Nazar

[ceph-users] Extremely slow small files rewrite performance

2014-10-21 Thread Sergey Nazarov
Hi I just built a new cluster using this quickstart instructions: http://ceph.com/docs/master/start/ And here is what I am seeing: # time for i in {1..10}; do echo $i > $i.txt ; done real 0m0.081s user 0m0.000s sys 0m0.004s And if I try to repeat the same command (when files already created):