On 03/08/2013 07:05 PM, Greg Farnum wrote: > On Friday, March 8, 2013 at 2:45 PM, Jim Schutt wrote: >> On 03/07/2013 08:15 AM, Jim Schutt wrote: >>> On 03/06/2013 05:18 PM, Greg Farnum wrote: >>>> On Wednesday, March 6, 2013 at 3:14 PM, Jim Schutt wrote: >>> >> >> >> >> [snip] >> >>>>> Do you want the MDS log at 10 or 20? >>>> More is better. ;) >>> >>> >>> >>> OK, thanks. >> >> I've sent some mds logs via private email... >> >> -- Jim > I'm going to need to probe into this a bit more, but on an initial > examination I see that most of your stats are actually happening very > quickly — it's just that occasionally they take quite a while.
Interesting... > Going > through the MDS log for one of those, the inode in question is > flagged with "needsrecover" from its first appearance in the log — > that really shouldn't happen unless a client had write caps on it and > the client disappeared. Any ideas? The slowness is being caused by > the MDS going out and looking at every object which could be in the > file — there are a lot since the file has a listed size of 8GB. For this run, the MDS logging slowed it down enough to cause the client caps to occasionally go stale. I don't think it's the cause of the issue, because I was having it before I turned MDS debugging up. My client caps never go stale at, e.g., debug mds 5. Otherwise, there were no signs of trouble while writing the files. Can you suggest which kernel client debugging I might enable that would help understand what is happening? Also, I have the full MDS log from writing the files, if that will help. It's big (~10 GiB). > (There are several other mysteries here that can probably be traced > to different varieties of non-optimal and buggy code as well — there > is a client which has write caps on the inode in question despite it > needing recovery, but the recovery isn't triggered until the stat > event occurs, etc). OK, thanks for taking a look. Let me know if there is other logging I can enable that will be helpful. -- Jim > -Greg > > Software Engineer #42 @ http://inktank.com | http://ceph.com > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
