Re: endless flying slow requests

2012-11-27 Thread Sage Weil
Hi Stefan, On Thu, 15 Nov 2012, Sage Weil wrote: > On Thu, 15 Nov 2012, Stefan Priebe - Profihost AG wrote: > > Am 14.11.2012 15:59, schrieb Sage Weil: > > > Hi Stefan, > > > > > > I would be nice to confirm that no clients are waiting on replies for > > > these requests; currently we suspect tha

Re: endless flying slow requests

2012-11-27 Thread Stefan Priebe - Profihost AG
Am 28.11.2012 02:51, schrieb Sage Weil: Hi Stefan, Yes it is. So i have to specify admin socket at the KVM host? Right. IIRC the disk line is a ; (or \;) separated list of key/value pairs. How do i query the admin socket for requests? ceph --admin-daemon /path/to/socket help ceph --admin

Re: endless flying slow requests

2012-11-27 Thread Andrey Korolyov
On Wed, Nov 28, 2012 at 5:51 AM, Sage Weil wrote: > Hi Stefan, > > On Thu, 15 Nov 2012, Sage Weil wrote: >> On Thu, 15 Nov 2012, Stefan Priebe - Profihost AG wrote: >> > Am 14.11.2012 15:59, schrieb Sage Weil: >> > > Hi Stefan, >> > > >> > > I would be nice to confirm that no clients are waiting o

Re: endless flying slow requests

2012-11-28 Thread Samuel Just
Did you observe hung io along with that error? Both sub_op_commit and sub_op_applied have happened, so the sub_op_reply should have been sent back to the primary. This looks more like a leak. If you also observed hung io, then it's possible that the problem is occurring between the sub_op_applie

Re: endless flying slow requests

2012-11-28 Thread Samuel Just
Also, these clusters aren't mixed argonaut and next, are they? (Not that that shouldn't work, but it would be a useful data point.) -Sam On Wed, Nov 28, 2012 at 1:11 PM, Samuel Just wrote: > Did you observe hung io along with that error? Both sub_op_commit and > sub_op_applied have happened, so

RE: endless flying slow requests

2012-11-28 Thread Moore, Shawn M
5c0b5a7d45d7c)" cluster. Regards, Shawn From: ceph-devel-ow...@vger.kernel.org [ceph-devel-ow...@vger.kernel.org] on behalf of Samuel Just [sam.j...@inktank.com] Sent: Wednesday, November 28, 2012 16:12 To: Andrey Korolyov Cc: Sage Weil; Stefan Priebe - Profihost AG; ceph-devel@vger.kernel.

Re: endless flying slow requests

2012-11-28 Thread Andrey Korolyov
On Thu, Nov 29, 2012 at 1:12 AM, Samuel Just wrote: > Also, these clusters aren't mixed argonaut and next, are they? (Not > that that shouldn't work, but it would be a useful data point.) > -Sam > > On Wed, Nov 28, 2012 at 1:11 PM, Samuel Just wrote: >> Did you observe hung io along with that er

Re: endless flying slow requests

2012-11-30 Thread Samuel Just
I've pushed a fix to next, 49f32cee647c5bd09f36ba7c9fd4f481a697b9d7. Let me know if the problem persists with this patch. -Sam On Wed, Nov 28, 2012 at 2:04 PM, Andrey Korolyov wrote: > On Thu, Nov 29, 2012 at 1:12 AM, Samuel Just wrote: >> Also, these clusters aren't mixed argonaut and next, are

Re: endless flying slow requests

2012-12-01 Thread Sage Weil
I pushed another fix that plugs a leak when requests race with peering that fixed the 'slow request' for osd_sub_op messages for me. It's in wip-osd-leak. Sam, when you get a chance, can you take a look and push it to next if it looks okay? Thanks! sage On Fri, 30 Nov 2012, Samuel Just wrot

Re: endless flying slow requests

2012-12-02 Thread Samuel Just
Looks good, pushed. -Sam On Sat, Dec 1, 2012 at 7:18 PM, Sage Weil wrote: > I pushed another fix that plugs a leak when requests race with peering > that fixed the 'slow request' for osd_sub_op messages for me. It's in > wip-osd-leak. > > Sam, when you get a chance, can you take a look and push

Re: endless flying slow requests

2012-11-14 Thread Sage Weil
Hi Stefan, I would be nice to confirm that no clients are waiting on replies for these requests; currently we suspect that the OSD request tracking is the buggy part. If you query the OSD admin socket you should be able to dump requests and see the client IP, and then query the client. Is i

Re: endless flying slow requests

2012-11-15 Thread Stefan Priebe - Profihost AG
Am 14.11.2012 15:59, schrieb Sage Weil: Hi Stefan, I would be nice to confirm that no clients are waiting on replies for these requests; currently we suspect that the OSD request tracking is the buggy part. If you query the OSD admin socket you should be able to dump requests and see the client

Re: endless flying slow requests

2012-11-15 Thread Josh Durgin
On 11/15/2012 12:09 AM, Stefan Priebe - Profihost AG wrote: Am 14.11.2012 15:59, schrieb Sage Weil: Hi Stefan, I would be nice to confirm that no clients are waiting on replies for these requests; currently we suspect that the OSD request tracking is the buggy part. If you query the OSD admin

Re: endless flying slow requests

2012-11-15 Thread Sage Weil
On Thu, 15 Nov 2012, Stefan Priebe - Profihost AG wrote: > Am 14.11.2012 15:59, schrieb Sage Weil: > > Hi Stefan, > > > > I would be nice to confirm that no clients are waiting on replies for > > these requests; currently we suspect that the OSD request tracking is the > > buggy part. If you quer