Hi,
My log is filling up with warnings about a single slow request that has
been around for a very long time:
osd.1 10.0.0.2:6800/900 162926 : [WRN] 1 slow requests, 1 included
below; oldest blocked for > 84446.312051 secs
osd.1 10.0.0.2:6800/900 162927 : [WRN] slow request 84446.312
health warnings like
HEALTH_WARN 4 requests are blocked > 32 sec;
2 osds have slow requests 3 ops are blocked > 32.768 sec on
osd.9
1 ops are blocked > 32.768 sec on osd.10
2 osds have slow requests
After a couple of minutes, health goes to
devel@vger.kernel.org
Subject: Slow requests
Hi,
My log is filling up with warnings about a single slow request that has been
around for a very long time:
osd.1 10.0.0.2:6800/900 162926 : [WRN] 1 slow requests, 1 included below;
oldest blocked for > 84446.312051 secs
osd.1 10.0.0.2:6800/900 162927 : [W
On Sun, Dec 16, 2012 at 5:59 PM, Jens Kristian Søgaard
wrote:
> Hi,
>
> My log is filling up with warnings about a single slow request that has been
> around for a very long time:
>
> osd.1 10.0.0.2:6800/900 162926 : [WRN] 1 slow requests, 1 included below;
> oldest block
Hi Andrey,
Thanks for your reply!
Please take a look to this thread:
http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/10843
I took your advice and restarted each of my three osd's individually.
The first two restarted within a minute or two. The last one took 20
minutes to resta
On Mon, Dec 17, 2012 at 2:42 AM, Jens Kristian Søgaard
wrote:
> Hi Andrey,
>
> Thanks for your reply!
>
>
>> Please take a look to this thread:
>> http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/10843
>
>
> I took your advice and restarted each of my three osd's individually. The
> fi
;
> -Chris
>
> -Original Message-
> From: ceph-devel-ow...@vger.kernel.org
> [mailto:ceph-devel-ow...@vger.kernel.org] On Behalf Of Jens Kristian S?gaard
> Sent: Sunday, December 16, 2012 9:00 AM
> To: ceph-devel@vger.kernel.org
> Subject: Slow requests
>
> Hi,
>
> My
Hi Sage,
v0.52 is also quite old. This was resolved in v0.55, I believe. Where
are you looking for the RPMs? v0.55 should be there...
The homepage said to look in:
http://ceph.com/rpms/fc17/x86_64/
The newest there is 0.52-6.fc17.x86_64.
I also looked at the EL6 folder, but the newest
Hi Andrey,
Oh, sorry, I maybe misguided you - solution is the patch from Sam,
restarts may help only on the short distance and you`re not able to
check some pgs for consistency until patch have been applied - they`ll
hang on scrub every time.
I'll need to update, yes!
I don't have any pgs in
Hi Jens,
Seems that the RPM packager likes to keep the latest and greatest
versions in http://ceph.com/rpm-testing/ but this path isn't defined
in the ceph yum repository.
Dino
On Sun, Dec 16, 2012 at 5:11 PM, Jens Kristian Søgaard
wrote:
> Hi Sage,
>
>
>> v0.52 is also quite old. This was res
Hi Dino,
Seems that the RPM packager likes to keep the latest and greatest
versions in http://ceph.com/rpm-testing/ but this path isn't defined
in the ceph yum repository.
Thanks for the link!
Perhaps the documentation should be updated with this URL?
The release notes link to:
http://ceph.
.6b8b4567.07a7 [write 4046848~147456] 2.54bcf797) currently
waiting for sub ops
2013-01-15 15:54:06.990355 7ff97dd0c700 0 log [WRN] : 2 slow requests, 2
included below; oldest blocked for > 33.225773 secs
2013-01-15 15:54:06.990366 7ff97dd0c700 0 log [WRN] : slow request 33.225773
seconds
--no-cleanup,
> the rados bench commands seem to finish OK but I often get health warnings
> like
> HEALTH_WARN 4 requests are blocked > 32 sec;
> 2 osds have slow requests 3 ops are blocked > 32.768 sec on
> osd.9
> 1 ops are blocked > 3
Hello list,
i see this several times. Endless flying slow requests. And they never
stop until i restart the mentioned osd.
2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included
below; oldest blocked for > 31789.858457 secs
2012-11-14 10:11:57.513399 osd.24 [WRN] slow requ
t; >
> >
> > > On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote:
> > >
> > > > Hello list,
> > > >
> > > > i see this several times. Endless flying slow requests. And they never
> > > > stop
> > > > until
Am 28.11.2012 02:51, schrieb Sage Weil:
Hi Stefan,
Yes it is. So i have to specify admin socket at the KVM host?
Right. IIRC the disk line is a ; (or \;) separated list of key/value
pairs.
How do i query the admin socket for requests?
ceph --admin-daemon /path/to/socket help
ceph --admin
me": "2012-11-28 03:54:45.360049",
"event": "write_thread_in_journal_buffer"},
{ "time": "2012-11-28 03:54:45.586183",
"event": "journaled_completion_queued"},
;event": "commit_queued_for_journal_write"},
> { "time": "2012-11-28 03:54:45.360049",
> "event": "write_thread_in_journal_buffer"},
> { "time": "2012-11-28 03:54:45.586183&
;,
>> "event": "started"},
>> { "time": "2012-11-28 03:54:43.386973",
>> "event": "commit_queued_for_journal_write"},
>> { "time": "20
I actually have this exact issue with hung io during these slow requests. The
odd thing is when I restarted the osd that was being mentioned via "ceph -w" it
didn't resume io. A bit later (unsure on exact time) another one started being
listed in "ceph -w" and when I
{ "time": "2012-11-28 03:54:43.094222",
>>> "event": "waiting_for_osdmap"},
>>> { "time": "2012-11-28 03:54:43.386890",
>>> "event": "reached_pg"},
>>>
2.676641",
>>>> "flag_point": "started",
>>>> "events": [
>>>> { "time": "2012-11-28 03:54:43.094222",
>>>> "event": "waiting_for_os
;>> "age": "27812.942680",
> >>>> "duration": "2.676641",
> >>>> "flag_point": "started",
> >>>> "events": [
> >>>>
snapset=0=[]:[] snapc=0=[])",
>> >>>> "received_at": "2012-11-28 03:54:43.094151",
>> >>>> "age": "27812.942680",
>> >>>> "duration": "2.676641",
>> >>>
it librbd? In that case you likely need to change the config so that
it is listening on an admin socket ('admin socket = path').
Thanks!
sage
On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote:
> Hello list,
>
> i see this several times. Endless flying slow requests.
On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote:
Hello list,
i see this several times. Endless flying slow requests. And they never stop
until i restart the mentioned osd.
2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included below;
oldest blocked for > 31789.858457 se
On 11/15/2012 12:09 AM, Stefan Priebe - Profihost AG wrote:
Am 14.11.2012 15:59, schrieb Sage Weil:
Hi Stefan,
I would be nice to confirm that no clients are waiting on replies for
these requests; currently we suspect that the OSD request tracking is the
buggy part. If you query the OSD admin
o/socket objecter_dump (i think)
sage
>
> Stefan
>
>
> > On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote:
> >
> > > Hello list,
> > >
> > > i see this several times. Endless flying slow requests. And they never
> > &
Hi,
What are likely causes for "slow requests" and "monclient: hunting for new
mon" messages? E.g.:
2013-02-12 16:27:07.318943 7f9c0bc16700 0 monclient: hunting for new mon
...
2013-02-12 16:27:45.892314 7f9c13c26700 0 log [WRN] : 6 slow requests, 6
included below
On Tue, Feb 12, 2013 at 06:28:15PM +1100, Chris Dunlop wrote:
> Hi,
>
> What are likely causes for "slow requests" and "monclient: hunting for new
> mon" messages? E.g.:
>
> 2013-02-12 16:27:07.318943 7f9c0bc16700 0 monclient: hunting for new mon
> ...
On 2013-02-12, Chris Dunlop wrote:
> Hi,
>
> What are likely causes for "slow requests" and "monclient: hunting for new
> mon" messages? E.g.:
>
> 2013-02-12 16:27:07.318943 7f9c0bc16700 0 monclient: hunting for new mon
> ...
> 2013-02-12 16:27:
On Fri, 15 Feb 2013, Chris Dunlop wrote:
> On 2013-02-12, Chris Dunlop wrote:
> > Hi,
> >
> > What are likely causes for "slow requests" and "monclient: hunting for new
> > mon" messages? E.g.:
> >
> > 2013-02-12 16:27:07.318943 7f9c0bc
32 matches
Mail list logo
