[ceph-users] slow requests after rocksdb delete wal or table_file_deletion

[lin zhou]

hi, cephers
recenty, I am testing ceph 12.2.12 with bluestore using cosbench.
both SATA osd and ssd osd has slow request.
many slow request occur, and most slow logs after rocksdb delete wal
or table_file_deletion logs

does it means the bottleneck of Rocksdb? if so how to improve. if not
how to find out the bottleneck?

Thanks.

ceph 12.2.12
42 nodes, each node 10*8TB sata , 2* 900G sata SSD
each SSD has 5*10G lv as wal and 5*60GB lv as DB for SATA osd
and  one 50GB SSD lv as osd to hold radosgw index\gc\lc pools.


ops/iostat logs
https://gist.github.com/hnuzhoulin/abb732b5df9dd0200247dfee56850293
detail osd logs
:https://drive.google.com/file/d/1moa8Ilgqj-300nVMUVnR9Cc4194Q2ABy/view?usp=sharing

2019-09-25 20:26:16.755001 7f159edc2700  4 rocksdb: (Original Log Time
2019/09/25-20:26:16.754716) EVENT_LOG_v1 {"time_micros":
1569414376754705
, "job": 1017, "event": "flush_finished", "lsm_state": [2, 4, 46, 316,
0, 0, 0], "immutable_memtables": 0}
2019-09-25 20:26:16.755006 7f159edc2700  4 rocksdb: (Original Log Time
2019/09/25-20:26:16.754928)
[/build/ceph-12.2.12/src/rocksdb/db/db_impl_c
ompaction_flush.cc:132] [default] Level summary: base level 1 max
bytes base 268435456 files[2 4 46 316 0 0 0] max score 0.99

2019-09-25 20:26:16.755133 7f159edc2700  4 rocksdb:
[/build/ceph-12.2.12/src/rocksdb/db/db_impl_files.cc:388] [JOB 1017]
Try to delete WAL files
 size 235707203, prev total WAL file size 237105615, number of live WAL files 2.

2019-09-25 20:27:31.576827 7f15bd5ff700  0 log_channel(cluster) log
[WRN] : 6 slow requests, 5 included below; oldest blocked for >
30.589488 secs
2019-09-25 20:27:31.576839 7f15bd5ff700  0 log_channel(cluster) log
[WRN] : slow request 30.589488 seconds old, received at 2019-09-25
20:27:00.987184: osd_op(client.127567.0:8759 31.eb5
31:ad7d21e3:::.dir.9612b61b-b07e-4b93-835e-4596b5b1b39b.127567.11.12:head
[call rgw.guard_bucket_resharding,call rgw.bucket_prepare_op] snapc
0=[] ondisk+write+known_if_redirected e11236) currently waiting for rw
locks
2019-09-25 20:27:31.576849 7f15bd5ff700  0 log_channel(cluster) log
[WRN] : slow request 30.212751 seconds old, received at 2019-09-25
20:27:01.363921: osd_op(client.99543.0:87452593 31.eb5
31:ad7d21e3:::.dir.9612b61b-b07e-4b93-835e-4596b5b1b39b.127567.11.12:head
[call rgw.guard_bucket_resharding,call rgw.bucket_prepare_op] snapc
0=[] ondisk+write+known_if_redirected e11236) currently waiting for rw
locks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests due to scrubbing of very small pg

[Igor Fedotov]

Hi Lukasz,

if this is filestore then most probably my comments are irrelevant. The 
issue I expected is BlueStore specific


Unfortunately I'm not an expert in filestore hence unable to help in 
further investigation. Sorry...



Thanks,

Igor


On 7/9/2019 11:39 AM, Luk wrote:

We have (still) on these OSDs filestore.

Regards
Lukasz


Hi Igor,
ThankYoufor   Your   input,  will  try  Your  suggestion  with
ceph-objectstore-tool.
But for now it looks like main problem is this:
2019-07-09 09:29:25.410839 7f5e4b64f700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f5e20e87700' had timed out after 15
2019-07-09 09:29:25.410842 7f5e4b64f700  1 heartbeat_map is_healthy
'FileStore::op_tp thread 0x7f5e41651700' had timed out after 60
after  this  (a  lot of) logs osd become unresponsive for monitors and
they are marked down for a few seconds/minutes, sometimes it suicide:
2019-07-09 09:29:32.271361 7f5e3d649700  0 log_channel(cluster) log
[WRN] : Monitor daemon marked osd.118 down, but it is still running
2019-07-09 09:29:32.271381 7f5e3d649700  0 log_channel(cluster) log
[DBG] : map e71903 wrongly marked me down at e71902
2019-07-09 09:29:32.271393 7f5e3d649700  1 osd.118 71903 
start_waiting_for_healthy



maybe You (or any other cepher) know how to dill with this problem ?
Regards
Lukasz

Hi Lukasz,
I've seen something like that - slow requests and relevant OSD reboots
on suicide timeout at least twice with two different clusters. The root
cause was slow omap listing for some objects which had started to happen
after massive removals from RocksDB.
To verify if this is the case you can create a script that uses
ceph-objectstore-tool to list objects for the specific pg and then
list-omap for every returned object.
If omap listing for some object(s) takes too long (minutes in my case) -
you're facing the same issue.
PR that implements automatic lookup for such "slow" objects in
ceph-objectstore-tool is under review:
https://github.com/ceph/ceph/pull/27985



The only known workaround for existing OSDs so far is manual DB
compaction. And https://github.com/ceph/ceph/pull/27627 hopefully fixes
the issue for newly deployed OSDs.




Relevant upstream tickets are:
http://tracker.ceph.com/issues/36482
http://tracker.ceph.com/issues/40557



Hope this helps,
Igor
On 7/3/2019 9:54 AM, Luk wrote:

Hello,

I have strange problem with scrubbing.

When  scrubbing starts on PG which belong to default.rgw.buckets.index
pool,  I  can  see that this OSD is very busy (see attachment), and starts 
showing many
slow  request,  after  the  scrubbing  of this PG stops, slow requests
stops immediately.

[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# zgrep scrub 
/var/log/ceph/ceph-osd.118.log.1.gz  | grep -w 20.2
2019-07-03 00:14:57.496308 7fd4c7a09700  0 log_channel(cluster) log [DBG] : 
20.2 deep-scrub starts
2019-07-03 05:36:13.274637 7fd4ca20e700  0 log_channel(cluster) log [DBG] : 
20.2 deep-scrub ok
[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]#

[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# du -sh 20.2_*
636K20.2_head
0   20.2_TEMP
[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# ls -1 -R 20.2_head | wc -l
4125
[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]#

and on mon:

2019-07-03 00:48:44.793893 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231090 : 
cluster [WRN] Health check failed: 105 slow requests are blocked > 32 sec. 
Implicated osds 118 (REQUEST_SLOW)
2019-07-03 00:48:54.086446 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231097 : 
cluster [WRN] Health check update: 102 slow requests are blocked > 32 sec. 
Implicated osds 118 (REQUEST_SLOW)
2019-07-03 00:48:59.088240 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231099 : 
cluster [WRN] Health check update: 91 slow requests are blocked > 32 sec. 
Implicated osds 118 (REQUEST_SLOW)

[...]

2019-07-03 05:36:19.695586 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6243211 : 
cluster [INF] Health check cleared: REQUEST_SLOW (was: 23 slow requests are 
blocked > 32 sec. Implicated osds 118)
2019-07-03 05:36:19.695700 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6243212 : 
cluster [INF] Cluster is now healthy

ceph version 12.2.9

it  might  be related to this (taken from:
https://ceph.com/releases/v12-2-11-luminous-released/) ? :

"
There have been fixes to RGW dynamic and manual resharding, which no longer
leaves behind stale bucket instances to be removed manually. For finding and
cleaning up older instances from a reshard a radosgw-admin command reshard
stale-instances list and reshard stale-instances rm should do the necessary
cleanup.
"


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com







___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests due to scrubbing of very small pg

[Luk]

We have (still) on these OSDs filestore.

Regards
Lukasz

> Hi Igor,

> ThankYoufor   Your   input,  will  try  Your  suggestion  with
> ceph-objectstore-tool.

> But for now it looks like main problem is this:

> 2019-07-09 09:29:25.410839 7f5e4b64f700  1 heartbeat_map is_healthy
> 'OSD::osd_op_tp thread 0x7f5e20e87700' had timed out after 15
> 2019-07-09 09:29:25.410842 7f5e4b64f700  1 heartbeat_map is_healthy
> 'FileStore::op_tp thread 0x7f5e41651700' had timed out after 60

> after  this  (a  lot of) logs osd become unresponsive for monitors and
> they are marked down for a few seconds/minutes, sometimes it suicide:

> 2019-07-09 09:29:32.271361 7f5e3d649700  0 log_channel(cluster) log
> [WRN] : Monitor daemon marked osd.118 down, but it is still running
> 2019-07-09 09:29:32.271381 7f5e3d649700  0 log_channel(cluster) log
> [DBG] : map e71903 wrongly marked me down at e71902
> 2019-07-09 09:29:32.271393 7f5e3d649700  1 osd.118 71903 
> start_waiting_for_healthy


> maybe You (or any other cepher) know how to dill with this problem ?

> Regards
> Lukasz
>> Hi Lukasz,

>> I've seen something like that - slow requests and relevant OSD reboots
>> on suicide timeout at least twice with two different clusters. The root
>> cause was slow omap listing for some objects which had started to happen
>> after massive removals from RocksDB.

>> To verify if this is the case you can create a script that uses 
>> ceph-objectstore-tool to list objects for the specific pg and then 
>> list-omap for every returned object.

>> If omap listing for some object(s) takes too long (minutes in my case) -
>> you're facing the same issue.

>> PR that implements automatic lookup for such "slow" objects in 
>> ceph-objectstore-tool is under review: 
>> https://github.com/ceph/ceph/pull/27985


>> The only known workaround for existing OSDs so far is manual DB 
>> compaction. And https://github.com/ceph/ceph/pull/27627 hopefully fixes
>> the issue for newly deployed OSDs.



>> Relevant upstream tickets are:

>> http://tracker.ceph.com/issues/36482

>> http://tracker.ceph.com/issues/40557


>> Hope this helps,

>> Igor

>> On 7/3/2019 9:54 AM, Luk wrote:
>>> Hello,
>>>
>>> I have strange problem with scrubbing.
>>>
>>> When  scrubbing starts on PG which belong to default.rgw.buckets.index
>>> pool,  I  can  see that this OSD is very busy (see attachment), and starts 
>>> showing many
>>> slow  request,  after  the  scrubbing  of this PG stops, slow requests
>>> stops immediately.
>>>
>>> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# zgrep scrub 
>>> /var/log/ceph/ceph-osd.118.log.1.gz  | grep -w 20.2
>>> 2019-07-03 00:14:57.496308 7fd4c7a09700  0 log_channel(cluster) log [DBG] : 
>>> 20.2 deep-scrub starts
>>> 2019-07-03 05:36:13.274637 7fd4ca20e700  0 log_channel(cluster) log [DBG] : 
>>> 20.2 deep-scrub ok
>>> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]#
>>>
>>> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# du -sh 20.2_*
>>> 636K20.2_head
>>> 0   20.2_TEMP
>>> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# ls -1 -R 20.2_head | wc 
>>> -l
>>> 4125
>>> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]#
>>>
>>> and on mon:
>>>
>>> 2019-07-03 00:48:44.793893 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231090 
>>> : cluster [WRN] Health check failed: 105 slow requests are blocked > 32 
>>> sec. Implicated osds 118 (REQUEST_SLOW)
>>> 2019-07-03 00:48:54.086446 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231097 
>>> : cluster [WRN] Health check update: 102 slow requests are blocked > 32 
>>> sec. Implicated osds 118 (REQUEST_SLOW)
>>> 2019-07-03 00:48:59.088240 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231099 
>>> : cluster [WRN] Health check update: 91 slow requests are blocked > 32 sec. 
>>> Implicated osds 118 (REQUEST_SLOW)
>>>
>>> [...]
>>>
>>> 2019-07-03 05:36:19.695586 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6243211 
>>> : cluster [INF] Health check cleared: REQUEST_SLOW (was: 23 slow requests 
>>> are blocked > 32 sec. Implicated osds 118)
>>> 2019-07-03 05:36:19.695700 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6243212 
>>> : cluster [INF] Cluster is now healthy
>>>
>>> ceph version 12.2.9
>>>
>>> it  might  be related to this (taken from:
>>> https://ceph.com/releases/v12-2-11-luminous-released/) ? :
>>>
>>> "
>>> There have been fixes to RGW dynamic and manual resharding, which no longer
>>> leaves behind stale bucket instances to be removed manually. For finding and
>>> cleaning up older instances from a reshard a radosgw-admin command reshard
>>> stale-instances list and reshard stale-instances rm should do the necessary
>>> cleanup.
>>> "
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com






-- 
Pozdrowienia,
 Luk

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/

Re: [ceph-users] slow requests due to scrubbing of very small pg

[Luk]

Hi Igor,

ThankYoufor   Your   input,  will  try  Your  suggestion  with
ceph-objectstore-tool.

But for now it looks like main problem is this:

2019-07-09 09:29:25.410839 7f5e4b64f700  1 heartbeat_map is_healthy 
'OSD::osd_op_tp thread 0x7f5e20e87700' had timed out after 15
2019-07-09 09:29:25.410842 7f5e4b64f700  1 heartbeat_map is_healthy 
'FileStore::op_tp thread 0x7f5e41651700' had timed out after 60

after  this  (a  lot of) logs osd become unresponsive for monitors and
they are marked down for a few seconds/minutes, sometimes it suicide:

2019-07-09 09:29:32.271361 7f5e3d649700  0 log_channel(cluster) log [WRN] : 
Monitor daemon marked osd.118 down, but it is still running
2019-07-09 09:29:32.271381 7f5e3d649700  0 log_channel(cluster) log [DBG] : map 
e71903 wrongly marked me down at e71902
2019-07-09 09:29:32.271393 7f5e3d649700  1 osd.118 71903 
start_waiting_for_healthy


maybe You (or any other cepher) know how to dill with this problem ?

Regards
Lukasz
> Hi Lukasz,

> I've seen something like that - slow requests and relevant OSD reboots
> on suicide timeout at least twice with two different clusters. The root
> cause was slow omap listing for some objects which had started to happen
> after massive removals from RocksDB.

> To verify if this is the case you can create a script that uses 
> ceph-objectstore-tool to list objects for the specific pg and then 
> list-omap for every returned object.

> If omap listing for some object(s) takes too long (minutes in my case) -
> you're facing the same issue.

> PR that implements automatic lookup for such "slow" objects in 
> ceph-objectstore-tool is under review: 
> https://github.com/ceph/ceph/pull/27985


> The only known workaround for existing OSDs so far is manual DB 
> compaction. And https://github.com/ceph/ceph/pull/27627 hopefully fixes
> the issue for newly deployed OSDs.



> Relevant upstream tickets are:

> http://tracker.ceph.com/issues/36482

> http://tracker.ceph.com/issues/40557


> Hope this helps,

> Igor

> On 7/3/2019 9:54 AM, Luk wrote:
>> Hello,
>>
>> I have strange problem with scrubbing.
>>
>> When  scrubbing starts on PG which belong to default.rgw.buckets.index
>> pool,  I  can  see that this OSD is very busy (see attachment), and starts 
>> showing many
>> slow  request,  after  the  scrubbing  of this PG stops, slow requests
>> stops immediately.
>>
>> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# zgrep scrub 
>> /var/log/ceph/ceph-osd.118.log.1.gz  | grep -w 20.2
>> 2019-07-03 00:14:57.496308 7fd4c7a09700  0 log_channel(cluster) log [DBG] : 
>> 20.2 deep-scrub starts
>> 2019-07-03 05:36:13.274637 7fd4ca20e700  0 log_channel(cluster) log [DBG] : 
>> 20.2 deep-scrub ok
>> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]#
>>
>> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# du -sh 20.2_*
>> 636K20.2_head
>> 0   20.2_TEMP
>> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# ls -1 -R 20.2_head | wc 
>> -l
>> 4125
>> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]#
>>
>> and on mon:
>>
>> 2019-07-03 00:48:44.793893 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231090 
>> : cluster [WRN] Health check failed: 105 slow requests are blocked > 32 sec. 
>> Implicated osds 118 (REQUEST_SLOW)
>> 2019-07-03 00:48:54.086446 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231097 
>> : cluster [WRN] Health check update: 102 slow requests are blocked > 32 sec. 
>> Implicated osds 118 (REQUEST_SLOW)
>> 2019-07-03 00:48:59.088240 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231099 
>> : cluster [WRN] Health check update: 91 slow requests are blocked > 32 sec. 
>> Implicated osds 118 (REQUEST_SLOW)
>>
>> [...]
>>
>> 2019-07-03 05:36:19.695586 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6243211 
>> : cluster [INF] Health check cleared: REQUEST_SLOW (was: 23 slow requests 
>> are blocked > 32 sec. Implicated osds 118)
>> 2019-07-03 05:36:19.695700 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6243212 
>> : cluster [INF] Cluster is now healthy
>>
>> ceph version 12.2.9
>>
>> it  might  be related to this (taken from:
>> https://ceph.com/releases/v12-2-11-luminous-released/) ? :
>>
>> "
>> There have been fixes to RGW dynamic and manual resharding, which no longer
>> leaves behind stale bucket instances to be removed manually. For finding and
>> cleaning up older instances from a reshard a radosgw-admin command reshard
>> stale-instances list and reshard stale-instances rm should do the necessary
>> cleanup.
>> "
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Pozdrowienia,
 Luk

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests due to scrubbing of very small pg

[Igor Fedotov]

Hi Lukasz,

I've seen something like that - slow requests and relevant OSD reboots 
on suicide timeout at least twice with two different clusters. The root 
cause was slow omap listing for some objects which had started to happen 
after massive removals from RocksDB.


To verify if this is the case you can create a script that uses 
ceph-objectstore-tool to list objects for the specific pg and then 
list-omap for every returned object.


If omap listing for some object(s) takes too long (minutes in my case) - 
you're facing the same issue.


PR that implements automatic lookup for such "slow" objects in 
ceph-objectstore-tool is under review: 
https://github.com/ceph/ceph/pull/27985



The only known workaround for existing OSDs so far is manual DB 
compaction. And https://github.com/ceph/ceph/pull/27627 hopefully fixes 
the issue for newly deployed OSDs.




Relevant upstream tickets are:

http://tracker.ceph.com/issues/36482

http://tracker.ceph.com/issues/40557


Hope this helps,

Igor

On 7/3/2019 9:54 AM, Luk wrote:

Hello,

I have strange problem with scrubbing.

When  scrubbing starts on PG which belong to default.rgw.buckets.index
pool,  I  can  see that this OSD is very busy (see attachment), and starts 
showing many
slow  request,  after  the  scrubbing  of this PG stops, slow requests
stops immediately.

[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# zgrep scrub 
/var/log/ceph/ceph-osd.118.log.1.gz  | grep -w 20.2
2019-07-03 00:14:57.496308 7fd4c7a09700  0 log_channel(cluster) log [DBG] : 
20.2 deep-scrub starts
2019-07-03 05:36:13.274637 7fd4ca20e700  0 log_channel(cluster) log [DBG] : 
20.2 deep-scrub ok
[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]#

[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# du -sh 20.2_*
636K20.2_head
0   20.2_TEMP
[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# ls -1 -R 20.2_head | wc -l
4125
[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]#

and on mon:

2019-07-03 00:48:44.793893 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231090 : 
cluster [WRN] Health check failed: 105 slow requests are blocked > 32 sec. 
Implicated osds 118 (REQUEST_SLOW)
2019-07-03 00:48:54.086446 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231097 : 
cluster [WRN] Health check update: 102 slow requests are blocked > 32 sec. 
Implicated osds 118 (REQUEST_SLOW)
2019-07-03 00:48:59.088240 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231099 : 
cluster [WRN] Health check update: 91 slow requests are blocked > 32 sec. 
Implicated osds 118 (REQUEST_SLOW)

[...]

2019-07-03 05:36:19.695586 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6243211 : 
cluster [INF] Health check cleared: REQUEST_SLOW (was: 23 slow requests are 
blocked > 32 sec. Implicated osds 118)
2019-07-03 05:36:19.695700 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6243212 : 
cluster [INF] Cluster is now healthy

ceph version 12.2.9

it  might  be related to this (taken from:
https://ceph.com/releases/v12-2-11-luminous-released/) ? :

"
There have been fixes to RGW dynamic and manual resharding, which no longer
leaves behind stale bucket instances to be removed manually. For finding and
cleaning up older instances from a reshard a radosgw-admin command reshard
stale-instances list and reshard stale-instances rm should do the necessary
cleanup.
"


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests due to scrubbing of very small pg

[Paul Emmerich]

On Wed, Jul 3, 2019 at 4:47 PM Luk  wrote:

>
>
> this pool is that 'big' :
>
> [root@ceph-mon-01 ~]# rados df | grep -e index -e WR
> POOL_NAME  USEDOBJECTS CLONES COPIES
>  MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS  RD  WR_OPS  WR
>
> default.rgw.buckets.index   0B   31763  095289
>   0   00   147679875 3.88TiB  8014040021  0B
>
>
You are running Luminous which doesn't account metadata properly here.
31763 objects in an an rgw index pool is a non-trivial size.

However, scrubbing that shouldn't be a problem for a properly configured
cluster (i.e., no gigantic shards). Have you tried to increase
osd_scrub_sleep?

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90


> [root@ceph-mon-01 ~]# ceph df | grep -e index -e N
> NAME   ID USED%USED MAX AVAIL
>OBJECTS
> default.rgw.buckets.index  20  0B 0   37.0TiB
>  31763
> [root@ceph-mon-01 ~]#
>
>
> I don't think it should have such big problem during scrubing...
>
> --
> Regards
> Lukasz
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests due to scrubbing of very small pg

[Luk]

Hi,

> Den ons 3 juli 2019 kl 09:01 skrev Luk :

> Hello,

>  I have strange problem with scrubbing.

>  When  scrubbing starts on PG which belong to default.rgw.buckets.index
>  pool,  I  can  see that this OSD is very busy (see attachment), and starts 
> showing many
>  slow  request,  after  the  scrubbing  of this PG stops, slow requests
>  stops immediately.

>  [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# du -sh 20.2_*
>  636K    20.2_head
>  0       20.2_TEMP
>  [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# ls -1 -R 20.2_head | wc 
> -l
>  4125
>  [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]#




> I think that pool might contain tons of metadata entries and
> non-obvious data for which it takes a while to process.
> But I see similar things on my rgw pools also during scrubs. :-( 


this pool is that 'big' :

[root@ceph-mon-01 ~]# rados df | grep -e index -e WR
POOL_NAME  USEDOBJECTS CLONES COPIES   MISSING_ON_PRIMARY 
UNFOUND DEGRADED RD_OPS  RD  WR_OPS  WR  
default.rgw.buckets.index   0B   31763  095289  0   
00   147679875 3.88TiB  8014040021  0B 

[root@ceph-mon-01 ~]# ceph df | grep -e index -e N
NAME   ID USED%USED MAX AVAIL 
OBJECTS 
default.rgw.buckets.index  20  0B 0   37.0TiB   
31763 
[root@ceph-mon-01 ~]#


I don't think it should have such big problem during scrubing...

-- 
Regards
Lukasz

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests due to scrubbing of very small pg

[Janne Johansson]

Den ons 3 juli 2019 kl 09:01 skrev Luk :

> Hello,
>
> I have strange problem with scrubbing.
>
> When  scrubbing starts on PG which belong to default.rgw.buckets.index
> pool,  I  can  see that this OSD is very busy (see attachment), and starts
> showing many
> slow  request,  after  the  scrubbing  of this PG stops, slow requests
> stops immediately.
>
> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# du -sh 20.2_*
> 636K20.2_head
> 0   20.2_TEMP
> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# ls -1 -R 20.2_head |
> wc -l
> 4125
> [root@stor-b02 /var/lib/ceph/osd/ceph-118/current]#
>

I think that pool might contain tons of metadata entries and non-obvious
data for which it takes a while to process.
But I see similar things on my rgw pools also during scrubs. :-(

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] slow requests due to scrubbing of very small pg

[Luk]

Hello,

I have strange problem with scrubbing.

When  scrubbing starts on PG which belong to default.rgw.buckets.index
pool,  I  can  see that this OSD is very busy (see attachment), and starts 
showing many
slow  request,  after  the  scrubbing  of this PG stops, slow requests
stops immediately.

[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# zgrep scrub 
/var/log/ceph/ceph-osd.118.log.1.gz  | grep -w 20.2
2019-07-03 00:14:57.496308 7fd4c7a09700  0 log_channel(cluster) log [DBG] : 
20.2 deep-scrub starts
2019-07-03 05:36:13.274637 7fd4ca20e700  0 log_channel(cluster) log [DBG] : 
20.2 deep-scrub ok
[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]#

[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# du -sh 20.2_*
636K20.2_head
0   20.2_TEMP
[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]# ls -1 -R 20.2_head | wc -l
4125
[root@stor-b02 /var/lib/ceph/osd/ceph-118/current]#

and on mon:

2019-07-03 00:48:44.793893 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231090 : 
cluster [WRN] Health check failed: 105 slow requests are blocked > 32 sec. 
Implicated osds 118 (REQUEST_SLOW)
2019-07-03 00:48:54.086446 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231097 : 
cluster [WRN] Health check update: 102 slow requests are blocked > 32 sec. 
Implicated osds 118 (REQUEST_SLOW)
2019-07-03 00:48:59.088240 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6231099 : 
cluster [WRN] Health check update: 91 slow requests are blocked > 32 sec. 
Implicated osds 118 (REQUEST_SLOW)

[...]

2019-07-03 05:36:19.695586 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6243211 : 
cluster [INF] Health check cleared: REQUEST_SLOW (was: 23 slow requests are 
blocked > 32 sec. Implicated osds 118)
2019-07-03 05:36:19.695700 mon.ceph-mon-01 mon.0 10.10.8.221:6789/0 6243212 : 
cluster [INF] Cluster is now healthy

ceph version 12.2.9

it  might  be related to this (taken from:
https://ceph.com/releases/v12-2-11-luminous-released/) ? :

"
There have been fixes to RGW dynamic and manual resharding, which no longer
leaves behind stale bucket instances to be removed manually. For finding and
cleaning up older instances from a reshard a radosgw-admin command reshard
stale-instances list and reshard stale-instances rm should do the necessary
cleanup.
"

-- 
Regads
 Lukasz___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests are blocked > 32 sec. Implicated osds 0, 2, 3, 4, 5 (REQUEST_SLOW)

[BASSAGET Cédric]

Hello Robert,
I did not make any changes, so I'm still using the prio queue.
Regards

Le lun. 10 juin 2019 à 17:44, Robert LeBlanc  a
écrit :

> I'm glad it's working, to be clear did you use wpq, or is it still the
> prio queue?
>
> Sent from a mobile device, please excuse any typos.
>
> On Mon, Jun 10, 2019, 4:45 AM BASSAGET Cédric <
> cedric.bassaget...@gmail.com> wrote:
>
>> an update from 12.2.9 to 12.2.12 seems to have fixed the problem !
>>
>> Le lun. 10 juin 2019 à 12:25, BASSAGET Cédric <
>> cedric.bassaget...@gmail.com> a écrit :
>>
>>> Hi Robert,
>>> Before doing anything on my prod env, I generate r/w on ceph cluster
>>> using fio .
>>> On my newest cluster, release 12.2.12, I did not manage to get
>>> the (REQUEST_SLOW) warning, even if my OSD disk usage goes above 95% (fio
>>> ran from 4 diffrent hosts)
>>>
>>> On my prod cluster, release 12.2.9, as soon as I run fio on a single
>>> host, I see a lot of REQUEST_SLOW warninr gmessages, but "iostat -xd 1"
>>> does not show me a usage more that 5-10% on disks...
>>>
>>> Le lun. 10 juin 2019 à 10:12, Robert LeBlanc  a
>>> écrit :
>>>
 On Mon, Jun 10, 2019 at 1:00 AM BASSAGET Cédric <
 cedric.bassaget...@gmail.com> wrote:

> Hello Robert,
> My disks did not reach 100% on the last warning, they climb to 70-80%
> usage. But I see rrqm / wrqm counters increasing...
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>
> sda   0.00 4.000.00   16.00 0.00   104.00
>  13.00 0.000.000.000.00   0.00   0.00
> sdb   0.00 2.001.00 3456.00 8.00 25996.00
>  15.04 5.761.670.001.67   0.03   9.20
> sdd   4.00 0.00 41462.00 1119.00 331272.00  7996.00
>  15.9419.890.470.480.21   0.02  66.00
>
> dm-0  0.00 0.00 6825.00  503.00 330856.00  7996.00
>  92.48 4.000.550.560.30   0.09  66.80
> dm-1  0.00 0.001.00 1129.00 8.00 25996.00
>  46.02 1.030.910.000.91   0.09  10.00
>
>
> sda is my system disk (SAMSUNG   MZILS480HEGR/007  GXL0), sdb and sdd
> are my OSDs
>
> would "osd op queue = wpq" help in this case ?
> Regards
>

 Your disk times look okay, just a lot more unbalanced than I would
 expect. I'd give wpq a try, I use it all the time, just be sure to also
 include the op_cutoff setting too or it doesn't have much effect. Let me
 know how it goes.
 
 Robert LeBlanc
 PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1

>>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests are blocked > 32 sec. Implicated osds 0, 2, 3, 4, 5 (REQUEST_SLOW)

[Robert LeBlanc]

I'm glad it's working, to be clear did you use wpq, or is it still the prio
queue?

Sent from a mobile device, please excuse any typos.

On Mon, Jun 10, 2019, 4:45 AM BASSAGET Cédric 
wrote:

> an update from 12.2.9 to 12.2.12 seems to have fixed the problem !
>
> Le lun. 10 juin 2019 à 12:25, BASSAGET Cédric <
> cedric.bassaget...@gmail.com> a écrit :
>
>> Hi Robert,
>> Before doing anything on my prod env, I generate r/w on ceph cluster
>> using fio .
>> On my newest cluster, release 12.2.12, I did not manage to get
>> the (REQUEST_SLOW) warning, even if my OSD disk usage goes above 95% (fio
>> ran from 4 diffrent hosts)
>>
>> On my prod cluster, release 12.2.9, as soon as I run fio on a single
>> host, I see a lot of REQUEST_SLOW warninr gmessages, but "iostat -xd 1"
>> does not show me a usage more that 5-10% on disks...
>>
>> Le lun. 10 juin 2019 à 10:12, Robert LeBlanc  a
>> écrit :
>>
>>> On Mon, Jun 10, 2019 at 1:00 AM BASSAGET Cédric <
>>> cedric.bassaget...@gmail.com> wrote:
>>>
 Hello Robert,
 My disks did not reach 100% on the last warning, they climb to 70-80%
 usage. But I see rrqm / wrqm counters increasing...

 Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
 avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

 sda   0.00 4.000.00   16.00 0.00   104.00
  13.00 0.000.000.000.00   0.00   0.00
 sdb   0.00 2.001.00 3456.00 8.00 25996.00
  15.04 5.761.670.001.67   0.03   9.20
 sdd   4.00 0.00 41462.00 1119.00 331272.00  7996.00
  15.9419.890.470.480.21   0.02  66.00

 dm-0  0.00 0.00 6825.00  503.00 330856.00  7996.00
  92.48 4.000.550.560.30   0.09  66.80
 dm-1  0.00 0.001.00 1129.00 8.00 25996.00
  46.02 1.030.910.000.91   0.09  10.00


 sda is my system disk (SAMSUNG   MZILS480HEGR/007  GXL0), sdb and sdd
 are my OSDs

 would "osd op queue = wpq" help in this case ?
 Regards

>>>
>>> Your disk times look okay, just a lot more unbalanced than I would
>>> expect. I'd give wpq a try, I use it all the time, just be sure to also
>>> include the op_cutoff setting too or it doesn't have much effect. Let me
>>> know how it goes.
>>> 
>>> Robert LeBlanc
>>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>>>
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests are blocked > 32 sec. Implicated osds 0, 2, 3, 4, 5 (REQUEST_SLOW)

[BASSAGET Cédric]

an update from 12.2.9 to 12.2.12 seems to have fixed the problem !

Le lun. 10 juin 2019 à 12:25, BASSAGET Cédric 
a écrit :

> Hi Robert,
> Before doing anything on my prod env, I generate r/w on ceph cluster using
> fio .
> On my newest cluster, release 12.2.12, I did not manage to get
> the (REQUEST_SLOW) warning, even if my OSD disk usage goes above 95% (fio
> ran from 4 diffrent hosts)
>
> On my prod cluster, release 12.2.9, as soon as I run fio on a single host,
> I see a lot of REQUEST_SLOW warninr gmessages, but "iostat -xd 1" does not
> show me a usage more that 5-10% on disks...
>
> Le lun. 10 juin 2019 à 10:12, Robert LeBlanc  a
> écrit :
>
>> On Mon, Jun 10, 2019 at 1:00 AM BASSAGET Cédric <
>> cedric.bassaget...@gmail.com> wrote:
>>
>>> Hello Robert,
>>> My disks did not reach 100% on the last warning, they climb to 70-80%
>>> usage. But I see rrqm / wrqm counters increasing...
>>>
>>> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
>>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>>>
>>> sda   0.00 4.000.00   16.00 0.00   104.00
>>>  13.00 0.000.000.000.00   0.00   0.00
>>> sdb   0.00 2.001.00 3456.00 8.00 25996.00
>>>  15.04 5.761.670.001.67   0.03   9.20
>>> sdd   4.00 0.00 41462.00 1119.00 331272.00  7996.00
>>>  15.9419.890.470.480.21   0.02  66.00
>>>
>>> dm-0  0.00 0.00 6825.00  503.00 330856.00  7996.00
>>>  92.48 4.000.550.560.30   0.09  66.80
>>> dm-1  0.00 0.001.00 1129.00 8.00 25996.00
>>>  46.02 1.030.910.000.91   0.09  10.00
>>>
>>>
>>> sda is my system disk (SAMSUNG   MZILS480HEGR/007  GXL0), sdb and sdd
>>> are my OSDs
>>>
>>> would "osd op queue = wpq" help in this case ?
>>> Regards
>>>
>>
>> Your disk times look okay, just a lot more unbalanced than I would
>> expect. I'd give wpq a try, I use it all the time, just be sure to also
>> include the op_cutoff setting too or it doesn't have much effect. Let me
>> know how it goes.
>> 
>> Robert LeBlanc
>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests are blocked > 32 sec. Implicated osds 0, 2, 3, 4, 5 (REQUEST_SLOW)

[BASSAGET Cédric]

Hi Robert,
Before doing anything on my prod env, I generate r/w on ceph cluster using
fio .
On my newest cluster, release 12.2.12, I did not manage to get
the (REQUEST_SLOW) warning, even if my OSD disk usage goes above 95% (fio
ran from 4 diffrent hosts)

On my prod cluster, release 12.2.9, as soon as I run fio on a single host,
I see a lot of REQUEST_SLOW warninr gmessages, but "iostat -xd 1" does not
show me a usage more that 5-10% on disks...

Le lun. 10 juin 2019 à 10:12, Robert LeBlanc  a
écrit :

> On Mon, Jun 10, 2019 at 1:00 AM BASSAGET Cédric <
> cedric.bassaget...@gmail.com> wrote:
>
>> Hello Robert,
>> My disks did not reach 100% on the last warning, they climb to 70-80%
>> usage. But I see rrqm / wrqm counters increasing...
>>
>> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>>
>> sda   0.00 4.000.00   16.00 0.00   104.00
>>  13.00 0.000.000.000.00   0.00   0.00
>> sdb   0.00 2.001.00 3456.00 8.00 25996.00
>>  15.04 5.761.670.001.67   0.03   9.20
>> sdd   4.00 0.00 41462.00 1119.00 331272.00  7996.00
>>  15.9419.890.470.480.21   0.02  66.00
>>
>> dm-0  0.00 0.00 6825.00  503.00 330856.00  7996.00
>>  92.48 4.000.550.560.30   0.09  66.80
>> dm-1  0.00 0.001.00 1129.00 8.00 25996.00
>>  46.02 1.030.910.000.91   0.09  10.00
>>
>>
>> sda is my system disk (SAMSUNG   MZILS480HEGR/007  GXL0), sdb and sdd are
>> my OSDs
>>
>> would "osd op queue = wpq" help in this case ?
>> Regards
>>
>
> Your disk times look okay, just a lot more unbalanced than I would expect.
> I'd give wpq a try, I use it all the time, just be sure to also include the
> op_cutoff setting too or it doesn't have much effect. Let me know how it
> goes.
> 
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests are blocked > 32 sec. Implicated osds 0, 2, 3, 4, 5 (REQUEST_SLOW)

[Robert LeBlanc]

On Mon, Jun 10, 2019 at 1:00 AM BASSAGET Cédric <
cedric.bassaget...@gmail.com> wrote:

> Hello Robert,
> My disks did not reach 100% on the last warning, they climb to 70-80%
> usage. But I see rrqm / wrqm counters increasing...
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   0.00 4.000.00   16.00 0.00   104.0013.00
> 0.000.000.000.00   0.00   0.00
> sdb   0.00 2.001.00 3456.00 8.00 25996.0015.04
> 5.761.670.001.67   0.03   9.20
> sdd   4.00 0.00 41462.00 1119.00 331272.00  7996.00
>  15.9419.890.470.480.21   0.02  66.00
>
> dm-0  0.00 0.00 6825.00  503.00 330856.00  7996.00
>  92.48 4.000.550.560.30   0.09  66.80
> dm-1  0.00 0.001.00 1129.00 8.00 25996.0046.02
> 1.030.910.000.91   0.09  10.00
>
>
> sda is my system disk (SAMSUNG   MZILS480HEGR/007  GXL0), sdb and sdd are
> my OSDs
>
> would "osd op queue = wpq" help in this case ?
> Regards
>

Your disk times look okay, just a lot more unbalanced than I would expect.
I'd give wpq a try, I use it all the time, just be sure to also include the
op_cutoff setting too or it doesn't have much effect. Let me know how it
goes.

Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests are blocked > 32 sec. Implicated osds 0, 2, 3, 4, 5 (REQUEST_SLOW)

[BASSAGET Cédric]

Hello Robert,
My disks did not reach 100% on the last warning, they climb to 70-80%
usage. But I see rrqm / wrqm counters increasing...

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util

sda   0.00 4.000.00   16.00 0.00   104.0013.00
0.000.000.000.00   0.00   0.00
sdb   0.00 2.001.00 3456.00 8.00 25996.0015.04
5.761.670.001.67   0.03   9.20
sdd   4.00 0.00 41462.00 1119.00 331272.00  7996.00
 15.9419.890.470.480.21   0.02  66.00

dm-0  0.00 0.00 6825.00  503.00 330856.00  7996.0092.48
4.000.550.560.30   0.09  66.80
dm-1  0.00 0.001.00 1129.00 8.00 25996.0046.02
1.030.910.000.91   0.09  10.00


sda is my system disk (SAMSUNG   MZILS480HEGR/007  GXL0), sdb and sdd are
my OSDs

would "osd op queue = wpq" help in this case ?
Regards

Le sam. 8 juin 2019 à 07:44, Robert LeBlanc  a écrit :

> With the low number of OSDs, you are probably satuarting the disks. Check
> with `iostat -xd 2` and see what the utilization of your disks are. A lot
> of SSDs don't perform well with Ceph's heavy sync writes and performance is
> terrible.
>
> If some of your drives are 100% while others are lower utilization, you
> can possibly get more performance and greatly reduce the blocked I/O with
> the WPQ scheduler. In the ceph.conf add this to the [osd] section and
> restart the processes:
>
> osd op queue = wpq
> osd op queue cut off = high
>
> This has helped our clusters with fairness between OSDs and making
> backfills not so disruptive.
> 
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>
> On Thu, Jun 6, 2019 at 1:43 AM BASSAGET Cédric <
> cedric.bassaget...@gmail.com> wrote:
>
>> Hello,
>>
>> I see messages related to REQUEST_SLOW a few times per day.
>>
>> here's my ceph -s  :
>>
>> root@ceph-pa2-1:/etc/ceph# ceph -s
>>   cluster:
>> id: 72d94815-f057-4127-8914-448dfd25f5bc
>> health: HEALTH_OK
>>
>>   services:
>> mon: 3 daemons, quorum ceph-pa2-1,ceph-pa2-2,ceph-pa2-3
>> mgr: ceph-pa2-3(active), standbys: ceph-pa2-1, ceph-pa2-2
>> osd: 6 osds: 6 up, 6 in
>>
>>   data:
>> pools:   1 pools, 256 pgs
>> objects: 408.79k objects, 1.49TiB
>> usage:   4.44TiB used, 37.5TiB / 41.9TiB avail
>> pgs: 256 active+clean
>>
>>   io:
>> client:   8.00KiB/s rd, 17.2MiB/s wr, 1op/s rd, 546op/s wr
>>
>>
>> Running ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217)
>> luminous (stable)
>>
>> I've check :
>> - all my network stack : OK ( 2*10G LAG )
>> - memory usage : ok (256G on each host, about 2% used per osd)
>> - cpu usage : OK (Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz)
>> - disk status : OK (SAMSUNG   AREA7680S5xnNTRI  3P04 => samsung DC series)
>>
>> I heard on IRC that it can be related to samsung PM / SM series.
>>
>> Do anybody here is facing the same problem ? What can I do to solve that ?
>> Regards,
>> Cédric
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests are blocked > 32 sec. Implicated osds 0, 2, 3, 4, 5 (REQUEST_SLOW)

[Robert LeBlanc]

With the low number of OSDs, you are probably satuarting the disks. Check
with `iostat -xd 2` and see what the utilization of your disks are. A lot
of SSDs don't perform well with Ceph's heavy sync writes and performance is
terrible.

If some of your drives are 100% while others are lower utilization, you can
possibly get more performance and greatly reduce the blocked I/O with the
WPQ scheduler. In the ceph.conf add this to the [osd] section and restart
the processes:

osd op queue = wpq
osd op queue cut off = high

This has helped our clusters with fairness between OSDs and making
backfills not so disruptive.

Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Thu, Jun 6, 2019 at 1:43 AM BASSAGET Cédric 
wrote:

> Hello,
>
> I see messages related to REQUEST_SLOW a few times per day.
>
> here's my ceph -s  :
>
> root@ceph-pa2-1:/etc/ceph# ceph -s
>   cluster:
> id: 72d94815-f057-4127-8914-448dfd25f5bc
> health: HEALTH_OK
>
>   services:
> mon: 3 daemons, quorum ceph-pa2-1,ceph-pa2-2,ceph-pa2-3
> mgr: ceph-pa2-3(active), standbys: ceph-pa2-1, ceph-pa2-2
> osd: 6 osds: 6 up, 6 in
>
>   data:
> pools:   1 pools, 256 pgs
> objects: 408.79k objects, 1.49TiB
> usage:   4.44TiB used, 37.5TiB / 41.9TiB avail
> pgs: 256 active+clean
>
>   io:
> client:   8.00KiB/s rd, 17.2MiB/s wr, 1op/s rd, 546op/s wr
>
>
> Running ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217)
> luminous (stable)
>
> I've check :
> - all my network stack : OK ( 2*10G LAG )
> - memory usage : ok (256G on each host, about 2% used per osd)
> - cpu usage : OK (Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz)
> - disk status : OK (SAMSUNG   AREA7680S5xnNTRI  3P04 => samsung DC series)
>
> I heard on IRC that it can be related to samsung PM / SM series.
>
> Do anybody here is facing the same problem ? What can I do to solve that ?
> Regards,
> Cédric
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] slow requests are blocked > 32 sec. Implicated osds 0, 2, 3, 4, 5 (REQUEST_SLOW)

[BASSAGET Cédric]

Hello,

I see messages related to REQUEST_SLOW a few times per day.

here's my ceph -s  :

root@ceph-pa2-1:/etc/ceph# ceph -s
  cluster:
id: 72d94815-f057-4127-8914-448dfd25f5bc
health: HEALTH_OK

  services:
mon: 3 daemons, quorum ceph-pa2-1,ceph-pa2-2,ceph-pa2-3
mgr: ceph-pa2-3(active), standbys: ceph-pa2-1, ceph-pa2-2
osd: 6 osds: 6 up, 6 in

  data:
pools:   1 pools, 256 pgs
objects: 408.79k objects, 1.49TiB
usage:   4.44TiB used, 37.5TiB / 41.9TiB avail
pgs: 256 active+clean

  io:
client:   8.00KiB/s rd, 17.2MiB/s wr, 1op/s rd, 546op/s wr


Running ceph version 12.2.9 (9e300932ef8a8916fb3fda78c58691a6ab0f4217)
luminous (stable)

I've check :
- all my network stack : OK ( 2*10G LAG )
- memory usage : ok (256G on each host, about 2% used per osd)
- cpu usage : OK (Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz)
- disk status : OK (SAMSUNG   AREA7680S5xnNTRI  3P04 => samsung DC series)

I heard on IRC that it can be related to samsung PM / SM series.

Do anybody here is facing the same problem ? What can I do to solve that ?
Regards,
Cédric
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests from bluestore osds / crashing rbd-nbd

[Charles Alva]

Got it. Thanks for the explanation, Jason!

Kind regards,

Charles Alva
Sent from Gmail Mobile


On Tue, May 21, 2019 at 5:16 PM Jason Dillaman  wrote:

> On Tue, May 21, 2019 at 12:03 PM Charles Alva 
> wrote:
> >
> > Hi Jason,
> >
> > Should we disable fstrim services inside VM which runs on top of RBD?
>
> It has a potential to be a thundering herd issue if you have lots of
> VMs all issuing discards all at the same time and your RBD images do
> not have object-map enabled. With object-map enabled, the discards
> will just get ignored if the backing objects do not already exist. You
> could hit a similar issue if you have hundreds or thousands of VMs all
> running a scheduled IO heavy task all at the same time (e.g. a yum/apt
> update every week at midnight), so it's not really tied to discard (or
> even Ceph/RBD) but more of a peak IOPS capacity issue.
>
> > I recall Ubuntu OS has weekly fstrim cronjob enabled by default, while
> we have to enable fstrim service manually on Debian and CentOS.
> >
> >
> > Kind regards,
> >
> > Charles Alva
> > Sent from Gmail Mobile
> >
> > On Tue, May 21, 2019, 4:49 AM Jason Dillaman 
> wrote:
> >>
> >> On Mon, May 20, 2019 at 2:17 PM Marc Schöchlin  wrote:
> >> >
> >> > Hello cephers,
> >> >
> >> > we have a few systems which utilize a rbd-bd map/mount to get access
> to a rbd volume.
> >> > (This problem seems to be related to "[ceph-users] Slow requests from
> bluestore osds" (the original thread))
> >> >
> >> > Unfortunately the rbd-nbd device of a system crashes three mondays in
> series at ~00:00 when the systemd fstrim timer executes "fstrim -av".
> >> > (which runs in parallel to deep scrub operations)
> >>
> >> That's probably not a good practice if you have lots of VMs doing this
> >> at the same time *and* you are not using object-map. The reason is
> >> that "fstrim" could discard huge extents that result around a thousand
> >> concurrent remove/truncate/zero ops per image being thrown at your
> >> cluster.
> >>
> >> > After that the device constantly reports io errors every time a
> access to the filesystem happens.
> >> > Unmounting, remapping and mounting helped to get the
> filesystem/device back into business :-)
> >>
> >> If the cluster was being DDoSed by the fstrims, the VM OSes' might
> >> have timed out thinking a controller failure.
> >>
> >> > Manual 30 minute stresstests using the following fio command, did not
> produce any problems on client side
> >> > (Ceph storage reported some slow requests while testing).
> >> >
> >> > fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
> --name=test --filename=test --bs=4k --iodepth=64 --size=4G
> --readwrite=randrw --rwmixread=50 --numjobs=50 --loops=10
> >> >
> >> > It seems that others also experienced this problem:
> https://ceph-users.ceph.narkive.com/2FIfyx1U/rbd-nbd-timeout-and-crash
> >> > The change for setting device timeouts by not seems to be merged to
> luminous.
> >> > Experiments setting the timeout manually after mapping using
> https://github.com/OnApp/nbd-kernel_mod/blob/master/nbd_set_timeout.c
> haven't change the situation.
> >> >
> >> > Do you have suggestions how to analyze/solve the situation?
> >> >
> >> > Regards
> >> > Marc
> >> > 
> >> >
> >> >
> >> >
> >> > The client kernel throws messages like this:
> >> >
> >> > May 19 23:59:01 int-nfs-001 CRON[836295]: (root) CMD (command -v
> debian-sa1 > /dev/null && debian-sa1 60 2)
> >> > May 20 00:00:30 int-nfs-001 systemd[1]: Starting Discard unused
> blocks...
> >> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623582] block nbd0:
> Connection timed out
> >> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623613] block nbd0:
> shutting down sockets
> >> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623617] print_req_error:
> I/O error, dev nbd0, sector 84082280
> >> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623632] block nbd0:
> Connection timed out
> >> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623636] print_req_error:
> I/O error, dev nbd0, sector 92470887
> >> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623642] block nbd0:
> Connection timed out
> >> >
> >> >

Re: [ceph-users] Slow requests from bluestore osds / crashing rbd-nbd

[Jason Dillaman]

On Tue, May 21, 2019 at 12:03 PM Charles Alva  wrote:
>
> Hi Jason,
>
> Should we disable fstrim services inside VM which runs on top of RBD?

It has a potential to be a thundering herd issue if you have lots of
VMs all issuing discards all at the same time and your RBD images do
not have object-map enabled. With object-map enabled, the discards
will just get ignored if the backing objects do not already exist. You
could hit a similar issue if you have hundreds or thousands of VMs all
running a scheduled IO heavy task all at the same time (e.g. a yum/apt
update every week at midnight), so it's not really tied to discard (or
even Ceph/RBD) but more of a peak IOPS capacity issue.

> I recall Ubuntu OS has weekly fstrim cronjob enabled by default, while we 
> have to enable fstrim service manually on Debian and CentOS.
>
>
> Kind regards,
>
> Charles Alva
> Sent from Gmail Mobile
>
> On Tue, May 21, 2019, 4:49 AM Jason Dillaman  wrote:
>>
>> On Mon, May 20, 2019 at 2:17 PM Marc Schöchlin  wrote:
>> >
>> > Hello cephers,
>> >
>> > we have a few systems which utilize a rbd-bd map/mount to get access to a 
>> > rbd volume.
>> > (This problem seems to be related to "[ceph-users] Slow requests from 
>> > bluestore osds" (the original thread))
>> >
>> > Unfortunately the rbd-nbd device of a system crashes three mondays in 
>> > series at ~00:00 when the systemd fstrim timer executes "fstrim -av".
>> > (which runs in parallel to deep scrub operations)
>>
>> That's probably not a good practice if you have lots of VMs doing this
>> at the same time *and* you are not using object-map. The reason is
>> that "fstrim" could discard huge extents that result around a thousand
>> concurrent remove/truncate/zero ops per image being thrown at your
>> cluster.
>>
>> > After that the device constantly reports io errors every time a access to 
>> > the filesystem happens.
>> > Unmounting, remapping and mounting helped to get the filesystem/device 
>> > back into business :-)
>>
>> If the cluster was being DDoSed by the fstrims, the VM OSes' might
>> have timed out thinking a controller failure.
>>
>> > Manual 30 minute stresstests using the following fio command, did not 
>> > produce any problems on client side
>> > (Ceph storage reported some slow requests while testing).
>> >
>> > fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 
>> > --name=test --filename=test --bs=4k --iodepth=64 --size=4G 
>> > --readwrite=randrw --rwmixread=50 --numjobs=50 --loops=10
>> >
>> > It seems that others also experienced this problem: 
>> > https://ceph-users.ceph.narkive.com/2FIfyx1U/rbd-nbd-timeout-and-crash
>> > The change for setting device timeouts by not seems to be merged to 
>> > luminous.
>> > Experiments setting the timeout manually after mapping using 
>> > https://github.com/OnApp/nbd-kernel_mod/blob/master/nbd_set_timeout.c 
>> > haven't change the situation.
>> >
>> > Do you have suggestions how to analyze/solve the situation?
>> >
>> > Regards
>> > Marc
>> > 
>> >
>> >
>> >
>> > The client kernel throws messages like this:
>> >
>> > May 19 23:59:01 int-nfs-001 CRON[836295]: (root) CMD (command -v 
>> > debian-sa1 > /dev/null && debian-sa1 60 2)
>> > May 20 00:00:30 int-nfs-001 systemd[1]: Starting Discard unused blocks...
>> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623582] block nbd0: 
>> > Connection timed out
>> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623613] block nbd0: shutting 
>> > down sockets
>> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623617] print_req_error: I/O 
>> > error, dev nbd0, sector 84082280
>> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623632] block nbd0: 
>> > Connection timed out
>> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623636] print_req_error: I/O 
>> > error, dev nbd0, sector 92470887
>> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623642] block nbd0: 
>> > Connection timed out
>> >
>> > Ceph throws messages like this:
>> >
>> > 2019-05-20 00:00:00.000124 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 
>> > 173572 : cluster [INF] overall HEALTH_OK
>> > 2019-05-20 00:00:54.249998 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 
>> > 173586 : cluster [WRN] Health check failed: 644 slow requests are

Re: [ceph-users] Slow requests from bluestore osds / crashing rbd-nbd

[Charles Alva]

Hi Jason,

Should we disable fstrim services inside VM which runs on top of RBD?

I recall Ubuntu OS has weekly fstrim cronjob enabled by default, while we
have to enable fstrim service manually on Debian and CentOS.


Kind regards,

Charles Alva
Sent from Gmail Mobile

On Tue, May 21, 2019, 4:49 AM Jason Dillaman  wrote:

> On Mon, May 20, 2019 at 2:17 PM Marc Schöchlin  wrote:
> >
> > Hello cephers,
> >
> > we have a few systems which utilize a rbd-bd map/mount to get access to
> a rbd volume.
> > (This problem seems to be related to "[ceph-users] Slow requests from
> bluestore osds" (the original thread))
> >
> > Unfortunately the rbd-nbd device of a system crashes three mondays in
> series at ~00:00 when the systemd fstrim timer executes "fstrim -av".
> > (which runs in parallel to deep scrub operations)
>
> That's probably not a good practice if you have lots of VMs doing this
> at the same time *and* you are not using object-map. The reason is
> that "fstrim" could discard huge extents that result around a thousand
> concurrent remove/truncate/zero ops per image being thrown at your
> cluster.
>
> > After that the device constantly reports io errors every time a access
> to the filesystem happens.
> > Unmounting, remapping and mounting helped to get the filesystem/device
> back into business :-)
>
> If the cluster was being DDoSed by the fstrims, the VM OSes' might
> have timed out thinking a controller failure.
>
> > Manual 30 minute stresstests using the following fio command, did not
> produce any problems on client side
> > (Ceph storage reported some slow requests while testing).
> >
> > fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
> --name=test --filename=test --bs=4k --iodepth=64 --size=4G
> --readwrite=randrw --rwmixread=50 --numjobs=50 --loops=10
> >
> > It seems that others also experienced this problem:
> https://ceph-users.ceph.narkive.com/2FIfyx1U/rbd-nbd-timeout-and-crash
> > The change for setting device timeouts by not seems to be merged to
> luminous.
> > Experiments setting the timeout manually after mapping using
> https://github.com/OnApp/nbd-kernel_mod/blob/master/nbd_set_timeout.c
> haven't change the situation.
> >
> > Do you have suggestions how to analyze/solve the situation?
> >
> > Regards
> > Marc
> > 
> >
> >
> >
> > The client kernel throws messages like this:
> >
> > May 19 23:59:01 int-nfs-001 CRON[836295]: (root) CMD (command -v
> debian-sa1 > /dev/null && debian-sa1 60 2)
> > May 20 00:00:30 int-nfs-001 systemd[1]: Starting Discard unused blocks...
> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623582] block nbd0:
> Connection timed out
> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623613] block nbd0:
> shutting down sockets
> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623617] print_req_error:
> I/O error, dev nbd0, sector 84082280
> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623632] block nbd0:
> Connection timed out
> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623636] print_req_error:
> I/O error, dev nbd0, sector 92470887
> > May 20 00:01:02 int-nfs-001 kernel: [1077851.623642] block nbd0:
> Connection timed out
> >
> > Ceph throws messages like this:
> >
> > 2019-05-20 00:00:00.000124 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 173572 : cluster [INF] overall HEALTH_OK
> > 2019-05-20 00:00:54.249998 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 173586 : cluster [WRN] Health check failed: 644 slow requests are blocked >
> 32 sec. Implicated osds 51 (REQUEST_SLOW)
> > 2019-05-20 00:01:00.330566 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 173587 : cluster [WRN] Health check update: 594 slow requests are blocked >
> 32 sec. Implicated osds 51 (REQUEST_SLOW)
> > 2019-05-20 00:01:09.768476 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 173591 : cluster [WRN] Health check update: 505 slow requests are blocked >
> 32 sec. Implicated osds 51 (REQUEST_SLOW)
> > 2019-05-20 00:01:14.768769 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 173592 : cluster [WRN] Health check update: 497 slow requests are blocked >
> 32 sec. Implicated osds 51 (REQUEST_SLOW)
> > 2019-05-20 00:01:20.610398 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 173593 : cluster [WRN] Health check update: 509 slow requests are blocked >
> 32 sec. Implicated osds 51 (REQUEST_SLOW)
> > 2019-05-20 00:01:28.721891 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 173594 : cluster [WRN] Health check update: 501 slow requests are blocked >
> 32 sec. Implicated osds 51 (

Re: [ceph-users] Slow requests from bluestore osds / crashing rbd-nbd

[Jason Dillaman]

On Tue, May 21, 2019 at 11:28 AM Marc Schöchlin  wrote:
>
> Hello Jason,
>
> Am 20.05.19 um 23:49 schrieb Jason Dillaman:
>
> On Mon, May 20, 2019 at 2:17 PM Marc Schöchlin  wrote:
>
> Hello cephers,
>
> we have a few systems which utilize a rbd-bd map/mount to get access to a rbd 
> volume.
> (This problem seems to be related to "[ceph-users] Slow requests from 
> bluestore osds" (the original thread))
>
> Unfortunately the rbd-nbd device of a system crashes three mondays in series 
> at ~00:00 when the systemd fstrim timer executes "fstrim -av".
> (which runs in parallel to deep scrub operations)
>
> That's probably not a good practice if you have lots of VMs doing this
> at the same time *and* you are not using object-map. The reason is
> that "fstrim" could discard huge extents that result around a thousand
> concurrent remove/truncate/zero ops per image being thrown at your
> cluster.
>
> Sure, currently we do not have lots of vms which are capable to run fstim on 
> rbd volumes.
> But the already involved RBD Images are multiple-tb images with a high 
> write/deletetion rate.
> Therefore i am already in progress to distribute fstrims by adding random 
> delays
>
> After that the device constantly reports io errors every time a access to the 
> filesystem happens.
> Unmounting, remapping and mounting helped to get the filesystem/device back 
> into business :-)
>
> If the cluster was being DDoSed by the fstrims, the VM OSes' might
> have timed out thinking a controller failure.
>
>
> Yes and no :-) Probably my problem is related to the kernel release, kernel 
> setting or the operating system release.
> Why?
>
> we run ~800 RBD images on that ceph cluster with rbd-nbd 12.2.5 in our xen 
> cluster as dom0-storage repository device without any timeout problems
> (kernel 4.4.0+10, centos 7)
> we run some 35TB kRBD images with multiples of the load of the crashed 
> rbd-nbd very write/read/deletion load without any timeout problems
> the timeout problem appears on two vms (ubuntu 18.04, ubuntu 16.04) which 
> utilize the described settings
>
> From my point of view, the error behavior is currently reproducible with a 
> good probability.
> Do you have suggestions how to find the root cause of this problem?

Can you provide any logs/backtraces/core dumps from the rbd-nbd process?

> Regards
> Marc


-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests from bluestore osds / crashing rbd-nbd

[Marc Schöchlin]

Hello Jason,

Am 20.05.19 um 23:49 schrieb Jason Dillaman:
> On Mon, May 20, 2019 at 2:17 PM Marc Schöchlin  wrote:
>> Hello cephers,
>>
>> we have a few systems which utilize a rbd-bd map/mount to get access to a 
>> rbd volume.
>> (This problem seems to be related to "[ceph-users] Slow requests from 
>> bluestore osds" (the original thread))
>>
>> Unfortunately the rbd-nbd device of a system crashes three mondays in series 
>> at ~00:00 when the systemd fstrim timer executes "fstrim -av".
>> (which runs in parallel to deep scrub operations)
> That's probably not a good practice if you have lots of VMs doing this
> at the same time *and* you are not using object-map. The reason is
> that "fstrim" could discard huge extents that result around a thousand
> concurrent remove/truncate/zero ops per image being thrown at your
> cluster.
Sure, currently we do not have lots of vms which are capable to run fstim on 
rbd volumes.
But the already involved RBD Images are multiple-tb images with a high 
write/deletetion rate.
Therefore i am already in progress to distribute fstrims by adding random delays
>
>> After that the device constantly reports io errors every time a access to 
>> the filesystem happens.
>> Unmounting, remapping and mounting helped to get the filesystem/device back 
>> into business :-)
> If the cluster was being DDoSed by the fstrims, the VM OSes' might
> have timed out thinking a controller failure.

Yes and no :-) Probably my problem is related to the kernel release, kernel 
setting or the operating system release.
Why?

  * we run ~800 RBD images on that ceph cluster with rbd-nbd 12.2.5 in our xen 
cluster as dom0-storage repository device without any timeout problems
(kernel 4.4.0+10, centos 7)
  * we run some 35TB kRBD images with multiples of the load of the crashed 
rbd-nbd very write/read/deletion load without any timeout problems
  * the timeout problem appears on two vms (ubuntu 18.04, ubuntu 16.04) which 
utilize the described settings

From my point of view, the error behavior is currently reproducible with a good 
probability.
Do you have suggestions how to find the root cause of this problem?

Regards
Marc

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests from bluestore osds / crashing rbd-nbd

[Jason Dillaman]

On Mon, May 20, 2019 at 2:17 PM Marc Schöchlin  wrote:
>
> Hello cephers,
>
> we have a few systems which utilize a rbd-bd map/mount to get access to a rbd 
> volume.
> (This problem seems to be related to "[ceph-users] Slow requests from 
> bluestore osds" (the original thread))
>
> Unfortunately the rbd-nbd device of a system crashes three mondays in series 
> at ~00:00 when the systemd fstrim timer executes "fstrim -av".
> (which runs in parallel to deep scrub operations)

That's probably not a good practice if you have lots of VMs doing this
at the same time *and* you are not using object-map. The reason is
that "fstrim" could discard huge extents that result around a thousand
concurrent remove/truncate/zero ops per image being thrown at your
cluster.

> After that the device constantly reports io errors every time a access to the 
> filesystem happens.
> Unmounting, remapping and mounting helped to get the filesystem/device back 
> into business :-)

If the cluster was being DDoSed by the fstrims, the VM OSes' might
have timed out thinking a controller failure.

> Manual 30 minute stresstests using the following fio command, did not produce 
> any problems on client side
> (Ceph storage reported some slow requests while testing).
>
> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test 
> --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw 
> --rwmixread=50 --numjobs=50 --loops=10
>
> It seems that others also experienced this problem: 
> https://ceph-users.ceph.narkive.com/2FIfyx1U/rbd-nbd-timeout-and-crash
> The change for setting device timeouts by not seems to be merged to luminous.
> Experiments setting the timeout manually after mapping using 
> https://github.com/OnApp/nbd-kernel_mod/blob/master/nbd_set_timeout.c haven't 
> change the situation.
>
> Do you have suggestions how to analyze/solve the situation?
>
> Regards
> Marc
> 
>
>
>
> The client kernel throws messages like this:
>
> May 19 23:59:01 int-nfs-001 CRON[836295]: (root) CMD (command -v debian-sa1 > 
> /dev/null && debian-sa1 60 2)
> May 20 00:00:30 int-nfs-001 systemd[1]: Starting Discard unused blocks...
> May 20 00:01:02 int-nfs-001 kernel: [1077851.623582] block nbd0: Connection 
> timed out
> May 20 00:01:02 int-nfs-001 kernel: [1077851.623613] block nbd0: shutting 
> down sockets
> May 20 00:01:02 int-nfs-001 kernel: [1077851.623617] print_req_error: I/O 
> error, dev nbd0, sector 84082280
> May 20 00:01:02 int-nfs-001 kernel: [1077851.623632] block nbd0: Connection 
> timed out
> May 20 00:01:02 int-nfs-001 kernel: [1077851.623636] print_req_error: I/O 
> error, dev nbd0, sector 92470887
> May 20 00:01:02 int-nfs-001 kernel: [1077851.623642] block nbd0: Connection 
> timed out
>
> Ceph throws messages like this:
>
> 2019-05-20 00:00:00.000124 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173572 
> : cluster [INF] overall HEALTH_OK
> 2019-05-20 00:00:54.249998 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173586 
> : cluster [WRN] Health check failed: 644 slow requests are blocked > 32 sec. 
> Implicated osds 51 (REQUEST_SLOW)
> 2019-05-20 00:01:00.330566 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173587 
> : cluster [WRN] Health check update: 594 slow requests are blocked > 32 sec. 
> Implicated osds 51 (REQUEST_SLOW)
> 2019-05-20 00:01:09.768476 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173591 
> : cluster [WRN] Health check update: 505 slow requests are blocked > 32 sec. 
> Implicated osds 51 (REQUEST_SLOW)
> 2019-05-20 00:01:14.768769 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173592 
> : cluster [WRN] Health check update: 497 slow requests are blocked > 32 sec. 
> Implicated osds 51 (REQUEST_SLOW)
> 2019-05-20 00:01:20.610398 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173593 
> : cluster [WRN] Health check update: 509 slow requests are blocked > 32 sec. 
> Implicated osds 51 (REQUEST_SLOW)
> 2019-05-20 00:01:28.721891 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173594 
> : cluster [WRN] Health check update: 501 slow requests are blocked > 32 sec. 
> Implicated osds 51 (REQUEST_SLOW)
> 2019-05-20 00:01:34.909842 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173596 
> : cluster [WRN] Health check update: 494 slow requests are blocked > 32 sec. 
> Implicated osds 51 (REQUEST_SLOW)
> 2019-05-20 00:01:44.770330 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173597 
> : cluster [WRN] Health check update: 500 slow requests are blocked > 32 sec. 
> Implicated osds 51 (REQUEST_SLOW)
> 2019-05-20 00:01:49.770625 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173599 
> : cluster [WRN] Health check update: 608 slow requests are blocked > 32 s

Re: [ceph-users] Slow requests from bluestore osds / crashing rbd-nbd

[Marc Schöchlin]

Hello cephers,

we have a few systems which utilize a rbd-bd map/mount to get access to a rbd 
volume.
(This problem seems to be related to "[ceph-users] Slow requests from bluestore 
osds" (the original thread))

Unfortunately the rbd-nbd device of a system crashes three mondays in series at 
~00:00 when the systemd fstrim timer executes "fstrim -av".
(which runs in parallel to deep scrub operations)

After that the device constantly reports io errors every time a access to the 
filesystem happens.
Unmounting, remapping and mounting helped to get the filesystem/device back 
into business :-)

Manual 30 minute stresstests using the following fio command, did not produce 
any problems on client side
(Ceph storage reported some slow requests while testing).

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test 
--filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw 
--rwmixread=50 --numjobs=50 --loops=10

It seems that others also experienced this problem: 
https://ceph-users.ceph.narkive.com/2FIfyx1U/rbd-nbd-timeout-and-crash
The change for setting device timeouts by not seems to be merged to luminous.
Experiments setting the timeout manually after mapping using 
https://github.com/OnApp/nbd-kernel_mod/blob/master/nbd_set_timeout.c haven't 
change the situation.

Do you have suggestions how to analyze/solve the situation?

Regards
Marc
--



The client kernel throws messages like this:

May 19 23:59:01 int-nfs-001 CRON[836295]: (root) CMD (command -v debian-sa1 > 
/dev/null && debian-sa1 60 2)
May 20 00:00:30 int-nfs-001 systemd[1]: Starting Discard unused blocks...
May 20 00:01:02 int-nfs-001 kernel: [1077851.623582] block nbd0: Connection 
timed out
May 20 00:01:02 int-nfs-001 kernel: [1077851.623613] block nbd0: shutting down 
sockets
May 20 00:01:02 int-nfs-001 kernel: [1077851.623617] print_req_error: I/O 
error, dev nbd0, sector 84082280
May 20 00:01:02 int-nfs-001 kernel: [1077851.623632] block nbd0: Connection 
timed out
May 20 00:01:02 int-nfs-001 kernel: [1077851.623636] print_req_error: I/O 
error, dev nbd0, sector 92470887
May 20 00:01:02 int-nfs-001 kernel: [1077851.623642] block nbd0: Connection 
timed out

Ceph throws messages like this:

2019-05-20 00:00:00.000124 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173572 : 
cluster [INF] overall HEALTH_OK
2019-05-20 00:00:54.249998 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173586 : 
cluster [WRN] Health check failed: 644 slow requests are blocked > 32 sec. 
Implicated osds 51 (REQUEST_SLOW)
2019-05-20 00:01:00.330566 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173587 : 
cluster [WRN] Health check update: 594 slow requests are blocked > 32 sec. 
Implicated osds 51 (REQUEST_SLOW)
2019-05-20 00:01:09.768476 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173591 : 
cluster [WRN] Health check update: 505 slow requests are blocked > 32 sec. 
Implicated osds 51 (REQUEST_SLOW)
2019-05-20 00:01:14.768769 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173592 : 
cluster [WRN] Health check update: 497 slow requests are blocked > 32 sec. 
Implicated osds 51 (REQUEST_SLOW)
2019-05-20 00:01:20.610398 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173593 : 
cluster [WRN] Health check update: 509 slow requests are blocked > 32 sec. 
Implicated osds 51 (REQUEST_SLOW)
2019-05-20 00:01:28.721891 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173594 : 
cluster [WRN] Health check update: 501 slow requests are blocked > 32 sec. 
Implicated osds 51 (REQUEST_SLOW)
2019-05-20 00:01:34.909842 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173596 : 
cluster [WRN] Health check update: 494 slow requests are blocked > 32 sec. 
Implicated osds 51 (REQUEST_SLOW)
2019-05-20 00:01:44.770330 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173597 : 
cluster [WRN] Health check update: 500 slow requests are blocked > 32 sec. 
Implicated osds 51 (REQUEST_SLOW)
2019-05-20 00:01:49.770625 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0 173599 : 
cluster [WRN]

Re: [ceph-users] Slow requests from bluestore osds

[Stefan Kooman]

Quoting Marc Schöchlin (m...@256bit.org):

> Out new setup is now:
> (12.2.10 on Ubuntu 16.04)
> 
> [osd]
> osd deep scrub interval = 2592000
> osd scrub begin hour = 19
> osd scrub end hour = 6
> osd scrub load threshold = 6
> osd scrub sleep = 0.3
> osd snap trim sleep = 0.4
> pg max concurrent snap trims = 1
> 
> [osd.51]
> osd memory target = 8589934592

I would upgrade to 12.2.12 and set the following:

[osd]
bluestore_allocator = bitmap
bluefs_allocator = bitmap

Just to make sure you're not hit by the "stupid allocator" behaviour,
which (also) might result in slow ops after $period of OSD uptime.

Gr. Stefan

-- 
| BIT BV  http://www.bit.nl/Kamer van Koophandel 09090351
| GPG: 0xD14839C6   +31 318 648 688 / i...@bit.nl
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests from bluestore osds

[Marc Schöchlin]

Hi Manuel, hello list,

what is the reason for manually compacting the osd?

In past debugging sessions we saw that slow requests appeared in correlation to 
compaction messages of the involved osds.
In this situation we haven't seen this (see logfile below in my last mail).

As described our cluster runs on 12.2.10 and our xen-dom-0 clients mainly 
utilize 10.2.5 with rbd-nbd.

It seems that there were some activities in in improving slow request problems 
in recent ceph releases due to the fact that users report problems in this area 
in the last months.
Can you recommend to update "rbd-rbd" or the cluster itself to improve the 
situation?

Best regards
Marc

Am 13.05.19 um 07:40 schrieb EDH - Manuel Rios Fernandez:
> Hi Marc,
>
> Try to compact OSD with slow request 
>
> ceph tell osd.[ID] compact
>
> This will make the OSD offline for some seconds(SSD) to minutes(HDD) and 
> perform a compact of OMAP database.
>
> Regards,
>
>
>
>
> -Mensaje original-
> De: ceph-users  En nombre de Marc Schöchlin
> Enviado el: lunes, 13 de mayo de 2019 6:59
> Para: ceph-users@lists.ceph.com
> Asunto: Re: [ceph-users] Slow requests from bluestore osds
>
> Hello cephers,
>
> one week ago we replaced the bluestore cache size by "osd memory target" and 
> removed the detail memory settings.
> This storage class now runs 42*8GB spinners with a permanent write workload 
> of 2000-3000 write IOPS, and 1200-8000 read IOPS.
>
> Out new setup is now:
> (12.2.10 on Ubuntu 16.04)
>
> [osd]
> osd deep scrub interval = 2592000
> osd scrub begin hour = 19
> osd scrub end hour = 6
> osd scrub load threshold = 6
> osd scrub sleep = 0.3
> osd snap trim sleep = 0.4
> pg max concurrent snap trims = 1
>
> [osd.51]
> osd memory target = 8589934592
> ...
>
> After that (restarting the entire cluster with these settings) we were very 
> happy to not seeany slow request for 7 days.
>
> Unfortunately this night the slow requests returned on one osd without any 
> known change of the workload of the last 14 days (according to our detailed 
> monitoring)
>
> 2019-05-12 22:00:00.000117 mon.ceph-mon-s43 [INF] overall HEALTH_OK
> 2019-05-12 23:00:00.000130 mon.ceph-mon-s43 [INF] overall HEALTH_OK
> 2019-05-13 00:00:00.000129 mon.ceph-mon-s43 [INF] overall HEALTH_OK
> 2019-05-13 00:00:44.069793 mon.ceph-mon-s43 [WRN] Health check failed: 416 
> slow requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
> 2019-05-13 00:00:50.151190 mon.ceph-mon-s43 [WRN] Health check update: 439 
> slow requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
> 2019-05-13 00:00:59.750398 mon.ceph-mon-s43 [WRN] Health check update: 452 
> slow requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
> 2019-05-13 00:01:04.750697 mon.ceph-mon-s43 [WRN] Health check update: 283 
> slow requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
> 2019-05-13 00:01:10.419801 mon.ceph-mon-s43 [WRN] Health check update: 230 
> slow requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
> 2019-05-13 00:01:19.751516 mon.ceph-mon-s43 [WRN] Health check update: 362 
> slow requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
> 2019-05-13 00:01:24.751822 mon.ceph-mon-s43 [WRN] Health check update: 324 
> slow requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
> 2019-05-13 00:01:30.675160 mon.ceph-mon-s43 [WRN] Health check update: 341 
> slow requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
> 2019-05-13 00:01:38.759012 mon.ceph-mon-s43 [WRN] Health check update: 390 
> slow requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
> 2019-05-13 00:01:44.858392 mon.ceph-mon-s43 [WRN] Health check update: 366 
> slow requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
> 2019-05-13 00:01:54.753388 mon.ceph-mon-s43 [WRN] Health check update: 352 
> slow requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
> 2019-05-13 00:01:59.045220 mon.ceph-mon-s43 [INF] Health check cleared: 
> REQUEST_SLOW (was: 168 slow requests are blocked > 32 sec. Implicated osds 51)
> 2019-05-13 00:01:59.045257 mon.ceph-mon-s43 [INF] Cluster is now healthy
> 2019-05-13 01:00:00.000114 mon.ceph-mon-s43 [INF] overall HEALTH_OK
> 2019-05-13 02:00:00.000130 mon.ceph-mon-s43 [INF] overall HEALTH_OK
>
>
> The output of a "ceph health detail" loop at the time the problem occurred:
>
> Mon May 13 00:01:27 CEST 2019
> HEALTH_WARN 324 slow requests are blocked > 32 sec. Implicated osds 51 
> REQUEST_SLOW 324 slow requests are blocked > 32 sec. Implicated osds 51
> 324 ops are blocked > 32.768 sec
> osd.51 has blocked requests > 32.768

Re: [ceph-users] Slow requests from bluestore osds

[EDH - Manuel Rios Fernandez]

Hi Marc,

Try to compact OSD with slow request 

ceph tell osd.[ID] compact

This will make the OSD offline for some seconds(SSD) to minutes(HDD) and 
perform a compact of OMAP database.

Regards,




-Mensaje original-
De: ceph-users  En nombre de Marc Schöchlin
Enviado el: lunes, 13 de mayo de 2019 6:59
Para: ceph-users@lists.ceph.com
Asunto: Re: [ceph-users] Slow requests from bluestore osds

Hello cephers,

one week ago we replaced the bluestore cache size by "osd memory target" and 
removed the detail memory settings.
This storage class now runs 42*8GB spinners with a permanent write workload of 
2000-3000 write IOPS, and 1200-8000 read IOPS.

Out new setup is now:
(12.2.10 on Ubuntu 16.04)

[osd]
osd deep scrub interval = 2592000
osd scrub begin hour = 19
osd scrub end hour = 6
osd scrub load threshold = 6
osd scrub sleep = 0.3
osd snap trim sleep = 0.4
pg max concurrent snap trims = 1

[osd.51]
osd memory target = 8589934592
...

After that (restarting the entire cluster with these settings) we were very 
happy to not seeany slow request for 7 days.

Unfortunately this night the slow requests returned on one osd without any 
known change of the workload of the last 14 days (according to our detailed 
monitoring)

2019-05-12 22:00:00.000117 mon.ceph-mon-s43 [INF] overall HEALTH_OK
2019-05-12 23:00:00.000130 mon.ceph-mon-s43 [INF] overall HEALTH_OK
2019-05-13 00:00:00.000129 mon.ceph-mon-s43 [INF] overall HEALTH_OK
2019-05-13 00:00:44.069793 mon.ceph-mon-s43 [WRN] Health check failed: 416 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:00:50.151190 mon.ceph-mon-s43 [WRN] Health check update: 439 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:00:59.750398 mon.ceph-mon-s43 [WRN] Health check update: 452 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:04.750697 mon.ceph-mon-s43 [WRN] Health check update: 283 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:10.419801 mon.ceph-mon-s43 [WRN] Health check update: 230 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:19.751516 mon.ceph-mon-s43 [WRN] Health check update: 362 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:24.751822 mon.ceph-mon-s43 [WRN] Health check update: 324 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:30.675160 mon.ceph-mon-s43 [WRN] Health check update: 341 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:38.759012 mon.ceph-mon-s43 [WRN] Health check update: 390 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:44.858392 mon.ceph-mon-s43 [WRN] Health check update: 366 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:54.753388 mon.ceph-mon-s43 [WRN] Health check update: 352 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:59.045220 mon.ceph-mon-s43 [INF] Health check cleared: 
REQUEST_SLOW (was: 168 slow requests are blocked > 32 sec. Implicated osds 51)
2019-05-13 00:01:59.045257 mon.ceph-mon-s43 [INF] Cluster is now healthy
2019-05-13 01:00:00.000114 mon.ceph-mon-s43 [INF] overall HEALTH_OK
2019-05-13 02:00:00.000130 mon.ceph-mon-s43 [INF] overall HEALTH_OK


The output of a "ceph health detail" loop at the time the problem occurred:

Mon May 13 00:01:27 CEST 2019
HEALTH_WARN 324 slow requests are blocked > 32 sec. Implicated osds 51 
REQUEST_SLOW 324 slow requests are blocked > 32 sec. Implicated osds 51
324 ops are blocked > 32.768 sec
osd.51 has blocked requests > 32.768 sec

The logfile of the OSD:

2019-05-12 23:57:28.767463 7f38da4e2700  4 rocksdb: (Original Log Time 
2019/05/12-23:57:28.767419) 
[/build/ceph-12.2.10/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] 
Level summary: base level 1 max b ytes base 268435456 files[2 4 21 122 0 0 0] 
max score 0.94

2019-05-12 23:57:28.767511 7f38da4e2700  4 rocksdb: 
[/build/ceph-12.2.10/src/rocksdb/db/db_impl_files.cc:388] [JOB 2991] Try to 
delete WAL files size 256700142, prev total WAL file size 257271487, number of 
live
 WAL files 2.

2019-05-12 23:58:07.816376 7f38ddce9700  0 log_channel(cluster) log [DBG] : 
34.ac scrub ok
2019-05-12 23:59:54.070025 7f38de4ea700  0 log_channel(cluster) log [DBG] : 
34.236 scrub starts
2019-05-13 00:02:21.818689 7f38de4ea700  0 log_channel(cluster) log [DBG] : 
34.236 scrub ok
2019-05-13 00:04:37.613094 7f38ead03700  4 rocksdb: 
[/build/ceph-12.2.10/src/rocksdb/db/db_impl_write.cc:684] reusing log 422507 
from recycle list

2019-05-13 00:04:37.613186 7f38ead03700  4 rocksdb: 
[/build/ceph-12.2.10/src/rocksdb/db/db_impl_write.cc:725] [default] New 
memtable created with log file: #422511. Immutable memtables: 0.

Any hints how to 

Re: [ceph-users] Slow requests from bluestore osds

[Marc Schöchlin]

Hello cephers,

one week ago we replaced the bluestore cache size by "osd memory target" and 
removed the detail memory settings.
This storage class now runs 42*8GB spinners with a permanent write workload of 
2000-3000 write IOPS, and 1200-8000 read IOPS.

Out new setup is now:
(12.2.10 on Ubuntu 16.04)

[osd]
osd deep scrub interval = 2592000
osd scrub begin hour = 19
osd scrub end hour = 6
osd scrub load threshold = 6
osd scrub sleep = 0.3
osd snap trim sleep = 0.4
pg max concurrent snap trims = 1

[osd.51]
osd memory target = 8589934592
...

After that (restarting the entire cluster with these settings) we were very 
happy to not seeany slow request for 7 days.

Unfortunately this night the slow requests returned on one osd without any 
known change of the workload of the last 14 days
(according to our detailed monitoring)

2019-05-12 22:00:00.000117 mon.ceph-mon-s43 [INF] overall HEALTH_OK
2019-05-12 23:00:00.000130 mon.ceph-mon-s43 [INF] overall HEALTH_OK
2019-05-13 00:00:00.000129 mon.ceph-mon-s43 [INF] overall HEALTH_OK
2019-05-13 00:00:44.069793 mon.ceph-mon-s43 [WRN] Health check failed: 416 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:00:50.151190 mon.ceph-mon-s43 [WRN] Health check update: 439 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:00:59.750398 mon.ceph-mon-s43 [WRN] Health check update: 452 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:04.750697 mon.ceph-mon-s43 [WRN] Health check update: 283 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:10.419801 mon.ceph-mon-s43 [WRN] Health check update: 230 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:19.751516 mon.ceph-mon-s43 [WRN] Health check update: 362 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:24.751822 mon.ceph-mon-s43 [WRN] Health check update: 324 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:30.675160 mon.ceph-mon-s43 [WRN] Health check update: 341 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:38.759012 mon.ceph-mon-s43 [WRN] Health check update: 390 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:44.858392 mon.ceph-mon-s43 [WRN] Health check update: 366 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:54.753388 mon.ceph-mon-s43 [WRN] Health check update: 352 slow 
requests are blocked > 32 sec. Implicated osds 51 (REQUEST_SLOW)
2019-05-13 00:01:59.045220 mon.ceph-mon-s43 [INF] Health check cleared: 
REQUEST_SLOW (was: 168 slow requests are blocked > 32 sec. Implicated osds 51)
2019-05-13 00:01:59.045257 mon.ceph-mon-s43 [INF] Cluster is now healthy
2019-05-13 01:00:00.000114 mon.ceph-mon-s43 [INF] overall HEALTH_OK
2019-05-13 02:00:00.000130 mon.ceph-mon-s43 [INF] overall HEALTH_OK


The output of a "ceph health detail" loop at the time the problem occurred:

Mon May 13 00:01:27 CEST 2019
HEALTH_WARN 324 slow requests are blocked > 32 sec. Implicated osds 51
REQUEST_SLOW 324 slow requests are blocked > 32 sec. Implicated osds 51
    324 ops are blocked > 32.768 sec
    osd.51 has blocked requests > 32.768 sec

The logfile of the OSD:

2019-05-12 23:57:28.767463 7f38da4e2700  4 rocksdb: (Original Log Time 
2019/05/12-23:57:28.767419) 
[/build/ceph-12.2.10/src/rocksdb/db/db_impl_compaction_flush.cc:132] [default] 
Level summary: base level 1 max b
ytes base 268435456 files[2 4 21 122 0 0 0] max score 0.94

2019-05-12 23:57:28.767511 7f38da4e2700  4 rocksdb: 
[/build/ceph-12.2.10/src/rocksdb/db/db_impl_files.cc:388] [JOB 2991] Try to 
delete WAL files size 256700142, prev total WAL file size 257271487, number of 
live
 WAL files 2.

2019-05-12 23:58:07.816376 7f38ddce9700  0 log_channel(cluster) log [DBG] : 
34.ac scrub ok
2019-05-12 23:59:54.070025 7f38de4ea700  0 log_channel(cluster) log [DBG] : 
34.236 scrub starts
2019-05-13 00:02:21.818689 7f38de4ea700  0 log_channel(cluster) log [DBG] : 
34.236 scrub ok
2019-05-13 00:04:37.613094 7f38ead03700  4 rocksdb: 
[/build/ceph-12.2.10/src/rocksdb/db/db_impl_write.cc:684] reusing log 422507 
from recycle list

2019-05-13 00:04:37.613186 7f38ead03700  4 rocksdb: 
[/build/ceph-12.2.10/src/rocksdb/db/db_impl_write.cc:725] [default] New 
memtable created with log file: #422511. Immutable memtables: 0.

Any hints how to find more details about the origin of this problem?
How can we solve that?

Regards
Marc

Am 28.01.19 um 22:27 schrieb Marc Schöchlin:
> Hello cephers,
>
> as described - we also have the slow requests in our setup.
>
> We recently updated from ceph 12.2.4 to 12.2.10, updated Ubuntu 16.04 to the 
> latest patchlevel (with kernel 4.15.0-43) and applied dell firmware 2.8.0.
>
> On 12.2.5 (before updating the cluster) we had in a frequency of 10min to 
> 30minutes in the entire de

Re: [ceph-users] Slow requests from bluestore osds

[Marc Schöchlin]

Hello cephers,

as described - we also have the slow requests in our setup.

We recently updated from ceph 12.2.4 to 12.2.10, updated Ubuntu 16.04 to the 
latest patchlevel (with kernel 4.15.0-43) and applied dell firmware 2.8.0.

On 12.2.5 (before updating the cluster) we had in a frequency of 10min to 
30minutes in the entire deepscrub-window between 8:00 PM and 6:00 AM.
Especially between 04:00AM and 06:00 AM when when we sequentially create a rbd 
snapshots for every rbd image and delete a outdated snapshot (we hold 3 
snapshots per rbd device).

After the upgrade to 12.2.10 (and the other patches) slow requests seems to be 
reduced, but they still occur after the snapshot creation/deletion procedure.
Today we changed the time of the creation/deletion procedure from 4:00 AM to 
7:30PM and we experienced slow request right in the the snapshot process at 
8:00PM.

The slow requests only happen on a certain storage class osds (30 * 8GB 
spinners)  - i.e ssd osds do not have this problem on the same cluster
The pools which use this storage class are loaded by 80% write requests.

Our configuration looks like this:
---
bluestore cache kv max = 2147483648
bluestore cache kv ratio = 0.9
bluestore cache meta ratio = 0.1
bluestore cache size hdd = 10737418240
osd deep scrub interval = 2592000
osd scrub begin hour = 19
osd scrub end hour = 6
osd scrub load threshold = 4
osd scrub sleep = 0.3
osd max trimming pgs = 2
---
We do not have so much devices in this storage class (a enhancement is in 
progress to get more iops)

What can i do to decrease the impact of snaptrims to prevent slow requests?
(i.e. reduce "osd max trimming pgs" to "1")

Regards
Marc Schöchlin

Am 03.09.18 um 10:13 schrieb Marc Schöchlin:
> Hi,
>
> we are also experiencing this type of behavior for some weeks on our not
> so performance critical hdd pools.
> We haven't spent so much time on this problem, because there are
> currently more important tasks - but here are a few details:
>
> Running the following loop results in the following output:
>
> while true; do ceph health|grep -q HEALTH_OK || (date;  ceph health
> detail); sleep 2; done
>
> Sun Sep  2 20:59:47 CEST 2018
> HEALTH_WARN 4 slow requests are blocked > 32 sec
> REQUEST_SLOW 4 slow requests are blocked > 32 sec
>     4 ops are blocked > 32.768 sec
>     osd.43 has blocked requests > 32.768 sec
> Sun Sep  2 20:59:50 CEST 2018
> HEALTH_WARN 4 slow requests are blocked > 32 sec
> REQUEST_SLOW 4 slow requests are blocked > 32 sec
>     4 ops are blocked > 32.768 sec
>     osd.43 has blocked requests > 32.768 sec
> Sun Sep  2 20:59:52 CEST 2018
> HEALTH_OK
> Sun Sep  2 21:00:28 CEST 2018
> HEALTH_WARN 1 slow requests are blocked > 32 sec
> REQUEST_SLOW 1 slow requests are blocked > 32 sec
>     1 ops are blocked > 32.768 sec
>     osd.41 has blocked requests > 32.768 sec
> Sun Sep  2 21:00:31 CEST 2018
> HEALTH_WARN 7 slow requests are blocked > 32 sec
> REQUEST_SLOW 7 slow requests are blocked > 32 sec
>     7 ops are blocked > 32.768 sec
>     osds 35,41 have blocked requests > 32.768 sec
> Sun Sep  2 21:00:33 CEST 2018
> HEALTH_WARN 7 slow requests are blocked > 32 sec
> REQUEST_SLOW 7 slow requests are blocked > 32 sec
>     7 ops are blocked > 32.768 sec
>     osds 35,51 have blocked requests > 32.768 sec
> Sun Sep  2 21:00:35 CEST 2018
> HEALTH_WARN 7 slow requests are blocked > 32 sec
> REQUEST_SLOW 7 slow requests are blocked > 32 sec
>     7 ops are blocked > 32.768 sec
>     osds 35,51 have blocked requests > 32.768 sec
>
> Our details:
>
>   * system details:
> * Ubuntu 16.04
>  * Kernel 4.13.0-39
>  * 30 * 8 TB Disk (SEAGATE/ST8000NM0075)
>  * 3* Dell Power Edge R730xd (Firmware 2.50.50.50)
>* Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
>* 2*10GBITS SFP+ Network Adapters
>* 192GB RAM
>  * Pools are using replication factor 3, 2MB object size,
>85% write load, 1700 write IOPS/sec
>(ops mainly between 4k and 16k size), 300 read IOPS/sec
>   * we have the impression that this appears on deepscrub/scrub activity.
>   * Ceph 12.2.5, we alread played with the osd settings OSD Settings
> (our assumtion was that the problem is related to rocksdb compaction)
> bluestore cache kv max = 2147483648
> bluestore cache kv ratio = 0.9
> bluestore cache meta ratio = 0.1
> bluestore cache size hdd = 10737418240
>   * this type problem only appears on hdd/bluestore osds, ssd/bluestore
> osds did never experienced that problem
>   * the system is healthy, no swapping, no high load, no errors in dmesg
>
> I attached a log excerpt of osd.35 - probably this is useful for
> investigating the problem is someone owns deeper bluestore knowledge.
> (slow requests appeared on Sun Sep  2 21:00:35)
>
> Regards
> Marc
>
>
> Am 02.09.2018 um 15:50 schrieb Brett Chancellor:
>> The warnings look like this. 
>>
>> 6 ops are blocked > 32.768 sec on osd.219
>> 1 osds have slow requests
>>
>> On Sun, Sep 2, 2018, 8:45 AM Alfredo 

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Mykola Golub]

On Fri, Jan 18, 2019 at 11:06:54AM -0600, Mark Nelson wrote:

> IE even though you guys set bluestore_cache_size to 1GB, it is being
> overridden by bluestore_cache_size_ssd.

Isn't it vice versa [1]?

[1] 
https://github.com/ceph/ceph/blob/luminous/src/os/bluestore/BlueStore.cc#L3976

-- 
Mykola Golub
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Mark Nelson]

,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 3 upset size
3 up 3
2019-01-16 15:19:23.503231 7fecb97ff700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f
1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472
pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=183,(3+0)=3}}] _update_calc_stats ml 183 upset
size 3 up 2

Greets,
Stefan
Am 16.01.19 um 09:12 schrieb Stefan Priebe - Profihost AG:

Hi,

no ok it was not. Bug still present. It was only working because the
osdmap was so far away that it has started backfill instead of
recovery.

So it happens only in the recovery case.

Greets,
Stefan

Am 15.01.19 um 16:02 schrieb Stefan Priebe - Profihost AG:

Am 15.01.19 um 12:45 schrieb Marc Roos:

   I upgraded this weekend from 12.2.8 to 12.2.10 without such
issues
(osd's are idle)

it turns out this was a kernel bug. Updating to a newer kernel -
has
solved this issue.

Greets,
Stefan



-Original Message-
From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag]
Sent: 15 January 2019 10:26
To: ceph-users@lists.ceph.com
Cc: n.fahldi...@profihost.ag
Subject: Re: [ceph-users] slow requests and high i/o / read
rate on
bluestore osds after upgrade 12.2.8 -> 12.2.10

Hello list,

i also tested current upstream/luminous branch and it happens
as well. A
clean install works fine. It only happens on upgraded bluestore
osds.

Greets,
Stefan

Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:

while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm

experience

issues with bluestore osds - so i canceled the upgrade and all

bluestore

osds are stopped now.

After starting a bluestore osd i'm seeing a lot of slow requests

caused

by very high read rates.


Device: rrqm/s   wrqm/s r/s w/s    rkB/s    wkB/s
avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda  45,00   187,00  767,00   39,00 482040,00
8660,00
1217,62    58,16   74,60   73,85   89,23   1,24 100,00

it reads permanently with 500MB/s from the disk and can't service

client

requests. Overall client read rate is at 10.9MiB/s rd

I can't reproduce this with 12.2.8. Is this a known bug /
regression?

Greets,
Stefan


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Nils Fahldieck - Profihost AG]

gt; ec=133405/133405 lis/c 1318472/1278145 les/c/f
>>>>>> 1318473/1278148/1211861 131
>>>>>> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472
>>>>>> pi=[1278145,1318472)/1
>>>>>> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
>>>>>> overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
>>>>>> mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 3 upset size
>>>>>> 3 up 3
>>>>>> 2019-01-16 15:19:23.503231 7fecb97ff700  0 osd.33 pg_epoch: 1318479
>>>>>> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
>>>>>> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
>>>>>> ec=133405/133405 lis/c 1318472/1278145 les/c/f
>>>>>> 1318473/1278148/1211861 131
>>>>>> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472
>>>>>> pi=[1278145,1318472)/1
>>>>>> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
>>>>>> overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
>>>>>> mbc={255={(2+0)=183,(3+0)=3}}] _update_calc_stats ml 183 upset
>>>>>> size 3 up 2
>>>>>>
>>>>>> Greets,
>>>>>> Stefan
>>>>>> Am 16.01.19 um 09:12 schrieb Stefan Priebe - Profihost AG:
>>>>>>> Hi,
>>>>>>>
>>>>>>> no ok it was not. Bug still present. It was only working because the
>>>>>>> osdmap was so far away that it has started backfill instead of
>>>>>>> recovery.
>>>>>>>
>>>>>>> So it happens only in the recovery case.
>>>>>>>
>>>>>>> Greets,
>>>>>>> Stefan
>>>>>>>
>>>>>>> Am 15.01.19 um 16:02 schrieb Stefan Priebe - Profihost AG:
>>>>>>>> Am 15.01.19 um 12:45 schrieb Marc Roos:
>>>>>>>>>   I upgraded this weekend from 12.2.8 to 12.2.10 without such
>>>>>>>>> issues
>>>>>>>>> (osd's are idle)
>>>>>>>>
>>>>>>>> it turns out this was a kernel bug. Updating to a newer kernel -
>>>>>>>> has
>>>>>>>> solved this issue.
>>>>>>>>
>>>>>>>> Greets,
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>>
>>>>>>>>> -Original Message-
>>>>>>>>> From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag]
>>>>>>>>> Sent: 15 January 2019 10:26
>>>>>>>>> To: ceph-users@lists.ceph.com
>>>>>>>>> Cc: n.fahldi...@profihost.ag
>>>>>>>>> Subject: Re: [ceph-users] slow requests and high i/o / read
>>>>>>>>> rate on
>>>>>>>>> bluestore osds after upgrade 12.2.8 -> 12.2.10
>>>>>>>>>
>>>>>>>>> Hello list,
>>>>>>>>>
>>>>>>>>> i also tested current upstream/luminous branch and it happens
>>>>>>>>> as well. A
>>>>>>>>> clean install works fine. It only happens on upgraded bluestore
>>>>>>>>> osds.
>>>>>>>>>
>>>>>>>>> Greets,
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>>> Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>> while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm
>>>>>>>>> experience
>>>>>>>>>> issues with bluestore osds - so i canceled the upgrade and all
>>>>>>>>> bluestore
>>>>>>>>>> osds are stopped now.
>>>>>>>>>>
>>>>>>>>>> After starting a bluestore osd i'm seeing a lot of slow requests
>>>>>>>>> caused
>>>>>>>>>> by very high read rates.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Device: rrqm/s   wrqm/s r/s w/s    rkB/s    wkB/s
>>>>>>>>>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>>>>>>>>>> sda  45,00   187,00  767,00   39,00 482040,00 
>>>>>>>>>> 8660,00
>>>>>>>>>> 1217,62    58,16   74,60   73,85   89,23   1,24 100,00
>>>>>>>>>>
>>>>>>>>>> it reads permanently with 500MB/s from the disk and can't service
>>>>>>>>> client
>>>>>>>>>> requests. Overall client read rate is at 10.9MiB/s rd
>>>>>>>>>>
>>>>>>>>>> I can't reproduce this with 12.2.8. Is this a known bug /
>>>>>>>>>> regression?
>>>>>>>>>>
>>>>>>>>>> Greets,
>>>>>>>>>> Stefan
>>>>>>>>>>
>>>>>>>>> ___
>>>>>>>>> ceph-users mailing list
>>>>>>>>> ceph-users@lists.ceph.com
>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>>>>
>>>>>>>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Mark Nelson]

)=2}}] _update_calc_stats ml 185 upset size 3 up 2
2019-01-16 15:19:13.568637 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=184 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=185,(3+0)=2}}] _update_calc_stats ml 2 upset size 3 up 3
2019-01-16 15:19:15.909327 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 184 upset size 3 up 2
2019-01-16 15:19:15.909446 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 3 upset size 3 up 3
2019-01-16 15:19:23.503231 7fecb97ff700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=183,(3+0)=3}}] _update_calc_stats ml 183 upset size 3 up 2

Greets,
Stefan
Am 16.01.19 um 09:12 schrieb Stefan Priebe - Profihost AG:

Hi,

no ok it was not. Bug still present. It was only working because the
osdmap was so far away that it has started backfill instead of recovery.

So it happens only in the recovery case.

Greets,
Stefan

Am 15.01.19 um 16:02 schrieb Stefan Priebe - Profihost AG:

Am 15.01.19 um 12:45 schrieb Marc Roos:
  
I upgraded this weekend from 12.2.8 to 12.2.10 without such issues

(osd's are idle)


it turns out this was a kernel bug. Updating to a newer kernel - has
solved this issue.

Greets,
Stefan



-Original Message-
From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag]
Sent: 15 January 2019 10:26
To: ceph-users@lists.ceph.com
Cc: n.fahldi...@profihost.ag
Subject: Re: [ceph-users] slow requests and high i/o / read rate on
bluestore osds after upgrade 12.2.8 -> 12.2.10

Hello list,

i also tested current upstream/luminous branch and it happens as well. A
clean install works fine. It only happens on upgraded bluestore osds.

Greets,
Stefan

Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:

while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm

experience

issues with bluestore osds - so i canceled the upgrade and all

bluestore

osds are stopped now.

After starting a bluestore osd i'm seeing a lot of slow requests

caused

by very high read rates.


Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda  45,00   187,00  767,00   39,00 482040,00  8660,00
1217,6258,16   74,60   73,85   89,23   1,24 100,00

it reads permanently with 500MB/s from the disk and can't service

client

requests. Overall client read rate is at 10.9MiB/s rd

I can't reproduce this with 12.2.8. Is this a known bug / regression?

Greets,
Stefan


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Stefan Priebe - Profihost AG]

/1211861 131
>>>> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
>>>> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
>>>> overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
>>>> mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 184 upset size 3 up 2
>>>> 2019-01-16 15:19:15.909446 7fecbf7da700  0 osd.33 pg_epoch: 1318479
>>>> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
>>>> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
>>>> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
>>>> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
>>>> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
>>>> overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
>>>> mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 3 upset size 3 up 3
>>>> 2019-01-16 15:19:23.503231 7fecb97ff700  0 osd.33 pg_epoch: 1318479
>>>> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
>>>> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
>>>> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
>>>> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
>>>> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
>>>> overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
>>>> mbc={255={(2+0)=183,(3+0)=3}}] _update_calc_stats ml 183 upset size 3 up 2
>>>>
>>>> Greets,
>>>> Stefan
>>>> Am 16.01.19 um 09:12 schrieb Stefan Priebe - Profihost AG:
>>>>> Hi,
>>>>>
>>>>> no ok it was not. Bug still present. It was only working because the
>>>>> osdmap was so far away that it has started backfill instead of recovery.
>>>>>
>>>>> So it happens only in the recovery case.
>>>>>
>>>>> Greets,
>>>>> Stefan
>>>>>
>>>>> Am 15.01.19 um 16:02 schrieb Stefan Priebe - Profihost AG:
>>>>>>
>>>>>> Am 15.01.19 um 12:45 schrieb Marc Roos:
>>>>>>>  
>>>>>>> I upgraded this weekend from 12.2.8 to 12.2.10 without such issues 
>>>>>>> (osd's are idle)
>>>>>>
>>>>>>
>>>>>> it turns out this was a kernel bug. Updating to a newer kernel - has
>>>>>> solved this issue.
>>>>>>
>>>>>> Greets,
>>>>>> Stefan
>>>>>>
>>>>>>
>>>>>>> -Original Message-
>>>>>>> From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] 
>>>>>>> Sent: 15 January 2019 10:26
>>>>>>> To: ceph-users@lists.ceph.com
>>>>>>> Cc: n.fahldi...@profihost.ag
>>>>>>> Subject: Re: [ceph-users] slow requests and high i/o / read rate on 
>>>>>>> bluestore osds after upgrade 12.2.8 -> 12.2.10
>>>>>>>
>>>>>>> Hello list,
>>>>>>>
>>>>>>> i also tested current upstream/luminous branch and it happens as well. A
>>>>>>> clean install works fine. It only happens on upgraded bluestore osds.
>>>>>>>
>>>>>>> Greets,
>>>>>>> Stefan
>>>>>>>
>>>>>>> Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:
>>>>>>>> while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm 
>>>>>>> experience
>>>>>>>> issues with bluestore osds - so i canceled the upgrade and all 
>>>>>>> bluestore
>>>>>>>> osds are stopped now.
>>>>>>>>
>>>>>>>> After starting a bluestore osd i'm seeing a lot of slow requests 
>>>>>>> caused
>>>>>>>> by very high read rates.
>>>>>>>>
>>>>>>>>
>>>>>>>> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
>>>>>>>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>>>>>>>> sda  45,00   187,00  767,00   39,00 482040,00  8660,00
>>>>>>>> 1217,6258,16   74,60   73,85   89,23   1,24 100,00
>>>>>>>>
>>>>>>>> it reads permanently with 500MB/s from the disk and can't service 
>>>>>>> client
>>>>>>>> requests. Overall client read rate is at 10.9MiB/s rd
>>>>>>>>
>>>>>>>> I can't reproduce this with 12.2.8. Is this a known bug / regression?
>>>>>>>>
>>>>>>>> Greets,
>>>>>>>> Stefan
>>>>>>>>
>>>>>>> ___
>>>>>>> ceph-users mailing list
>>>>>>> ceph-users@lists.ceph.com
>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>>
>>>>>>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Stefan Priebe - Profihost AG]

ts ml 3 upset size 3 up 3
>>> 2019-01-16 15:19:23.503231 7fecb97ff700  0 osd.33 pg_epoch: 1318479
>>> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
>>> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
>>> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
>>> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
>>> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
>>> overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
>>> mbc={255={(2+0)=183,(3+0)=3}}] _update_calc_stats ml 183 upset size 3 up 2
>>>
>>> Greets,
>>> Stefan
>>> Am 16.01.19 um 09:12 schrieb Stefan Priebe - Profihost AG:
>>>> Hi,
>>>>
>>>> no ok it was not. Bug still present. It was only working because the
>>>> osdmap was so far away that it has started backfill instead of recovery.
>>>>
>>>> So it happens only in the recovery case.
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>>> Am 15.01.19 um 16:02 schrieb Stefan Priebe - Profihost AG:
>>>>>
>>>>> Am 15.01.19 um 12:45 schrieb Marc Roos:
>>>>>>  
>>>>>> I upgraded this weekend from 12.2.8 to 12.2.10 without such issues 
>>>>>> (osd's are idle)
>>>>>
>>>>>
>>>>> it turns out this was a kernel bug. Updating to a newer kernel - has
>>>>> solved this issue.
>>>>>
>>>>> Greets,
>>>>> Stefan
>>>>>
>>>>>
>>>>>> -Original Message-
>>>>>> From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] 
>>>>>> Sent: 15 January 2019 10:26
>>>>>> To: ceph-users@lists.ceph.com
>>>>>> Cc: n.fahldi...@profihost.ag
>>>>>> Subject: Re: [ceph-users] slow requests and high i/o / read rate on 
>>>>>> bluestore osds after upgrade 12.2.8 -> 12.2.10
>>>>>>
>>>>>> Hello list,
>>>>>>
>>>>>> i also tested current upstream/luminous branch and it happens as well. A
>>>>>> clean install works fine. It only happens on upgraded bluestore osds.
>>>>>>
>>>>>> Greets,
>>>>>> Stefan
>>>>>>
>>>>>> Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:
>>>>>>> while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm 
>>>>>> experience
>>>>>>> issues with bluestore osds - so i canceled the upgrade and all 
>>>>>> bluestore
>>>>>>> osds are stopped now.
>>>>>>>
>>>>>>> After starting a bluestore osd i'm seeing a lot of slow requests 
>>>>>> caused
>>>>>>> by very high read rates.
>>>>>>>
>>>>>>>
>>>>>>> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
>>>>>>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>>>>>>> sda  45,00   187,00  767,00   39,00 482040,00  8660,00
>>>>>>> 1217,6258,16   74,60   73,85   89,23   1,24 100,00
>>>>>>>
>>>>>>> it reads permanently with 500MB/s from the disk and can't service 
>>>>>> client
>>>>>>> requests. Overall client read rate is at 10.9MiB/s rd
>>>>>>>
>>>>>>> I can't reproduce this with 12.2.8. Is this a known bug / regression?
>>>>>>>
>>>>>>> Greets,
>>>>>>> Stefan
>>>>>>>
>>>>>> ___
>>>>>> ceph-users mailing list
>>>>>> ceph-users@lists.ceph.com
>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>
>>>>>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Mark Nelson]

Hi Stefan,


I'm taking a stab at reproducing this in-house.  Any details you can 
give me that might help would be much appreciated.  I'll let you know 
what I find.



Thanks,

Mark


On 1/16/19 1:56 PM, Stefan Priebe - Profihost AG wrote:

i reverted the whole cluster back to 12.2.8 - recovery speed also
dropped from 300-400MB/s to 20MB/s on 12.2.10. So something is really
broken.

Greets,
Stefan
Am 16.01.19 um 16:00 schrieb Stefan Priebe - Profihost AG:

This is not the case with 12.2.8 - it happens with 12.2.9 as well. After
boot all pgs are instantly active - not inactive pgs at least not
noticable in ceph -s.

With 12.2.9 or 12.2.10 or eben current upstream/luminous it takes
minutes until all pgs are active again.

Greets,
Stefan
Am 16.01.19 um 15:22 schrieb Stefan Priebe - Profihost AG:

Hello,

while digging into this further i saw that it takes ages until all pgs
are active. After starting the OSD 3% of all pgs are inactive and it
takes minutes after they're active.

The log of the OSD is full of:


2019-01-16 15:19:13.568527 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=184 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=185,(3+0)=2}}] _update_calc_stats ml 185 upset size 3 up 2
2019-01-16 15:19:13.568637 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=184 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=185,(3+0)=2}}] _update_calc_stats ml 2 upset size 3 up 3
2019-01-16 15:19:15.909327 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 184 upset size 3 up 2
2019-01-16 15:19:15.909446 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 3 upset size 3 up 3
2019-01-16 15:19:23.503231 7fecb97ff700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=183,(3+0)=3}}] _update_calc_stats ml 183 upset size 3 up 2

Greets,
Stefan
Am 16.01.19 um 09:12 schrieb Stefan Priebe - Profihost AG:

Hi,

no ok it was not. Bug still present. It was only working because the
osdmap was so far away that it has started backfill instead of recovery.

So it happens only in the recovery case.

Greets,
Stefan

Am 15.01.19 um 16:02 schrieb Stefan Priebe - Profihost AG:

Am 15.01.19 um 12:45 schrieb Marc Roos:
  
I upgraded this weekend from 12.2.8 to 12.2.10 without such issues

(osd's are idle)


it turns out this was a kernel bug. Updating to a newer kernel - has
solved this issue.

Greets,
Stefan



-Original Message-----
From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag]
Sent: 15 January 2019 10:26
To: ceph-users@lists.ceph.com
Cc: n.fahldi...@profihost.ag
Subject: Re: [ceph-users] slow requests and high i/o / read rate on
bluestore osds after upgrade 12.2.8 -> 12.2.10

Hello list,

i also tested current upstream/luminous branch and it happens as well. A
clean install works fine. It only happens on upgraded bluestore osds.

Greets,
Stefan

Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:

while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Stefan Priebe - Profihost AG]

i reverted the whole cluster back to 12.2.8 - recovery speed also
dropped from 300-400MB/s to 20MB/s on 12.2.10. So something is really
broken.

Greets,
Stefan
Am 16.01.19 um 16:00 schrieb Stefan Priebe - Profihost AG:
> This is not the case with 12.2.8 - it happens with 12.2.9 as well. After
> boot all pgs are instantly active - not inactive pgs at least not
> noticable in ceph -s.
> 
> With 12.2.9 or 12.2.10 or eben current upstream/luminous it takes
> minutes until all pgs are active again.
> 
> Greets,
> Stefan
> Am 16.01.19 um 15:22 schrieb Stefan Priebe - Profihost AG:
>> Hello,
>>
>> while digging into this further i saw that it takes ages until all pgs
>> are active. After starting the OSD 3% of all pgs are inactive and it
>> takes minutes after they're active.
>>
>> The log of the OSD is full of:
>>
>>
>> 2019-01-16 15:19:13.568527 7fecbf7da700  0 osd.33 pg_epoch: 1318479
>> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
>> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
>> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
>> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
>> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
>> overing+degraded m=184 snaptrimq=[ec1a0~1,ec808~1]
>> mbc={255={(2+0)=185,(3+0)=2}}] _update_calc_stats ml 185 upset size 3 up 2
>> 2019-01-16 15:19:13.568637 7fecbf7da700  0 osd.33 pg_epoch: 1318479
>> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
>> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
>> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
>> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
>> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
>> overing+degraded m=184 snaptrimq=[ec1a0~1,ec808~1]
>> mbc={255={(2+0)=185,(3+0)=2}}] _update_calc_stats ml 2 upset size 3 up 3
>> 2019-01-16 15:19:15.909327 7fecbf7da700  0 osd.33 pg_epoch: 1318479
>> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
>> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
>> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
>> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
>> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
>> overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
>> mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 184 upset size 3 up 2
>> 2019-01-16 15:19:15.909446 7fecbf7da700  0 osd.33 pg_epoch: 1318479
>> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
>> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
>> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
>> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
>> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
>> overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
>> mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 3 upset size 3 up 3
>> 2019-01-16 15:19:23.503231 7fecb97ff700  0 osd.33 pg_epoch: 1318479
>> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
>> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
>> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
>> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
>> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
>> overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
>> mbc={255={(2+0)=183,(3+0)=3}}] _update_calc_stats ml 183 upset size 3 up 2
>>
>> Greets,
>> Stefan
>> Am 16.01.19 um 09:12 schrieb Stefan Priebe - Profihost AG:
>>> Hi,
>>>
>>> no ok it was not. Bug still present. It was only working because the
>>> osdmap was so far away that it has started backfill instead of recovery.
>>>
>>> So it happens only in the recovery case.
>>>
>>> Greets,
>>> Stefan
>>>
>>> Am 15.01.19 um 16:02 schrieb Stefan Priebe - Profihost AG:
>>>>
>>>> Am 15.01.19 um 12:45 schrieb Marc Roos:
>>>>>  
>>>>> I upgraded this weekend from 12.2.8 to 12.2.10 without such issues 
>>>>> (osd's are idle)
>>>>
>>>>
>>>> it turns out this was a kernel bug. Updating to a newer kernel - has
>>>> solved this issue.
>>>>
>>>> Greets,
>&g

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Mark Nelson]

Hi Stefan,


12.2.9 included the pg hard limit patches and the osd_memory_autotuning 
patches.  While at first I was wondering if this was autotuning, it 
sounds like it may be more related to the pg hard limit.  I'm not 
terribly familiar with those patches though so some of the other members 
from the core team may need to take a look.



Mark


On 1/16/19 9:00 AM, Stefan Priebe - Profihost AG wrote:

This is not the case with 12.2.8 - it happens with 12.2.9 as well. After
boot all pgs are instantly active - not inactive pgs at least not
noticable in ceph -s.

With 12.2.9 or 12.2.10 or eben current upstream/luminous it takes
minutes until all pgs are active again.

Greets,
Stefan
Am 16.01.19 um 15:22 schrieb Stefan Priebe - Profihost AG:

Hello,

while digging into this further i saw that it takes ages until all pgs
are active. After starting the OSD 3% of all pgs are inactive and it
takes minutes after they're active.

The log of the OSD is full of:


2019-01-16 15:19:13.568527 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=184 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=185,(3+0)=2}}] _update_calc_stats ml 185 upset size 3 up 2
2019-01-16 15:19:13.568637 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=184 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=185,(3+0)=2}}] _update_calc_stats ml 2 upset size 3 up 3
2019-01-16 15:19:15.909327 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 184 upset size 3 up 2
2019-01-16 15:19:15.909446 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 3 upset size 3 up 3
2019-01-16 15:19:23.503231 7fecb97ff700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=183,(3+0)=3}}] _update_calc_stats ml 183 upset size 3 up 2

Greets,
Stefan
Am 16.01.19 um 09:12 schrieb Stefan Priebe - Profihost AG:

Hi,

no ok it was not. Bug still present. It was only working because the
osdmap was so far away that it has started backfill instead of recovery.

So it happens only in the recovery case.

Greets,
Stefan

Am 15.01.19 um 16:02 schrieb Stefan Priebe - Profihost AG:

Am 15.01.19 um 12:45 schrieb Marc Roos:
  
I upgraded this weekend from 12.2.8 to 12.2.10 without such issues

(osd's are idle)


it turns out this was a kernel bug. Updating to a newer kernel - has
solved this issue.

Greets,
Stefan



-Original Message-
From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag]
Sent: 15 January 2019 10:26
To: ceph-users@lists.ceph.com
Cc: n.fahldi...@profihost.ag
Subject: Re: [ceph-users] slow requests and high i/o / read rate on
bluestore osds after upgrade 12.2.8 -> 12.2.10

Hello list,

i also tested current upstream/luminous branch and it happens as well. A
clean install works fine. It only happens on upgraded bluestore osds.

Greets,
Stefan

Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:

while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm

experience

issues with bluestore osds - so i canceled the u

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Stefan Priebe - Profihost AG]

This is not the case with 12.2.8 - it happens with 12.2.9 as well. After
boot all pgs are instantly active - not inactive pgs at least not
noticable in ceph -s.

With 12.2.9 or 12.2.10 or eben current upstream/luminous it takes
minutes until all pgs are active again.

Greets,
Stefan
Am 16.01.19 um 15:22 schrieb Stefan Priebe - Profihost AG:
> Hello,
> 
> while digging into this further i saw that it takes ages until all pgs
> are active. After starting the OSD 3% of all pgs are inactive and it
> takes minutes after they're active.
> 
> The log of the OSD is full of:
> 
> 
> 2019-01-16 15:19:13.568527 7fecbf7da700  0 osd.33 pg_epoch: 1318479
> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
> overing+degraded m=184 snaptrimq=[ec1a0~1,ec808~1]
> mbc={255={(2+0)=185,(3+0)=2}}] _update_calc_stats ml 185 upset size 3 up 2
> 2019-01-16 15:19:13.568637 7fecbf7da700  0 osd.33 pg_epoch: 1318479
> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
> overing+degraded m=184 snaptrimq=[ec1a0~1,ec808~1]
> mbc={255={(2+0)=185,(3+0)=2}}] _update_calc_stats ml 2 upset size 3 up 3
> 2019-01-16 15:19:15.909327 7fecbf7da700  0 osd.33 pg_epoch: 1318479
> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
> overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
> mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 184 upset size 3 up 2
> 2019-01-16 15:19:15.909446 7fecbf7da700  0 osd.33 pg_epoch: 1318479
> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
> overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
> mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 3 upset size 3 up 3
> 2019-01-16 15:19:23.503231 7fecb97ff700  0 osd.33 pg_epoch: 1318479
> pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
> 21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
> ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
> 8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
> rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
> overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
> mbc={255={(2+0)=183,(3+0)=3}}] _update_calc_stats ml 183 upset size 3 up 2
> 
> Greets,
> Stefan
> Am 16.01.19 um 09:12 schrieb Stefan Priebe - Profihost AG:
>> Hi,
>>
>> no ok it was not. Bug still present. It was only working because the
>> osdmap was so far away that it has started backfill instead of recovery.
>>
>> So it happens only in the recovery case.
>>
>> Greets,
>> Stefan
>>
>> Am 15.01.19 um 16:02 schrieb Stefan Priebe - Profihost AG:
>>>
>>> Am 15.01.19 um 12:45 schrieb Marc Roos:
>>>>  
>>>> I upgraded this weekend from 12.2.8 to 12.2.10 without such issues 
>>>> (osd's are idle)
>>>
>>>
>>> it turns out this was a kernel bug. Updating to a newer kernel - has
>>> solved this issue.
>>>
>>> Greets,
>>> Stefan
>>>
>>>
>>>> -Original Message-
>>>> From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] 
>>>> Sent: 15 January 2019 10:26
>>>> To: ceph-users@lists.ceph.com
>>>> Cc: n.fahldi...@profihost.ag
>>>> Subject: Re: [ceph-users] slow requests and high i/o / read rate on 
>>>> bluestore osds after upgrade 12.2.8 -> 12.2.10
>>>>
>>>> Hello list,
>>>>
>>>> i also tested current upstream/lu

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Stefan Priebe - Profihost AG]

Hello,

while digging into this further i saw that it takes ages until all pgs
are active. After starting the OSD 3% of all pgs are inactive and it
takes minutes after they're active.

The log of the OSD is full of:


2019-01-16 15:19:13.568527 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=184 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=185,(3+0)=2}}] _update_calc_stats ml 185 upset size 3 up 2
2019-01-16 15:19:13.568637 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=184 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=185,(3+0)=2}}] _update_calc_stats ml 2 upset size 3 up 3
2019-01-16 15:19:15.909327 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 184 upset size 3 up 2
2019-01-16 15:19:15.909446 7fecbf7da700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=184,(3+0)=3}}] _update_calc_stats ml 3 upset size 3 up 3
2019-01-16 15:19:23.503231 7fecb97ff700  0 osd.33 pg_epoch: 1318479
pg[5.563( v 1318474'61584855 lc 1318356'61576253 (1318287'615747
21,1318474'61584855] local-lis/les=1318472/1318473 n=1912
ec=133405/133405 lis/c 1318472/1278145 les/c/f 1318473/1278148/1211861 131
8472/1318472/1318472) [33,3,22] r=0 lpr=1318472 pi=[1278145,1318472)/1
rops=4 crt=1318474'61584855 mlcod 1318356'61576253 active+rec
overing+degraded m=183 snaptrimq=[ec1a0~1,ec808~1]
mbc={255={(2+0)=183,(3+0)=3}}] _update_calc_stats ml 183 upset size 3 up 2

Greets,
Stefan
Am 16.01.19 um 09:12 schrieb Stefan Priebe - Profihost AG:
> Hi,
> 
> no ok it was not. Bug still present. It was only working because the
> osdmap was so far away that it has started backfill instead of recovery.
> 
> So it happens only in the recovery case.
> 
> Greets,
> Stefan
> 
> Am 15.01.19 um 16:02 schrieb Stefan Priebe - Profihost AG:
>>
>> Am 15.01.19 um 12:45 schrieb Marc Roos:
>>>  
>>> I upgraded this weekend from 12.2.8 to 12.2.10 without such issues 
>>> (osd's are idle)
>>
>>
>> it turns out this was a kernel bug. Updating to a newer kernel - has
>> solved this issue.
>>
>> Greets,
>> Stefan
>>
>>
>>> -Original Message-
>>> From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] 
>>> Sent: 15 January 2019 10:26
>>> To: ceph-users@lists.ceph.com
>>> Cc: n.fahldi...@profihost.ag
>>> Subject: Re: [ceph-users] slow requests and high i/o / read rate on 
>>> bluestore osds after upgrade 12.2.8 -> 12.2.10
>>>
>>> Hello list,
>>>
>>> i also tested current upstream/luminous branch and it happens as well. A
>>> clean install works fine. It only happens on upgraded bluestore osds.
>>>
>>> Greets,
>>> Stefan
>>>
>>> Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:
>>>> while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm 
>>> experience
>>>> issues with bluestore osds - so i canceled the upgrade and all 
>>> bluestore
>>>> osds are stopped now.
>>>>
>>>> After starting a bluestore osd i'm seeing a lot of slow requests 
>>> caused
>>>> by very high read rates.
>>>>
>>>>
>>>> Device:  

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Stefan Priebe - Profihost AG]

Hi,

no ok it was not. Bug still present. It was only working because the
osdmap was so far away that it has started backfill instead of recovery.

So it happens only in the recovery case.

Greets,
Stefan

Am 15.01.19 um 16:02 schrieb Stefan Priebe - Profihost AG:
> 
> Am 15.01.19 um 12:45 schrieb Marc Roos:
>>  
>> I upgraded this weekend from 12.2.8 to 12.2.10 without such issues 
>> (osd's are idle)
> 
> 
> it turns out this was a kernel bug. Updating to a newer kernel - has
> solved this issue.
> 
> Greets,
> Stefan
> 
> 
>> -Original Message-
>> From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] 
>> Sent: 15 January 2019 10:26
>> To: ceph-users@lists.ceph.com
>> Cc: n.fahldi...@profihost.ag
>> Subject: Re: [ceph-users] slow requests and high i/o / read rate on 
>> bluestore osds after upgrade 12.2.8 -> 12.2.10
>>
>> Hello list,
>>
>> i also tested current upstream/luminous branch and it happens as well. A
>> clean install works fine. It only happens on upgraded bluestore osds.
>>
>> Greets,
>> Stefan
>>
>> Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:
>>> while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm 
>> experience
>>> issues with bluestore osds - so i canceled the upgrade and all 
>> bluestore
>>> osds are stopped now.
>>>
>>> After starting a bluestore osd i'm seeing a lot of slow requests 
>> caused
>>> by very high read rates.
>>>
>>>
>>> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
>>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>>> sda  45,00   187,00  767,00   39,00 482040,00  8660,00
>>> 1217,6258,16   74,60   73,85   89,23   1,24 100,00
>>>
>>> it reads permanently with 500MB/s from the disk and can't service 
>> client
>>> requests. Overall client read rate is at 10.9MiB/s rd
>>>
>>> I can't reproduce this with 12.2.8. Is this a known bug / regression?
>>>
>>> Greets,
>>> Stefan
>>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Mark Nelson]

On 1/15/19 9:02 AM, Stefan Priebe - Profihost AG wrote:

Am 15.01.19 um 12:45 schrieb Marc Roos:
  
I upgraded this weekend from 12.2.8 to 12.2.10 without such issues

(osd's are idle)


it turns out this was a kernel bug. Updating to a newer kernel - has
solved this issue.

Greets,
Stefan



Hi Stefan, can you tell me what kernel you were on and what hardware was 
involved?  I want to make sure that it's recorded for the community in 
case others run into the same issue.



Thanks,

Mark






-Original Message-
From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag]
Sent: 15 January 2019 10:26
To: ceph-users@lists.ceph.com
Cc: n.fahldi...@profihost.ag
Subject: Re: [ceph-users] slow requests and high i/o / read rate on
bluestore osds after upgrade 12.2.8 -> 12.2.10

Hello list,

i also tested current upstream/luminous branch and it happens as well. A
clean install works fine. It only happens on upgraded bluestore osds.

Greets,
Stefan

Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:

while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm

experience

issues with bluestore osds - so i canceled the upgrade and all

bluestore

osds are stopped now.

After starting a bluestore osd i'm seeing a lot of slow requests

caused

by very high read rates.


Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda  45,00   187,00  767,00   39,00 482040,00  8660,00
1217,6258,16   74,60   73,85   89,23   1,24 100,00

it reads permanently with 500MB/s from the disk and can't service

client

requests. Overall client read rate is at 10.9MiB/s rd

I can't reproduce this with 12.2.8. Is this a known bug / regression?

Greets,
Stefan


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Stefan Priebe - Profihost AG]

Am 15.01.19 um 12:45 schrieb Marc Roos:
>  
> I upgraded this weekend from 12.2.8 to 12.2.10 without such issues 
> (osd's are idle)


it turns out this was a kernel bug. Updating to a newer kernel - has
solved this issue.

Greets,
Stefan


> -Original Message-
> From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] 
> Sent: 15 January 2019 10:26
> To: ceph-users@lists.ceph.com
> Cc: n.fahldi...@profihost.ag
> Subject: Re: [ceph-users] slow requests and high i/o / read rate on 
> bluestore osds after upgrade 12.2.8 -> 12.2.10
> 
> Hello list,
> 
> i also tested current upstream/luminous branch and it happens as well. A
> clean install works fine. It only happens on upgraded bluestore osds.
> 
> Greets,
> Stefan
> 
> Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:
>> while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm 
> experience
>> issues with bluestore osds - so i canceled the upgrade and all 
> bluestore
>> osds are stopped now.
>>
>> After starting a bluestore osd i'm seeing a lot of slow requests 
> caused
>> by very high read rates.
>>
>>
>> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
>> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
>> sda  45,00   187,00  767,00   39,00 482040,00  8660,00
>> 1217,6258,16   74,60   73,85   89,23   1,24 100,00
>>
>> it reads permanently with 500MB/s from the disk and can't service 
> client
>> requests. Overall client read rate is at 10.9MiB/s rd
>>
>> I can't reproduce this with 12.2.8. Is this a known bug / regression?
>>
>> Greets,
>> Stefan
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Marc Roos]

 
I upgraded this weekend from 12.2.8 to 12.2.10 without such issues 
(osd's are idle)




-Original Message-
From: Stefan Priebe - Profihost AG [mailto:s.pri...@profihost.ag] 
Sent: 15 January 2019 10:26
To: ceph-users@lists.ceph.com
Cc: n.fahldi...@profihost.ag
Subject: Re: [ceph-users] slow requests and high i/o / read rate on 
bluestore osds after upgrade 12.2.8 -> 12.2.10

Hello list,

i also tested current upstream/luminous branch and it happens as well. A
clean install works fine. It only happens on upgraded bluestore osds.

Greets,
Stefan

Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:
> while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm 
experience
> issues with bluestore osds - so i canceled the upgrade and all 
bluestore
> osds are stopped now.
> 
> After starting a bluestore osd i'm seeing a lot of slow requests 
caused
> by very high read rates.
> 
> 
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> sda  45,00   187,00  767,00   39,00 482040,00  8660,00
> 1217,6258,16   74,60   73,85   89,23   1,24 100,00
> 
> it reads permanently with 500MB/s from the disk and can't service 
client
> requests. Overall client read rate is at 10.9MiB/s rd
> 
> I can't reproduce this with 12.2.8. Is this a known bug / regression?
> 
> Greets,
> Stefan
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Stefan Priebe - Profihost AG]

Hello list,

i also tested current upstream/luminous branch and it happens as well. A
clean install works fine. It only happens on upgraded bluestore osds.

Greets,
Stefan

Am 14.01.19 um 20:35 schrieb Stefan Priebe - Profihost AG:
> while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm experience
> issues with bluestore osds - so i canceled the upgrade and all bluestore
> osds are stopped now.
> 
> After starting a bluestore osd i'm seeing a lot of slow requests caused
> by very high read rates.
> 
> 
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> sda  45,00   187,00  767,00   39,00 482040,00  8660,00
> 1217,6258,16   74,60   73,85   89,23   1,24 100,00
> 
> it reads permanently with 500MB/s from the disk and can't service client
> requests. Overall client read rate is at 10.9MiB/s rd
> 
> I can't reproduce this with 12.2.8. Is this a known bug / regression?
> 
> Greets,
> Stefan
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Mark Nelson]

Hi Stefan,


Any idea if the reads are constant or bursty?  One cause of heavy reads 
is when rocksdb is compacting and has to read SST files from disk.  It's 
also possible you could see heavy read traffic during writes if data has 
to be read from SST files rather than cache. It's possible this could be 
related to the osd_memory_autotune feature.  It will try to keep OSD 
memory usage within a certain footprint (4GB by default) which 
supercedes the bluestore cache size (it automatically sets the cache 
size based on the osd_memory_target).



To see what's happening during compaction, you can run this script 
against one of your bluestore OSD logs:


https://github.com/ceph/cbt/blob/master/tools/ceph_rocksdb_log_parser.py


Mark

On 1/14/19 1:35 PM, Stefan Priebe - Profihost AG wrote:

Hi,

while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm experience
issues with bluestore osds - so i canceled the upgrade and all bluestore
osds are stopped now.

After starting a bluestore osd i'm seeing a lot of slow requests caused
by very high read rates.


Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda  45,00   187,00  767,00   39,00 482040,00  8660,00
1217,6258,16   74,60   73,85   89,23   1,24 100,00

it reads permanently with 500MB/s from the disk and can't service client
requests. Overall client read rate is at 10.9MiB/s rd

I can't reproduce this with 12.2.8. Is this a known bug / regression?

Greets,
Stefan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Stefan Priebe - Profihost AG]

Hi Paul,

Am 14.01.19 um 21:39 schrieb Paul Emmerich:
> What's the output of "ceph daemon osd. status" on one of the OSDs
> while it's starting?


{
"cluster_fsid": "b338193d-39e0-40e9-baba-4965ef3868a3",
"osd_fsid": "d95d0e3b-7441-4ab0-869c-fe0551d3bd52",
"whoami": 2,
"state": "active",
"oldest_map": 1313325,
"newest_map": 1314026,
"num_pgs": 212
}


> Is the OSD crashing and being restarted all the time?

no it's just running

> Anything weird in the log files

no

> Was there recovery or backfill during the upgrade?

No - it was clean and healthy.

Greets,
Stefan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Paul Emmerich]

What's the output of "ceph daemon osd. status" on one of the OSDs
while it's starting?

Is the OSD crashing and being restarted all the time? Anything weird
in the log files? Was there recovery or backfill during the upgrade?

Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Mon, Jan 14, 2019 at 8:35 PM Stefan Priebe - Profihost AG
 wrote:
>
> Hi,
>
> while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm experience
> issues with bluestore osds - so i canceled the upgrade and all bluestore
> osds are stopped now.
>
> After starting a bluestore osd i'm seeing a lot of slow requests caused
> by very high read rates.
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
> avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
> sda  45,00   187,00  767,00   39,00 482040,00  8660,00
> 1217,6258,16   74,60   73,85   89,23   1,24 100,00
>
> it reads permanently with 500MB/s from the disk and can't service client
> requests. Overall client read rate is at 10.9MiB/s rd
>
> I can't reproduce this with 12.2.8. Is this a known bug / regression?
>
> Greets,
> Stefan
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] slow requests and high i/o / read rate on bluestore osds after upgrade 12.2.8 -> 12.2.10

[Stefan Priebe - Profihost AG]

Hi,

while trying to upgrade a cluster from 12.2.8 to 12.2.10 i'm experience
issues with bluestore osds - so i canceled the upgrade and all bluestore
osds are stopped now.

After starting a bluestore osd i'm seeing a lot of slow requests caused
by very high read rates.


Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s
avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda  45,00   187,00  767,00   39,00 482040,00  8660,00
1217,6258,16   74,60   73,85   89,23   1,24 100,00

it reads permanently with 500MB/s from the disk and can't service client
requests. Overall client read rate is at 10.9MiB/s rd

I can't reproduce this with 12.2.8. Is this a known bug / regression?

Greets,
Stefan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] slow requests and degraded cluster, but not really ?

[Ben Morrice]

Hello all,

We have an issue with our ceph cluster where 'ceph -s' shows that 
several requests are blocked, however querying further with 'ceph health 
detail' indicates that the PGs affected are either active+clean or do 
not currently exist.
OSD 32 appears to be working fine, and the cluster is performing as 
expected with no clients seemingly affected.


Note - we had just upgraded to Luminous - and despite having "mon max pg 
per osd = 400" set in ceph.conf, we still have the message "too many PGs 
per OSD (278 > max 200)"


In order to improve the situation above, I removed several pools that 
were not used anymore. I assume the PGs that ceph cannot find now are 
related to this pool deletion.


Does anyone have any ideas on how to get out of this state?

Details below - and full 'ceph health detail' attached to this email.

Kind regards,

Ben Morrice

[root@ceph03 ~]# ceph -s
  cluster:
    id: 6c21c4ba-9c4d-46ef-93a3-441b8055cdc6
    health: HEALTH_WARN
    Degraded data redundancy: 443765/14311983 objects degraded 
(3.101%), 162 pgs degraded, 241 pgs undersized

    75 slow requests are blocked > 32 sec. Implicated osds 32
    too many PGs per OSD (278 > max 200)

  services:
    mon: 5 daemons, quorum bbpocn01,bbpocn02,bbpocn03,bbpocn04,bbpocn07
    mgr: bbpocn03(active, starting)
    osd: 36 osds: 36 up, 36 in
    rgw: 1 daemon active

  data:
    pools:   24 pools, 3440 pgs
    objects: 4.77M objects, 7.69TiB
    usage:   23.1TiB used, 104TiB / 127TiB avail
    pgs: 443765/14311983 objects degraded (3.101%)
 3107 active+clean
 170  active+undersized
 109  active+undersized+degraded
 43   active+recovery_wait+degraded
 10   active+recovering+degraded
 1    active+recovery_wait

[root@ceph03 ~]# for i in `ceph health detail |grep stuck | awk '{print 
$2}'`; do echo -n "$i: " ; ceph pg $i query -f plain | cut -d: -f2 | cut 
-d\" -f2; done

150.270: active+clean
150.2a0: active+clean
150.2b6: active+clean
150.2c2: active+clean
150.2cc: active+clean
150.2d5: active+clean
150.2d6: active+clean
150.2e1: active+clean
150.2ef: active+clean
150.2f5: active+clean
150.2f7: active+clean
150.2fc: active+clean
150.315: active+clean
150.318: active+clean
150.31a: active+clean
150.320: active+clean
150.326: active+clean
150.36e: active+clean
150.380: active+clean
150.389: active+clean
150.3a4: active+clean
150.3ad: active+clean
150.3b4: active+clean
150.3bb: active+clean
150.3ce: active+clean
150.3d0: active+clean
150.3d8: active+clean
150.3e0: active+clean
150.3f6: active+clean
165.24c: Error ENOENT: problem getting command descriptions from pg.165.24c
165.28f: Error ENOENT: problem getting command descriptions from pg.165.28f
165.2b3: Error ENOENT: problem getting command descriptions from pg.165.2b3
165.2b4: Error ENOENT: problem getting command descriptions from pg.165.2b4
165.2d6: Error ENOENT: problem getting command descriptions from pg.165.2d6
165.2f4: Error ENOENT: problem getting command descriptions from pg.165.2f4
165.2fd: Error ENOENT: problem getting command descriptions from pg.165.2fd
165.30f: Error ENOENT: problem getting command descriptions from pg.165.30f
165.322: Error ENOENT: problem getting command descriptions from pg.165.322
165.325: Error ENOENT: problem getting command descriptions from pg.165.325
165.334: Error ENOENT: problem getting command descriptions from pg.165.334
165.36e: Error ENOENT: problem getting command descriptions from pg.165.36e
165.37c: Error ENOENT: problem getting command descriptions from pg.165.37c
165.382: Error ENOENT: problem getting command descriptions from pg.165.382
165.387: Error ENOENT: problem getting command descriptions from pg.165.387
165.3af: Error ENOENT: problem getting command descriptions from pg.165.3af
165.3da: Error ENOENT: problem getting command descriptions from pg.165.3da
165.3e0: Error ENOENT: problem getting command descriptions from pg.165.3e0
165.3e2: Error ENOENT: problem getting command descriptions from pg.165.3e2
165.3e9: Error ENOENT: problem getting command descriptions from pg.165.3e9
165.3fb: Error ENOENT: problem getting command descriptions from pg.165.3fb

[root@ceph03 ~]# ceph pg 165.24c query
Error ENOENT: problem getting command descriptions from pg.165.24c
[root@ceph03 ~]# ceph pg 165.24c delete
Error ENOENT: problem getting command descriptions from pg.165.24c

--
Kind regards,

Ben Morrice

__
Ben Morrice | e: ben.morr...@epfl.ch | t: +41-21-693-9670
EPFL / BBP
Biotech Campus
Chemin des Mines 9
1202 Geneva
Switzerland

HEALTH_WARN Degraded data redundancy: 443765/14311983 objects degraded 
(3.101%), 162 pgs degraded, 241 pgs undersized; 75 slow requests are blocked > 
32 sec. Implicated osds 32; too many PGs per OSD (278 > max 200)
pg 150.270 is stuck undersized for 1871.987162, current state 
active+undersized, last acting [17,30]
pg 150.2a0 is s

Re: [ceph-users] Slow requests blocked. No rebalancing

[Jaime Ibar]

Hi all,

after increasing mon_max_pg_per_osd number ceph starts rebalancing as usual.

However, the slow requests warnings are still there, even after setting

primary-affinity to 0 beforehand.

By the other hand, if I destroy the osd, ceph will start rebalancing unless

noout flag is set, am I right?

Thanks

Jaime


On 20/09/18 14:25, Paul Emmerich wrote:

You can prevent creation of the PGs on the old filestore OSDs (which
seems to be the culprit here) during replacement by replacing the
disks the hard way:

* ceph osd destroy osd.X
* re-create with bluestore under the same id (ceph volume ... --osd-id X)

it will then just backfill onto the same disk without moving any PG.

Keep in mind that this means that you are running with one missing
copy during the recovery, so that's not the recommended way to do
that.

Paul


2018-09-20 13:51 GMT+02:00 Eugen Block :

Hi,

to reduce impact on clients during migration I would set the OSD's
primary-affinity to 0 beforehand. This should prevent the slow requests, at
least this setting has helped us a lot with problematic OSDs.

Regards
Eugen


Zitat von Jaime Ibar :



Hi all,

we recently upgrade from Jewel 10.2.10 to Luminous 12.2.7, now we're
trying to migrate the

osd's to Bluestore following this document[0], however when I mark the osd
as out,

I'm getting warnings similar to these ones

2018-09-20 09:32:46.079630 mon.dri-ceph01 [WRN] Health check failed: 2
slow requests are blocked > 32 sec. Implicated osds 16,28 (REQUEST_SLOW)
2018-09-20 09:32:52.841123 mon.dri-ceph01 [WRN] Health check update: 7
slow requests are blocked > 32 sec. Implicated osds 10,16,28,32,59
(REQUEST_SLOW)
2018-09-20 09:32:57.842230 mon.dri-ceph01 [WRN] Health check update: 15
slow requests are blocked > 32 sec. Implicated osds 10,16,28,31,32,59,78,80
(REQUEST_SLOW)

2018-09-20 09:32:58.851142 mon.dri-ceph01 [WRN] Health check update:
244944/40100780 objects misplaced (0.611%) (OBJECT_MISPLACED)
2018-09-20 09:32:58.851160 mon.dri-ceph01 [WRN] Health check update: 249
PGs pending on creation (PENDING_CREATING_PGS)

which prevent ceph start rebalancing and the vm's running on ceph start
hanging and we have to mark the osd back in.

I tried to reweight the osd to 0.90 in order to minimize the impact on the
cluster but the warnings are the same.

I tried to increased these settings to

mds cache memory limit = 2147483648
rocksdb cache size = 2147483648

but with no luck, same warnings.

We also have cephfs for storing files from different projects(no directory
fragmentation enabled).

The problem here is that if one osd dies, all the services will be blocked
as ceph won't be able to

start rebalancing.

The cluster is

- 3 mons

- 3 mds(running on the same hosts as the mons). 2 mds active and 1 standby

- 3 mgr(running on the same hosts as the mons)

- 6 servers, 12 osd's each.

- 1GB private network


Does anyone know how to fix or where the problem could be?

Thanks a lot in advance.

Jaime


[0]
http://docs.ceph.com/docs/luminous/rados/operations/bluestore-migration/

--

Jaime Ibar
High Performance & Research Computing, IS Services
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
http://www.tchpc.tcd.ie/  |ja...@tchpc.tcd.ie
Tel: +353-1-896-3725




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





--

Jaime Ibar
High Performance & Research Computing, IS Services
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
http://www.tchpc.tcd.ie/ | ja...@tchpc.tcd.ie
Tel: +353-1-896-3725

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests blocked. No rebalancing

[Paul Emmerich]

You can prevent creation of the PGs on the old filestore OSDs (which
seems to be the culprit here) during replacement by replacing the
disks the hard way:

* ceph osd destroy osd.X
* re-create with bluestore under the same id (ceph volume ... --osd-id X)

it will then just backfill onto the same disk without moving any PG.

Keep in mind that this means that you are running with one missing
copy during the recovery, so that's not the recommended way to do
that.

Paul


2018-09-20 13:51 GMT+02:00 Eugen Block :
> Hi,
>
> to reduce impact on clients during migration I would set the OSD's
> primary-affinity to 0 beforehand. This should prevent the slow requests, at
> least this setting has helped us a lot with problematic OSDs.
>
> Regards
> Eugen
>
>
> Zitat von Jaime Ibar :
>
>
>> Hi all,
>>
>> we recently upgrade from Jewel 10.2.10 to Luminous 12.2.7, now we're
>> trying to migrate the
>>
>> osd's to Bluestore following this document[0], however when I mark the osd
>> as out,
>>
>> I'm getting warnings similar to these ones
>>
>> 2018-09-20 09:32:46.079630 mon.dri-ceph01 [WRN] Health check failed: 2
>> slow requests are blocked > 32 sec. Implicated osds 16,28 (REQUEST_SLOW)
>> 2018-09-20 09:32:52.841123 mon.dri-ceph01 [WRN] Health check update: 7
>> slow requests are blocked > 32 sec. Implicated osds 10,16,28,32,59
>> (REQUEST_SLOW)
>> 2018-09-20 09:32:57.842230 mon.dri-ceph01 [WRN] Health check update: 15
>> slow requests are blocked > 32 sec. Implicated osds 10,16,28,31,32,59,78,80
>> (REQUEST_SLOW)
>>
>> 2018-09-20 09:32:58.851142 mon.dri-ceph01 [WRN] Health check update:
>> 244944/40100780 objects misplaced (0.611%) (OBJECT_MISPLACED)
>> 2018-09-20 09:32:58.851160 mon.dri-ceph01 [WRN] Health check update: 249
>> PGs pending on creation (PENDING_CREATING_PGS)
>>
>> which prevent ceph start rebalancing and the vm's running on ceph start
>> hanging and we have to mark the osd back in.
>>
>> I tried to reweight the osd to 0.90 in order to minimize the impact on the
>> cluster but the warnings are the same.
>>
>> I tried to increased these settings to
>>
>> mds cache memory limit = 2147483648
>> rocksdb cache size = 2147483648
>>
>> but with no luck, same warnings.
>>
>> We also have cephfs for storing files from different projects(no directory
>> fragmentation enabled).
>>
>> The problem here is that if one osd dies, all the services will be blocked
>> as ceph won't be able to
>>
>> start rebalancing.
>>
>> The cluster is
>>
>> - 3 mons
>>
>> - 3 mds(running on the same hosts as the mons). 2 mds active and 1 standby
>>
>> - 3 mgr(running on the same hosts as the mons)
>>
>> - 6 servers, 12 osd's each.
>>
>> - 1GB private network
>>
>>
>> Does anyone know how to fix or where the problem could be?
>>
>> Thanks a lot in advance.
>>
>> Jaime
>>
>>
>> [0]
>> http://docs.ceph.com/docs/luminous/rados/operations/bluestore-migration/
>>
>> --
>>
>> Jaime Ibar
>> High Performance & Research Computing, IS Services
>> Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
>> http://www.tchpc.tcd.ie/  |ja...@tchpc.tcd.ie
>> Tel: +353-1-896-3725
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests blocked. No rebalancing

[Eugen Block]

Hi,

to reduce impact on clients during migration I would set the OSD's  
primary-affinity to 0 beforehand. This should prevent the slow  
requests, at least this setting has helped us a lot with problematic  
OSDs.


Regards
Eugen


Zitat von Jaime Ibar :


Hi all,

we recently upgrade from Jewel 10.2.10 to Luminous 12.2.7, now we're  
trying to migrate the


osd's to Bluestore following this document[0], however when I mark  
the osd as out,


I'm getting warnings similar to these ones

2018-09-20 09:32:46.079630 mon.dri-ceph01 [WRN] Health check failed:  
2 slow requests are blocked > 32 sec. Implicated osds 16,28  
(REQUEST_SLOW)
2018-09-20 09:32:52.841123 mon.dri-ceph01 [WRN] Health check update:  
7 slow requests are blocked > 32 sec. Implicated osds 10,16,28,32,59  
(REQUEST_SLOW)
2018-09-20 09:32:57.842230 mon.dri-ceph01 [WRN] Health check update:  
15 slow requests are blocked > 32 sec. Implicated osds  
10,16,28,31,32,59,78,80 (REQUEST_SLOW)


2018-09-20 09:32:58.851142 mon.dri-ceph01 [WRN] Health check update:  
244944/40100780 objects misplaced (0.611%) (OBJECT_MISPLACED)
2018-09-20 09:32:58.851160 mon.dri-ceph01 [WRN] Health check update:  
249 PGs pending on creation (PENDING_CREATING_PGS)


which prevent ceph start rebalancing and the vm's running on ceph  
start hanging and we have to mark the osd back in.


I tried to reweight the osd to 0.90 in order to minimize the impact  
on the cluster but the warnings are the same.


I tried to increased these settings to

mds cache memory limit = 2147483648
rocksdb cache size = 2147483648

but with no luck, same warnings.

We also have cephfs for storing files from different projects(no  
directory fragmentation enabled).


The problem here is that if one osd dies, all the services will be  
blocked as ceph won't be able to


start rebalancing.

The cluster is

- 3 mons

- 3 mds(running on the same hosts as the mons). 2 mds active and 1 standby

- 3 mgr(running on the same hosts as the mons)

- 6 servers, 12 osd's each.

- 1GB private network


Does anyone know how to fix or where the problem could be?

Thanks a lot in advance.

Jaime


[0] http://docs.ceph.com/docs/luminous/rados/operations/bluestore-migration/

--

Jaime Ibar
High Performance & Research Computing, IS Services
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
http://www.tchpc.tcd.ie/  |ja...@tchpc.tcd.ie
Tel: +353-1-896-3725




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests blocked. No rebalancing

[Darius Kasparavičius]

Hello,


2018-09-20 09:32:58.851160 mon.dri-ceph01 [WRN] Health check update:
249 PGs pending on creation (PENDING_CREATING_PGS)

This error might indicate that you are hitting a PG limit per osd.
Here some information on it
https://ceph.com/community/new-luminous-pg-overdose-protection/ . You
might need to increase mon_max_pg_per_osd for OSD to start balancing
out.


On Thu, Sep 20, 2018 at 2:25 PM Jaime Ibar  wrote:
>
> Hi all,
>
> we recently upgrade from Jewel 10.2.10 to Luminous 12.2.7, now we're trying 
> to migrate the
>
> osd's to Bluestore following this document[0], however when I mark the osd as 
> out,
>
> I'm getting warnings similar to these ones
>
> 2018-09-20 09:32:46.079630 mon.dri-ceph01 [WRN] Health check failed: 2 slow 
> requests are blocked > 32 sec. Implicated osds 16,28 (REQUEST_SLOW)
> 2018-09-20 09:32:52.841123 mon.dri-ceph01 [WRN] Health check update: 7 slow 
> requests are blocked > 32 sec. Implicated osds 10,16,28,32,59 (REQUEST_SLOW)
> 2018-09-20 09:32:57.842230 mon.dri-ceph01 [WRN] Health check update: 15 slow 
> requests are blocked > 32 sec. Implicated osds 10,16,28,31,32,59,78,80 
> (REQUEST_SLOW)
>
> 2018-09-20 09:32:58.851142 mon.dri-ceph01 [WRN] Health check update: 
> 244944/40100780 objects misplaced (0.611%) (OBJECT_MISPLACED)
> 2018-09-20 09:32:58.851160 mon.dri-ceph01 [WRN] Health check update: 249 PGs 
> pending on creation (PENDING_CREATING_PGS)
>
> which prevent ceph start rebalancing and the vm's running on ceph start 
> hanging and we have to mark the osd back in.
>
> I tried to reweight the osd to 0.90 in order to minimize the impact on the 
> cluster but the warnings are the same.
>
> I tried to increased these settings to
>
> mds cache memory limit = 2147483648
> rocksdb cache size = 2147483648
>
> but with no luck, same warnings.
>
> We also have cephfs for storing files from different projects(no directory 
> fragmentation enabled).
>
> The problem here is that if one osd dies, all the services will be blocked as 
> ceph won't be able to
>
> start rebalancing.
>
> The cluster is
>
> - 3 mons
>
> - 3 mds(running on the same hosts as the mons). 2 mds active and 1 standby
>
> - 3 mgr(running on the same hosts as the mons)
>
> - 6 servers, 12 osd's each.
>
> - 1GB private network
>
>
> Does anyone know how to fix or where the problem could be?
>
> Thanks a lot in advance.
>
> Jaime
>
>
> [0] http://docs.ceph.com/docs/luminous/rados/operations/bluestore-migration/
>
> --
>
> Jaime Ibar
> High Performance & Research Computing, IS Services
> Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
> http://www.tchpc.tcd.ie/ | ja...@tchpc.tcd.ie
> Tel: +353-1-896-3725
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Slow requests blocked. No rebalancing

[Jaime Ibar]

Hi all,

we recently upgrade from Jewel 10.2.10 to Luminous 12.2.7, now we're 
trying to migrate the


osd's to Bluestore following this document[0], however when I mark the 
osd as out,


I'm getting warnings similar to these ones

2018-09-20 09:32:46.079630 mon.dri-ceph01 [WRN] Health check failed: 2 
slow requests are blocked > 32 sec. Implicated osds 16,28 (REQUEST_SLOW)
2018-09-20 09:32:52.841123 mon.dri-ceph01 [WRN] Health check update: 7 
slow requests are blocked > 32 sec. Implicated osds 10,16,28,32,59 
(REQUEST_SLOW)
2018-09-20 09:32:57.842230 mon.dri-ceph01 [WRN] Health check update: 15 
slow requests are blocked > 32 sec. Implicated osds 
10,16,28,31,32,59,78,80 (REQUEST_SLOW)


2018-09-20 09:32:58.851142 mon.dri-ceph01 [WRN] Health check update: 
244944/40100780 objects misplaced (0.611%) (OBJECT_MISPLACED)
2018-09-20 09:32:58.851160 mon.dri-ceph01 [WRN] Health check update: 249 
PGs pending on creation (PENDING_CREATING_PGS)


which prevent ceph start rebalancing and the vm's running on ceph start 
hanging and we have to mark the osd back in.


I tried to reweight the osd to 0.90 in order to minimize the impact on 
the cluster but the warnings are the same.


I tried to increased these settings to

mds cache memory limit = 2147483648
rocksdb cache size = 2147483648

but with no luck, same warnings.

We also have cephfs for storing files from different projects(no 
directory fragmentation enabled).


The problem here is that if one osd dies, all the services will be 
blocked as ceph won't be able to


start rebalancing.

The cluster is

- 3 mons

- 3 mds(running on the same hosts as the mons). 2 mds active and 1 standby

- 3 mgr(running on the same hosts as the mons)

- 6 servers, 12 osd's each.

- 1GB private network


Does anyone know how to fix or where the problem could be?

Thanks a lot in advance.

Jaime


[0] http://docs.ceph.com/docs/luminous/rados/operations/bluestore-migration/

--

Jaime Ibar
High Performance & Research Computing, IS Services
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
http://www.tchpc.tcd.ie/  |ja...@tchpc.tcd.ie
Tel: +353-1-896-3725

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests from bluestore osds

[Augusto Rodrigues]

I solved my slow requests by increasing the size of block.db. Calculate 4% per 
stored TB and preferably host the DB in NVME.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests from bluestore osds

[Marc Schöchlin]

Hello Uwe,

as described in my mail we are running 4.13.0-39.

In conjunction with some later mails of this thread it seems that this problem 
might related to os/microcode (spectre) updates.
I am planning a ceph/ubuntu upgrade in the next week because of various 
reasons, let's see what happens.

Regards Marc


Am 05.09.2018 um 20:24 schrieb Uwe Sauter:
> I'm also experiencing slow requests though I cannot point it to scrubbing.
>
> Which kernel do you run? Would you be able to test against the same kernel 
> with Spectre/Meltdown mitigations disabled ("noibrs noibpb nopti 
> nospectre_v2" as boot option)?
>
> Uwe
>
> Am 05.09.18 um 19:30 schrieb Brett Chancellor:
>> Marc,
>>    As with you, this problem manifests itself only when the bluestore OSD is 
>> involved in some form of deep scrub.  Anybody have any insight on what might 
>> be causing this?
>>
>> -Brett
>>
>> On Mon, Sep 3, 2018 at 4:13 AM, Marc Schöchlin > > wrote:
>>
>>     Hi,
>>
>>     we are also experiencing this type of behavior for some weeks on our not
>>     so performance critical hdd pools.
>>     We haven't spent so much time on this problem, because there are
>>     currently more important tasks - but here are a few details:
>>
>>     Running the following loop results in the following output:
>>
>>     while true; do ceph health|grep -q HEALTH_OK || (date;  ceph health
>>     detail); sleep 2; done
>>
>>     Sun Sep  2 20:59:47 CEST 2018
>>     HEALTH_WARN 4 slow requests are blocked > 32 sec
>>     REQUEST_SLOW 4 slow requests are blocked > 32 sec
>>      4 ops are blocked > 32.768 sec
>>      osd.43 has blocked requests > 32.768 sec
>>     Sun Sep  2 20:59:50 CEST 2018
>>     HEALTH_WARN 4 slow requests are blocked > 32 sec
>>     REQUEST_SLOW 4 slow requests are blocked > 32 sec
>>      4 ops are blocked > 32.768 sec
>>      osd.43 has blocked requests > 32.768 sec
>>     Sun Sep  2 20:59:52 CEST 2018
>>     HEALTH_OK
>>     Sun Sep  2 21:00:28 CEST 2018
>>     HEALTH_WARN 1 slow requests are blocked > 32 sec
>>     REQUEST_SLOW 1 slow requests are blocked > 32 sec
>>      1 ops are blocked > 32.768 sec
>>      osd.41 has blocked requests > 32.768 sec
>>     Sun Sep  2 21:00:31 CEST 2018
>>     HEALTH_WARN 7 slow requests are blocked > 32 sec
>>     REQUEST_SLOW 7 slow requests are blocked > 32 sec
>>      7 ops are blocked > 32.768 sec
>>      osds 35,41 have blocked requests > 32.768 sec
>>     Sun Sep  2 21:00:33 CEST 2018
>>     HEALTH_WARN 7 slow requests are blocked > 32 sec
>>     REQUEST_SLOW 7 slow requests are blocked > 32 sec
>>      7 ops are blocked > 32.768 sec
>>      osds 35,51 have blocked requests > 32.768 sec
>>     Sun Sep  2 21:00:35 CEST 2018
>>     HEALTH_WARN 7 slow requests are blocked > 32 sec
>>     REQUEST_SLOW 7 slow requests are blocked > 32 sec
>>      7 ops are blocked > 32.768 sec
>>      osds 35,51 have blocked requests > 32.768 sec
>>
>>     Our details:
>>
>>    * system details:
>>      * Ubuntu 16.04
>>       * Kernel 4.13.0-39
>>       * 30 * 8 TB Disk (SEAGATE/ST8000NM0075)
>>       * 3* Dell Power Edge R730xd (Firmware 2.50.50.50)
>>         * Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
>>         * 2*10GBITS SFP+ Network Adapters
>>         * 192GB RAM
>>       * Pools are using replication factor 3, 2MB object size,
>>         85% write load, 1700 write IOPS/sec
>>         (ops mainly between 4k and 16k size), 300 read IOPS/sec
>>    * we have the impression that this appears on deepscrub/scrub 
>> activity.
>>    * Ceph 12.2.5, we alread played with the osd settings OSD Settings
>>      (our assumtion was that the problem is related to rocksdb 
>> compaction)
>>      bluestore cache kv max = 2147483648
>>      bluestore cache kv ratio = 0.9
>>      bluestore cache meta ratio = 0.1
>>      bluestore cache size hdd = 10737418240
>>    * this type problem only appears on hdd/bluestore osds, ssd/bluestore
>>      osds did never experienced that problem
>>    * the system is healthy, no swapping, no high load, no errors in dmesg
>>
>>     I attached a log excerpt of osd.35 - probably this is useful for
>>     investigating the problem is someone owns deeper bluestore knowledge.
>>     (slow requests appeared on Sun Sep  2 21:00:35)
>>
>>     Regards
>>     Marc
>>
>>
>>     Am 02.09.2018 um 15:50 schrieb Brett Chancellor:
>>     > The warnings look like this.     >
>>     > 6 ops are blocked > 32.768 sec on osd.219
>>     > 1 osds have slow requests
>>     >
>>     > On Sun, Sep 2, 2018, 8:45 AM Alfredo Deza > 
>>     > >> wrote:
>>     >
>>     >     On Sat, Sep 1, 2018 at 12:45 PM, Brett Chancellor
>>  >     mailto:bchancel...@salesforce.com> 
>> >     >>
>>  >     wrote:
>> 

Re: [ceph-users] Slow requests from bluestore osds

[Tim Bishop]

On Sat, Sep 01, 2018 at 12:45:06PM -0400, Brett Chancellor wrote:
> Hi Cephers,
>   I am in the process of upgrading a cluster from Filestore to bluestore,
> but I'm concerned about frequent warnings popping up against the new
> bluestore devices. I'm frequently seeing messages like this, although the
> specific osd changes, it's always one of the few hosts I've converted to
> bluestore.
> 
> 6 ops are blocked > 32.768 sec on osd.219
> 1 osds have slow requests
> 
> I'm running 12.2.4, have any of you seen similar issues? It seems as though
> these messages pop up more frequently when one of the bluestore pgs is
> involved in a scrub.  I'll include my bluestore creation process below, in
> case that might cause an issue. (sdb, sdc, sdd are SATA, sde and sdf are
> SSD)

I had the same issue for some time after a bunch of updates including
switching to Luminous and to Bluestore from Filestore. I was unable to
track down what was causing it.

Then recently I did another bunch of updates including Ceph 12.2.7 and
the latest Ubuntu 16.04 updates (which would have included microcode
updates), and the problem has as good as gone away. My monitoring also
shows reduced latency on my RBD volumes and end users are reporting
improved performance.

Just throwing this out there in case it helps anyone else - appreciate
it's very anecdotal.

Tim.

-- 
Tim Bishop
http://www.bishnet.net/tim/
PGP Key: 0x6C226B37FDF38D55

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests from bluestore osds

[Brett Chancellor]

Mine is currently at 1000 due to the high number of pgs we had coming from
Jewel. I do find it odd that only the bluestore OSDs have this issue.
Filestore OSDs seem to be unaffected.

On Wed, Sep 5, 2018, 3:43 PM Samuel Taylor Liston 
wrote:

> Just a thought - have you looked at increasing your "—mon_max_pg_per_osd”
> both on the mons and osds?  I was having a similar issue while trying to
> add more OSDs to my cluster (12.2.27, CentOS7.5,
> 3.10.0-862.9.1.el7.x86_64).   I increased mine to 300 temporarily while
> adding OSDs and stopped having blocked requests.
> --
> Sam Liston (sam.lis...@utah.edu)
> 
> Center for High Performance Computing
> 155 S. 1452 E. Rm 405
> Salt Lake City, Utah 84112 (801)232-6932
> 
>
>
>
>
> On Sep 5, 2018, at 12:46 PM, Daniel Pryor  wrote:
>
> I've experienced the same thing during scrubbing and/or any kind of
> expansion activity.
>
> *Daniel Pryor*
>
> On Mon, Sep 3, 2018 at 2:13 AM Marc Schöchlin  wrote:
>
>> Hi,
>>
>> we are also experiencing this type of behavior for some weeks on our not
>> so performance critical hdd pools.
>> We haven't spent so much time on this problem, because there are
>> currently more important tasks - but here are a few details:
>>
>> Running the following loop results in the following output:
>>
>> while true; do ceph health|grep -q HEALTH_OK || (date;  ceph health
>> detail); sleep 2; done
>>
>> Sun Sep  2 20:59:47 CEST 2018
>> HEALTH_WARN 4 slow requests are blocked > 32 sec
>> REQUEST_SLOW 4 slow requests are blocked > 32 sec
>> 4 ops are blocked > 32.768 sec
>> osd.43 has blocked requests > 32.768 sec
>> Sun Sep  2 20:59:50 CEST 2018
>> HEALTH_WARN 4 slow requests are blocked > 32 sec
>> REQUEST_SLOW 4 slow requests are blocked > 32 sec
>> 4 ops are blocked > 32.768 sec
>> osd.43 has blocked requests > 32.768 sec
>> Sun Sep  2 20:59:52 CEST 2018
>> HEALTH_OK
>> Sun Sep  2 21:00:28 CEST 2018
>> HEALTH_WARN 1 slow requests are blocked > 32 sec
>> REQUEST_SLOW 1 slow requests are blocked > 32 sec
>> 1 ops are blocked > 32.768 sec
>> osd.41 has blocked requests > 32.768 sec
>> Sun Sep  2 21:00:31 CEST 2018
>> HEALTH_WARN 7 slow requests are blocked > 32 sec
>> REQUEST_SLOW 7 slow requests are blocked > 32 sec
>> 7 ops are blocked > 32.768 sec
>> osds 35,41 have blocked requests > 32.768 sec
>> Sun Sep  2 21:00:33 CEST 2018
>> HEALTH_WARN 7 slow requests are blocked > 32 sec
>> REQUEST_SLOW 7 slow requests are blocked > 32 sec
>> 7 ops are blocked > 32.768 sec
>> osds 35,51 have blocked requests > 32.768 sec
>> Sun Sep  2 21:00:35 CEST 2018
>> HEALTH_WARN 7 slow requests are blocked > 32 sec
>> REQUEST_SLOW 7 slow requests are blocked > 32 sec
>> 7 ops are blocked > 32.768 sec
>> osds 35,51 have blocked requests > 32.768 sec
>>
>> Our details:
>>
>>   * system details:
>> * Ubuntu 16.04
>>  * Kernel 4.13.0-39
>>  * 30 * 8 TB Disk (SEAGATE/ST8000NM0075)
>>  * 3* Dell Power Edge R730xd (Firmware 2.50.50.50)
>>* Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
>>* 2*10GBITS SFP+ Network Adapters
>>* 192GB RAM
>>  * Pools are using replication factor 3, 2MB object size,
>>85% write load, 1700 write IOPS/sec
>>(ops mainly between 4k and 16k size), 300 read IOPS/sec
>>   * we have the impression that this appears on deepscrub/scrub activity.
>>   * Ceph 12.2.5, we alread played with the osd settings OSD Settings
>> (our assumtion was that the problem is related to rocksdb compaction)
>> bluestore cache kv max = 2147483648
>> bluestore cache kv ratio = 0.9
>> bluestore cache meta ratio = 0.1
>> bluestore cache size hdd = 10737418240
>>   * this type problem only appears on hdd/bluestore osds, ssd/bluestore
>> osds did never experienced that problem
>>   * the system is healthy, no swapping, no high load, no errors in dmesg
>>
>> I attached a log excerpt of osd.35 - probably this is useful for
>> investigating the problem is someone owns deeper bluestore knowledge.
>> (slow requests appeared on Sun Sep  2 21:00:35)
>>
>> Regards
>> Marc
>>
>>
>> Am 02.09.2018 um 15:50 schrieb Brett Chancellor:
>> > The warnings look like this.
>> >
>> > 6 ops are blocked > 32.768 sec on osd.219
>> > 1 osds have slow requests
>> >
>> > On Sun, Sep 2, 2018, 8:45 AM Alfredo Deza > > > wrote:
>> >
>> > On Sat, Sep 1, 2018 at 12:45 PM, Brett Chancellor
>> > mailto:bchancel...@salesforce.com>>
>> > wrote:
>> > > Hi Cephers,
>> > >   I am in the process of upgrading a cluster from Filestore to
>> > bluestore,
>> > > but I'm concerned about frequent warnings popping up against the
>> new
>> > > bluestore devices. I'm frequently seeing messages like this,
>> > although the
>> > > specific osd changes, it's always one of the few hosts I've
>> > converted to
>> > > bluestore.
>> > >
>> > > 6 ops a

Re: [ceph-users] Slow requests from bluestore osds

[Samuel Taylor Liston]

Just a thought - have you looked at increasing your "—mon_max_pg_per_osd” both 
on the mons and osds?  I was having a similar issue while trying to add more 
OSDs to my cluster (12.2.27, CentOS7.5, 3.10.0-862.9.1.el7.x86_64).   I 
increased mine to 300 temporarily while adding OSDs and stopped having blocked 
requests.
--
Sam Liston (sam.lis...@utah.edu)

Center for High Performance Computing
155 S. 1452 E. Rm 405
Salt Lake City, Utah 84112 (801)232-6932





On Sep 5, 2018, at 12:46 PM, Daniel Pryor 
mailto:dpr...@parchment.com>> wrote:

I've experienced the same thing during scrubbing and/or any kind of expansion 
activity.

Daniel Pryor

On Mon, Sep 3, 2018 at 2:13 AM Marc Schöchlin 
mailto:m...@256bit.org>> wrote:
Hi,

we are also experiencing this type of behavior for some weeks on our not
so performance critical hdd pools.
We haven't spent so much time on this problem, because there are
currently more important tasks - but here are a few details:

Running the following loop results in the following output:

while true; do ceph health|grep -q HEALTH_OK || (date;  ceph health
detail); sleep 2; done

Sun Sep  2 20:59:47 CEST 2018
HEALTH_WARN 4 slow requests are blocked > 32 sec
REQUEST_SLOW 4 slow requests are blocked > 32 sec
4 ops are blocked > 32.768 sec
osd.43 has blocked requests > 32.768 sec
Sun Sep  2 20:59:50 CEST 2018
HEALTH_WARN 4 slow requests are blocked > 32 sec
REQUEST_SLOW 4 slow requests are blocked > 32 sec
4 ops are blocked > 32.768 sec
osd.43 has blocked requests > 32.768 sec
Sun Sep  2 20:59:52 CEST 2018
HEALTH_OK
Sun Sep  2 21:00:28 CEST 2018
HEALTH_WARN 1 slow requests are blocked > 32 sec
REQUEST_SLOW 1 slow requests are blocked > 32 sec
1 ops are blocked > 32.768 sec
osd.41 has blocked requests > 32.768 sec
Sun Sep  2 21:00:31 CEST 2018
HEALTH_WARN 7 slow requests are blocked > 32 sec
REQUEST_SLOW 7 slow requests are blocked > 32 sec
7 ops are blocked > 32.768 sec
osds 35,41 have blocked requests > 32.768 sec
Sun Sep  2 21:00:33 CEST 2018
HEALTH_WARN 7 slow requests are blocked > 32 sec
REQUEST_SLOW 7 slow requests are blocked > 32 sec
7 ops are blocked > 32.768 sec
osds 35,51 have blocked requests > 32.768 sec
Sun Sep  2 21:00:35 CEST 2018
HEALTH_WARN 7 slow requests are blocked > 32 sec
REQUEST_SLOW 7 slow requests are blocked > 32 sec
7 ops are blocked > 32.768 sec
osds 35,51 have blocked requests > 32.768 sec

Our details:

  * system details:
* Ubuntu 16.04
 * Kernel 4.13.0-39
 * 30 * 8 TB Disk (SEAGATE/ST8000NM0075)
 * 3* Dell Power Edge R730xd (Firmware 2.50.50.50)
   * Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
   * 2*10GBITS SFP+ Network Adapters
   * 192GB RAM
 * Pools are using replication factor 3, 2MB object size,
   85% write load, 1700 write IOPS/sec
   (ops mainly between 4k and 16k size), 300 read IOPS/sec
  * we have the impression that this appears on deepscrub/scrub activity.
  * Ceph 12.2.5, we alread played with the osd settings OSD Settings
(our assumtion was that the problem is related to rocksdb compaction)
bluestore cache kv max = 2147483648
bluestore cache kv ratio = 0.9
bluestore cache meta ratio = 0.1
bluestore cache size hdd = 10737418240
  * this type problem only appears on hdd/bluestore osds, ssd/bluestore
osds did never experienced that problem
  * the system is healthy, no swapping, no high load, no errors in dmesg

I attached a log excerpt of osd.35 - probably this is useful for
investigating the problem is someone owns deeper bluestore knowledge.
(slow requests appeared on Sun Sep  2 21:00:35)

Regards
Marc


Am 02.09.2018 um 15:50 schrieb Brett Chancellor:
> The warnings look like this.
>
> 6 ops are blocked > 32.768 sec on osd.219
> 1 osds have slow requests
>
> On Sun, Sep 2, 2018, 8:45 AM Alfredo Deza 
> mailto:ad...@redhat.com>
> >> wrote:
>
> On Sat, Sep 1, 2018 at 12:45 PM, Brett Chancellor
> mailto:bchancel...@salesforce.com> 
> >>
> wrote:
> > Hi Cephers,
> >   I am in the process of upgrading a cluster from Filestore to
> bluestore,
> > but I'm concerned about frequent warnings popping up against the new
> > bluestore devices. I'm frequently seeing messages like this,
> although the
> > specific osd changes, it's always one of the few hosts I've
> converted to
> > bluestore.
> >
> > 6 ops are blocked > 32.768 sec on osd.219
> > 1 osds have slow requests
> >
> > I'm running 12.2.4, have any of you seen similar issues? It
> seems as though
> > these messages pop up more frequently when one of the bluestore
> pgs is
> > involved in a scrub.  I'll include my bluestore creation process
> below, in
> > case that might 

Re: [ceph-users] Slow requests from bluestore osds

[Daniel Pryor]

I've experienced the same thing during scrubbing and/or any kind of
expansion activity.

*Daniel Pryor*

On Mon, Sep 3, 2018 at 2:13 AM Marc Schöchlin  wrote:

> Hi,
>
> we are also experiencing this type of behavior for some weeks on our not
> so performance critical hdd pools.
> We haven't spent so much time on this problem, because there are
> currently more important tasks - but here are a few details:
>
> Running the following loop results in the following output:
>
> while true; do ceph health|grep -q HEALTH_OK || (date;  ceph health
> detail); sleep 2; done
>
> Sun Sep  2 20:59:47 CEST 2018
> HEALTH_WARN 4 slow requests are blocked > 32 sec
> REQUEST_SLOW 4 slow requests are blocked > 32 sec
> 4 ops are blocked > 32.768 sec
> osd.43 has blocked requests > 32.768 sec
> Sun Sep  2 20:59:50 CEST 2018
> HEALTH_WARN 4 slow requests are blocked > 32 sec
> REQUEST_SLOW 4 slow requests are blocked > 32 sec
> 4 ops are blocked > 32.768 sec
> osd.43 has blocked requests > 32.768 sec
> Sun Sep  2 20:59:52 CEST 2018
> HEALTH_OK
> Sun Sep  2 21:00:28 CEST 2018
> HEALTH_WARN 1 slow requests are blocked > 32 sec
> REQUEST_SLOW 1 slow requests are blocked > 32 sec
> 1 ops are blocked > 32.768 sec
> osd.41 has blocked requests > 32.768 sec
> Sun Sep  2 21:00:31 CEST 2018
> HEALTH_WARN 7 slow requests are blocked > 32 sec
> REQUEST_SLOW 7 slow requests are blocked > 32 sec
> 7 ops are blocked > 32.768 sec
> osds 35,41 have blocked requests > 32.768 sec
> Sun Sep  2 21:00:33 CEST 2018
> HEALTH_WARN 7 slow requests are blocked > 32 sec
> REQUEST_SLOW 7 slow requests are blocked > 32 sec
> 7 ops are blocked > 32.768 sec
> osds 35,51 have blocked requests > 32.768 sec
> Sun Sep  2 21:00:35 CEST 2018
> HEALTH_WARN 7 slow requests are blocked > 32 sec
> REQUEST_SLOW 7 slow requests are blocked > 32 sec
> 7 ops are blocked > 32.768 sec
> osds 35,51 have blocked requests > 32.768 sec
>
> Our details:
>
>   * system details:
> * Ubuntu 16.04
>  * Kernel 4.13.0-39
>  * 30 * 8 TB Disk (SEAGATE/ST8000NM0075)
>  * 3* Dell Power Edge R730xd (Firmware 2.50.50.50)
>* Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
>* 2*10GBITS SFP+ Network Adapters
>* 192GB RAM
>  * Pools are using replication factor 3, 2MB object size,
>85% write load, 1700 write IOPS/sec
>(ops mainly between 4k and 16k size), 300 read IOPS/sec
>   * we have the impression that this appears on deepscrub/scrub activity.
>   * Ceph 12.2.5, we alread played with the osd settings OSD Settings
> (our assumtion was that the problem is related to rocksdb compaction)
> bluestore cache kv max = 2147483648
> bluestore cache kv ratio = 0.9
> bluestore cache meta ratio = 0.1
> bluestore cache size hdd = 10737418240
>   * this type problem only appears on hdd/bluestore osds, ssd/bluestore
> osds did never experienced that problem
>   * the system is healthy, no swapping, no high load, no errors in dmesg
>
> I attached a log excerpt of osd.35 - probably this is useful for
> investigating the problem is someone owns deeper bluestore knowledge.
> (slow requests appeared on Sun Sep  2 21:00:35)
>
> Regards
> Marc
>
>
> Am 02.09.2018 um 15:50 schrieb Brett Chancellor:
> > The warnings look like this.
> >
> > 6 ops are blocked > 32.768 sec on osd.219
> > 1 osds have slow requests
> >
> > On Sun, Sep 2, 2018, 8:45 AM Alfredo Deza  > > wrote:
> >
> > On Sat, Sep 1, 2018 at 12:45 PM, Brett Chancellor
> > mailto:bchancel...@salesforce.com>>
> > wrote:
> > > Hi Cephers,
> > >   I am in the process of upgrading a cluster from Filestore to
> > bluestore,
> > > but I'm concerned about frequent warnings popping up against the
> new
> > > bluestore devices. I'm frequently seeing messages like this,
> > although the
> > > specific osd changes, it's always one of the few hosts I've
> > converted to
> > > bluestore.
> > >
> > > 6 ops are blocked > 32.768 sec on osd.219
> > > 1 osds have slow requests
> > >
> > > I'm running 12.2.4, have any of you seen similar issues? It
> > seems as though
> > > these messages pop up more frequently when one of the bluestore
> > pgs is
> > > involved in a scrub.  I'll include my bluestore creation process
> > below, in
> > > case that might cause an issue. (sdb, sdc, sdd are SATA, sde and
> > sdf are
> > > SSD)
> >
> > Would be useful to include what those warnings say. The ceph-volume
> > commands look OK to me
> >
> > >
> > >
> > > ## Process used to create osds
> > > sudo ceph-disk zap /dev/sdb /dev/sdc /dev/sdd /dev/sdd /dev/sde
> > /dev/sdf
> > > sudo ceph-volume lvm zap /dev/sdb
> > > sudo ceph-volume lvm zap /dev/sdc
> > > sudo ceph-volume lvm zap /dev/sdd
> > > sudo ceph-volume lvm zap /dev/sde
> > > sudo ceph-volume lvm zap /dev/sdf
> > > sudo sgdisk -n 0:

Re: [ceph-users] Slow requests from bluestore osds

[Brett Chancellor]

I'm running Centos 7.5. If I turn off spectre/meltdown protection then a
security sweep will disconnect it from the network.

-Brett

On Wed, Sep 5, 2018 at 2:24 PM, Uwe Sauter  wrote:

> I'm also experiencing slow requests though I cannot point it to scrubbing.
>
> Which kernel do you run? Would you be able to test against the same kernel
> with Spectre/Meltdown mitigations disabled ("noibrs noibpb nopti
> nospectre_v2" as boot option)?
>
> Uwe
>
> Am 05.09.18 um 19:30 schrieb Brett Chancellor:
>
>> Marc,
>>As with you, this problem manifests itself only when the bluestore OSD
>> is involved in some form of deep scrub.  Anybody have any insight on what
>> might be causing this?
>>
>> -Brett
>>
>> On Mon, Sep 3, 2018 at 4:13 AM, Marc Schöchlin > m...@256bit.org>> wrote:
>>
>> Hi,
>>
>> we are also experiencing this type of behavior for some weeks on our
>> not
>> so performance critical hdd pools.
>> We haven't spent so much time on this problem, because there are
>> currently more important tasks - but here are a few details:
>>
>> Running the following loop results in the following output:
>>
>> while true; do ceph health|grep -q HEALTH_OK || (date;  ceph health
>> detail); sleep 2; done
>>
>> Sun Sep  2 20:59:47 CEST 2018
>> HEALTH_WARN 4 slow requests are blocked > 32 sec
>> REQUEST_SLOW 4 slow requests are blocked > 32 sec
>>  4 ops are blocked > 32.768 sec
>>  osd.43 has blocked requests > 32.768 sec
>> Sun Sep  2 20:59:50 CEST 2018
>> HEALTH_WARN 4 slow requests are blocked > 32 sec
>> REQUEST_SLOW 4 slow requests are blocked > 32 sec
>>  4 ops are blocked > 32.768 sec
>>  osd.43 has blocked requests > 32.768 sec
>> Sun Sep  2 20:59:52 CEST 2018
>> HEALTH_OK
>> Sun Sep  2 21:00:28 CEST 2018
>> HEALTH_WARN 1 slow requests are blocked > 32 sec
>> REQUEST_SLOW 1 slow requests are blocked > 32 sec
>>  1 ops are blocked > 32.768 sec
>>  osd.41 has blocked requests > 32.768 sec
>> Sun Sep  2 21:00:31 CEST 2018
>> HEALTH_WARN 7 slow requests are blocked > 32 sec
>> REQUEST_SLOW 7 slow requests are blocked > 32 sec
>>  7 ops are blocked > 32.768 sec
>>  osds 35,41 have blocked requests > 32.768 sec
>> Sun Sep  2 21:00:33 CEST 2018
>> HEALTH_WARN 7 slow requests are blocked > 32 sec
>> REQUEST_SLOW 7 slow requests are blocked > 32 sec
>>  7 ops are blocked > 32.768 sec
>>  osds 35,51 have blocked requests > 32.768 sec
>> Sun Sep  2 21:00:35 CEST 2018
>> HEALTH_WARN 7 slow requests are blocked > 32 sec
>> REQUEST_SLOW 7 slow requests are blocked > 32 sec
>>  7 ops are blocked > 32.768 sec
>>  osds 35,51 have blocked requests > 32.768 sec
>>
>> Our details:
>>
>>* system details:
>>  * Ubuntu 16.04
>>   * Kernel 4.13.0-39
>>   * 30 * 8 TB Disk (SEAGATE/ST8000NM0075)
>>   * 3* Dell Power Edge R730xd (Firmware 2.50.50.50)
>> * Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
>> * 2*10GBITS SFP+ Network Adapters
>> * 192GB RAM
>>   * Pools are using replication factor 3, 2MB object size,
>> 85% write load, 1700 write IOPS/sec
>> (ops mainly between 4k and 16k size), 300 read IOPS/sec
>>* we have the impression that this appears on deepscrub/scrub
>> activity.
>>* Ceph 12.2.5, we alread played with the osd settings OSD Settings
>>  (our assumtion was that the problem is related to rocksdb
>> compaction)
>>  bluestore cache kv max = 2147483648
>>  bluestore cache kv ratio = 0.9
>>  bluestore cache meta ratio = 0.1
>>  bluestore cache size hdd = 10737418240
>>* this type problem only appears on hdd/bluestore osds,
>> ssd/bluestore
>>  osds did never experienced that problem
>>* the system is healthy, no swapping, no high load, no errors in
>> dmesg
>>
>> I attached a log excerpt of osd.35 - probably this is useful for
>> investigating the problem is someone owns deeper bluestore knowledge.
>> (slow requests appeared on Sun Sep  2 21:00:35)
>>
>> Regards
>> Marc
>>
>>
>> Am 02.09.2018 um 15:50 schrieb Brett Chancellor:
>> > The warnings look like this. >
>> > 6 ops are blocked > 32.768 sec on osd.219
>> > 1 osds have slow requests
>> >
>> > On Sun, Sep 2, 2018, 8:45 AM Alfredo Deza > 
>> > >> wrote:
>> >
>> > On Sat, Sep 1, 2018 at 12:45 PM, Brett Chancellor
>>  > mailto:bchancel...@salesforce.com>
>> >
>> >>
>>  > wrote:
>>  > > Hi Cephers,
>>  > >   I am in the process of upgrading a cluster from Filestore
>> to
>>  > bluestore,
>>  > > but I'm concerned about 

Re: [ceph-users] Slow requests from bluestore osds

[Uwe Sauter]

I'm also experiencing slow requests though I cannot point it to scrubbing.

Which kernel do you run? Would you be able to test against the same kernel with Spectre/Meltdown mitigations disabled 
("noibrs noibpb nopti nospectre_v2" as boot option)?


Uwe

Am 05.09.18 um 19:30 schrieb Brett Chancellor:

Marc,
   As with you, this problem manifests itself only when the bluestore OSD is involved in some form of deep scrub.  
Anybody have any insight on what might be causing this?


-Brett

On Mon, Sep 3, 2018 at 4:13 AM, Marc Schöchlin mailto:m...@256bit.org>> wrote:

Hi,

we are also experiencing this type of behavior for some weeks on our not
so performance critical hdd pools.
We haven't spent so much time on this problem, because there are
currently more important tasks - but here are a few details:

Running the following loop results in the following output:

while true; do ceph health|grep -q HEALTH_OK || (date;  ceph health
detail); sleep 2; done

Sun Sep  2 20:59:47 CEST 2018
HEALTH_WARN 4 slow requests are blocked > 32 sec
REQUEST_SLOW 4 slow requests are blocked > 32 sec
     4 ops are blocked > 32.768 sec
     osd.43 has blocked requests > 32.768 sec
Sun Sep  2 20:59:50 CEST 2018
HEALTH_WARN 4 slow requests are blocked > 32 sec
REQUEST_SLOW 4 slow requests are blocked > 32 sec
     4 ops are blocked > 32.768 sec
     osd.43 has blocked requests > 32.768 sec
Sun Sep  2 20:59:52 CEST 2018
HEALTH_OK
Sun Sep  2 21:00:28 CEST 2018
HEALTH_WARN 1 slow requests are blocked > 32 sec
REQUEST_SLOW 1 slow requests are blocked > 32 sec
     1 ops are blocked > 32.768 sec
     osd.41 has blocked requests > 32.768 sec
Sun Sep  2 21:00:31 CEST 2018
HEALTH_WARN 7 slow requests are blocked > 32 sec
REQUEST_SLOW 7 slow requests are blocked > 32 sec
     7 ops are blocked > 32.768 sec
     osds 35,41 have blocked requests > 32.768 sec
Sun Sep  2 21:00:33 CEST 2018
HEALTH_WARN 7 slow requests are blocked > 32 sec
REQUEST_SLOW 7 slow requests are blocked > 32 sec
     7 ops are blocked > 32.768 sec
     osds 35,51 have blocked requests > 32.768 sec
Sun Sep  2 21:00:35 CEST 2018
HEALTH_WARN 7 slow requests are blocked > 32 sec
REQUEST_SLOW 7 slow requests are blocked > 32 sec
     7 ops are blocked > 32.768 sec
     osds 35,51 have blocked requests > 32.768 sec

Our details:

   * system details:
     * Ubuntu 16.04
      * Kernel 4.13.0-39
      * 30 * 8 TB Disk (SEAGATE/ST8000NM0075)
      * 3* Dell Power Edge R730xd (Firmware 2.50.50.50)
        * Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
        * 2*10GBITS SFP+ Network Adapters
        * 192GB RAM
      * Pools are using replication factor 3, 2MB object size,
        85% write load, 1700 write IOPS/sec
        (ops mainly between 4k and 16k size), 300 read IOPS/sec
   * we have the impression that this appears on deepscrub/scrub activity.
   * Ceph 12.2.5, we alread played with the osd settings OSD Settings
     (our assumtion was that the problem is related to rocksdb compaction)
     bluestore cache kv max = 2147483648
     bluestore cache kv ratio = 0.9
     bluestore cache meta ratio = 0.1
     bluestore cache size hdd = 10737418240
   * this type problem only appears on hdd/bluestore osds, ssd/bluestore
     osds did never experienced that problem
   * the system is healthy, no swapping, no high load, no errors in dmesg

I attached a log excerpt of osd.35 - probably this is useful for
investigating the problem is someone owns deeper bluestore knowledge.
(slow requests appeared on Sun Sep  2 21:00:35)

Regards
Marc


Am 02.09.2018 um 15:50 schrieb Brett Chancellor:
> The warnings look like this. 
>

> 6 ops are blocked > 32.768 sec on osd.219
> 1 osds have slow requests
>
> On Sun, Sep 2, 2018, 8:45 AM Alfredo Deza mailto:ad...@redhat.com>
> >> wrote:
>
>     On Sat, Sep 1, 2018 at 12:45 PM, Brett Chancellor
 >     mailto:bchancel...@salesforce.com> 
>>
 >     wrote:
 >     > Hi Cephers,
 >     >   I am in the process of upgrading a cluster from Filestore to
 >     bluestore,
 >     > but I'm concerned about frequent warnings popping up against the 
new
 >     > bluestore devices. I'm frequently seeing messages like this,
 >     although the
 >     > specific osd changes, it's always one of the few hosts I've
 >     converted to
 >     > bluestore.
 >     >
 >     > 6 ops are blocked > 32.768 sec on osd.219
 >     > 1 osds have slow requests
 >     >
 >     > I'm running 12.2.4, have any of you seen similar issues? It
 >     seems as though
 >     > these messa

Re: [ceph-users] Slow requests from bluestore osds

[Brett Chancellor]

Marc,
  As with you, this problem manifests itself only when the bluestore OSD is
involved in some form of deep scrub.  Anybody have any insight on what
might be causing this?

-Brett

On Mon, Sep 3, 2018 at 4:13 AM, Marc Schöchlin  wrote:

> Hi,
>
> we are also experiencing this type of behavior for some weeks on our not
> so performance critical hdd pools.
> We haven't spent so much time on this problem, because there are
> currently more important tasks - but here are a few details:
>
> Running the following loop results in the following output:
>
> while true; do ceph health|grep -q HEALTH_OK || (date;  ceph health
> detail); sleep 2; done
>
> Sun Sep  2 20:59:47 CEST 2018
> HEALTH_WARN 4 slow requests are blocked > 32 sec
> REQUEST_SLOW 4 slow requests are blocked > 32 sec
> 4 ops are blocked > 32.768 sec
> osd.43 has blocked requests > 32.768 sec
> Sun Sep  2 20:59:50 CEST 2018
> HEALTH_WARN 4 slow requests are blocked > 32 sec
> REQUEST_SLOW 4 slow requests are blocked > 32 sec
> 4 ops are blocked > 32.768 sec
> osd.43 has blocked requests > 32.768 sec
> Sun Sep  2 20:59:52 CEST 2018
> HEALTH_OK
> Sun Sep  2 21:00:28 CEST 2018
> HEALTH_WARN 1 slow requests are blocked > 32 sec
> REQUEST_SLOW 1 slow requests are blocked > 32 sec
> 1 ops are blocked > 32.768 sec
> osd.41 has blocked requests > 32.768 sec
> Sun Sep  2 21:00:31 CEST 2018
> HEALTH_WARN 7 slow requests are blocked > 32 sec
> REQUEST_SLOW 7 slow requests are blocked > 32 sec
> 7 ops are blocked > 32.768 sec
> osds 35,41 have blocked requests > 32.768 sec
> Sun Sep  2 21:00:33 CEST 2018
> HEALTH_WARN 7 slow requests are blocked > 32 sec
> REQUEST_SLOW 7 slow requests are blocked > 32 sec
> 7 ops are blocked > 32.768 sec
> osds 35,51 have blocked requests > 32.768 sec
> Sun Sep  2 21:00:35 CEST 2018
> HEALTH_WARN 7 slow requests are blocked > 32 sec
> REQUEST_SLOW 7 slow requests are blocked > 32 sec
> 7 ops are blocked > 32.768 sec
> osds 35,51 have blocked requests > 32.768 sec
>
> Our details:
>
>   * system details:
> * Ubuntu 16.04
>  * Kernel 4.13.0-39
>  * 30 * 8 TB Disk (SEAGATE/ST8000NM0075)
>  * 3* Dell Power Edge R730xd (Firmware 2.50.50.50)
>* Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
>* 2*10GBITS SFP+ Network Adapters
>* 192GB RAM
>  * Pools are using replication factor 3, 2MB object size,
>85% write load, 1700 write IOPS/sec
>(ops mainly between 4k and 16k size), 300 read IOPS/sec
>   * we have the impression that this appears on deepscrub/scrub activity.
>   * Ceph 12.2.5, we alread played with the osd settings OSD Settings
> (our assumtion was that the problem is related to rocksdb compaction)
> bluestore cache kv max = 2147483648
> bluestore cache kv ratio = 0.9
> bluestore cache meta ratio = 0.1
> bluestore cache size hdd = 10737418240
>   * this type problem only appears on hdd/bluestore osds, ssd/bluestore
> osds did never experienced that problem
>   * the system is healthy, no swapping, no high load, no errors in dmesg
>
> I attached a log excerpt of osd.35 - probably this is useful for
> investigating the problem is someone owns deeper bluestore knowledge.
> (slow requests appeared on Sun Sep  2 21:00:35)
>
> Regards
> Marc
>
>
> Am 02.09.2018 um 15:50 schrieb Brett Chancellor:
> > The warnings look like this.
> >
> > 6 ops are blocked > 32.768 sec on osd.219
> > 1 osds have slow requests
> >
> > On Sun, Sep 2, 2018, 8:45 AM Alfredo Deza  > > wrote:
> >
> > On Sat, Sep 1, 2018 at 12:45 PM, Brett Chancellor
> > mailto:bchancel...@salesforce.com>>
> > wrote:
> > > Hi Cephers,
> > >   I am in the process of upgrading a cluster from Filestore to
> > bluestore,
> > > but I'm concerned about frequent warnings popping up against the
> new
> > > bluestore devices. I'm frequently seeing messages like this,
> > although the
> > > specific osd changes, it's always one of the few hosts I've
> > converted to
> > > bluestore.
> > >
> > > 6 ops are blocked > 32.768 sec on osd.219
> > > 1 osds have slow requests
> > >
> > > I'm running 12.2.4, have any of you seen similar issues? It
> > seems as though
> > > these messages pop up more frequently when one of the bluestore
> > pgs is
> > > involved in a scrub.  I'll include my bluestore creation process
> > below, in
> > > case that might cause an issue. (sdb, sdc, sdd are SATA, sde and
> > sdf are
> > > SSD)
> >
> > Would be useful to include what those warnings say. The ceph-volume
> > commands look OK to me
> >
> > >
> > >
> > > ## Process used to create osds
> > > sudo ceph-disk zap /dev/sdb /dev/sdc /dev/sdd /dev/sdd /dev/sde
> > /dev/sdf
> > > sudo ceph-volume lvm zap /dev/sdb
> > > sudo ceph-volume lvm zap /dev/sdc
> > > sudo ceph-volume lvm zap /dev/sdd
> > > sudo ceph-volume lvm zap

[ceph-users] Slow requests from bluestore osds

[Marc Schöchlin]

Hi,

we are also experiencing this type of behavior for some weeks on our not
so performance critical hdd pools.
We haven't spent so much time on this problem, because there are
currently more important tasks - but here are a few details:

Running the following loop results in the following output:

while true; do ceph health|grep -q HEALTH_OK || (date;  ceph health
detail); sleep 2; done

Sun Sep  2 20:59:47 CEST 2018
HEALTH_WARN 4 slow requests are blocked > 32 sec
REQUEST_SLOW 4 slow requests are blocked > 32 sec
    4 ops are blocked > 32.768 sec
    osd.43 has blocked requests > 32.768 sec
Sun Sep  2 20:59:50 CEST 2018
HEALTH_WARN 4 slow requests are blocked > 32 sec
REQUEST_SLOW 4 slow requests are blocked > 32 sec
    4 ops are blocked > 32.768 sec
    osd.43 has blocked requests > 32.768 sec
Sun Sep  2 20:59:52 CEST 2018
HEALTH_OK
Sun Sep  2 21:00:28 CEST 2018
HEALTH_WARN 1 slow requests are blocked > 32 sec
REQUEST_SLOW 1 slow requests are blocked > 32 sec
    1 ops are blocked > 32.768 sec
    osd.41 has blocked requests > 32.768 sec
Sun Sep  2 21:00:31 CEST 2018
HEALTH_WARN 7 slow requests are blocked > 32 sec
REQUEST_SLOW 7 slow requests are blocked > 32 sec
    7 ops are blocked > 32.768 sec
    osds 35,41 have blocked requests > 32.768 sec
Sun Sep  2 21:00:33 CEST 2018
HEALTH_WARN 7 slow requests are blocked > 32 sec
REQUEST_SLOW 7 slow requests are blocked > 32 sec
    7 ops are blocked > 32.768 sec
    osds 35,51 have blocked requests > 32.768 sec
Sun Sep  2 21:00:35 CEST 2018
HEALTH_WARN 7 slow requests are blocked > 32 sec
REQUEST_SLOW 7 slow requests are blocked > 32 sec
    7 ops are blocked > 32.768 sec
    osds 35,51 have blocked requests > 32.768 sec

Our details:

  * system details:
* Ubuntu 16.04
 * Kernel 4.13.0-39
 * 30 * 8 TB Disk (SEAGATE/ST8000NM0075)
 * 3* Dell Power Edge R730xd (Firmware 2.50.50.50)
   * Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
   * 2*10GBITS SFP+ Network Adapters
   * 192GB RAM
 * Pools are using replication factor 3, 2MB object size,
   85% write load, 1700 write IOPS/sec
   (ops mainly between 4k and 16k size), 300 read IOPS/sec
  * we have the impression that this appears on deepscrub/scrub activity.
  * Ceph 12.2.5, we alread played with the osd settings OSD Settings
(our assumtion was that the problem is related to rocksdb compaction)
bluestore cache kv max = 2147483648
bluestore cache kv ratio = 0.9
bluestore cache meta ratio = 0.1
bluestore cache size hdd = 10737418240
  * this type problem only appears on hdd/bluestore osds, ssd/bluestore
osds did never experienced that problem
  * the system is healthy, no swapping, no high load, no errors in dmesg

I attached a log excerpt of osd.35 - probably this is useful for
investigating the problem is someone owns deeper bluestore knowledge.
(slow requests appeared on Sun Sep  2 21:00:35)

Regards
Marc


Am 02.09.2018 um 15:50 schrieb Brett Chancellor:
> The warnings look like this. 
>
> 6 ops are blocked > 32.768 sec on osd.219
> 1 osds have slow requests
>
> On Sun, Sep 2, 2018, 8:45 AM Alfredo Deza  > wrote:
>
> On Sat, Sep 1, 2018 at 12:45 PM, Brett Chancellor
> mailto:bchancel...@salesforce.com>>
> wrote:
> > Hi Cephers,
> >   I am in the process of upgrading a cluster from Filestore to
> bluestore,
> > but I'm concerned about frequent warnings popping up against the new
> > bluestore devices. I'm frequently seeing messages like this,
> although the
> > specific osd changes, it's always one of the few hosts I've
> converted to
> > bluestore.
> >
> > 6 ops are blocked > 32.768 sec on osd.219
> > 1 osds have slow requests
> >
> > I'm running 12.2.4, have any of you seen similar issues? It
> seems as though
> > these messages pop up more frequently when one of the bluestore
> pgs is
> > involved in a scrub.  I'll include my bluestore creation process
> below, in
> > case that might cause an issue. (sdb, sdc, sdd are SATA, sde and
> sdf are
> > SSD)
>
> Would be useful to include what those warnings say. The ceph-volume
> commands look OK to me
>
> >
> >
> > ## Process used to create osds
> > sudo ceph-disk zap /dev/sdb /dev/sdc /dev/sdd /dev/sdd /dev/sde
> /dev/sdf
> > sudo ceph-volume lvm zap /dev/sdb
> > sudo ceph-volume lvm zap /dev/sdc
> > sudo ceph-volume lvm zap /dev/sdd
> > sudo ceph-volume lvm zap /dev/sde
> > sudo ceph-volume lvm zap /dev/sdf
> > sudo sgdisk -n 0:2048:+133GiB -t 0: -c 1:"ceph block.db sdb"
> /dev/sdf
> > sudo sgdisk -n 0:0:+133GiB -t 0: -c 2:"ceph block.db sdc"
> /dev/sdf
> > sudo sgdisk -n 0:0:+133GiB -t 0: -c 3:"ceph block.db sdd"
> /dev/sdf
> > sudo sgdisk -n 0:0:+133GiB -t 0: -c 4:"ceph block.db sde"
> /dev/sdf
> > sudo ceph-volume lvm create --bluestore --crush-device-class hdd
> --data

Re: [ceph-users] Slow requests from bluestore osds

[Brett Chancellor]

The warnings look like this.

6 ops are blocked > 32.768 sec on osd.219
1 osds have slow requests

On Sun, Sep 2, 2018, 8:45 AM Alfredo Deza  wrote:

> On Sat, Sep 1, 2018 at 12:45 PM, Brett Chancellor
>  wrote:
> > Hi Cephers,
> >   I am in the process of upgrading a cluster from Filestore to bluestore,
> > but I'm concerned about frequent warnings popping up against the new
> > bluestore devices. I'm frequently seeing messages like this, although the
> > specific osd changes, it's always one of the few hosts I've converted to
> > bluestore.
> >
> > 6 ops are blocked > 32.768 sec on osd.219
> > 1 osds have slow requests
> >
> > I'm running 12.2.4, have any of you seen similar issues? It seems as
> though
> > these messages pop up more frequently when one of the bluestore pgs is
> > involved in a scrub.  I'll include my bluestore creation process below,
> in
> > case that might cause an issue. (sdb, sdc, sdd are SATA, sde and sdf are
> > SSD)
>
> Would be useful to include what those warnings say. The ceph-volume
> commands look OK to me
>
> >
> >
> > ## Process used to create osds
> > sudo ceph-disk zap /dev/sdb /dev/sdc /dev/sdd /dev/sdd /dev/sde /dev/sdf
> > sudo ceph-volume lvm zap /dev/sdb
> > sudo ceph-volume lvm zap /dev/sdc
> > sudo ceph-volume lvm zap /dev/sdd
> > sudo ceph-volume lvm zap /dev/sde
> > sudo ceph-volume lvm zap /dev/sdf
> > sudo sgdisk -n 0:2048:+133GiB -t 0: -c 1:"ceph block.db sdb" /dev/sdf
> > sudo sgdisk -n 0:0:+133GiB -t 0: -c 2:"ceph block.db sdc" /dev/sdf
> > sudo sgdisk -n 0:0:+133GiB -t 0: -c 3:"ceph block.db sdd" /dev/sdf
> > sudo sgdisk -n 0:0:+133GiB -t 0: -c 4:"ceph block.db sde" /dev/sdf
> > sudo ceph-volume lvm create --bluestore --crush-device-class hdd --data
> > /dev/sdb --block.db /dev/sdf1
> > sudo ceph-volume lvm create --bluestore --crush-device-class hdd --data
> > /dev/sdc --block.db /dev/sdf2
> > sudo ceph-volume lvm create --bluestore --crush-device-class hdd --data
> > /dev/sdd --block.db /dev/sdf3
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests from bluestore osds

[Alfredo Deza]

On Sat, Sep 1, 2018 at 12:45 PM, Brett Chancellor
 wrote:
> Hi Cephers,
>   I am in the process of upgrading a cluster from Filestore to bluestore,
> but I'm concerned about frequent warnings popping up against the new
> bluestore devices. I'm frequently seeing messages like this, although the
> specific osd changes, it's always one of the few hosts I've converted to
> bluestore.
>
> 6 ops are blocked > 32.768 sec on osd.219
> 1 osds have slow requests
>
> I'm running 12.2.4, have any of you seen similar issues? It seems as though
> these messages pop up more frequently when one of the bluestore pgs is
> involved in a scrub.  I'll include my bluestore creation process below, in
> case that might cause an issue. (sdb, sdc, sdd are SATA, sde and sdf are
> SSD)

Would be useful to include what those warnings say. The ceph-volume
commands look OK to me

>
>
> ## Process used to create osds
> sudo ceph-disk zap /dev/sdb /dev/sdc /dev/sdd /dev/sdd /dev/sde /dev/sdf
> sudo ceph-volume lvm zap /dev/sdb
> sudo ceph-volume lvm zap /dev/sdc
> sudo ceph-volume lvm zap /dev/sdd
> sudo ceph-volume lvm zap /dev/sde
> sudo ceph-volume lvm zap /dev/sdf
> sudo sgdisk -n 0:2048:+133GiB -t 0: -c 1:"ceph block.db sdb" /dev/sdf
> sudo sgdisk -n 0:0:+133GiB -t 0: -c 2:"ceph block.db sdc" /dev/sdf
> sudo sgdisk -n 0:0:+133GiB -t 0: -c 3:"ceph block.db sdd" /dev/sdf
> sudo sgdisk -n 0:0:+133GiB -t 0: -c 4:"ceph block.db sde" /dev/sdf
> sudo ceph-volume lvm create --bluestore --crush-device-class hdd --data
> /dev/sdb --block.db /dev/sdf1
> sudo ceph-volume lvm create --bluestore --crush-device-class hdd --data
> /dev/sdc --block.db /dev/sdf2
> sudo ceph-volume lvm create --bluestore --crush-device-class hdd --data
> /dev/sdd --block.db /dev/sdf3
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Slow requests from bluestore osds

[Brett Chancellor]

Hi Cephers,
  I am in the process of upgrading a cluster from Filestore to bluestore,
but I'm concerned about frequent warnings popping up against the new
bluestore devices. I'm frequently seeing messages like this, although the
specific osd changes, it's always one of the few hosts I've converted to
bluestore.

6 ops are blocked > 32.768 sec on osd.219
1 osds have slow requests

I'm running 12.2.4, have any of you seen similar issues? It seems as though
these messages pop up more frequently when one of the bluestore pgs is
involved in a scrub.  I'll include my bluestore creation process below, in
case that might cause an issue. (sdb, sdc, sdd are SATA, sde and sdf are
SSD)


## Process used to create osds
sudo ceph-disk zap /dev/sdb /dev/sdc /dev/sdd /dev/sdd /dev/sde /dev/sdf
sudo ceph-volume lvm zap /dev/sdb
sudo ceph-volume lvm zap /dev/sdc
sudo ceph-volume lvm zap /dev/sdd
sudo ceph-volume lvm zap /dev/sde
sudo ceph-volume lvm zap /dev/sdf
sudo sgdisk -n 0:2048:+133GiB -t 0: -c 1:"ceph block.db sdb" /dev/sdf
sudo sgdisk -n 0:0:+133GiB -t 0: -c 2:"ceph block.db sdc" /dev/sdf
sudo sgdisk -n 0:0:+133GiB -t 0: -c 3:"ceph block.db sdd" /dev/sdf
sudo sgdisk -n 0:0:+133GiB -t 0: -c 4:"ceph block.db sde" /dev/sdf
sudo ceph-volume lvm create --bluestore --crush-device-class hdd --data
/dev/sdb --block.db /dev/sdf1
sudo ceph-volume lvm create --bluestore --crush-device-class hdd --data
/dev/sdc --block.db /dev/sdf2
sudo ceph-volume lvm create --bluestore --crush-device-class hdd --data
/dev/sdd --block.db /dev/sdf3
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests during OSD maintenance

[Konstantin Shalygin]

2. What is the best way to remove an OSD node from the cluster during
maintenance? ceph osd set noout is not the way to go, since no OSD's are
out during yum update and the node is still part of the cluster and will
handle I/O.
I think the best way is the combination of "ceph osd set noout" + stopping
the OSD services so the OSD node does not have any traffic anymore.



You are right.

ceph osd net noout

systemctl stop ceph-osd.target




k

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Slow requests during OSD maintenance

[sinan]

Hi,

On one of our OSD nodes I performed a "yum update" with Ceph repositories
disabled. So only the OS packages were being updated.

During and namely at the end of the yum update, the cluster started to
have slow/blocked requests and all VM's with Ceph storage backend had high
I/O load. After ~15 minutes the cluster health was OK.

The load on the OSD was not high, plenty of available memory + CPU.

1. How is it possible that an yum update on 1 node causes slow requests?
2. What is the best way to remove an OSD node from the cluster during
maintenance? ceph osd set noout is not the way to go, since no OSD's are
out during yum update and the node is still part of the cluster and will
handle I/O.
I think the best way is the combination of "ceph osd set noout" + stopping
the OSD services so the OSD node does not have any traffic anymore.

Any thoughts on this?

Thanks!
Sinan

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Slow Requests when deep scrubbing PGs that hold Bucket Index

[Christian Wimmer]

Hi,

I'm using ceph primarily for block storage (which works quite well) and as
an object gateway using the S3 API.

Here is some info about my system:
Ceph: 12.2.4, OS: Ubuntu 18.04
OSD: Bluestore
6 servers in total, about 60 OSDs, 2TB SSDs each, no HDDs, CFQ scheduler
20 GBit private network
20 GBit public network
Block storage and object storage runs on separate disks

Main use case:
Saving small (30KB - 2MB) objects in rgw buckets.
- dynamic bucket index resharding is disabled for now but I keep the index
objects per shard at about 100k.
- data pool: EC4+2
- index pool: replicated (3)
- atm around 500k objects in each bucket

My problem:
Sometimes, I get "slow request" warnings like so:
"[WRN] Health check update: 7 slow requests are blocked > 32 sec
(REQUEST_SLOW)"

It turned out that these warnings appear, whenever specific PGs are being
deep scrubbed.
After further investigation, I figured out that these PG's hold the bucket
index of the rados gateway.

I already tried some configuration changes like:
ceph tell osd.* injectargs '--osd_disk_thread_ioprio_priority 0'
ceph tell osd.* injectargs '--osd_disk_thread_ioprio_class idle'
ceph tell osd.* injectargs '--osd_scrub_sleep 1';
ceph tell osd.* injectargs '--osd_deep_scrub_stride 1048576'
ceph tell osd.* injectargs '--osd_scrub_chunk_max 1'
ceph tell osd.* injectargs '--osd_scrub_chunk_min 1'

This helped a lot to mitigate the effects but the problem is still there.

Does anybody else have this issue?

I have a few questions to better understand what's going on:

As far as I know, the bucket index is stored in rocksdb and the (empty)
objects in the index pool are just references to the data in rocksdb. Is
that correct?

How does a deep scrub affect rocksdb?
Does the index pool even need deep scrubbing or could I just disable it?

Also:

Does it make sense to create more index shards to get the objects per shard
down to let's say 50k or 20k?

Right now, I have about 500k objects per bucket. I want to increase that
number to a couple of hundred million objects. Do you see any problems with
that, provided that the bucket index is sharded appropriately?

Any help is appreciated. Let me know if you need anything like logs,
configs, etc.

Thanks!

Christian
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests

[Brad Hubbard]

On Mon, Jul 9, 2018 at 5:28 PM, Benjamin Naber
 wrote:
> Hi @all,
>
> Problem seems to be solved, afther downgrading from Kernel 4.17.2 to 
> 3.10.0-862.
> Anyone other have issues with newer Kernels and osd nodes?

I'd suggest you pursue that with whoever supports the kernel
exhibiting the problem.

>
> kind regards
>
> Ben
>
>> Brad Hubbard  hat am 5. Juli 2018 um 01:16 geschrieben:
>>
>>
>> On Wed, Jul 4, 2018 at 6:26 PM, Benjamin Naber  
>> wrote:
>> > Hi @all,
>> >
>> > im currently in testing for setup an production environment based on the 
>> > following OSD Nodes:
>> >
>> > CEPH Version: luminous 12.2.5
>> >
>> > 5x OSD Nodes with following specs:
>> >
>> > - 8 Core Intel Xeon 2,0 GHZ
>> >
>> > - 96GB Ram
>> >
>> > - 10x 1,92 TB Intel DC S4500 connectet via SATA
>> >
>> > - 4x 10 Gbit NIC 2 bonded via LACP for Backend Network 2 bonded via LACP 
>> > for Backend Network.
>> >
>> > if i run some fio benchmark via a VM that ist running on a RBD Device on a 
>> > KVM testing Host. the cluster always runs into slow request warning. Also 
>> > the performance goes heavy down.
>> >
>> > If i dump the osd that stucks, i get the following output:
>> >
>> > {
>> > "ops": [
>> > {
>> > "description": "osd_op(client.141944.0:359346834 13.1da 
>> > 13:5b8b7fd3:::rbd_data.170a3238e1f29.00be:head [write 
>> > 2097152~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.475879",
>> > "age": 287.180328,
>> > "duration": 287.180355,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346834
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.475879",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.476935",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.477547",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.477578",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.477614",
>> > "event": "waiting for subops from 5,26"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.484679",
>> > "event": "op_commit"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.484681",
>> > "event": "op_applied"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.485588",
>> > "event": "sub_op_commit_rec from 5"
>> > }
>> > ]
>> > }
>> > },
>> > {
>> > "description": "osd_op(client.141944.0:359346835 13.1da 
>> > 13:5b8b7fd3:::rbd_data.170a3238e1f29.00be:head [write 
>> > 3145728~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
>> > "initiated_at": "2018-07-04 10:00:49.477065",
>> > "age": 287.179143,
>> > "duration": 287.179221,
>> > "type_data": {
>> > "flag_point": "waiting for sub ops",
>> > "client_info": {
>> > "client": "client.141944",
>> > "client_addr": "10.111.90.1:0/3532639465",
>> > "tid": 359346835
>> > },
>> > "events": [
>> > {
>> > "time": "2018-07-04 10:00:49.477065",
>> > "event": "initiated"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.478116",
>> > "event": "queued_for_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.478178",
>> > "event": "reached_pg"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.478201",
>> > "event": "started"
>> > },
>> > {
>> > "time": "2018-07-04 10:00:49.478232",
>> > "event": "waiting for subops from 5,26"
>> > },
>> > 

Re: [ceph-users] Slow requests

[Brad Hubbard]

On Wed, Jul 4, 2018 at 6:26 PM, Benjamin Naber  wrote:
> Hi @all,
>
> im currently in testing for setup an production environment based on the 
> following OSD Nodes:
>
> CEPH Version: luminous 12.2.5
>
> 5x OSD Nodes with following specs:
>
> - 8 Core Intel Xeon 2,0 GHZ
>
> - 96GB Ram
>
> - 10x 1,92 TB Intel DC S4500 connectet via SATA
>
> - 4x 10 Gbit NIC 2 bonded via LACP for Backend Network 2 bonded via LACP for 
> Backend Network.
>
> if i run some fio benchmark via a VM that ist running on a RBD Device on a 
> KVM testing Host. the cluster always runs into slow request warning. Also the 
> performance goes heavy down.
>
> If i dump the osd that stucks, i get the following output:
>
> {
> "ops": [
> {
> "description": "osd_op(client.141944.0:359346834 13.1da 
> 13:5b8b7fd3:::rbd_data.170a3238e1f29.00be:head [write 
> 2097152~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
> "initiated_at": "2018-07-04 10:00:49.475879",
> "age": 287.180328,
> "duration": 287.180355,
> "type_data": {
> "flag_point": "waiting for sub ops",
> "client_info": {
> "client": "client.141944",
> "client_addr": "10.111.90.1:0/3532639465",
> "tid": 359346834
> },
> "events": [
> {
> "time": "2018-07-04 10:00:49.475879",
> "event": "initiated"
> },
> {
> "time": "2018-07-04 10:00:49.476935",
> "event": "queued_for_pg"
> },
> {
> "time": "2018-07-04 10:00:49.477547",
> "event": "reached_pg"
> },
> {
> "time": "2018-07-04 10:00:49.477578",
> "event": "started"
> },
> {
> "time": "2018-07-04 10:00:49.477614",
> "event": "waiting for subops from 5,26"
> },
> {
> "time": "2018-07-04 10:00:49.484679",
> "event": "op_commit"
> },
> {
> "time": "2018-07-04 10:00:49.484681",
> "event": "op_applied"
> },
> {
> "time": "2018-07-04 10:00:49.485588",
> "event": "sub_op_commit_rec from 5"
> }
> ]
> }
> },
> {
> "description": "osd_op(client.141944.0:359346835 13.1da 
> 13:5b8b7fd3:::rbd_data.170a3238e1f29.00be:head [write 
> 3145728~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
> "initiated_at": "2018-07-04 10:00:49.477065",
> "age": 287.179143,
> "duration": 287.179221,
> "type_data": {
> "flag_point": "waiting for sub ops",
> "client_info": {
> "client": "client.141944",
> "client_addr": "10.111.90.1:0/3532639465",
> "tid": 359346835
> },
> "events": [
> {
> "time": "2018-07-04 10:00:49.477065",
> "event": "initiated"
> },
> {
> "time": "2018-07-04 10:00:49.478116",
> "event": "queued_for_pg"
> },
> {
> "time": "2018-07-04 10:00:49.478178",
> "event": "reached_pg"
> },
> {
> "time": "2018-07-04 10:00:49.478201",
> "event": "started"
> },
> {
> "time": "2018-07-04 10:00:49.478232",
> "event": "waiting for subops from 5,26"
> },
> {
> "time": "2018-07-04 10:00:49.484695",
> "event": "op_commit"
> },
> {
> "time": "2018-07-04 10:00:49.484696",
> "event": "op_applied"
> },
> {
> "time": "2018-07-04 10:00:49.485621",
> "event": "sub_op_commit_rec from 5"
> }
> ]
> }
> },
> {
> "description": "osd_op(client.141944.0:359346440 13.11d 
> 13:b8afbe4a:::rbd_data.170a3238e1f29.005c:head [write 0~1048576] 
> snapc 0=[] ondisk+write+known_

Re: [ceph-users] Slow requests

[Benjamin Naber]

hi Caspar,

ty for the reply. ive updatet all SSDs to actual firmware. Still having the 
same error. the strange thing is that this issue switches from node to node and 
from osd to osd.

HEALTH_WARN 4 slow requests are blocked > 32 sec
REQUEST_SLOW 4 slow requests are blocked > 32 sec
    1 ops are blocked > 1048.58 sec
    3 ops are blocked > 262.144 sec
    osds 20,21 have blocked requests > 262.144 sec
    osd.12 has blocked requests > 1048.58 sec

also there is no symptom that gives me any idea what could be the problem. 
Syslog etc. also say nothing. 

any other ideas ?

Kind regards and thanks for your help

Ben

> Caspar Smit  hat am 4. Juli 2018 um 11:32 
> geschrieben: 
> 
> Hi Ben,
> 
> At first glance i would say the CPU's are a bit weak for this setup. 
> Recommended is to have at least 1 core per OSD. Since you have 8 cores and 10 
> OSD's there isn't much left for other processes.
> 
> Furthermore, did you upgrade the firmware of those DC S4500's to the latest 
> firmware? ( SCV10142)
> If not, you can upgrade them via the Intel Data Center upgrade tool: 
> https://downloadcenter.intel.com/download/27863?v=t
> 
> Kind regards,
> 
> Caspar 
> 
> 2018-07-04 10:26 GMT+02:00 Benjamin Naber : 
> 
> > Hi @all, 
> > 
> >  im currently in testing for setup an production environment based on the 
> > following OSD Nodes: 
> > 
> >  CEPH Version: luminous 12.2.5 
> > 
> >  5x OSD Nodes with following specs: 
> > 
> >  - 8 Core Intel Xeon 2,0 GHZ 
> > 
> >  - 96GB Ram 
> > 
> >  - 10x 1,92 TB Intel DC S4500 connectet via SATA 
> > 
> >  - 4x 10 Gbit NIC 2 bonded via LACP for Backend Network 2 bonded via LACP 
> > for Backend Network. 
> > 
> >  if i run some fio benchmark via a VM that ist running on a RBD Device on a 
> > KVM testing Host. the cluster always runs into slow request warning. Also 
> > the performance goes heavy down. 
> > 
> >  If i dump the osd that stucks, i get the following output: 
> > 
> >  { 
> >      "ops": [ 
> >      { 
> >      "description": "osd_op(client.141944.0: 359346834 13.1da 
> > 13:5b8b7fd3:::rbd_data. 170a3238e1f29. 00be:head [write 
> > 2097152~1048576] snapc 0=[] ondisk+write+known_if_ redirected e2755)", 
> >      "initiated_at": "2018-07-04 10:00:49.475879", 
> >      "age": 287.180328, 
> >      "duration": 287.180355, 
> >      "type_data": { 
> >      "flag_point": "waiting for sub ops", 
> >      "client_info": { 
> >      "client": "client.141944", 
> >      "client_addr": " 
> > [10.111.90.1:0/3532639465](http://10.111.90.1:0/3532639465)", 
> >      "tid": 359346834 
> >      }, 
> >      "events": [ 
> >      { 
> >      "time": "2018-07-04 10:00:49.475879", 
> >      "event": "initiated" 
> >      }, 
> >      { 
> >      "time": "2018-07-04 10:00:49.476935", 
> >      "event": "queued_for_pg" 
> >      }, 
> >      { 
> >      "time": "2018-07-04 10:00:49.477547", 
> >      "event": "reached_pg" 
> >      }, 
> >      { 
> >      "time": "2018-07-04 10:00:49.477578", 
> >      "event": "started" 
> >      }, 
> >      { 
> >      "time": "2018-07-04 10:00:49.477614", 
> >      "event": "waiting for subops from 5,26" 
> >      }, 
> >      { 
> >      "time": "2018-07-04 10:00:49.484679", 
> >      "event": "op_commit" 
> >      }, 
> >      { 
> >      "time": "2018-07-04 10:00:49.484681", 
> >      "event": "op_applied" 
> >      }, 
> >      { 
> >      "time": "2018-07-04 10:00:49.485588", 
> >      "event": "sub_op_commit_rec from 5" 
> >      } 
> >      ] 
> >      } 
> >      }, 
> >      { 
> >      "description": "osd_op(client.141944.0: 359346835 13.1da 
> > 13:5b8b7fd3:::rbd_data. 170a3238e1f29. 00be:head [write 
> > 3145728~1048576] snapc 0=[] ondisk+write+known_if_ redirected e2755)", 
> >      "initiated_at": "2018-07-04 10:00:49.477065", 
> >      "age": 287.179143, 
> >      "duration": 287.179221, 
> >      "type_data": { 
> >      "flag_point": "waiting for sub ops", 
> >      "client_info": { 
> >      "client": "client.141944", 
> >      "client_addr": " 
> > [10.111.90.1:0/3532639465](http://10.111.90.1:0/3532639465)", 
> >      "tid": 359346835 
> >      }

Re: [ceph-users] Slow requests

[Caspar Smit]

Hi Ben,

At first glance i would say the CPU's are a bit weak for this setup.
Recommended is to have at least 1 core per OSD. Since you have 8 cores and
10 OSD's there isn't much left for other processes.

Furthermore, did you upgrade the firmware of those DC S4500's to the latest
firmware? (SCV10142)
If not, you can upgrade them via the Intel Data Center upgrade tool:
https://downloadcenter.intel.com/download/27863?v=t

Kind regards,
Caspar

2018-07-04 10:26 GMT+02:00 Benjamin Naber :

> Hi @all,
>
> im currently in testing for setup an production environment based on the
> following OSD Nodes:
>
> CEPH Version: luminous 12.2.5
>
> 5x OSD Nodes with following specs:
>
> - 8 Core Intel Xeon 2,0 GHZ
>
> - 96GB Ram
>
> - 10x 1,92 TB Intel DC S4500 connectet via SATA
>
> - 4x 10 Gbit NIC 2 bonded via LACP for Backend Network 2 bonded via LACP
> for Backend Network.
>
> if i run some fio benchmark via a VM that ist running on a RBD Device on a
> KVM testing Host. the cluster always runs into slow request warning. Also
> the performance goes heavy down.
>
> If i dump the osd that stucks, i get the following output:
>
> {
> "ops": [
> {
> "description": "osd_op(client.141944.0:359346834 13.1da
> 13:5b8b7fd3:::rbd_data.170a3238e1f29.00be:head [write
> 2097152~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
> "initiated_at": "2018-07-04 10:00:49.475879",
> "age": 287.180328,
> "duration": 287.180355,
> "type_data": {
> "flag_point": "waiting for sub ops",
> "client_info": {
> "client": "client.141944",
> "client_addr": "10.111.90.1:0/3532639465",
> "tid": 359346834
> },
> "events": [
> {
> "time": "2018-07-04 10:00:49.475879",
> "event": "initiated"
> },
> {
> "time": "2018-07-04 10:00:49.476935",
> "event": "queued_for_pg"
> },
> {
> "time": "2018-07-04 10:00:49.477547",
> "event": "reached_pg"
> },
> {
> "time": "2018-07-04 10:00:49.477578",
> "event": "started"
> },
> {
> "time": "2018-07-04 10:00:49.477614",
> "event": "waiting for subops from 5,26"
> },
> {
> "time": "2018-07-04 10:00:49.484679",
> "event": "op_commit"
> },
> {
> "time": "2018-07-04 10:00:49.484681",
> "event": "op_applied"
> },
> {
> "time": "2018-07-04 10:00:49.485588",
> "event": "sub_op_commit_rec from 5"
> }
> ]
> }
> },
> {
> "description": "osd_op(client.141944.0:359346835 13.1da
> 13:5b8b7fd3:::rbd_data.170a3238e1f29.00be:head [write
> 3145728~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
> "initiated_at": "2018-07-04 10:00:49.477065",
> "age": 287.179143,
> "duration": 287.179221,
> "type_data": {
> "flag_point": "waiting for sub ops",
> "client_info": {
> "client": "client.141944",
> "client_addr": "10.111.90.1:0/3532639465",
> "tid": 359346835
> },
> "events": [
> {
> "time": "2018-07-04 10:00:49.477065",
> "event": "initiated"
> },
> {
> "time": "2018-07-04 10:00:49.478116",
> "event": "queued_for_pg"
> },
> {
> "time": "2018-07-04 10:00:49.478178",
> "event": "reached_pg"
> },
> {
> "time": "2018-07-04 10:00:49.478201",
> "event": "started"
> },
> {
> "time": "2018-07-04 10:00:49.478232",
> "event": "waiting for subops from 5,26"
> },
> {
> "time": "2018-07-04 10:00:49.484695",
> "event": "op_commit"
> },
> {
> "time": "2018-07-04 10:00:49.484696",
> "event": "op_applied"
> 

[ceph-users] Slow requests

[Benjamin Naber]

Hi @all,

im currently in testing for setup an production environment based on the 
following OSD Nodes:

CEPH Version: luminous 12.2.5

5x OSD Nodes with following specs:

- 8 Core Intel Xeon 2,0 GHZ

- 96GB Ram

- 10x 1,92 TB Intel DC S4500 connectet via SATA

- 4x 10 Gbit NIC 2 bonded via LACP for Backend Network 2 bonded via LACP for 
Backend Network.

if i run some fio benchmark via a VM that ist running on a RBD Device on a KVM 
testing Host. the cluster always runs into slow request warning. Also the 
performance goes heavy down.

If i dump the osd that stucks, i get the following output:

{
    "ops": [
    {
    "description": "osd_op(client.141944.0:359346834 13.1da 
13:5b8b7fd3:::rbd_data.170a3238e1f29.00be:head [write 
2097152~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
    "initiated_at": "2018-07-04 10:00:49.475879",
    "age": 287.180328,
    "duration": 287.180355,
    "type_data": {
    "flag_point": "waiting for sub ops",
    "client_info": {
    "client": "client.141944",
    "client_addr": "10.111.90.1:0/3532639465",
    "tid": 359346834
    },
    "events": [
    {
    "time": "2018-07-04 10:00:49.475879",
    "event": "initiated"
    },
    {
    "time": "2018-07-04 10:00:49.476935",
    "event": "queued_for_pg"
    },
    {
    "time": "2018-07-04 10:00:49.477547",
    "event": "reached_pg"
    },
    {
    "time": "2018-07-04 10:00:49.477578",
    "event": "started"
    },
    {
    "time": "2018-07-04 10:00:49.477614",
    "event": "waiting for subops from 5,26"
    },
    {
    "time": "2018-07-04 10:00:49.484679",
    "event": "op_commit"
    },
    {
    "time": "2018-07-04 10:00:49.484681",
    "event": "op_applied"
    },
    {
    "time": "2018-07-04 10:00:49.485588",
    "event": "sub_op_commit_rec from 5"
    }
    ]
    }
    },
    {
    "description": "osd_op(client.141944.0:359346835 13.1da 
13:5b8b7fd3:::rbd_data.170a3238e1f29.00be:head [write 
3145728~1048576] snapc 0=[] ondisk+write+known_if_redirected e2755)",
    "initiated_at": "2018-07-04 10:00:49.477065",
    "age": 287.179143,
    "duration": 287.179221,
    "type_data": {
    "flag_point": "waiting for sub ops",
    "client_info": {
    "client": "client.141944",
    "client_addr": "10.111.90.1:0/3532639465",
    "tid": 359346835
    },
    "events": [
    {
    "time": "2018-07-04 10:00:49.477065",
    "event": "initiated"
    },
    {
    "time": "2018-07-04 10:00:49.478116",
    "event": "queued_for_pg"
    },
    {
    "time": "2018-07-04 10:00:49.478178",
    "event": "reached_pg"
    },
    {
    "time": "2018-07-04 10:00:49.478201",
    "event": "started"
    },
    {
    "time": "2018-07-04 10:00:49.478232",
    "event": "waiting for subops from 5,26"
    },
    {
    "time": "2018-07-04 10:00:49.484695",
    "event": "op_commit"
    },
    {
    "time": "2018-07-04 10:00:49.484696",
    "event": "op_applied"
    },
    {
    "time": "2018-07-04 10:00:49.485621",
    "event": "sub_op_commit_rec from 5"
    }
    ]
    }
    },
    {
    "description": "osd_op(client.141944.0:359346440 13.11d 
13:b8afbe4a:::rbd_data.170a3238e1f29.005c:head [write 0~1048576] 
snapc 0=[] ondisk+write+known_if_redirected e2755)",
    "initiated_at": "2018-07-04 10:00:49.091127",
    "age": 287.565080,
    "duration": 287.565196,
    "type_data": {
    "flag_point": "waiting for sub ops",
    "client_info": {
    "client": "client.141944

Re: [ceph-users] slow requests are blocked

[Paul Emmerich]

By looking at the operations that are slow in your dump_*_ops command.
We've found that it's best to move all the metadata stuff for RGW onto
SSDs, i.e., all pools except the actual data pool.

But that depends on your use case and whether the slow requests you are
seeing is actually a problem for you. Maybe you
don't care too much about the metadata operations.

Paul

2018-05-16 12:25 GMT+02:00 Grigory Murashov :

> Hello Paul!
>
> Thanks for your answer.
>
> How did you understand it's RGW Metadata stuff?
>
> No, I don't use any SSDs. Where I can find out more about Metadata pools,
> using SSD etc?..
>
> Thanks.
>
> Grigory Murashov
> Voximplant
>
> 15.05.2018 23:42, Paul Emmerich пишет:
>
> Looks like it's mostly RGW metadata stuff; are you running your non-data
> RGW pools on SSDs (you should, that can help *a lot*)?
>
>
> Paul
>
> 2018-05-15 18:49 GMT+02:00 Grigory Murashov :
>
>> Hello guys!
>>
>> I collected output of ceph daemon osd.16 dump_ops_in_flight and ceph
>> daemon osd.16 dump_historic_ops.
>>
>> Here is the output of ceph heath details in the moment of problem
>>
>> HEALTH_WARN 20 slow requests are blocked > 32 sec
>> REQUEST_SLOW 20 slow requests are blocked > 32 sec
>> 20 ops are blocked > 65.536 sec
>> osds 16,27,29 have blocked requests > 65.536 sec
>>
>> So I grab logs from osd.16.
>>
>> The file is attached.  Could you please help to translate?
>>
>> Thanks in advance.
>>
>> Grigory Murashov
>> Voximplant
>>
>> 14.05.2018 18:14, Grigory Murashov пишет:
>>
>> Hello David!
>>
>> 2. I set it up 10/10
>>
>> 3. Thanks, my problem was I did it on host where was no osd.15 daemon.
>>
>> Could you please help to read osd logs?
>>
>> Here is a part from ceph.log
>>
>> 2018-05-14 13:46:32.644323 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553895 : cluster [INF] Cluster is now healthy
>> 2018-05-14 13:46:43.741921 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553896 : cluster [WRN] Health check failed: 21 slow
>> requests are blocked > 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:46:49.746994 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553897 : cluster [WRN] Health check update: 23 slow
>> requests are blocked > 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:46:55.752314 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553900 : cluster [WRN] Health check update: 3 slow
>> requests are blocked > 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:47:01.030686 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553901 : cluster [WRN] Health check update: 4 slow
>> requests are blocked > 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:47:07.764236 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553903 : cluster [WRN] Health check update: 32 slow
>> requests are blocked > 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:47:13.770833 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553904 : cluster [WRN] Health check update: 21 slow
>> requests are blocked > 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:47:17.774530 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553905 : cluster [INF] Health check cleared:
>> REQUEST_SLOW (was: 12 slow requests are blocked > 32 sec)
>> 2018-05-14 13:47:17.774582 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553906 : cluster [INF] Cluster is now healthy
>>
>> At 13-47 I had a problem with osd.21
>>
>> 1. Ceph Health (storage-ru1-osd1.voximplant.com:ceph.health): HEALTH_WARN
>> {u'REQUEST_SLOW': {u'severity': u'HEALTH_WARN', u'summary': {u'message': u'4 
>> slow requests are blocked > 32 sec'}}}
>> HEALTH_WARN 4 slow requests are blocked > 32 sec
>> REQUEST_SLOW 4 slow requests are blocked > 32 sec
>> 2 ops are blocked > 65.536 sec
>> 2 ops are blocked > 32.768 sec
>> osd.21 has blocked requests > 65.536 sec
>>
>> Here is a part from ceph-osd.21.log
>>
>> 2018-05-14 13:47:06.891399 7fb806dd6700 10 osd.21 pg_epoch: 236 pg[2.0( v
>> 236'297 (0'0,236'297] local-lis/les=223/224 n=1 ec=119/119 lis/c 223/223
>> les/c/f 224/224/0 223/223/212) [21,29,15]
>> r=0 lpr=223 crt=236'297 lcod 236'296 mlcod 236'296 active+clean]
>> dropping ondisk_read_lock
>> 2018-05-14 13:47:06.891435 7fb806dd6700 10 osd.21 236 dequeue_op
>> 0x56453b753f80 finish
>> 2018-05-14 13:47:07.111388 7fb8185f9700 10 osd.21 236 tick
>> 2018-05-14 13:47:07.111398 7fb8185f9700 10 osd.21 236 do_waiters -- start
>> 2018-05-14 13:47:07.111401 7fb8185f9700 10 osd.21 236 do_waiters -- finish
>> 2018-05-14 13:47:07.800421 7fb817df8700 10 osd.21 236
>> tick_without_osd_lock
>> 2018-05-14 13:47:07.800444 7fb817df8700 10 osd.21 236
>> promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0  bytes;
>> target 25 obj/sec or 5120 k bytes/sec
>> 2018-05-14 13:47:07.800449 7fb817df8700 10 osd.21 236
>> promote_throttle_recalibrate  actual 0, actual/prob ratio 1, adjusted
>> new_prob 1000, prob 1000 -> 1000
>> 2018-05-14 13:47:08.111470 7fb8185f9700 10 osd.21 236 tick
>> 2018-05-14 13:47:08.111483 7fb8185f9700 10 osd.21 236 do_waiters -- start
>> 2018-05-14 13:47:08.111485 7fb8185f9700 10 osd.21 236 do_waiters -- f

Re: [ceph-users] slow requests are blocked

[Grigory Murashov]

Hello Paul!

Thanks for your answer.

How did you understand it's RGW Metadata stuff?

No, I don't use any SSDs. Where I can find out more about Metadata 
pools, using SSD etc?..


Thanks.

Grigory Murashov
Voximplant

15.05.2018 23:42, Paul Emmerich пишет:
Looks like it's mostly RGW metadata stuff; are you running your 
non-data RGW pools on SSDs (you should, that can help *a lot*)?



Paul

2018-05-15 18:49 GMT+02:00 Grigory Murashov >:


Hello guys!

I collected output of ceph daemon osd.16 dump_ops_in_flight and
ceph daemon osd.16 dump_historic_ops.

Here is the output of ceph heath details in the moment of problem

HEALTH_WARN 20 slow requests are blocked > 32 sec
REQUEST_SLOW 20 slow requests are blocked > 32 sec
    20 ops are blocked > 65.536 sec
    osds 16,27,29 have blocked requests > 65.536 sec

So I grab logs from osd.16.

The file is attached.  Could you please help to translate?

Thanks in advance.

Grigory Murashov
Voximplant

14.05.2018 18:14, Grigory Murashov пишет:


Hello David!

2. I set it up 10/10

3. Thanks, my problem was I did it on host where was no osd.15
daemon.

Could you please help to read osd logs?

Here is a part from ceph.log

2018-05-14 13:46:32.644323 mon.storage-ru1-osd1 mon.0
185.164.149.2:6789/0  553895 :
cluster [INF] Cluster is now healthy
2018-05-14 13:46:43.741921 mon.storage-ru1-osd1 mon.0
185.164.149.2:6789/0  553896 :
cluster [WRN] Health check failed: 21 slow requests are blocked >
32 sec (REQUEST_SLOW)
2018-05-14 13:46:49.746994 mon.storage-ru1-osd1 mon.0
185.164.149.2:6789/0  553897 :
cluster [WRN] Health check update: 23 slow requests are blocked >
32 sec (REQUEST_SLOW)
2018-05-14 13:46:55.752314 mon.storage-ru1-osd1 mon.0
185.164.149.2:6789/0  553900 :
cluster [WRN] Health check update: 3 slow requests are blocked >
32 sec (REQUEST_SLOW)
2018-05-14 13:47:01.030686 mon.storage-ru1-osd1 mon.0
185.164.149.2:6789/0  553901 :
cluster [WRN] Health check update: 4 slow requests are blocked >
32 sec (REQUEST_SLOW)
2018-05-14 13:47:07.764236 mon.storage-ru1-osd1 mon.0
185.164.149.2:6789/0  553903 :
cluster [WRN] Health check update: 32 slow requests are blocked >
32 sec (REQUEST_SLOW)
2018-05-14 13:47:13.770833 mon.storage-ru1-osd1 mon.0
185.164.149.2:6789/0  553904 :
cluster [WRN] Health check update: 21 slow requests are blocked >
32 sec (REQUEST_SLOW)
2018-05-14 13:47:17.774530 mon.storage-ru1-osd1 mon.0
185.164.149.2:6789/0  553905 :
cluster [INF] Health check cleared: REQUEST_SLOW (was: 12 slow
requests are blocked > 32 sec)
2018-05-14 13:47:17.774582 mon.storage-ru1-osd1 mon.0
185.164.149.2:6789/0  553906 :
cluster [INF] Cluster is now healthy

At 13-47 I had a problem with osd.21

1. Ceph Health (storage-ru1-osd1.voximplant.com:ceph.health): HEALTH_WARN
{u'REQUEST_SLOW': {u'severity': u'HEALTH_WARN', u'summary': {u'message': u'4 
slow requests are blocked > 32 sec'}}}
HEALTH_WARN 4 slow requests are blocked > 32 sec
REQUEST_SLOW 4 slow requests are blocked > 32 sec
 2 ops are blocked > 65.536 sec
 2 ops are blocked > 32.768 sec
 osd.21 has blocked requests > 65.536 sec

Here is a part from ceph-osd.21.log

2018-05-14 13:47:06.891399 7fb806dd6700 10 osd.21 pg_epoch: 236
pg[2.0( v 236'297 (0'0,236'297] local-lis/les=223/224 n=1
ec=119/119 lis/c 223/223 les/c/f 224/224/0 223/223/212) [21,29,15]
r=0 lpr=223 crt=236'297 lcod 236'296 mlcod 236'296 active+clean] 
dropping ondisk_read_lock
2018-05-14 13:47:06.891435 7fb806dd6700 10 osd.21 236 dequeue_op
0x56453b753f80 finish
2018-05-14 13:47:07.111388 7fb8185f9700 10 osd.21 236 tick
2018-05-14 13:47:07.111398 7fb8185f9700 10 osd.21 236 do_waiters
-- start
2018-05-14 13:47:07.111401 7fb8185f9700 10 osd.21 236 do_waiters
-- finish
2018-05-14 13:47:07.800421 7fb817df8700 10 osd.21 236
tick_without_osd_lock
2018-05-14 13:47:07.800444 7fb817df8700 10 osd.21 236
promote_throttle_recalibrate 0 attempts, promoted 0 objects and
0  bytes; target 25 obj/sec or 5120 k bytes/sec
2018-05-14 13:47:07.800449 7fb817df8700 10 osd.21 236
promote_throttle_recalibrate  actual 0, actual/prob ratio 1,
adjusted new_prob 1000, prob 1000 -> 1000
2018-05-14 13:47:08.111470 7fb8185f9700 10 osd.21 236 tick
2018-05-14 13:47:08.111483 7fb8185f9700 10 osd.21 236 do_waiters
-- start
2018-05-14 13:47:08.111485 7fb8185f9700 10 osd.21 236 do_waiters
-- finish
2018-05-14 13:47:08.181070 7fb8055d3700 10

Re: [ceph-users] slow requests are blocked

[David Turner]

I've been happening into slow requests with my rgw metadata pools just this
week. I tracked it down because the slow requests were on my nmve osds. I
haven't solved the issue yet, but I can confirm that no resharding was
taking place and that the auto-resharder is working as all of my larger
buckets have different amounts of index shards.

I'm interested to see if there are some settings we should try, or things
we should check on, to resolve the rgw meta-pools slow requests.

On Tue, May 15, 2018, 4:42 PM Paul Emmerich  wrote:

> Looks like it's mostly RGW metadata stuff; are you running your non-data
> RGW pools on SSDs (you should, that can help *a lot*)?
>
>
> Paul
>
> 2018-05-15 18:49 GMT+02:00 Grigory Murashov :
>
>> Hello guys!
>>
>> I collected output of ceph daemon osd.16 dump_ops_in_flight and ceph
>> daemon osd.16 dump_historic_ops.
>>
>> Here is the output of ceph heath details in the moment of problem
>>
>> HEALTH_WARN 20 slow requests are blocked > 32 sec
>> REQUEST_SLOW 20 slow requests are blocked > 32 sec
>> 20 ops are blocked > 65.536 sec
>> osds 16,27,29 have blocked requests > 65.536 sec
>>
>> So I grab logs from osd.16.
>>
>> The file is attached.  Could you please help to translate?
>>
>> Thanks in advance.
>>
>> Grigory Murashov
>> Voximplant
>>
>> 14.05.2018 18:14, Grigory Murashov пишет:
>>
>> Hello David!
>>
>> 2. I set it up 10/10
>>
>> 3. Thanks, my problem was I did it on host where was no osd.15 daemon.
>>
>> Could you please help to read osd logs?
>>
>> Here is a part from ceph.log
>>
>> 2018-05-14 13:46:32.644323 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553895 : cluster [INF] Cluster is now healthy
>> 2018-05-14 13:46:43.741921 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553896 : cluster [WRN] Health check failed: 21 slow
>> requests are blocked > 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:46:49.746994 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553897 : cluster [WRN] Health check update: 23 slow
>> requests are blocked > 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:46:55.752314 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553900 : cluster [WRN] Health check update: 3 slow
>> requests are blocked > 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:47:01.030686 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553901 : cluster [WRN] Health check update: 4 slow
>> requests are blocked > 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:47:07.764236 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553903 : cluster [WRN] Health check update: 32 slow
>> requests are blocked > 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:47:13.770833 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553904 : cluster [WRN] Health check update: 21 slow
>> requests are blocked > 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:47:17.774530 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553905 : cluster [INF] Health check cleared:
>> REQUEST_SLOW (was: 12 slow requests are blocked > 32 sec)
>> 2018-05-14 13:47:17.774582 mon.storage-ru1-osd1 mon.0
>> 185.164.149.2:6789/0 553906 : cluster [INF] Cluster is now healthy
>>
>> At 13-47 I had a problem with osd.21
>>
>> 1. Ceph Health (storage-ru1-osd1.voximplant.com:ceph.health): HEALTH_WARN
>> {u'REQUEST_SLOW': {u'severity': u'HEALTH_WARN', u'summary': {u'message': u'4 
>> slow requests are blocked > 32 sec'}}}
>> HEALTH_WARN 4 slow requests are blocked > 32 sec
>> REQUEST_SLOW 4 slow requests are blocked > 32 sec
>> 2 ops are blocked > 65.536 sec
>> 2 ops are blocked > 32.768 sec
>> osd.21 has blocked requests > 65.536 sec
>>
>> Here is a part from ceph-osd.21.log
>>
>> 2018-05-14 13:47:06.891399 7fb806dd6700 10 osd.21 pg_epoch: 236 pg[2.0( v
>> 236'297 (0'0,236'297] local-lis/les=223/224 n=1 ec=119/119 lis/c 223/223
>> les/c/f 224/224/0 223/223/212) [21,29,15]
>> r=0 lpr=223 crt=236'297 lcod 236'296 mlcod 236'296 active+clean]
>> dropping ondisk_read_lock
>> 2018-05-14 13:47:06.891435 7fb806dd6700 10 osd.21 236 dequeue_op
>> 0x56453b753f80 finish
>> 2018-05-14 13:47:07.111388 7fb8185f9700 10 osd.21 236 tick
>> 2018-05-14 13:47:07.111398 7fb8185f9700 10 osd.21 236 do_waiters -- start
>> 2018-05-14 13:47:07.111401 7fb8185f9700 10 osd.21 236 do_waiters -- finish
>> 2018-05-14 13:47:07.800421 7fb817df8700 10 osd.21 236
>> tick_without_osd_lock
>> 2018-05-14 13:47:07.800444 7fb817df8700 10 osd.21 236
>> promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0  bytes;
>> target 25 obj/sec or 5120 k bytes/sec
>> 2018-05-14 13:47:07.800449 7fb817df8700 10 osd.21 236
>> promote_throttle_recalibrate  actual 0, actual/prob ratio 1, adjusted
>> new_prob 1000, prob 1000 -> 1000
>> 2018-05-14 13:47:08.111470 7fb8185f9700 10 osd.21 236 tick
>> 2018-05-14 13:47:08.111483 7fb8185f9700 10 osd.21 236 do_waiters -- start
>> 2018-05-14 13:47:08.111485 7fb8185f9700 10 osd.21 236 do_waiters -- finish
>> 2018-05-14 13:47:08.181070 7fb8055d3700 10 osd.21 236 dequeue_op
>> 0x564539651000 prio 63 cost 0 latency 0.000143
>> osd_op(client.2597258.0:213844298 6.1d4

Re: [ceph-users] slow requests are blocked

[Paul Emmerich]

Looks like it's mostly RGW metadata stuff; are you running your non-data
RGW pools on SSDs (you should, that can help *a lot*)?


Paul

2018-05-15 18:49 GMT+02:00 Grigory Murashov :

> Hello guys!
>
> I collected output of ceph daemon osd.16 dump_ops_in_flight and ceph
> daemon osd.16 dump_historic_ops.
>
> Here is the output of ceph heath details in the moment of problem
>
> HEALTH_WARN 20 slow requests are blocked > 32 sec
> REQUEST_SLOW 20 slow requests are blocked > 32 sec
> 20 ops are blocked > 65.536 sec
> osds 16,27,29 have blocked requests > 65.536 sec
>
> So I grab logs from osd.16.
>
> The file is attached.  Could you please help to translate?
>
> Thanks in advance.
>
> Grigory Murashov
> Voximplant
>
> 14.05.2018 18:14, Grigory Murashov пишет:
>
> Hello David!
>
> 2. I set it up 10/10
>
> 3. Thanks, my problem was I did it on host where was no osd.15 daemon.
>
> Could you please help to read osd logs?
>
> Here is a part from ceph.log
>
> 2018-05-14 13:46:32.644323 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0
> 553895 : cluster [INF] Cluster is now healthy
> 2018-05-14 13:46:43.741921 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0
> 553896 : cluster [WRN] Health check failed: 21 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-05-14 13:46:49.746994 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0
> 553897 : cluster [WRN] Health check update: 23 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-05-14 13:46:55.752314 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0
> 553900 : cluster [WRN] Health check update: 3 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-05-14 13:47:01.030686 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0
> 553901 : cluster [WRN] Health check update: 4 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-05-14 13:47:07.764236 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0
> 553903 : cluster [WRN] Health check update: 32 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-05-14 13:47:13.770833 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0
> 553904 : cluster [WRN] Health check update: 21 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-05-14 13:47:17.774530 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0
> 553905 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 12 slow
> requests are blocked > 32 sec)
> 2018-05-14 13:47:17.774582 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0
> 553906 : cluster [INF] Cluster is now healthy
>
> At 13-47 I had a problem with osd.21
>
> 1. Ceph Health (storage-ru1-osd1.voximplant.com:ceph.health): HEALTH_WARN
> {u'REQUEST_SLOW': {u'severity': u'HEALTH_WARN', u'summary': {u'message': u'4 
> slow requests are blocked > 32 sec'}}}
> HEALTH_WARN 4 slow requests are blocked > 32 sec
> REQUEST_SLOW 4 slow requests are blocked > 32 sec
> 2 ops are blocked > 65.536 sec
> 2 ops are blocked > 32.768 sec
> osd.21 has blocked requests > 65.536 sec
>
> Here is a part from ceph-osd.21.log
>
> 2018-05-14 13:47:06.891399 7fb806dd6700 10 osd.21 pg_epoch: 236 pg[2.0( v
> 236'297 (0'0,236'297] local-lis/les=223/224 n=1 ec=119/119 lis/c 223/223
> les/c/f 224/224/0 223/223/212) [21,29,15]
> r=0 lpr=223 crt=236'297 lcod 236'296 mlcod 236'296 active+clean]  dropping
> ondisk_read_lock
> 2018-05-14 13:47:06.891435 7fb806dd6700 10 osd.21 236 dequeue_op
> 0x56453b753f80 finish
> 2018-05-14 13:47:07.111388 7fb8185f9700 10 osd.21 236 tick
> 2018-05-14 13:47:07.111398 7fb8185f9700 10 osd.21 236 do_waiters -- start
> 2018-05-14 13:47:07.111401 7fb8185f9700 10 osd.21 236 do_waiters -- finish
> 2018-05-14 13:47:07.800421 7fb817df8700 10 osd.21 236 tick_without_osd_lock
> 2018-05-14 13:47:07.800444 7fb817df8700 10 osd.21 236
> promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0  bytes;
> target 25 obj/sec or 5120 k bytes/sec
> 2018-05-14 13:47:07.800449 7fb817df8700 10 osd.21 236
> promote_throttle_recalibrate  actual 0, actual/prob ratio 1, adjusted
> new_prob 1000, prob 1000 -> 1000
> 2018-05-14 13:47:08.111470 7fb8185f9700 10 osd.21 236 tick
> 2018-05-14 13:47:08.111483 7fb8185f9700 10 osd.21 236 do_waiters -- start
> 2018-05-14 13:47:08.111485 7fb8185f9700 10 osd.21 236 do_waiters -- finish
> 2018-05-14 13:47:08.181070 7fb8055d3700 10 osd.21 236 dequeue_op
> 0x564539651000 prio 63 cost 0 latency 0.000143 
> osd_op(client.2597258.0:213844298
> 6.1d4 6.4079fd4 (undecoded) ondisk+read+kno
> wn_if_redirected e236) v8 pg pg[6.1d4( v 236'20882 (236'19289,236'20882]
> local-lis/les=223/224 n=20791 ec=145/132 lis/c 223/223 les/c/f 224/224/0
> 223/223/212) [21,29,17] r=0 lpr=223 crt=236
> '20882 lcod 236'20881 mlcod 236'20881 active+clean]
> 2018-05-14 13:47:08.181112 7fb8055d3700 10 osd.21 pg_epoch: 236 pg[6.1d4(
> v 236'20882 (236'19289,236'20882] local-lis/les=223/224 n=20791 ec=145/132
> lis/c 223/223 les/c/f 224/224/0 223/223/
> 212) [21,29,17] r=0 lpr=223 crt=236'20882 lcod 236'20881 mlcod 236'20881
> active+clean] _handle_message: 0x564539651000
> 2018-05-14 13:47:08.181141 7fb8

Re: [ceph-users] slow requests are blocked

[LOPEZ Jean-Charles]

Hi Grigory,

looks like osd.16 is having a hard time acknowledging the write request (for 
bucket resharding operations from what it looks like) as it takes about 15 
seconds for osd.16 to receive the commit confirmation from osd.21 on subop 
communication.

Have a go and check at the journal device for osd.21 or if the machine where 
osd.21 is running is either overloaded or has a network issue.

Regards
JC

> On 15 May 2018, at 19:49, Grigory Murashov  wrote:
> 
> Hello guys!
> 
> I collected output of ceph daemon osd.16 dump_ops_in_flight and ceph daemon 
> osd.16 dump_historic_ops.
> 
> Here is the output of ceph heath details in the moment of problem
> 
> HEALTH_WARN 20 slow requests are blocked > 32 sec
> REQUEST_SLOW 20 slow requests are blocked > 32 sec
> 20 ops are blocked > 65.536 sec
> osds 16,27,29 have blocked requests > 65.536 sec
> So I grab logs from osd.16.
> 
> The file is attached.  Could you please help to translate?
> 
> Thanks in advance.
> Grigory Murashov
> Voximplant
> 14.05.2018 18:14, Grigory Murashov пишет:
>> Hello David!
>> 
>> 2. I set it up 10/10
>> 
>> 3. Thanks, my problem was I did it on host where was no osd.15 daemon.
>> 
>> Could you please help to read osd logs?
>> 
>> Here is a part from ceph.log
>> 
>> 2018-05-14 13:46:32.644323 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553895 : cluster [INF] Cluster is now healthy
>> 2018-05-14 13:46:43.741921 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553896 : cluster [WRN] Health check failed: 21 slow requests are blocked > 
>> 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:46:49.746994 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553897 : cluster [WRN] Health check update: 23 slow requests are blocked > 
>> 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:46:55.752314 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553900 : cluster [WRN] Health check update: 3 slow requests are blocked > 32 
>> sec (REQUEST_SLOW)
>> 2018-05-14 13:47:01.030686 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553901 : cluster [WRN] Health check update: 4 slow requests are blocked > 32 
>> sec (REQUEST_SLOW)
>> 2018-05-14 13:47:07.764236 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553903 : cluster [WRN] Health check update: 32 slow requests are blocked > 
>> 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:47:13.770833 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553904 : cluster [WRN] Health check update: 21 slow requests are blocked > 
>> 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:47:17.774530 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553905 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 12 slow 
>> requests are blocked > 32 sec)
>> 2018-05-14 13:47:17.774582 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553906 : cluster [INF] Cluster is now healthy
>> At 13-47 I had a problem with osd.21
>> 
>> 1. Ceph Health (storage-ru1-osd1.voximplant.com:ceph.health): HEALTH_WARN
>> {u'REQUEST_SLOW': {u'severity': u'HEALTH_WARN', u'summary': {u'message': u'4 
>> slow requests are blocked > 32 sec'}}}
>> HEALTH_WARN 4 slow requests are blocked > 32 sec
>> REQUEST_SLOW 4 slow requests are blocked > 32 sec
>> 2 ops are blocked > 65.536 sec
>> 2 ops are blocked > 32.768 sec
>> osd.21 has blocked requests > 65.536 sec
>> Here is a part from ceph-osd.21.log
>> 2018-05-14 13:47:06.891399 7fb806dd6700 10 osd.21 pg_epoch: 236 pg[2.0( v 
>> 236'297 (0'0,236'297] local-lis/les=223/224 n=1 ec=119/119 lis/c 223/223 
>> les/c/f 224/224/0 223/223/212) [21,29,15]
>> r=0 lpr=223 crt=236'297 lcod 236'296 mlcod 236'296 active+clean]  dropping 
>> ondisk_read_lock
>> 2018-05-14 13:47:06.891435 7fb806dd6700 10 osd.21 236 dequeue_op 
>> 0x56453b753f80 finish
>> 2018-05-14 13:47:07.111388 7fb8185f9700 10 osd.21 236 tick
>> 2018-05-14 13:47:07.111398 7fb8185f9700 10 osd.21 236 do_waiters -- start
>> 2018-05-14 13:47:07.111401 7fb8185f9700 10 osd.21 236 do_waiters -- finish
>> 2018-05-14 13:47:07.800421 7fb817df8700 10 osd.21 236 tick_without_osd_lock
>> 2018-05-14 13:47:07.800444 7fb817df8700 10 osd.21 236 
>> promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0  bytes; 
>> target 25 obj/sec or 5120 k bytes/sec
>> 2018-05-14 13:47:07.800449 7fb817df8700 10 osd.21 236 
>> promote_throttle_recalibrate  actual 0, actual/prob ratio 1, adjusted 
>> new_prob 1000, prob 1000 -> 1000
>> 2018-05-14 13:47:08.111470 7fb8185f9700 10 osd.21 236 tick
>> 2018-05-14 13:47:08.111483 7fb8185f9700 10 osd.21 236 do_waiters -- start
>> 2018-05-14 13:47:08.111485 7fb8185f9700 10 osd.21 236 do_waiters -- finish
>> 2018-05-14 13:47:08.181070 7fb8055d3700 10 osd.21 236 dequeue_op 
>> 0x564539651000 prio 63 cost 0 latency 0.000143 
>> osd_op(client.2597258.0:213844298 6.1d4 6.4079fd4 (undecoded) ondisk+read+kno
>> wn_if_redirected e236) v8 pg pg[6.1d4( v 236'20882 (236'19289,236'20882] 
>> local-lis/les=223/224 n=20791 ec=145/132 lis/c 223/223 les/c/f 224/224/0 
>> 223/223/212) [21,29,17] r=0 lpr=223 crt=236
>> '20882 lcod 236'20881 

Re: [ceph-users] slow requests are blocked

[Grigory Murashov]

Hello guys!

I collected output of ceph daemon osd.16 dump_ops_in_flight and ceph 
daemon osd.16 dump_historic_ops.


Here is the output of ceph heath details in the moment of problem

HEALTH_WARN 20 slow requests are blocked > 32 sec
REQUEST_SLOW 20 slow requests are blocked > 32 sec
    20 ops are blocked > 65.536 sec
    osds 16,27,29 have blocked requests > 65.536 sec

So I grab logs from osd.16.

The file is attached.  Could you please help to translate?

Thanks in advance.

Grigory Murashov
Voximplant

14.05.2018 18:14, Grigory Murashov пишет:


Hello David!

2. I set it up 10/10

3. Thanks, my problem was I did it on host where was no osd.15 daemon.

Could you please help to read osd logs?

Here is a part from ceph.log

2018-05-14 13:46:32.644323 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553895 : cluster [INF] Cluster is now healthy
2018-05-14 13:46:43.741921 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553896 : cluster [WRN] Health check failed: 21 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-05-14 13:46:49.746994 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553897 : cluster [WRN] Health check update: 23 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-05-14 13:46:55.752314 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553900 : cluster [WRN] Health check update: 3 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-05-14 13:47:01.030686 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553901 : cluster [WRN] Health check update: 4 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-05-14 13:47:07.764236 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553903 : cluster [WRN] Health check update: 32 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-05-14 13:47:13.770833 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553904 : cluster [WRN] Health check update: 21 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-05-14 13:47:17.774530 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553905 : cluster [INF] Health check cleared: 
REQUEST_SLOW (was: 12 slow requests are blocked > 32 sec)
2018-05-14 13:47:17.774582 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553906 : cluster [INF] Cluster is now healthy


At 13-47 I had a problem with osd.21

1. Ceph Health (storage-ru1-osd1.voximplant.com:ceph.health): HEALTH_WARN
{u'REQUEST_SLOW': {u'severity': u'HEALTH_WARN', u'summary': {u'message': u'4 slow 
requests are blocked > 32 sec'}}}
HEALTH_WARN 4 slow requests are blocked > 32 sec
REQUEST_SLOW 4 slow requests are blocked > 32 sec
 2 ops are blocked > 65.536 sec
 2 ops are blocked > 32.768 sec
 osd.21 has blocked requests > 65.536 sec

Here is a part from ceph-osd.21.log

2018-05-14 13:47:06.891399 7fb806dd6700 10 osd.21 pg_epoch: 236 
pg[2.0( v 236'297 (0'0,236'297] local-lis/les=223/224 n=1 ec=119/119 
lis/c 223/223 les/c/f 224/224/0 223/223/212) [21,29,15]
r=0 lpr=223 crt=236'297 lcod 236'296 mlcod 236'296 active+clean]  
dropping ondisk_read_lock
2018-05-14 13:47:06.891435 7fb806dd6700 10 osd.21 236 dequeue_op 
0x56453b753f80 finish

2018-05-14 13:47:07.111388 7fb8185f9700 10 osd.21 236 tick
2018-05-14 13:47:07.111398 7fb8185f9700 10 osd.21 236 do_waiters -- start
2018-05-14 13:47:07.111401 7fb8185f9700 10 osd.21 236 do_waiters -- finish
2018-05-14 13:47:07.800421 7fb817df8700 10 osd.21 236 
tick_without_osd_lock
2018-05-14 13:47:07.800444 7fb817df8700 10 osd.21 236 
promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0  
bytes; target 25 obj/sec or 5120 k bytes/sec
2018-05-14 13:47:07.800449 7fb817df8700 10 osd.21 236 
promote_throttle_recalibrate  actual 0, actual/prob ratio 1, adjusted 
new_prob 1000, prob 1000 -> 1000

2018-05-14 13:47:08.111470 7fb8185f9700 10 osd.21 236 tick
2018-05-14 13:47:08.111483 7fb8185f9700 10 osd.21 236 do_waiters -- start
2018-05-14 13:47:08.111485 7fb8185f9700 10 osd.21 236 do_waiters -- finish
2018-05-14 13:47:08.181070 7fb8055d3700 10 osd.21 236 dequeue_op 
0x564539651000 prio 63 cost 0 latency 0.000143 
osd_op(client.2597258.0:213844298 6.1d4 6.4079fd4 (undecoded) 
ondisk+read+kno
wn_if_redirected e236) v8 pg pg[6.1d4( v 236'20882 
(236'19289,236'20882] local-lis/les=223/224 n=20791 ec=145/132 lis/c 
223/223 les/c/f 224/224/0 223/223/212) [21,29,17] r=0 lpr=223 crt=236

'20882 lcod 236'20881 mlcod 236'20881 active+clean]
2018-05-14 13:47:08.181112 7fb8055d3700 10 osd.21 pg_epoch: 236 
pg[6.1d4( v 236'20882 (236'19289,236'20882] local-lis/les=223/224 
n=20791 ec=145/132 lis/c 223/223 les/c/f 224/224/0 223/223/
212) [21,29,17] r=0 lpr=223 crt=236'20882 lcod 236'20881 mlcod 
236'20881 active+clean] _handle_message: 0x564539651000
2018-05-14 13:47:08.181141 7fb8055d3700 10 osd.21 pg_epoch: 236 
pg[6.1d4( v 236'20882 (236'19289,236'20882] local-lis/les=223/224 
n=20791 ec=145/132 lis/c 223/223 les/c/f 224/224/0 223/223/
212) [21,29,17] r=0 lpr=223 crt=236'20882 lcod 236'20881 mlcod 
236'20881 active+clean] do_op osd_op(client.2597258.0:213844298 6.1d4 
6:2bf9e020:::eb359f44-3316-4cd3-9006-d

Re: [ceph-users] slow requests are blocked

[Grigory Murashov]

Hello David!

2. I set it up 10/10

3. Thanks, my problem was I did it on host where was no osd.15 daemon.

Could you please help to read osd logs?

Here is a part from ceph.log

2018-05-14 13:46:32.644323 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553895 : cluster [INF] Cluster is now healthy
2018-05-14 13:46:43.741921 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553896 : cluster [WRN] Health check failed: 21 slow 
requests are blocked > 32 sec (REQUEST_SLOW)
2018-05-14 13:46:49.746994 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553897 : cluster [WRN] Health check update: 23 slow 
requests are blocked > 32 sec (REQUEST_SLOW)
2018-05-14 13:46:55.752314 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553900 : cluster [WRN] Health check update: 3 slow 
requests are blocked > 32 sec (REQUEST_SLOW)
2018-05-14 13:47:01.030686 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553901 : cluster [WRN] Health check update: 4 slow 
requests are blocked > 32 sec (REQUEST_SLOW)
2018-05-14 13:47:07.764236 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553903 : cluster [WRN] Health check update: 32 slow 
requests are blocked > 32 sec (REQUEST_SLOW)
2018-05-14 13:47:13.770833 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553904 : cluster [WRN] Health check update: 21 slow 
requests are blocked > 32 sec (REQUEST_SLOW)
2018-05-14 13:47:17.774530 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553905 : cluster [INF] Health check cleared: 
REQUEST_SLOW (was: 12 slow requests are blocked > 32 sec)
2018-05-14 13:47:17.774582 mon.storage-ru1-osd1 mon.0 
185.164.149.2:6789/0 553906 : cluster [INF] Cluster is now healthy


At 13-47 I had a problem with osd.21

1. Ceph Health (storage-ru1-osd1.voximplant.com:ceph.health): HEALTH_WARN
{u'REQUEST_SLOW': {u'severity': u'HEALTH_WARN', u'summary': {u'message': u'4 slow 
requests are blocked > 32 sec'}}}
HEALTH_WARN 4 slow requests are blocked > 32 sec
REQUEST_SLOW 4 slow requests are blocked > 32 sec
2 ops are blocked > 65.536 sec
2 ops are blocked > 32.768 sec
osd.21 has blocked requests > 65.536 sec

Here is a part from ceph-osd.21.log

2018-05-14 13:47:06.891399 7fb806dd6700 10 osd.21 pg_epoch: 236 pg[2.0( 
v 236'297 (0'0,236'297] local-lis/les=223/224 n=1 ec=119/119 lis/c 
223/223 les/c/f 224/224/0 223/223/212) [21,29,15]
r=0 lpr=223 crt=236'297 lcod 236'296 mlcod 236'296 active+clean] 
dropping ondisk_read_lock
2018-05-14 13:47:06.891435 7fb806dd6700 10 osd.21 236 dequeue_op 
0x56453b753f80 finish

2018-05-14 13:47:07.111388 7fb8185f9700 10 osd.21 236 tick
2018-05-14 13:47:07.111398 7fb8185f9700 10 osd.21 236 do_waiters -- start
2018-05-14 13:47:07.111401 7fb8185f9700 10 osd.21 236 do_waiters -- finish
2018-05-14 13:47:07.800421 7fb817df8700 10 osd.21 236 tick_without_osd_lock
2018-05-14 13:47:07.800444 7fb817df8700 10 osd.21 236 
promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0 bytes; 
target 25 obj/sec or 5120 k bytes/sec
2018-05-14 13:47:07.800449 7fb817df8700 10 osd.21 236 
promote_throttle_recalibrate  actual 0, actual/prob ratio 1, adjusted 
new_prob 1000, prob 1000 -> 1000

2018-05-14 13:47:08.111470 7fb8185f9700 10 osd.21 236 tick
2018-05-14 13:47:08.111483 7fb8185f9700 10 osd.21 236 do_waiters -- start
2018-05-14 13:47:08.111485 7fb8185f9700 10 osd.21 236 do_waiters -- finish
2018-05-14 13:47:08.181070 7fb8055d3700 10 osd.21 236 dequeue_op 
0x564539651000 prio 63 cost 0 latency 0.000143 
osd_op(client.2597258.0:213844298 6.1d4 6.4079fd4 (undecoded) 
ondisk+read+kno
wn_if_redirected e236) v8 pg pg[6.1d4( v 236'20882 (236'19289,236'20882] 
local-lis/les=223/224 n=20791 ec=145/132 lis/c 223/223 les/c/f 224/224/0 
223/223/212) [21,29,17] r=0 lpr=223 crt=236

'20882 lcod 236'20881 mlcod 236'20881 active+clean]
2018-05-14 13:47:08.181112 7fb8055d3700 10 osd.21 pg_epoch: 236 
pg[6.1d4( v 236'20882 (236'19289,236'20882] local-lis/les=223/224 
n=20791 ec=145/132 lis/c 223/223 les/c/f 224/224/0 223/223/
212) [21,29,17] r=0 lpr=223 crt=236'20882 lcod 236'20881 mlcod 236'20881 
active+clean] _handle_message: 0x564539651000
2018-05-14 13:47:08.181141 7fb8055d3700 10 osd.21 pg_epoch: 236 
pg[6.1d4( v 236'20882 (236'19289,236'20882] local-lis/les=223/224 
n=20791 ec=145/132 lis/c 223/223 les/c/f 224/224/0 223/223/
212) [21,29,17] r=0 lpr=223 crt=236'20882 lcod 236'20881 mlcod 236'20881 
active+clean] do_op osd_op(client.2597258.0:213844298 6.1d4 
6:2bf9e020:::eb359f44-3316-4cd3-9006-d416c21e0745.120446

4.6_2018%2f05%2f14%2fYWRjNmZmNzQzODI2ZGQzOTc3ZjFiNGMxZjIxOGZlYzQvaHR0cDovL3d3dy1sdS0wMS0zNi52b3hpbXBsYW50LmNvbS9yZWNvcmRzLzIwMTgvMDUvMTQvOTRlNjYxY2JiZjU3MTk4NS4xNTI2MjkwMzQ0Ljk2NjQ5MS5tcDM-
:head [getxattrs,stat,read 0~4194304] snapc 0=[] 
ondisk+read+known_if_redirected e236) v8 may_read -> read-ordered flags 
ondisk+read+known_if_redirected
2018-05-14 13:47:08.181179 7fb8055d3700 10 osd.21 pg_epoch: 236 
pg[6.1d4( v 236'20882 (236'19289,236'20882] local-lis/les=223/224 
n=20791 ec=145/132 lis/c 223/223 les/c/f 224/224/0 223/223/
212) [2

Re: [ceph-users] slow requests are blocked

[David Turner]

2. When logging the 1/5 is what's written to the log file/what's
temporarily stored in memory.  If you want to increase logging, you need to
increase both numbers to 20/20 or 10/10.  You can also just set it to 20 or
10 and ceph will set them to the same number.  I personally do both numbers
to remind myself that the defaults aren't the same to set it back to.

3. You are not accessing something stored within the ceph cluster, so it
isn't using your admin cephx key that you have.  It is accessing the daemon
socket in the OS so you need to have proper permissions to be able to
access it.  The daemon sockets are located in /var/run/ceph/ which is a
folder without any public permissions.  You could use `sudo -u ceph ceph
daemon osd.15 dump_historic_opsor` or just `sudo ceph daemon osd.15
dump_historic_ops` as root can access the daemon socket as well.

On Thu, May 10, 2018 at 7:14 AM Grigory Murashov 
wrote:

> Hi JC!
>
> Thanks for your answer first.
>
> 1. I have added output of  ceph health detail to Zabbix in case of
> warning. So every time I will see with which OSD the problem is.
>
> 2. I have default level of all logs. As I see here
> http://docs.ceph.com/docs/master/rados/troubleshooting/log-and-debug/
> default log level for OSD is 1/5. Should I try
>
> debug osd = 1/20 of 1/10 would be enough here?
>
> 3. Any thoughts why do I have Permission denied? All of my sockets are
> also defaults.
>
> [cephuser@osd3 ~]$ ceph daemon osd.15 dump_historic_ops
> admin_socket: exception getting command descriptions: [Errno 13]
> Permission denied
>
> Thanks in advance.
>
> Grigory Murashov
> Voximplant
>
> 08.05.2018 17:31, Jean-Charles Lopez пишет:
> > Hi Grigory,
> >
> > are these lines the only lines in your log file for OSD 15?
> >
> > Just for sanity, what are the log levels you have set, if any, in your
> config file away from the default? If you set all log levels to 0 like some
> people do you may want to simply go back to the default by commenting out
> the debug_ lines in your config file. If you want to see something more
> detailed you can indeed increase the log level to 5 or 10.
> >
> > What you can also do is to use the admin socket on the machine to see
> what operations are actually blocked: ceph daemon osd.15 dump_ops_in_flight
> and ceph daemon osd.15 dump_historic_ops.
> >
> > These two commands and their output will show you what exact operations
> are blocked and will also point you to the other OSDs this OSD is working
> with to serve the IO. May be the culprit is actually one of the OSDs
> handling the subops or it could be a network problem.
> >
> > Regards
> > JC
> >
> >> On May 8, 2018, at 03:11, Grigory Murashov 
> wrote:
> >>
> >> Hello Jean-Charles!
> >>
> >> I have finally catch the problem, It was at 13-02.
> >>
> >> [cephuser@storage-ru1-osd3 ~]$ ceph health detail
> >> HEALTH_WARN 18 slow requests are blocked > 32 sec
> >> REQUEST_SLOW 18 slow requests are blocked > 32 sec
> >>  3 ops are blocked > 65.536 sec
> >>  15 ops are blocked > 32.768 sec
> >>  osd.15 has blocked requests > 65.536 sec
> >> [cephuser@storage-ru1-osd3 ~]$
> >>
> >>
> >> But surprise - there is no information in ceph-osd.15.log that time
> >>
> >>
> >> 2018-05-08 12:54:26.105919 7f003f5f9700  4 rocksdb: (Original Log Time
> 2018/05/08-12:54:26.105843) EVENT_LOG_v1 {"time_micros": 1525773266105834,
> "job": 2793, "event": "trivial_move", "dest
> >> ination_level": 3, "files": 1, "total_files_size": 68316970}
> >> 2018-05-08 12:54:26.105926 7f003f5f9700  4 rocksdb: (Original Log Time
> 2018/05/08-12:54:26.105854)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABL
> >>
> E_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:1537]
> [default] Moved #1 files to level-3 68316970 bytes OK
> >> : base level 1 max bytes base 268435456 files[0 4 45 403 722 0 0] max
> score 0.98
> >>
> >> 2018-05-08 13:07:29.711425 7f004f619700  4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/r
> >>
> elease/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_write.cc:684]
> reusing log 8051 from recycle list
> >>
> >> 2018-05-08 13:07:29.711497 7f004f619700  4 rocksdb:
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/r
> >>
> elease/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_write.cc:725]
> [default] New memtable created with log file: #8089. Immutable memtables: 0.
> >>
> >> 2018-05-08 13:07:29.726107 7f003fdfa700  4 rocksdb: (Original Log Time
> 2018/05/08-13:07:29.711524)
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABL
> >>
> E_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:11

Re: [ceph-users] slow requests are blocked

[Grigory Murashov]

Hi JC!

Thanks for your answer first.

1. I have added output of  ceph health detail to Zabbix in case of 
warning. So every time I will see with which OSD the problem is.


2. I have default level of all logs. As I see here 
http://docs.ceph.com/docs/master/rados/troubleshooting/log-and-debug/ 
default log level for OSD is 1/5. Should I try


debug osd = 1/20 of 1/10 would be enough here?

3. Any thoughts why do I have Permission denied? All of my sockets are 
also defaults.


[cephuser@osd3 ~]$ ceph daemon osd.15 dump_historic_ops
admin_socket: exception getting command descriptions: [Errno 13] 
Permission denied


Thanks in advance.

Grigory Murashov
Voximplant

08.05.2018 17:31, Jean-Charles Lopez пишет:

Hi Grigory,

are these lines the only lines in your log file for OSD 15?

Just for sanity, what are the log levels you have set, if any, in your config 
file away from the default? If you set all log levels to 0 like some people do 
you may want to simply go back to the default by commenting out the debug_ 
lines in your config file. If you want to see something more detailed you can 
indeed increase the log level to 5 or 10.

What you can also do is to use the admin socket on the machine to see what 
operations are actually blocked: ceph daemon osd.15 dump_ops_in_flight and ceph 
daemon osd.15 dump_historic_ops.

These two commands and their output will show you what exact operations are 
blocked and will also point you to the other OSDs this OSD is working with to 
serve the IO. May be the culprit is actually one of the OSDs handling the 
subops or it could be a network problem.

Regards
JC


On May 8, 2018, at 03:11, Grigory Murashov  wrote:

Hello Jean-Charles!

I have finally catch the problem, It was at 13-02.

[cephuser@storage-ru1-osd3 ~]$ ceph health detail
HEALTH_WARN 18 slow requests are blocked > 32 sec
REQUEST_SLOW 18 slow requests are blocked > 32 sec
 3 ops are blocked > 65.536 sec
 15 ops are blocked > 32.768 sec
 osd.15 has blocked requests > 65.536 sec
[cephuser@storage-ru1-osd3 ~]$


But surprise - there is no information in ceph-osd.15.log that time


2018-05-08 12:54:26.105919 7f003f5f9700  4 rocksdb: (Original Log Time 2018/05/08-12:54:26.105843) EVENT_LOG_v1 
{"time_micros": 1525773266105834, "job": 2793, "event": "trivial_move", "dest
ination_level": 3, "files": 1, "total_files_size": 68316970}
2018-05-08 12:54:26.105926 7f003f5f9700  4 rocksdb: (Original Log Time 
2018/05/08-12:54:26.105854) 
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABL
E_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:1537]
 [default] Moved #1 files to level-3 68316970 bytes OK
: base level 1 max bytes base 268435456 files[0 4 45 403 722 0 0] max score 0.98

2018-05-08 13:07:29.711425 7f004f619700  4 rocksdb: 
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/r
elease/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_write.cc:684] 
reusing log 8051 from recycle list

2018-05-08 13:07:29.711497 7f004f619700  4 rocksdb: 
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/r
elease/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_write.cc:725] 
[default] New memtable created with log file: #8089. Immutable memtables: 0.

2018-05-08 13:07:29.726107 7f003fdfa700  4 rocksdb: (Original Log Time 
2018/05/08-13:07:29.711524) 
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABL
E_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:1158]
 Calling FlushMemTableToOutputFile with column family
[default], flush slots available 1, compaction slots allowed 1, compaction 
slots scheduled 1
2018-05-08 13:07:29.726124 7f003fdfa700  4 rocksdb: 
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/r
elease/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/flush_job.cc:264] 
[default] [JOB 2794] Flushing memtable with next log file: 8089

Should I have some deeply logging?


Grigory Murashov
Voximplant

07.05.2018 18:59, Jean-Charles Lopez пишет:

Hi,

ceph health detail

This will tell you which OSDs are experiencing the problem so you can then go 
and inspect the logs and use the admin socket to find out which requests are at 
the source.

Regards
JC


On May 7, 2018, at 03:52, Grigory Murashov  wrote:

Hello!

I'm not much experiensed in ceph troubleshouting that why I ask for help.

I have multiple warnings coming from zabbix as a result of ceph -s

REQUEST_SLOW: HEALTH_WARN : 21 slow requests are blocked > 32 sec

I don't see any hardware problems that time.

I'm able to find the same strings in ceph.log and ceph-mon.l

Re: [ceph-users] slow requests are blocked

[Jean-Charles Lopez]

Hi Grigory,

are these lines the only lines in your log file for OSD 15?

Just for sanity, what are the log levels you have set, if any, in your config 
file away from the default? If you set all log levels to 0 like some people do 
you may want to simply go back to the default by commenting out the debug_ 
lines in your config file. If you want to see something more detailed you can 
indeed increase the log level to 5 or 10.

What you can also do is to use the admin socket on the machine to see what 
operations are actually blocked: ceph daemon osd.15 dump_ops_in_flight and ceph 
daemon osd.15 dump_historic_ops.

These two commands and their output will show you what exact operations are 
blocked and will also point you to the other OSDs this OSD is working with to 
serve the IO. May be the culprit is actually one of the OSDs handling the 
subops or it could be a network problem.

Regards
JC

> On May 8, 2018, at 03:11, Grigory Murashov  wrote:
> 
> Hello Jean-Charles!
> 
> I have finally catch the problem, It was at 13-02.
> 
> [cephuser@storage-ru1-osd3 ~]$ ceph health detail
> HEALTH_WARN 18 slow requests are blocked > 32 sec
> REQUEST_SLOW 18 slow requests are blocked > 32 sec
> 3 ops are blocked > 65.536 sec
> 15 ops are blocked > 32.768 sec
> osd.15 has blocked requests > 65.536 sec
> [cephuser@storage-ru1-osd3 ~]$
> 
> 
> But surprise - there is no information in ceph-osd.15.log that time
> 
> 
> 2018-05-08 12:54:26.105919 7f003f5f9700  4 rocksdb: (Original Log Time 
> 2018/05/08-12:54:26.105843) EVENT_LOG_v1 {"time_micros": 1525773266105834, 
> "job": 2793, "event": "trivial_move", "dest
> ination_level": 3, "files": 1, "total_files_size": 68316970}
> 2018-05-08 12:54:26.105926 7f003f5f9700  4 rocksdb: (Original Log Time 
> 2018/05/08-12:54:26.105854) 
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABL
> E_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:1537]
>  [default] Moved #1 files to level-3 68316970 bytes OK
> : base level 1 max bytes base 268435456 files[0 4 45 403 722 0 0] max score 
> 0.98
> 
> 2018-05-08 13:07:29.711425 7f004f619700  4 rocksdb: 
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/r
> elease/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_write.cc:684] 
> reusing log 8051 from recycle list
> 
> 2018-05-08 13:07:29.711497 7f004f619700  4 rocksdb: 
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/r
> elease/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_write.cc:725] 
> [default] New memtable created with log file: #8089. Immutable memtables: 0.
> 
> 2018-05-08 13:07:29.726107 7f003fdfa700  4 rocksdb: (Original Log Time 
> 2018/05/08-13:07:29.711524) 
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABL
> E_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:1158]
>  Calling FlushMemTableToOutputFile with column family
> [default], flush slots available 1, compaction slots allowed 1, compaction 
> slots scheduled 1
> 2018-05-08 13:07:29.726124 7f003fdfa700  4 rocksdb: 
> [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/r
> elease/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/flush_job.cc:264] 
> [default] [JOB 2794] Flushing memtable with next log file: 8089
> 
> Should I have some deeply logging?
> 
> 
> Grigory Murashov
> Voximplant
> 
> 07.05.2018 18:59, Jean-Charles Lopez пишет:
>> Hi,
>> 
>> ceph health detail
>> 
>> This will tell you which OSDs are experiencing the problem so you can then 
>> go and inspect the logs and use the admin socket to find out which requests 
>> are at the source.
>> 
>> Regards
>> JC
>> 
>>> On May 7, 2018, at 03:52, Grigory Murashov  wrote:
>>> 
>>> Hello!
>>> 
>>> I'm not much experiensed in ceph troubleshouting that why I ask for help.
>>> 
>>> I have multiple warnings coming from zabbix as a result of ceph -s
>>> 
>>> REQUEST_SLOW: HEALTH_WARN : 21 slow requests are blocked > 32 sec
>>> 
>>> I don't see any hardware problems that time.
>>> 
>>> I'm able to find the same strings in ceph.log and ceph-mon.log like
>>> 
>>> 2018-05-07 12:37:57.375546 7f3037dae700  0 log_channel(cluster) log [WRN] : 
>>> Health check failed: 12 slow requests are blocked > 32 sec (REQUEST_SLOW)
>>> 
>>> Now It's important to find out the root of the problem.
>>> 
>>> How to find out:
>>> 
>>> 1. which OSDs are affected
>>> 
>>> 2. which particular requests were slowed and blocked?
>>> 
>>> I assume I need more detailed logging - how to do that?
>>> 
>>> Appreciate your help.
>>> 
>>> -- 
>>> Grigory Murashov
>>> 
>>> 

Re: [ceph-users] slow requests are blocked

[Grigory Murashov]

Hello Jean-Charles!

I have finally catch the problem, It was at 13-02.

[cephuser@storage-ru1-osd3 ~]$ ceph health detail
HEALTH_WARN 18 slow requests are blocked > 32 sec
REQUEST_SLOW 18 slow requests are blocked > 32 sec
    3 ops are blocked > 65.536 sec
    15 ops are blocked > 32.768 sec
    osd.15 has blocked requests > 65.536 sec
[cephuser@storage-ru1-osd3 ~]$


But surprise - there is no information in ceph-osd.15.log that time


2018-05-08 12:54:26.105919 7f003f5f9700  4 rocksdb: (Original Log Time 
2018/05/08-12:54:26.105843) EVENT_LOG_v1 {"time_micros": 
1525773266105834, "job": 2793, "event": "trivial_move", "dest

ination_level": 3, "files": 1, "total_files_size": 68316970}
2018-05-08 12:54:26.105926 7f003f5f9700  4 rocksdb: (Original Log Time 
2018/05/08-12:54:26.105854) 
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABL
E_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:1537] 
[default] Moved #1 files to level-3 68316970 bytes OK
: base level 1 max bytes base 268435456 files[0 4 45 403 722 0 0] max 
score 0.98


2018-05-08 13:07:29.711425 7f004f619700  4 rocksdb: 
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/r
elease/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_write.cc:684] 
reusing log 8051 from recycle list


2018-05-08 13:07:29.711497 7f004f619700  4 rocksdb: 
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/r
elease/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_write.cc:725] 
[default] New memtable created with log file: #8089. Immutable memtables: 0.


2018-05-08 13:07:29.726107 7f003fdfa700  4 rocksdb: (Original Log Time 
2018/05/08-13:07:29.711524) 
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABL
E_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/db_impl_compaction_flush.cc:1158] 
Calling FlushMemTableToOutputFile with column family
[default], flush slots available 1, compaction slots allowed 1, 
compaction slots scheduled 1
2018-05-08 13:07:29.726124 7f003fdfa700  4 rocksdb: 
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/r
elease/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/flush_job.cc:264] 
[default] [JOB 2794] Flushing memtable with next log file: 8089


Should I have some deeply logging?


Grigory Murashov
Voximplant

07.05.2018 18:59, Jean-Charles Lopez пишет:

Hi,

ceph health detail

This will tell you which OSDs are experiencing the problem so you can then go 
and inspect the logs and use the admin socket to find out which requests are at 
the source.

Regards
JC


On May 7, 2018, at 03:52, Grigory Murashov  wrote:

Hello!

I'm not much experiensed in ceph troubleshouting that why I ask for help.

I have multiple warnings coming from zabbix as a result of ceph -s

REQUEST_SLOW: HEALTH_WARN : 21 slow requests are blocked > 32 sec

I don't see any hardware problems that time.

I'm able to find the same strings in ceph.log and ceph-mon.log like

2018-05-07 12:37:57.375546 7f3037dae700  0 log_channel(cluster) log [WRN] : Health 
check failed: 12 slow requests are blocked > 32 sec (REQUEST_SLOW)

Now It's important to find out the root of the problem.

How to find out:

1. which OSDs are affected

2. which particular requests were slowed and blocked?

I assume I need more detailed logging - how to do that?

Appreciate your help.

--
Grigory Murashov

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests are blocked

[Jean-Charles Lopez]

Hi,

ceph health detail

This will tell you which OSDs are experiencing the problem so you can then go 
and inspect the logs and use the admin socket to find out which requests are at 
the source.

Regards
JC

> On May 7, 2018, at 03:52, Grigory Murashov  wrote:
> 
> Hello!
> 
> I'm not much experiensed in ceph troubleshouting that why I ask for help.
> 
> I have multiple warnings coming from zabbix as a result of ceph -s
> 
> REQUEST_SLOW: HEALTH_WARN : 21 slow requests are blocked > 32 sec
> 
> I don't see any hardware problems that time.
> 
> I'm able to find the same strings in ceph.log and ceph-mon.log like
> 
> 2018-05-07 12:37:57.375546 7f3037dae700  0 log_channel(cluster) log [WRN] : 
> Health check failed: 12 slow requests are blocked > 32 sec (REQUEST_SLOW)
> 
> Now It's important to find out the root of the problem.
> 
> How to find out:
> 
> 1. which OSDs are affected
> 
> 2. which particular requests were slowed and blocked?
> 
> I assume I need more detailed logging - how to do that?
> 
> Appreciate your help.
> 
> -- 
> Grigory Murashov
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] slow requests are blocked

[Grigory Murashov]

Hello!

I'm not much experiensed in ceph troubleshouting that why I ask for help.

I have multiple warnings coming from zabbix as a result of ceph -s

REQUEST_SLOW: HEALTH_WARN : 21 slow requests are blocked > 32 sec

I don't see any hardware problems that time.

I'm able to find the same strings in ceph.log and ceph-mon.log like

2018-05-07 12:37:57.375546 7f3037dae700  0 log_channel(cluster) log 
[WRN] : Health check failed: 12 slow requests are blocked > 32 sec 
(REQUEST_SLOW)


Now It's important to find out the root of the problem.

How to find out:

1. which OSDs are affected

2. which particular requests were slowed and blocked?

I assume I need more detailed logging - how to do that?

Appreciate your help.

--
Grigory Murashov

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests troubleshooting in Luminous - details missing

[Alex Gorbachev]

On Mon, Mar 5, 2018 at 11:20 PM, Brad Hubbard  wrote:
> On Fri, Mar 2, 2018 at 3:54 PM, Alex Gorbachev  
> wrote:
>> On Thu, Mar 1, 2018 at 10:57 PM, David Turner  wrote:
>>> Blocked requests and slow requests are synonyms in ceph. They are 2 names
>>> for the exact same thing.
>>>
>>>
>>> On Thu, Mar 1, 2018, 10:21 PM Alex Gorbachev  
>>> wrote:

 On Thu, Mar 1, 2018 at 2:47 PM, David Turner 
 wrote:
 > `ceph health detail` should show you more information about the slow
 > requests.  If the output is too much stuff, you can grep out for blocked
 > or
 > something.  It should tell you which OSDs are involved, how long they've
 > been slow, etc.  The default is for them to show '> 32 sec' but that may
 > very well be much longer and `ceph health detail` will show that.

 Hi David,

 Thank you for the reply.  Unfortunately, the health detail only shows
 blocked requests.  This seems to be related to a compression setting
 on the pool, nothing in OSD logs.

 I replied to another compression thread.  This makes sense since
 compression is new, and in the past all such issues were reflected in
 OSD logs and related to either network or OSD hardware.

 Regards,
 Alex

 >
 > On Thu, Mar 1, 2018 at 2:23 PM Alex Gorbachev 
 > wrote:
 >>
 >> Is there a switch to turn on the display of specific OSD issues?  Or
 >> does the below indicate a generic problem, e.g. network and no any
 >> specific OSD?
 >>
 >> 2018-02-28 18:09:36.438300 7f6dead56700  0
 >> mon.roc-vm-sc3c234@0(leader).data_health(46) update_stats avail 56%
 >> total 15997 MB, used 6154 MB, avail 9008 MB
 >> 2018-02-28 18:09:41.477216 7f6dead56700  0 log_channel(cluster) log
 >> [WRN] : Health check failed: 73 slow requests are blocked > 32 sec
 >> (REQUEST_SLOW)
 >> 2018-02-28 18:09:47.552669 7f6dead56700  0 log_channel(cluster) log
 >> [WRN] : Health check update: 74 slow requests are blocked > 32 sec
 >> (REQUEST_SLOW)
 >> 2018-02-28 18:09:53.794882 7f6de8551700  0
 >> mon.roc-vm-sc3c234@0(leader) e1 handle_command mon_command({"prefix":
 >> "status", "format": "json"} v 0) v1
 >>
 >> --
>>
>> I was wrong where the pool compression does not matter, even
>> uncompressed pool also generates these slow messages.
>>
>> Question is why no subsequent message relating to specific OSDs (like
>> in Jewel and prior, like this example from RH:
>>
>> 2015-08-24 13:18:10.024659 osd.1 127.0.0.1:6812/3032 9 : cluster [WRN]
>> 6 slow requests, 6 included below; oldest blocked for > 61.758455 secs
>>
>> 2016-07-25 03:44:06.510583 osd.50 [WRN] slow request 30.005692 seconds
>> old, received at {date-time}: osd_op(client.4240.0:8
>> benchmark_data_ceph-1_39426_object7 [write 0~4194304] 0.69848840) v4
>> currently waiting for subops from [610]
>>
>> In comparison, my Luminous cluster only shows the general slow/blocked 
>> message:
>>
>> 2018-03-01 21:52:54.237270 7f7e419e3700  0 log_channel(cluster) log
>> [WRN] : Health check failed: 116 slow requests are blocked > 32 sec
>> (REQUEST_SLOW)
>> 2018-03-01 21:53:00.282721 7f7e419e3700  0 log_channel(cluster) log
>> [WRN] : Health check update: 66 slow requests are blocked > 32 sec
>> (REQUEST_SLOW)
>> 2018-03-01 21:53:08.534244 7f7e419e3700  0 log_channel(cluster) log
>> [WRN] : Health check update: 5 slow requests are blocked > 32 sec
>> (REQUEST_SLOW)
>> 2018-03-01 21:53:10.382510 7f7e419e3700  0 log_channel(cluster) log
>> [INF] : Health check cleared: REQUEST_SLOW (was: 5 slow requests are
>> blocked > 32 sec)
>> 2018-03-01 21:53:10.382546 7f7e419e3700  0 log_channel(cluster) log
>> [INF] : Cluster is now healthy
>>
>> So where are the details?
>
> Working on this, thanks.
>
> See https://tracker.ceph.com/issues/23236

As in the tracker, but I think it would be useful to others:

If something like below comes up, how do you troubleshoot the cause of
past events, especially if this is specific to just a handful of 1000s
of OSDs on many hosts?

2018-03-11 22:00:00.000132 mon.roc-vm-sc3c234 [INF] overall HEALTH_OK
2018-03-11 22:44:46.173825 mon.roc-vm-sc3c234 [WRN] Health check
failed: 12 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-03-11 22:44:52.245738 mon.roc-vm-sc3c234 [WRN] Health check
update: 9 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-03-11 22:44:57.925686 mon.roc-vm-sc3c234 [WRN] Health check
update: 10 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-03-11 22:45:02.926031 mon.roc-vm-sc3c234 [WRN] Health check
update: 14 slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-03-11 22:45:06.413741 mon.roc-vm-sc3c234 [INF] Health check
cleared: REQUEST_SLOW (was: 14 slow requests are blocked > 32 sec)
2018-03-11 22:45:06.413814 mon.roc-vm-sc3c234 [INF] Cluster is now healthy
2018-03-11 23:00:00.000136 mon.roc-vm-sc3c234 [INF] overall HEALTH_OK

Thank you,
Alex


>
>>
>> Thanks,
>> Alex
>> _

Re: [ceph-users] Slow requests troubleshooting in Luminous - details missing

[Brad Hubbard]

On Fri, Mar 2, 2018 at 3:54 PM, Alex Gorbachev  wrote:
> On Thu, Mar 1, 2018 at 10:57 PM, David Turner  wrote:
>> Blocked requests and slow requests are synonyms in ceph. They are 2 names
>> for the exact same thing.
>>
>>
>> On Thu, Mar 1, 2018, 10:21 PM Alex Gorbachev  
>> wrote:
>>>
>>> On Thu, Mar 1, 2018 at 2:47 PM, David Turner 
>>> wrote:
>>> > `ceph health detail` should show you more information about the slow
>>> > requests.  If the output is too much stuff, you can grep out for blocked
>>> > or
>>> > something.  It should tell you which OSDs are involved, how long they've
>>> > been slow, etc.  The default is for them to show '> 32 sec' but that may
>>> > very well be much longer and `ceph health detail` will show that.
>>>
>>> Hi David,
>>>
>>> Thank you for the reply.  Unfortunately, the health detail only shows
>>> blocked requests.  This seems to be related to a compression setting
>>> on the pool, nothing in OSD logs.
>>>
>>> I replied to another compression thread.  This makes sense since
>>> compression is new, and in the past all such issues were reflected in
>>> OSD logs and related to either network or OSD hardware.
>>>
>>> Regards,
>>> Alex
>>>
>>> >
>>> > On Thu, Mar 1, 2018 at 2:23 PM Alex Gorbachev 
>>> > wrote:
>>> >>
>>> >> Is there a switch to turn on the display of specific OSD issues?  Or
>>> >> does the below indicate a generic problem, e.g. network and no any
>>> >> specific OSD?
>>> >>
>>> >> 2018-02-28 18:09:36.438300 7f6dead56700  0
>>> >> mon.roc-vm-sc3c234@0(leader).data_health(46) update_stats avail 56%
>>> >> total 15997 MB, used 6154 MB, avail 9008 MB
>>> >> 2018-02-28 18:09:41.477216 7f6dead56700  0 log_channel(cluster) log
>>> >> [WRN] : Health check failed: 73 slow requests are blocked > 32 sec
>>> >> (REQUEST_SLOW)
>>> >> 2018-02-28 18:09:47.552669 7f6dead56700  0 log_channel(cluster) log
>>> >> [WRN] : Health check update: 74 slow requests are blocked > 32 sec
>>> >> (REQUEST_SLOW)
>>> >> 2018-02-28 18:09:53.794882 7f6de8551700  0
>>> >> mon.roc-vm-sc3c234@0(leader) e1 handle_command mon_command({"prefix":
>>> >> "status", "format": "json"} v 0) v1
>>> >>
>>> >> --
>
> I was wrong where the pool compression does not matter, even
> uncompressed pool also generates these slow messages.
>
> Question is why no subsequent message relating to specific OSDs (like
> in Jewel and prior, like this example from RH:
>
> 2015-08-24 13:18:10.024659 osd.1 127.0.0.1:6812/3032 9 : cluster [WRN]
> 6 slow requests, 6 included below; oldest blocked for > 61.758455 secs
>
> 2016-07-25 03:44:06.510583 osd.50 [WRN] slow request 30.005692 seconds
> old, received at {date-time}: osd_op(client.4240.0:8
> benchmark_data_ceph-1_39426_object7 [write 0~4194304] 0.69848840) v4
> currently waiting for subops from [610]
>
> In comparison, my Luminous cluster only shows the general slow/blocked 
> message:
>
> 2018-03-01 21:52:54.237270 7f7e419e3700  0 log_channel(cluster) log
> [WRN] : Health check failed: 116 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-03-01 21:53:00.282721 7f7e419e3700  0 log_channel(cluster) log
> [WRN] : Health check update: 66 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-03-01 21:53:08.534244 7f7e419e3700  0 log_channel(cluster) log
> [WRN] : Health check update: 5 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-03-01 21:53:10.382510 7f7e419e3700  0 log_channel(cluster) log
> [INF] : Health check cleared: REQUEST_SLOW (was: 5 slow requests are
> blocked > 32 sec)
> 2018-03-01 21:53:10.382546 7f7e419e3700  0 log_channel(cluster) log
> [INF] : Cluster is now healthy
>
> So where are the details?

Working on this, thanks.

See https://tracker.ceph.com/issues/23236

>
> Thanks,
> Alex
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests troubleshooting in Luminous - details missing

[Alex Gorbachev]

On Fri, Mar 2, 2018 at 9:56 AM, Alex Gorbachev  wrote:
>
> On Fri, Mar 2, 2018 at 4:17 AM Maged Mokhtar  wrote:
>>
>> On 2018-03-02 07:54, Alex Gorbachev wrote:
>>
>> On Thu, Mar 1, 2018 at 10:57 PM, David Turner 
>> wrote:
>>
>> Blocked requests and slow requests are synonyms in ceph. They are 2 names
>> for the exact same thing.
>>
>>
>> On Thu, Mar 1, 2018, 10:21 PM Alex Gorbachev 
>> wrote:
>>
>>
>> On Thu, Mar 1, 2018 at 2:47 PM, David Turner 
>> wrote:
>>
>> `ceph health detail` should show you more information about the slow
>> requests.  If the output is too much stuff, you can grep out for blocked
>> or
>> something.  It should tell you which OSDs are involved, how long they've
>> been slow, etc.  The default is for them to show '> 32 sec' but that may
>> very well be much longer and `ceph health detail` will show that.
>>
>>
>> Hi David,
>>
>> Thank you for the reply.  Unfortunately, the health detail only shows
>> blocked requests.  This seems to be related to a compression setting
>> on the pool, nothing in OSD logs.
>>
>> I replied to another compression thread.  This makes sense since
>> compression is new, and in the past all such issues were reflected in
>> OSD logs and related to either network or OSD hardware.
>>
>> Regards,
>> Alex
>>
>>
>> On Thu, Mar 1, 2018 at 2:23 PM Alex Gorbachev 
>> wrote:
>>
>>
>> Is there a switch to turn on the display of specific OSD issues?  Or
>> does the below indicate a generic problem, e.g. network and no any
>> specific OSD?
>>
>> 2018-02-28 18:09:36.438300 7f6dead56700  0
>> mon.roc-vm-sc3c234@0(leader).data_health(46) update_stats avail 56%
>> total 15997 MB, used 6154 MB, avail 9008 MB
>> 2018-02-28 18:09:41.477216 7f6dead56700  0 log_channel(cluster) log
>> [WRN] : Health check failed: 73 slow requests are blocked > 32 sec
>> (REQUEST_SLOW)
>> 2018-02-28 18:09:47.552669 7f6dead56700  0 log_channel(cluster) log
>> [WRN] : Health check update: 74 slow requests are blocked > 32 sec
>> (REQUEST_SLOW)
>> 2018-02-28 18:09:53.794882 7f6de8551700  0
>> mon.roc-vm-sc3c234@0(leader) e1 handle_command mon_command({"prefix":
>> "status", "format": "json"} v 0) v1
>>
>> --
>>
>>
>> I was wrong where the pool compression does not matter, even
>> uncompressed pool also generates these slow messages.
>>
>> Question is why no subsequent message relating to specific OSDs (like
>> in Jewel and prior, like this example from RH:
>>
>> 2015-08-24 13:18:10.024659 osd.1 127.0.0.1:6812/3032 9 : cluster [WRN]
>> 6 slow requests, 6 included below; oldest blocked for > 61.758455 secs
>>
>> 2016-07-25 03:44:06.510583 osd.50 [WRN] slow request 30.005692 seconds
>> old, received at {date-time}: osd_op(client.4240.0:8
>> benchmark_data_ceph-1_39426_object7 [write 0~4194304] 0.69848840) v4
>> currently waiting for subops from [610]
>>
>> In comparison, my Luminous cluster only shows the general slow/blocked
>> message:
>>
>> 2018-03-01 21:52:54.237270 7f7e419e3700  0 log_channel(cluster) log
>> [WRN] : Health check failed: 116 slow requests are blocked > 32 sec
>> (REQUEST_SLOW)
>> 2018-03-01 21:53:00.282721 7f7e419e3700  0 log_channel(cluster) log
>> [WRN] : Health check update: 66 slow requests are blocked > 32 sec
>> (REQUEST_SLOW)
>> 2018-03-01 21:53:08.534244 7f7e419e3700  0 log_channel(cluster) log
>> [WRN] : Health check update: 5 slow requests are blocked > 32 sec
>> (REQUEST_SLOW)
>> 2018-03-01 21:53:10.382510 7f7e419e3700  0 log_channel(cluster) log
>> [INF] : Health check cleared: REQUEST_SLOW (was: 5 slow requests are
>> blocked > 32 sec)
>> 2018-03-01 21:53:10.382546 7f7e419e3700  0 log_channel(cluster) log
>> [INF] : Cluster is now healthy
>>
>> So where are the details?
>>
>> Thanks,
>> Alex
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>>
>> Hi Alex,
>>
>> I agree, the health detail should display osd info.
>> Regarding the slow requests: ESX is very sensitive to io delay, this could
>> be due to cluster overload or disk going bad. If you have a monitoring tool,
>> do you see high load spikes (such as disk % busy ) on your disks ? Generally
>> it is good (if you can) simulate the max client workload + simulate
>> concurrent recovery and scrubbing and monitor your resources and see if they
>> get overloaded. Most if not all issues with ESX will go away if you have
>> enough hardware. I suspect this is what you are see-ing, specially the
>> VMotion and compression could be the extra load which triggered the issue,
>> in such case the problem is not specific to a particular osd. The other
>> possibility is that a particular disk is going bad, the monitoring tool
>> should be able to identify this.
>>
>> Note that VMotion generates an io pattern that keeps hitting the same osd
>> with concurrent small writes, if you can use rbd striping, creating 64k
>> stripes may help. A controller with write back cache will also help.
>>
>> Maged
>
>
> Tha

Re: [ceph-users] Slow requests troubleshooting in Luminous - details missing

[Alex Gorbachev]

On Fri, Mar 2, 2018 at 4:17 AM Maged Mokhtar  wrote:

> On 2018-03-02 07:54, Alex Gorbachev wrote:
>
> On Thu, Mar 1, 2018 at 10:57 PM, David Turner 
> wrote:
>
> Blocked requests and slow requests are synonyms in ceph. They are 2 names
> for the exact same thing.
>
>
> On Thu, Mar 1, 2018, 10:21 PM Alex Gorbachev 
> wrote:
>
>
> On Thu, Mar 1, 2018 at 2:47 PM, David Turner 
> wrote:
>
> `ceph health detail` should show you more information about the slow
> requests.  If the output is too much stuff, you can grep out for blocked
> or
> something.  It should tell you which OSDs are involved, how long they've
> been slow, etc.  The default is for them to show '> 32 sec' but that may
> very well be much longer and `ceph health detail` will show that.
>
>
> Hi David,
>
> Thank you for the reply.  Unfortunately, the health detail only shows
> blocked requests.  This seems to be related to a compression setting
> on the pool, nothing in OSD logs.
>
> I replied to another compression thread.  This makes sense since
> compression is new, and in the past all such issues were reflected in
> OSD logs and related to either network or OSD hardware.
>
> Regards,
> Alex
>
>
> On Thu, Mar 1, 2018 at 2:23 PM Alex Gorbachev 
> wrote:
>
>
> Is there a switch to turn on the display of specific OSD issues?  Or
> does the below indicate a generic problem, e.g. network and no any
> specific OSD?
>
> 2018-02-28 18:09:36.438300 7f6dead56700  0
> mon.roc-vm-sc3c234@0(leader).data_health(46) update_stats avail 56%
> total 15997 MB, used 6154 MB, avail 9008 MB
> 2018-02-28 18:09:41.477216 7f6dead56700  0 log_channel(cluster) log
> [WRN] : Health check failed: 73 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-02-28 18:09:47.552669 7f6dead56700  0 log_channel(cluster) log
> [WRN] : Health check update: 74 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-02-28 18:09:53.794882 7f6de8551700  0
> mon.roc-vm-sc3c234@0(leader) e1 handle_command mon_command({"prefix":
> "status", "format": "json"} v 0) v1
>
> --
>
>
> I was wrong where the pool compression does not matter, even
> uncompressed pool also generates these slow messages.
>
> Question is why no subsequent message relating to specific OSDs (like
> in Jewel and prior, like this example from RH:
>
> 2015-08-24 13:18:10.024659 osd.1 127.0.0.1:6812/3032 9 : cluster [WRN]
> 6 slow requests, 6 included below; oldest blocked for > 61.758455 secs
>
> 2016-07-25 03:44:06.510583 osd.50 [WRN] slow request 30.005692 seconds
> old, received at {date-time}: osd_op(client.4240.0:8
> benchmark_data_ceph-1_39426_object7 [write 0~4194304] 0.69848840) v4
> currently waiting for subops from [610]
>
> In comparison, my Luminous cluster only shows the general slow/blocked
> message:
>
> 2018-03-01 21:52:54.237270 7f7e419e3700  0 log_channel(cluster) log
> [WRN] : Health check failed: 116 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-03-01 21:53:00.282721 7f7e419e3700  0 log_channel(cluster) log
> [WRN] : Health check update: 66 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-03-01 21:53:08.534244 7f7e419e3700  0 log_channel(cluster) log
> [WRN] : Health check update: 5 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-03-01 21:53:10.382510 7f7e419e3700  0 log_channel(cluster) log
> [INF] : Health check cleared: REQUEST_SLOW (was: 5 slow requests are
> blocked > 32 sec)
> 2018-03-01 21:53:10.382546 7f7e419e3700  0 log_channel(cluster) log
> [INF] : Cluster is now healthy
>
> So where are the details?
>
> Thanks,
> Alex
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
> Hi Alex,
>
> I agree, the health detail should display osd info.
> Regarding the slow requests: ESX is very sensitive to io delay, this could
> be due to cluster overload or disk going bad. If you have a monitoring
> tool, do you see high load spikes (such as disk % busy ) on your disks ?
> Generally it is good (if you can) simulate the max client workload +
> simulate concurrent recovery and scrubbing and monitor your resources and
> see if they get overloaded. Most if not all issues with ESX will go away if
> you have enough hardware. I suspect this is what you are see-ing, specially
> the VMotion and compression could be the extra load which triggered the
> issue, in such case the problem is not specific to a particular osd. The
> other possibility is that a particular disk is going bad, the monitoring
> tool should be able to identify this.
>
> Note that VMotion generates an io pattern that keeps hitting the same osd
> with concurrent small writes, if you can use rbd striping, creating 64k
> stripes may help. A controller with write back cache will also help.
>
> Maged
>

Thank you, Maged. This does seem to be related to compression. I did two
things - turned off compression on pool and tried vmotion to same image -
failed.

Then I made another image and that tim

Re: [ceph-users] Slow requests troubleshooting in Luminous - details missing

[Maged Mokhtar]

On 2018-03-02 07:54, Alex Gorbachev wrote:

> On Thu, Mar 1, 2018 at 10:57 PM, David Turner  wrote: 
> Blocked requests and slow requests are synonyms in ceph. They are 2 names
> for the exact same thing.
> 
> On Thu, Mar 1, 2018, 10:21 PM Alex Gorbachev  
> wrote: 
> On Thu, Mar 1, 2018 at 2:47 PM, David Turner 
> wrote: `ceph health detail` should show you more information about the slow
> requests.  If the output is too much stuff, you can grep out for blocked
> or
> something.  It should tell you which OSDs are involved, how long they've
> been slow, etc.  The default is for them to show '> 32 sec' but that may
> very well be much longer and `ceph health detail` will show that. 
> Hi David,
> 
> Thank you for the reply.  Unfortunately, the health detail only shows
> blocked requests.  This seems to be related to a compression setting
> on the pool, nothing in OSD logs.
> 
> I replied to another compression thread.  This makes sense since
> compression is new, and in the past all such issues were reflected in
> OSD logs and related to either network or OSD hardware.
> 
> Regards,
> Alex
> 
> On Thu, Mar 1, 2018 at 2:23 PM Alex Gorbachev 
> wrote: 
> Is there a switch to turn on the display of specific OSD issues?  Or
> does the below indicate a generic problem, e.g. network and no any
> specific OSD?
> 
> 2018-02-28 18:09:36.438300 7f6dead56700  0
> mon.roc-vm-sc3c234@0(leader).data_health(46) update_stats avail 56%
> total 15997 MB, used 6154 MB, avail 9008 MB
> 2018-02-28 18:09:41.477216 7f6dead56700  0 log_channel(cluster) log
> [WRN] : Health check failed: 73 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-02-28 18:09:47.552669 7f6dead56700  0 log_channel(cluster) log
> [WRN] : Health check update: 74 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-02-28 18:09:53.794882 7f6de8551700  0
> mon.roc-vm-sc3c234@0(leader) e1 handle_command mon_command({"prefix":
> "status", "format": "json"} v 0) v1
> 
> --

I was wrong where the pool compression does not matter, even
uncompressed pool also generates these slow messages.

Question is why no subsequent message relating to specific OSDs (like
in Jewel and prior, like this example from RH:

2015-08-24 13:18:10.024659 osd.1 127.0.0.1:6812/3032 9 : cluster [WRN]
6 slow requests, 6 included below; oldest blocked for > 61.758455 secs

2016-07-25 03:44:06.510583 osd.50 [WRN] slow request 30.005692 seconds
old, received at {date-time}: osd_op(client.4240.0:8
benchmark_data_ceph-1_39426_object7 [write 0~4194304] 0.69848840) v4
currently waiting for subops from [610]

In comparison, my Luminous cluster only shows the general slow/blocked
message:

2018-03-01 21:52:54.237270 7f7e419e3700  0 log_channel(cluster) log
[WRN] : Health check failed: 116 slow requests are blocked > 32 sec
(REQUEST_SLOW)
2018-03-01 21:53:00.282721 7f7e419e3700  0 log_channel(cluster) log
[WRN] : Health check update: 66 slow requests are blocked > 32 sec
(REQUEST_SLOW)
2018-03-01 21:53:08.534244 7f7e419e3700  0 log_channel(cluster) log
[WRN] : Health check update: 5 slow requests are blocked > 32 sec
(REQUEST_SLOW)
2018-03-01 21:53:10.382510 7f7e419e3700  0 log_channel(cluster) log
[INF] : Health check cleared: REQUEST_SLOW (was: 5 slow requests are
blocked > 32 sec)
2018-03-01 21:53:10.382546 7f7e419e3700  0 log_channel(cluster) log
[INF] : Cluster is now healthy

So where are the details?

Thanks,
Alex
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

Hi Alex,

I agree, the health detail should display osd info.
Regarding the slow requests: ESX is very sensitive to io delay, this
could be due to cluster overload or disk going bad. If you have a
monitoring tool, do you see high load spikes (such as disk % busy ) on
your disks ? Generally it is good (if you can) simulate the max client
workload + simulate concurrent recovery and scrubbing and monitor your
resources and see if they get overloaded. Most if not all issues with
ESX will go away if you have enough hardware. I suspect this is what you
are see-ing, specially the VMotion and compression could be the extra
load which triggered the issue, in such case the problem is not specific
to a particular osd. The other possibility is that a particular disk is
going bad, the monitoring tool should be able to identify this. 

Note that VMotion generates an io pattern that keeps hitting the same
osd with concurrent small writes, if you can use rbd striping, creating
64k stripes may help. A controller with write back cache will also help.

Maged___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests troubleshooting in Luminous - details missing

[Alex Gorbachev]

On Thu, Mar 1, 2018 at 10:57 PM, David Turner  wrote:
> Blocked requests and slow requests are synonyms in ceph. They are 2 names
> for the exact same thing.
>
>
> On Thu, Mar 1, 2018, 10:21 PM Alex Gorbachev  wrote:
>>
>> On Thu, Mar 1, 2018 at 2:47 PM, David Turner 
>> wrote:
>> > `ceph health detail` should show you more information about the slow
>> > requests.  If the output is too much stuff, you can grep out for blocked
>> > or
>> > something.  It should tell you which OSDs are involved, how long they've
>> > been slow, etc.  The default is for them to show '> 32 sec' but that may
>> > very well be much longer and `ceph health detail` will show that.
>>
>> Hi David,
>>
>> Thank you for the reply.  Unfortunately, the health detail only shows
>> blocked requests.  This seems to be related to a compression setting
>> on the pool, nothing in OSD logs.
>>
>> I replied to another compression thread.  This makes sense since
>> compression is new, and in the past all such issues were reflected in
>> OSD logs and related to either network or OSD hardware.
>>
>> Regards,
>> Alex
>>
>> >
>> > On Thu, Mar 1, 2018 at 2:23 PM Alex Gorbachev 
>> > wrote:
>> >>
>> >> Is there a switch to turn on the display of specific OSD issues?  Or
>> >> does the below indicate a generic problem, e.g. network and no any
>> >> specific OSD?
>> >>
>> >> 2018-02-28 18:09:36.438300 7f6dead56700  0
>> >> mon.roc-vm-sc3c234@0(leader).data_health(46) update_stats avail 56%
>> >> total 15997 MB, used 6154 MB, avail 9008 MB
>> >> 2018-02-28 18:09:41.477216 7f6dead56700  0 log_channel(cluster) log
>> >> [WRN] : Health check failed: 73 slow requests are blocked > 32 sec
>> >> (REQUEST_SLOW)
>> >> 2018-02-28 18:09:47.552669 7f6dead56700  0 log_channel(cluster) log
>> >> [WRN] : Health check update: 74 slow requests are blocked > 32 sec
>> >> (REQUEST_SLOW)
>> >> 2018-02-28 18:09:53.794882 7f6de8551700  0
>> >> mon.roc-vm-sc3c234@0(leader) e1 handle_command mon_command({"prefix":
>> >> "status", "format": "json"} v 0) v1
>> >>
>> >> --

I was wrong where the pool compression does not matter, even
uncompressed pool also generates these slow messages.

Question is why no subsequent message relating to specific OSDs (like
in Jewel and prior, like this example from RH:

2015-08-24 13:18:10.024659 osd.1 127.0.0.1:6812/3032 9 : cluster [WRN]
6 slow requests, 6 included below; oldest blocked for > 61.758455 secs

2016-07-25 03:44:06.510583 osd.50 [WRN] slow request 30.005692 seconds
old, received at {date-time}: osd_op(client.4240.0:8
benchmark_data_ceph-1_39426_object7 [write 0~4194304] 0.69848840) v4
currently waiting for subops from [610]

In comparison, my Luminous cluster only shows the general slow/blocked message:

2018-03-01 21:52:54.237270 7f7e419e3700  0 log_channel(cluster) log
[WRN] : Health check failed: 116 slow requests are blocked > 32 sec
(REQUEST_SLOW)
2018-03-01 21:53:00.282721 7f7e419e3700  0 log_channel(cluster) log
[WRN] : Health check update: 66 slow requests are blocked > 32 sec
(REQUEST_SLOW)
2018-03-01 21:53:08.534244 7f7e419e3700  0 log_channel(cluster) log
[WRN] : Health check update: 5 slow requests are blocked > 32 sec
(REQUEST_SLOW)
2018-03-01 21:53:10.382510 7f7e419e3700  0 log_channel(cluster) log
[INF] : Health check cleared: REQUEST_SLOW (was: 5 slow requests are
blocked > 32 sec)
2018-03-01 21:53:10.382546 7f7e419e3700  0 log_channel(cluster) log
[INF] : Cluster is now healthy

So where are the details?

Thanks,
Alex
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests troubleshooting in Luminous - details missing

[David Turner]

Blocked requests and slow requests are synonyms in ceph. They are 2 names
for the exact same thing.

On Thu, Mar 1, 2018, 10:21 PM Alex Gorbachev  wrote:

> On Thu, Mar 1, 2018 at 2:47 PM, David Turner 
> wrote:
> > `ceph health detail` should show you more information about the slow
> > requests.  If the output is too much stuff, you can grep out for blocked
> or
> > something.  It should tell you which OSDs are involved, how long they've
> > been slow, etc.  The default is for them to show '> 32 sec' but that may
> > very well be much longer and `ceph health detail` will show that.
>
> Hi David,
>
> Thank you for the reply.  Unfortunately, the health detail only shows
> blocked requests.  This seems to be related to a compression setting
> on the pool, nothing in OSD logs.
>
> I replied to another compression thread.  This makes sense since
> compression is new, and in the past all such issues were reflected in
> OSD logs and related to either network or OSD hardware.
>
> Regards,
> Alex
>
> >
> > On Thu, Mar 1, 2018 at 2:23 PM Alex Gorbachev 
> > wrote:
> >>
> >> Is there a switch to turn on the display of specific OSD issues?  Or
> >> does the below indicate a generic problem, e.g. network and no any
> >> specific OSD?
> >>
> >> 2018-02-28 18:09:36.438300 7f6dead56700  0
> >> mon.roc-vm-sc3c234@0(leader).data_health(46) update_stats avail 56%
> >> total 15997 MB, used 6154 MB, avail 9008 MB
> >> 2018-02-28 18:09:41.477216 7f6dead56700  0 log_channel(cluster) log
> >> [WRN] : Health check failed: 73 slow requests are blocked > 32 sec
> >> (REQUEST_SLOW)
> >> 2018-02-28 18:09:47.552669 7f6dead56700  0 log_channel(cluster) log
> >> [WRN] : Health check update: 74 slow requests are blocked > 32 sec
> >> (REQUEST_SLOW)
> >> 2018-02-28 18:09:53.794882 7f6de8551700  0
> >> mon.roc-vm-sc3c234@0(leader) e1 handle_command mon_command({"prefix":
> >> "status", "format": "json"} v 0) v1
> >>
> >> --
> >> Alex Gorbachev
> >> Storcium
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests troubleshooting in Luminous - details missing

[Alex Gorbachev]

On Thu, Mar 1, 2018 at 2:47 PM, David Turner  wrote:
> `ceph health detail` should show you more information about the slow
> requests.  If the output is too much stuff, you can grep out for blocked or
> something.  It should tell you which OSDs are involved, how long they've
> been slow, etc.  The default is for them to show '> 32 sec' but that may
> very well be much longer and `ceph health detail` will show that.

Hi David,

Thank you for the reply.  Unfortunately, the health detail only shows
blocked requests.  This seems to be related to a compression setting
on the pool, nothing in OSD logs.

I replied to another compression thread.  This makes sense since
compression is new, and in the past all such issues were reflected in
OSD logs and related to either network or OSD hardware.

Regards,
Alex

>
> On Thu, Mar 1, 2018 at 2:23 PM Alex Gorbachev 
> wrote:
>>
>> Is there a switch to turn on the display of specific OSD issues?  Or
>> does the below indicate a generic problem, e.g. network and no any
>> specific OSD?
>>
>> 2018-02-28 18:09:36.438300 7f6dead56700  0
>> mon.roc-vm-sc3c234@0(leader).data_health(46) update_stats avail 56%
>> total 15997 MB, used 6154 MB, avail 9008 MB
>> 2018-02-28 18:09:41.477216 7f6dead56700  0 log_channel(cluster) log
>> [WRN] : Health check failed: 73 slow requests are blocked > 32 sec
>> (REQUEST_SLOW)
>> 2018-02-28 18:09:47.552669 7f6dead56700  0 log_channel(cluster) log
>> [WRN] : Health check update: 74 slow requests are blocked > 32 sec
>> (REQUEST_SLOW)
>> 2018-02-28 18:09:53.794882 7f6de8551700  0
>> mon.roc-vm-sc3c234@0(leader) e1 handle_command mon_command({"prefix":
>> "status", "format": "json"} v 0) v1
>>
>> --
>> Alex Gorbachev
>> Storcium
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow requests troubleshooting in Luminous - details missing

[David Turner]

`ceph health detail` should show you more information about the slow
requests.  If the output is too much stuff, you can grep out for blocked or
something.  It should tell you which OSDs are involved, how long they've
been slow, etc.  The default is for them to show '> 32 sec' but that may
very well be much longer and `ceph health detail` will show that.

On Thu, Mar 1, 2018 at 2:23 PM Alex Gorbachev 
wrote:

> Is there a switch to turn on the display of specific OSD issues?  Or
> does the below indicate a generic problem, e.g. network and no any
> specific OSD?
>
> 2018-02-28 18:09:36.438300 7f6dead56700  0
> mon.roc-vm-sc3c234@0(leader).data_health(46) update_stats avail 56%
> total 15997 MB, used 6154 MB, avail 9008 MB
> 2018-02-28 18:09:41.477216 7f6dead56700  0 log_channel(cluster) log
> [WRN] : Health check failed: 73 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-02-28 18:09:47.552669 7f6dead56700  0 log_channel(cluster) log
> [WRN] : Health check update: 74 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-02-28 18:09:53.794882 7f6de8551700  0
> mon.roc-vm-sc3c234@0(leader) e1 handle_command mon_command({"prefix":
> "status", "format": "json"} v 0) v1
>
> --
> Alex Gorbachev
> Storcium
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Slow requests troubleshooting in Luminous - details missing

[Alex Gorbachev]

Is there a switch to turn on the display of specific OSD issues?  Or
does the below indicate a generic problem, e.g. network and no any
specific OSD?

2018-02-28 18:09:36.438300 7f6dead56700  0
mon.roc-vm-sc3c234@0(leader).data_health(46) update_stats avail 56%
total 15997 MB, used 6154 MB, avail 9008 MB
2018-02-28 18:09:41.477216 7f6dead56700  0 log_channel(cluster) log
[WRN] : Health check failed: 73 slow requests are blocked > 32 sec
(REQUEST_SLOW)
2018-02-28 18:09:47.552669 7f6dead56700  0 log_channel(cluster) log
[WRN] : Health check update: 74 slow requests are blocked > 32 sec
(REQUEST_SLOW)
2018-02-28 18:09:53.794882 7f6de8551700  0
mon.roc-vm-sc3c234@0(leader) e1 handle_command mon_command({"prefix":
"status", "format": "json"} v 0) v1

--
Alex Gorbachev
Storcium
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow requests on a specific osd

[lists]

Hi Wes,

On 15-1-2018 20:57, Wes Dillingham wrote:
My understanding is that the exact same objects would move back to the 
OSD if weight went 1 -> 0 -> 1 given the same Cluster state and same 
object names, CRUSH is deterministic so that would be the almost certain 
result.




Ok, thanks! So this would be a useless exercise. :-|

Thanks very much for your feedback, Wes!

MJ
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com