Re: [ceph-users] Upgrading journals to BlueStore: a conundrum

[Gregory Farnum]

You could try flushing out the FileStore journals off the SSD and creating
new ones elsewhere (eg, colocated). This will obviously have a substantial
impact on performance but perhaps that’s acceptable during your upgrade
window?

On Mon, Aug 6, 2018 at 12:32 PM Robert Stanford 
wrote:

>
>  Eugen: I've tried similar approaches in the past and it seems like it
> won't work like that.  I have to zap the entire journal disk.  Also I plan
> to use the configuration tunable for making the bluestore partition (wal,
> db) larger than the default
>
> On Mon, Aug 6, 2018 at 2:30 PM, Eugen Block  wrote:
>
>> Hi,
>>
>>  How then can one upgrade journals to BlueStore when there is more than
>>> one
>>> journal on the same disk?
>>>
>>
>> if you're using one SSD for multiple OSDs the disk probably has several
>> partitions. So you could just zap one partition at a time and replace the
>> OSD. Or am I misunderstanding the question?
>>
>> Regards,
>> Eugen
>>
>>
>> Zitat von Bastiaan Visser :
>>
>>
>> As long as your fault domain is host (or even rack) you're good, just
>>> take out the entire host and recreate all osd's on it.
>>>
>>>
>>> - Original Message -
>>> From: "Robert Stanford" 
>>> To: "ceph-users" 
>>> Sent: Monday, August 6, 2018 8:39:07 PM
>>> Subject: [ceph-users] Upgrading journals to BlueStore: a conundrum
>>>
>>> According to the instructions to upgrade a journal to BlueStore (
>>> http://docs.ceph.com/docs/master/rados/operations/bluestore-migration/),
>>> the OSD that uses the journal is destroyed and recreated.
>>>
>>>  I am using SSD journals, and want to use them with BlueStore.  Reusing
>>> the
>>> SSD requires zapping the disk (ceph-disk zap).  But this would take down
>>> all OSDs that use this journal, not just the one-at-a-time that I destroy
>>> and recreate when following the upgrade instructions.
>>>
>>>  How then can one upgrade journals to BlueStore when there is more than
>>> one
>>> journal on the same disk?
>>>
>>>  R
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Ceph-deploy] Cluster Name

[Thode Jocelyn]

Hi Erik,

The thing is that the rbd-mirror service uses the /etc/sysconfig/ceph file to 
determine which configuration file to use (from CLUSTER_NAME). So you need to 
set this to the name you chose for rbd-mirror to work. However setting this 
CLUSTER_NAME variable in /etc/sysconfig/ceph makes it so that the mon, osd etc 
services will also use this variable. Because of this they cannot start anymore 
as all their path are set with "ceph" as cluster name.

However there might be something that I missed which would make this point moot

Best Regards
Jocelyn Thode 

-Original Message-
From: Erik McCormick [mailto:emccorm...@cirrusseven.com] 
Sent: mercredi, 8 août 2018 16:39
To: Thode Jocelyn 
Cc: Glen Baars ; Vasu Kulkarni 
; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name

I'm not using this feature, so maybe I'm missing something, but from the way I 
understand cluster naming to work...

I still don't understand why this is blocking for you. Unless you are 
attempting to mirror between two clusters running on the same hosts (why would 
you do this?) then systemd doesn't come into play. The --cluster flag on the 
rbd command will simply set the name of a configuration file with the FSID and 
settings of the appropriate cluster. Cluster name is just a way of telling ceph 
commands and systemd units where to find the configs.

So, what you end up with is something like:

/etc/ceph/ceph.conf (your local cluster configuration) on both clusters 
/etc/ceph/local.conf (config of the source cluster. Just a copy of ceph.conf of 
the source clsuter) /etc/ceph/remote.conf (config of destination peer cluster. 
Just a copy of ceph.conf of the remote cluster).

Run all your rbd mirror commands against local and remote names.
However when starting things like mons, osds, mds, etc. you need no cluster 
name as it can use ceph.conf (cluster name of ceph).

Am I making sense, or have I completely missed something?

-Erik

On Wed, Aug 8, 2018 at 8:34 AM, Thode Jocelyn  wrote:
> Hi,
>
>
>
> We are still blocked by this problem on our end. Glen did you  or 
> someone else figure out something for this ?
>
>
>
> Regards
>
> Jocelyn Thode
>
>
>
> From: Glen Baars [mailto:g...@onsitecomputers.com.au]
> Sent: jeudi, 2 août 2018 05:43
> To: Erik McCormick 
> Cc: Thode Jocelyn ; Vasu Kulkarni 
> ; ceph-users@lists.ceph.com
> Subject: RE: [ceph-users] [Ceph-deploy] Cluster Name
>
>
>
> Hello Erik,
>
>
>
> We are going to use RBD-mirror to replicate the clusters. This seems 
> to need separate cluster names.
>
> Kind regards,
>
> Glen Baars
>
>
>
> From: Erik McCormick 
> Sent: Thursday, 2 August 2018 9:39 AM
> To: Glen Baars 
> Cc: Thode Jocelyn ; Vasu Kulkarni 
> ; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name
>
>
>
> Don't set a cluster name. It's no longer supported. It really only 
> matters if you're running two or more independent clusters on the same 
> boxes. That's generally inadvisable anyway.
>
>
>
> Cheers,
>
> Erik
>
>
>
> On Wed, Aug 1, 2018, 9:17 PM Glen Baars  wrote:
>
> Hello Ceph Users,
>
> Does anyone know how to set the Cluster Name when deploying with 
> Ceph-deploy? I have 3 clusters to configure and need to correctly set 
> the name.
>
> Kind regards,
> Glen Baars
>
> -Original Message-
> From: ceph-users  On Behalf Of Glen 
> Baars
> Sent: Monday, 23 July 2018 5:59 PM
> To: Thode Jocelyn ; Vasu Kulkarni 
> 
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name
>
> How very timely, I am facing the exact same issue.
>
> Kind regards,
> Glen Baars
>
> -Original Message-
> From: ceph-users  On Behalf Of 
> Thode Jocelyn
> Sent: Monday, 23 July 2018 1:42 PM
> To: Vasu Kulkarni 
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name
>
> Hi,
>
> Yes my rbd-mirror is coloctaed with my mon/osd. It only affects nodes 
> where they are collocated as they all use the "/etc/sysconfig/ceph" 
> configuration file.
>
> Best
> Jocelyn Thode
>
> -Original Message-
> From: Vasu Kulkarni [mailto:vakul...@redhat.com]
> Sent: vendredi, 20 juillet 2018 17:25
> To: Thode Jocelyn 
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name
>
> On Fri, Jul 20, 2018 at 7:29 AM, Thode Jocelyn 
> wrote:
>> Hi,
>>
>>
>>
>> I noticed that in commit
>> https://github.com/ceph/ceph-deploy/commit/b1c27b85d524f2553af2487a98
>> 0 23b60efe421f3, the ability to specify a cluster name was removed. 
>> Is there a reason for this removal ?
>>
>>
>>
>> Because right now, there are no possibility to create a ceph cluster 
>> with a different name with ceph-deploy which is a big problem when 
>> having two clusters replicating with rbd-mirror as we need different 
>> names.
>>
>>
>>
>> And even when following the doc here:
>> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/
>> h 
>> 

Re: [ceph-users] permission errors rolling back ceph cluster to v13

[Raju Rangoju]

Thanks Greg.

I think I have to re-install ceph v13 from scratch then.

-Raju
From: Gregory Farnum 
Sent: 09 August 2018 01:54
To: Raju Rangoju 
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] permission errors rolling back ceph cluster to v13

On Tue, Aug 7, 2018 at 6:27 PM Raju Rangoju 
mailto:ra...@chelsio.com>> wrote:
Hi,

I have been running into some connection issues with the latest ceph-14 
version, so we thought the feasible solution would be to roll back the cluster 
to previous version (ceph-13.0.1) where things are known to work properly.

I’m wondering if rollback/downgrade is supported at all ?

After compiling/starting ceph-13 I’m running into a permission error. Basically 
it complains about the incompatibility of disk layout (ceph-13 mimic vs ceph-14 
nautilus)

2018-08-07 10:41:00.580 2b391528e080 -1 ERROR: on disk data includes 
unsupported features: compat={},rocompat={},incompat={11=nautilus ondisk layout}
2018-08-07 10:41:00.580 2b391528e080 -1 error checking features: (1) Operation 
not permitted
2018-08-07 10:41:11.161 2b16a7d14080  0 set uid:gid to 167:167 (ceph:ceph)
2018-08-07 10:41:11.161 2b16a7d14080  0 ceph version 13.0.1-3266-g6b59fbf 
(6b59fbfcc6bbfd67193e1c1e142b478ddd68aab4) mimic (dev), process (unknown), pid 
14013
2018-08-07 10:41:11.161 2b16a7d14080  0 pidfile_write: ignore empty --pid-file


I thought allowing permissions to mon would fix it (see below), but apparently 
ceph command hangs. So it didn’t allow permissions.

ceph auth add osd.44 osd 'allow *' mon 'allow profile osd' -i 
/var/lib/ceph/osd/ceph-44/keyring

[root@hadoop1 my-ceph]# ceph -s
2018-08-07 10:59:59.325 2b5c26347700  0 monclient(hunting): authenticate timed 
out after 300
2018-08-07 10:59:59.325 2b5c26347700  0 librados: client.admin authentication 
error (110) Connection timed out
[errno 110] error connecting to the cluster

Has anyone tried ceph rollback before? Any help is greatly appreciated.

Unfortunately this is generally not possible. Disk encodings change across 
major versions and the old code can't understand what's been written down once 
the new code runs.
-Greg


Thanks,
Raj

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD had suicide timed out

[Brad Hubbard]

If, in the above case, osd 13 was not too busy to respond (resource
shortage) then you need to find out why else osd 5, etc. could not
contact it.

On Wed, Aug 8, 2018 at 6:47 PM, Josef Zelenka
 wrote:
> Checked the system load on the host with the OSD that is suiciding currently
> and it's fine, however i can see a noticeably higher IO (around 700), though
> that seems rather like a symptom of the constant flapping/attempting to come
> up to me(it's an SSD based Ceph so this shouldn't cause much harm to it).
> Had a look at one of the osds sending the you_died messages and it seems
> it's attempting to contact osd.13, but ultimately fails.
>
> 8/0 13574/13574/13574) [5,11] r=0 lpr=13574 crt=13592'3654839 lcod
> 13592'3654838 mlcod 13592'3654838 active+clean] publish_stats_to_osd
> 13593:9552151
> 2018-08-08 10:45:16.112344 7effa1d8c700 15 osd.5 pg_epoch: 13593 pg[14.6( v
> 13592'3654839 (13294'3653334,13592'3654839] local-lis/les=13574/13575 n=945
> ec=126/126 lis/c 13574/13574 les/c/f 13575/13578/0 13574/13574/13574) [5,11]
> r=0 lpr=13574 crt=13592'3654839 lcod 13592'3654838 mlcod 13592'3654838
> active+clean] publish_stats_to_osd 13593:9552152
> 2018-08-08 10:45:16.679484 7eff9a57d700 15 osd.5 pg_epoch: 13593 pg[11.15( v
> 13575'34486 (9987'32956,13575'34486] local-lis/les=13574/13575 n=1
> ec=115/115 lis/c 13574/13574 les/c/f 13575/13575/0 13574/13574/13574) [5,10]
> r=0 lpr=13574 crt=13572'34485 lcod 13572'34485 mlcod 13572'34485
> active+clean] publish_stats_to_osd 13593:2966967
> 2018-08-08 10:45:17.818135 7effb95a4700  1 -- 10.12.125.1:6803/1319081 <==
> osd.13 10.12.125.3:0/735946 18  osd_ping(ping e13589 stamp 2018-08-08
> 10:45:17.817238) v4  2004+0+0 (4218069135 0 0) 0x55bb638ba800 con
> 0x55bb65e79800
> 2018-08-08 10:45:17.818176 7effb9da5700  1 -- 10.12.3.15:6809/1319081 <==
> osd.13 10.12.3.17:0/735946 18  osd_ping(ping e13589 stamp 2018-08-08
> 10:45:17.817238) v4  2004+0+0 (4218069135 0 0) 0x55bb63cd8c00 con
> 0x55bb65e7b000
> 2018-08-08 10:45:18.919053 7effb95a4700  1 -- 10.12.125.1:6803/1319081 <==
> osd.13 10.12.125.3:0/735946 19  osd_ping(ping e13589 stamp 2018-08-08
> 10:45:18.918149) v4  2004+0+0 (1428835292 0 0) 0x55bb638bb200 con
> 0x55bb65e79800
> 2018-08-08 10:45:18.919598 7effb9da5700  1 -- 10.12.3.15:6809/1319081 <==
> osd.13 10.12.3.17:0/735946 19  osd_ping(ping e13589 stamp 2018-08-08
> 10:45:18.918149) v4  2004+0+0 (1428835292 0 0) 0x55bb63cd8a00 con
> 0x55bb65e7b000
> 2018-08-08 10:45:21.679563 7eff9a57d700 15 osd.5 pg_epoch: 13593 pg[11.15( v
> 13575'34486 (9987'32956,13575'34486] local-lis/les=13574/13575 n=1
> ec=115/115 lis/c 13574/13574 les/c/f 13575/13575/0 13574/13574/13574) [5,10]
> r=0 lpr=13574 crt=13572'34485 lcod 13572'34485 mlcod 13572'34485
> active+clean] publish_stats_to_osd 13593:2966968
> 2018-08-08 10:45:23.020715 7effb95a4700  1 -- 10.12.125.1:6803/1319081 <==
> osd.13 10.12.125.3:0/735946 20  osd_ping(ping e13589 stamp 2018-08-08
> 10:45:23.018994) v4  2004+0+0 (1018071233 0 0) 0x55bb63bb7200 con
> 0x55bb65e79800
> 2018-08-08 10:45:23.020837 7effb9da5700  1 -- 10.12.3.15:6809/1319081 <==
> osd.13 10.12.3.17:0/735946 20  osd_ping(ping e13589 stamp 2018-08-08
> 10:45:23.018994) v4  2004+0+0 (1018071233 0 0) 0x55bb63cd8c00 con
> 0x55bb65e7b000
> 2018-08-08 10:45:26.679513 7eff8e565700 15 osd.5 pg_epoch: 13593 pg[11.15( v
> 13575'34486 (9987'32956,13575'34486] local-lis/les=13574/13575 n=1
> ec=115/115 lis/c 13574/13574 les/c/f 13575/13575/0 13574/13574/13574) [5,10]
> r=0 lpr=13574 crt=13572'34485 lcod 13572'34485 mlcod 13572'34485
> active+clean] publish_stats_to_osd 13593:2966969
> 2018-08-08 10:45:28.921091 7effb95a4700  1 -- 10.12.125.1:6803/1319081 <==
> osd.13 10.12.125.3:0/735946 21  osd_ping(ping e13589 stamp 2018-08-08
> 10:45:28.920140) v4  2004+0+0 (2459835898 0 0) 0x55bb638ba800 con
> 0x55bb65e79800
> 2018-08-08 10:45:28.922026 7effb9da5700  1 -- 10.12.3.15:6809/1319081 <==
> osd.13 10.12.3.17:0/735946 21  osd_ping(ping e13589 stamp 2018-08-08
> 10:45:28.920140) v4  2004+0+0 (2459835898 0 0) 0x55bb63cd8c00 con
> 0x55bb65e7b000
> 2018-08-08 10:45:31.679828 7eff9a57d700 15 osd.5 pg_epoch: 13593 pg[11.15( v
> 13575'34486 (9987'32956,13575'34486] local-lis/les=13574/13575 n=1
> ec=115/115 lis/c 13574/13574 les/c/f 13575/13575/0 13574/13574/13574) [5,10]
> r=0 lpr=13574 crt=13572'34485 lcod 13572'34485 mlcod 13572'34485
> active+clean] publish_stats_to_osd 13593:2966970
> 2018-08-08 10:45:33.022697 7effb95a4700  1 -- 10.12.125.1:6803/1319081 <==
> osd.13 10.12.125.3:0/735946 22  osd_ping(ping e13589 stamp 2018-08-08
> 10:45:33.021217) v4  2004+0+0 (3639738084 0 0) 0x55bb63bb7200 con
> 0x55bb65e79800
>
> Regarding heartbeat messages, all i can see on the failing osd is "heartbeat
> map is healthy" before the timeout messages. I'm thinking this might be a
> pool issue of some sort? as this happens on random osds throughout the
> cluster.
>
>
>
> On 08/08/18 09:45, Brad Hubbard 

Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?

[Alexandre DERUMIER]

Hi,

I have upgraded to 12.2.7 , 2 weeks ago,
and I don't see anymore memory increase !  (can't confirm that it was related 
to your patch).


Thanks again for helping !

Regards,

Alexandre Derumier


- Mail original -
De: "Zheng Yan" 
À: "aderumier" 
Cc: "ceph-users" 
Envoyé: Mardi 29 Mai 2018 04:40:27
Objet: Re: [ceph-users] ceph mds memory usage 20GB : is it normal ?

Could you try path https://github.com/ceph/ceph/pull/22240/files. 

The leakage of MMDSBeacon messages can explain your issue. 

Regards 
Yan, Zheng 





On Mon, May 28, 2018 at 12:06 PM, Alexandre DERUMIER 
 wrote: 
>>>could you send me full output of dump_mempools 
> 
> # ceph daemon mds.ceph4-2.odiso.net dump_mempools 
> { 
> "bloom_filter": { 
> "items": 41262668, 
> "bytes": 41262668 
> }, 
> "bluestore_alloc": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "bluestore_cache_data": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "bluestore_cache_onode": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "bluestore_cache_other": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "bluestore_fsck": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "bluestore_txc": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "bluestore_writing_deferred": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "bluestore_writing": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "bluefs": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "buffer_anon": { 
> "items": 712726, 
> "bytes": 106964870 
> }, 
> "buffer_meta": { 
> "items": 15, 
> "bytes": 1320 
> }, 
> "osd": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "osd_mapbl": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "osd_pglog": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "osdmap": { 
> "items": 216, 
> "bytes": 12168 
> }, 
> "osdmap_mapping": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "pgmap": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "mds_co": { 
> "items": 50741038, 
> "bytes": 5114319203 
> }, 
> "unittest_1": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "unittest_2": { 
> "items": 0, 
> "bytes": 0 
> }, 
> "total": { 
> "items": 92716663, 
> "bytes": 5262560229 
> } 
> } 
> 
> 
> 
> 
> 
> ceph daemon mds.ceph4-2.odiso.net perf dump 
> { 
> "AsyncMessenger::Worker-0": { 
> "msgr_recv_messages": 1276789161, 
> "msgr_send_messages": 1317625246, 
> "msgr_recv_bytes": 10630409633633, 
> "msgr_send_bytes": 1093972769957, 
> "msgr_created_connections": 207, 
> "msgr_active_connections": 204, 
> "msgr_running_total_time": 63745.463077594, 
> "msgr_running_send_time": 22210.867549070, 
> "msgr_running_recv_time": 51944.624353942, 
> "msgr_running_fast_dispatch_time": 9185.274084187 
> }, 
> "AsyncMessenger::Worker-1": { 
> "msgr_recv_messages": 641622644, 
> "msgr_send_messages": 616664293, 
> "msgr_recv_bytes": 7287546832466, 
> "msgr_send_bytes": 588278035895, 
> "msgr_created_connections": 494, 
> "msgr_active_connections": 494, 
> "msgr_running_total_time": 35390.081250881, 
> "msgr_running_send_time": 11559.689889195, 
> "msgr_running_recv_time": 29844.885712902, 
> "msgr_running_fast_dispatch_time": 6361.466445253 
> }, 
> "AsyncMessenger::Worker-2": { 
> "msgr_recv_messages": 1972469623, 
> "msgr_send_messages": 1886060294, 
> "msgr_recv_bytes": 7924136565846, 
> "msgr_send_bytes": 5072502101797, 
> "msgr_created_connections": 181, 
> "msgr_active_connections": 176, 
> "msgr_running_total_time": 93257.811989806, 
> "msgr_running_send_time": 35556.662488302, 
> "msgr_running_recv_time": 81686.262228047, 
> "msgr_running_fast_dispatch_time": 6476.875317930 
> }, 
> "finisher-PurgeQueue": { 
> "queue_len": 0, 
> "complete_latency": { 
> "avgcount": 3390753, 
> "sum": 44364.742135193, 
> "avgtime": 0.013084038 
> } 
> }, 
> "mds": { 
> "request": 2780760988, 
> "reply": 2780760950, 
> "reply_latency": { 
> "avgcount": 2780760950, 
> "sum": 8467119.492491407, 
> "avgtime": 0.003044892 
> }, 
> "forward": 0, 
> "dir_fetch": 173374097, 
> "dir_commit": 3235888, 
> "dir_split": 23, 
> "dir_merge": 45, 
> "inode_max": 2147483647, 
> "inodes": 1762555, 
> "inodes_top": 388540, 
> "inodes_bottom": 173389, 
> "inodes_pin_tail": 1200626, 
> "inodes_pinned": 1207497, 
> "inodes_expired": 32837415801, 
> "inodes_with_caps": 1206864, 
> "caps": 1565063, 
> "subtrees": 2, 
> "traverse": 2976675748, 
> "traverse_hit": 1725898480, 
> "traverse_forward": 0, 
> "traverse_discover": 0, 
> "traverse_dir_fetch": 157542892, 
> "traverse_remote_ino": 46197, 
> "traverse_lock": 294516, 
> "load_cent": 18446743922292121894, 
> "q": 169, 
> "exported": 0, 
> "exported_inodes": 0, 
> "imported": 0, 
> "imported_inodes": 0 
> }, 
> "mds_cache": { 
> "num_strays": 6004, 
> "num_strays_delayed": 23, 
> "num_strays_enqueuing": 0, 
> "strays_created": 3123475, 
> "strays_enqueued": 3118819, 
> "strays_reintegrated": 1279, 
> "strays_migrated": 0, 
> "num_recovering_processing": 0, 
> "num_recovering_enqueued": 0, 
> "num_recovering_prioritized": 0, 
> "recovery_started": 17, 
> "recovery_completed": 17, 
> "ireq_enqueue_scrub": 0, 
> "ireq_exportdir": 0, 
> "ireq_flush": 0, 
> "ireq_fragmentdir": 68, 
> "ireq_fragstats": 0, 
> 

Re: [ceph-users] Slack-IRC integration

[Gregory Farnum]

I looked at this a bit and it turns out anybody who's already in the slack
group can invite people with unrestricted domains. I think it's just part
of Slack that you need to specify which domains are allowed in by default?
Patrick set things up a couple years ago so I suppose our next community
manager will be the lucky inheritor of responsibility.

Anyway, as I have one of the allowed emails I set it up and sent out
invites. In future hopefully somebody remembers that more quickly! :)
-Greg

On Sat, Jul 28, 2018 at 3:39 PM Dan van der Ster  wrote:

> It's here https://ceph-storage.slack.com/ but for some reason the list of
> accepted email domains is limited. I have no idea who is maintaining this.
>
> Anyway, the slack is just mirroring #ceph and #ceph-devel on IRC so better
> to connect there directly.
>
> Cheers, Dan
>
>
>
>
> On Sat, Jul 28, 2018, 6:59 PM Alex Gorbachev 
> wrote:
>
>> -- Forwarded message --
>> From: Matt.Brown 
>>
>>
>> Can you please add me to the ceph-storage slack channel?  Thanks!
>>
>>
>> Me too, please
>>
>> --
>> Alex Gorbachev
>> Storcium
>>
>>
>> - Matt Brown | Lead Engineer | Infrastructure Services – Cloud &
>> Compute | Target | 7000 Target Pkwy N., NCE-0706 | Brooklyn Park, MN
>> 55445 | 612.304.4956 <(612)%20304-4956>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] removing auids and auid-based cephx capabilities

[Sage Weil]

There is an undocumented part of the cephx authentication framework called 
the 'auid' (auth uid) that assigns an integer identifier to cephx users 
and to rados pools and allows you to craft cephx capabilities that apply 
to those pools.  This is leftover infrastructure from an ancient time in 
which RGW buckets mapped 1:1 to rados pools (pre-argonaut!) and it was 
expected the cephx capabilities would line up with that.

Although in theory parts of the auid infrastructure might work and be in 
use, it is undocumented, untested, and a messy artifact in the code.  I'd 
like to remove it.

***

  If you are using auid-based cephx capabilities, now is the time to tell 
  us!  Or, if you know of any reason we should keep it around, now is 
  the time to speak up.

  Otherwise we will remove it!

***

Thanks!
sage

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] permission errors rolling back ceph cluster to v13

[Gregory Farnum]

On Tue, Aug 7, 2018 at 6:27 PM Raju Rangoju  wrote:

> Hi,
>
>
>
> I have been running into some connection issues with the latest ceph-14
> version, so we thought the feasible solution would be to roll back the
> cluster to previous version (ceph-13.0.1) where things are known to work
> properly.
>
>
>
> I’m wondering if rollback/downgrade is supported at all ?
>
>
>
> After compiling/starting ceph-13 I’m running into a permission error.
> Basically it complains about the incompatibility of disk layout (ceph-13
> mimic vs ceph-14 nautilus)
>
>
>
> 2018-08-07 10:41:00.580 2b391528e080 -1 ERROR: on disk data includes
> unsupported features: compat={},rocompat={},incompat={11=nautilus ondisk
> layout}
>
> 2018-08-07 10:41:00.580 2b391528e080 -1 error checking features: (1)
> Operation not permitted
>
> 2018-08-07 10:41:11.161 2b16a7d14080  0 set uid:gid to 167:167 (ceph:ceph)
>
> 2018-08-07 10:41:11.161 2b16a7d14080  0 ceph version 13.0.1-3266-g6b59fbf
> (6b59fbfcc6bbfd67193e1c1e142b478ddd68aab4) mimic (dev), process (unknown),
> pid 14013
>
> 2018-08-07 10:41:11.161 2b16a7d14080  0 pidfile_write: ignore empty
> --pid-file
>
>
>
>
>
> I thought allowing permissions to mon would fix it (see below), but
> apparently ceph command hangs. So it didn’t allow permissions.
>
>
>
> ceph auth add osd.44 osd 'allow *' mon 'allow profile osd' -i
> /var/lib/ceph/osd/ceph-44/keyring
>
>
>
> [root@hadoop1 my-ceph]# ceph -s
>
> 2018-08-07 10:59:59.325 2b5c26347700  0 monclient(hunting): authenticate
> timed out after 300
>
> 2018-08-07 10:59:59.325 2b5c26347700  0 librados: client.admin
> authentication error (110) Connection timed out
>
> [errno 110] error connecting to the cluster
>
>
>
> Has anyone tried ceph rollback before? Any help is greatly appreciated.
>

Unfortunately this is generally not possible. Disk encodings change across
major versions and the old code can't understand what's been written down
once the new code runs.
-Greg


>
>
> Thanks,
>
> Raj
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs kernel client hangs

[John Spray]

On Wed, Aug 8, 2018 at 4:46 PM Jake Grimmett  wrote:
>
> Hi John,
>
> With regard to memory pressure; Does the cephfs fuse client also cause a
> deadlock - or is this just the kernel client?

TBH, I'm not expert enough on the kernel-side implementation of fuse
to say.  Ceph does have the fuse_disable_pagecache that might reduce
the probability of issues if you're committed to running clients and
servers on the same node.

> We run the fuse client on ten OSD nodes, and use parsync (parallel
> rsync) to backup two beegfs systems (~1PB).
>
> Ordinarily fuse works OK, but any OSD problems can cause an out of
> memory error on other osd threads as they recover, e.g.:
>
> kernel: [] out_of_memory+0x4b6/0x4f0
> kernel: Out of memory: Kill process 1927903 (ceph-osd) score 27 or
> sacrifice child
>
> Limiting bluestore_cache (as follows) prevents the OOM error, and allows
> us to run the cephfs fuse client reliably:
>
> bluestore_cache_size = 209715200
> bluestore_cache_kv_max = 134217728
>
> We have 45 OSD's per box, 128GB RAM. Dual E5-2620 v4
> mimic 13.2.1, Load average 16 or so is normal...
>
> Could our OOM errors (with a default config) be caused by us running
> cephfs fuse on the osd servers?

I wouldn't rule it out, but this is also a pretty high density of OSDs
per node to begin with.  If each OSD is at least a few terabytes,
you're the wrong side of the rule of thumb on resources (1GB RAM per
TB of OSD storage).  I'd also be concerned about having only one
quarter of a CPU core for each OSD.  Sounds like you've got your
settings tuned to something that's working in practice though, so I
wouldn't mess with it :-)

John

>
> many thanks!
>
> Jake
>
> On 07/08/18 20:36, John Spray wrote:
> > On Tue, Aug 7, 2018 at 5:42 PM Reed Dier  wrote:
> >>
> >> This is the first I am hearing about this as well.
> >
> > This is not a Ceph-specific thing -- it can also affect similar
> > systems like Lustre.
> >
> > The classic case is when under some memory pressure, the kernel tries
> > to free memory by flushing the client's page cache, but doing the
> > flush means allocating more memory on the server, making the memory
> > pressure worse, until the whole thing just seizes up.
> >
> > John
> >
> >> Granted, I am using ceph-fuse rather than the kernel client at this point, 
> >> but that isn’t etched in stone.
> >>
> >> Curious if there is more to share.
> >>
> >> Reed
> >>
> >> On Aug 7, 2018, at 9:47 AM, Webert de Souza Lima  
> >> wrote:
> >>
> >>
> >> Yan, Zheng  于2018年8月7日周二 下午7:51写道：
> >>>
> >>> On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou  wrote:
> >>> this can cause memory deadlock. you should avoid doing this
> >>>
>  Yan, Zheng 于2018年8月7日 周二19:12写道：
> >
> > did you mount cephfs on the same machines that run ceph-osd?
> >
> >>
> >>
> >> I didn't know about this. I run this setup in production. :P
> >>
> >> Regards,
> >>
> >> Webert Lima
> >> DevOps Engineer at MAV Tecnologia
> >> Belo Horizonte - Brasil
> >> IRC NICK - WebertRLZ
> >>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs kernel client hangs

[Jake Grimmett]

Hi John,

With regard to memory pressure; Does the cephfs fuse client also cause a
deadlock - or is this just the kernel client?

We run the fuse client on ten OSD nodes, and use parsync (parallel
rsync) to backup two beegfs systems (~1PB).

Ordinarily fuse works OK, but any OSD problems can cause an out of
memory error on other osd threads as they recover, e.g.:

kernel: [] out_of_memory+0x4b6/0x4f0
kernel: Out of memory: Kill process 1927903 (ceph-osd) score 27 or
sacrifice child

Limiting bluestore_cache (as follows) prevents the OOM error, and allows
us to run the cephfs fuse client reliably:

bluestore_cache_size = 209715200
bluestore_cache_kv_max = 134217728

We have 45 OSD's per box, 128GB RAM. Dual E5-2620 v4
mimic 13.2.1, Load average 16 or so is normal...

Could our OOM errors (with a default config) be caused by us running
cephfs fuse on the osd servers?

many thanks!

Jake

On 07/08/18 20:36, John Spray wrote:
> On Tue, Aug 7, 2018 at 5:42 PM Reed Dier  wrote:
>>
>> This is the first I am hearing about this as well.
> 
> This is not a Ceph-specific thing -- it can also affect similar
> systems like Lustre.
> 
> The classic case is when under some memory pressure, the kernel tries
> to free memory by flushing the client's page cache, but doing the
> flush means allocating more memory on the server, making the memory
> pressure worse, until the whole thing just seizes up.
> 
> John
> 
>> Granted, I am using ceph-fuse rather than the kernel client at this point, 
>> but that isn’t etched in stone.
>>
>> Curious if there is more to share.
>>
>> Reed
>>
>> On Aug 7, 2018, at 9:47 AM, Webert de Souza Lima  
>> wrote:
>>
>>
>> Yan, Zheng  于2018年8月7日周二 下午7:51写道：
>>>
>>> On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou  wrote:
>>> this can cause memory deadlock. you should avoid doing this
>>>
 Yan, Zheng 于2018年8月7日 周二19:12写道：
>
> did you mount cephfs on the same machines that run ceph-osd?
>
>>
>>
>> I didn't know about this. I run this setup in production. :P
>>
>> Regards,
>>
>> Webert Lima
>> DevOps Engineer at MAV Tecnologia
>> Belo Horizonte - Brasil
>> IRC NICK - WebertRLZ
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs kernel client hangs

[Webert de Souza Lima]

You can only try to remount the cephs dir. It will probably not work,
giving you I/O Errors, so the fallback would be to use a fuse-mount.

If I recall correctly you could do a lazy umount on the current dir (umount
-fl /mountdir) and remount it using the FUSE client.
it will work for new sessions but the currently hanging ones will still be
hanging.

with fuse you'll only be able to mount cephfs root dir, so if you have
multiple directories, you'll need to:
 - mount root cephfs dir in another directory
 - mount each subdir (after root mounted) to the desired directory via bind
mount.


Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
*IRC NICK - WebertRLZ*


On Wed, Aug 8, 2018 at 11:46 AM Zhenshi Zhou  wrote:

> Hi,
> Is there any other way excpet rebooting the server when the client hangs?
> If the server is in production environment, I can't restart it everytime.
>
> Webert de Souza Lima  于2018年8月8日周三 下午10:33写道：
>
>> Hi Zhenshi,
>>
>> if you still have the client mount hanging but no session is connected,
>> you probably have some PID waiting with blocked IO from cephfs mount.
>> I face that now and then and the only solution is to reboot the server,
>> as you won't be able to kill a process with pending IO.
>>
>> Regards,
>>
>> Webert Lima
>> DevOps Engineer at MAV Tecnologia
>> *Belo Horizonte - Brasil*
>> *IRC NICK - WebertRLZ*
>>
>>
>> On Wed, Aug 8, 2018 at 11:17 AM Zhenshi Zhou  wrote:
>>
>>> Hi Webert,
>>> That command shows the current sessions, whereas the server which I get
>>> the files(osdc,mdsc,monc) disconnect for a long time.
>>> So I cannot get useful infomation from the command you provide.
>>>
>>> Thanks
>>>
>>> Webert de Souza Lima  于2018年8月8日周三 下午10:10写道：
>>>
 You could also see open sessions at the MDS server by issuing  `ceph
 daemon mds.XX session ls`

 Regards,

 Webert Lima
 DevOps Engineer at MAV Tecnologia
 *Belo Horizonte - Brasil*
 *IRC NICK - WebertRLZ*


 On Wed, Aug 8, 2018 at 5:08 AM Zhenshi Zhou 
 wrote:

> Hi, I find an old server which mounted cephfs and has the debug files.
> # cat osdc
> REQUESTS 0 homeless 0
> LINGER REQUESTS
> BACKOFFS
> # cat monc
> have monmap 2 want 3+
> have osdmap 3507
> have fsmap.user 0
> have mdsmap 55 want 56+
> fs_cluster_id -1
> # cat mdsc
> 194 mds0getattr  #1036ae3
>
> What does it mean?
>
> Zhenshi Zhou  于2018年8月8日周三 下午1:58写道：
>
>> I restarted the client server so that there's no file in that
>> directory. I will take care of it if the client hangs next time.
>>
>> Thanks
>>
>> Yan, Zheng  于2018年8月8日周三 上午11:23写道：
>>
>>> On Wed, Aug 8, 2018 at 11:02 AM Zhenshi Zhou 
>>> wrote:
>>> >
>>> > Hi,
>>> > I check all my ceph servers and they are not mount cephfs on each
>>> of them(maybe I umount after testing). As a result, the cluster didn't
>>> encounter a memory deadlock. Besides, I check the monitoring system and 
>>> the
>>> memory and cpu usage were at common level while the clients hung.
>>> > Back to my question, there must be something else cause the client
>>> hang.
>>> >
>>>
>>> Check if there are hang requests in
>>> /sys/kernel/debug/ceph//{osdc,mdsc},
>>>
>>> > Zhenshi Zhou  于2018年8月8日周三 上午4:16写道：
>>> >>
>>> >> Hi, I'm not sure if it just mounts the cephfs without using or
>>> doing any operation within the mounted directory would be affected by
>>> flushing cache. I mounted cephfs on osd servers only for testing and 
>>> then
>>> left it there. Anyway I will umount it.
>>> >>
>>> >> Thanks
>>> >>
>>> >> John Spray 于2018年8月8日 周三03:37写道：
>>> >>>
>>> >>> On Tue, Aug 7, 2018 at 5:42 PM Reed Dier 
>>> wrote:
>>> >>> >
>>> >>> > This is the first I am hearing about this as well.
>>> >>>
>>> >>> This is not a Ceph-specific thing -- it can also affect similar
>>> >>> systems like Lustre.
>>> >>>
>>> >>> The classic case is when under some memory pressure, the kernel
>>> tries
>>> >>> to free memory by flushing the client's page cache, but doing the
>>> >>> flush means allocating more memory on the server, making the
>>> memory
>>> >>> pressure worse, until the whole thing just seizes up.
>>> >>>
>>> >>> John
>>> >>>
>>> >>> > Granted, I am using ceph-fuse rather than the kernel client at
>>> this point, but that isn’t etched in stone.
>>> >>> >
>>> >>> > Curious if there is more to share.
>>> >>> >
>>> >>> > Reed
>>> >>> >
>>> >>> > On Aug 7, 2018, at 9:47 AM, Webert de Souza Lima <
>>> webert.b...@gmail.com> wrote:
>>> >>> >
>>> >>> >
>>> >>> > Yan, Zheng  于2018年8月7日周二 下午7:51写道：
>>> >>> >>
>>> >>> >> On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou <
>>> deader...@gmail.com> wrote:
>>> >>> >> 

Re: [ceph-users] cephfs kernel client hangs

[Zhenshi Zhou]

Hi,
Is there any other way excpet rebooting the server when the client hangs?
If the server is in production environment, I can't restart it everytime.

Webert de Souza Lima  于2018年8月8日周三 下午10:33写道：

> Hi Zhenshi,
>
> if you still have the client mount hanging but no session is connected,
> you probably have some PID waiting with blocked IO from cephfs mount.
> I face that now and then and the only solution is to reboot the server, as
> you won't be able to kill a process with pending IO.
>
> Regards,
>
> Webert Lima
> DevOps Engineer at MAV Tecnologia
> *Belo Horizonte - Brasil*
> *IRC NICK - WebertRLZ*
>
>
> On Wed, Aug 8, 2018 at 11:17 AM Zhenshi Zhou  wrote:
>
>> Hi Webert,
>> That command shows the current sessions, whereas the server which I get
>> the files(osdc,mdsc,monc) disconnect for a long time.
>> So I cannot get useful infomation from the command you provide.
>>
>> Thanks
>>
>> Webert de Souza Lima  于2018年8月8日周三 下午10:10写道：
>>
>>> You could also see open sessions at the MDS server by issuing  `ceph
>>> daemon mds.XX session ls`
>>>
>>> Regards,
>>>
>>> Webert Lima
>>> DevOps Engineer at MAV Tecnologia
>>> *Belo Horizonte - Brasil*
>>> *IRC NICK - WebertRLZ*
>>>
>>>
>>> On Wed, Aug 8, 2018 at 5:08 AM Zhenshi Zhou  wrote:
>>>
 Hi, I find an old server which mounted cephfs and has the debug files.
 # cat osdc
 REQUESTS 0 homeless 0
 LINGER REQUESTS
 BACKOFFS
 # cat monc
 have monmap 2 want 3+
 have osdmap 3507
 have fsmap.user 0
 have mdsmap 55 want 56+
 fs_cluster_id -1
 # cat mdsc
 194 mds0getattr  #1036ae3

 What does it mean?

 Zhenshi Zhou  于2018年8月8日周三 下午1:58写道：

> I restarted the client server so that there's no file in that
> directory. I will take care of it if the client hangs next time.
>
> Thanks
>
> Yan, Zheng  于2018年8月8日周三 上午11:23写道：
>
>> On Wed, Aug 8, 2018 at 11:02 AM Zhenshi Zhou 
>> wrote:
>> >
>> > Hi,
>> > I check all my ceph servers and they are not mount cephfs on each
>> of them(maybe I umount after testing). As a result, the cluster didn't
>> encounter a memory deadlock. Besides, I check the monitoring system and 
>> the
>> memory and cpu usage were at common level while the clients hung.
>> > Back to my question, there must be something else cause the client
>> hang.
>> >
>>
>> Check if there are hang requests in
>> /sys/kernel/debug/ceph//{osdc,mdsc},
>>
>> > Zhenshi Zhou  于2018年8月8日周三 上午4:16写道：
>> >>
>> >> Hi, I'm not sure if it just mounts the cephfs without using or
>> doing any operation within the mounted directory would be affected by
>> flushing cache. I mounted cephfs on osd servers only for testing and then
>> left it there. Anyway I will umount it.
>> >>
>> >> Thanks
>> >>
>> >> John Spray 于2018年8月8日 周三03:37写道：
>> >>>
>> >>> On Tue, Aug 7, 2018 at 5:42 PM Reed Dier 
>> wrote:
>> >>> >
>> >>> > This is the first I am hearing about this as well.
>> >>>
>> >>> This is not a Ceph-specific thing -- it can also affect similar
>> >>> systems like Lustre.
>> >>>
>> >>> The classic case is when under some memory pressure, the kernel
>> tries
>> >>> to free memory by flushing the client's page cache, but doing the
>> >>> flush means allocating more memory on the server, making the
>> memory
>> >>> pressure worse, until the whole thing just seizes up.
>> >>>
>> >>> John
>> >>>
>> >>> > Granted, I am using ceph-fuse rather than the kernel client at
>> this point, but that isn’t etched in stone.
>> >>> >
>> >>> > Curious if there is more to share.
>> >>> >
>> >>> > Reed
>> >>> >
>> >>> > On Aug 7, 2018, at 9:47 AM, Webert de Souza Lima <
>> webert.b...@gmail.com> wrote:
>> >>> >
>> >>> >
>> >>> > Yan, Zheng  于2018年8月7日周二 下午7:51写道：
>> >>> >>
>> >>> >> On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou <
>> deader...@gmail.com> wrote:
>> >>> >> this can cause memory deadlock. you should avoid doing this
>> >>> >>
>> >>> >> > Yan, Zheng 于2018年8月7日 周二19:12写道：
>> >>> >> >>
>> >>> >> >> did you mount cephfs on the same machines that run ceph-osd?
>> >>> >> >>
>> >>> >
>> >>> >
>> >>> > I didn't know about this. I run this setup in production. :P
>> >>> >
>> >>> > Regards,
>> >>> >
>> >>> > Webert Lima
>> >>> > DevOps Engineer at MAV Tecnologia
>> >>> > Belo Horizonte - Brasil
>> >>> > IRC NICK - WebertRLZ
>> >>> >
>> >>> > ___
>> >>> > ceph-users mailing list
>> >>> > ceph-users@lists.ceph.com
>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>> >
>> >>> >
>> >>> > ___
>> >>> > ceph-users mailing list
>> >>> > 

Re: [ceph-users] [Ceph-deploy] Cluster Name

[Erik McCormick]

I'm not using this feature, so maybe I'm missing something, but from
the way I understand cluster naming to work...

I still don't understand why this is blocking for you. Unless you are
attempting to mirror between two clusters running on the same hosts
(why would you do this?) then systemd doesn't come into play. The
--cluster flag on the rbd command will simply set the name of a
configuration file with the FSID and settings of the appropriate
cluster. Cluster name is just a way of telling ceph commands and
systemd units where to find the configs.

So, what you end up with is something like:

/etc/ceph/ceph.conf (your local cluster configuration) on both clusters
/etc/ceph/local.conf (config of the source cluster. Just a copy of
ceph.conf of the source clsuter)
/etc/ceph/remote.conf (config of destination peer cluster. Just a copy
of ceph.conf of the remote cluster).

Run all your rbd mirror commands against local and remote names.
However when starting things like mons, osds, mds, etc. you need no
cluster name as it can use ceph.conf (cluster name of ceph).

Am I making sense, or have I completely missed something?

-Erik

On Wed, Aug 8, 2018 at 8:34 AM, Thode Jocelyn  wrote:
> Hi,
>
>
>
> We are still blocked by this problem on our end. Glen did you  or someone
> else figure out something for this ?
>
>
>
> Regards
>
> Jocelyn Thode
>
>
>
> From: Glen Baars [mailto:g...@onsitecomputers.com.au]
> Sent: jeudi, 2 août 2018 05:43
> To: Erik McCormick 
> Cc: Thode Jocelyn ; Vasu Kulkarni
> ; ceph-users@lists.ceph.com
> Subject: RE: [ceph-users] [Ceph-deploy] Cluster Name
>
>
>
> Hello Erik,
>
>
>
> We are going to use RBD-mirror to replicate the clusters. This seems to need
> separate cluster names.
>
> Kind regards,
>
> Glen Baars
>
>
>
> From: Erik McCormick 
> Sent: Thursday, 2 August 2018 9:39 AM
> To: Glen Baars 
> Cc: Thode Jocelyn ; Vasu Kulkarni
> ; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name
>
>
>
> Don't set a cluster name. It's no longer supported. It really only matters
> if you're running two or more independent clusters on the same boxes. That's
> generally inadvisable anyway.
>
>
>
> Cheers,
>
> Erik
>
>
>
> On Wed, Aug 1, 2018, 9:17 PM Glen Baars  wrote:
>
> Hello Ceph Users,
>
> Does anyone know how to set the Cluster Name when deploying with
> Ceph-deploy? I have 3 clusters to configure and need to correctly set the
> name.
>
> Kind regards,
> Glen Baars
>
> -Original Message-
> From: ceph-users  On Behalf Of Glen Baars
> Sent: Monday, 23 July 2018 5:59 PM
> To: Thode Jocelyn ; Vasu Kulkarni
> 
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name
>
> How very timely, I am facing the exact same issue.
>
> Kind regards,
> Glen Baars
>
> -Original Message-
> From: ceph-users  On Behalf Of Thode
> Jocelyn
> Sent: Monday, 23 July 2018 1:42 PM
> To: Vasu Kulkarni 
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name
>
> Hi,
>
> Yes my rbd-mirror is coloctaed with my mon/osd. It only affects nodes where
> they are collocated as they all use the "/etc/sysconfig/ceph" configuration
> file.
>
> Best
> Jocelyn Thode
>
> -Original Message-
> From: Vasu Kulkarni [mailto:vakul...@redhat.com]
> Sent: vendredi, 20 juillet 2018 17:25
> To: Thode Jocelyn 
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name
>
> On Fri, Jul 20, 2018 at 7:29 AM, Thode Jocelyn 
> wrote:
>> Hi,
>>
>>
>>
>> I noticed that in commit
>> https://github.com/ceph/ceph-deploy/commit/b1c27b85d524f2553af2487a980
>> 23b60efe421f3, the ability to specify a cluster name was removed. Is
>> there a reason for this removal ?
>>
>>
>>
>> Because right now, there are no possibility to create a ceph cluster
>> with a different name with ceph-deploy which is a big problem when
>> having two clusters replicating with rbd-mirror as we need different
>> names.
>>
>>
>>
>> And even when following the doc here:
>> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/h
>> tml/block_device_guide/block_device_mirroring#rbd-mirroring-clusters-w
>> ith-the-same-name
>>
>>
>>
>> This is not sufficient as once we change the CLUSTER variable in the
>> sysconfig file, mon,osd, mds etc. all use it and fail to start on a
>> reboot as they then try to load data from a path in /var/lib/ceph
>> containing the cluster name.
>
> Is you rbd-mirror client also colocated with mon/osd? This needs to be
> changed only on the client side where you are doing mirroring, rest of the
> nodes are not affected?
>
>
>>
>>
>>
>> Is there a solution to this problem ?
>>
>>
>>
>> Best Regards
>>
>> Jocelyn Thode
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> 

Re: [ceph-users] cephfs kernel client hangs

[Webert de Souza Lima]

Hi Zhenshi,

if you still have the client mount hanging but no session is connected, you
probably have some PID waiting with blocked IO from cephfs mount.
I face that now and then and the only solution is to reboot the server, as
you won't be able to kill a process with pending IO.

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
*IRC NICK - WebertRLZ*


On Wed, Aug 8, 2018 at 11:17 AM Zhenshi Zhou  wrote:

> Hi Webert,
> That command shows the current sessions, whereas the server which I get
> the files(osdc,mdsc,monc) disconnect for a long time.
> So I cannot get useful infomation from the command you provide.
>
> Thanks
>
> Webert de Souza Lima  于2018年8月8日周三 下午10:10写道：
>
>> You could also see open sessions at the MDS server by issuing  `ceph
>> daemon mds.XX session ls`
>>
>> Regards,
>>
>> Webert Lima
>> DevOps Engineer at MAV Tecnologia
>> *Belo Horizonte - Brasil*
>> *IRC NICK - WebertRLZ*
>>
>>
>> On Wed, Aug 8, 2018 at 5:08 AM Zhenshi Zhou  wrote:
>>
>>> Hi, I find an old server which mounted cephfs and has the debug files.
>>> # cat osdc
>>> REQUESTS 0 homeless 0
>>> LINGER REQUESTS
>>> BACKOFFS
>>> # cat monc
>>> have monmap 2 want 3+
>>> have osdmap 3507
>>> have fsmap.user 0
>>> have mdsmap 55 want 56+
>>> fs_cluster_id -1
>>> # cat mdsc
>>> 194 mds0getattr  #1036ae3
>>>
>>> What does it mean?
>>>
>>> Zhenshi Zhou  于2018年8月8日周三 下午1:58写道：
>>>
 I restarted the client server so that there's no file in that
 directory. I will take care of it if the client hangs next time.

 Thanks

 Yan, Zheng  于2018年8月8日周三 上午11:23写道：

> On Wed, Aug 8, 2018 at 11:02 AM Zhenshi Zhou 
> wrote:
> >
> > Hi,
> > I check all my ceph servers and they are not mount cephfs on each of
> them(maybe I umount after testing). As a result, the cluster didn't
> encounter a memory deadlock. Besides, I check the monitoring system and 
> the
> memory and cpu usage were at common level while the clients hung.
> > Back to my question, there must be something else cause the client
> hang.
> >
>
> Check if there are hang requests in
> /sys/kernel/debug/ceph//{osdc,mdsc},
>
> > Zhenshi Zhou  于2018年8月8日周三 上午4:16写道：
> >>
> >> Hi, I'm not sure if it just mounts the cephfs without using or
> doing any operation within the mounted directory would be affected by
> flushing cache. I mounted cephfs on osd servers only for testing and then
> left it there. Anyway I will umount it.
> >>
> >> Thanks
> >>
> >> John Spray 于2018年8月8日 周三03:37写道：
> >>>
> >>> On Tue, Aug 7, 2018 at 5:42 PM Reed Dier 
> wrote:
> >>> >
> >>> > This is the first I am hearing about this as well.
> >>>
> >>> This is not a Ceph-specific thing -- it can also affect similar
> >>> systems like Lustre.
> >>>
> >>> The classic case is when under some memory pressure, the kernel
> tries
> >>> to free memory by flushing the client's page cache, but doing the
> >>> flush means allocating more memory on the server, making the memory
> >>> pressure worse, until the whole thing just seizes up.
> >>>
> >>> John
> >>>
> >>> > Granted, I am using ceph-fuse rather than the kernel client at
> this point, but that isn’t etched in stone.
> >>> >
> >>> > Curious if there is more to share.
> >>> >
> >>> > Reed
> >>> >
> >>> > On Aug 7, 2018, at 9:47 AM, Webert de Souza Lima <
> webert.b...@gmail.com> wrote:
> >>> >
> >>> >
> >>> > Yan, Zheng  于2018年8月7日周二 下午7:51写道：
> >>> >>
> >>> >> On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou <
> deader...@gmail.com> wrote:
> >>> >> this can cause memory deadlock. you should avoid doing this
> >>> >>
> >>> >> > Yan, Zheng 于2018年8月7日 周二19:12写道：
> >>> >> >>
> >>> >> >> did you mount cephfs on the same machines that run ceph-osd?
> >>> >> >>
> >>> >
> >>> >
> >>> > I didn't know about this. I run this setup in production. :P
> >>> >
> >>> > Regards,
> >>> >
> >>> > Webert Lima
> >>> > DevOps Engineer at MAV Tecnologia
> >>> > Belo Horizonte - Brasil
> >>> > IRC NICK - WebertRLZ
> >>> >
> >>> > ___
> >>> > ceph-users mailing list
> >>> > ceph-users@lists.ceph.com
> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>> >
> >>> >
> >>> > ___
> >>> > ceph-users mailing list
> >>> > ceph-users@lists.ceph.com
> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>> ___
> >>> ceph-users mailing list
> >>> ceph-users@lists.ceph.com
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> > ___
> > ceph-users mailing list
> > 

Re: [ceph-users] cephfs kernel client hangs

[Zhenshi Zhou]

Hi Webert,
That command shows the current sessions, whereas the server which I get the
files(osdc,mdsc,monc) disconnect for a long time.
So I cannot get useful infomation from the command you provide.

Thanks

Webert de Souza Lima  于2018年8月8日周三 下午10:10写道：

> You could also see open sessions at the MDS server by issuing  `ceph
> daemon mds.XX session ls`
>
> Regards,
>
> Webert Lima
> DevOps Engineer at MAV Tecnologia
> *Belo Horizonte - Brasil*
> *IRC NICK - WebertRLZ*
>
>
> On Wed, Aug 8, 2018 at 5:08 AM Zhenshi Zhou  wrote:
>
>> Hi, I find an old server which mounted cephfs and has the debug files.
>> # cat osdc
>> REQUESTS 0 homeless 0
>> LINGER REQUESTS
>> BACKOFFS
>> # cat monc
>> have monmap 2 want 3+
>> have osdmap 3507
>> have fsmap.user 0
>> have mdsmap 55 want 56+
>> fs_cluster_id -1
>> # cat mdsc
>> 194 mds0getattr  #1036ae3
>>
>> What does it mean?
>>
>> Zhenshi Zhou  于2018年8月8日周三 下午1:58写道：
>>
>>> I restarted the client server so that there's no file in that directory.
>>> I will take care of it if the client hangs next time.
>>>
>>> Thanks
>>>
>>> Yan, Zheng  于2018年8月8日周三 上午11:23写道：
>>>
 On Wed, Aug 8, 2018 at 11:02 AM Zhenshi Zhou 
 wrote:
 >
 > Hi,
 > I check all my ceph servers and they are not mount cephfs on each of
 them(maybe I umount after testing). As a result, the cluster didn't
 encounter a memory deadlock. Besides, I check the monitoring system and the
 memory and cpu usage were at common level while the clients hung.
 > Back to my question, there must be something else cause the client
 hang.
 >

 Check if there are hang requests in
 /sys/kernel/debug/ceph//{osdc,mdsc},

 > Zhenshi Zhou  于2018年8月8日周三 上午4:16写道：
 >>
 >> Hi, I'm not sure if it just mounts the cephfs without using or doing
 any operation within the mounted directory would be affected by flushing
 cache. I mounted cephfs on osd servers only for testing and then left it
 there. Anyway I will umount it.
 >>
 >> Thanks
 >>
 >> John Spray 于2018年8月8日 周三03:37写道：
 >>>
 >>> On Tue, Aug 7, 2018 at 5:42 PM Reed Dier 
 wrote:
 >>> >
 >>> > This is the first I am hearing about this as well.
 >>>
 >>> This is not a Ceph-specific thing -- it can also affect similar
 >>> systems like Lustre.
 >>>
 >>> The classic case is when under some memory pressure, the kernel
 tries
 >>> to free memory by flushing the client's page cache, but doing the
 >>> flush means allocating more memory on the server, making the memory
 >>> pressure worse, until the whole thing just seizes up.
 >>>
 >>> John
 >>>
 >>> > Granted, I am using ceph-fuse rather than the kernel client at
 this point, but that isn’t etched in stone.
 >>> >
 >>> > Curious if there is more to share.
 >>> >
 >>> > Reed
 >>> >
 >>> > On Aug 7, 2018, at 9:47 AM, Webert de Souza Lima <
 webert.b...@gmail.com> wrote:
 >>> >
 >>> >
 >>> > Yan, Zheng  于2018年8月7日周二 下午7:51写道：
 >>> >>
 >>> >> On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou 
 wrote:
 >>> >> this can cause memory deadlock. you should avoid doing this
 >>> >>
 >>> >> > Yan, Zheng 于2018年8月7日 周二19:12写道：
 >>> >> >>
 >>> >> >> did you mount cephfs on the same machines that run ceph-osd?
 >>> >> >>
 >>> >
 >>> >
 >>> > I didn't know about this. I run this setup in production. :P
 >>> >
 >>> > Regards,
 >>> >
 >>> > Webert Lima
 >>> > DevOps Engineer at MAV Tecnologia
 >>> > Belo Horizonte - Brasil
 >>> > IRC NICK - WebertRLZ
 >>> >
 >>> > ___
 >>> > ceph-users mailing list
 >>> > ceph-users@lists.ceph.com
 >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 >>> >
 >>> >
 >>> > ___
 >>> > ceph-users mailing list
 >>> > ceph-users@lists.ceph.com
 >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 >>> ___
 >>> ceph-users mailing list
 >>> ceph-users@lists.ceph.com
 >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 >
 > ___
 > ceph-users mailing list
 > ceph-users@lists.ceph.com
 > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

>>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Whole cluster flapping

[Will Marley]

Hi again Frederic,

It may be worth looking at a recovery sleep.
osd recovery sleep
Description:

Time in seconds to sleep before next recovery or backfill op. Increasing this 
value will slow down recovery operation while client operations will be less 
impacted.

Type:

Float

Default:

0

osd recovery sleep hdd
Description:

Time in seconds to sleep before next recovery or backfill op for HDDs.

Type:

Float

Default:

0.1

osd recovery sleep ssd
Description:

Time in seconds to sleep before next recovery or backfill op for SSDs.

Type:

Float

Default:

0

osd recovery sleep hybrid
Description:

Time in seconds to sleep before next recovery or backfill op when osd data is 
on HDD and osd journal is on SSD.

Type:

Float

Default:

0.025


(Pulled from 
http://docs.ceph.com/docs/master/rados/configuration/osd-config-ref/)

When we faced similar issues, using the command ceph tell osd.* injectargs 
'--osd-recovery-sleep 2 allowed the OSDs to respond with a heartbeat whilst 
taking a break between recovery operations. I’d suggest tweaking the sleep wait 
time to find a sweet spot.

This may be worth a try, so let us know how you get on.

Regards,
Will

From: ceph-users  On Behalf Of Webert de 
Souza Lima
Sent: 08 August 2018 15:06
To: frederic.c...@sib.fr
Cc: ceph-users 
Subject: Re: [ceph-users] Whole cluster flapping

So your OSDs are really too busy to respond heartbeats.
You'll be facing this for sometime until cluster loads get lower.

I would set `ceph osd set nodeep-scrub` until the heavy disk IO stops.
maybe you can schedule it for enable during the night and disabling in the 
morning.

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
Belo Horizonte - Brasil
IRC NICK - WebertRLZ


On Wed, Aug 8, 2018 at 9:18 AM CUZA Frédéric 
mailto:frederic.c...@sib.fr>> wrote:
Thx for the command line, I did take a look too it what I don’t really know 
what to search for, my bad….
All this flapping is due to deep-scrub when it starts on an OSD things start to 
go bad.

I set out all the OSDs that were flapping the most (1 by 1 after rebalancing) 
and it looks better even if some osds keep going down/up with the same message 
in logs :

1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fdabd897700' had timed out 
after 90

(I update it to 90 instead of 15s)

Regards,



De : ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
De la part de Webert de Souza Lima
Envoyé : 07 August 2018 16:28
À : ceph-users mailto:ceph-users@lists.ceph.com>>
Objet : Re: [ceph-users] Whole cluster flapping

oops, my bad, you're right.

I don't know much you can see but maybe you can dig around performance counters 
and see what's happening on those OSDs, try these:

~# ceph daemonperf osd.XX
~# ceph daemon osd.XX perf dump

change XX to your OSD numbers.

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
Belo Horizonte - Brasil
IRC NICK - WebertRLZ


On Tue, Aug 7, 2018 at 10:47 AM CUZA Frédéric 
mailto:frederic.c...@sib.fr>> wrote:
Pool is already deleted and no longer present in stats.

Regards,

De : ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
De la part de Webert de Souza Lima
Envoyé : 07 August 2018 15:08
À : ceph-users mailto:ceph-users@lists.ceph.com>>
Objet : Re: [ceph-users] Whole cluster flapping

Frédéric,

see if the number of objects is decreasing in the pool with `ceph df [detail]`

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
Belo Horizonte - Brasil
IRC NICK - WebertRLZ


On Tue, Aug 7, 2018 at 5:46 AM CUZA Frédéric 
mailto:frederic.c...@sib.fr>> wrote:
It’s been over a week now and the whole cluster keeps flapping, it is never the 
same OSDs that go down.
Is there a way to get the progress of this recovery ? (The pool hat I deleted 
is no longer present (for a while now))
In fact, there is a lot of i/o activity on the server where osds go down.

Regards,

De : ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
De la part de Webert de Souza Lima
Envoyé : 31 July 2018 16:25
À : ceph-users mailto:ceph-users@lists.ceph.com>>
Objet : Re: [ceph-users] Whole cluster flapping

The pool deletion might have triggered a lot of IO operations on the disks and 
the process might be too busy to respond to hearbeats, so the mons mark them as 
down due to no response.
Check also the OSD logs to see if they are actually crashing and restarting, 
and disk IO usage (i.e. iostat).

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
Belo Horizonte - Brasil
IRC NICK - WebertRLZ


On Tue, Jul 31, 2018 at 7:23 AM CUZA Frédéric 
mailto:frederic.c...@sib.fr>> wrote:
Hi Everyone,

I just upgrade our cluster to Luminous 12.2.7 and I delete a quite large pool 
that we had (120 TB).
Our cluster is made of 14 Nodes with each composed of 12 OSDs (1 HDD -> 1 OSD), 
we have SDD for journal.

After I deleted the large pool my cluster started to flapping on all OSDs.
Osds are marked down and then marked up as follow :

2018-07-31 10:42:51.504319 mon.ceph_monitor01 [INF] osd.97 

Re: [ceph-users] cephfs kernel client hangs

[Webert de Souza Lima]

You could also see open sessions at the MDS server by issuing  `ceph daemon
mds.XX session ls`

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
*IRC NICK - WebertRLZ*


On Wed, Aug 8, 2018 at 5:08 AM Zhenshi Zhou  wrote:

> Hi, I find an old server which mounted cephfs and has the debug files.
> # cat osdc
> REQUESTS 0 homeless 0
> LINGER REQUESTS
> BACKOFFS
> # cat monc
> have monmap 2 want 3+
> have osdmap 3507
> have fsmap.user 0
> have mdsmap 55 want 56+
> fs_cluster_id -1
> # cat mdsc
> 194 mds0getattr  #1036ae3
>
> What does it mean?
>
> Zhenshi Zhou  于2018年8月8日周三 下午1:58写道：
>
>> I restarted the client server so that there's no file in that directory.
>> I will take care of it if the client hangs next time.
>>
>> Thanks
>>
>> Yan, Zheng  于2018年8月8日周三 上午11:23写道：
>>
>>> On Wed, Aug 8, 2018 at 11:02 AM Zhenshi Zhou 
>>> wrote:
>>> >
>>> > Hi,
>>> > I check all my ceph servers and they are not mount cephfs on each of
>>> them(maybe I umount after testing). As a result, the cluster didn't
>>> encounter a memory deadlock. Besides, I check the monitoring system and the
>>> memory and cpu usage were at common level while the clients hung.
>>> > Back to my question, there must be something else cause the client
>>> hang.
>>> >
>>>
>>> Check if there are hang requests in
>>> /sys/kernel/debug/ceph//{osdc,mdsc},
>>>
>>> > Zhenshi Zhou  于2018年8月8日周三 上午4:16写道：
>>> >>
>>> >> Hi, I'm not sure if it just mounts the cephfs without using or doing
>>> any operation within the mounted directory would be affected by flushing
>>> cache. I mounted cephfs on osd servers only for testing and then left it
>>> there. Anyway I will umount it.
>>> >>
>>> >> Thanks
>>> >>
>>> >> John Spray 于2018年8月8日 周三03:37写道：
>>> >>>
>>> >>> On Tue, Aug 7, 2018 at 5:42 PM Reed Dier 
>>> wrote:
>>> >>> >
>>> >>> > This is the first I am hearing about this as well.
>>> >>>
>>> >>> This is not a Ceph-specific thing -- it can also affect similar
>>> >>> systems like Lustre.
>>> >>>
>>> >>> The classic case is when under some memory pressure, the kernel tries
>>> >>> to free memory by flushing the client's page cache, but doing the
>>> >>> flush means allocating more memory on the server, making the memory
>>> >>> pressure worse, until the whole thing just seizes up.
>>> >>>
>>> >>> John
>>> >>>
>>> >>> > Granted, I am using ceph-fuse rather than the kernel client at
>>> this point, but that isn’t etched in stone.
>>> >>> >
>>> >>> > Curious if there is more to share.
>>> >>> >
>>> >>> > Reed
>>> >>> >
>>> >>> > On Aug 7, 2018, at 9:47 AM, Webert de Souza Lima <
>>> webert.b...@gmail.com> wrote:
>>> >>> >
>>> >>> >
>>> >>> > Yan, Zheng  于2018年8月7日周二 下午7:51写道：
>>> >>> >>
>>> >>> >> On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou 
>>> wrote:
>>> >>> >> this can cause memory deadlock. you should avoid doing this
>>> >>> >>
>>> >>> >> > Yan, Zheng 于2018年8月7日 周二19:12写道：
>>> >>> >> >>
>>> >>> >> >> did you mount cephfs on the same machines that run ceph-osd?
>>> >>> >> >>
>>> >>> >
>>> >>> >
>>> >>> > I didn't know about this. I run this setup in production. :P
>>> >>> >
>>> >>> > Regards,
>>> >>> >
>>> >>> > Webert Lima
>>> >>> > DevOps Engineer at MAV Tecnologia
>>> >>> > Belo Horizonte - Brasil
>>> >>> > IRC NICK - WebertRLZ
>>> >>> >
>>> >>> > ___
>>> >>> > ceph-users mailing list
>>> >>> > ceph-users@lists.ceph.com
>>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >>> >
>>> >>> >
>>> >>> > ___
>>> >>> > ceph-users mailing list
>>> >>> > ceph-users@lists.ceph.com
>>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >>> ___
>>> >>> ceph-users mailing list
>>> >>> ceph-users@lists.ceph.com
>>> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> >
>>> > ___
>>> > ceph-users mailing list
>>> > ceph-users@lists.ceph.com
>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Whole cluster flapping

[Webert de Souza Lima]

So your OSDs are really too busy to respond heartbeats.
You'll be facing this for sometime until cluster loads get lower.

I would set `ceph osd set nodeep-scrub` until the heavy disk IO stops.
maybe you can schedule it for enable during the night and disabling in the
morning.

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
*IRC NICK - WebertRLZ*


On Wed, Aug 8, 2018 at 9:18 AM CUZA Frédéric  wrote:

> Thx for the command line, I did take a look too it what I don’t really
> know what to search for, my bad….
>
> All this flapping is due to deep-scrub when it starts on an OSD things
> start to go bad.
>
>
>
> I set out all the OSDs that were flapping the most (1 by 1 after
> rebalancing) and it looks better even if some osds keep going down/up with
> the same message in logs :
>
>
>
> 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fdabd897700' had
> timed out after 90
>
>
>
> (I update it to 90 instead of 15s)
>
>
>
> Regards,
>
>
>
>
>
>
>
> *De :* ceph-users  *De la part de*
> Webert de Souza Lima
> *Envoyé :* 07 August 2018 16:28
> *À :* ceph-users 
> *Objet :* Re: [ceph-users] Whole cluster flapping
>
>
>
> oops, my bad, you're right.
>
>
>
> I don't know much you can see but maybe you can dig around performance
> counters and see what's happening on those OSDs, try these:
>
>
>
> ~# ceph daemonperf osd.XX
>
> ~# ceph daemon osd.XX perf dump
>
>
>
> change XX to your OSD numbers.
>
>
>
> Regards,
>
>
>
> Webert Lima
>
> DevOps Engineer at MAV Tecnologia
>
> *Belo Horizonte - Brasil*
>
> *IRC NICK - WebertRLZ*
>
>
>
>
>
> On Tue, Aug 7, 2018 at 10:47 AM CUZA Frédéric 
> wrote:
>
> Pool is already deleted and no longer present in stats.
>
>
>
> Regards,
>
>
>
> *De :* ceph-users  *De la part de*
> Webert de Souza Lima
> *Envoyé :* 07 August 2018 15:08
> *À :* ceph-users 
> *Objet :* Re: [ceph-users] Whole cluster flapping
>
>
>
> Frédéric,
>
>
>
> see if the number of objects is decreasing in the pool with `ceph df
> [detail]`
>
>
>
> Regards,
>
>
>
> Webert Lima
>
> DevOps Engineer at MAV Tecnologia
>
> *Belo Horizonte - Brasil*
>
> *IRC NICK - WebertRLZ*
>
>
>
>
>
> On Tue, Aug 7, 2018 at 5:46 AM CUZA Frédéric  wrote:
>
> It’s been over a week now and the whole cluster keeps flapping, it is
> never the same OSDs that go down.
>
> Is there a way to get the progress of this recovery ? (The pool hat I
> deleted is no longer present (for a while now))
>
> In fact, there is a lot of i/o activity on the server where osds go down.
>
>
>
> Regards,
>
>
>
> *De :* ceph-users  *De la part de*
> Webert de Souza Lima
> *Envoyé :* 31 July 2018 16:25
> *À :* ceph-users 
> *Objet :* Re: [ceph-users] Whole cluster flapping
>
>
>
> The pool deletion might have triggered a lot of IO operations on the disks
> and the process might be too busy to respond to hearbeats, so the mons mark
> them as down due to no response.
>
> Check also the OSD logs to see if they are actually crashing and
> restarting, and disk IO usage (i.e. iostat).
>
>
>
> Regards,
>
>
>
> Webert Lima
>
> DevOps Engineer at MAV Tecnologia
>
> *Belo Horizonte - Brasil*
>
> *IRC NICK - WebertRLZ*
>
>
>
>
>
> On Tue, Jul 31, 2018 at 7:23 AM CUZA Frédéric 
> wrote:
>
> Hi Everyone,
>
>
>
> I just upgrade our cluster to Luminous 12.2.7 and I delete a quite large
> pool that we had (120 TB).
>
> Our cluster is made of 14 Nodes with each composed of 12 OSDs (1 HDD -> 1
> OSD), we have SDD for journal.
>
>
>
> After I deleted the large pool my cluster started to flapping on all OSDs.
>
> Osds are marked down and then marked up as follow :
>
>
>
> 2018-07-31 10:42:51.504319 mon.ceph_monitor01 [INF] osd.97
> 172.29.228.72:6800/95783 boot
>
> 2018-07-31 10:42:55.330993 mon.ceph_monitor01 [WRN] Health check update:
> 5798/5845200 objects misplaced (0.099%) (OBJECT_MISPLACED)
>
> 2018-07-31 10:42:55.331065 mon.ceph_monitor01 [WRN] Health check update:
> Degraded data redundancy: 221365/5845200 objects degraded (3.787%), 98 pgs
> degraded, 317 pgs undersized (PG_DEGRADED)
>
> 2018-07-31 10:42:55.331093 mon.ceph_monitor01 [WRN] Health check update:
> 81 slow requests are blocked > 32 sec (REQUEST_SLOW)
>
> 2018-07-31 10:42:55.548385 mon.ceph_monitor01 [WRN] Health check update:
> Reduced data availability: 13 pgs inactive, 4 pgs peering (PG_AVAILABILITY)
>
> 2018-07-31 10:42:55.610556 mon.ceph_monitor01 [INF] osd.96
> 172.29.228.72:6803/95830 boot
>
> 2018-07-31 10:43:00.331787 mon.ceph_monitor01 [WRN] Health check update: 5
> osds down (OSD_DOWN)
>
> 2018-07-31 10:43:00.331930 mon.ceph_monitor01 [WRN] Health check update:
> 5782/5845401 objects misplaced (0.099%) (OBJECT_MISPLACED)
>
> 2018-07-31 10:43:00.331950 mon.ceph_monitor01 [WRN] Health check update:
> Degraded data redundancy: 167757/5845401 objects degraded (2.870%), 77 pgs
> degraded, 223 pgs undersized (PG_DEGRADED)
>
> 2018-07-31 10:43:00.331966 mon.ceph_monitor01 [WRN] Health check update:
> 76 slow requests are blocked > 32 sec (REQUEST_SLOW)
>
> 2018-07-31 

Re: [ceph-users] [Ceph-deploy] Cluster Name

[Thode Jocelyn]

Hi,

We are still blocked by this problem on our end. Glen did you  or someone else 
figure out something for this ?

Regards
Jocelyn Thode

From: Glen Baars [mailto:g...@onsitecomputers.com.au]
Sent: jeudi, 2 août 2018 05:43
To: Erik McCormick 
Cc: Thode Jocelyn ; Vasu Kulkarni ; 
ceph-users@lists.ceph.com
Subject: RE: [ceph-users] [Ceph-deploy] Cluster Name

Hello Erik,

We are going to use RBD-mirror to replicate the clusters. This seems to need 
separate cluster names.
Kind regards,
Glen Baars

From: Erik McCormick 
mailto:emccorm...@cirrusseven.com>>
Sent: Thursday, 2 August 2018 9:39 AM
To: Glen Baars mailto:g...@onsitecomputers.com.au>>
Cc: Thode Jocelyn mailto:jocelyn.th...@elca.ch>>; Vasu 
Kulkarni mailto:vakul...@redhat.com>>; 
ceph-users@lists.ceph.com
Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name

Don't set a cluster name. It's no longer supported. It really only matters if 
you're running two or more independent clusters on the same boxes. That's 
generally inadvisable anyway.

Cheers,
Erik

On Wed, Aug 1, 2018, 9:17 PM Glen Baars 
mailto:g...@onsitecomputers.com.au>> wrote:
Hello Ceph Users,

Does anyone know how to set the Cluster Name when deploying with Ceph-deploy? I 
have 3 clusters to configure and need to correctly set the name.

Kind regards,
Glen Baars

-Original Message-
From: ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
On Behalf Of Glen Baars
Sent: Monday, 23 July 2018 5:59 PM
To: Thode Jocelyn mailto:jocelyn.th...@elca.ch>>; Vasu 
Kulkarni mailto:vakul...@redhat.com>>
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name

How very timely, I am facing the exact same issue.

Kind regards,
Glen Baars

-Original Message-
From: ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
On Behalf Of Thode Jocelyn
Sent: Monday, 23 July 2018 1:42 PM
To: Vasu Kulkarni mailto:vakul...@redhat.com>>
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name

Hi,

Yes my rbd-mirror is coloctaed with my mon/osd. It only affects nodes where 
they are collocated as they all use the "/etc/sysconfig/ceph" configuration 
file.

Best
Jocelyn Thode

-Original Message-
From: Vasu Kulkarni [mailto:vakul...@redhat.com]
Sent: vendredi, 20 juillet 2018 17:25
To: Thode Jocelyn mailto:jocelyn.th...@elca.ch>>
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name

On Fri, Jul 20, 2018 at 7:29 AM, Thode Jocelyn 
mailto:jocelyn.th...@elca.ch>> wrote:
> Hi,
>
>
>
> I noticed that in commit
> https://github.com/ceph/ceph-deploy/commit/b1c27b85d524f2553af2487a980
> 23b60efe421f3, the ability to specify a cluster name was removed. Is
> there a reason for this removal ?
>
>
>
> Because right now, there are no possibility to create a ceph cluster
> with a different name with ceph-deploy which is a big problem when
> having two clusters replicating with rbd-mirror as we need different names.
>
>
>
> And even when following the doc here:
> https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/h
> tml/block_device_guide/block_device_mirroring#rbd-mirroring-clusters-w
> ith-the-same-name
>
>
>
> This is not sufficient as once we change the CLUSTER variable in the
> sysconfig file, mon,osd, mds etc. all use it and fail to start on a
> reboot as they then try to load data from a path in /var/lib/ceph
> containing the cluster name.

Is you rbd-mirror client also colocated with mon/osd? This needs to be changed 
only on the client side where you are doing mirroring, rest of the nodes are 
not affected?


>
>
>
> Is there a solution to this problem ?
>
>
>
> Best Regards
>
> Jocelyn Thode
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
This e-mail is intended solely for the benefit of the addressee(s) and any 
other named recipient. It is confidential and may contain legally privileged or 
confidential information. If you are not the recipient, any use, distribution, 
disclosure or copying of this e-mail is prohibited. The confidentiality and 
legal privilege attached to this communication is not waived or lost by reason 
of the mistaken transmission or delivery to you. If you have received this 
e-mail in error, please notify us immediately.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
This e-mail is intended solely for the benefit of the addressee(s) and 

Re: [ceph-users] Whole cluster flapping

[CUZA Frédéric]

Thx for the command line, I did take a look too it what I don’t really know 
what to search for, my bad….
All this flapping is due to deep-scrub when it starts on an OSD things start to 
go bad.

I set out all the OSDs that were flapping the most (1 by 1 after rebalancing) 
and it looks better even if some osds keep going down/up with the same message 
in logs :

1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fdabd897700' had timed out 
after 90

(I update it to 90 instead of 15s)

Regards,



De : ceph-users  De la part de Webert de 
Souza Lima
Envoyé : 07 August 2018 16:28
À : ceph-users 
Objet : Re: [ceph-users] Whole cluster flapping

oops, my bad, you're right.

I don't know much you can see but maybe you can dig around performance counters 
and see what's happening on those OSDs, try these:

~# ceph daemonperf osd.XX
~# ceph daemon osd.XX perf dump

change XX to your OSD numbers.

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
Belo Horizonte - Brasil
IRC NICK - WebertRLZ


On Tue, Aug 7, 2018 at 10:47 AM CUZA Frédéric 
mailto:frederic.c...@sib.fr>> wrote:
Pool is already deleted and no longer present in stats.

Regards,

De : ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
De la part de Webert de Souza Lima
Envoyé : 07 August 2018 15:08
À : ceph-users mailto:ceph-users@lists.ceph.com>>
Objet : Re: [ceph-users] Whole cluster flapping

Frédéric,

see if the number of objects is decreasing in the pool with `ceph df [detail]`

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
Belo Horizonte - Brasil
IRC NICK - WebertRLZ


On Tue, Aug 7, 2018 at 5:46 AM CUZA Frédéric 
mailto:frederic.c...@sib.fr>> wrote:
It’s been over a week now and the whole cluster keeps flapping, it is never the 
same OSDs that go down.
Is there a way to get the progress of this recovery ? (The pool hat I deleted 
is no longer present (for a while now))
In fact, there is a lot of i/o activity on the server where osds go down.

Regards,

De : ceph-users 
mailto:ceph-users-boun...@lists.ceph.com>> 
De la part de Webert de Souza Lima
Envoyé : 31 July 2018 16:25
À : ceph-users mailto:ceph-users@lists.ceph.com>>
Objet : Re: [ceph-users] Whole cluster flapping

The pool deletion might have triggered a lot of IO operations on the disks and 
the process might be too busy to respond to hearbeats, so the mons mark them as 
down due to no response.
Check also the OSD logs to see if they are actually crashing and restarting, 
and disk IO usage (i.e. iostat).

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
Belo Horizonte - Brasil
IRC NICK - WebertRLZ


On Tue, Jul 31, 2018 at 7:23 AM CUZA Frédéric 
mailto:frederic.c...@sib.fr>> wrote:
Hi Everyone,

I just upgrade our cluster to Luminous 12.2.7 and I delete a quite large pool 
that we had (120 TB).
Our cluster is made of 14 Nodes with each composed of 12 OSDs (1 HDD -> 1 OSD), 
we have SDD for journal.

After I deleted the large pool my cluster started to flapping on all OSDs.
Osds are marked down and then marked up as follow :

2018-07-31 10:42:51.504319 mon.ceph_monitor01 [INF] osd.97 
172.29.228.72:6800/95783 boot
2018-07-31 10:42:55.330993 mon.ceph_monitor01 [WRN] Health check update: 
5798/5845200 objects misplaced (0.099%) (OBJECT_MISPLACED)
2018-07-31 10:42:55.331065 mon.ceph_monitor01 [WRN] Health check update: 
Degraded data redundancy: 221365/5845200 objects degraded (3.787%), 98 pgs 
degraded, 317 pgs undersized (PG_DEGRADED)
2018-07-31 10:42:55.331093 mon.ceph_monitor01 [WRN] Health check update: 81 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-07-31 10:42:55.548385 mon.ceph_monitor01 [WRN] Health check update: 
Reduced data availability: 13 pgs inactive, 4 pgs peering (PG_AVAILABILITY)
2018-07-31 10:42:55.610556 mon.ceph_monitor01 [INF] osd.96 
172.29.228.72:6803/95830 boot
2018-07-31 10:43:00.331787 mon.ceph_monitor01 [WRN] Health check update: 5 osds 
down (OSD_DOWN)
2018-07-31 10:43:00.331930 mon.ceph_monitor01 [WRN] Health check update: 
5782/5845401 objects misplaced (0.099%) (OBJECT_MISPLACED)
2018-07-31 10:43:00.331950 mon.ceph_monitor01 [WRN] Health check update: 
Degraded data redundancy: 167757/5845401 objects degraded (2.870%), 77 pgs 
degraded, 223 pgs undersized (PG_DEGRADED)
2018-07-31 10:43:00.331966 mon.ceph_monitor01 [WRN] Health check update: 76 
slow requests are blocked > 32 sec (REQUEST_SLOW)
2018-07-31 10:43:01.729891 mon.ceph_monitor01 [WRN] Health check update: 
Reduced data availability: 7 pgs inactive, 6 pgs peering (PG_AVAILABILITY)
2018-07-31 10:43:01.753867 mon.ceph_monitor01 [INF] osd.4 
172.29.228.246:6812/3144542 boot
2018-07-31 10:43:05.332624 mon.ceph_monitor01 [WRN] Health check update: 4 osds 
down (OSD_DOWN)
2018-07-31 10:43:05.332691 mon.ceph_monitor01 [WRN] Health check update: 
5767/5845569 objects misplaced (0.099%) (OBJECT_MISPLACED)
2018-07-31 10:43:05.332718 mon.ceph_monitor01 [WRN] Health 

Re: [ceph-users] CephFS - Mounting a second Ceph file system

[John Spray]

On Tue, Aug 7, 2018 at 11:41 PM Scott Petersen 
wrote:

> We are using kernel 4.15.17 and we keep receiving this error
> mount.ceph: unrecognized mount option "mds_namespace", passing to kernel.
>

That message is harmless -- it just means that the userspace mount.ceph
utility doesn't do anything with the option, and is passing it through.
You're seeing it because of the "-v" flag.

John


>
> Running Luminous 12.2.7 on the client.
>
> It seems like mds_namespace is not a recognized option.
>
> The actual command is as follows:
> mount -t ceph -v -o
> name=admin,secretfile=/root/BlueCEPH.keyring,noatime,mds_namespace=depots_ec
> ceph-mon-r1z1n1,ceph-mon-r1z1n2,ceph-mon-r1z1n4:/ /mnt/pve/BlueCEPHfs/
>
> Scott Petersen - System Administrator
>
> 1901 Butterfield Road • Suite 600 • Downers Grove, IL 60515
> 630-960-8700
> www.medstrat.com
>
>
> On Wed, Nov 29, 2017 at 7:11 AM Daniel Baumann 
> wrote:
>
>> On 11/29/17 00:06, Nigel Williams wrote:
>> > Are their opinions on how stable multiple filesystems per single Ceph
>> > cluster is in practice?
>>
>> we're using a single cephfs in production since february, and switched
>> to three cephfs in september - without any problem so far (running
>> 12.2.1).
>>
>> workload is backend for smb, hpc number crunching, and running generic
>> linux containers on it.
>>
>> Regards,
>> Daniel
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Tons of "cls_rgw.cc:3284: gc_iterate_entries end_key=" records in OSD logs

[Jakub Jaszewski]

Hi All, exactly the same story today, same 8 OSDs and a lot of garbage
collection objects to process

Below is the number of "cls_rgw.cc:3284: gc_iterate_entries end_key="
entries per OSD log file
hostA:
  /var/log/ceph/ceph-osd.58.log
  1826467
hostB:
  /var/log/ceph/ceph-osd.88.log
  2924241
hostC:
  /var/log/ceph/ceph-osd.153.log
  581002
  /var/log/ceph/ceph-osd.164.log
  3278606
hostD:
  /var/log/ceph/ceph-osd.95.log
  1426963
hostE:
  /var/log/ceph/ceph-osd.4.log
  2716914
  /var/log/ceph/ceph-osd.53.log
  943749
hostF:
  /var/log/ceph/ceph-osd.172.log
  4085334


# radosgw-admin gc list --include-all|grep oid |wc -l
302357
#

Can anyone please explain what is going on ?

Thanks!
Jakub

On Tue, Aug 7, 2018 at 3:03 PM Jakub Jaszewski 
wrote:

> Hi,
>
> 8 out of 192 OSDs in our cluster (version 12.2.5) write plenty of records
> like "cls_rgw.cc:3284: gc_iterate_entries end_key=" to the corresponding
> log files, e.g.
>
> 2018-08-07 04:34:06.000585 7fdd8f012700  0 
> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
> end_key=1_01533616446.000580407
> 2018-08-07 04:34:06.001888 7fdd8f012700  0 
> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
> end_key=1_01533616446.001886318
> 2018-08-07 04:34:06.003395 7fdd8f012700  0 
> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
> end_key=1_01533616446.003390299
> 2018-08-07 04:34:06.005205 7fdd8f012700  0 
> /build/ceph-12.2.5/src/cls/rgw/cls_rgw.cc:3284: gc_iterate_entries
> end_key=1_01533616446.005200341
>
> # grep '2018-08-07 04:34:06' /var/log/ceph/ceph-osd.4.log |wc -l
> 712
> #
>
> At the same time there were like 500 000 expired garbage collection
> objects.
>
> Log level of OSD subsystem is set to default 1/5 across all OSDs.
>
> I wonder why only few OSDs record this information and is it something to
> be logged in log level = 1 or maybe higher?
> https://github.com/ceph/ceph/blob/v12.2.5/src/cls/rgw/cls_rgw.cc#L3284
>
> Thanks
> Jakub
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD had suicide timed out

[Brad Hubbard]

Do you see "internal heartbeat not healthy" messages in the log of the
osd that suicides?

On Wed, Aug 8, 2018 at 5:45 PM, Brad Hubbard  wrote:
> What is the load like on the osd host at the time and what does the
> disk utilization look like?
>
> Also, what does the transaction look like from one of the osds that
> sends the "you died" message with debugging osd 20 and ms 1 enabled?
>
> On Wed, Aug 8, 2018 at 5:34 PM, Josef Zelenka
>  wrote:
>> Thank you for your suggestion, tried it,  really seems like the other osds
>> think the osd is dead(if I understand this right), however the networking
>> seems absolutely fine between the nodes(no issues in graphs etc).
>>
>>-13> 2018-08-08 09:13:58.466119 7fe053d41700  1 -- 10.12.3.17:0/706864
>> <== osd.12 10.12.3.17:6807/4624236 81  osd_ping(ping_reply e13452 stamp
>> 2018-08-08 09:13:58.464608) v4  2004+0+0 (687351303 0 0) 0x55731eb73e00
>> con 0x55731e7d4800
>>-12> 2018-08-08 09:13:58.466140 7fe054542700  1 -- 10.12.3.17:0/706864
>> <== osd.11 10.12.3.16:6812/19232 81  osd_ping(ping_reply e13452 stamp
>> 2018-08-08 09:13:58.464608) v4  2004+0+0 (687351303 0 0) 0x55733c391200
>> con 0x55731e7a5800
>>-11> 2018-08-08 09:13:58.466147 7fe053540700  1 -- 10.12.125.3:0/706864
>> <== osd.11 10.12.125.2:6811/19232 82  osd_ping(you_died e13452 stamp
>> 2018-08-08 09:13:58.464608) v4  2004+0+0 (3111562112 0 0) 0x55731eb66800
>> con 0x55731e7a4000
>>-10> 2018-08-08 09:13:58.466164 7fe054542700  1 -- 10.12.3.17:0/706864
>> <== osd.11 10.12.3.16:6812/19232 82  osd_ping(you_died e13452 stamp
>> 2018-08-08 09:13:58.464608) v4  2004+0+0 (3111562112 0 0) 0x55733c391200
>> con 0x55731e7a5800
>> -9> 2018-08-08 09:13:58.466164 7fe053d41700  1 -- 10.12.3.17:0/706864
>> <== osd.12 10.12.3.17:6807/4624236 82  osd_ping(you_died e13452 stamp
>> 2018-08-08 09:13:58.464608) v4  2004+0+0 (3111562112 0 0) 0x55731eb73e00
>> con 0x55731e7d4800
>> -8> 2018-08-08 09:13:58.466176 7fe053540700  1 -- 10.12.3.17:0/706864
>> <== osd.9 10.12.3.16:6813/10016600 81  osd_ping(ping_reply e13452 stamp
>> 2018-08-08 09:13:58.464608) v4  2004+0+0 (687351303 0 0) 0x55731eb66800
>> con 0x55731e732000
>> -7> 2018-08-08 09:13:58.466200 7fe053d41700  1 -- 10.12.3.17:0/706864
>> <== osd.10 10.12.3.16:6810/2017908 81  osd_ping(ping_reply e13452 stamp
>> 2018-08-08 09:13:58.464608) v4  2004+0+0 (687351303 0 0) 0x55731eb73e00
>> con 0x55731e796800
>> -6> 2018-08-08 09:13:58.466208 7fe053540700  1 -- 10.12.3.17:0/706864
>> <== osd.9 10.12.3.16:6813/10016600 82  osd_ping(you_died e13452 stamp
>> 2018-08-08 09:13:58.464608) v4  2004+0+0 (3111562112 0 0) 0x55731eb66800
>> con 0x55731e732000
>> -5> 2018-08-08 09:13:58.466222 7fe053d41700  1 -- 10.12.3.17:0/706864
>> <== osd.10 10.12.3.16:6810/2017908 82  osd_ping(you_died e13452 stamp
>> 2018-08-08 09:13:58.464608) v4  2004+0+0 (3111562112 0 0) 0x55731eb73e00
>> con 0x55731e796800
>> -4> 2018-08-08 09:13:59.748336 7fe040531700  1 -- 10.12.3.17:6802/706864
>> --> 10.12.3.16:6800/1677830 -- mgrreport(unknown.13 +0-0 packed 742
>> osd_metrics=1) v5 -- 0x55731fa4af00 con 0
>> -3> 2018-08-08 09:13:59.748538 7fe040531700  1 -- 10.12.3.17:6802/706864
>> --> 10.12.3.16:6800/1677830 -- pg_stats(64 pgs tid 0 v 0) v1 --
>> 0x55733cbf4c00 con 0
>> -2> 2018-08-08 09:14:00.953804 7fe0525a1700  1 heartbeat_map is_healthy
>> 'OSD::peering_tp thread 0x7fe03f52f700' had timed out after 15
>> -1> 2018-08-08 09:14:00.953857 7fe0525a1700  1 heartbeat_map is_healthy
>> 'OSD::peering_tp thread 0x7fe03f52f700' had suicide timed out after 150
>>  0> 2018-08-08 09:14:00.970742 7fe03f52f700 -1 *** Caught signal
>> (Aborted) **
>>
>>
>> Could it be that the suiciding OSDs are rejecting the ping somehow? I'm
>> quite confused as on what's really going on here, it seems completely random
>> to me.
>>
>>
>> On 08/08/18 01:51, Brad Hubbard wrote:
>>>
>>> Try to work out why the other osds are saying this one is down. Is it
>>> because this osd is too busy to respond or something else.
>>>
>>> debug_ms = 1 will show you some message debugging which may help.
>>>
>>> On Tue, Aug 7, 2018 at 10:34 PM, Josef Zelenka
>>>  wrote:

 To follow up, I did some further digging with debug_osd=20/20 and it
 appears
 as if there's no traffic to the OSD, even though it comes UP for the
 cluster
 (this started happening on another OSD in the cluster today, same stuff):

 -27> 2018-08-07 14:10:55.146531 7f9fce3cd700 10 osd.0 12560
 handle_osd_ping osd.17 10.12.3.17:6811/19661 says i am down in 12566
 -26> 2018-08-07 14:10:55.146542 7f9fcebce700 10 osd.0 12560
 handle_osd_ping osd.12 10.12.125.3:6807/4624236 says i am down in 12566
 -25> 2018-08-07 14:10:55.146551 7f9fcf3cf700 10 osd.0 12560
 handle_osd_ping osd.13 10.12.3.17:6805/186262 says i am down in 12566
 -24> 2018-08-07 14:10:55.146564 7f9fce3cd700 20 osd.0 12559

Re: [ceph-users] OSD had suicide timed out

[Brad Hubbard]

What is the load like on the osd host at the time and what does the
disk utilization look like?

Also, what does the transaction look like from one of the osds that
sends the "you died" message with debugging osd 20 and ms 1 enabled?

On Wed, Aug 8, 2018 at 5:34 PM, Josef Zelenka
 wrote:
> Thank you for your suggestion, tried it,  really seems like the other osds
> think the osd is dead(if I understand this right), however the networking
> seems absolutely fine between the nodes(no issues in graphs etc).
>
>-13> 2018-08-08 09:13:58.466119 7fe053d41700  1 -- 10.12.3.17:0/706864
> <== osd.12 10.12.3.17:6807/4624236 81  osd_ping(ping_reply e13452 stamp
> 2018-08-08 09:13:58.464608) v4  2004+0+0 (687351303 0 0) 0x55731eb73e00
> con 0x55731e7d4800
>-12> 2018-08-08 09:13:58.466140 7fe054542700  1 -- 10.12.3.17:0/706864
> <== osd.11 10.12.3.16:6812/19232 81  osd_ping(ping_reply e13452 stamp
> 2018-08-08 09:13:58.464608) v4  2004+0+0 (687351303 0 0) 0x55733c391200
> con 0x55731e7a5800
>-11> 2018-08-08 09:13:58.466147 7fe053540700  1 -- 10.12.125.3:0/706864
> <== osd.11 10.12.125.2:6811/19232 82  osd_ping(you_died e13452 stamp
> 2018-08-08 09:13:58.464608) v4  2004+0+0 (3111562112 0 0) 0x55731eb66800
> con 0x55731e7a4000
>-10> 2018-08-08 09:13:58.466164 7fe054542700  1 -- 10.12.3.17:0/706864
> <== osd.11 10.12.3.16:6812/19232 82  osd_ping(you_died e13452 stamp
> 2018-08-08 09:13:58.464608) v4  2004+0+0 (3111562112 0 0) 0x55733c391200
> con 0x55731e7a5800
> -9> 2018-08-08 09:13:58.466164 7fe053d41700  1 -- 10.12.3.17:0/706864
> <== osd.12 10.12.3.17:6807/4624236 82  osd_ping(you_died e13452 stamp
> 2018-08-08 09:13:58.464608) v4  2004+0+0 (3111562112 0 0) 0x55731eb73e00
> con 0x55731e7d4800
> -8> 2018-08-08 09:13:58.466176 7fe053540700  1 -- 10.12.3.17:0/706864
> <== osd.9 10.12.3.16:6813/10016600 81  osd_ping(ping_reply e13452 stamp
> 2018-08-08 09:13:58.464608) v4  2004+0+0 (687351303 0 0) 0x55731eb66800
> con 0x55731e732000
> -7> 2018-08-08 09:13:58.466200 7fe053d41700  1 -- 10.12.3.17:0/706864
> <== osd.10 10.12.3.16:6810/2017908 81  osd_ping(ping_reply e13452 stamp
> 2018-08-08 09:13:58.464608) v4  2004+0+0 (687351303 0 0) 0x55731eb73e00
> con 0x55731e796800
> -6> 2018-08-08 09:13:58.466208 7fe053540700  1 -- 10.12.3.17:0/706864
> <== osd.9 10.12.3.16:6813/10016600 82  osd_ping(you_died e13452 stamp
> 2018-08-08 09:13:58.464608) v4  2004+0+0 (3111562112 0 0) 0x55731eb66800
> con 0x55731e732000
> -5> 2018-08-08 09:13:58.466222 7fe053d41700  1 -- 10.12.3.17:0/706864
> <== osd.10 10.12.3.16:6810/2017908 82  osd_ping(you_died e13452 stamp
> 2018-08-08 09:13:58.464608) v4  2004+0+0 (3111562112 0 0) 0x55731eb73e00
> con 0x55731e796800
> -4> 2018-08-08 09:13:59.748336 7fe040531700  1 -- 10.12.3.17:6802/706864
> --> 10.12.3.16:6800/1677830 -- mgrreport(unknown.13 +0-0 packed 742
> osd_metrics=1) v5 -- 0x55731fa4af00 con 0
> -3> 2018-08-08 09:13:59.748538 7fe040531700  1 -- 10.12.3.17:6802/706864
> --> 10.12.3.16:6800/1677830 -- pg_stats(64 pgs tid 0 v 0) v1 --
> 0x55733cbf4c00 con 0
> -2> 2018-08-08 09:14:00.953804 7fe0525a1700  1 heartbeat_map is_healthy
> 'OSD::peering_tp thread 0x7fe03f52f700' had timed out after 15
> -1> 2018-08-08 09:14:00.953857 7fe0525a1700  1 heartbeat_map is_healthy
> 'OSD::peering_tp thread 0x7fe03f52f700' had suicide timed out after 150
>  0> 2018-08-08 09:14:00.970742 7fe03f52f700 -1 *** Caught signal
> (Aborted) **
>
>
> Could it be that the suiciding OSDs are rejecting the ping somehow? I'm
> quite confused as on what's really going on here, it seems completely random
> to me.
>
>
> On 08/08/18 01:51, Brad Hubbard wrote:
>>
>> Try to work out why the other osds are saying this one is down. Is it
>> because this osd is too busy to respond or something else.
>>
>> debug_ms = 1 will show you some message debugging which may help.
>>
>> On Tue, Aug 7, 2018 at 10:34 PM, Josef Zelenka
>>  wrote:
>>>
>>> To follow up, I did some further digging with debug_osd=20/20 and it
>>> appears
>>> as if there's no traffic to the OSD, even though it comes UP for the
>>> cluster
>>> (this started happening on another OSD in the cluster today, same stuff):
>>>
>>> -27> 2018-08-07 14:10:55.146531 7f9fce3cd700 10 osd.0 12560
>>> handle_osd_ping osd.17 10.12.3.17:6811/19661 says i am down in 12566
>>> -26> 2018-08-07 14:10:55.146542 7f9fcebce700 10 osd.0 12560
>>> handle_osd_ping osd.12 10.12.125.3:6807/4624236 says i am down in 12566
>>> -25> 2018-08-07 14:10:55.146551 7f9fcf3cf700 10 osd.0 12560
>>> handle_osd_ping osd.13 10.12.3.17:6805/186262 says i am down in 12566
>>> -24> 2018-08-07 14:10:55.146564 7f9fce3cd700 20 osd.0 12559
>>> share_map_peer 0x56308a9d already has epoch 12566
>>> -23> 2018-08-07 14:10:55.146576 7f9fcebce700 20 osd.0 12559
>>> share_map_peer 0x56308abb9800 already has epoch 12566
>>> -22> 2018-08-07 14:10:55.146590 7f9fcf3cf700 20 

Re: [ceph-users] cephfs kernel client hangs

[Zhenshi Zhou]

Hi, I find an old server which mounted cephfs and has the debug files.
# cat osdc
REQUESTS 0 homeless 0
LINGER REQUESTS
BACKOFFS
# cat monc
have monmap 2 want 3+
have osdmap 3507
have fsmap.user 0
have mdsmap 55 want 56+
fs_cluster_id -1
# cat mdsc
194 mds0getattr  #1036ae3

What does it mean?

Zhenshi Zhou  于2018年8月8日周三 下午1:58写道：

> I restarted the client server so that there's no file in that directory. I
> will take care of it if the client hangs next time.
>
> Thanks
>
> Yan, Zheng  于2018年8月8日周三 上午11:23写道：
>
>> On Wed, Aug 8, 2018 at 11:02 AM Zhenshi Zhou  wrote:
>> >
>> > Hi,
>> > I check all my ceph servers and they are not mount cephfs on each of
>> them(maybe I umount after testing). As a result, the cluster didn't
>> encounter a memory deadlock. Besides, I check the monitoring system and the
>> memory and cpu usage were at common level while the clients hung.
>> > Back to my question, there must be something else cause the client hang.
>> >
>>
>> Check if there are hang requests in
>> /sys/kernel/debug/ceph//{osdc,mdsc},
>>
>> > Zhenshi Zhou  于2018年8月8日周三 上午4:16写道：
>> >>
>> >> Hi, I'm not sure if it just mounts the cephfs without using or doing
>> any operation within the mounted directory would be affected by flushing
>> cache. I mounted cephfs on osd servers only for testing and then left it
>> there. Anyway I will umount it.
>> >>
>> >> Thanks
>> >>
>> >> John Spray 于2018年8月8日 周三03:37写道：
>> >>>
>> >>> On Tue, Aug 7, 2018 at 5:42 PM Reed Dier 
>> wrote:
>> >>> >
>> >>> > This is the first I am hearing about this as well.
>> >>>
>> >>> This is not a Ceph-specific thing -- it can also affect similar
>> >>> systems like Lustre.
>> >>>
>> >>> The classic case is when under some memory pressure, the kernel tries
>> >>> to free memory by flushing the client's page cache, but doing the
>> >>> flush means allocating more memory on the server, making the memory
>> >>> pressure worse, until the whole thing just seizes up.
>> >>>
>> >>> John
>> >>>
>> >>> > Granted, I am using ceph-fuse rather than the kernel client at this
>> point, but that isn’t etched in stone.
>> >>> >
>> >>> > Curious if there is more to share.
>> >>> >
>> >>> > Reed
>> >>> >
>> >>> > On Aug 7, 2018, at 9:47 AM, Webert de Souza Lima <
>> webert.b...@gmail.com> wrote:
>> >>> >
>> >>> >
>> >>> > Yan, Zheng  于2018年8月7日周二 下午7:51写道：
>> >>> >>
>> >>> >> On Tue, Aug 7, 2018 at 7:15 PM Zhenshi Zhou 
>> wrote:
>> >>> >> this can cause memory deadlock. you should avoid doing this
>> >>> >>
>> >>> >> > Yan, Zheng 于2018年8月7日 周二19:12写道：
>> >>> >> >>
>> >>> >> >> did you mount cephfs on the same machines that run ceph-osd?
>> >>> >> >>
>> >>> >
>> >>> >
>> >>> > I didn't know about this. I run this setup in production. :P
>> >>> >
>> >>> > Regards,
>> >>> >
>> >>> > Webert Lima
>> >>> > DevOps Engineer at MAV Tecnologia
>> >>> > Belo Horizonte - Brasil
>> >>> > IRC NICK - WebertRLZ
>> >>> >
>> >>> > ___
>> >>> > ceph-users mailing list
>> >>> > ceph-users@lists.ceph.com
>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>> >
>> >>> >
>> >>> > ___
>> >>> > ceph-users mailing list
>> >>> > ceph-users@lists.ceph.com
>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>> ___
>> >>> ceph-users mailing list
>> >>> ceph-users@lists.ceph.com
>> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] pg count question

[Sébastien VIGNERON]

The formula seems correct for a 100 pg/OSD target.


> Le 8 août 2018 à 04:21, Satish Patel  a écrit :
> 
> Thanks!
> 
> Do you have any comments on Question: 1 ?
> 
> On Tue, Aug 7, 2018 at 10:59 AM, Sébastien VIGNERON
>  wrote:
>> Question 2:
>> 
>> ceph osd pool set-quota  max_objects|max_bytes
>>set object or byte limit on 
>> pool
>> 
>> 
>>> Le 7 août 2018 à 16:50, Satish Patel  a écrit :
>>> 
>>> Folks,
>>> 
>>> I am little confused so just need clarification, I have 14 osd in my
>>> cluster and i want to create two pool  (pool-1 & pool-2) how do i
>>> device pg between two pool with replication 3
>>> 
>>> Question: 1
>>> 
>>> Is this correct formula?
>>> 
>>> 14 * 100 / 3 / 2 =  233  ( power of 2 would be 256)
>>> 
>>> So should i give 256 PG per pool right?
>>> 
>>> pool-1 = 256 pg & pgp
>>> poo-2 = 256 pg & pgp
>>> 
>>> 
>>> Question: 2
>>> 
>>> How do i set limit on pool for example if i want pool-1 can only use
>>> 500GB and pool-2 can use rest of the space?
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD had suicide timed out

[Josef Zelenka]

Thank you for your suggestion, tried it,  really seems like the other 
osds think the osd is dead(if I understand this right), however the 
networking seems absolutely fine between the nodes(no issues in graphs 
etc).


   -13> 2018-08-08 09:13:58.466119 7fe053d41700  1 -- 
10.12.3.17:0/706864 <== osd.12 10.12.3.17:6807/4624236 81  
osd_ping(ping_reply e13452 stamp 2018-08-08 09:13:58.464608) v4  
2004+0+0 (687351303 0 0) 0x55731eb73e00 con 0x55731e7d4800
   -12> 2018-08-08 09:13:58.466140 7fe054542700  1 -- 
10.12.3.17:0/706864 <== osd.11 10.12.3.16:6812/19232 81  
osd_ping(ping_reply e13452 stamp 2018-08-08 09:13:58.464608) v4  
2004+0+0 (687351303 0 0) 0x55733c391200 con 0x55731e7a5800
   -11> 2018-08-08 09:13:58.466147 7fe053540700  1 -- 
10.12.125.3:0/706864 <== osd.11 10.12.125.2:6811/19232 82  
osd_ping(you_died e13452 stamp 2018-08-08 09:13:58.464608) v4  
2004+0+0 (3111562112 0 0) 0x55731eb66800 con 0x55731e7a4000
   -10> 2018-08-08 09:13:58.466164 7fe054542700  1 -- 
10.12.3.17:0/706864 <== osd.11 10.12.3.16:6812/19232 82  
osd_ping(you_died e13452 stamp 2018-08-08 09:13:58.464608) v4  
2004+0+0 (3111562112 0 0) 0x55733c391200 con 0x55731e7a5800
    -9> 2018-08-08 09:13:58.466164 7fe053d41700  1 -- 
10.12.3.17:0/706864 <== osd.12 10.12.3.17:6807/4624236 82  
osd_ping(you_died e13452 stamp 2018-08-08 09:13:58.464608) v4  
2004+0+0 (3111562112 0 0) 0x55731eb73e00 con 0x55731e7d4800
    -8> 2018-08-08 09:13:58.466176 7fe053540700  1 -- 
10.12.3.17:0/706864 <== osd.9 10.12.3.16:6813/10016600 81  
osd_ping(ping_reply e13452 stamp 2018-08-08 09:13:58.464608) v4  
2004+0+0 (687351303 0 0) 0x55731eb66800 con 0x55731e732000
    -7> 2018-08-08 09:13:58.466200 7fe053d41700  1 -- 
10.12.3.17:0/706864 <== osd.10 10.12.3.16:6810/2017908 81  
osd_ping(ping_reply e13452 stamp 2018-08-08 09:13:58.464608) v4  
2004+0+0 (687351303 0 0) 0x55731eb73e00 con 0x55731e796800
    -6> 2018-08-08 09:13:58.466208 7fe053540700  1 -- 
10.12.3.17:0/706864 <== osd.9 10.12.3.16:6813/10016600 82  
osd_ping(you_died e13452 stamp 2018-08-08 09:13:58.464608) v4  
2004+0+0 (3111562112 0 0) 0x55731eb66800 con 0x55731e732000
    -5> 2018-08-08 09:13:58.466222 7fe053d41700  1 -- 
10.12.3.17:0/706864 <== osd.10 10.12.3.16:6810/2017908 82  
osd_ping(you_died e13452 stamp 2018-08-08 09:13:58.464608) v4  
2004+0+0 (3111562112 0 0) 0x55731eb73e00 con 0x55731e796800
    -4> 2018-08-08 09:13:59.748336 7fe040531700  1 -- 
10.12.3.17:6802/706864 --> 10.12.3.16:6800/1677830 -- 
mgrreport(unknown.13 +0-0 packed 742 osd_metrics=1) v5 -- 0x55731fa4af00 
con 0
    -3> 2018-08-08 09:13:59.748538 7fe040531700  1 -- 
10.12.3.17:6802/706864 --> 10.12.3.16:6800/1677830 -- pg_stats(64 pgs 
tid 0 v 0) v1 -- 0x55733cbf4c00 con 0
    -2> 2018-08-08 09:14:00.953804 7fe0525a1700  1 heartbeat_map 
is_healthy 'OSD::peering_tp thread 0x7fe03f52f700' had timed out after 15
    -1> 2018-08-08 09:14:00.953857 7fe0525a1700  1 heartbeat_map 
is_healthy 'OSD::peering_tp thread 0x7fe03f52f700' had suicide timed out 
after 150
 0> 2018-08-08 09:14:00.970742 7fe03f52f700 -1 *** Caught signal 
(Aborted) **



Could it be that the suiciding OSDs are rejecting the ping somehow? I'm 
quite confused as on what's really going on here, it seems completely 
random to me.


On 08/08/18 01:51, Brad Hubbard wrote:

Try to work out why the other osds are saying this one is down. Is it
because this osd is too busy to respond or something else.

debug_ms = 1 will show you some message debugging which may help.

On Tue, Aug 7, 2018 at 10:34 PM, Josef Zelenka
 wrote:

To follow up, I did some further digging with debug_osd=20/20 and it appears
as if there's no traffic to the OSD, even though it comes UP for the cluster
(this started happening on another OSD in the cluster today, same stuff):

-27> 2018-08-07 14:10:55.146531 7f9fce3cd700 10 osd.0 12560
handle_osd_ping osd.17 10.12.3.17:6811/19661 says i am down in 12566
-26> 2018-08-07 14:10:55.146542 7f9fcebce700 10 osd.0 12560
handle_osd_ping osd.12 10.12.125.3:6807/4624236 says i am down in 12566
-25> 2018-08-07 14:10:55.146551 7f9fcf3cf700 10 osd.0 12560
handle_osd_ping osd.13 10.12.3.17:6805/186262 says i am down in 12566
-24> 2018-08-07 14:10:55.146564 7f9fce3cd700 20 osd.0 12559
share_map_peer 0x56308a9d already has epoch 12566
-23> 2018-08-07 14:10:55.146576 7f9fcebce700 20 osd.0 12559
share_map_peer 0x56308abb9800 already has epoch 12566
-22> 2018-08-07 14:10:55.146590 7f9fcf3cf700 20 osd.0 12559
share_map_peer 0x56308abb1000 already has epoch 12566
-21> 2018-08-07 14:10:55.146600 7f9fce3cd700 10 osd.0 12560
handle_osd_ping osd.15 10.12.125.3:6813/49064793 says i am down in 12566
-20> 2018-08-07 14:10:55.146609 7f9fcebce700 10 osd.0 12560
handle_osd_ping osd.16 10.12.3.17:6801/1018363 says i am down in 12566
-19> 2018-08-07 14:10:55.146619 7f9fcf3cf700 10 osd.0 12560
handle_osd_ping osd.11 10.12.3.16:6812/19232 says