[ceph-users] Re: Scrubs Randomly Starting/Stopping

[Ashley Merrick]

So I have done some further digging. Seems similar to this : Bug #54172: ceph version 16.2.7 PG scrubs not progressing - RADOS - Ceph Apart from: 1/ I have restarted all OSD's/forced a re-peer 
and the issue is still there 2/ Setting noscrub stops the scrubs "appearing" Checking a PG seems its just stuck in an endless schedule loop "scrubber": { "active": 
false, "must_scrub": false, "must_deep_scrub": false, "must_repair": false, "need_auto": false, "scrub_reg_stamp": 
"2024-02-21T13:22:37.654028+", "schedule": "queued for deep scrub" On 24 Feb 2024, at 0:45, ash...@amerrick.co.uk wrote: Have just upgraded a cluster from 
17.2.7 to 18.2.1 Everything is working as expected apart from the amount of scrubs & deep scrubs is bouncing all over the place every second. I have the value set to 1 per OSD but currently 
the cluster reckons one minute it’s doing 60+ scrubs, and then second this will drop to 40, then back to 70. If I check the ceph live log’s I can see every second it’s reporting multiple PG’s 
starting either a scrub or deep scrub, it does not look like these are actually running as isn’t having a negative effect on the cluster’s performance. Is this something to be expected off the 
back of the upgrade and should sort it self out? A sample of the logs: 2024-02-24T00:41:20.055401+ osd.54 (osd.54) 3160 : cluster 0 12.9a deep-scrub starts 2024-02-24T00:41:19.658144+ 
osd.41 (osd.41) 4103 : cluster 0 12.cd deep-scrub starts 2024-02-24T00:41:19.823910+ osd.33 (osd.33) 5625 : cluster 0 12.ae deep-scrub starts 2024-02-24T00:41:19.846736+ osd.65 (osd.65) 
3947 : cluster 0 12.53 deep-scrub starts 2024-02-24T00:41:20.007331+ osd.20 (osd.20) 7214 : cluster 0 12.142 scrub starts 2024-02-24T00:41:20.114748+ osd.10 (osd.10) 6538 : cluster 0 
12.2c deep-scrub starts 2024-02-24T00:41:20.247205+ osd.36 (osd.36) 4789 : cluster 0 12.16f deep-scrub starts 2024-02-24T00:41:20.908051+ osd.68 (osd.68) 3869 : cluster 0 12.d7 
deep-scrub starts ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph Upgrade 16.2.5 stuck completing

[Ashley Merrick]

Just upgraded to 16.2.5 all OSD,MDS,MON,MGR,CRASH services updated.

However ceph -s shows repeated line's like:

Updating alertmanager deployment (+1 -1 -> 1) (0s)
[]
Updating node-exporter deployment (+7 -7 -> 7) (0s)
[]
Updating prometheus deployment (+1 -1 -> 1) (0s)
[]
Updating node-exporter deployment (+7 -7 -> 7) (0s)
[]
Updating node-exporter deployment (+7 -7 -> 7) (0s)

ceph orch upgrade status
{
"target_image": null,
"in_progress": false,
"services_complete": [],
"progress": null,
"message": ""
}

If I start a new upgrade via the ceph orch command for a few seconds it shows

ceph orch upgrade status
{
"target_image": 
"docker.io/ceph/ceph@sha256:829ebf54704f2d827de00913b171e5da741aad9b53c1f35ad59251524790eceb",
"in_progress": true,
"services_complete": [
"mgr",
"mds",
"mon",
"osd",
"crash"
],
"progress": "66/76 ceph daemons upgraded",
"message": "Doing first pull of docker.io/ceph/ceph:v16.2.5 image"
}

But then goes back to complete, seems it's stuck completing the last few 
services.

Thanks
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: mon vanished after cephadm upgrade

[Ashley Merrick]

Hello,Is not listed under ceph -s, ceph-s reports no issues on the cluster.Is 
lised under orch ps and dashboard but reports "mon.sn-m01 sn-m01 stopped  
114s ago  4M  -  "Let me know if anything 
else useful you would like before I try remove and redeploy.Thanks
> On Fri May 14 2021 21:44:11 GMT+0800 (Singapore Standard Time), Sebastian 
> Wagner  wrote:
> Hi Ashley,
> 
> is sn-m01 listed in `ceph -s`? Which hosts are listed in `ceph orch ps 
> --daemon-type mon ?
> 
> 
> Otherwise, there are a two helpful commands now:
> 
>   * `cpeh orch daemon rm mon.sn-m01` to remove the mon
>   * `ceph orch daemon start mon.sn-m01` to start it again
> 
> Am 14.05.21 um 14:14 schrieb Ashley Merrick:
>> I had a 3 mon CEPH cluster, after updating from 15.2.x to 16.2.x one of my 
>> mon's is showing as a stopped state in the Ceph Dashboard.And checking the 
>> cephadm logs on the server in question I can see "/usr/bin/docker: Error: No 
>> such object: ceph-30449cba-44e4-11eb-ba64-dda10beff041-mon.sn-m01"There is a 
>> few OSD services running on the same physical server and they all are 
>> starting/running fine via docker.I tried to do a cephadm apply mon to push a 
>> new mon to the same host, but it seems to not do anything, nothing shows in 
>> the same log file on sn-m01Also ceph -s shows full health and no errors and 
>> has no trace of the "failed" mon (not sure if this is expected), only in the 
>> ceph dashboard under services can I see the stopped not running mon.
>>   
>> Sent via MXlogin
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
> 

 
Sent via MXlogin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] mon vanished after cephadm upgrade

[Ashley Merrick]

I had a 3 mon CEPH cluster, after updating from 15.2.x to 16.2.x one of my 
mon's is showing as a stopped state in the Ceph Dashboard.And checking the 
cephadm logs on the server in question I can see "/usr/bin/docker: Error: No 
such object: ceph-30449cba-44e4-11eb-ba64-dda10beff041-mon.sn-m01"There is a 
few OSD services running on the same physical server and they all are 
starting/running fine via docker.I tried to do a cephadm apply mon to push a 
new mon to the same host, but it seems to not do anything, nothing shows in the 
same log file on sn-m01Also ceph -s shows full health and no errors and has no 
trace of the "failed" mon (not sure if this is expected), only in the ceph 
dashboard under services can I see the stopped not running mon.
 
Sent via MXlogin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Failed cephadm Upgrade - ValueError

[Ashley Merrick]

Created BugTicket : https://tracker.ceph.com/issues/50616
> On Mon May 03 2021 21:49:41 GMT+0800 (Singapore Standard Time), Ashley 
> Merrick  wrote:
> Just checked cluster logs and they are full of:cephadm exited with an error 
> code: 1, stderr:Reconfig daemon osd.16 ... Traceback (most recent call last): 
> File 
> "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
>  line 7931, in  main() File 
> "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
>  line 7919, in main r = ctx.func(ctx) File 
> "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
>  line 1717, in defaultimage return func(ctx) File 
> "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
>  line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, 
> daemon_id, File 
> "/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a
 3b697d119482", line 2451, in get_container 
volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 6451, in selinux_enabled return (self.kernel_security['type'] == 
'SELinux') and \ File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 6434, in kernel_security ret = _fetch_apparmor() File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not 
enough values to unpack (expected 2, got 1) Traceback (most recent ca
 ll last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in 
_remote_connection yield (conn, connr) File 
"/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm code, 
'\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with 
an error code: 1, stderr:Reconfig daemon osd.16 ... Traceback (most recent call 
last): File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 7931, in  main() File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 7919, in main r = ctx.func(ctx) File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 1717, in _default_image return func(ctx) File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 lin
 e 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, 
daemon_id, File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, 
daemon_type, daemon_id), File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 6451, in selinux_enabled return (self.kernel_security['type'] == 
'SELinux') and \ File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 6434, in kernel_security ret = _fetch_apparmor() File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484
 bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in 
_fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to 
unpack (expected 2, got 1)being repeated over and over again for each OSD.Again 
listing "ValueError: not enough values to unpack (expected 2, got 1)"
>> On Mon May 03 2021 17:20:59 GMT+0800 (Singapore Standard Time), Ashley 
>> Merrick  wrote:
>> Hello,Wondering if anyone had any feedback 

[ceph-users] Failed cephadm Upgrade - ValueError

[Ashley Merrick]

Just checked cluster logs and they are full of:cephadm exited with an error 
code: 1, stderr:Reconfig daemon osd.16 ... Traceback (most recent call last): 
File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 7931, in  main() File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 7919, in main r = ctx.func(ctx) File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 1717, in defaultimage return func(ctx) File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, 
daemon_id, File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b
 697d119482", line 2451, in get_container 
volume_mounts=get_container_mounts(ctx, fsid, daemon_type, daemon_id), File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 6451, in selinux_enabled return (self.kernel_security['type'] == 
'SELinux') and \ File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 6434, in kernel_security ret = _fetch_apparmor() File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 6415, in _fetch_apparmor item, mode = line.split(' ') ValueError: not 
enough values to unpack (expected 2, got 1) Traceback (most recent call
  last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1172, in 
_remote_connection yield (conn, connr) File 
"/usr/share/ceph/mgr/cephadm/serve.py", line 1087, in _run_cephadm code, 
'\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with 
an error code: 1, stderr:Reconfig daemon osd.16 ... Traceback (most recent call 
last): File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 7931, in  main() File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 7919, in main r = ctx.func(ctx) File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 1717, in _default_image return func(ctx) File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 
 4162, in command_deploy c = get_container(ctx, ctx.fsid, daemon_type, 
daemon_id, File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 2451, in get_container volume_mounts=get_container_mounts(ctx, fsid, 
daemon_type, daemon_id), File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 2292, in get_container_mounts if HostFacts(ctx).selinux_enabled: File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 6451, in selinux_enabled return (self.kernel_security['type'] == 
'SELinux') and \ File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bdc911a9c50d6408adfca696c2faaa65c018d660a3b697d119482",
 line 6434, in kernel_security ret = _fetch_apparmor() File 
"/var/lib/ceph/30449cba-44e4-11eb-ba64-dda10beff041/cephadm.17068a0b484bd
 c911a9c50d6408adfca696c2faaa65c018d660a3b697d119482", line 6415, in 
_fetch_apparmor item, mode = line.split(' ') ValueError: not enough values to 
unpack (expected 2, got 1)being repeated over and over again for each OSD.Again 
listing "ValueError: not enough values to unpack (expected 2, got 1)"
> On Mon May 03 2021 17:20:59 GMT+0800 (Singapore Standard Time), Ashley 
> Merrick  wrote:
> Hello,Wondering if anyone had any feedback on some commands I could try to 
> manually update the current OSD that is down to 16.2.1 so I can at least get 
> around this upgrade bug and back to 100%?If there is any log's or if it seems 
> a new bug and I should create a b

[ceph-users] Failed cephadm Upgrade - ValueError

[Ashley Merrick]

Hello,Wondering if anyone had any feedback on some commands I could try to 
manually update the current OSD that is down to 16.2.1 so I can at least get 
around this upgrade bug and back to 100%?If there is any log's or if it seems a 
new bug and I should create a bugzilla report do let me know.Thanks
> On Fri Apr 30 2021 21:54:30 GMT+0800 (Singapore Standard Time), Ashley 
> Merrick  wrote:
> Hello All,I was running 15.2.8 via cephadm on docker Ubuntu 20.04I just 
> attempted to upgrade to 16.2.1 via the automated method, it successfully 
> upgraded the mon/mgr/mds and some OSD's, however it then failed on an OSD and 
> hasn't been able to pass even after stopping and restarting the upgrade.It 
> reported the following ""message": "Error: UPGRADEREDEPLOYDAEMON: Upgrading 
> daemon osd.35 on host sn-s01 failed.""If I run 'ceph health detail' I get 
> lot's of the following error : "ValueError: not enough values to unpack 
> (expected 2, got 1)" throughout the detail reportUpon googling, it looks like 
> I am hitting something along the lines of https://158.69.68.89/issues/48924 & 
> https://tracker.ceph.com/issues/49522What do I need to do to either get 
> around this bug, or a way I can manually upgrade the remaining ceph OSD's to 
> 16.2.1, currently my cluster is working but the last OSD it failed to upgrade 
> is currently offline (I guess as no image attached to it now as it failed to 
> pull it), and I 
 have a cluster with OSD's from not 15.2.8 and 16.2.1Thanks
>  
> Sent via MXlogin

 
Sent via MXlogin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Failed cephadm Upgrade - ValueError

[Ashley Merrick]

Hello All,I was running 15.2.8 via cephadm on docker Ubuntu 20.04I just 
attempted to upgrade to 16.2.1 via the automated method, it successfully 
upgraded the mon/mgr/mds and some OSD's, however it then failed on an OSD and 
hasn't been able to pass even after stopping and restarting the upgrade.It 
reported the following ""message": "Error: UPGRADEREDEPLOYDAEMON: Upgrading 
daemon osd.35 on host sn-s01 failed.""If I run 'ceph health detail' I get lot's 
of the following error : "ValueError: not enough values to unpack (expected 2, 
got 1)" throughout the detail reportUpon googling, it looks like I am hitting 
something along the lines of https://158.69.68.89/issues/48924 & 
https://tracker.ceph.com/issues/49522What do I need to do to either get around 
this bug, or a way I can manually upgrade the remaining ceph OSD's to 16.2.1, 
currently my cluster is working but the last OSD it failed to upgrade is 
currently offline (I guess as no image attached to it now as it failed to pull 
it), and I ha
 ve a cluster with OSD's from not 15.2.8 and 16.2.1Thanks
 
Sent via MXlogin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Failing OSD RocksDB Corrupt

[Ashley Merrick]

Hello,I had some faulty power cables on some OSD's in one server which caused 
lots of IO issues/disks appearing/disappearing.This has been corrected now, 2 
of the 10 OSD's are working, however 8 are failing to start due to what looks 
to be a corrupt DB.When running a ceph-bluestore-tool fsck I get the following 
output:rocksdb: [db/db_impl_open.cc:516] db.wal/002221.log: dropping 1302 
bytes; Corruption: missing start of fragmented 
record(2)2020-12-22T16:21:52.715+0100 7f7b6a1500c0 4 rocksdb: 
[db/db_impl.cc:389] Shutdown: canceling all background 
work2020-12-22T16:21:52.715+0100 7f7b6a1500c0 4 rocksdb: [db/db_impl.cc:563] 
Shutdown complete2020-12-22T16:21:52.715+0100 7f7b6a1500c0 -1 rocksdb: 
Corruption: missing start of fragmented record(2)2020-12-22T16:21:52.715+0100 
7f7b6a1500c0 -1 
bluestore(/var/lib/ceph/b1db6b36-0c4c-4bce-9cda-18834be0632d/osd.28) opendb 
erroring opening db:Trying to start the OSD leads to:ceph_abort_msg("Bad table 
magic number: expected 9863518390377041911, found 9372993859750765257 in 
db/002442.sst")It looks like the last write to these OSD's never fully 
completed, sadly as I was adding this new node to move from OSD to Host 
redundancy (EC Pool) I have 20% down PG's currently, is there anything I can do 
to remove the last entry in the DB or somehow clean up the rocksDB to get these 
OSD's atleast started? Understand may end up with some corrupted files.Thanks
 
Sent via MXlogin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 15.2.2 Upgrade - Corruption: error in middle of record

[Ashley Merrick]

Hello,Great news can you confirm the exact command you used to inject the value 
so I can replicate you exact steps.I will do that and then leave it a good 
couple of days before trying a reboot to make sure the WAL is completely 
flushed Thanks Ashley  On Sat, 23 May 2020 23:20:45 +0800  
chris.pal...@pobox.com  wrote Status date:

We seem to have success. I followed the steps below. Only one more OSD 
(on node3) failed to restart, showing the same WAL corruption messages. 
After replacing that & backfilling I could then restart it. So we have a 
healthy cluster with restartable OSDs again, with 
bluefs_preextend_wal_files=false until its deemed safe to re-enable it.

Many thanks Igor!

Regards, Chris

On 23/05/2020 11:06, Chris Palmer wrote:
> Hi Ashley
>
> Setting bluefs_preextend_wal_files to false should stop any further 
> corruption of the WAL (subject to the small risk of doing this while 
> the OSD is active). Over time WAL blocks will be recycled and 
> overwritten with new good blocks, so the extent of the corruption may 
> decrease or even eliminate. However you can't tell whether this has 
> happened. But leaving each running for a while may decrease the 
> chances of having to recreate it.
>
> Having tried changing the parameter on one, then another, I've taken 
> the risk of resetting it on all (running) OSDs, and nothing untoward 
> seems to have happened. I have removed and recreated both failed OSDs 
> (both on the node that was rebooted). They are in different crush 
> device classes so I know that they are used by discrete sets of pgs. 
> osd.9 has been recreated, backfilled, and stopped/started without 
> issue. osd,2 has been recreated and is currently backfilling. When 
> that has finished I will restart osd.2 and expect that the restart 
> will not find any corruption.
>
> Following that I will cycle through all other OSDs, stopping and 
> starting each in turn. If one fails to restart, I will replace it, 
> wait until it backfills, then stop/start it.
>
> Do be aware that you can set the parameter globally (for all OSDs) 
> and/or individually. I made sure the global setting was in place 
> before creating new OSDs. (There might be other ways to achieve this 
> on the command line for creating a new one).
>
> Hope that's clear. But once again, please don't take this as advice on 
> what you should do. That should come from the experts!
>
> Regards, Chris
>
> On 23/05/2020 10:03, Ashley Merrick wrote:
>> Hello Chris,
>>
>> Great to hear, few questions.
>>
>> Once you have injected the bluefs_preextend_wal_files to false, are 
>> you just rebuilding the OSD's that failed? Or are you going through 
>> and rebuilding every OSD even the working one's?
>>
>> Or does setting the bluefs_preextend_wal_files value to false and 
>> leaving the OSD running fix the WAL automatically?
>>
>> Thanks
>>
>>
>>  On Sat, 23 May 2020 15:53:42 +0800 *Chris Palmer 
>> * wrote 
>>
>>     Hi Ashley
>>
>>     Igor has done a great job of tracking down the problem, and we
>>     have finally shown evidence of the type of corruption it would
>>     produce in one of my WALs. Our feeling at the moment is that the
>>     problem can be detoured by setting bluefs_preextend_wal_files to
>>     false on affected OSDs while they are running (but see below),
>>     although Igor does note that there is a small risk in doing this.
>>     I've agreed a plan of action based on this route, recreating the
>>     failed OSDs, and then cycling through the others until all are
>>     healthy. I've started this now, and so far it looks promising,
>>     although of course I have to wait for recovery/rebalancing. This
>>     is the fastest route to recovery, although there other options.
>>
>>     I'll post as it progresses. The good news seems to be that there
>>     shouldn't be any actual data corruption or loss, providing that
>>     this can be done before OSDs are taken down (other than as part of
>>     this process). My understanding is that there will some degree of
>>     performance penalty until the root cause is fixed in the next
>>     release and preextending can be turned back on. However it does
>>     seem like I can get back to a stable/safe position without waiting
>>     for a software release.
>>
>>     I'm just working through this at the moment though, so please
>>     don't take the above as any form of recommendation. It is
>>     important not to try to restart OSDs though in the meantime. I'm
>>     sure Igor will publish some more expert recommendatio

[ceph-users] Re: 15.2.2 Upgrade - Corruption: error in middle of record

[Ashley Merrick]

Hello Chris,



Great to hear, few questions.



Once you have injected the bluefs_preextend_wal_files to false, are you just 
rebuilding the OSD's that failed? Or are you going through and rebuilding every 
OSD even the working one's?



Or does setting the bluefs_preextend_wal_files value to false and leaving the 
OSD running fix the WAL automatically?



Thanks





 On Sat, 23 May 2020 15:53:42 +0800 Chris Palmer  
wrote 



Hi Ashley

 

 Igor has done a great job of tracking down the problem, and we have
finally shown evidence of the type of corruption it would produce in
one of my WALs. Our feeling at the moment is that the problem can be
detoured by setting bluefs_preextend_wal_files to false on affected
OSDs while they are running (but see below), although Igor does note
that there is a small risk in doing this. I've agreed a plan of
action based on this route, recreating the failed OSDs, and then
cycling through the others until all are healthy. I've started this
now, and so far it looks promising, although of course I have to
wait for recovery/rebalancing. This is the fastest route to
recovery, although there other options.

 

 I'll post as it progresses. The good news seems to be that there
shouldn't be any actual data corruption or loss, providing that this
can be done before OSDs are taken down (other than as part of this
process). My understanding is that there will some degree of
performance penalty until the root cause is fixed in the next
release and preextending can be turned back on. However it does seem
like I can get back to a stable/safe position without waiting for a
software release.

 

 I'm just working through this at the moment though, so please don't
take the above as any form of recommendation. It is important not to
try to restart OSDs though in the meantime. I'm sure Igor will
publish some more expert recommendations in due course...

 

 Regards, Chris

 

 

On 23/05/2020 06:54, Ashley Merrick
  wrote:






Thanks Igor,



Do you have any idea on a e.t.a or plan for people that are
  running 15.2.2 to be able to patch / fix the issue.



I had a read of the ticket and seems the corruption is
  happening but the WAL is not read till OSD restart, so I
  imagine we will need some form of fix / patch we can apply to
  a running OSD before we then restart the OSD, as a normal OSD
  upgrade will require the OSD to restart to apply the code
  resulting in a corrupt OSD.



Thanks





 On Sat, 23 May 2020 00:12:59 +0800 Igor Fedotov mailto:ifedo...@suse.de 
wrote 



Status update: 
 
 Finally we have the first patch to fix the issue in
  master: 
 https://github.com/ceph/ceph/pull/35201 
 
 And ticket has been updated with root cause 
 analysis:https://tracker.ceph.com/issues/45613On 5/21/2020 2:07 PM, Igor 
 Fedotov wrote: 
 
 @Chris - unfortunately it looks like the corruption is
  permanent since  
 valid WAL data are presumably overwritten with another
  stuff. Hence I 
 don't know any way to recover - perhaps you can try
  cutting 
 
 WAL file off which will allow OSD to start. With some
  latest ops lost. 
 Once can use exported BlueFS as a drop in replacement for
  regular DB 
 volume but I'm not aware of details. 
 
 And the above are just speculations, can't say for sure if
  it helps... 
 
 I can't explain why WAL doesn't have zero block in your
  case though. 
 Little chances this is a different issue. Just in case -
  could you 
 please search for 32K zero blocks over the whole file? And
  the same for 
 another OSD? 
 
 
 Thanks, 
 
 Igor 
 
 > Short update on the issue: 
 > 
 > Finally we're able to reproduce the issue in master
  (not octopus), 
 > investigating further.. 
 > 
 > @Chris - to make sure you're facing the same issue
  could you please 
 > check the content of the broken file. To do so: 
 > 
 > 1) run "ceph-bluestore-tool --path
   --our-dir  dir> --command bluefs-export 
 > 
 > This will export bluefs files to  
 > 
 > 2) Check the content for file db.wal/002040.log at
  offset 0x47 
 > 
 > This will presumably contain 32K of zero bytes. Is
  this the case? 
 > 
 > 
 > No hurry as I'm just making sure symptoms in Octopus
  are the same... 
 > 
 > 
 > Thanks, 
 > 
 > Igor 
 > 
 > On 5/20/2020 5:24 PM, Igor Fedotov wrote: 
 >> Chris, 
 >> 
 >> got them, thanks! 
 >> 
 >> Investigating 
 >> 
 >> 
 >> Thanks, 
 >> 
 >> Igor 
 >> 
 >> On 5/20/2020 5:23 PM, Chris Palmer wro

[ceph-users] Re: 15.2.2 Upgrade - Corruption: error in middle of record

[Ashley Merrick]

Thanks Igor,



Do you have any idea on a e.t.a or plan for people that are running 15.2.2 to 
be able to patch / fix the issue.



I had a read of the ticket and seems the corruption is happening but the WAL is 
not read till OSD restart, so I imagine we will need some form of fix / patch 
we can apply to a running OSD before we then restart the OSD, as a normal OSD 
upgrade will require the OSD to restart to apply the code resulting in a 
corrupt OSD.



Thanks



 On Sat, 23 May 2020 00:12:59 +0800 Igor Fedotov  wrote 



Status update: 
 
Finally we have the first patch to fix the issue in master: 
https://github.com/ceph/ceph/pull/35201 
 
And ticket has been updated with root cause 
analysis:https://tracker.ceph.com/issues/45613On 5/21/2020 2:07 PM, Igor 
Fedotov wrote: 
 
@Chris - unfortunately it looks like the corruption is permanent since  
valid WAL data are presumably overwritten with another stuff. Hence I 
don't know any way to recover - perhaps you can try cutting 
 
WAL file off which will allow OSD to start. With some latest ops lost. 
Once can use exported BlueFS as a drop in replacement for regular DB 
volume but I'm not aware of details. 
 
And the above are just speculations, can't say for sure if it helps... 
 
I can't explain why WAL doesn't have zero block in your case though. 
Little chances this is a different issue. Just in case - could you 
please search for 32K zero blocks over the whole file? And the same for 
another OSD? 
 
 
Thanks, 
 
Igor 
 
> Short update on the issue: 
> 
> Finally we're able to reproduce the issue in master (not octopus), 
> investigating further.. 
> 
> @Chris - to make sure you're facing the same issue could you please 
> check the content of the broken file. To do so: 
> 
> 1) run "ceph-bluestore-tool --path  --our-dir  dir> --command bluefs-export 
> 
> This will export bluefs files to  
> 
> 2) Check the content for file db.wal/002040.log at offset 0x47 
> 
> This will presumably contain 32K of zero bytes. Is this the case? 
> 
> 
> No hurry as I'm just making sure symptoms in Octopus are the same... 
> 
> 
> Thanks, 
> 
> Igor 
> 
> On 5/20/2020 5:24 PM, Igor Fedotov wrote: 
>> Chris, 
>> 
>> got them, thanks! 
>> 
>> Investigating 
>> 
>> 
>> Thanks, 
>> 
>> Igor 
>> 
>> On 5/20/2020 5:23 PM, Chris Palmer wrote: 
>>> Hi Igor 
>>> I've sent you these directly as they're a bit chunky. Let me know if 
>>> you haven't got them. 
>>> Thx, Chris 
>>> 
>>> On 20/05/2020 14:43, Igor Fedotov wrote: 
 Hi Cris, 
 
 could you please share the full log prior to the first failure? 
 
 Also if possible please set debug-bluestore/debug bluefs to 20 and 
 collect another one for failed OSD startup. 
 
 
 Thanks, 
 
 Igor 
 
 
 On 5/20/2020 4:39 PM, Chris Palmer wrote: 
> I'm getting similar errors after rebooting a node. Cluster was 
> upgraded 15.2.1 -> 15.2.2 yesterday. No problems after rebooting 
> during upgrade. 
> 
> On the node I just rebooted, 2/4 OSDs won't restart. Similar logs 
> from both. Logs from one below. 
> Neither OSDs have compression enabled, although there is a 
> compression-related error in the log. 
> Both are replicated x3. One has data on HDD & separate WAL/DB on 
> NVMe partition, the other is everything on NVMe partition only. 
> 
> Feeling kinda nervous here - advice welcomed!! 
> 
> Thx, Chris 
> 
> 
> 
> 2020-05-20T13:14:00.837+0100 7f2e0d273700  3 rocksdb: 
> [table/block_based_table_reader.cc:1117] Encountered error while 
> reading data from compression dictionary block Corruption: block 
> checksum mismatch: expected 0, got 3423870535  in db/000304.sst 
> offset 18446744073709551615 size 18446744073709551615 
> 2020-05-20T13:14:00.841+0100 7f2e1957ee00  4 rocksdb: 
> [db/version_set.cc:3757] Recovered from manifest 
> file:db/MANIFEST-000312 succeeded,manifest_file_number is 312, 
> next_file_number is 314, last_sequence is 22320582, log_number is 
> 309,prev_log_number is 0,max_column_family is 
> 0,min_log_number_to_keep is 0 
> 
> 2020-05-20T13:14:00.841+0100 7f2e1957ee00  4 rocksdb: 
> [db/version_set.cc:3766] Column family [default] (ID 0), log 
> number is 309 
> 
> 2020-05-20T13:14:00.841+0100 7f2e1957ee00  4 rocksdb: EVENT_LOG_v1 
> {"time_micros": 1589976840843199, "job": 1, "event": 
> "recovery_started", "log_files": [313]} 
> 2020-05-20T13:14:00.841+0100 7f2e1957ee00  4 rocksdb: 
> [db/db_impl_open.cc:583] Recovering log #313 mode 0 
> 2020-05-20T13:14:00.937+0100 7f2e1957ee00  3 rocksdb: 
> [db/db_impl_open.cc:518] db.wal/000313.log: dropping 9044 bytes; 
> Corruption: error in middle of record 
> 2020-05-20T13:14:00.937+0100 7f2e1957ee00  3 rocksdb: 
> [db/db_impl_open.cc:518] db.wal/000313.log: dropping 86 bytes; 
> Corruption: missing start of fragmented record(2) 
>

[ceph-users] Re: ceph orch upgrade stuck at the beginning.

[Ashley Merrick]

Hello,Yes I did but wasn't able to suggest anything further to get around it, 
however:1/ There is currently an issue with 15.2.2 so I would advise holding 
off any upgrade2/ Another mail list user replied to one of your older emails in 
the thread asking for some manager logs not sure if you have seen this.Thanks 
 On Fri, 22 May 2020 01:21:26 +0800  gen...@gencgiyen.com  wrote Hi 
Ashley,Have you seen my previous reply? If so, and no solution then does anyone 
has any idea how can this be done with 2 node?Thanks,Gencer.

On 20.05.2020 16:33:53, Gencer W. Genç 
 wrote:





This is 2 node setup. I have no third 
node :(

I am planning to add more in the future 
but currently 2 nodes only.At the moment, is there a --force command for such 
usage?
On 20.05.2020 16:32:15, Ashley Merrick 
 wrote:Correct, however it will need to stop one to 
do the upgrade leaving you with only one working MON (this is what I would 
suggest the error means seeing i had the same thing when I only had a single 
MGR), normally is suggested to have 3 MONs due to quorum.Do you not have a node 
you can run a mon for the few minutes to complete the upgrade? On Wed, 20 
May 2020 21:28:19 +0800 Gencer W. Genç  wrote I have 
2 mons and 2 mgrs.  cluster:    id:     7d308992-8899-11ea-8537-7d489fa7c193    
health: HEALTH_OK  services:    mon: 2 daemons, quorum 
vx-rg23-rk65-u43-130,vx-rg23-rk65-u43-130-1 (age 91s)    mgr: 
vx-rg23-rk65-u43-130.arnvag(active, since 28m), standbys: 
vx-rg23-rk65-u43-130-1.pxmyie    mds: cephfs:1 
{0=cephfs.vx-rg23-rk65-u43-130.kzjznt=up:active} 1 up:standby    osd: 24 osds: 
24 up (since 69m), 24 in (since 3w)  task status:    scrub status:        
mds.cephfs.vx-rg23-rk65-u43-130.kzjznt: idle  data:    pools:   4 pools, 97 pgs 
   objects: 1.38k objects, 4.8 GiB    usage:   35 GiB used, 87 TiB / 87 TiB 
avail    pgs:     97 active+clean  io:    client:   5.3 KiB/s wr, 0 op/s rd, 0 
op/s wr  progress:    Upgrade to docker.io/ceph/ceph:v15.2.2 (33s)      
[=...] (remaining: 9m)Isn't both mons already up? I 
have no way to add third mon btw.Thnaks,Gencer.On 20.05.2020 16:21:03, Ashley 
Merrick  wrote:Yes, I think it's because your only 
running two mons, so the script is halting at a check to stop you being in the 
position of just one running (no backup).I had the same issue with a single MGR 
instance and had to add a second to allow to upgrade to continue, can you bring 
up an extra MON?Thanks On Wed, 20 May 2020 21:18:09 +0800 Gencer W. Genç 
 wrote Hi Ashley,I see this:[INF] Upgrade: Target is 
docker.io/ceph/ceph:v15.2.2 with id 
4569944bbW86c3f9b5286057a558a3f852156079f759c9734e54d4f64092be9fa[INF] Upgrade: 
It is NOT safe to stop mon.vx-rg23-rk65-u43-130Does this meaning anything to 
you?I've also attached full log. See especially after line #49. I stopped and 
restart upgrade there.Thanks,Gencer.On 20.05.2020 16:13:00, Ashley Merrick 
 wrote:ceph config set mgr 
mgr/cephadm/log_to_cluster_level debugceph -W cephadm --watch-debugSee if you 
see anything that stands out as an issue with the update, seems it has 
completed only the two MGR instancesIf not:ceph orch upgrade stopceph orch 
upgrade start --ceph-version 15.2.2and monitor the watch-debug logMake sure at 
the end you run:ceph config set mgr mgr/cephadm/log_to_cluster_level info 
On Wed, 20 May 2020 21:07:43 +0800 Gencer W. Genç  wrote 
Ah yes,{    "mon": {        "ceph version 15.2.1 
(9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 2    },    "mgr": 
{        "ceph version 15.2.2 (0c857e985a29d90501a285f242ea9c008df49eb8) 
octopus (stable)": 2    },    "osd": {        "ceph version 15.2.1 
(9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 24    },    
"mds": {        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) 
octopus (stable)": 2    },    "overall": {        "ceph version 15.2.1 
(9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable)": 28,        "ceph 
version 15.2.2 (0c857e985a29d90501a285f242ea9c008df49eb8) octopus (stable)": 2  
  }}How can i fix this?Gencer.On 20.05.2020 16:04:33, Ashley Merrick 
 wrote:Does:ceph versionsshow any services yet 
running on 15.2.2? On Wed, 20 May 2020 21:01:12 +0800 Gencer W. Genç 
 wrote Hi Ashley,$ ceph orch upgrade status {    
"target_image": "docker.io/ceph/ceph:v15.2.2",    "in_progress": true,    
"services_complete

[ceph-users] Re: 15.2.2 Upgrade - Corruption: error in middle of record

[Ashley Merrick]

Hey Igor,



The OSDs only back two metadata pools, so only hold a couple of MB of data 
(hence they was easy and quick to rebuild), there actually NVME LVM devices 
passed through QEMU into a VM (hence only 10GB and showing as rotational)



I have large 10TB disks that back the EC(RBD/FS) them self, but I don't dare 
touch them right now.



Thanks



 On Wed, 20 May 2020 21:34:14 +0800 Igor Fedotov  wrote 



Thanks!

So for now I can see the following similarities between you case
  and the ticket:

1) Single main spinner as an OSD backing device.

2) Corruption happens to RocksDB WAL file

3) OSD has user data compression enabled.



And one more question. Fro the following line:

May 20 06:05:14 sn-m03 bash[86466]: debug
  2020-05-20T06:05:14.466+ 7f60da5a5ec0  1 bdev(0x558e8fce
  /var/lib/ceph/osd/ceph-26/block) open size 10733223936
  (0x27fc0, 10 GiB) block_size 4096 (4 KiB) rotational discard
  supported



I can see that your main device is pretty small - 10GiB only?
  Which looks rather small for any real usage. Is this a correct
  size? 

Surprisingly failed QA run had 9.5 GiB volume too. Don't know if
  this has any meaning or just a coincidence though



Thanks,

Igor



On 5/20/2020 4:01 PM, Ashley Merrick
  wrote:




I attached the log but was too big and got moderated.



Here is it in a paste bin : https://pastebin.pl/view/69b2beb9



I have cut the log to start from the point of the original
  upgrade.



Thanks



 On Wed, 20 May 2020 20:55:51 +0800 Igor Fedotov mailto:ifedo...@suse.de 
wrote 


Dan, thanks for the info. Good to know. 
 
 Failed QA run in the ticket uses snappy though. 
 
 And in fact any stuff writing to process memory can 
  introduce data 
 corruption in the similar manner. 
 
 So will keep that in mind but IMO relation to compression
  is still not 
 evident... 
 
 
 Kind regards, 
 
 Igor 
 
 
 On 5/20/2020 3:32 PM, Dan van der Ster wrote: 
 > lz4 ? It's not obviously related, but I've seen it
  involved in really 
 > non-obvious ways: https://tracker.ceph.com/issues/39525 
 > 
 > -- dan 
 > 
 > On Wed, May 20, 2020 at 2:27 PM Ashley Merrick 
 > <mailto:singap...@amerrick.co.uk>
  wrote: 
 >> Thanks, fyi the OSD's that went down back two
  pools, an Erasure code Meta (RBD) and cephFS Meta. The
  cephFS Pool does have compresison enabled ( I noticed it
  mentioned in the ceph tracker) 
 >> 
 >> 
 >> 
 >> Thanks 
 >> 
 >> 
 >> 
 >> 
 >> 
 >>  On Wed, 20 May 2020 20:17:33 +0800 Igor
  Fedotov <mailto:ifedo...@suse.de>
  wrote  
 >> 
 >> 
 >> 
 >> Hi Ashley, 
 >> 
 >> looks like this is a regression. Neha observed
  similar error(s) during 
 >> here QA run, see https://tracker.ceph.com/issues/45613 
 >> 
 >> 
 >> Please preserve broken OSDs for a while if
  possible, likely I'll come 
 >> back to you for more information to troubleshoot. 
 >> 
 >> 
 >> Thanks, 
 >> 
 >> Igor 
 >> 
 >> On 5/20/2020 1:26 PM, Ashley Merrick wrote: 
 >> 
 >>> So reading online it looked a dead end error,
  so I recreated the 3 OSD's on that node and now working
  fine after a reboot. 
 >>> 
 >>> 
 >>> 
 >>> However I restarted the next server with 3
  OSD's and one of them is now facing the same issue. 
 >>> 
 >>> 
 >>> 
 >>> Let me know if you need any more logs. 
 >>> 
 >>> 
 >>> 
 >>> Thanks 
 >>> 
 >>> 
 >>> 
 >>>  On Wed, 20 May 2020 17:02:31 +0800
  Ashley Merrick <mailto:mailto:singap...@amerrick.co.uk>
  wrote  
 >>> 
 >>> 
 >>> I just upgraded a cephadm cluster from 15.2.1
  to 15.2.2. 
 >>> 
 >>> 
 >>> 
 >>> Everything went fine on the upgrade, however
  after restarting one node that has 3 OSD's for ecmeta two
  of the 3 ODS's now wont boot with the following error: 
 >>> 
 >>> 
 >>> 
 >>> May 20 08:29:42 sn-m01 bash[6833]: debug
  2020-05-20T08:29:42.598+ 7fbcc46f7ec0 4 rocksdb:
  [db/version_set.cc:3757] Recovered from manifest  
succeeded,manifest_file_number is
  2768, next_file_number is 2775, last_sequence is
  188026749, log_number is 2767,prev_log_number is
  0,max_column_family is 0,min_log_number_to_keep is 0 
 >>> 
 >>> May 20 08:29:42 sn-m01 bash[6833]: deb

[ceph-users] Re: ceph orch upgrade stuck at the beginning.

[Ashley Merrick]

Correct, however it will need to stop one to do the upgrade leaving you with 
only one working MON (this is what I would suggest the error means seeing i had 
the same thing when I only had a single MGR), normally is suggested to have 3 
MONs due to quorum.



Do you not have a node you can run a mon for the few minutes to complete the 
upgrade?





 On Wed, 20 May 2020 21:28:19 +0800 Gencer W. Genç  
wrote 



I have 2 mons and 2 mgrs.



  cluster:

    id:     7d308992-8899-11ea-8537-7d489fa7c193

    health: HEALTH_OK



  services:

    mon: 2 daemons, quorum vx-rg23-rk65-u43-130,vx-rg23-rk65-u43-130-1 (age 91s)

    mgr: vx-rg23-rk65-u43-130.arnvag(active, since 28m), standbys: 
vx-rg23-rk65-u43-130-1.pxmyie

    mds: cephfs:1 {0=cephfs.vx-rg23-rk65-u43-130.kzjznt=up:active} 1 up:standby

    osd: 24 osds: 24 up (since 69m), 24 in (since 3w)



  task status:

    scrub status:

        mds.cephfs.vx-rg23-rk65-u43-130.kzjznt: idle



  data:

    pools:   4 pools, 97 pgs

    objects: 1.38k objects, 4.8 GiB

    usage:   35 GiB used, 87 TiB / 87 TiB avail

    pgs:     97 active+clean



  io:

    client:   5.3 KiB/s wr, 0 op/s rd, 0 op/s wr



  progress:

    Upgrade to docker.io/ceph/ceph:v15.2.2 (33s)

      [=...] (remaining: 9m)




Isn't both mons already up? I have no way to add third mon btw.



Thnaks,

Gencer.



On 20.05.2020 16:21:03, Ashley Merrick <mailto:singap...@amerrick.co.uk> wrote:

Yes, I think it's because your only running two mons, so the script is halting 
at a check to stop you being in the position of just one running (no backup).



I had the same issue with a single MGR instance and had to add a second to 
allow to upgrade to continue, can you bring up an extra MON?



Thanks

 On Wed, 20 May 2020 21:18:09 +0800 Gencer W. Genç 
<mailto:gen...@gencgiyen.com> wrote 



Hi Ashley,





I see this:

[INF] Upgrade: Target is docker.io/ceph/ceph:v15.2.2 with id 
4569944bbW86c3f9b5286057a558a3f852156079f759c9734e54d4f64092be9fa


[INF] Upgrade: It is NOT safe to stop mon.vx-rg23-rk65-u43-130



Does this meaning anything to you?



I've also attached full log. See especially after line #49. I stopped and 
restart upgrade there.



Thanks,

Gencer.

On 20.05.2020 16:13:00, Ashley Merrick <mailto:singap...@amerrick.co.uk> wrote:

ceph config set mgr mgr/cephadm/log_to_cluster_level debug

ceph -W cephadm --watch-debug



See if you see anything that stands out as an issue with the update, seems it 
has completed only the two MGR instances



If not:



ceph orch upgrade stop

ceph orch upgrade start --ceph-version 15.2.2



and monitor the watch-debug log



Make sure at the end you run:



ceph config set mgr mgr/cephadm/log_to_cluster_level info





 On Wed, 20 May 2020 21:07:43 +0800 Gencer W. Genç 
<mailto:gen...@gencgiyen.com> wrote 



Ah yes,





{

    "mon": {

        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus 
(stable)": 2

    },

    "mgr": {

        "ceph version 15.2.2 (0c857e985a29d90501a285f242ea9c008df49eb8) octopus 
(stable)": 2

    },

    "osd": {

        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus 
(stable)": 24

    },

    "mds": {

        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus 
(stable)": 2

    },

    "overall": {

        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus 
(stable)": 28,

        "ceph version 15.2.2 (0c857e985a29d90501a285f242ea9c008df49eb8) octopus 
(stable)": 2

    }

}




How can i fix this?



Gencer.

On 20.05.2020 16:04:33, Ashley Merrick <mailto:singap...@amerrick.co.uk> wrote:

Does:



ceph versions



show any services yet running on 15.2.2?





 On Wed, 20 May 2020 21:01:12 +0800 Gencer W. Genç 
<mailto:gen...@gencgiyen.com> wrote 



Hi Ashley,

$ ceph orch upgrade status 




{

    "target_image": "docker.io/ceph/ceph:v15.2.2",

    "in_progress": true,

    "services_complete": [],

    "message": ""

}




Thanks,

Gencer.



On 20.05.2020 15:58:34, Ashley Merrick <mailto:singap...@amerrick.co.uk> wrote:

What does 

ceph orch upgrade status 

show?





 On Wed, 20 May 2020 20:52:39 +0800 Gencer W. Genç 
<mailto:gen...@gencgiyen.com> wrote 



Hi, 
 
I've 15.2.1 installed on all machines. On primary machine I executed ceph 
upgrade command: 
 
$ ceph orch upgrade start --ceph-version 15.2.2 
 
 
When I check ceph -s I see this: 
 
  progress: 
    Upgrade to docker.io/ceph/ceph:v15.2.2 (30m) 
      [=...] (remaining: 8h) 
 
It says 8 hours. It is already ran for 3 hours. No upgrade processed. It get 
stuck at this point. 
 
Is there any way to know why this has stuck? 
 
Th

[ceph-users] Re: ceph orch upgrade stuck at the beginning.

[Ashley Merrick]

Yes, I think it's because your only running two mons, so the script is halting 
at a check to stop you being in the position of just one running (no backup).



I had the same issue with a single MGR instance and had to add a second to 
allow to upgrade to continue, can you bring up an extra MON?


Thanks
 On Wed, 20 May 2020 21:18:09 +0800 Gencer W. Genç  
wrote 


Hi Ashley,



I see this:

[INF] Upgrade: Target is docker.io/ceph/ceph:v15.2.2 with id 
4569944bbW86c3f9b5286057a558a3f852156079f759c9734e54d4f64092be9fa


[INF] Upgrade: It is NOT safe to stop mon.vx-rg23-rk65-u43-130



Does this meaning anything to you?



I've also attached full log. See especially after line #49. I stopped and 
restart upgrade there.



Thanks,

Gencer.

On 20.05.2020 16:13:00, Ashley Merrick <mailto:singap...@amerrick.co.uk> wrote:

ceph config set mgr mgr/cephadm/log_to_cluster_level debug

ceph -W cephadm --watch-debug



See if you see anything that stands out as an issue with the update, seems it 
has completed only the two MGR instances



If not:



ceph orch upgrade stop

ceph orch upgrade start --ceph-version 15.2.2



and monitor the watch-debug log



Make sure at the end you run:



ceph config set mgr mgr/cephadm/log_to_cluster_level info



 On Wed, 20 May 2020 21:07:43 +0800 Gencer W. Genç 
<mailto:gen...@gencgiyen.com> wrote 


Ah yes,



{

    "mon": {

        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus 
(stable)": 2

    },

    "mgr": {

        "ceph version 15.2.2 (0c857e985a29d90501a285f242ea9c008df49eb8) octopus 
(stable)": 2

    },

    "osd": {

        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus 
(stable)": 24

    },

    "mds": {

        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus 
(stable)": 2

    },

    "overall": {

        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus 
(stable)": 28,

        "ceph version 15.2.2 (0c857e985a29d90501a285f242ea9c008df49eb8) octopus 
(stable)": 2

    }

}




How can i fix this?



Gencer.

On 20.05.2020 16:04:33, Ashley Merrick <mailto:singap...@amerrick.co.uk> wrote:

Does:



ceph versions



show any services yet running on 15.2.2?



 On Wed, 20 May 2020 21:01:12 +0800 Gencer W. Genç 
<mailto:gen...@gencgiyen.com> wrote 


Hi Ashley,$ ceph orch upgrade status 




{

    "target_image": "docker.io/ceph/ceph:v15.2.2",

    "in_progress": true,

    "services_complete": [],

    "message": ""

}




Thanks,

Gencer.



On 20.05.2020 15:58:34, Ashley Merrick <mailto:singap...@amerrick.co.uk> wrote:

What does 

ceph orch upgrade status 

show?



 On Wed, 20 May 2020 20:52:39 +0800 Gencer W. Genç 
<mailto:gen...@gencgiyen.com> wrote 


Hi, 
 
I've 15.2.1 installed on all machines. On primary machine I executed ceph 
upgrade command: 
 
$ ceph orch upgrade start --ceph-version 15.2.2 
 
 
When I check ceph -s I see this: 
 
  progress: 
    Upgrade to docker.io/ceph/ceph:v15.2.2 (30m) 
      [=...] (remaining: 8h) 
 
It says 8 hours. It is already ran for 3 hours. No upgrade processed. It get 
stuck at this point. 
 
Is there any way to know why this has stuck? 
 
Thanks, 
Gencer.
___
ceph-users mailing list -- mailto:ceph-users@ceph.io
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph orch upgrade stuck at the beginning.

[Ashley Merrick]

ceph config set mgr mgr/cephadm/log_to_cluster_level debug

ceph -W cephadm --watch-debug



See if you see anything that stands out as an issue with the update, seems it 
has completed only the two MGR instances



If not:



ceph orch upgrade stop

ceph orch upgrade start --ceph-version 15.2.2



and monitor the watch-debug log



Make sure at the end you run:



ceph config set mgr mgr/cephadm/log_to_cluster_level info



 On Wed, 20 May 2020 21:07:43 +0800 Gencer W. Genç  
wrote 


Ah yes,



{

    "mon": {

        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus 
(stable)": 2

    },

    "mgr": {

        "ceph version 15.2.2 (0c857e985a29d90501a285f242ea9c008df49eb8) octopus 
(stable)": 2

    },

    "osd": {

        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus 
(stable)": 24

    },

    "mds": {

        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus 
(stable)": 2

    },

    "overall": {

        "ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus 
(stable)": 28,

        "ceph version 15.2.2 (0c857e985a29d90501a285f242ea9c008df49eb8) octopus 
(stable)": 2

    }

}




How can i fix this?



Gencer.

On 20.05.2020 16:04:33, Ashley Merrick <mailto:singap...@amerrick.co.uk> wrote:

Does:



ceph versions



show any services yet running on 15.2.2?



 On Wed, 20 May 2020 21:01:12 +0800 Gencer W. Genç 
<mailto:gen...@gencgiyen.com> wrote 


Hi Ashley,$ ceph orch upgrade status 




{

    "target_image": "docker.io/ceph/ceph:v15.2.2",

    "in_progress": true,

    "services_complete": [],

    "message": ""

}




Thanks,

Gencer.



On 20.05.2020 15:58:34, Ashley Merrick <mailto:singap...@amerrick.co.uk> wrote:

What does 

ceph orch upgrade status 

show?



 On Wed, 20 May 2020 20:52:39 +0800 Gencer W. Genç 
<mailto:gen...@gencgiyen.com> wrote 


Hi, 
 
I've 15.2.1 installed on all machines. On primary machine I executed ceph 
upgrade command: 
 
$ ceph orch upgrade start --ceph-version 15.2.2 
 
 
When I check ceph -s I see this: 
 
  progress: 
    Upgrade to docker.io/ceph/ceph:v15.2.2 (30m) 
      [=...] (remaining: 8h) 
 
It says 8 hours. It is already ran for 3 hours. No upgrade processed. It get 
stuck at this point. 
 
Is there any way to know why this has stuck? 
 
Thanks, 
Gencer.
___
ceph-users mailing list -- mailto:ceph-users@ceph.io
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph orch upgrade stuck at the beginning.

[Ashley Merrick]

Does:



ceph versions



show any services yet running on 15.2.2?



 On Wed, 20 May 2020 21:01:12 +0800 Gencer W. Genç  
wrote 


Hi Ashley,$ ceph orch upgrade status 




{

    "target_image": "docker.io/ceph/ceph:v15.2.2",

    "in_progress": true,

    "services_complete": [],

    "message": ""

}




Thanks,

Gencer.



On 20.05.2020 15:58:34, Ashley Merrick <mailto:singap...@amerrick.co.uk> wrote:

What does 

ceph orch upgrade status 

show?



 On Wed, 20 May 2020 20:52:39 +0800 Gencer W. Genç 
<mailto:gen...@gencgiyen.com> wrote 


Hi, 
 
I've 15.2.1 installed on all machines. On primary machine I executed ceph 
upgrade command: 
 
$ ceph orch upgrade start --ceph-version 15.2.2 
 
 
When I check ceph -s I see this: 
 
  progress: 
    Upgrade to docker.io/ceph/ceph:v15.2.2 (30m) 
      [=...] (remaining: 8h) 
 
It says 8 hours. It is already ran for 3 hours. No upgrade processed. It get 
stuck at this point. 
 
Is there any way to know why this has stuck? 
 
Thanks, 
Gencer.
___
ceph-users mailing list -- mailto:ceph-users@ceph.io
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 15.2.2 Upgrade - Corruption: error in middle of record

[Ashley Merrick]

I attached the log but was too big and got moderated.



Here is it in a paste bin : https://pastebin.pl/view/69b2beb9



I have cut the log to start from the point of the original upgrade.



Thanks



 On Wed, 20 May 2020 20:55:51 +0800 Igor Fedotov  wrote 



Dan, thanks for the info. Good to know. 
 
Failed QA run in the ticket uses snappy though. 
 
And in fact any stuff writing to process memory can  introduce data 
corruption in the similar manner. 
 
So will keep that in mind but IMO relation to compression is still not 
evident... 
 
 
Kind regards, 
 
Igor 
 
 
On 5/20/2020 3:32 PM, Dan van der Ster wrote: 
> lz4 ? It's not obviously related, but I've seen it involved in really 
> non-obvious ways: https://tracker.ceph.com/issues/39525 
> 
> -- dan 
> 
> On Wed, May 20, 2020 at 2:27 PM Ashley Merrick 
> <mailto:singap...@amerrick.co.uk> wrote: 
>> Thanks, fyi the OSD's that went down back two pools, an Erasure code Meta 
>> (RBD) and cephFS Meta. The cephFS Pool does have compresison enabled ( I 
>> noticed it mentioned in the ceph tracker) 
>> 
>> 
>> 
>> Thanks 
>> 
>> 
>> 
>> 
>> 
>>  On Wed, 20 May 2020 20:17:33 +0800 Igor Fedotov 
>> <mailto:ifedo...@suse.de> wrote  
>> 
>> 
>> 
>> Hi Ashley, 
>> 
>> looks like this is a regression. Neha observed similar error(s) during 
>> here QA run, see https://tracker.ceph.com/issues/45613 
>> 
>> 
>> Please preserve broken OSDs for a while if possible, likely I'll come 
>> back to you for more information to troubleshoot. 
>> 
>> 
>> Thanks, 
>> 
>> Igor 
>> 
>> On 5/20/2020 1:26 PM, Ashley Merrick wrote: 
>> 
>>> So reading online it looked a dead end error, so I recreated the 3 OSD's on 
>>> that node and now working fine after a reboot. 
>>> 
>>> 
>>> 
>>> However I restarted the next server with 3 OSD's and one of them is now 
>>> facing the same issue. 
>>> 
>>> 
>>> 
>>> Let me know if you need any more logs. 
>>> 
>>> 
>>> 
>>> Thanks 
>>> 
>>> 
>>> 
>>>  On Wed, 20 May 2020 17:02:31 +0800 Ashley Merrick 
>>> <mailto:mailto:singap...@amerrick.co.uk> wrote  
>>> 
>>> 
>>> I just upgraded a cephadm cluster from 15.2.1 to 15.2.2. 
>>> 
>>> 
>>> 
>>> Everything went fine on the upgrade, however after restarting one node that 
>>> has 3 OSD's for ecmeta two of the 3 ODS's now wont boot with the following 
>>> error: 
>>> 
>>> 
>>> 
>>> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
>>> 7fbcc46f7ec0  4 rocksdb: [db/version_set.cc:3757] Recovered from manifest 
>>> file:db/MANIFEST-002768 succeeded,manifest_file_number is 2768, 
>>> next_file_number is 2775, last_sequence is 188026749, log_number is 
>>> 2767,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 
>>> 0 
>>> 
>>> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
>>> 7fbcc46f7ec0  4 rocksdb: [db/version_set.cc:3766] Column family [default] 
>>> (ID 0), log number is 2767 
>>> 
>>> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
>>> 7fbcc46f7ec0  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1589963382599157, 
>>> "job": 1, "event": "recovery_started", "log_files": [2769]} 
>>> 
>>> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
>>> 7fbcc46f7ec0  4 rocksdb: [db/db_impl_open.cc:583] Recovering log #2769 mode 
>>> 0 
>>> 
>>> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
>>> 7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
>>> 537526 bytes; Corruption: error in middle of record 
>>> 
>>> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
>>> 7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
>>> 32757 bytes; Corruption: missing start of fragmented record(1) 
>>> 
>>> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
>>> 7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
>>> 32757 bytes; Corruption: missing start of fragmented record(1) 
>>> 
>>> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
>>> 7fbcc46f7ec

[ceph-users] Re: ceph orch upgrade stuck at the beginning.

[Ashley Merrick]

What does 

ceph orch upgrade status 

show?



 On Wed, 20 May 2020 20:52:39 +0800 Gencer W. Genç  
wrote 


Hi, 
 
I've 15.2.1 installed on all machines. On primary machine I executed ceph 
upgrade command: 
 
$ ceph orch upgrade start --ceph-version 15.2.2 
 
 
When I check ceph -s I see this: 
 
  progress: 
    Upgrade to docker.io/ceph/ceph:v15.2.2 (30m) 
      [=...] (remaining: 8h) 
 
It says 8 hours. It is already ran for 3 hours. No upgrade processed. It get 
stuck at this point. 
 
Is there any way to know why this has stuck? 
 
Thanks, 
Gencer.
___
ceph-users mailing list -- mailto:ceph-users@ceph.io
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 15.2.2 Upgrade - Corruption: error in middle of record

[Ashley Merrick]

Is a single shared main device.



Sadly I had already rebuilt the failed OSD's to bring me back in the green 
after a while.

I have just tried a few restarts and none are failing (seems after a rebuild 
using 15.2.2 they are stable?)



I don't have any other servers/OSD's I am willing to risk not starting right 
this minute ,if it does happen again I will grab the logs.



@Dan yeah is using lz4



Thanks

 On Wed, 20 May 2020 20:30:27 +0800 Igor Fedotov  wrote 




I don't believe compression is related to be honest.

Wondering if these OSDs have standalone WAL and/or DB devices or
  just a single shared main device.

Also could you please set debug-bluefs/debug-bluestore to 20 and
  collect startup log for broken OSD.



Kind regards,

Igor

On 5/20/2020 3:27 PM, Ashley Merrick
  wrote:




Thanks, fyi the OSD's that went down back two pools, an
  Erasure code Meta (RBD) and cephFS Meta. The cephFS Pool does
  have compresison enabled ( I noticed it mentioned in the ceph
  tracker)



Thanks





 On Wed, 20 May 2020 20:17:33 +0800 Igor Fedotov mailto:ifedo...@suse.de 
wrote 



Hi Ashley, 
 
 looks like this is a regression. Neha observed similar
  error(s) during 
 here QA run, see https://tracker.ceph.com/issues/45613 
 
 
 Please preserve broken OSDs for a while if possible,
  likely I'll come 
 back to you for more information to troubleshoot. 
 
 
 Thanks, 
 
 Igor 
 
 On 5/20/2020 1:26 PM, Ashley Merrick wrote: 
 
 > So reading online it looked a dead end error, so I
  recreated the 3 OSD's on that node and now working fine
  after a reboot. 
 > 
 > 
 > 
 > However I restarted the next server with 3 OSD's and
  one of them is now facing the same issue. 
 > 
 > 
 > 
 > Let me know if you need any more logs. 
 > 
 > 
 > 
 > Thanks 
 > 
 > 
 > 
 >  On Wed, 20 May 2020 17:02:31 +0800 Ashley
  Merrick <mailto:singap...@amerrick.co.uk>
  wrote  
 > 
 > 
 > I just upgraded a cephadm cluster from 15.2.1 to
  15.2.2. 
 > 
 > 
 > 
 > Everything went fine on the upgrade, however after
  restarting one node that has 3 OSD's for ecmeta two of the
  3 ODS's now wont boot with the following error: 
 > 
 > 
 > 
 > May 20 08:29:42 sn-m01 bash[6833]: debug
  2020-05-20T08:29:42.598+ 7fbcc46f7ec0  4 rocksdb:
  [db/version_set.cc:3757] Recovered from manifest  
succeeded,manifest_file_number is
  2768, next_file_number is 2775, last_sequence is
  188026749, log_number is 2767,prev_log_number is
  0,max_column_family is 0,min_log_number_to_keep is 0 
 > 
 > May 20 08:29:42 sn-m01 bash[6833]: debug
  2020-05-20T08:29:42.598+ 7fbcc46f7ec0  4 rocksdb:
  [db/version_set.cc:3766] Column family [default] (ID 0),
  log number is 2767 
 > 
 > May 20 08:29:42 sn-m01 bash[6833]: debug
  2020-05-20T08:29:42.598+ 7fbcc46f7ec0  4 rocksdb:
  EVENT_LOG_v1 {"time_micros": 1589963382599157, "job": 1,
  "event": "recovery_started", "log_files": [2769]} 
 > 
 > May 20 08:29:42 sn-m01 bash[6833]: debug
  2020-05-20T08:29:42.598+ 7fbcc46f7ec0  4 rocksdb:
  [db/db_impl_open.cc:583] Recovering log #2769 mode 0 
 > 
 > May 20 08:29:42 sn-m01 bash[6833]: debug
  2020-05-20T08:29:42.598+ 7fbcc46f7ec0  3 rocksdb:
  [db/db_impl_open.cc:518] db/002769.log: dropping 537526
  bytes; Corruption: error in middle of record 
 > 
 > May 20 08:29:42 sn-m01 bash[6833]: debug
  2020-05-20T08:29:42.598+ 7fbcc46f7ec0  3 rocksdb:
  [db/db_impl_open.cc:518] db/002769.log: dropping 32757
  bytes; Corruption: missing start of fragmented record(1) 
 > 
 > May 20 08:29:42 sn-m01 bash[6833]: debug
  2020-05-20T08:29:42.602+ 7fbcc46f7ec0  3 rocksdb:
  [db/db_impl_open.cc:518] db/002769.log: dropping 32757
  bytes; Corruption: missing start of fragmented record(1) 
 > 
 > May 20 08:29:42 sn-m01 bash[6833]: debug
  2020-05-20T08:29:42.602+ 7fbcc46f7ec0  3 rocksdb:
  [db/db_impl_open.cc:518] db/002769.log: dropping 32757
  bytes; Corruption: missing start of fragmented record(1) 
 > 
 > May 20 08:29:42 sn-m01 bash[6833]: debug
  2020-05-20T08:29:42.602+ 7fbcc46f7ec0  3 rocksdb:
  [db/db_impl_open.cc:518] db/002769.log: dropping 32757
  bytes; Corruption: missing start of fragmented record(1) 
 > 
 > May 20 08:29:42 sn-m01 bash[6833]: debug
  2020-05-20T08:29:42.602+ 

[ceph-users] Re: 15.2.2 Upgrade - Corruption: error in middle of record

[Ashley Merrick]

Thanks, fyi the OSD's that went down back two pools, an Erasure code Meta (RBD) 
and cephFS Meta. The cephFS Pool does have compresison enabled ( I noticed it 
mentioned in the ceph tracker)



Thanks





 On Wed, 20 May 2020 20:17:33 +0800 Igor Fedotov  wrote 




Hi Ashley, 
 
looks like this is a regression. Neha observed similar error(s) during 
here QA run, see https://tracker.ceph.com/issues/45613 
 
 
Please preserve broken OSDs for a while if possible, likely I'll come 
back to you for more information to troubleshoot. 
 
 
Thanks, 
 
Igor 
 
On 5/20/2020 1:26 PM, Ashley Merrick wrote: 
 
> So reading online it looked a dead end error, so I recreated the 3 OSD's on 
> that node and now working fine after a reboot. 
> 
> 
> 
> However I restarted the next server with 3 OSD's and one of them is now 
> facing the same issue. 
> 
> 
> 
> Let me know if you need any more logs. 
> 
> 
> 
> Thanks 
> 
> 
> 
>  On Wed, 20 May 2020 17:02:31 +0800 Ashley Merrick 
> <mailto:singap...@amerrick.co.uk> wrote  
> 
> 
> I just upgraded a cephadm cluster from 15.2.1 to 15.2.2. 
> 
> 
> 
> Everything went fine on the upgrade, however after restarting one node that 
> has 3 OSD's for ecmeta two of the 3 ODS's now wont boot with the following 
> error: 
> 
> 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
> 7fbcc46f7ec0  4 rocksdb: [db/version_set.cc:3757] Recovered from manifest 
> file:db/MANIFEST-002768 succeeded,manifest_file_number is 2768, 
> next_file_number is 2775, last_sequence is 188026749, log_number is 
> 2767,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
> 7fbcc46f7ec0  4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 
> 0), log number is 2767 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
> 7fbcc46f7ec0  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1589963382599157, 
> "job": 1, "event": "recovery_started", "log_files": [2769]} 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
> 7fbcc46f7ec0  4 rocksdb: [db/db_impl_open.cc:583] Recovering log #2769 mode 0 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
> 7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
> 537526 bytes; Corruption: error in middle of record 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
> 7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
> 32757 bytes; Corruption: missing start of fragmented record(1) 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
> 7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
> 32757 bytes; Corruption: missing start of fragmented record(1) 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
> 7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
> 32757 bytes; Corruption: missing start of fragmented record(1) 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
> 7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
> 32757 bytes; Corruption: missing start of fragmented record(1) 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
> 7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
> 32757 bytes; Corruption: missing start of fragmented record(1) 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
> 7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
> 32757 bytes; Corruption: missing start of fragmented record(1) 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
> 7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
> 23263 bytes; Corruption: missing start of fragmented record(2) 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
> 7fbcc46f7ec0  4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all 
> background work 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
> 7fbcc46f7ec0  4 rocksdb: [db/db_impl.cc:563] Shutdown complete 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
> 7fbcc46f7ec0 -1 rocksdb: Corruption: error in middle of record 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
> 7fbcc46f7ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db erroring opening 
> db: 
> 
> May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05

[ceph-users] Re: 15.2.2 Upgrade - Corruption: error in middle of record

[Ashley Merrick]

So reading online it looked a dead end error, so I recreated the 3 OSD's on 
that node and now working fine after a reboot.



However I restarted the next server with 3 OSD's and one of them is now facing 
the same issue.



Let me know if you need any more logs.



Thanks



 On Wed, 20 May 2020 17:02:31 +0800 Ashley Merrick 
 wrote 


I just upgraded a cephadm cluster from 15.2.1 to 15.2.2. 
 
 
 
Everything went fine on the upgrade, however after restarting one node that has 
3 OSD's for ecmeta two of the 3 ODS's now wont boot with the following error: 
 
 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
7fbcc46f7ec0  4 rocksdb: [db/version_set.cc:3757] Recovered from manifest 
file:db/MANIFEST-002768 succeeded,manifest_file_number is 2768, 
next_file_number is 2775, last_sequence is 188026749, log_number is 
2767,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
7fbcc46f7ec0  4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 
0), log number is 2767 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
7fbcc46f7ec0  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1589963382599157, "job": 
1, "event": "recovery_started", "log_files": [2769]} 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
7fbcc46f7ec0  4 rocksdb: [db/db_impl_open.cc:583] Recovering log #2769 mode 0 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
537526 bytes; Corruption: error in middle of record 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 
bytes; Corruption: missing start of fragmented record(1) 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 
bytes; Corruption: missing start of fragmented record(1) 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 
bytes; Corruption: missing start of fragmented record(1) 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 
bytes; Corruption: missing start of fragmented record(1) 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 
bytes; Corruption: missing start of fragmented record(1) 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 
bytes; Corruption: missing start of fragmented record(1) 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 23263 
bytes; Corruption: missing start of fragmented record(2) 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background 
work 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  4 rocksdb: [db/db_impl.cc:563] Shutdown complete 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0 -1 rocksdb: Corruption: error in middle of record 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db erroring opening 
db: 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  1 bdev(0x558a28dd0700 /var/lib/ceph/osd/ceph-0/block) close 
 
May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.870+ 
7fbcc46f7ec0  1 bdev(0x558a28dd /var/lib/ceph/osd/ceph-0/block) close 
 
May 20 08:29:43 sn-m01 bash[6833]: debug 2020-05-20T08:29:43.118+ 
7fbcc46f7ec0 -1 osd.0 0 OSD:init: unable to mount object store 
 
May 20 08:29:43 sn-m01 bash[6833]: debug 2020-05-20T08:29:43.118+ 
7fbcc46f7ec0 -1  ** ERROR: osd init failed: (5) Input/output error 
 
 
 
Have I hit a bug, or is there something I can do to try and fix these OSD's? 
 
 
 
Thanks
___
ceph-users mailing list -- mailto:ceph-users@ceph.io
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] 15.2.2 Upgrade - Corruption: error in middle of record

[Ashley Merrick]

I just upgraded a cephadm cluster from 15.2.1 to 15.2.2.



Everything went fine on the upgrade, however after restarting one node that has 
3 OSD's for ecmeta two of the 3 ODS's now wont boot with the following error:



May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
7fbcc46f7ec0  4 rocksdb: [db/version_set.cc:3757] Recovered from manifest 
file:db/MANIFEST-002768 succeeded,manifest_file_number is 2768, 
next_file_number is 2775, last_sequence is 188026749, log_number is 
2767,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
7fbcc46f7ec0  4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 
0), log number is 2767

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
7fbcc46f7ec0  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1589963382599157, "job": 
1, "event": "recovery_started", "log_files": [2769]}

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
7fbcc46f7ec0  4 rocksdb: [db/db_impl_open.cc:583] Recovering log #2769 mode 0

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 
537526 bytes; Corruption: error in middle of record

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.598+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 
bytes; Corruption: missing start of fragmented record(1)

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 
bytes; Corruption: missing start of fragmented record(1)

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 
bytes; Corruption: missing start of fragmented record(1)

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 
bytes; Corruption: missing start of fragmented record(1)

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 
bytes; Corruption: missing start of fragmented record(1)

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 32757 
bytes; Corruption: missing start of fragmented record(1)

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  3 rocksdb: [db/db_impl_open.cc:518] db/002769.log: dropping 23263 
bytes; Corruption: missing start of fragmented record(2)

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background 
work

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  4 rocksdb: [db/db_impl.cc:563] Shutdown complete

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0 -1 rocksdb: Corruption: error in middle of record

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db erroring opening 
db:

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.602+ 
7fbcc46f7ec0  1 bdev(0x558a28dd0700 /var/lib/ceph/osd/ceph-0/block) close

May 20 08:29:42 sn-m01 bash[6833]: debug 2020-05-20T08:29:42.870+ 
7fbcc46f7ec0  1 bdev(0x558a28dd /var/lib/ceph/osd/ceph-0/block) close

May 20 08:29:43 sn-m01 bash[6833]: debug 2020-05-20T08:29:43.118+ 
7fbcc46f7ec0 -1 osd.0 0 OSD:init: unable to mount object store

May 20 08:29:43 sn-m01 bash[6833]: debug 2020-05-20T08:29:43.118+ 
7fbcc46f7ec0 -1  ** ERROR: osd init failed: (5) Input/output error



Have I hit a bug, or is there something I can do to try and fix these OSD's?



Thanks
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: v15.2.2 Octopus released

[Ashley Merrick]

I am getting the following error when trying to upgrade via cephadm



ceph orch upgrade status

{

    "target_image": "docker.io/ceph/ceph:v15.2.2",

    "in_progress": true,

    "services_complete": [],

    "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target 
image"

}


Are the packages not yet built?

Thanks



 On Tue, 19 May 2020 04:27:21 +0800 Abhishek Lekshmanan  
wrote 



We're happy to announce the second bugfix release of Ceph Octopus stable 
release series, we recommend that all Octopus users upgrade. This 
release has a range of fixes across all components and a security fix. 
 
Notable Changes 
--- 
* CVE-2020-10736: Fixed an authorization bypass in mons & mgrs (Olle 
SegerDahl, Josh Durgin) 
 
For the complete changelog please refer to the full release blog at 
https://ceph.io/releases/v15-2-2-octopus-released/ 
 
Getting Ceph 
 
* Git at git://github.com/ceph/ceph.git 
* Tarball at http://download.ceph.com/tarballs/ceph-15.2.2.tar.gz 
* For packages, see 
http://docs.ceph.com/docs/master/install/get-packages/ 
* Release git sha1: 0c857e985a29d90501a285f242ea9c008df49eb8 
 
-- 
Abhishek Lekshmanan 
SUSE Software Solutions Germany GmbH 
GF: Felix Imendörffer, HRB 36809 (AG Nürnberg)
___
ceph-users mailing list -- mailto:ceph-users@ceph.io
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSD Inbalance - upmap mode

[Ashley Merrick]

 GiB  1.1 GiB  142 MiB   33 MiB  991 MiB  8.9 
GiB  11.39  0.16   31  up

41    hdd  0.00999   1.0   10 GiB  1.2 GiB  162 MiB   28 MiB  996 MiB  8.8 
GiB  11.58  0.16   32  up

   TOTAL  273 TiB  200 TiB  199 TiB  336 MiB  588 GiB   73 
TiB  73.21



Thanks


 On Tue, 05 May 2020 14:23:54 +0800 Ashley Merrick 
 wrote 


I have a cluster running 15.2.1, was originally running 14.x, the cluster is 
running the balance module in upmap mode (I have tried crush-compat in the past)

Most OSD's are around the same & used give or take 0.x, however there is one 
OSD that is down a good few % and a few that are above average by 1 or 2 %, I 
have been trying to get the balance to fix this.



I have tried running a manual osdmaptool command on an export of my map, but it 
lists no fixed however does display the underfall OSD in it's output (overfull 
3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,20 underfull [36])



The debug output is just lots of:



2020-05-05T06:15:39.172+ 7f3dfb0c3c40 10  trying 2.55

2020-05-05T06:15:39.172+ 7f3dfb0c3c40 10  2.55 [12,3,7,6,33,34,30,35,21,18] 
-> [12,3,7,6,33,34,30,35,21,16]

2020-05-05T06:15:39.172+ 7f3dfb0c3c40 10  will try adding new remapping 
pair 18 -> 16 for 2.55 NOT selected osd

2020-05-05T06:15:39.172+ 7f3dfb0c3c40 10  stddev 528.667 -> 528.667

2020-05-05T06:15:39.172+ 7f3dfb0c3c40 10  Overfull search osd.7 target 
170.667 deviation 9.33327



Is there anything I can to try and balance the overfull onto the underful OSDs 
to balance out the last bit.# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 1
tunable straw_calc_version 1
tunable allowed_bucket_algs 54

# devices
device 0 osd.0 class hdd
device 1 osd.1 class hdd
device 2 osd.2 class hdd
device 3 osd.3 class hdd
device 4 osd.4 class hdd
device 5 osd.5 class hdd
device 6 osd.6 class hdd
device 7 osd.7 class hdd
device 8 osd.8 class hdd
device 9 osd.9 class hdd
device 10 osd.10 class hdd
device 11 osd.11 class hdd
device 12 osd.12 class hdd
device 13 osd.13 class hdd
device 14 osd.14 class hdd
device 15 osd.15 class hdd
device 16 osd.16 class hdd
device 17 osd.17 class hdd
device 18 osd.18 class hdd
device 19 osd.19 class hdd
device 20 osd.20 class hdd
device 21 osd.21 class hdd
device 22 osd.22 class hdd
device 26 osd.26 class hdd
device 27 osd.27 class hdd
device 28 osd.28 class hdd
device 29 osd.29 class hdd
device 30 osd.30 class hdd
device 31 osd.31 class hdd
device 32 osd.32 class hdd
device 33 osd.33 class hdd
device 34 osd.34 class hdd
device 35 osd.35 class hdd
device 36 osd.36 class hdd
device 37 osd.37 class hdd
device 38 osd.38 class hdd
device 39 osd.39 class hdd
device 40 osd.40 class hdd
device 41 osd.41 class hdd

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host sn-m01 {
id -3   # do not change unnecessarily
id -4 class hdd # do not change unnecessarily
# weight 0.030
alg straw2
hash 0  # rjenkins1
item osd.0 weight 0.010
item osd.1 weight 0.010
item osd.2 weight 0.010
}
host sn-m03 {
id -15  # do not change unnecessarily
id -16 class hdd# do not change unnecessarily
# weight 0.030
alg straw2
hash 0  # rjenkins1
item osd.26 weight 0.010
item osd.27 weight 0.010
item osd.28 weight 0.010
}
host sn-m04 {
id -19  # do not change unnecessarily
id -20 class hdd# do not change unnecessarily
# weight 0.030
alg straw2
hash 0  # rjenkins1
item osd.39 weight 0.010
item osd.40 weight 0.010
item osd.41 weight 0.010
}
root meta {
id -1   # do not change unnecessarily
id -2 class hdd # do not change unnecessarily
# weight 0.087
alg straw2
hash 0  # rjenkins1
item sn-m01 weight 0.029
item sn-m03 weight 0.029
item sn-m04 weight 0.029
}
host sn-s01 {
id -5   # do not change unnecessarily
id -7 class hdd # do not change unnecessarily
# weight 90.960
alg straw2
hash 0  # rjenkins1
item osd.3 weight 9.096
item osd.4 weight 9.096
item osd.5 weight 9.096
item osd.6 weight 9.096
item osd.7 weight 9.096
item osd.8 weight 9.096
item osd.9 weight 9.096
item osd.10 weight 9.096
item osd.11 weight 9.096
item osd.12 weight 9.096
}
host sn-s02 {
id -9   # do not change unnecessarily
id -10 class hdd# do not change unnecessarily
# weight 90.960
alg straw2
hash 0  # rje

[ceph-users] Re: Fwd: Octopus on CentOS 7: lacking some packages

[Ashley Merrick]

As per the release notes : https://docs.ceph.com/docs/master/releases/octopus/



The dashboard and a few other modules aren't supported on CentOS 7.x due to 
python version / dependencies.



 On Wed, 06 May 2020 17:18:06 +0800 Sam Huracan  
wrote 


Hi Cephers, 
 
I am trying to install Ceph Octopus using ceph-deploy on CentOS 7, 
as installing ceph-mgr-dashboard, it required these packages: 
 - python3-cherrypy 
 - python3-jwt 
 - python3-routes 
https://pastebin.com/dSQPgGJD 
 
But, when I installed these packages, they were not available. 
https://pastebin.com/Hguf8kJe 
 
Is there anyone installed Octopus successfully on CentOS 7? Can you help me? 
 
Thanks in advance. 
___ 
ceph-users mailing list -- mailto:ceph-users@ceph.io 
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] OSD Inbalance - upmap mode

[Ashley Merrick]

I have a cluster running 15.2.1, was originally running 14.x, the cluster is 
running the balance module in upmap mode (I have tried crush-compat in the past)

Most OSD's are around the same & used give or take 0.x, however there is one 
OSD that is down a good few % and a few that are above average by 1 or 2 %, I 
have been trying to get the balance to fix this.



I have tried running a manual osdmaptool command on an export of my map, but it 
lists no fixed however does display the underfall OSD in it's output (overfull 
3,4,5,6,7,8,9,10,11,12,13,14,15,18,19,20 underfull [36])



The debug output is just lots of:



2020-05-05T06:15:39.172+ 7f3dfb0c3c40 10  trying 2.55

2020-05-05T06:15:39.172+ 7f3dfb0c3c40 10  2.55 [12,3,7,6,33,34,30,35,21,18] 
-> [12,3,7,6,33,34,30,35,21,16]

2020-05-05T06:15:39.172+ 7f3dfb0c3c40 10  will try adding new remapping 
pair 18 -> 16 for 2.55 NOT selected osd

2020-05-05T06:15:39.172+ 7f3dfb0c3c40 10  stddev 528.667 -> 528.667

2020-05-05T06:15:39.172+ 7f3dfb0c3c40 10  Overfull search osd.7 target 
170.667 deviation 9.33327



Is there anything I can to try and balance the overfull onto the underful OSDs 
to balance out the last bit.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 14.2.9 MDS Failing

[Ashley Merrick]

Quickly checking the code that calls that assert




if (version > omap_version) {

omap_version = version;

omap_num_objs = num_objs;

omap_num_items.resize(omap_num_objs);

journal_state = jstate;

} else if (version == omap_version) {

ceph_assert(omap_num_objs == num_objs);

if (jstate > journal_state)

journal_state = jstate;

}

}


Im not a dev, but not sure if this will help, seems could mean that MDS thinks 
its behind on omaps/too far ahead.


Anything happened recently? Just running a single MDS?


Hopefully someone else may see this and shine some light on what could be 
causing it.



 On Sat, 02 May 2020 02:10:58 +0800 marcopizz...@gmail.com wrote 


Hello,

Hoping you can help me.

Ceph had been largely problem free for us for the better part of a year.
We have a high file count in a single CephFS filesystem, and are seeing
this error in the logs:

/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.9/rpm/el7/BUILD/ceph-14.2.9/src/mds/OpenFileTable.cc:
777: FAILED ceph_assert(omap_num_objs == num_objs)

The issued seemed to occur this morning, and restarting the MDS as well as
rebooting the servers doesn't correct the problem.

Not really sure where to look next as the MDS daemons crash.

Appreciate any help you can provide

Marco
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Existing Cluster to cephadm - mds start failing

[Ashley Merrick]

So I'm still stuck against this bug which is stopping me adding any new service 
either OSD / MON / MDS e.t.c


>From my understanding if I enable cephx I should be able to get around this 
>but, is there a particular way I should tell cephadm to set cephx enabled on 
>the cluster before I reboot all the services with it enabled?


I know in a normal environment it would just be updating all the ceph.conf 
files and rebooting all the services (I have already checked every service has 
a valid key)


Thanks





 On Wed, 15 Apr 2020 13:42:02 +0800 singap...@amerrick.co.uk wrote 


I have logged the following bug ticket for it : 
https://tracker.ceph.com/issues/45091



I have also noticed another bug with cephadm which I have logged under : 
https://tracker.ceph.com/issues/45092



Thanks



 On Mon, 13 Apr 2020 12:36:01 +0800 Ashley Merrick 
 wrote 


Completed the migration of an existing Ceph cluster on Octopus to cephadm.



All OSD/MON/MGR moved fine, however upon running the command to setup some new 
MDS for cephfs they both failed to start.



After looking into the cephadm log's I found the following error:



Apr 13 06:26:15 sn-s01 systemd[1]: Started Ceph mds.cephfs.sn-s01.snkhfd for 
b1db6b36-0c4c-4bce-9cda-18834be0632d.

Apr 13 06:26:16 sn-s01 bash[3520809]: debug 2020-04-13T04:26:16.445+ 
7f380b908700 -1 monclient(hunting): handle_auth_bad_method server 
allowed_methods [1] but i only support [2]

Apr 13 06:26:16 sn-s01 bash[3520809]: debug 2020-04-13T04:26:16.445+ 
7f380b107700 -1 monclient(hunting): handle_auth_bad_method server 
allowed_methods [1] but i only support [2]

Apr 13 06:26:16 sn-s01 bash[3520809]: debug 2020-04-13T04:26:16.445+ 
7f380a906700 -1 monclient(hunting): handle_auth_bad_method server 
allowed_methods [1] but i only support [2]

Apr 13 06:26:16 sn-s01 bash[3520809]: failed to fetch mon config 
(--no-mon-config to skip



This cluster is running with cephx disabled, I imported the ceph.conf into 
cephadm fine and this has worked fine when the other services have started, but 
from the above error looks like maybe the mds is not checking if cephx is 
enabled or disabled before trying to communicate to the mon's?
___
ceph-users mailing list -- mailto:ceph-users@ceph.io
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Device /dev/rbd0 excluded by a filter.

[Ashley Merrick]

A quick google shows this:



You need to change your /etc/lvm/lvm.conf device name filter to include it. May 
be that your LVM filter is not allowing rbdX type disks to be used.



 On Mon, 27 Apr 2020 17:49:53 +0800 Marc Roos  
wrote 


It is a new image. -vvv says "Unrecognised LVM device type 252" Could 
this related to rbd features not being enabled/disabled? 
 
 
-Original Message- 
Cc: ceph-users 
Subject: Re: [ceph-users] Device /dev/rbd0 excluded by a filter. 
 
This normally means you have some form of partition data on the RBD 
disk. 
 
 
If you use -vvv on the pv command it should show you the reason, but yes 
redhat solutions require an active support subscription. 
 
 
 
 On Mon, 27 Apr 2020 17:43:02 +0800 Marc Roos 
 wrote  
 
 
 
 
Why is this info not available??? 
 
https://access.redhat.com/solutions/4009431 
 
___ 
ceph-users mailing list -- mailto:ceph-users@ceph.io 
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Device /dev/rbd0 excluded by a filter.

[Ashley Merrick]

This normally means you have some form of partition data on the RBD disk.



If you use -vvv on the pv command it should show you the reason, but yes redhat 
solutions require an active support subscription.



 On Mon, 27 Apr 2020 17:43:02 +0800 Marc Roos  
wrote 



Why is this info not available??? 
 
https://access.redhat.com/solutions/4009431 
 
___ 
ceph-users mailing list -- mailto:ceph-users@ceph.io 
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Existing Cluster to cephadm - mds start failing

[Ashley Merrick]

I have logged the following bug ticket for it : 
https://tracker.ceph.com/issues/45091



I have also noticed another bug with cephadm which I have logged under : 
https://tracker.ceph.com/issues/45092



Thanks



 On Mon, 13 Apr 2020 12:36:01 +0800 Ashley Merrick 
 wrote 


Completed the migration of an existing Ceph cluster on Octopus to cephadm. 
 
 
 
All OSD/MON/MGR moved fine, however upon running the command to setup some new 
MDS for cephfs they both failed to start. 
 
 
 
After looking into the cephadm log's I found the following error: 
 
 
 
Apr 13 06:26:15 sn-s01 systemd[1]: Started Ceph mds.cephfs.sn-s01.snkhfd for 
b1db6b36-0c4c-4bce-9cda-18834be0632d. 
 
Apr 13 06:26:16 sn-s01 bash[3520809]: debug 2020-04-13T04:26:16.445+ 
7f380b908700 -1 monclient(hunting): handle_auth_bad_method server 
allowed_methods [1] but i only support [2] 
 
Apr 13 06:26:16 sn-s01 bash[3520809]: debug 2020-04-13T04:26:16.445+ 
7f380b107700 -1 monclient(hunting): handle_auth_bad_method server 
allowed_methods [1] but i only support [2] 
 
Apr 13 06:26:16 sn-s01 bash[3520809]: debug 2020-04-13T04:26:16.445+ 
7f380a906700 -1 monclient(hunting): handle_auth_bad_method server 
allowed_methods [1] but i only support [2] 
 
Apr 13 06:26:16 sn-s01 bash[3520809]: failed to fetch mon config 
(--no-mon-config to skip 
 
 
 
This cluster is running with cephx disabled, I imported the ceph.conf into 
cephadm fine and this has worked fine when the other services have started, but 
from the above error looks like maybe the mds is not checking if cephx is 
enabled or disabled before trying to communicate to the mon's? 
___ 
ceph-users mailing list -- mailto:ceph-users@ceph.io 
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Existing Cluster to cephadm - mds start failing

[Ashley Merrick]

Completed the migration of an existing Ceph cluster on Octopus to cephadm.



All OSD/MON/MGR moved fine, however upon running the command to setup some new 
MDS for cephfs they both failed to start.



After looking into the cephadm log's I found the following error:



Apr 13 06:26:15 sn-s01 systemd[1]: Started Ceph mds.cephfs.sn-s01.snkhfd for 
b1db6b36-0c4c-4bce-9cda-18834be0632d.

Apr 13 06:26:16 sn-s01 bash[3520809]: debug 2020-04-13T04:26:16.445+ 
7f380b908700 -1 monclient(hunting): handle_auth_bad_method server 
allowed_methods [1] but i only support [2]

Apr 13 06:26:16 sn-s01 bash[3520809]: debug 2020-04-13T04:26:16.445+ 
7f380b107700 -1 monclient(hunting): handle_auth_bad_method server 
allowed_methods [1] but i only support [2]

Apr 13 06:26:16 sn-s01 bash[3520809]: debug 2020-04-13T04:26:16.445+ 
7f380a906700 -1 monclient(hunting): handle_auth_bad_method server 
allowed_methods [1] but i only support [2]

Apr 13 06:26:16 sn-s01 bash[3520809]: failed to fetch mon config 
(--no-mon-config to skip



This cluster is running with cephx disabled, I imported the ceph.conf into 
cephadm fine and this has worked fine when the other services have started, but 
from the above error looks like maybe the mds is not checking if cephx is 
enabled or disabled before trying to communicate to the mon's?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Octopus] OSD overloading

[Ashley Merrick]

Are you sure your not being hit by:



ceph config set osd bluestore_fsck_quick_fix_on_mount false @ 
https://docs.ceph.com/docs/master/releases/octopus/

Have all your OSD's successfully completed the fsck?



Reasons I say that is I can see "20 OSD(s) reporting legacy (not per-pool) 
BlueStore omap usage stats"





 On Thu, 09 Apr 2020 02:15:02 +0800 Jack  wrote 




Just to confirm this does not get better: 
 
root@backup1:~# ceph status 
 cluster: 
 id: 9cd41f0f-936d-4b59-8e5d-9b679dae9140 
 health: HEALTH_WARN 
 20 OSD(s) reporting legacy (not per-pool) BlueStore omap 
usage stats 
 4/50952060 objects unfound (0.000%) 
 nobackfill,norecover,noscrub,nodeep-scrub flag(s) set 
 1 osds down 
 3 nearfull osd(s) 
 Reduced data availability: 826 pgs inactive, 616 pgs down, 
185 pgs peering, 158 pgs stale 
 Low space hindering backfill (add storage if this doesn't 
resolve itself): 93 pgs backfill_toofull 
 Degraded data redundancy: 13285415/101904120 objects 
degraded (13.037%), 706 pgs degraded, 696 pgs undersized 
 989 pgs not deep-scrubbed in time 
 378 pgs not scrubbed in time 
 10 pool(s) nearfull 
 2216 slow ops, oldest one blocked for 13905 sec, daemons 
[osd.1,osd.11,osd.20,osd.24,osd.25,osd.29,osd.31,osd.37,osd.4,osd.5]... 
have slow ops. 
 
 services: 
 mon: 1 daemons, quorum backup1 (age 8d) 
 mgr: backup1(active, since 8d) 
 osd: 37 osds: 26 up (since 9m), 27 in (since 2h); 626 remapped pgs 
 flags nobackfill,norecover,noscrub,nodeep-scrub 
 rgw: 1 daemon active (backup1.odiso.net) 
 
 task status: 
 
 data: 
 pools:   10 pools, 2785 pgs 
 objects: 50.95M objects, 92 TiB 
 usage:   121 TiB used, 39 TiB / 160 TiB avail 
 pgs: 29.659% pgs not active 
 13285415/101904120 objects degraded (13.037%) 
 433992/101904120 objects misplaced (0.426%) 
 4/50952060 objects unfound (0.000%) 
 840 active+clean+snaptrim_wait 
 536 down 
 490 active+undersized+degraded+remapped+backfilling 
 326 active+clean 
 113 peering 
 88  active+undersized+degraded 
 83  active+undersized+degraded+remapped+backfill_toofull 
 79  stale+down 
 63  stale+peering 
 51  active+clean+snaptrim 
 24  activating 
 22  active+recovering+degraded 
 19  active+remapped+backfilling 
 13  stale+active+undersized+degraded 
 9   remapped+peering 
 9   active+undersized+remapped+backfilling 
 9 
active+undersized+degraded+remapped+backfill_wait+backfill_toofull 
 2   stale+active+clean+snaptrim 
 2   active+undersized 
 1   stale+active+clean+snaptrim_wait 
 1   active+remapped+backfill_toofull 
 1   active+clean+snaptrim_wait+laggy 
 1   active+recovering+undersized+remapped 
 1   down+remapped 
 1   activating+undersized+degraded+remapped 
 1   active+recovering+laggy 
 
On 4/8/20 3:27 PM, Jack wrote: 
> The CPU is used by userspace, not kernelspace 
> 
> Here is the perf top, see attachment 
> 
> Rocksdb eats everything :/ 
> 
> 
> On 4/8/20 3:14 PM, Paul Emmerich wrote: 
>> What's the CPU busy with while spinning at 100%? 
>> 
>> Check "perf top" for a quick overview 
>> 
>> 
>> Paul 
>> 
> 
> 
> ___ 
> ceph-users mailing list -- mailto:ceph-users@ceph.io 
> To unsubscribe send an email to mailto:ceph-users-le...@ceph.io 
> 
___ 
ceph-users mailing list -- mailto:ceph-users@ceph.io 
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: HEALTH_WARN 1 pools have too few placement groups

[Ashley Merrick]

This was a bug in 14.2.7 and calculation for EC pools.



It has been fixed in 14.2.8





 On Mon, 16 Mar 2020 16:21:41 +0800 Dietmar Rieder 
 wrote 



Hi, 
 
I was planing to activate the pg_autoscaler on a EC (6+3) pool which I 
created two years ago. 
Back then I calculated the total # of pgs for this pool with a target 
per ods pg # of 150 (this was the recommended /osd pg number as far as I 
recall). 
 
I used the RedHat ceph pg per pool calculator [1] with the following 
settings: 
PoolyType: EC 
K: 6 
M: 3 
OSD #: 220 
%Data: 100.00 
Target PGs per OSD: 150 
 
This resulted into a suggested PG count of 4096, which I used to create 
that pool. 
(BTW: I get the same using the ceph.io pgcalc [2], setting size to 9 
(=EC 6+3)) 
 
# ceph osd df shows that I have now 162-169 pgs / osd 
 
Now I enabled the pg_autoscaler and set autoscale_mode to "warn", using 
the following commands: 
 
# ceph mgr module  enable pg_autoscaler 
# ceph osd pool set hdd-ec-data-pool pg_autoscale_mode warn 
 
when i set the target_ratio_size to 1, using: 
 
# ceph osd pool set hdd-ec-data-pool target_size_ratio 1 
 
I get the warning: 
1 pools have too few placement groups 
 
The autoscale status shows that pg number in the EC pool would get 
scaled up to 16384 pgs: 
 
# ceph osd pool autoscale-status 
POOLSIZE TARGET SIZE RATE RAW CAPACITY  RATIO TARGET 
RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE 
ssd-rep-metadata-pool 11146M  3.0   29805G 0.0011 
 1.0 64  4 off 
ssd-rep-data-pool 837.8G  3.0   29805G 0.0843 
 1.0   1024 64 off 
hdd-ec-data-pool  391.3T  1.51614T 0.3636 
1.  1.0   4096  16384 warn 
 
How does this relate the the calculated number of 4096 pgs for that EC 
pool? I mean 16384 is quite a jump from the original value. 
Is the autoscaler miscalculating something or is the pg calculator wrong? 
 
Can someone explain it? (My cluster is on ceph version 14.2.7) 
 
 
 
Thanks so much 
 Dietmar 
 
[1]: https://access.redhat.com/labs/cephpgc/ 
[2]: https://ceph.io/pgcalc/ 
 
 
-- 
_ 
D i e t m a r  R i e d e r, Mag.Dr. 
Innsbruck Medical University 
Biocenter - Institute of Bioinformatics 
Innrain 80, 6020 Innsbruck 
Email: mailto:dietmar.rie...@i-med.ac.at 
Web: http://www.icbi.at 
___ 
ceph-users mailing list -- mailto:ceph-users@ceph.io 
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: New 3 node Ceph cluster

[Ashley Merrick]

I would say you definitely need more RAM with that many disks.




 On Sat, 14 Mar 2020 15:17:14 +0800 amudha...@gmail.com wrote 


Hi,

I am planning to create a new 3 node ceph storage cluster.

I will be using Cephfs + with samba for max 10 clients for upload and
download.

Storage Node HW is Intel Xeon E5v2 8 core single Proc, 32GB RAM and 10Gb
Nic 2 nos., 6TB SATA HDD 24 Nos. each node, OS separate SSD disk.

Earlier I have tested orchestration using ceph-deploy in the test setup.
now, is there any other alternative to ceph-deploy?

Can I restrict folder access to the user using cephfs + vfs samba or should
I use ceph client + samba?

Ubuntu or Centos?

Any block size consideration for object size, metadata when using cephfs?

Idea or suggestion from existing users. I am also going to start to explore
all the above.

regards
Amudhan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Near Perfect PG distrubtion apart from two OSD

[Ashley Merrick]

    up

24   hdd 0.00999  1.0  10 GiB 1.5 GiB 467 MiB  24 KiB 1024 MiB 8.5 GiB 
14.57 0.22  34 up

25   hdd 0.00999  1.0  10 GiB 1.5 GiB 462 MiB  28 KiB 1024 MiB 8.5 GiB 
14.52 0.22  34 up

26   hdd 0.00999  1.0  10 GiB 1.5 GiB 464 MiB 1.4 MiB 1023 MiB 8.5 GiB 
14.53 0.22  34 up

27   hdd 0.00999  1.0  10 GiB 1.5 GiB 464 MiB  40 KiB 1024 MiB 8.5 GiB 
14.53 0.22  33 up

28   hdd 0.00999  1.0  10 GiB 1.5 GiB 462 MiB  12 KiB 1024 MiB 8.5 GiB 
14.52 0.22  34 up

    TOTAL 273 TiB 183 TiB 182 TiB 6.1 MiB  574 GiB  90 TiB 67.02

MIN/MAX VAR: 0.22/1.15  STDDEV: 30.40



 On Fri, 10 Jan 2020 14:57:05 +0800 Ashley Merrick 
<mailto:singap...@amerrick.co.uk> wrote 










Hey,



I have a cluster of 30 OSD's that is near perfect distribution minus two OSD's.



I am running ceph version 14.2.6 however has been the same for the previous 
versions, I have the balance module enabled in upmap and it says no 
improvements, I have also tried in crush mode.



ceph balancer status

{

    "last_optimize_duration": "0:00:01.123659",

    "plans": [],

    "mode": "upmap",

    "active": true,

    "optimize_result": "Unable to find further optimization, or pool(s)' pg_num 
is decreasing, or distribution is already perfect",

    "last_optimize_started": "Fri Jan 10 06:11:08 2020"

}



I have read a few email threads on the ML recently about similar cases but not 
sure if I am hitting the same "bug" as its only two that are off the rest are 
almost perfect.



ceph osd df

ID CLASS WEIGHT  REWEIGHT SIZE    RAW USE DATA    OMAP    META AVAIL   %USE 
 VAR  PGS STATUS

23   hdd 0.00999  1.0  10 GiB 1.4 GiB 434 MiB 1.4 MiB 1023 MiB 8.6 GiB 
14.24 0.21  33 up

24   hdd 0.00999  1.0  10 GiB 1.4 GiB 441 MiB  48 KiB 1024 MiB 8.6 GiB 
14.31 0.21  34 up

25   hdd 0.00999  1.0  10 GiB 1.4 GiB 435 MiB  24 KiB 1024 MiB 8.6 GiB 
14.26 0.21  34 up

26   hdd 0.00999  1.0  10 GiB 1.4 GiB 436 MiB 1.4 MiB 1023 MiB 8.6 GiB 
14.27 0.21  34 up

27   hdd 0.00999  1.0  10 GiB 1.4 GiB 437 MiB  16 KiB 1024 MiB 8.6 GiB 
14.27 0.21  33 up

28   hdd 0.00999  1.0  10 GiB 1.4 GiB 436 MiB  36 KiB 1024 MiB 8.6 GiB 
14.26 0.21  34 up

 3   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB  76 KiB   19 GiB 3.0 TiB 
67.26 1.00 170 up

 4   hdd 9.09599  1.0 9.1 TiB 6.2 TiB 6.1 TiB  44 KiB   19 GiB 2.9 TiB 
67.77 1.01 172 up

 5   hdd 9.09599  1.0 9.1 TiB 6.3 TiB 6.3 TiB 112 KiB   20 GiB 2.8 TiB 
69.50 1.03 176 up

 6   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB  17 KiB   19 GiB 2.9 TiB 
67.58 1.01 171 up

 7   hdd 9.09599  1.0 9.1 TiB 6.7 TiB 6.7 TiB  88 KiB   21 GiB 2.4 TiB 
73.98 1.10 187 up

 8   hdd 9.09599  1.0 9.1 TiB 6.5 TiB 6.5 TiB  76 KiB   20 GiB 2.6 TiB 
71.84 1.07 182 up

 9   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB 120 KiB   19 GiB 3.0 TiB 
67.24 1.00 170 up

10   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB  72 KiB   19 GiB 3.0 TiB 
67.19 1.00 170 up

11   hdd 9.09599  1.0 9.1 TiB 6.2 TiB 6.2 TiB  40 KiB   19 GiB 2.9 TiB 
68.06 1.01 172 up

12   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB  28 KiB   19 GiB 3.0 TiB 
67.48 1.00 170 up

13   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB  36 KiB   19 GiB 3.0 TiB 
67.04 1.00 170 up

14   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB 108 KiB   19 GiB 3.0 TiB 
67.30 1.00 170 up

15   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB  68 KiB   19 GiB 3.0 TiB 
67.41 1.00 170 up

16   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB 152 KiB   19 GiB 2.9 TiB 
67.61 1.01 171 up

17   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB  36 KiB   19 GiB 3.0 TiB 
67.16 1.00 170 up

18   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB  41 KiB   19 GiB 3.0 TiB 
67.19 1.00 170 up

19   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB  64 KiB   19 GiB 3.0 TiB 
67.49 1.00 171 up

20   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB  12 KiB   19 GiB 3.0 TiB 
67.55 1.01 171 up

21   hdd 9.09599  1.0 9.1 TiB 6.2 TiB 6.1 TiB  76 KiB   19 GiB 2.9 TiB 
67.76 1.01 171 up

22   hdd 9.09599  1.0 9.1 TiB 6.2 TiB 6.2 TiB  12 KiB   19 GiB 2.9 TiB 
68.05 1.01 172 up

29   hdd 9.09599  1.0 9.1 TiB 5.8 TiB 5.8 TiB 108 KiB   17 GiB 3.3 TiB 
63.59 0.95 163 up

30   hdd 9.09599  1.0 9.1 TiB 5.9 TiB 5.9 TiB  24 KiB   18 GiB 3.2 TiB 
65.18 0.97 167 up

31   hdd 9.09599  1.0 9.1 TiB 6.1 TiB 6.1 TiB  44 KiB   18 GiB 3.0 TiB 
66.74 0.99 171 up

32   hdd 9.09599  1.0 9.1 TiB 6.0 TiB 6.0 TiB 220 KiB   18 GiB 3.1 TiB 
66.31 0.99 170 up

33   hdd 9.09599  1.0 9.1 TiB 6.0 TiB 5.9 TiB  36 KiB   18 GiB 3.1 TiB 
65.54 0.98 168 up

34   hdd 9.09599  1.0 9.1 TiB 6.0 TiB 6.0 TiB  44 KiB   18 GiB 3.1 TiB 
66.33 0.99 170 up

35   hdd 9.09599  1.0 9.1 TiB 5.9 TiB 5.9 TiB  68 KiB  

[ceph-users] Re: Recovering from a Failed Disk (replication 1)

[Ashley Merrick]

I think your better off doing the DD method, you can export and import a PG at 
a time (ceph-objectstore-tool)

But if the disk is failing a DD is probably your best method.




 On Thu, 17 Oct 2019 11:44:20 +0800 vladimir franciz blando 
 wrote 


Sorry for not being clear, when I say healthy disk, I mean those are already an 
OSD, so I need to transfer the data from the failed OSD to the other OSDs that 
are healthy.


- Vlad










ᐧ


On Thu, Oct 17, 2019 at 11:31 AM Konstantin Shalygin  
wrote:


On 10/17/19 10:29 AM, vladimir franciz blando wrote:
 > I have a not ideal setup on one of my cluster,  3 ceph  nodes but 
 > using replication 1 on all pools (don't ask me why replication 1, it's 
 > a long story).
 >
 > So it has come to this situation that a disk keeps on crashing, 
 > possible a hardware failure and I need to recover from that.
 >
 > What's my best option for me to recover the data from the failed disk 
 > and transfer it to the other healthy disks?
 >
 > This cluster is using Firefly
 
 
 `dd if=/dev/old_drive of=/dev/new_drive` I guess.
 
 
 
 k
 


___
ceph-users mailing list -- mailto:ceph-users@ceph.io 
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Constant write load on 4 node ceph cluster

[Ashley Merrick]

Is the storage being used for the whole VM disk?



If so have you checked none of your software is writing constant log's? Or 
something that could continuously write to disk.



If your running a new version you can use : 
https://docs.ceph.com/docs/mimic/mgr/iostat/ to locate the exact RBD image.





 On Mon, 14 Oct 2019 20:32:53 +0800 Ingo Schmidt  
wrote 



Hi all 
 
We have a 4 node ceph cluster that runs generally fine. It is the storage 
backend for our virtualization cluster with Proxmox, that runs about 40 virtual 
machines (80% various Linux Servers). Now that we have implemented monitoring, 
i see that there is a quite constant write load of about 4-10MB/s throughout 
the whole day and even weekends, while read load clearly is bound to daily work 
hours. 
I am wondering whats up with this. Is this normal? 
Outside working hours, there is almost no read load, well under 1MB/s, while 
write Load stays relatively constant at around 5MB/s. During rebalancing of the 
Cluster, normal write load decreases significantly to under 1MB/s. Overall 
performance is not greatly affected by rebalancing as it is tuned to relatively 
slow backfills (osd_max_backfills = '3' | osd_recovery_max_active = '3') 
After rebalance has been finished, the average write load of about 5MB/s 
reappears. 
I have attached a little screenshot for reference. 
 
 
Can someone shed a little light on this? 
Thanks. 
 
kind regards 
Ingo Schmidt 
 
 
IT-Department 
Island Municipality Langeoog 
with in-house operations 
Tourismusservice and Schiffahrt 
 
Hauptstraße 28 
26465 Langeoog 
Deutschland 
 
[ http://www.langeoog.de/ ] 
___ 
ceph-users mailing list -- mailto:ceph-users@ceph.io 
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cannot start virtual machines KVM / LXC

[Ashley Merrick]

gress in "Rebalancing after osd.9 
marked out", the number of objects degraded dropped to 360047/153249771 
and the number of objects misplaced to 1998603/153249771. 
 
I think this is very little progress in 2 days with no activity (over 
the weekend) on the cluster. 
 
 
 
Am 21.09.2019 um 20:39 schrieb Paul Emmerich: 
> On Sat, Sep 21, 2019 at 6:47 PM Thomas <mailto:74cmo...@gmail.com> wrote: 
>> Hello, 
>> 
>> I have re-created the OSDs using these disks. 
>> Can I still export the affected PGs manually? 
> No, the data probably cannot be recovered in this case :( 
> (It might still be somewhere on the disk if it hasn't been overwritten 
> yet, but it's virtually impossible to recover it: the metadata has 
> almost certainly long been overwritten) 
> 
> But that's only for 17 PGs showing up as unknown. The ones stuck in 
> peering can probably be revived but without the latest writes. Can you 
> run "ceph pg X.YX query" on one of the PGs stuck in peering? 
> It might tell you what's wrong and how to proceed. 
> 
> But even 17 PGs will probably affect almost all of your VMs/containers... 
> 
> Paul 
> 
>> Regards 
>> Thomas 
>> 
>> 
>> Am 20.09.19 um 21:15 schrieb Paul Emmerich: 
>>> On Fri, Sep 20, 2019 at 1:31 PM Thomas Schneider 
>>> <mailto:74cmo...@gmail.com> wrote: 
>>>> Hi, 
>>>> 
>>>> I cannot get rid of 
>>>>   pgs unknown 
>>>> because there were 3 disks that couldn't be started. 
>>>> Therefore I destroyed the relevant OSD and re-created it for the 
>>>> relevant disks. 
>>> and you had it configured to run with replica 3? Well, I guess the 
>>> down PGs where located on these three disks that you wiped. 
>>> 
>>> Do you still have the disks? Use ceph-objectstore-tool to export the 
>>> affected PGs manually and inject them into another OSD. 
>>> 
>>> 
>>> Paul 
>>> 
>>>> Then I added the 3 OSDs to crushmap. 
>>>> 
>>>> Regards 
>>>> Thomas 
>>>> 
>>>> Am 20.09.2019 um 08:19 schrieb Ashley Merrick: 
>>>>> Your need to fix this first. 
>>>>> 
>>>>>  pgs: 0.056% pgs unknown 
>>>>>   0.553% pgs not active 
>>>>> 
>>>>> The back filling will cause slow I/O, but having pgs unknown and not 
>>>>> active will cause I/O blocking which your seeing with the VM booting. 
>>>>> 
>>>>> Seems you have 4 OSD's down, if you get them back online you should be 
>>>>> able to get all the PG's online. 
>>>>> 
>>>>> 
>>>>>  On Fri, 20 Sep 2019 14:14:01 +0800 *Thomas 
>>>>> <mailto:74cmo...@gmail.com>* 
>>>>> wrote  
>>>>> 
>>>>>  Hi, 
>>>>> 
>>>>>  here I describe 1 of the 2 major issues I'm currently facing in my 8 
>>>>>  node ceph cluster (2x MDS, 6x ODS). 
>>>>> 
>>>>>  The issue is that I cannot start any virtual machine KVM or 
>>>>> container 
>>>>>  LXC; the boot process just hangs after a few seconds. 
>>>>>  All these KVMs and LXCs have in common that their virtual disks 
>>>>>  reside 
>>>>>  in the same pool: hdd 
>>>>> 
>>>>>  This pool hdd is relatively small compared to the largest pool: 
>>>>>  hdb_backup 
>>>>>  root@ld3955:~# rados df 
>>>>>  POOL_NAME  USED  OBJECTS CLONESCOPIES 
>>>>>  MISSING_ON_PRIMARY 
>>>>>  UNFOUND DEGRADEDRD_OPS   RDWR_OPS  WR USED COMPR 
>>>>>  UNDER COMPR 
>>>>>  backup  0 B0  0 
>>>>>  0 
>>>>>  0   00 0  0 B 0 0 B0 
>>>>>  B 0 B 
>>>>>  hdb_backup  589 TiB 51262212  0 
>>>>>  153786636 
>>>>>  0   0   124895  12266095  4.3 TiB 247132863 463 TiB0 
>>>>>  B 0 B 
>>>>>  hdd 3.2 TiB   281884   6568 
>>>>>  845652 
>>>>>  0   0 1658 275277357   16 TiB 208213922  10 TiB0 
>>>>>  B 0 B 
>>>>>  pve_cephfs_data 955

[ceph-users] Re: How to reduce or control memory usage during recovery?

[Ashley Merrick]

I'm not aware of any memory settings that control rebuild memory usage.

You are running very under on RAM, have you tried adding more swap or adjusting 
/proc/sys/vm/swappiness




 On Fri, 20 Sep 2019 20:41:09 +0800 Amudhan P  
wrote 


Hi,

I am using ceph mimic in a small test setup using the below configuration.



OS: ubuntu 18.04



1 node running (mon,mds,mgr) + 4 core cpu and 4GB RAM and 1 Gb lan

3 nodes each having 2 osd's, disks are 2TB + 2 core cpu and 4G RAM 

 and 1 Gb lan

1 node acting as cephfs client 

+ 2 core cpu and 4G RAM 

 and 1 Gb lan



configured cephfs_metadata_pool (3 replica) and cephfs_data_pool erasure 2+1.



When running a script doing multiple folders creation ceph started throwing 
error late IO due to high metadata workload.

once after folder creation complete PG's degraded and I am waiting for PG to 
complete recovery but my OSD's starting to crash due to OOM and restarting 
after some time.



Now my question is I can wait for recovery to complete but how do I stop OOM 
and OSD crash? basically want to know the way to control memory usage during 
recovery and make it stable.



I have also set very low PG metadata_pool 8 and data_pool 16.



I have already set "mon osd memory target to 1Gb" and I have set max-backfill 
from 1 to 8.



Attached msg from "kern.log" from one of the node and snippet of error msg in 
this mail.



-error msg snippet --

-bash: fork: Cannot allocate memory



Sep 18 19:01:57 test-node1 kernel: [341246.765644] msgr-worker-0 invoked 
oom-killer: gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, 
oom_score_adj=0
Sep 18 19:02:00 test-node1 kernel: [341246.765645] msgr-worker-0 cpuset=/ 
mems_allowed=0
Sep 18 19:02:00 test-node1 kernel: [341246.765650] CPU: 1 PID: 1737 Comm: 
msgr-worker-0 Not tainted 4.15.0-45-generic #48-Ubuntu



Sep 18 19:02:02 test-node1 kernel: [341246.765833] Out of memory: Kill process 
1727 (ceph-osd) score 489 or sacrifice child
Sep 18 19:02:03 test-node1 kernel: [341246.765919] Killed process 1727 
(ceph-osd) total-vm:3483844kB, anon-rss:1992708kB, file-rss:0kB, shmem-rss:0kB
Sep 18 19:02:03 test-node1 kernel: [341246.899395] oom_reaper: reaped process 
1727 (ceph-osd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 18 22:09:57 test-node1 kernel: [352529.433155] perf: interrupt took too 
long (4965 > 4938), lowering kernel.perf_event_max_sample_rate to 40250



regards

Amudhan


___
ceph-users mailing list -- mailto:ceph-users@ceph.io 
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cannot start virtual machines KVM / LXC

[Ashley Merrick]

Your need to fix this first.



    pgs: 0.056% pgs unknown 

 0.553% pgs not active



The back filling will cause slow I/O, but having pgs unknown and not active 
will cause I/O blocking which your seeing with the VM booting.



Seems you have 4 OSD's down, if you get them back online you should be able to 
get all the PG's online.



 On Fri, 20 Sep 2019 14:14:01 +0800 Thomas <74cmo...@gmail.com> wrote 


Hi, 
 
here I describe 1 of the 2 major issues I'm currently facing in my 8 
node ceph cluster (2x MDS, 6x ODS). 
 
The issue is that I cannot start any virtual machine KVM or container 
LXC; the boot process just hangs after a few seconds. 
All these KVMs and LXCs have in common that their virtual disks reside 
in the same pool: hdd 
 
This pool hdd is relatively small compared to the largest pool: hdb_backup 
root@ld3955:~# rados df 
POOL_NAME  USED  OBJECTS CLONES    COPIES MISSING_ON_PRIMARY 
UNFOUND DEGRADED    RD_OPS   RD    WR_OPS  WR USED COMPR UNDER COMPR 
backup  0 B    0  0 0  
0   0    0 0  0 B 0 0 B    0 
B 0 B 
hdb_backup  589 TiB 51262212  0 153786636  
0   0   124895  12266095  4.3 TiB 247132863 463 TiB    0 
B 0 B 
hdd 3.2 TiB   281884   6568    845652  
0   0 1658 275277357   16 TiB 208213922  10 TiB    0 
B 0 B 
pve_cephfs_data 955 GiB    91832  0    275496  
0   0 3038  2103 1021 MiB    102170 318 GiB    0 
B 0 B 
pve_cephfs_metadata 486 MiB   62  0   186  
0   0    7   860  1.4 GiB 12393 166 MiB    0 
B 0 B 
 
total_objects    51635990 
total_used   597 TiB 
total_avail  522 TiB 
total_space  1.1 PiB 
 
This is the current health status of the ceph cluster: 
  cluster: 
    id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae 
    health: HEALTH_ERR 
    1 filesystem is degraded 
    1 MDSs report slow metadata IOs 
    1 backfillfull osd(s) 
    87 nearfull osd(s) 
    1 pool(s) backfillfull 
    Reduced data availability: 54 pgs inactive, 47 pgs peering, 
1 pg stale 
    Degraded data redundancy: 129598/154907946 objects degraded 
(0.084%), 33 pgs degraded, 33 pgs undersized 
    Degraded data redundancy (low space): 322 pgs backfill_toofull 
    1 subtrees have overcommitted pool target_size_bytes 
    1 subtrees have overcommitted pool target_size_ratio 
    1 pools have too many placement groups 
    21 slow requests are blocked > 32 sec 
 
  services: 
    mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 14h) 
    mgr: ld5507(active, since 16h), standbys: ld5506, ld5505 
    mds: pve_cephfs:1/1 {0=ld3955=up:replay} 1 up:standby 
    osd: 360 osds: 356 up, 356 in; 382 remapped pgs 
 
  data: 
    pools:   5 pools, 8868 pgs 
    objects: 51.64M objects, 197 TiB 
    usage:   597 TiB used, 522 TiB / 1.1 PiB avail 
    pgs: 0.056% pgs unknown 
 0.553% pgs not active 
 129598/154907946 objects degraded (0.084%) 
 229/154907946 objects misplaced (1.427%) 
 8458 active+clean 
 298  active+remapped+backfill_toofull 
 29   remapped+peering 
 24   active+undersized+degraded+remapped+backfill_toofull 
 22   active+remapped+backfill_wait 
 17   peering 
 5    unknown 
 5    active+recovery_wait+undersized+degraded+remapped 
 3    active+undersized+degraded+remapped+backfill_wait 
 2    activating+remapped 
 1    active+clean+remapped 
 1    stale+peering 
 1    active+remapped+backfilling 
 1    active+recovering+undersized+remapped 
 1    active+recovery_wait+degraded 
 
  io: 
    client:   9.2 KiB/s wr, 0 op/s rd, 1 op/s wr 
 
I believe the cluster is busy with rebalancing pool hdb_backup. 
I set the balance mode upmap recently after the 589TB data was written. 
root@ld3955:~# ceph balancer status 
{ 
    "active": true, 
    "plans": [], 
    "mode": "upmap" 
} 
 
 
In order to resolve the issue with pool hdd I started some investigation. 
First step was to install drivers for the NIC provided Mellanox. 
Then I configured some kernel parameters recommended 
 by Mellanox. 
 
However this didn't fix the issue. 
In my opinion I must get rid of all "slow requests are blocked". 
 
When I check the output of ceph health detail any OSD listed under 
REQUEST_SLOW points to an OSD that belongs to pool hdd. 
This means none of the disks belonging to pool hdb_backup is showing a 
comparable behaviour. 
 
Then I checked the running processes on the different OSD nodes; I use 
tool "glanc

[ceph-users] 14.2.4 Packages Avaliable

[Ashley Merrick]

Have just noticed their is packages available for 14.2.4..

I know with the whole 14.2.3 release and the notes not going out to a good day 
or so later.. but this is not long after the 14.2.3 release..?

Was this release even meant to have come out? Makes it difficult for people 
installing a new node if they can't reply on the current "stable" packages that 
apt/yum will give them.___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Activate Cache Tier on Running Pools

[Ashley Merrick]

I hope the data your running the CEPH server isn't important if your looking to 
run a Cache tier with just 2 SSDS / Replication of 2.

If your cache tier fails, you basically corrupt most data on the pool below.

Also as Wido said, as much as you may get it to work, I don't think it will 
give you the performance you expected, can you not create a separate small SSD 
pool as a scratch disk for the VM's for when they are doing heavy temp I/O?

Leaving your current data and setup untouched.




 On Mon, 16 Sep 2019 18:21:47 +0800 Eikermann, Robert 
 wrote 




We have terrible IO performance when multiple VMs do some file IO. Mainly do 
some java compilation on that servers. If we have 2 parallel jobs everything is 
fine, but having 10 jobs we see the warning
 “HEALTH_WARN X requests are blocked > 32 sec; Y osds have slow requests”. I 
have two enterprise SSDs which gained good results in ceph tested with fio. 
They are too small to have a separate “ssd_vms” pool and advertise it in 
openstack as a separate storage
 backend

 

--
 -
Robert Eikermann M.Sc.RWTH   | Software Engineering

Lehrstuhl für Software Engineering   | RWTH Aachen University

Ahornstr. 55, 52074 Aachen, Germany      | http://www.se-rwth.de/

Phone ++49 241 80-21306 / Fax -22218 |


 

Von: Wido den Hollander [mailto:mailto:w...@42on.com] 
 Gesendet: Montag, 16. September 2019 11:52
 An: Eikermann, Robert ; mailto:ceph-users@ceph.io
 Betreff: Re: [ceph-users] Activate Cache Tier on Running Pools


 

 

On 9/16/19 11:36 AM, Eikermann, Robert wrote:


Hi,

 

I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d like to 
enable writeback caching tier, like described here: 
https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ .

 


Can you explain why? The cache tiering has some serious flaws and can even 
decrease performance instead of improve it.

What are you trying to solve?

Wido

Should it be possible to do that on a running pool? I tried to do so and 
immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly 
filesystems. No errors were shown in ceph (but also no traffic arrived
 after enabling the cache tier). Removing the cache tier , rebooting the VMs 
and doing a filesystemcheck repaired everything.

 

Best

Robert

 




___

ceph-users mailing list -- mailto:ceph-users@ceph.io

To unsubscribe send an email to mailto:ceph-users-le...@ceph.io




___
ceph-users mailing list -- mailto:ceph-users@ceph.io 
To unsubscribe send an email to mailto:ceph-users-le...@ceph.io___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Activate Cache Tier on Running Pools

[Ashley Merrick]

Have you checked that the user/keys that your VMs are connecting to have access 
rights to the cache pool?





 On Mon, 16 Sep 2019 17:36:38 +0800 Eikermann, Robert 
 wrote 





Hi,

 

I’m using Ceph in combination with Openstack. For the “VMs” Pool I’d like to 
enable writeback caching tier, like described here: 
https://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ .

 

Should it be possible to do that on a running pool? I tried to do so and 
immediately all VMs (Linux Ubuntu OS) running on Ceph disks got readonly 
filesystems. No errors were shown in ceph (but also no traffic arrived
 after enabling the cache tier). Removing the cache tier , rebooting the VMs 
and doing a filesystemcheck repaired everything.

 

Best

Robert

 



___

ceph-users mailing list -- ceph-users@ceph.io 

To unsubscribe send an email to ceph-users-le...@ceph.io___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Host failure trigger " Cannot allocate memory"

[Ashley Merrick]

What's specs ate the machines?


Recovery work will use more memory the general clean operation and looks like 
your maxing out the available memory on the machines during CEPH trying to 
recover.




 On Tue, 10 Sep 2019 18:10:50 +0800 amudha...@gmail.com wrote 


I have also found below error in dmesg.



[332884.028810] systemd-journald[6240]: Failed to parse kernel command line, 
ignoring: Cannot allocate memory
[332885.054147] systemd-journald[6240]: Out of memory.
[332894.844765] systemd[1]: systemd-journald.service: Main process exited, 
code=exited, status=1/FAILURE
[332897.199736] systemd[1]: systemd-journald.service: Failed with result 
'exit-code'.
[332906.503076] systemd[1]: Failed to start Journal Service.
[332937.909198] systemd[1]: ceph-crash.service: Main process exited, 
code=exited, status=1/FAILURE
[332939.308341] systemd[1]: ceph-crash.service: Failed with result 'exit-code'.
[332949.545907] systemd[1]: systemd-journald.service: Service has no hold-off 
time, scheduling restart.
[332949.546631] systemd[1]: systemd-journald.service: Scheduled restart job, 
restart counter is at 7.
[332949.546781] systemd[1]: Stopped Journal Service.
[332949.566402] systemd[1]: Starting Journal Service...
[332950.190332] systemd[1]: ceph-osd@1.service: Main process exited, 
code=killed, status=6/ABRT
[332950.190477] systemd[1]: ceph-osd@1.service: Failed with result 'signal'.
[332950.842297] systemd-journald[6249]: File 
/var/log/journal/8f2559099bf54865adc95e5340d04447/system.journal corrupted or 
uncleanly shut down, renaming and replacing.
[332951.019531] systemd[1]: Started Journal Service.



On Tue, Sep 10, 2019 at 3:04 PM Amudhan P  wrote:

Hi,


I am using ceph version 13.2.6 (mimic) on test setup trying with cephfs.


My current setup:

3 nodes, 1 node contain two bricks and other 2 nodes contain single brick each.


Volume is a 3 replica, I am trying to simulate node failure.


I powered down one host and started getting msg in other systems when running 
any command
"-bash: fork: Cannot allocate memory" and system not responding to commands.


what could be the reason for this?
at this stage, I could able to read some of the data stored in the volume and 
some just waiting for IO.


output from "sudo ceph -s"
  cluster:
    id:     7c138e13-7b98-4309-b591-d4091a1742b4
    health: HEALTH_WARN
            1 osds down
            2 hosts (3 osds) down
            Degraded data redundancy: 5313488/7970232 objects degraded 
(66.667%), 64 pgs degraded

  services:
    mon: 1 daemons, quorum mon01
    mgr: mon01(active)
    mds: cephfs-tst-1/1/1 up  {0=mon01=up:active}
    osd: 4 osds: 1 up, 2 in

  data:
    pools:   2 pools, 64 pgs
    objects: 2.66 M objects, 206 GiB
    usage:   421 GiB used, 3.2 TiB / 3.6 TiB avail
    pgs:     5313488/7970232 objects degraded (66.667%)
             64 active+undersized+degraded

  io:
    client:   79 MiB/s rd, 24 op/s rd, 0 op/s wr



output from : sudo ceph osd df
ID CLASS WEIGHT  REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS
 0   hdd 1.81940        0     0 B     0 B     0 B     0    0   0
 3   hdd 1.81940        0     0 B     0 B     0 B     0    0   0
 1   hdd 1.81940  1.0 1.8 TiB 211 GiB 1.6 TiB 11.34 1.00   0
 2   hdd 1.81940  1.0 1.8 TiB 210 GiB 1.6 TiB 11.28 1.00  64
                    TOTAL 3.6 TiB 421 GiB 3.2 TiB 11.31
MIN/MAX VAR: 1.00/1.00  STDDEV: 0.03



regards
Amudhan

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: disk failure

[Ashley Merrick]

I would suggest checking the logs and seeing the exact reason its being marked 
out.

If the disk is being hit hard and their is heavy I/O delays then Ceph may see 
that as a delayed reply outside of the set windows and mark as out.

There is some variables that can be changed to give an OSD more time to reply 
to a heartbeat, but I would definitely suggest checking the OSD log at the time 
of the disk being marked out to see exactly what's going on.

As the last thing you want to do is just patch an actually issue if there is 
one. 


 On Fri, 06 Sep 2019 02:11:06 +0800 solarflo...@gmail.com wrote 


no, I mean ceph sees it as a failure and marks it out for a while



On Thu, Sep 5, 2019 at 11:00 AM Ashley Merrick  wrote:

Is your HD actually failing and vanishing from the OS and then coming back 
shortly?

Or do you just mean your OSD is crashing and then restarting it self shortly 
later?



 On Fri, 06 Sep 2019 01:55:25 +0800 solarflo...@gmail.com wrote 


One of the things i've come to notice is when HDD drives fail, they often 
recover in a short time and get added back to the cluster.  This causes the 
data to rebalance back and forth, and if I set the noout flag I get a health 
warning.  Is there a better way to avoid this?




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: disk failure

[Ashley Merrick]

Is your HD actually failing and vanishing from the OS and then coming back 
shortly?

Or do you just mean your OSD is crashing and then restarting it self shortly 
later? 


 On Fri, 06 Sep 2019 01:55:25 +0800 solarflo...@gmail.com wrote 


One of the things i've come to notice is when HDD drives fail, they often 
recover in a short time and get added back to the cluster.  This causes the 
data to rebalance back and forth, and if I set the noout flag I get a health 
warning.  Is there a better way to avoid this?




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph mons stuck in electing state

[Ashley Merrick]

What change did you make in ceph.conf

Id check that hasn't caused an issue first. 


 On Tue, 27 Aug 2019 04:37:15 +0800 nkern...@gmail.com wrote 


Hello,


I have an old ceph 0.94.10 cluster that had 10 storage nodes with one extra 
management node used for running commands on the cluster. Over time we'd had 
some hardware failures on some of the storage nodes, so we're down to 6, with 
ceph-mon running on the management server and 4 of the storage nodes. We 
attempted deploying a ceph.conf change and restarted ceph-mon and ceph-osd 
services, but the cluster went down on us. We found all the ceph-mons are stuck 
in the electing state, I can't get any response from any ceph commands but I 
found I can contact the daemon directly and get this information (hostnames 
removed for privacy reasons):


root@:~# ceph daemon mon. mon_status
{
    "name": "",
    "rank": 0,
    "state": "electing",
    "election_epoch": 4327,
    "quorum": [],
    "outside_quorum": [],
    "extra_probe_peers": [],
    "sync_provider": [],
    "monmap": {
        "epoch": 10,
        "fsid": "69611c75-200f-4861-8709-8a0adc64a1c9",
        "modified": "2019-08-23 08:20:57.620147",
        "created": "0.00",
        "mons": [
            {
                "rank": 0,
                "name": "",
                "addr": "[fdc4:8570:e14c:132d::15]:6789\/0"
            },
            {
                "rank": 1,
                "name": "",
                "addr": "[fdc4:8570:e14c:132d::16]:6789\/0"
            },
            {
                "rank": 2,
                "name": "",
                "addr": "[fdc4:8570:e14c:132d::28]:6789\/0"
            },
            {
                "rank": 3,
                "name": "",
                "addr": "[fdc4:8570:e14c:132d::29]:6789\/0"
            },
            {
                "rank": 4,
                "name": "",
                "addr": "[fdc4:8570:e14c:132d::151]:6789\/0"
            }
        ]
    }
}





Is there any way to force the cluster back into a quorum even if it's just one 
mon running to start it up? I've tried exporting the mgmt's monmap and 
injecting it into the other nodes, but it didn't make any difference.


Thanks!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph mons stuck in electing state

[Ashley Merrick]

What change did you make in your ceph.conf?

Id say it be a good idea to check and make sure that hasn't caused the issue. 

,Ashley


 On Tue, 27 Aug 2019 04:37:15 +0800 nkern...@gmail.com wrote 


Hello,


I have an old ceph 0.94.10 cluster that had 10 storage nodes with one extra 
management node used for running commands on the cluster. Over time we'd had 
some hardware failures on some of the storage nodes, so we're down to 6, with 
ceph-mon running on the management server and 4 of the storage nodes. We 
attempted deploying a ceph.conf change and restarted ceph-mon and ceph-osd 
services, but the cluster went down on us. We found all the ceph-mons are stuck 
in the electing state, I can't get any response from any ceph commands but I 
found I can contact the daemon directly and get this information (hostnames 
removed for privacy reasons):


root@:~# ceph daemon mon. mon_status
{
    "name": "",
    "rank": 0,
    "state": "electing",
    "election_epoch": 4327,
    "quorum": [],
    "outside_quorum": [],
    "extra_probe_peers": [],
    "sync_provider": [],
    "monmap": {
        "epoch": 10,
        "fsid": "69611c75-200f-4861-8709-8a0adc64a1c9",
        "modified": "2019-08-23 08:20:57.620147",
        "created": "0.00",
        "mons": [
            {
                "rank": 0,
                "name": "",
                "addr": "[fdc4:8570:e14c:132d::15]:6789\/0"
            },
            {
                "rank": 1,
                "name": "",
                "addr": "[fdc4:8570:e14c:132d::16]:6789\/0"
            },
            {
                "rank": 2,
                "name": "",
                "addr": "[fdc4:8570:e14c:132d::28]:6789\/0"
            },
            {
                "rank": 3,
                "name": "",
                "addr": "[fdc4:8570:e14c:132d::29]:6789\/0"
            },
            {
                "rank": 4,
                "name": "",
                "addr": "[fdc4:8570:e14c:132d::151]:6789\/0"
            }
        ]
    }
}





Is there any way to force the cluster back into a quorum even if it's just one 
mon running to start it up? I've tried exporting the mgmt's monmap and 
injecting it into the other nodes, but it didn't make any difference.


Thanks!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] EC Compression

[Ashley Merrick]

I have an EC RBD pool I want to add compression on.

I have the meta pool and the data pool, do I need to enable compression on both 
for it to function correctly or only on one pool?

Thanks___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io