[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

2022-10-15 Thread Frank Schilder
rank Schilder; ceph-users Subject: Re: [ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus Just a datapoint - we upgraded several large Mimic-born clusters straight to 15.2.12 with the quick fsck disabled in ceph.conf, then did require-osd-release, and finally

[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

2022-09-26 Thread Tyler Stachecki
Just a datapoint - we upgraded several large Mimic-born clusters straight to 15.2.12 with the quick fsck disabled in ceph.conf, then did require-osd-release, and finally did the omap conversion offline after the cluster was upgraded using the bluestore tool while the OSDs were down (all done in

[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

2022-09-26 Thread Marc
Hi Frank, Thank you very much for this! :) > > we just completed a third upgrade test. There are 2 ways to convert the > OSDs: > > A) convert along with the upgrade (quick-fix-on-start=true) > B) convert after setting require-osd-release=octopus (quick-fix-on- > start=false until

[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

2022-09-13 Thread Boris Behrens
back log of the OSDs. >> > >> > If its not that, I don't know what else to look for. >> > >> > Best regards, >> > = >> > Frank Schilder >> > AIT Risø Campus >> > Bygning 109, rum S14 >> > >> >

[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

2022-09-13 Thread Wesley Dillingham
gt; > > > ____________ > > From: Boris Behrens > > Sent: 13 September 2022 12:58:19 > > To: Frank Schilder > > Cc: ceph-users@ceph.io > > Subject: Re: [ceph-users] laggy OSDs and staling krbd IO after upgrade > > from nautilus to oc

[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

2022-09-13 Thread Boris Behrens
gt; Sent: 13 September 2022 12:58:19 > To: Frank Schilder > Cc: ceph-users@ceph.io > Subject: Re: [ceph-users] laggy OSDs and staling krbd IO after upgrade > from nautilus to octopus > > Hi Frank, > we converted the OSDs directly on the upgrade. > > 1. installing new ceph v

[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

2022-09-13 Thread Marc
> > It might be possible that converting OSDs before setting require-osd- > release=octopus leads to a broken state of the converted OSDs. I could > not yet find a way out of this situation. We will soon perform a third > upgrade test to test this hypothesis. > So with upgrading one should

[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

2022-09-13 Thread Boris Behrens
Hi Frank, we converted the OSDs directly on the upgrade. 1. installing new ceph versions 2. restart all OSD daemons 3. wait some time (took around 5-20 minutes) 4. all OSDs were online again. So I would expect, that the OSDs are all upgraded correctly. I also checked when the trimming happens,

[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

2022-09-13 Thread Boris Behrens
I checked the cluster for other snaptrim operations and they happen all over the place, so for me it looks like they just happend to be done when the issue occured, but were not the driving factor. Am Di., 13. Sept. 2022 um 12:04 Uhr schrieb Boris Behrens : > Because someone mentioned that the

[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

2022-09-13 Thread Boris Behrens
Because someone mentioned that the attachments did not went through I created pastebin links: monlog: https://pastebin.com/jiNPUrtL osdlog: https://pastebin.com/dxqXgqDz Am Di., 13. Sept. 2022 um 11:43 Uhr schrieb Boris Behrens : > Hi, I need you help really bad. > > we are currently