[ceph-users] Re: Upgrade tips from Luminous to Nautilus?
I believe it was nautilus that started requiring ms_bind_ipv4 = false ms_bind_ipv6 = true if you run IPv6 only clusters. OSDs prior to nautilus worked without these settings for us. I'm not sure if the port change (v1->v2) was part of luminous->nautilus as well, but you might want to check your firewalling (if any). Overall I recall luminous->nautilus a bit more rocky than usual (compared to the previous releases), but nothing too serious. Cheers, Nico Mark Schouten writes: > Hi, > > We've done our fair share of Ceph cluster upgrades since Hammer, and > have not seen much problems with them. I'm now at the point that I have > to upgrade a rather large cluster running Luminous and I would like to > hear from other users if they have experiences with issues I can expect > so that I can anticipate on them beforehand. > > As said, the cluster is running Luminous (12.2.13) and has the following > services active: > services: > mon: 3 daemons, quorum osdnode01,osdnode02,osdnode04 > mgr: osdnode01(active), standbys: osdnode02, osdnode03 > mds: pmrb-3/3/3 up > {0=osdnode06=up:active,1=osdnode08=up:active,2=osdnode07=up:active}, 1 > up:standby > osd: 116 osds: 116 up, 116 in; > rgw: 3 daemons active > > > Of the OSD's, we have 11 SSD's and 105 HDD. The capacity of the cluster > is 1.01PiB. > > We have 2 active crush-rules on 18 pools. All pools have a size of 3 there is > a total of 5760 pgs. > { > "rule_id": 1, > "rule_name": "hdd-data", > "ruleset": 1, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -10, > "item_name": "default~hdd" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "host" > }, > { > "op": "emit" > } > ] > }, > { > "rule_id": 2, > "rule_name": "ssd-data", > "ruleset": 2, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -21, > "item_name": "default~ssd" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "host" > }, > { > "op": "emit" > } > ] > } > > rbd -> crush_rule: hdd-data > .rgw.root -> crush_rule: hdd-data > default.rgw.control -> crush_rule: hdd-data > default.rgw.data.root -> crush_rule: ssd-data > default.rgw.gc -> crush_rule: ssd-data > default.rgw.log -> crush_rule: ssd-data > default.rgw.users.uid -> crush_rule: hdd-data > default.rgw.usage -> crush_rule: ssd-data > default.rgw.users.email -> crush_rule: hdd-data > default.rgw.users.keys -> crush_rule: hdd-data > default.rgw.meta -> crush_rule: hdd-data > default.rgw.buckets.index -> crush_rule: ssd-data > default.rgw.buckets.data -> crush_rule: hdd-data > default.rgw.users.swift -> crush_rule: hdd-data > default.rgw.buckets.non-ec -> crush_rule: ssd-data > DB0475 -> crush_rule: hdd-data > cephfs_pmrb_data -> crush_rule: hdd-data > cephfs_pmrb_metadata -> crush_rule: ssd-data > > > All but four clients are running Luminous, the four are running Jewel > (that needs upgrading before proceeding with this upgrade). > > So, normally, I would 'just' upgrade all Ceph packages on the > monitor-nodes and restart mons and then mgrs. > > After that, I would upgrade all Ceph packages on the OSD nodes and > restart all the OSD's. Then, after that, the MDSes and RGWs. Restarting > the OSD's will probably take a while. > > If anyone has a hint on what I should expect to cause some extra load or > waiting time, that would be great. > > Obviously, we have read > https://ceph.com/releases/v14-2-0-nautilus-released/ , but I'm looking > for real world experiences. > > Thanks! -- Sustainable and modern Infrastructures by ungleich.ch ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrade tips from Luminous to Nautilus?
Hi, I haven't had any issues upgrading from Luminous to Nautilus in multiple clusters (mostly RBD usage, but also CephFS), including a couple of different setups in my lab (RGW, iGW). Just recently I upgraded a customer cluster with around 220 OSDs, it was pretty straight forward without any hickups. The only thing sticking out is your multi-active mds setup which I didn't have yet. But I don't see any reason why that could be an issue. Regards, Eugen Zitat von Mark Schouten : Hi, We've done our fair share of Ceph cluster upgrades since Hammer, and have not seen much problems with them. I'm now at the point that I have to upgrade a rather large cluster running Luminous and I would like to hear from other users if they have experiences with issues I can expect so that I can anticipate on them beforehand. As said, the cluster is running Luminous (12.2.13) and has the following services active: services: mon: 3 daemons, quorum osdnode01,osdnode02,osdnode04 mgr: osdnode01(active), standbys: osdnode02, osdnode03 mds: pmrb-3/3/3 up {0=osdnode06=up:active,1=osdnode08=up:active,2=osdnode07=up:active}, 1 up:standby osd: 116 osds: 116 up, 116 in; rgw: 3 daemons active Of the OSD's, we have 11 SSD's and 105 HDD. The capacity of the cluster is 1.01PiB. We have 2 active crush-rules on 18 pools. All pools have a size of 3 there is a total of 5760 pgs. { "rule_id": 1, "rule_name": "hdd-data", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -10, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "ssd-data", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -21, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } rbd -> crush_rule: hdd-data .rgw.root -> crush_rule: hdd-data default.rgw.control -> crush_rule: hdd-data default.rgw.data.root -> crush_rule: ssd-data default.rgw.gc -> crush_rule: ssd-data default.rgw.log -> crush_rule: ssd-data default.rgw.users.uid -> crush_rule: hdd-data default.rgw.usage -> crush_rule: ssd-data default.rgw.users.email -> crush_rule: hdd-data default.rgw.users.keys -> crush_rule: hdd-data default.rgw.meta -> crush_rule: hdd-data default.rgw.buckets.index -> crush_rule: ssd-data default.rgw.buckets.data -> crush_rule: hdd-data default.rgw.users.swift -> crush_rule: hdd-data default.rgw.buckets.non-ec -> crush_rule: ssd-data DB0475 -> crush_rule: hdd-data cephfs_pmrb_data -> crush_rule: hdd-data cephfs_pmrb_metadata -> crush_rule: ssd-data All but four clients are running Luminous, the four are running Jewel (that needs upgrading before proceeding with this upgrade). So, normally, I would 'just' upgrade all Ceph packages on the monitor-nodes and restart mons and then mgrs. After that, I would upgrade all Ceph packages on the OSD nodes and restart all the OSD's. Then, after that, the MDSes and RGWs. Restarting the OSD's will probably take a while. If anyone has a hint on what I should expect to cause some extra load or waiting time, that would be great. Obviously, we have read https://ceph.com/releases/v14-2-0-nautilus-released/ , but I'm looking for real world experiences. Thanks! -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | i...@tuxis.nl ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrade tips from Luminous to Nautilus?
Mark, My main note was to make sure and NOT enable msgr2 until all OSDs are upgraded to Nautilus. I made that mistake early in the lab, and had to work hard to get it back together. Otherwise, pretty smooth process. -- Alex Gorbachev ISS/Storcium On Thu, Apr 29, 2021 at 4:58 AM Mark Schouten wrote: > Hi, > > We've done our fair share of Ceph cluster upgrades since Hammer, and > have not seen much problems with them. I'm now at the point that I have > to upgrade a rather large cluster running Luminous and I would like to > hear from other users if they have experiences with issues I can expect > so that I can anticipate on them beforehand. > > As said, the cluster is running Luminous (12.2.13) and has the following > services active: > services: > mon: 3 daemons, quorum osdnode01,osdnode02,osdnode04 > mgr: osdnode01(active), standbys: osdnode02, osdnode03 > mds: pmrb-3/3/3 up > {0=osdnode06=up:active,1=osdnode08=up:active,2=osdnode07=up:active}, 1 > up:standby > osd: 116 osds: 116 up, 116 in; > rgw: 3 daemons active > > > Of the OSD's, we have 11 SSD's and 105 HDD. The capacity of the cluster > is 1.01PiB. > > We have 2 active crush-rules on 18 pools. All pools have a size of 3 there > is a total of 5760 pgs. > { > "rule_id": 1, > "rule_name": "hdd-data", > "ruleset": 1, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -10, > "item_name": "default~hdd" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "host" > }, > { > "op": "emit" > } > ] > }, > { > "rule_id": 2, > "rule_name": "ssd-data", > "ruleset": 2, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -21, > "item_name": "default~ssd" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "host" > }, > { > "op": "emit" > } > ] > } > > rbd -> crush_rule: hdd-data > .rgw.root -> crush_rule: hdd-data > default.rgw.control -> crush_rule: hdd-data > default.rgw.data.root -> crush_rule: ssd-data > default.rgw.gc -> crush_rule: ssd-data > default.rgw.log -> crush_rule: ssd-data > default.rgw.users.uid -> crush_rule: hdd-data > default.rgw.usage -> crush_rule: ssd-data > default.rgw.users.email -> crush_rule: hdd-data > default.rgw.users.keys -> crush_rule: hdd-data > default.rgw.meta -> crush_rule: hdd-data > default.rgw.buckets.index -> crush_rule: ssd-data > default.rgw.buckets.data -> crush_rule: hdd-data > default.rgw.users.swift -> crush_rule: hdd-data > default.rgw.buckets.non-ec -> crush_rule: ssd-data > DB0475 -> crush_rule: hdd-data > cephfs_pmrb_data -> crush_rule: hdd-data > cephfs_pmrb_metadata -> crush_rule: ssd-data > > > All but four clients are running Luminous, the four are running Jewel > (that needs upgrading before proceeding with this upgrade). > > So, normally, I would 'just' upgrade all Ceph packages on the > monitor-nodes and restart mons and then mgrs. > > After that, I would upgrade all Ceph packages on the OSD nodes and > restart all the OSD's. Then, after that, the MDSes and RGWs. Restarting > the OSD's will probably take a while. > > If anyone has a hint on what I should expect to cause some extra load or > waiting time, that would be great. > > Obviously, we have read > https://ceph.com/releases/v14-2-0-nautilus-released/ , but I'm looking > for real world experiences. > > Thanks! > > > -- > Mark Schouten | Tuxis B.V. > KvK: 74698818 | http://www.tuxis.nl/ > T: +31 318 200208 | i...@tuxis.nl > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrade tips from Luminous to Nautilus?
On Thu, Apr 29, 2021 at 10:58:15AM +0200, Mark Schouten wrote: > We've done our fair share of Ceph cluster upgrades since Hammer, and > have not seen much problems with them. I'm now at the point that I have > to upgrade a rather large cluster running Luminous and I would like to > hear from other users if they have experiences with issues I can expect > so that I can anticipate on them beforehand. Thanks for the replies! Just one question though. Step one for me was to lower max_mds to one. Documentation seems to suggest that the cluster automagically moves > 1 mds'es to a standby state. However, nothing really happens. root@osdnode01:~# ceph fs get dadup_pmrb | grep max_mds max_mds 1 I still have three active ranks. Do I simply restart two of the MDS'es and force max_mds to one daemon, or is there a nicer way to move two mds'es from active to standby? Thanks again! -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | i...@tuxis.nl ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrade tips from Luminous to Nautilus?
I don't have a Luminous cluster at hand right now but setting max_mds to 1 already should take care and stop MDS services. Do you have have pinning enabled (subdirectories pinned to a specific MDS)? Zitat von Mark Schouten : On Thu, Apr 29, 2021 at 10:58:15AM +0200, Mark Schouten wrote: We've done our fair share of Ceph cluster upgrades since Hammer, and have not seen much problems with them. I'm now at the point that I have to upgrade a rather large cluster running Luminous and I would like to hear from other users if they have experiences with issues I can expect so that I can anticipate on them beforehand. Thanks for the replies! Just one question though. Step one for me was to lower max_mds to one. Documentation seems to suggest that the cluster automagically moves > 1 mds'es to a standby state. However, nothing really happens. root@osdnode01:~# ceph fs get dadup_pmrb | grep max_mds max_mds 1 I still have three active ranks. Do I simply restart two of the MDS'es and force max_mds to one daemon, or is there a nicer way to move two mds'es from active to standby? Thanks again! -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | i...@tuxis.nl ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrade tips from Luminous to Nautilus?
On Tue, May 11, 2021 at 08:47:26AM +, Eugen Block wrote: > I don't have a Luminous cluster at hand right now but setting max_mds to 1 > already should take care and stop MDS services. Do you have have pinning > enabled (subdirectories pinned to a specific MDS)? Not on this cluster, AFAIK. How can I check that? -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | i...@tuxis.nl ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrade tips from Luminous to Nautilus?
You can check the remaining active daemons if they have pinned subtrees: ceph daemon mds.daemon-a get subtrees | jq '.[] | [.dir.path, .auth_first]' [ "/dir1/subdir1", 6 ] [ "", 0 ] [ "~mds6", 6 ] If there's no pinning enabled it should probably look like this: [ "", 0 ] [ "~mds0", 0 ] [ "/dir2", 0 ] If you mount the cephfs root directory you can check the subdirectories with getfattr: host:~ # getfattr -n ceph.dir.pin /mnt/dir1/subdir1 getfattr: Removing leading '/' from absolute path names # file: mnt/dir1/subdir1 ceph.dir.pin="1" Does that help? Zitat von Mark Schouten : On Tue, May 11, 2021 at 08:47:26AM +, Eugen Block wrote: I don't have a Luminous cluster at hand right now but setting max_mds to 1 already should take care and stop MDS services. Do you have have pinning enabled (subdirectories pinned to a specific MDS)? Not on this cluster, AFAIK. How can I check that? -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | i...@tuxis.nl ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrade tips from Luminous to Nautilus?
On Tue, May 11, 2021 at 09:13:51AM +, Eugen Block wrote: > You can check the remaining active daemons if they have pinned subtrees: > > ceph daemon mds.daemon-a get subtrees | jq '.[] | [.dir.path, .auth_first]' This gives me output, a whole lot of lines. However, none of the directories are directories anyone has ever actively put pinning on... -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | i...@tuxis.nl ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrade tips from Luminous to Nautilus?
On Mon, May 10, 2021 at 10:46:45PM +0200, Mark Schouten wrote: > I still have three active ranks. Do I simply restart two of the MDS'es > and force max_mds to one daemon, or is there a nicer way to move two > mds'es from active to standby? It seems (documentation was no longer available, so ik took some searching) that I needed to run ceph mds deactivate $fs:$rank for every MDS I wanted to deactivate. That helped! -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | i...@tuxis.nl ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrade tips from Luminous to Nautilus?
On Fri, May 14, 2021 at 09:12:07PM +0200, Mark Schouten wrote: > It seems (documentation was no longer available, so ik took some > searching) that I needed to run ceph mds deactivate $fs:$rank for every > MDS I wanted to deactivate. Ok, so that helped for one of the MDS'es. Trying to deactivate another mds, it started to release inos and dns'es, until it was almost done. When it had a 50-ish left, a client started to complain and be blacklisted until I restarted the deactivated MDS.. So no joy yet, not deactivated until a single active MDS. Any ideas to achieve that are appreciated. Thanks! -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | i...@tuxis.nl ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Upgrade tips from Luminous to Nautilus?
Hi, Op 15-05-2021 om 22:17 schreef Mark Schouten: Ok, so that helped for one of the MDS'es. Trying to deactivate another mds, it started to release inos and dns'es, until it was almost done. When it had a 50-ish left, a client started to complain and be blacklisted until I restarted the deactivated MDS.. So no joy yet, not deactivated until a single active MDS. Any ideas to achieve that are appreciated. I've been able to deactivate the second MDS. Not sure why, but I had a lot of stray entries, which I've cleaned up by running a `find -ls` on the whole CephFS tree. That already was a few weeks ago, but I decided to just try and deactivate the second MDS, which now worked. So now I can finally do the upgrade. :) Thanks! -- Mark Schouten CTO, Tuxis B.V. | https://www.tuxis.nl/ | +31 318 200208 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io