[ceph-users] Re: Trying to throttle global backfill
I received a few suggestions, and resolved my issue. Anthony D'Atri suggested mclock (newer than my nautilus version), adding "--osd_recovery_max_single_start 1” (didn’t seem to take), “osd_op_queue_cut_off=high” (which I didn’t get to checking), and pgremapper (from github). Pgremapper did the trick to cancel the backfill which had been initiated by an unfortunate OSD name-changing sequence. Big winner, achieved EXACTLY what I needed, which was to undo an unfortunate recalculation of placement groups. Before: 310842802/17308319325 objects misplaced (1.796%) Ran: pgremapper cancel-backfill --yes After: 421709/17308356309 objects misplaced (0.002%) The “before” scenario was causing over 10GiB/s of backfill traffic. The “after” scenario was a very cool 300-400MiB/s, entirely within the realm of sanity. The cluster is temporarily split between two datacenters, being physically lifted and shifted over a period of a month. Alex Gorbachev also suggested setting osd-recovery-sleep. That was probably the solution I was looking for to throttle backfill operations at the beginning, and I’ll be keeping that in my toolbox, as well. As always, I’m HUGELY appreciative of the community response. I learned a lot in the process, had an outage-inducing scenario rectified very quickly, and got back to work. Thanks so much! Happy to answer any followup questions and return the favor when I can. From: Rice, Christian Date: Wednesday, March 8, 2023 at 3:57 PM To: ceph-users Subject: [EXTERNAL] [ceph-users] Trying to throttle global backfill I have a large number of misplaced objects, and I have all osd settings to “1” already: sudo ceph tell osd.\* injectargs '--osd_max_backfills=1 --osd_recovery_max_active=1 --osd_recovery_op_priority=1' How can I slow it down even more? The cluster is too large, it’s impacting other network traffic 😉 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Trying to throttle global backfill
I have a large number of misplaced objects, and I have all osd settings to “1” already: sudo ceph tell osd.\* injectargs '--osd_max_backfills=1 --osd_recovery_max_active=1 --osd_recovery_op_priority=1' How can I slow it down even more? The cluster is too large, it’s impacting other network traffic 😉 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [EXTERNAL] Re: Renaming a ceph node
Hi all, so I used the rename-bucket option this morning for OSD node renames, and it was a success. Works great even on Luminous. I looked at the swap-bucket command and I felt it was leaning toward real data migration from old OSDs to new OSDs and I was a bit timid because there wasn’t a second host, just a name change. So when I looked at rename-bucket, it just seemed too simple not to try first. And I did, and it was. I renamed two host buckets (they housed discrete storage classes, so no dangerous loss of data redundancy), and even some rack buckets. sudo ceph osd crush rename-bucket and no data moved. I first thought I’d wait til the hosts were shutdown, but after I stopped the OSDs on the nodes, it seemed safe enough, and it was. In my particular case, I was moving migrating nodes to a new datacenter, just new names and IPs. I also moved a mon/mgr/rgw; and I merely had to delete the mon first, then reprovision it later. The rgw and mgr worked fine. I pre-edited ceph.conf to add the new networks, remove the old mon name, add the new mon name, so on startup it worked. I’m not a ceph admin but I play one on the tele. From: Eugen Block Date: Wednesday, February 15, 2023 at 12:44 AM To: ceph-users@ceph.io Subject: [EXTERNAL] [ceph-users] Re: Renaming a ceph node Hi, I haven't done this in a production cluster yet, only in small test clusters without data. But there's a rename-bucket command: ceph osd crush rename-bucket rename bucket to It should do exactly that, just rename the bucket within the crushmap without changing the ID. That command also exists in Luminous, I believe. To have an impression of the impact I'd recommend to test in a test cluster first. Regards, Eugen Zitat von Manuel Lausch : > Hi, > > yes you can rename a node without massive rebalancing. > > The following I tested with pacific. But I think this should work with > older versions as well. > You need to rename the node in the crushmap between shutting down the > node with the old name and starting it with the new name. > You only must keep the ID from the node in the crushmap! > > Regards > Manuel > > > On Mon, 13 Feb 2023 22:22:35 + > "Rice, Christian" wrote: > >> Can anyone please point me at a doc that explains the most >> efficient procedure to rename a ceph node WITHOUT causing a massive >> misplaced objects churn? >> >> When my node came up with a new name, it properly joined the >> cluster and owned the OSDs, but the original node with no devices >> remained. I expect this affected the crush map such that a large >> qty of objects got reshuffled. I want no object movement, if >> possible. >> >> BTW this old cluster is on luminous. ☹ >> >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Renaming a ceph node
Can anyone please point me at a doc that explains the most efficient procedure to rename a ceph node WITHOUT causing a massive misplaced objects churn? When my node came up with a new name, it properly joined the cluster and owned the OSDs, but the original node with no devices remained. I expect this affected the crush map such that a large qty of objects got reshuffled. I want no object movement, if possible. BTW this old cluster is on luminous. ☹ ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: [EXTERNAL] Laggy OSDs
we had issues with slow ops on ssd AND nvme; mostly fixed by raising aio-max-nr from 64K to 1M, eg "fs.aio-max-nr=1048576" if I remember correctly. On 3/29/22, 2:13 PM, "Alex Closs" wrote: Hey folks, We have a 16.2.7 cephadm cluster that's had slow ops and several (constantly changing) laggy PGs. The set of OSDs with slow ops seems to change at random, among all 6 OSD hosts in the cluster. All drives are enterprise SATA SSDs, by either Intel or Micron. We're still not ruling out a network issue, but wanted to troubleshoot from the Ceph side in case something broke there. ceph -s: health: HEALTH_WARN 3 slow ops, oldest one blocked for 246 sec, daemons [osd.124,osd.130,osd.141,osd.152,osd.27] have slow ops. services: mon: 5 daemons, quorum ceph-osd10,ceph-mon0,ceph-mon1,ceph-osd9,ceph-osd11 (age 28h) mgr: ceph-mon0.sckxhj(active, since 25m), standbys: ceph-osd10.xmdwfh, ceph-mon1.iogajr osd: 143 osds: 143 up (since 92m), 143 in (since 2w) rgw: 3 daemons active (3 hosts, 1 zones) data: pools: 26 pools, 3936 pgs objects: 33.14M objects, 144 TiB usage: 338 TiB used, 162 TiB / 500 TiB avail pgs: 3916 active+clean 19 active+clean+laggy 1 active+clean+scrubbing+deep io: client: 59 MiB/s rd, 98 MiB/s wr, 1.66k op/s rd, 1.68k op/s wr This is actually much faster than it's been for much of the past hour, it's been as low as 50 kb/s and dozens of iops in both directions (where the cluster typically does 300MB to a few gigs, and ~4k iops) The cluster has been on 16.2.7 since a few days after release without issue. The only recent change was an apt upgrade and reboot on the hosts (which was last Friday and didn't show signs of problems). Happy to provide logs, let me know what would be useful. Thanks for reading this wall :) -Alex MIT CSAIL he/they ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
