With the balancer in crush-compat mode each OSDs has around 100 PGs.
But during the purge of the host the last remaining OSD gets assigned
too many PGs, my tests with the osdmaptool show 266 PGs, and the same
happens after the first OSD is up after the rebuild. In the logs I
counted 751 uni
Hi,
You have 2 or 4 osd/disk?
no such thing, only one OSD per HDD (with DB on SSD).
Zitat von "Szabo, Istvan (Agoda)" :
You have 2 or 4 osd/disk?
Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.c
Just to update this thread, apparently you were right, we did hit the
limit of mon_max_pg_per_osd * osd_max_pg_per_osd_hard_ratio (250 * 3 =
750), this was found in the logs:
2022-04-06 14:24:55.256 7f8bb5a0e700 1 osd.8 43377
maybe_wait_for_max_pg withhold creation of pg 75.56s16: 750 >= 7
Thanks, I’ll take a closer look at that.
Zitat von Josh Baergen :
Hi Eugen,
how did you determine how many PGs were assigned to the OSDs? I looked
at one of the OSD's logs and checked how many times each PG chunk of
the affected pool was logged during startup. I got around 580 unique
entries,
Hi,
It could be, yes. I've seen a case on a test cluster where thousands
of PGs were assigned to a single OSD even when the steady state was
far fewer than that.
how did you determine how many PGs were assigned to the OSDs? I looked
at one of the OSD's logs and checked how many times each PG
Hi,
thanks for your explanation, Josh. I think In understand now how
mon_max_pg_per_osd could have an impact here. The default seems to be
250. Each OSD currently has around 100 PGs, is this a potential
bottleneck? In my test cluster I have around 150 PGs per OSD and
couldn't reproduce it
dy
exists?) and the recreated osd was a orphant...
Greetings,
Dominique.
Van: Eugen Block
Verzonden: donderdag 7 april 2022 8:15
Aan: Josh Baergen
CC: ceph-users@ceph.io
Onderwerp: [ceph-users] Re: Ceph PGs stuck inactive after rebuild node
Basically, th
Basically, these are the steps to remove all OSDs from that host (OSDs
are not "replaced" so they aren't marked "destroyed") [1]:
1) Call 'ceph osd out $id'
2) Call systemctl stop ceph-osd@$id
3) ceph osd purge $id --yes-i-really-mean-it
4) call ceph-volume lvm zap --osd-id $id --destroy
After
Thanks for the comments, I'll get the log files to see if there's any
hint. Getting the PGs in an active state is one thing, I'm sure
multiple approaches would have worked. The main question is why this
happens, we have 19 hosts to rebuild and can't risk the application
outage everytime.
Something worth a try before restarting an OSD in situations like this:
ceph osd down 9
This marks the OSD down in the osdmap, but doesn’t touch the daemon.
Typically the subject OSD will see this and tell the mons “I’m not dead yet!”
and repeer, which sometimes suffices to clear glitch
Thanks everyone!
/Zakhar
On Wed, Apr 6, 2022 at 6:24 PM Josh Baergen
wrote:
> For future reference, "ceph pg repeer " might have helped here.
>
> Was the PG stuck in the "activating" state? If so, I wonder if you
> temporarily exceeded mon_max_pg_per_osd on some OSDs when rebuilding
> your host
Sure, from the output of 'ceph pg map ' you get the acting set,
for example:
cephadmin:~ # ceph pg map 32.18
osdmap e7198 pg 32.18 (32.18) -> up [9,2,1] acting [9,2,1]
Then I restarted OSD.9 and the inactive PG became active again.
I remember this has been discussed a couple of times in the pa
Hi Eugen,
Can you please elaborate on what you mean by "restarting the primary PG"?
Best regards,
Zakhar
On Wed, Apr 6, 2022 at 5:15 PM Eugen Block wrote:
> Update: Restarting the primary PG helped to bring the PGs back to
> active state. Consider this thread closed.
>
>
> Zitat von Eugen Bloc
Update: Restarting the primary PG helped to bring the PGs back to
active state. Consider this thread closed.
Zitat von Eugen Block :
Hi all,
I have a strange situation here, a Nautilus cluster with two DCs,
the main pool is an EC pool with k7 m11, min_size = 8 (failure
domain host). We
14 matches
Mail list logo
