Not really, as unfortunately the cache eviction fails for some rbd
objects that still hace some "lock", right now we need to understand
why the eviction fails on these objects, and find a solution to have
the cache eviction fully working. I will provide more information
later on.
If you have any p
Hi,
great that you found a solution. Maybe that also helps to get rid of
the cache-tier entirely?
Zitat von Cedric :
Hello,
Sorry for the late reply, so yes we finally find a solution, which
was to split apart the cache_pool on dedicated OSD. It had the
effect to clear off slow ops and
Hello,
Sorry for the late reply, so yes we finally find a solution, which was to split
apart the cache_pool on dedicated OSD. It had the effect to clear off slow ops
and allow the cluster to serves clients again, after 5 days of lock down,
hopefully the majority of VM resume well, thanks to the
Hi,
thanks for the context. Was there any progress over the weekend? The
hanging commands seem to be MGR related, and there's only one in your
cluster according to your output. Can you deploy a second one
manually, then adopt it with cephadm? Can you add 'ceph versions' as
well?
Zitat
Hello Eugen,
We used to have cache tiering (hdd+ssd) for openstack nova/glance in the past
before we move to nvme hardware. But we were not able to evict all objects
because it required to shutdown all virtual instances and then do the eviction.
So we decided to set the cache mode to "proxy" an
Hi,
A bit of history might help to understand why we have the cache tier.
We run openstack on top ceph since many years now (started with mimic, then an
upgrade to nautilus (years 2 ago) and today and upgrade to pacific). At the
beginning of the setup, we used to have a mix of hdd+ssd devices i
Hi,
Have you already tried to set the primary PG out and wait for the
backfill to finish?
Of course I meant the primary OSD for that PG, I hope that was clear. ;-)
We are thinking about the use of "ceph pg_mark_unfound_lost revert"
I'm not a developer, but how I read the code [2] is that s
On Thu, Feb 22, 2024 at 12:37 PM Eugen Block wrote:
> You haven't told yet if you changed the hit_set_count to 0.
Not yet, we will give it a try ASAP
> Have you already tried to set the primary PG out and wait for the
> backfill to finish?
No, we will try also
> And another question, are all s
We are thinking about the use of "ceph pg_mark_unfound_lost revert"
action, but we wonder if there is a risk of data loss.
You don't seem to have unfound objects so I don't think that command
would make sense.
You haven't told yet if you changed the hit_set_count to 0.
Have you already tried
Yes the osd_scrub_invalid_stats is set to true.
We are thinking about the use of "ceph pg_mark_unfound_lost revert"
action, but we wonder if there is a risk of data loss.
On Thu, Feb 22, 2024 at 11:50 AM Eugen Block wrote:
>
> I found a config to force scrub invalid PGs, what is your current
> s
I found a config to force scrub invalid PGs, what is your current
setting on that?
ceph config get osd osd_scrub_invalid_stats
true
The config reference states:
Forces extra scrub to fix stats marked as invalid.
But the default seems to be true, so I'd expect it's true in your case
as we
Thanks Eugen for the suggestion, yes we have tried, also repeering
concerned PGs, still the same issue.
Looking at the code it seems the split-mode message is triggered when
the PG as ""stats_invalid": true,", here is the result of a query:
"stats_invalid": true,
"dirty_stats_inva
Hm, I wonder if setting (and unsetting after a while) noscrub and
nodeep-scrub has any effect. Have you tried that?
Zitat von Cedric :
Update: we have run fsck and re-shard on all bluestore volume, seems
sharding were not applied.
Unfortunately scrubs and deep-scrubs are still stuck on PGs
Update: we have run fsck and re-shard on all bluestore volume, seems sharding
were not applied.
Unfortunately scrubs and deep-scrubs are still stuck on PGs of the pool that is
suffering the issue, but other PGs scrubs well.
The next step will be to remove the cache tier as suggested, but its no
Please don't drop the list from your response.
The first question coming to mind is, why do you have a cache-tier if
all your pools are on nvme decices anyway? I don't see any benefit here.
Did you try the suggested workaround and disable the cache-tier?
Zitat von Cedric :
Thanks Eugen, see
Hi,
some more details would be helpful, for example what's the pool size
of the cache pool? Did you issue a PG split before or during the
upgrade? This thread [1] deals with the same problem, the described
workaround was to set hit_set_count to 0 and disable the cache layer
until that is
16 matches
Mail list logo
