Sure, I tried in screen session before but it did not reduce the queue.
Eventually managed to zero the queue by increasing these params
radosgw-admin gc process --include-all --debug-rgw=20
--rgw-gc-max-concurrent-io=20 --rgw-gc-max-trim-chunk=64
--rgw-gc-processor-max-time=7200
I think it was
You could try to do this in a screen session for a while.
while true; do radosgw-admin gc process; done
Maybe your normal RGW daemons are too busy for GC processing.
We have this in our config and have started extra RGW instances for GC only:
[global]
...
# disable garbage collector default
I have this problem also with OSDs that are with SSDs as block.db.
>
> Am 26.04.2022 um 17:10 schrieb Konstantin Shalygin :
>
> Hi,
>
> After some load HDD's will be not perform well. You should move block.db's to
> NVMe for avoid database vacuuming problems
>
> k
> Sent from my iPhone
>
Hi,
After some load HDD's will be not perform well. You should move block.db's to
NVMe for avoid database vacuuming problems
k
Sent from my iPhone
> On 26 Apr 2022, at 13:58, Boris Behrens wrote:
>
> The cluster contains 12x8TB OSDs without any SSDs as cache
So,
I just checked the logs on one of our smaller cluster and it looks like
this error happened twice last week.
The cluster contains 12x8TB OSDs without any SSDs as cache. And it started
with octopus (so no upgrade from nautilus was performed)
root@3cecef08a104:~# zgrep -i marked
So,
I just checked the logs on one of our smaller cluster and it looks like
this error happened twice last week.
The cluster contains 12x8TB OSDs without any SSDs as cache. And it started
with octopus (so no upgrade from nautilus was performed)
root@3cecef08a104:~# zgrep -i marked
I would not host multiple OSD on a spinning drive (unless it's one of those
Seagate MACH.2 drives that have two independent heads) - head seek time
will most likely kill performance. The main reason to host multiple OSD on
a single SSD or NVME is typically to make use of the large IOPS capacity
What should go to the SSDs? We have not enough Slots to have a 3/1 ratio
for block.db. Most of the block.db SSDs served 10 OSDs and were mostly
idling so we are now removing them as we haven't seen any benefits. (But
maybe I am just blind and ignorant and just not see it)
Am Mi., 23. März 2022 um
Unfortunately there is no silver bullet here so far. Just one note after
looking at your configuration - I would strongly encourage you to add
SSD disks for spinner-only OSDs.
Particularly when they are used for s3 payload which is pretty DB
intensive.
Thanks,
Igor
On 3/23/2022 5:03 PM,
Hi Igor,
yes, I've compacted them all.
So is there a solution for the problem, because I can imagine this happens
when we remove large files from s3 (we use it as backup storage for lz4
compressed rbd exports).
Maybe I missed it.
Cheers
Boris
Am Mi., 23. März 2022 um 13:43 Uhr schrieb Igor
Hi Boris,
Curious if you tried to compact RocksdDB for all your OSDs? Sorry I this
has been already discussed, haven't read through all the thread...
From my experience the symptoms you're facing are pretty common for DB
performance degradation caused by bulk data removal. In that case OSDs
You mean in the OSD logfiles?
Am Mi., 23. März 2022 um 08:23 Uhr schrieb Szabo, Istvan (Agoda) <
istvan.sz...@agoda.com>:
> Can you see in the pg dump like waiting for reading or something like
> this? In each step how much time it spends?
>
>
>
> Istvan Szabo
> Senior Infrastructure Engineer
>
Good morning Istvan,
those are rotating disks and we don't use EC. Splitting up the 16TB disks
into two 8TB partitions and have two OSDs on one disk also sounds
interesting, but would it solve the problem?
I also thought to adjust the PGs for the data pool from 4096 to 8192. But I
am not sure if
The number 180 PGs is because of the 16TB disks. 3/4 of our OSDs had cache
SSDs (not nvme though and most of them are 10OSDs one SSD) but this problem
only came in with octopus.
We also thought this might be the db compactation, but it doesn't match up.
It might happen when the compactation run,
180PG per OSD is usually overhead, also 40k obj per PG is not much, but I don't
think this will works without block.db NVMe. I think your "wrong out marks"
evulate in time of rocksdb compaction. With default log settings you can try to
grep 'latency' strings
Also,
Norf, I missed half of the answers...
* the 8TB disks hold around 80-90 PGs (16TB around 160-180)
* per PG we've around 40k objects 170m objects in 1.2PiB of storage
Am Di., 22. März 2022 um 09:29 Uhr schrieb Boris Behrens :
> Good morning K,
>
> the "freshly done" host, where it happened last
Good morning K,
the "freshly done" host, where it happened last got:
* 21x 8TB TOSHIBA MG06ACA800E (Spinning)
* No block.db devices (just removed the 2 cache SSDs by syncing the disks
out, wiping them and adding them back without block.db)
* 1x Intel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz
* 256GB ECC
Hi,
What is actual hardware (CPU, spinners, NVMe, network)?
This is HDD with block.db on NVMe?
How much PG per osd?
How much obj per PG?
k
Sent from my iPhone
> On 20 Mar 2022, at 19:59, Boris Behrens wrote:
>
> So,
> I have tried to remove the OSDs, wipe the disks and sync them back in
>
So,
I have tried to remove the OSDs, wipe the disks and sync them back in
without block.db SSD. (Still in progress, 212 spinning disks take time to
out and in again)
And I just experienced them same behavior on one OSD on a host where all
disks got synced in new. This disk was marked as in
Hi,
The last 2 osd I recreated were on december 30 and february 8.
I totally agree that ssd cache are a terrible spof. I think that's an
option if you use 1 ssd/nvme for 1 or 2 osd, but the cost is then very
high. Using 1 ssd for 10 osd increase the risk for almost no gain
because the ssd is
Hi Francois,
thanks for the reminder. We offline compacted all of the OSDs when we
reinstalled the hosts with the new OS.
But actually reinstalling them was never on my list.
I could try that and in the same go I can remove all the cache SSDs (when
one SSD share the cache for 10 OSDs this is a
Hi,
We also had this kind of problems after upgrading to octopus. Maybe you
can play with the hearthbeat grace time (
https://docs.ceph.com/en/latest/rados/configuration/mon-osd-interaction/
) to tell osds to wait a little more before declaring another osd down !
We also try to fix the problem
Yes, this is something we know and we disabled it, because we ran into the
problem that PGs went unavailable when two or more OSDs went offline.
I am searching for the reason WHY this happens.
Currently we have set the service file to restart=always and removed the
StartLimitBurst from the
Here's the reason they exit:
7f1605dc9700 -1 osd.97 486896 _committed_osd_maps marked down 6 >
osd_max_markdown_count 5 in last 600.00 seconds, shutting down
If an osd flaps (marked down, then up) 6 times in 10 minutes, it
exits. (This is a safety measure).
It's normally caused by a network
24 matches
Mail list logo
