On 31/08/2020 12:31, Igor Fedotov wrote:
Hi Wido,
'b' prefix relates to free list manager which keeps all the free extents
for main device in a bitmap. Its records have fixed size hence you can
easily estimate the overall size for these type of data.
Yes, so I figured.
But I doubt it takes that much. I presume that DB just lacks the proper
compaction. Which could happen eventually but looks like you interrupted
the process by going offline.
May be try manual compaction with ceph-kvstore-tool?
This cluster is suffering from a lot of spillovers. So we tested with
marking one OSD as out.
After being marked as out it still had this large DB. A compact didn't
work, the RocksDB database just stayed so large.
New OSDs coming into the cluster aren't suffering from this and they
have a RocksDB of a couple of MB in size.
Old OSDs installed with Luminous and now upgraded to Nautilus are
suffering from this.
It kind of seems like that garbage data stays behind in RocksDB which is
never clean up.
Wido
Thanks,
Igor
On 8/31/2020 10:57 AM, Wido den Hollander wrote:
Hello,
On a Nautilus 14.2.8 cluster I am seeing large RocksDB database with
many slow DB bytes in use.
To investigate this further I marked one OSD as out and waited for the
all the backfilling to complete.
Once the backfilling was completed I exported BlueFS and investigated
the RocksDB using 'ceph-kvstore-tool'. This resulted in 22GB of data.
Listing all the keys in the RocksDB shows me there are 747.000 keys in
the DB. A small portion are osdmaps, but the biggest amount are keys
prefixed with 'b'.
I dumped the stats of the RocksDB and this shows me:
L1: 1/0: 439.32 KB
L2: 1/0: 2.65 MB
L3: 5/0: 14.36 MB
L4: 127/0: 7.22 GB
L5: 217/0: 13.73 GB
Sum: 351/0: 20.98 GB
So there is almost 21GB of data in this RocksDB database. Why? Where
is this coming from?
Throughout this cluster OSDs are suffering from many slow bytes used
and I can't figure out why.
Has anybody seen this or has a clue on what is going on?
I have an external copy of this RocksDB database to do investigations on.
Thank you,
Wido
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
