, 2022 18:54
To: tao xiao ; David Morávek
Cc: Yun Tang ; user
Subject: RE: RocksDB state not cleaned up
May I ask if anyone tested RocksDB’s periodic compaction in the meantime? And
if yes, if it helped with this case.
Regards,
Alexis.
From: tao xiao
Sent: Samstag, 18. September 2021 05:01
May I ask if anyone tested RocksDB’s periodic compaction in the meantime? And
if yes, if it helped with this case.
Regards,
Alexis.
From: tao xiao
Sent: Samstag, 18. September 2021 05:01
To: David Morávek
Cc: Yun Tang ; user
Subject: Re: RocksDB state not cleaned up
Thanks for the feedback
Thanks for the feedback! However TTL already proves that the state cannot
be cleaned up on time due to too many levels built up in RocksDB.
Hi @Yun Tang do you have any suggestions to tune RocksDB
to accelerate the compaction progress?
On Fri, Sep 17, 2021 at 8:01 PM David Morávek wrote:
>
Cleaning up with timers should solve this. Both approaches have some
advantages and disadvantages though.
Timers:
- No "side effects".
- Can be set in event time. Deletes are regular tombstones that will get
compacted later on.
TTL:
- Performance. This costs literally nothing compared to an
Hi David,
Confirmed with RocksDB log Stephan's observation is the root cause that
compaction doesn't clean up the high level sst files fast enough. Do you
think manual clean up by registering a timer is the way to go or any
RocksDB parameter can be tuned to mitigate this issue?
On Wed, Sep 15,
Hi David,
If I read Stephan's comment correctly TTL doesn't work well for cases where
we have too many levels, like fast growing state, as compaction doesn't
clean up high level SST files in time, Is this correct? If yes should we
register a timer with TTL time and manual clean up the state
Hi Tao,
my intuition is that the compaction of SST files is not triggering. By
default, it's only triggered by the size ratios of different levels [1] and
the TTL mechanism has no effect on it.
Some reasoning from Stephan:
It's very likely to have large files in higher levels that haven't been
