[
https://issues.apache.org/jira/browse/HBASE-28463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wellington Chevreuil updated HBASE-28463:
-----------------------------------------
Description:
This Jira introduces the feature of time-based priority in BucketCache, where a
configurable "age" is used as a threshold limit for data caching. Data blocks
with a more recent age then this limit should be kept in the cache, while older
data would be picked for eviction (or not considered for caching). The data age
based priority would be applied when deciding if a block should be added to
BucketCache (i.e. during reads, writes, compaction and prefetch), as well as
during the cache freeSpace run (mass eviction), before applying the LRU logic.
Because blocks don't hold any specific meta information other than type, it's
necessary to group blocks of same "age group" on separate files. We already
have DateTieredCompation for that, which allows for grouping blocks according
to its cells timestamps values in different time window groups.
DateTieredCompaction can be configured to provide two windows (one older and
one younger than the threshold limit), so that a cell timestamp based age
priority can be implemented. Additionally, we are extended DateTieredCompaction
so that the "age" value to be used for comparison can be provided in a
pluggable way, giving extra flexibility for different use cases to implement
their own concept of time priority.
The initial scope is to allow for data age to be determined in the following
different ways, all configurable:
* Cell timestamps: Uses the timestamp portion of HBase cells for comparing the
data age, requires DateTieredCompaction to be configured to provide two time
windows, one older and one younger than the time limit threshold.
* Custom cell qualifiers: Uses a custom-defined qualifier for comparing the
data age. It uses that value to tier the entire row containing the given
qualifier value. This requires that the custom qualifier be a valid Java long
timestamp, and must use the "new" compaction implementation defined as part of
this feature, the CustomTieredCompaction.
* Custom value provider: Allows for defining a pluggable implementation that
contains the logic for identifying the date value to be used for comparison.
This also requires the "new" compaction implementation defined as part of this
feature, the CustomTieredCompaction.
was:
This Jira introduces the feature of time-based data tiering in HBase to
optimize storage efficiency and access performance by segregating data based on
its recency. By keeping recent data in the bucket cache (backed by faster
storage types like SSDs) and evicting older data, the system aims to provide a
more flexible control over the cache allocation and eviction logic via
configuration, allowing for defining time priorities for cached data.
The need for a more extensive cache allocation mechanism becomes even more
critical on HBase deployments where cache access reflects on significant
performance gains, such as when using cloud storage as the underlying file
system.
The data is segregated into hot or cold categories based on its age. The recent
data within a specific time range (configured as hot-data-age) is treated as
hot and is stored in the cache, while the older data is stored and accessed
from the file system.
This feature intends to provide the TCO gains by optimizing the utilization of
high cost bucket cache. Perfect fit for the use cases that have the date-based
data writes while the scans focus on the recently written data.
Please find the detailed design document of the feature attached with the Jira.
Thanks,
Janardhan
> Time Based Priority for BucketCache
> -----------------------------------
>
> Key: HBASE-28463
> URL: https://issues.apache.org/jira/browse/HBASE-28463
> Project: HBase
> Issue Type: New Feature
> Components: BucketCache
> Reporter: Janardhan Hungund
> Assignee: Janardhan Hungund
> Priority: Major
>
> This Jira introduces the feature of time-based priority in BucketCache, where
> a configurable "age" is used as a threshold limit for data caching. Data
> blocks with a more recent age then this limit should be kept in the cache,
> while older data would be picked for eviction (or not considered for
> caching). The data age based priority would be applied when deciding if a
> block should be added to BucketCache (i.e. during reads, writes, compaction
> and prefetch), as well as during the cache freeSpace run (mass eviction),
> before applying the LRU logic.
> Because blocks don't hold any specific meta information other than type, it's
> necessary to group blocks of same "age group" on separate files. We already
> have DateTieredCompation for that, which allows for grouping blocks according
> to its cells timestamps values in different time window groups.
> DateTieredCompaction can be configured to provide two windows (one older and
> one younger than the threshold limit), so that a cell timestamp based age
> priority can be implemented. Additionally, we are extended
> DateTieredCompaction so that the "age" value to be used for comparison can be
> provided in a pluggable way, giving extra flexibility for different use cases
> to implement their own concept of time priority.
> The initial scope is to allow for data age to be determined in the following
> different ways, all configurable:
> * Cell timestamps: Uses the timestamp portion of HBase cells for comparing
> the data age, requires DateTieredCompaction to be configured to provide two
> time windows, one older and one younger than the time limit threshold.
> * Custom cell qualifiers: Uses a custom-defined qualifier for comparing the
> data age. It uses that value to tier the entire row containing the given
> qualifier value. This requires that the custom qualifier be a valid Java long
> timestamp, and must use the "new" compaction implementation defined as part
> of this feature, the CustomTieredCompaction.
> * Custom value provider: Allows for defining a pluggable implementation that
> contains the logic for identifying the date value to be used for comparison.
> This also requires the "new" compaction implementation defined as part of
> this feature, the CustomTieredCompaction.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
