Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on PR #8062: URL: https://github.com/apache/hudi/pull/8062#issuecomment-1846475981 > @geserdugarov I'm gonna merge this one first, can you fire your suggestion chagne PR incrementally based on this one? And we can move the discussions there. @danny0405 Ok, I will work on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 merged PR #8062: URL: https://github.com/apache/hudi/pull/8062 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on PR #8062: URL: https://github.com/apache/hudi/pull/8062#issuecomment-1846473187 @geserdugarov I'm gonna merge this one first, can you fire your suggestion chagne PR incrementally based on this one? And we can move the discussions there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1418323849
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,210 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+For 1.0.0 and later hudi version which supports efficient completion time
queries on the timeline(#9565), we can get partition's `lastModifiedTime` by
scanning the timeline and get the last write commit for the partition. Also for
efficiency,
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1418322239
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,210 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
Review Comment:
Then make it clear in the pseudocode code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062: URL: https://github.com/apache/hudi/pull/8062#discussion_r1418307128 ## rfc/rfc-65/rfc-65.md: ## @@ -0,0 +1,248 @@ +## Proposers + +- @stream2000 +- @hujincalrin +- @huberylee +- @YuweiXiao + +## Approvers + +## Status + +JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823) + +## Abstract + +In some classic hudi use cases, users partition hudi data by time and are only interested in data from a recent period +of time. The outdated data is useless and costly, we need a lifecycle management mechanism to prevent the +dataset from growing infinitely. Review Comment: To clarify: this is very different, the proposal for this doc only yields target partitons, it does not remove the whole partition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on PR #8062: URL: https://github.com/apache/hudi/pull/8062#issuecomment-1840714919 > @danny0405 @geserdugarov Thanks for your review! I have addressed all comments, hope for your opinions. @stream2000 Thank you for reply. As you mentioned, I've wrote all my ideas in a separate RFC to simplify details discussion: https://github.com/apache/hudi/pull/10248 @danny0405 @stream2000 Do you mind me asking for your comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1415523349
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions und
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1415523349
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions und
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062: URL: https://github.com/apache/hudi/pull/8062#discussion_r1415515672 ## rfc/rfc-65/rfc-65.md: ## @@ -0,0 +1,209 @@ +## Proposers + +- @stream2000 +- @hujincalrin +- @huberylee +- @YuweiXiao + +## Approvers + +## Status + +JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823) + +## Abstract + +In some classic hudi use cases, users partition hudi data by time and are only interested in data from a recent period +of time. The outdated data is useless and costly, we need a TTL(Time-To-Live) management mechanism to prevent the +dataset from growing infinitely. +This proposal introduces Partition TTL Management strategies to hudi, people can config the strategies by table config +directly or by call commands. With proper configs set, Hudi can find out which partitions are outdated and delete them. + + +This proposal introduces Partition TTL Management service to hudi. TTL management is like other table services such as Clean/Compaction/Clustering. +The user can config their ttl strategies through write configs and Hudi will help users find expired partitions and delete them automatically. + +## Background + +TTL management mechanism is an important feature for databases. Hudi already provides a `delete_partition` interface to +delete outdated partitions. However, users still need to detect which partitions are outdated and +call `delete_partition` manually, which means that users need to define and implement some kind of TTL strategies, find expired partitions and call call `delete_partition` by themselves. As the scale of installations grew, it is becoming increasingly important to implement a user-friendly TTL management mechanism for hudi. + +## Implementation + +Our main goals are as follows: + +* Providing an extensible framework for partition TTL management. +* Implement a simple KEEP_BY_TIME strategy, which can be executed through independent Spark job, synchronous or asynchronous table services. + +### Strategy Definition + +The TTL strategies is similar to existing table service strategies. We can define TTL strategies like defining a clustering/clean/compaction strategy: + +```properties +hoodie.partition.ttl.management.strategy=KEEP_BY_TIME Review Comment: > For example, user wants to add a new strategy that delete partitions by their specified logic. Do you have any example of this logic? Unfortunately, I couldn't come up with any of them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
stream2000 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1414833910
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
Review Comment:
Again, leverage `.hoodie_partition_metadata` will bring format change, and
it doesn't support any kind of transaction currently. As discussed with
@danny0405 , In 1.0.0 and later version which supports efficient completion
time queries on the timeline(#9565), we will have a more elegant way to get the
`lastCommitTime`.
You can see the the updated RFC for details.
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-582
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413459990
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions und
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413457079
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions und
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413454783
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions und
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413454783
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions und
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413436193
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
Review Comment:
> The TTL policy will default to KEEP_BY_TIME and we can automatically
detect expired partitions by their last modified time and delete them.
Unfortunately, I don't see any other strategies except KEEP_BY_TIME for
partition level TTL.
> Customized implementation of PartitionTTLManagementStrategy will only
provide a list of expired partitions, and hudi will help delete them. We can
rename the word PartitionTTLManagementStrategy to
PartitionLifecycleManagementStrategy since we do not always delete the
partitions by time.
I see a partition on different object levels as:
| level | time | size
|
| | -- |
--- |
| the whole partition | TTL | not applicable, could be trigger for lower
levels|
| file groups | managed by | other services |
| file slices| managed by | other services |
| records | TTL | oldest could be removed by partition
size threshold |
So, we don't need anything else except KEEP_BY_TIME for partition level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please con
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on PR #8062: URL: https://github.com/apache/hudi/pull/8062#issuecomment-1837977625 > @geserdugarov Thanks very much for your review! I think the most important part of the design to be confirmed is whether we need to provide a feature-rich but complicated TTL policy in the first place or just implement a simple but extensible policy. > > @geserdugarov @codope @nsivabalan @vinothchandar Hope for your opinion for this ~ Yes, you're right about the main difference in our ideas. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413459990
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions und
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413457079
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions und
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413454783
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions und
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413453307
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions und
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413452143
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions und
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413444578
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
Review Comment:
Currently, HoodiePartitionMetadata, `.hoodie_partition_metadata` files,
provides only `commitTime` and `partitionDepth` properties. This file is
written during partition creation and is not updated later. We can make this
file more usable by adding a new property, for instance, `lastUpdateTime` for
saving time of the last upsert/delete operation on the partition with each
commit/deltacommit/replacecommit. And use this property to handle for Partition
TTL check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abov
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413444578
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
Review Comment:
Currently, HoodiePartitionMetadata, `.hoodie_partition_metadata` files,
provides only commitTime and partitionDepth properties. This file is written
during partition creation and is not updated later. We can make this file more
usable by adding a new property, for instance, `lastUpdateTime` for saving time
of the last upsert/delete operation on the partition with each
commit/deltacommit/replacecommit. And use this property to handle for Partition
TTL check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413444578
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
Review Comment:
Currently, HoodiePartitionMetadata, `.hoodie_partition_metadata` files,
provides only commitTime, commit time of partition creation, and partitionDepth
properties. This file is written during partition creation and is not updated
later.
We can add new property, for instance, `lastUpdateTime` to save time of the
last upsert/delete operation on the partition with each
commit/deltacommit/replacecommit. And use this property to handle for Partition
TTL check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abo
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1413436193
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
Review Comment:
> The TTL policy will default to KEEP_BY_TIME and we can automatically
detect expired partitions by their last modified time and delete them.
Unfortunately, I don't see any other strategies except KEEP_BY_TIME for
partition level TTL.
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users stil
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062: URL: https://github.com/apache/hudi/pull/8062#discussion_r1413434722 ## rfc/rfc-65/rfc-65.md: ## @@ -0,0 +1,209 @@ +## Proposers + +- @stream2000 +- @hujincalrin +- @huberylee +- @YuweiXiao + +## Approvers + +## Status + +JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823) + +## Abstract + +In some classic hudi use cases, users partition hudi data by time and are only interested in data from a recent period +of time. The outdated data is useless and costly, we need a TTL(Time-To-Live) management mechanism to prevent the +dataset from growing infinitely. +This proposal introduces Partition TTL Management strategies to hudi, people can config the strategies by table config +directly or by call commands. With proper configs set, Hudi can find out which partitions are outdated and delete them. + + +This proposal introduces Partition TTL Management service to hudi. TTL management is like other table services such as Clean/Compaction/Clustering. +The user can config their ttl strategies through write configs and Hudi will help users find expired partitions and delete them automatically. + +## Background + +TTL management mechanism is an important feature for databases. Hudi already provides a `delete_partition` interface to +delete outdated partitions. However, users still need to detect which partitions are outdated and +call `delete_partition` manually, which means that users need to define and implement some kind of TTL strategies, find expired partitions and call call `delete_partition` by themselves. As the scale of installations grew, it is becoming increasingly important to implement a user-friendly TTL management mechanism for hudi. + +## Implementation + +Our main goals are as follows: + +* Providing an extensible framework for partition TTL management. +* Implement a simple KEEP_BY_TIME strategy, which can be executed through independent Spark job, synchronous or asynchronous table services. + +### Strategy Definition + +The TTL strategies is similar to existing table service strategies. We can define TTL strategies like defining a clustering/clean/compaction strategy: + +```properties +hoodie.partition.ttl.management.strategy=KEEP_BY_TIME Review Comment: I suppose it's better to implement only one processing for Partition TTL with accounting time only. For KEEP_BY_SIZE partition level looks not suitable. It's appropriate for record level processing. So, we don't need this setting: `hoodie.partition.ttl.strategy`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
geserdugarov commented on code in PR #8062: URL: https://github.com/apache/hudi/pull/8062#discussion_r1413434722 ## rfc/rfc-65/rfc-65.md: ## @@ -0,0 +1,209 @@ +## Proposers + +- @stream2000 +- @hujincalrin +- @huberylee +- @YuweiXiao + +## Approvers + +## Status + +JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823) + +## Abstract + +In some classic hudi use cases, users partition hudi data by time and are only interested in data from a recent period +of time. The outdated data is useless and costly, we need a TTL(Time-To-Live) management mechanism to prevent the +dataset from growing infinitely. +This proposal introduces Partition TTL Management strategies to hudi, people can config the strategies by table config +directly or by call commands. With proper configs set, Hudi can find out which partitions are outdated and delete them. + + +This proposal introduces Partition TTL Management service to hudi. TTL management is like other table services such as Clean/Compaction/Clustering. +The user can config their ttl strategies through write configs and Hudi will help users find expired partitions and delete them automatically. + +## Background + +TTL management mechanism is an important feature for databases. Hudi already provides a `delete_partition` interface to +delete outdated partitions. However, users still need to detect which partitions are outdated and +call `delete_partition` manually, which means that users need to define and implement some kind of TTL strategies, find expired partitions and call call `delete_partition` by themselves. As the scale of installations grew, it is becoming increasingly important to implement a user-friendly TTL management mechanism for hudi. + +## Implementation + +Our main goals are as follows: + +* Providing an extensible framework for partition TTL management. +* Implement a simple KEEP_BY_TIME strategy, which can be executed through independent Spark job, synchronous or asynchronous table services. + +### Strategy Definition + +The TTL strategies is similar to existing table service strategies. We can define TTL strategies like defining a clustering/clean/compaction strategy: + +```properties +hoodie.partition.ttl.management.strategy=KEEP_BY_TIME Review Comment: I suppose it's better to implement only one processing for Partition TTL with accounting time only. For KEEP_BY_SIZE partition level looks not suitable. It's appropriate for record level processing. So, we don't need `hoodie.partition.ttl.strategy` setting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1411533758
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,210 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+For 1.0.0 and later hudi version which supports efficient completion time
queries on the timeline(#9565), we can get partition's `lastModifiedTime` by
scanning the timeline and get the last write commit for the partition. Also for
efficiency,
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1411533758
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,210 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+For 1.0.0 and later hudi version which supports efficient completion time
queries on the timeline(#9565), we can get partition's `lastModifiedTime` by
scanning the timeline and get the last write commit for the partition. Also for
efficiency,
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1411533324
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,210 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+For 1.0.0 and later hudi version which supports efficient completion time
queries on the timeline(#9565), we can get partition's `lastModifiedTime` by
scanning the timeline and get the last write commit for the partition. Also for
efficiency,
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1411532246
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
Review Comment:
We have no notion of `lastModifiedTime`, do you mean `lastCommitTime` or
`latestCommitTime`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1411531558
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,210 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
Review Comment:
Do we need to pass around basic info like table path or meta client so that
user can customize more easily? For example, how can a customized class knows
the table path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062: URL: https://github.com/apache/hudi/pull/8062#discussion_r1411530672 ## rfc/rfc-65/rfc-65.md: ## @@ -0,0 +1,210 @@ +## Proposers + +- @stream2000 +- @hujincalrin +- @huberylee +- @YuweiXiao + +## Approvers + +## Status + +JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823) + +## Abstract + +In some classic hudi use cases, users partition hudi data by time and are only interested in data from a recent period +of time. The outdated data is useless and costly, we need a TTL(Time-To-Live) management mechanism to prevent the +dataset from growing infinitely. +This proposal introduces Partition TTL Management strategies to hudi, people can config the strategies by table config +directly or by call commands. With proper configs set, Hudi can find out which partitions are outdated and delete them. + +This proposal introduces Partition TTL Management service to hudi. TTL management is like other table services such as Clean/Compaction/Clustering. +The user can config their ttl strategies through write configs and Hudi will help users find expired partitions and delete them automatically. + +## Background + +TTL management mechanism is an important feature for databases. Hudi already provides a `delete_partition` interface to +delete outdated partitions. However, users still need to detect which partitions are outdated and +call `delete_partition` manually, which means that users need to define and implement some kind of TTL strategies, find expired partitions and call call `delete_partition` by themselves. As the scale of installations grew, it is becoming increasingly important to implement a user-friendly TTL management mechanism for hudi. + +## Implementation + +Our main goals are as follows: + +* Providing an extensible framework for partition TTL management. +* Implement a simple KEEP_BY_TIME strategy, which can be executed through independent Spark job, synchronous or asynchronous table services. + +### Strategy Definition + +The TTL strategies is similar to existing table service strategies. We can define TTL strategies like defining a clustering/clean/compaction strategy: + +```properties +hoodie.partition.ttl.management.strategy=KEEP_BY_TIME +hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy Review Comment: hoodie.partition.ttl.strategy.class ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062: URL: https://github.com/apache/hudi/pull/8062#discussion_r1411530357 ## rfc/rfc-65/rfc-65.md: ## @@ -0,0 +1,209 @@ +## Proposers + +- @stream2000 +- @hujincalrin +- @huberylee +- @YuweiXiao + +## Approvers + +## Status + +JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823) + +## Abstract + +In some classic hudi use cases, users partition hudi data by time and are only interested in data from a recent period +of time. The outdated data is useless and costly, we need a TTL(Time-To-Live) management mechanism to prevent the +dataset from growing infinitely. +This proposal introduces Partition TTL Management strategies to hudi, people can config the strategies by table config +directly or by call commands. With proper configs set, Hudi can find out which partitions are outdated and delete them. + + +This proposal introduces Partition TTL Management service to hudi. TTL management is like other table services such as Clean/Compaction/Clustering. +The user can config their ttl strategies through write configs and Hudi will help users find expired partitions and delete them automatically. + +## Background + +TTL management mechanism is an important feature for databases. Hudi already provides a `delete_partition` interface to +delete outdated partitions. However, users still need to detect which partitions are outdated and +call `delete_partition` manually, which means that users need to define and implement some kind of TTL strategies, find expired partitions and call call `delete_partition` by themselves. As the scale of installations grew, it is becoming increasingly important to implement a user-friendly TTL management mechanism for hudi. + +## Implementation + +Our main goals are as follows: + +* Providing an extensible framework for partition TTL management. +* Implement a simple KEEP_BY_TIME strategy, which can be executed through independent Spark job, synchronous or asynchronous table services. + +### Strategy Definition + +The TTL strategies is similar to existing table service strategies. We can define TTL strategies like defining a clustering/clean/compaction strategy: + +```properties +hoodie.partition.ttl.management.strategy=KEEP_BY_TIME Review Comment: `hoodie.partition.ttl.strategy` ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
stream2000 commented on PR #8062: URL: https://github.com/apache/hudi/pull/8062#issuecomment-1833238919 For 1.0.0 and later hudi version which supports efficient completion time queries on the timeline(#9565), we can get partition's `lastModifiedTime` by scanning the timeline and get the last write commit for the partition. Also for efficiency, we can store the partitions' last modified time and current completion time in the replace commit metadata. The next time we need to calculate the partitions' last modified time, we can build incrementally from the replace commit metadata of the last ttl management. @danny0405 Added new `lastModifiedTime` calculation method for 1.0.0 and later hudi version. We plan to implement the file listing based `lastModifiedTime` at first and implement the timeline-based `lastModifiedTime` calculation in a separate PR. This will help users with earlier hudi versions easy to pick the function to their code base. I have addressed all comments according to online/offline discussions. If there is no other concern, we can move on this~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
stream2000 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1406076695
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions under
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
stream2000 commented on code in PR #8062: URL: https://github.com/apache/hudi/pull/8062#discussion_r1406043167 ## rfc/rfc-65/rfc-65.md: ## @@ -0,0 +1,110 @@ +## Proposers +- @stream2000 +- @hujincalrin +- @huberylee +- @YuweiXiao +## Approvers +## Status +JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823) +## Abstract +In some classic hudi use cases, users partition hudi data by time and are only interested in data from a recent period of time. The outdated data is useless and costly, we need a TTL(Time-To-Live) management mechanism to prevent the dataset from growing infinitely. +This proposal introduces Partition TTL Management policies to hudi, people can config the policies by table config directly or by call commands. With proper configs set, Hudi can find out which partitions are outdated and delete them. +## Background +TTL management mechanism is an important feature for databases. Hudi already provides a delete_partition interface to delete outdated partitions. However, users still need to detect which partitions are outdated and call `delete_partition` manually, which means that users need to define and implement some kind of TTL policies and maintain proper statistics to find expired partitions by themself. As the scale of installations grew, it's more important to implement a user-friendly TTL management mechanism for hudi. +## Implementation +There are 3 components to implement Partition TTL Management + +- TTL policy definition & storage +- Partition statistics for TTL management +- Appling policies +### TTL Policy Definition +We have three main considerations when designing TTL policy: + +1. User hopes to manage partition TTL not only by expired time but also by sub-partitions count and sub-partitions size. So we need to support the following three different TTL policy types. +1. **KEEP_BY_TIME**. Partitions will expire N days after their last modified time. Review Comment: In the latest version of the RFC, we use the max instant time of the committed file slices in the partition as the partition's last modified time for simplicity. Otherwise, we need some extra mechanism to get the last modified time. In our inner version, we maintain an extra JSON file and update it incrementally as new instants committed to get the real modified time for the partition. Also, we can use metadata table to track the last modify time. What do you think about this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1405592067
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,209 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a TTL(Time-To-Live)
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces Partition TTL Management strategies to hudi, people
can config the strategies by table config
+directly or by call commands. With proper configs set, Hudi can find out which
partitions are outdated and delete them.
+
+
+This proposal introduces Partition TTL Management service to hudi. TTL
management is like other table services such as Clean/Compaction/Clustering.
+The user can config their ttl strategies through write configs and Hudi will
help users find expired partitions and delete them automatically.
+
+## Background
+
+TTL management mechanism is an important feature for databases. Hudi already
provides a `delete_partition` interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of TTL strategies, find expired partitions and call call
`delete_partition` by themselves. As the scale of installations grew, it is
becoming increasingly important to implement a user-friendly TTL management
mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition TTL management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or asynchronous table services.
+
+### Strategy Definition
+
+The TTL strategies is similar to existing table service strategies. We can
define TTL strategies like defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.ttl.management.strategy=KEEP_BY_TIME
+hoodie.partition.ttl.management.strategy.class=org.apache.hudi.table.action.ttl.strategy.KeepByTimePartitionTTLManagementStrategy
+hoodie.partition.ttl.days.retain=10
+```
+
+The config `hoodie.partition.ttl.management.strategy.class` is to provide a
strategy class (subclass of `PartitionTTLManagementStrategy`) to get expired
partition paths to delete. And `hoodie.partition.ttl.days.retain` is the
strategy value used by `KeepByTimePartitionTTLManagementStrategy` which means
that we will expire partitions that haven't been modified for this strategy
value set. We will cover the `KeepByTimeTTLManagementStrategy` strategy in
detail in the next section.
+
+The core definition of `PartitionTTLManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level TTL management.
+ */
+public abstract class PartitionTTLManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition ttl management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of `PartitionTTLManagementStrategy`
and hudi will help delete the expired partitions.
+
+### KeepByTimeTTLManagementStrategy
+
+We will provide a strategy call `KeepByTimePartitionTTLManagementStrategy` in
the first version of partition TTL management implementation.
+
+The `KeepByTimePartitionTTLManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If duration between now and
'lastModifiedTime' for the partition is larger than what
`hoodie.partition.ttl.days.retain` configured,
`KeepByTimePartitionTTLManagementStrategy` will mark this partition as an
expired partition. We use day as the unit of expired time since it is very
common-used for datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However, we can assume that we won't do
clustering for a partition without new writes for a long time when using the
strategy. And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example using metadata table.
+
+### Apply different strategies for different partitions
+
+For some specific users, they may want to apply different strategies for
different partitions. For example, they may have multi partition
fileds(productId, day). For partitions under
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062: URL: https://github.com/apache/hudi/pull/8062#discussion_r1405590162 ## rfc/rfc-65/rfc-65.md: ## @@ -0,0 +1,110 @@ +## Proposers +- @stream2000 +- @hujincalrin +- @huberylee +- @YuweiXiao +## Approvers +## Status +JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823) +## Abstract +In some classic hudi use cases, users partition hudi data by time and are only interested in data from a recent period of time. The outdated data is useless and costly, we need a TTL(Time-To-Live) management mechanism to prevent the dataset from growing infinitely. +This proposal introduces Partition TTL Management policies to hudi, people can config the policies by table config directly or by call commands. With proper configs set, Hudi can find out which partitions are outdated and delete them. +## Background +TTL management mechanism is an important feature for databases. Hudi already provides a delete_partition interface to delete outdated partitions. However, users still need to detect which partitions are outdated and call `delete_partition` manually, which means that users need to define and implement some kind of TTL policies and maintain proper statistics to find expired partitions by themself. As the scale of installations grew, it's more important to implement a user-friendly TTL management mechanism for hudi. +## Implementation +There are 3 components to implement Partition TTL Management + +- TTL policy definition & storage +- Partition statistics for TTL management +- Appling policies +### TTL Policy Definition +We have three main considerations when designing TTL policy: + +1. User hopes to manage partition TTL not only by expired time but also by sub-partitions count and sub-partitions size. So we need to support the following three different TTL policy types. +1. **KEEP_BY_TIME**. Partitions will expire N days after their last modified time. Review Comment: How to fetch the info when a commit is archived? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
zyclove commented on PR #8062: URL: https://github.com/apache/hudi/pull/8062#issuecomment-1826719670 Will version 1.0 support this feature? This feature is greatly needed. @danny0405 @stream2000 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
stream2000 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1399985395
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,248 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a lifecycle
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces partition lifecycle management strategies to hudi,
people can config the strategies by write
+configs. With proper configs set, Hudi can find out which partitions are
expired and delete them.
+
+This proposal introduces partition lifecycle management service to hudi.
Lifecycle management is like other table
+services such as Clean/Compaction/Clustering.
+Users can config their partition lifecycle management strategies through write
configs and Hudi will help users find
+expired partitions and delete them automatically.
+
+## Background
+
+Lifecycle management mechanism is an important feature for databases. Hudi
already provides a `delete_partition`
+interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of partition lifecycle
+management strategies, find expired partitions and call `delete_partition` by
themselves. As the scale of installations
+grew, it is becoming increasingly important to implement a user-friendly
lifecycle management mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition lifecycle management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or
+ asynchronous table services.
+
+### Strategy Definition
+
+The lifecycle strategies is similar to existing table service strategies. We
can define lifecycle strategies like
+defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.lifecycle.management.strategy=KEEP_BY_TIME
+hoodie.partition.lifecycle.management.strategy.class=org.apache.hudi.table.action.lifecycle.strategy.KeepByTimePartitionLifecycleManagementStrategy
+hoodie.partition.lifecycle.days.retain=10
+```
+
+The config `hoodie.partition.lifecycle.management.strategy.class` is to
provide a strategy class (subclass
+of `PartitionLifecycleManagementStrategy`) to get expired partition paths to
delete.
+And `hoodie.partition.lifecycle.days.retain` is the strategy value used
+by `KeepByTimePartitionLifecycleManagementStrategy` which means that we will
expire partitions that haven't been
+modified for this strategy value set. We will cover the
`KeepByTimePartitionLifecycleManagementStrategy` strategy in
+detail in the next section.
+
+The core definition of `PartitionLifecycleManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level lifecycle management.
+ */
+public abstract class PartitionLifecycleManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition lifecycle management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of
`PartitionLifecycleManagementStrategy` and hudi will help delete the
+expired partitions.
+
+### KeepByTimePartitionLifecycleManagementStrategy
+
+We will provide a strategy call
`KeepByTimePartitionLifecycleManagementStrategy` in the first version of
partition
+lifecycle management implementation.
+
+The `KeepByTimePartitionLifecycleManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If
+duration between now and 'lastModifiedTime' for the partition is larger than
+what `hoodie.partition.lifecycle.days.retain` configured,
`KeepByTimePartitionLifecycleManagementStrategy` will mark
+this partition as an expired partition. We use day as the unit of expired time
since it is very common-used for
+datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the
+partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However,
+we can assume that we won't do clustering for a partition without new writes
for a long time when using the strategy.
+And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example
+using metadata table.
+
+### Apply different strategies
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1398634206
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,248 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a lifecycle
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces partition lifecycle management strategies to hudi,
people can config the strategies by write
+configs. With proper configs set, Hudi can find out which partitions are
expired and delete them.
+
+This proposal introduces partition lifecycle management service to hudi.
Lifecycle management is like other table
+services such as Clean/Compaction/Clustering.
+Users can config their partition lifecycle management strategies through write
configs and Hudi will help users find
+expired partitions and delete them automatically.
+
+## Background
+
+Lifecycle management mechanism is an important feature for databases. Hudi
already provides a `delete_partition`
+interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of partition lifecycle
+management strategies, find expired partitions and call `delete_partition` by
themselves. As the scale of installations
+grew, it is becoming increasingly important to implement a user-friendly
lifecycle management mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition lifecycle management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or
+ asynchronous table services.
+
+### Strategy Definition
+
+The lifecycle strategies is similar to existing table service strategies. We
can define lifecycle strategies like
+defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.lifecycle.management.strategy=KEEP_BY_TIME
+hoodie.partition.lifecycle.management.strategy.class=org.apache.hudi.table.action.lifecycle.strategy.KeepByTimePartitionLifecycleManagementStrategy
+hoodie.partition.lifecycle.days.retain=10
+```
+
+The config `hoodie.partition.lifecycle.management.strategy.class` is to
provide a strategy class (subclass
+of `PartitionLifecycleManagementStrategy`) to get expired partition paths to
delete.
+And `hoodie.partition.lifecycle.days.retain` is the strategy value used
+by `KeepByTimePartitionLifecycleManagementStrategy` which means that we will
expire partitions that haven't been
+modified for this strategy value set. We will cover the
`KeepByTimePartitionLifecycleManagementStrategy` strategy in
+detail in the next section.
+
+The core definition of `PartitionLifecycleManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level lifecycle management.
+ */
+public abstract class PartitionLifecycleManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition lifecycle management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of
`PartitionLifecycleManagementStrategy` and hudi will help delete the
+expired partitions.
+
+### KeepByTimePartitionLifecycleManagementStrategy
+
+We will provide a strategy call
`KeepByTimePartitionLifecycleManagementStrategy` in the first version of
partition
+lifecycle management implementation.
+
+The `KeepByTimePartitionLifecycleManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If
+duration between now and 'lastModifiedTime' for the partition is larger than
+what `hoodie.partition.lifecycle.days.retain` configured,
`KeepByTimePartitionLifecycleManagementStrategy` will mark
+this partition as an expired partition. We use day as the unit of expired time
since it is very common-used for
+datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the
+partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However,
+we can assume that we won't do clustering for a partition without new writes
for a long time when using the strategy.
+And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example
+using metadata table.
+
+### Apply different strategies
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062:
URL: https://github.com/apache/hudi/pull/8062#discussion_r1398633783
##
rfc/rfc-65/rfc-65.md:
##
@@ -0,0 +1,248 @@
+## Proposers
+
+- @stream2000
+- @hujincalrin
+- @huberylee
+- @YuweiXiao
+
+## Approvers
+
+## Status
+
+JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823)
+
+## Abstract
+
+In some classic hudi use cases, users partition hudi data by time and are only
interested in data from a recent period
+of time. The outdated data is useless and costly, we need a lifecycle
management mechanism to prevent the
+dataset from growing infinitely.
+This proposal introduces partition lifecycle management strategies to hudi,
people can config the strategies by write
+configs. With proper configs set, Hudi can find out which partitions are
expired and delete them.
+
+This proposal introduces partition lifecycle management service to hudi.
Lifecycle management is like other table
+services such as Clean/Compaction/Clustering.
+Users can config their partition lifecycle management strategies through write
configs and Hudi will help users find
+expired partitions and delete them automatically.
+
+## Background
+
+Lifecycle management mechanism is an important feature for databases. Hudi
already provides a `delete_partition`
+interface to
+delete outdated partitions. However, users still need to detect which
partitions are outdated and
+call `delete_partition` manually, which means that users need to define and
implement some kind of partition lifecycle
+management strategies, find expired partitions and call `delete_partition` by
themselves. As the scale of installations
+grew, it is becoming increasingly important to implement a user-friendly
lifecycle management mechanism for hudi.
+
+## Implementation
+
+Our main goals are as follows:
+
+* Providing an extensible framework for partition lifecycle management.
+* Implement a simple KEEP_BY_TIME strategy, which can be executed through
independent Spark job, synchronous or
+ asynchronous table services.
+
+### Strategy Definition
+
+The lifecycle strategies is similar to existing table service strategies. We
can define lifecycle strategies like
+defining a clustering/clean/compaction strategy:
+
+```properties
+hoodie.partition.lifecycle.management.strategy=KEEP_BY_TIME
+hoodie.partition.lifecycle.management.strategy.class=org.apache.hudi.table.action.lifecycle.strategy.KeepByTimePartitionLifecycleManagementStrategy
+hoodie.partition.lifecycle.days.retain=10
+```
+
+The config `hoodie.partition.lifecycle.management.strategy.class` is to
provide a strategy class (subclass
+of `PartitionLifecycleManagementStrategy`) to get expired partition paths to
delete.
+And `hoodie.partition.lifecycle.days.retain` is the strategy value used
+by `KeepByTimePartitionLifecycleManagementStrategy` which means that we will
expire partitions that haven't been
+modified for this strategy value set. We will cover the
`KeepByTimePartitionLifecycleManagementStrategy` strategy in
+detail in the next section.
+
+The core definition of `PartitionLifecycleManagementStrategy` looks like this:
+
+```java
+/**
+ * Strategy for partition-level lifecycle management.
+ */
+public abstract class PartitionLifecycleManagementStrategy {
+ /**
+ * Get expired partition paths for a specific partition lifecycle management
strategy.
+ *
+ * @return Expired partition paths.
+ */
+ public abstract List getExpiredPartitionPaths();
+}
+```
+
+Users can provide their own implementation of
`PartitionLifecycleManagementStrategy` and hudi will help delete the
+expired partitions.
+
+### KeepByTimePartitionLifecycleManagementStrategy
+
+We will provide a strategy call
`KeepByTimePartitionLifecycleManagementStrategy` in the first version of
partition
+lifecycle management implementation.
+
+The `KeepByTimePartitionLifecycleManagementStrategy` will calculate the
`lastModifiedTime` for each input partitions. If
+duration between now and 'lastModifiedTime' for the partition is larger than
+what `hoodie.partition.lifecycle.days.retain` configured,
`KeepByTimePartitionLifecycleManagementStrategy` will mark
+this partition as an expired partition. We use day as the unit of expired time
since it is very common-used for
+datalakes. Open to ideas for this.
+
+we will to use the largest commit time of committed file groups in the
partition as the partition's
+`lastModifiedTime`. So any write (including normal DMLs, clustering etc.) with
larger instant time will change the
+partition's `lastModifiedTime`.
+
+For file groups generated by replace commit, it may not reveal the real
insert/update time for the file group. However,
+we can assume that we won't do clustering for a partition without new writes
for a long time when using the strategy.
+And in the future, we may introduce a more accurate mechanism to get
`lastModifiedTime` of a partition, for example
+using metadata table.
+
+### Apply different strategies
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062: URL: https://github.com/apache/hudi/pull/8062#discussion_r1398624021 ## rfc/rfc-65/rfc-65.md: ## @@ -0,0 +1,248 @@ +## Proposers + +- @stream2000 +- @hujincalrin +- @huberylee +- @YuweiXiao + +## Approvers + +## Status + +JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823) + +## Abstract + +In some classic hudi use cases, users partition hudi data by time and are only interested in data from a recent period +of time. The outdated data is useless and costly, we need a lifecycle management mechanism to prevent the +dataset from growing infinitely. +This proposal introduces partition lifecycle management strategies to hudi, people can config the strategies by write +configs. With proper configs set, Hudi can find out which partitions are expired and delete them. + +This proposal introduces partition lifecycle management service to hudi. Lifecycle management is like other table +services such as Clean/Compaction/Clustering. +Users can config their partition lifecycle management strategies through write configs and Hudi will help users find +expired partitions and delete them automatically. + +## Background + +Lifecycle management mechanism is an important feature for databases. Hudi already provides a `delete_partition` +interface to +delete outdated partitions. However, users still need to detect which partitions are outdated and +call `delete_partition` manually, which means that users need to define and implement some kind of partition lifecycle +management strategies, find expired partitions and call `delete_partition` by themselves. As the scale of installations +grew, it is becoming increasingly important to implement a user-friendly lifecycle management mechanism for hudi. + +## Implementation + +Our main goals are as follows: + +* Providing an extensible framework for partition lifecycle management. +* Implement a simple KEEP_BY_TIME strategy, which can be executed through independent Spark job, synchronous or + asynchronous table services. + +### Strategy Definition + +The lifecycle strategies is similar to existing table service strategies. We can define lifecycle strategies like +defining a clustering/clean/compaction strategy: + +```properties +hoodie.partition.lifecycle.management.strategy=KEEP_BY_TIME +hoodie.partition.lifecycle.management.strategy.class=org.apache.hudi.table.action.lifecycle.strategy.KeepByTimePartitionLifecycleManagementStrategy Review Comment: `hoodie.partition.lifecycle.management.strategy` -> `hoodie.partition.ttl.strategy` ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
danny0405 commented on code in PR #8062: URL: https://github.com/apache/hudi/pull/8062#discussion_r1398623725 ## rfc/rfc-65/rfc-65.md: ## @@ -0,0 +1,248 @@ +## Proposers + +- @stream2000 +- @hujincalrin +- @huberylee +- @YuweiXiao + +## Approvers + +## Status + +JIRA: [HUDI-5823](https://issues.apache.org/jira/browse/HUDI-5823) + +## Abstract + +In some classic hudi use cases, users partition hudi data by time and are only interested in data from a recent period +of time. The outdated data is useless and costly, we need a lifecycle management mechanism to prevent the +dataset from growing infinitely. Review Comment: Guess you are talking about GDPR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5823][RFC-65] RFC for Partition Lifecycle Management [hudi]
stream2000 commented on PR #8062: URL: https://github.com/apache/hudi/pull/8062#issuecomment-1798178547 Hi community, I have implemeted a poc version of partition lifecycle management in this PR: #9723. This PR provide a simple `KEEP_BY_TIME` strategy and can do the partition lifecycle management inline after commit. This POC implementation will help understand the design mentioned in this RFC. Hope for your review😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
