tzphh opened a new issue, #10715: URL: https://github.com/apache/paimon/issues/10715
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation We use a partitioned table for periodically refreshed data and want to expire old partitions while keeping the latest available partition as a fallback when upstream refreshes are delayed or paused. With the current partition expiration policy, all existing partitions can become eligible for expiration when no newer partition arrives within `partition.expiration-time`. A subsequent expiration run can remove all of them from the latest table state. For this use case, keeping the last available data is preferable to returning an empty table solely because time has passed. For example, with a seven-day TTL and `values-time` expiration: | Situation | Expected behavior with the proposed option enabled | | --- | --- | | On October 21, the table contains only partitions dated October 1, 2, and 3 | Expire October 1 and 2; retain October 3 even though it is also expired | | Expiration runs again and no new partition has arrived | Continue retaining October 3 | | A new partition dated October 22 is committed | Allow October 3 to expire; October 22 becomes the latest available partition | | No newer partition arrives, and October 22 eventually exceeds the TTL | Continue retaining October 22 | The retained partition should remain visible through ordinary queries of the latest table state. Keeping historical snapshots or tags alone does not provide that behavior. ### Solution Add an opt-in safeguard to partition expiration for normal partitioned tables. Whenever all existing partitions are eligible for expiration, exclude the latest partition from the deletion candidates before applying deletion limits and batching. Suggested semantics: - Keep the current behavior by default. The option name can be discussed with the community. - Apply the normal TTL policy while at least one partition will remain. If all partitions are eligible, retain the latest one regardless of its age. - Determine the protected partition from committed table state. Uncommitted files from an incoming write must not cause the existing fallback partition to lose protection. - For `values-time`, select the latest partition by the parsed partition timestamp. For `update-time`, use the partition's last file creation time. Resolve ties deterministically. - Keep the safeguard effective across repeated expiration runs, deletion limits, and batch boundaries. Setting a small deletion limit alone must not eventually remove the protected partition. - Apply the same policy to writer-triggered expiration and the `expire_partitions` procedure. Once a newer partition is committed, the previously protected partition can expire normally. - An already empty table remains empty. The safeguard applies to partition expiration; explicit `DROP PARTITION` and `TRUNCATE` keep their existing semantics. The initial request is to retain one partition for the table as a whole. A configurable minimum count or per-group retention could be considered separately. ### Anything else? Checked against Apache Paimon `master` at `33a8cd34d3fc23c994cbbe0b1644f61035ee01e6` on October 21, 2026. This proposal is based on source and test inspection; no runtime reproduction was performed. - [NormalPartitionExpire](https://github.com/apache/paimon/blob/33a8cd34d3fc23c994cbbe0b1644f61035ee01e6/paimon-core/src/main/java/org/apache/paimon/operation/NormalPartitionExpire.java) selects expired partitions and applies deletion limits without a minimum remaining partition check. - [PartitionExpireTest](https://github.com/apache/paimon/blob/33a8cd34d3fc23c994cbbe0b1644f61035ee01e6/paimon-core/src/test/java/org/apache/paimon/operation/PartitionExpireTest.java#L479) explicitly expects an empty result after all partitions expire. `partition.expiration-max-num` limits deletions per run, and `snapshot.num-retained.min` retains table versions. Neither guarantees that the latest table state contains a partition. The [custom partition expiration SPI](https://github.com/apache/paimon/blob/33a8cd34d3fc23c994cbbe0b1644f61035ee01e6/paimon-core/src/main/java/org/apache/paimon/partition/PartitionExpireStrategyFactory.java) can be used to implement a custom policy, but a built-in option would avoid maintaining and deploying a separate strategy JAR for this use case. [Chain Table](https://paimon.apache.org/docs/2.1/primary-key-table/chain-table/#partition-expiration) already preserves an anchor partition for its snapshot/delta model; this request concerns ordinary partitioned tables. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
