zyclove opened a new issue, #10316: URL: https://github.com/apache/hudi/issues/10316
How to configure TTL policy in hudi data table? Can the metadata （.hoodie ）be separated into a directory? Only configure the appropriate TTL for the data directory, so that data cleaning can also use hierarchical storage and different life cycles, and the data can also be automatically cleaned by relying on the object storage service, and there is no cost. EG: -------------------------------------------------------------------- s3://big-data-eu/hudi/data/bi_ods/table_name/dt=20231211/data < 30 days STANDARD S3 > 30 days delete by TTL with no cost. ================================= < 15 days STANDARD S3 > 15 days GLACIER_IR > 105 days delete by TTL with no cost. s3://big-data-eu/hudi/table_meta/bi_ods/table_name/.hoodie As mentioned above, if there are many data tables under the data storage and the storage periods are the same, I can just configure the storage period for the directory and rely on the object storage to automatically clean up the historical data at no cost. EG: -------------------------------------------------------------------- s3://big-data-eu/hudi/data/30days/bi_ods/table_name/dt=20231211/data < 30 days STANDARD S3 > 30 days delete by TTL with no cost. s3://big-data-eu/hudi/data/90days/bi_ods/table_name/dt=20231211/data < 30 days STANDARD S3 30days < 90days GLACIER_IR > 90 days delete by TTL with no cost. .... ================================= < 15 days STANDARD S3 > 15 days GLACIER_IR > 105 days delete by TTL with no cost. s3://big-data-eu/hudi/table_meta/bi_ods/table_name/.hoodie -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
