+1 Very nice idea. Looking forward to the RFC! On Wed, Oct 19, 2022 at 10:13 AM Shiyan Xu <xu.shiyan.raym...@gmail.com> wrote:
> great proposal. Partition TTL is a good starting point. we can extend it to > other TTL strategies like column-based, and make it customizable and > pluggable. Looking forward to the RFC! > > On Wed, Oct 19, 2022 at 11:40 AM Jian Feng <jian.f...@shopee.com.invalid> > wrote: > > > Good idea, > > this is definitely worth an RFC > > btw should it only depend on Hudi's partition? I feel it should be a more > > common feature since sometimes customers' data can not update across > > partitions > > > > > > On Wed, Oct 19, 2022 at 11:07 AM stream2000 <18889897...@163.com> wrote: > > > > > Hi all, we have implemented a partition based data ttl management, > which > > > we can manage ttl for hudi partition by size, expired time and > > > sub-partition count. When a partition is detected as outdated, we use > > > delete partition interface to delete it, which will generate a replace > > > commit to mark the data as deleted. The real deletion will then done by > > > clean service. > > > > > > > > > If community is interested in this idea, maybe we can propose a RFC to > > > discuss it in detail. > > > > > > > > > > On Oct 19, 2022, at 10:06, Vinoth Chandar <vin...@apache.org> wrote: > > > > > > > > +1 love to discuss this on a RFC proposal. > > > > > > > > On Tue, Oct 18, 2022 at 13:11 Alexey Kudinkin <ale...@onehouse.ai> > > > wrote: > > > > > > > >> That's a very interesting idea. > > > >> > > > >> Do you want to take a stab at writing a full proposal (in the form > of > > > RFC) > > > >> for it? > > > >> > > > >> On Tue, Oct 18, 2022 at 10:20 AM Bingeng Huang <hbgstc...@gmail.com > > > > > >> wrote: > > > >> > > > >>> Hi all, > > > >>> > > > >>> Do we have plan to integrate data TTL into HUDI, so we don't have > to > > > >>> schedule a offline spark job to delete outdated data, just set a > TTL > > > >>> config, then writer or some offline service will delete old data as > > > >>> expected. > > > >>> > > > >> > > > > > > > > > > -- > > *Jian Feng,冯健* > > Shopee | Engineer | Data Infrastructure > > > > > -- > Best, > Shiyan >
