Yes we can have a talk about it. We will try our best to write the RFC, maybe publish it in a few weeks.
> On Oct 21, 2022, at 10:18, JerryYue <272614...@qq.com.INVALID> wrote: > > Looking forward to the RFC > It's a good idea, we also need hudi data TTL in some case > Do we have any plan or time to do this? We also had some simple designs to > implement it > Maybe we can had a talk about it > > 在 2022/10/20 上午9:47，“Bingeng > Huang”<dev-return-5022-272614347=qq....@hudi.apache.org 代表 > hbgstc...@gmail.com> 写入: > > Looking forward to the RFC. > We can propose RFC about support TTL config using non-partition field after > > > > sagar sumit <cod...@apache.org> 于2022年10月19日周三 14:42写道： > >> +1 Very nice idea. Looking forward to the RFC! >> >> On Wed, Oct 19, 2022 at 10:13 AM Shiyan Xu <xu.shiyan.raym...@gmail.com> >> wrote: >> >>> great proposal. Partition TTL is a good starting point. we can extend it >> to >>> other TTL strategies like column-based, and make it customizable and >>> pluggable. Looking forward to the RFC! >>> >>> On Wed, Oct 19, 2022 at 11:40 AM Jian Feng <jian.f...@shopee.com.invalid >>> >>> wrote: >>> >>>> Good idea, >>>> this is definitely worth an RFC >>>> btw should it only depend on Hudi's partition? I feel it should be a >> more >>>> common feature since sometimes customers' data can not update across >>>> partitions >>>> >>>> >>>> On Wed, Oct 19, 2022 at 11:07 AM stream2000 <18889897...@163.com> >> wrote: >>>> >>>>> Hi all, we have implemented a partition based data ttl management, >>> which >>>>> we can manage ttl for hudi partition by size, expired time and >>>>> sub-partition count. When a partition is detected as outdated, we use >>>>> delete partition interface to delete it, which will generate a >> replace >>>>> commit to mark the data as deleted. The real deletion will then done >> by >>>>> clean service. >>>>> >>>>> >>>>> If community is interested in this idea, maybe we can propose a RFC >> to >>>>> discuss it in detail. >>>>> >>>>> >>>>>> On Oct 19, 2022, at 10:06, Vinoth Chandar <vin...@apache.org> >> wrote: >>>>>> >>>>>> +1 love to discuss this on a RFC proposal. >>>>>> >>>>>> On Tue, Oct 18, 2022 at 13:11 Alexey Kudinkin <ale...@onehouse.ai> >>>>> wrote: >>>>>> >>>>>>> That's a very interesting idea. >>>>>>> >>>>>>> Do you want to take a stab at writing a full proposal (in the form >>> of >>>>> RFC) >>>>>>> for it? >>>>>>> >>>>>>> On Tue, Oct 18, 2022 at 10:20 AM Bingeng Huang < >> hbgstc...@gmail.com >>>> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Do we have plan to integrate data TTL into HUDI, so we don't have >>> to >>>>>>>> schedule a offline spark job to delete outdated data, just set a >>> TTL >>>>>>>> config, then writer or some offline service will delete old data >> as >>>>>>>> expected. >>>>>>>> >>>>>>> >>>>> >>>>> >>>> >>>> -- >>>> *Jian Feng,冯健* >>>> Shopee | Engineer | Data Infrastructure >>>> >>> >>> >>> -- >>> Best, >>> Shiyan >>> >> >
