Re: Deleting Hudi Partitons

2020-10-21 Thread Satish Kotha
Yes, that would work. You would typically add below option on dataframe to use insert overwrite (InsertOverwrite is a new API, I haven't updated documentation yet). - hoodie.datasource.write.operation: insert_overwrite Let me know if you have any questions. @Balaji Thanks for creating the f

Re: Deleting Hudi Partitons

2020-10-21 Thread Balaji Varadarajan
Fixing incorrect Satish's email.On Wednesday, October 21, 2020, 06:19:43 PM PDT, Balaji Varadarajan wrote: cc Satish who implemented Insert Overwrite support. We have recently landed Insert Overwrite support in Hudi. Partition level deletion is a logical extension of this feature but

Re: Deleting Hudi Partitons

2020-10-21 Thread Balaji Varadarajan
cc Satish who implemented Insert Overwrite support. We have recently landed Insert Overwrite support in Hudi. Partition level deletion is a logical extension of this feature but not currently available yet.  I have added a jira to track this :  https://issues.apache.org/jira/browse/HUDI-1350 Mea

Deleting Hudi Partitons

2020-10-20 Thread selvaraj periyasamy
Team , I have a COW table which has sub partition columns Date/Hour . For some of the use case , I need to totally remove free petitions (removing few hours alone) . Hudi maintains metadata info. Manually removing folders as well as in hive megastore , may mess up hudi metadata. What is the best w