Re: Dynamic INSERT OVERWRITE

2021-01-30 Thread Zoltán Borók-Nagy
Thanks for your answer, Ryan. In the short term we'll only have INSERT INTO/OVERWRITE for Icebeg tables in Impala (so the way to go is AppendFiles and ReplacePartitions accordingly). But I agree that a MERGE INTO statement would be super useful. Hopefully we'll add support for it as well in the not

Re: Dynamic INSERT OVERWRITE

2021-01-29 Thread Ryan Blue
Zoltan, The warning is that dynamic overwrites in general aren't recommended. ReplacePartitions is the right operation to use for dynamic overwrite, we just want to steer users away from dynamic overwrites in general. The problem with dynamic overwrite is that its behavior depends on the underlyi

Dynamic INSERT OVERWRITE

2021-01-29 Thread Zoltán Borók-Nagy
Hey everyone, I'm currently working on the INSERT OVERWRITE statement for Iceberg tables in Impala. Seems like ReplacePartitions is the perfect interface for this job: https://github.infra.cloudera.com/CDH/iceberg/blob/cdpd-master/api/src/main/java/org/apache/iceberg/ReplacePartitions.java IIUC