Re: Is Insert Overwrite table partition on s3 is an atomic operation ?

2021-01-11 Thread Austin Hackett
Hi Mark It’s my understanding that when you do an INSERT OVERWRITE into a partition, Hive will take out an exclusive lock on the partition and a shared lock on the table itself. This blocks are read and write operations on the partition, and allows reads against the other partitions to

Re: Is Insert Overwrite table partition on s3 is an atomic operation ?

2021-01-11 Thread Mich Talebzadeh
Hi Mark, By atomic operation I gather you mean INSERT/OVERWRITE affects that partition only? According to my somehow dated scripts yes you can do that. The idea being that you only want to overwrite data for that partition ONLY. --show create table marketData; --Populate target table select

Is Insert Overwrite table partition on s3 is an atomic operation ?

2021-01-11 Thread Mark Norkin
Hello Hive users, We are using AWS Glue as Hive compatible metastore when running queries on EMR. For Hive external tables we are using AWS S3. After looking at the docs we didn't find a conclusive answer on whether an Insert Overwrite table partition is an atomic operation, maybe we've missed