[
https://issues.apache.org/jira/browse/HIVE-15199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15687108#comment-15687108
]
Steve Loughran commented on HIVE-15199:
---------------------------------------
I do think I'd rather fix this in s3, because it is adding 2 GET calls and a
LIST before each rename, calls which take place in the rename itself. And of
course, when Hadoop 2.8 or derivatives change s3a's rename to == HDFS, the
check will be superfluous. Similarly, once you have a consistent FS view
(s3guard, etc), you are less likely to see a mismatch between listing and
stat-ing. If you are, it means something else is writing to the same dir,
putting you in trouble.
Would it be possible to set this up to make it easy to turn off in future. For
example: create the JIRA on stripping the exists check out?
> INSERT INTO data on S3 is replacing the old rows with the new ones
> ------------------------------------------------------------------
>
> Key: HIVE-15199
> URL: https://issues.apache.org/jira/browse/HIVE-15199
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: Sergio Peña
> Assignee: Sergio Peña
> Priority: Critical
> Attachments: HIVE-15199.1.patch, HIVE-15199.2.patch,
> HIVE-15199.3.patch, HIVE-15199.4.patch, HIVE-15199.5.patch,
> HIVE-15199.6.patch, HIVE-15199.7.patch, HIVE-15199.8.patch
>
>
> Any INSERT INTO statement run on S3 tables and when the scratch directory is
> saved on S3 is deleting old rows of the table.
> {noformat}
> hive> set hive.blobstore.use.blobstore.as.scratchdir=true;
> hive> create table t1 (id int, name string) location 's3a://spena-bucket/t1';
> hive> insert into table t1 values (1,'name1');
> hive> select * from t1;
> 1 name1
> hive> insert into table t1 values (2,'name2');
> hive> select * from t1;
> 2 name2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
