-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72333/
-----------------------------------------------------------
(Updated April 9, 2020, 7:44 a.m.)
Review request for hive, Ashutosh Chauhan and Thejas Nair.
Bugs: HIVE-23154
https://issues.apache.org/jira/browse/HIVE-23154
Repository: hive-git
Description
-------
With rename(), we could run into a race condition between snapshot of files to
be moved ("filesKept") and when the fs.rename happens. It is possible that run
awaay task could have added more files to this.
1. Patch fixes the problem by relying on local threadpool to move the files
instead of fs.rename (where S3AFileSystem's rename is inherently parallel).
2. Same race condition issue persists in "insert into" mode as well, which was
relying on "fs.rename". Patch fixes this issue as well.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java e25dc54e7d
Diff: https://reviews.apache.org/r/72333/diff/2/
Changes: https://reviews.apache.org/r/72333/diff/1-2/
Testing
-------
File Attachments (updated)
----------------
HIVE-23154.3.patch
https://reviews.apache.org/media/uploaded/files/2020/04/09/38ab8dfe-18c2-4174-84da-e1028ad4133c__HIVE-23154.3.patch
Thanks,
Rajesh Balamohan
