[
https://issues.apache.org/jira/browse/HIVE-17608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-17608 started by Sankar Hariappan.
-----------------------------------------------
> REPL LOAD should overwrite the data files if exists instead of duplicating it
> -----------------------------------------------------------------------------
>
> Key: HIVE-17608
> URL: https://issues.apache.org/jira/browse/HIVE-17608
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, repl
> Affects Versions: 3.0.0
> Reporter: Sankar Hariappan
> Assignee: Sankar Hariappan
> Labels: DR, replication
> Fix For: 3.0.0
>
>
> This is to make insert event idempotent.
> Currently, MoveTask would create a new file if the destination folder
> contains a file of the same name. This is wrong if we have the same file in
> both bootstrap dump and incremental dump (by design, duplicate file in
> incremental dump will be ignored for idempotent reason), we will get
> duplicate files eventually. Also it is wrong to just retain the filename in
> the staging folder. Suppose we get the same insert event twice, the first
> time we get the file from source table folder, the second time we get the
> file from cm, we still end up with duplicate copy. The right solution is to
> keep the same file name as the source table folder.
> To do that, we can put the original filename in MoveWork, and in MoveTask, if
> original filename is set, don't generate a new name, simply overwrite. We
> need to do it in both bootstrap and incremental load.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
