[ https://issues.apache.org/jira/browse/HIVE-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-3302: -------------------------------- Status: Patch Available (was: Open) > Race condition in query plan for merging at the end of a query > -------------------------------------------------------------- > > Key: HIVE-3302 > URL: https://issues.apache.org/jira/browse/HIVE-3302 > Project: Hive > Issue Type: Bug > Components: Query Processor > Affects Versions: 0.10.0 > Reporter: Kevin Wilfong > Assignee: Kevin Wilfong > Priority: Critical > Attachments: HIVE-3302.1.patch.txt > > > In the query plan that's used to merge files at the end of a query, the > dependency tree looks something like: > MoveTask(1) > / \ > ...ConditionalTask MoveTask(2)... > \ / > MergeTask > Here MoveTask(1) moves the partition data to a temporary location, and > MoveTask(2) moves it to the final location. > However if there are dynamic partitions generated and some of these > partitions are merged and others are moved, the dependency tree is changed at > runtime to: > ...ConditionalTask MoveTask(2)... > \ / > MergeTask > \ > MoveTask(1) > This produces a race condition between the two MoveTasks where if MoveTask(2) > runs before MoveTask(1) the partitions moved by MoveTask(1) will get moved to > an intermediate location and never moved to the final location. In this case > those partitions are quietly lost. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira