[
https://issues.apache.org/jira/browse/HIVE-29702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryu Kobayashi reassigned HIVE-29702:
------------------------------------
Assignee: Ryu Kobayashi
> Reduce-side merge join can produce NULL values or fail with
> IllegalStateException under Tez container reuse
> -----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-29702
> URL: https://issues.apache.org/jira/browse/HIVE-29702
> Project: Hive
> Issue Type: Bug
> Reporter: Ryu Kobayashi
> Assignee: Ryu Kobayashi
> Priority: Major
>
> h2. Summary
> Reduce-side merge join (a DummyStore + CommonMergeJoinOperator pipeline used
> for SMB / merge-join fallback) can misbehave when Tez reuses a container
> across multiple tasks of the same reducer vertex. Depending on timing, this
> manifests as either:
> (a) a task failure:
> {code}
> java.lang.IllegalStateException: Was expecting dummy store operator but
> found: FS[...]
> {code}
> (b) or, if no exception happens to be thrown, silent NULL values in the
> columns produced by the join, with no error reported to the user.
> h2. Root cause 1: stale operator-tree wiring on container reuse (crash)
> Hive's per-query {{ObjectCache}} (keyed by vertex name) can hand a task the
> same already-initialized {{DummyStoreOperator}} / {{CommonMergeJoinOperator}}
> instances that a previous task (sharing the same JVM/container) already used.
> {{TezDummyStoreOperator#closeOp}} does not remove itself from its child's
> {{parentOperators}}, nor clear its own {{childOperators}}, when closing. When
> the cached operator instances are reused by a subsequent task, the
> DummyStoreOperator still has a stale child reference left over from the
> previous task's execution. This causes
> {{ReduceRecordProcessor#getJoinParentOp}} to walk past the DummyStoreOperator
> into the wrong subtree, eventually reaching a non-DummyStoreOperator leaf and
> throwing {{IllegalStateException}}.
> Additionally, {{ReduceRecordProcessor#close()}} closes the main {{reducer}}
> before the {{mergeWorkList}} (DummyStore chain). Since
> {{DummyStoreOperator#closeOp}} is what removes the stale parent reference
> from the MergeJoin side, closing the main reducer first means
> {{CommonMergeJoinOperator#allInitializedParentsAreClosed()}} cannot detect
> all parents as closed in the correct order to flush the final join group.
> h2. Root cause 2: reducer count / partitioning mismatch (NULL values)
> (to be filled in once the corresponding fix is finalized)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
