Shohei Okumiya created TEZ-4569:
-----------------------------------
Summary: SCATTER_GATHER + BROADCAST hangs on DAG Recovery
Key: TEZ-4569
URL: https://issues.apache.org/jira/browse/TEZ-4569
Project: Apache Tez
Issue Type: Improvement
Affects Versions: 0.10.3
Reporter: Shohei Okumiya
Assignee: Shohei Okumiya
Attachments: image-2024-06-11-20-45-12-540.png
A Tez DAG fails to initialize itself when an Application Master is timely
preempted.
The problem typically happens with Map Join(Broadcast Hash Join) of Hive when
the broadcast edge is multi-staged. In the following case, the smaller side
includes one aggregation, and the condition is satisfied.
{code:java}
CREATE TABLE small AS SELECT 1 AS id;
CREATE TABLE big AS SELECT 1 AS id UNION ALL SELECT 2 AS id UNION ALL SELECT 3
AS id;
SELECT *
FROM big
JOIN (SELECT id, count(*) AS num FROM small GROUP BY id) s ON big.id = s.id
{code}
Once it happens, a retried AM fails to configure the Map Join vertex. In the
following case, Map 1 never starts.
{code:java}
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
----------------------------------------------------------------------------------------------
Map 2 .......... container SUCCEEDED 1 1 0 0
0 1
Reducer 3 ...... container SUCCEEDED 1 1 0 0
0 0
Map 1 container INITIALIZING -1 0 0 -1
0 0
----------------------------------------------------------------------------------------------
{code}
Tez starts Map 2 and Map 1 once their splits are configured. The hang issue
happens when an AM is retried before it starts Reducer 3.
!image-2024-06-11-20-45-12-540.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
