[ https://issues.apache.org/jira/browse/TEZ-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391159#comment-14391159 ]
Gopal V commented on TEZ-2251: ------------------------------ Is vertex auto-parallelism kicking in when it receives 25% of all events for a vertex? Can we tweak it to kick in when the min-src fraction of all edges are satisfied? (i.e 100 small tasks, 3 big tasks feeding a JOIN will wait for 25 + 1 tasks, instead of triggering when 26 small tasks finish?). > Enabling auto reduce parallelism in certain jobs causes DAG to hang > ------------------------------------------------------------------- > > Key: TEZ-2251 > URL: https://issues.apache.org/jira/browse/TEZ-2251 > Project: Apache Tez > Issue Type: Bug > Reporter: Rajesh Balamohan > Attachments: hive_console.png, tez_2251_dag.png > > > Scenario: > - Run TPCH query20 > (https://github.com/cartershanklin/hive-testbench/blob/master/sample-queries-tpch/tpch_query20.sql) > at 1 TB scale (tez-master branch, hive trunk) > - Enable auto reduce parallelism > - DAG didn't complete and got stuck in "Reducer 6" > Vertex parallelism of "Reducer 5 & 6" happens within a span of 3 > milliseconds, and tasks of "reducer 5" ends up producing wrong partition > details as it sees the updated task numbers of reducer 6 when scheduled. > This causes, job to hang. -- This message was sent by Atlassian JIRA (v6.3.4#6332)