[
https://issues.apache.org/jira/browse/TEZ-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated TEZ-1163:
-------------------------
Description:
When the per-reducer estimate is set to 256Mb and over-all data output is
~200Mb, gets a divide-by-zero.
{code}
2014-05-31 16:17:29,125 FATAL [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.ArithmeticException: / by zero
at
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.determineParallelismAndApply(ShuffleVertexManager.java:392)
at
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.schedulePendingTasks(ShuffleVertexManager.java:446)
{code}
Because of incorrect value set for minTasks
{code}
vertexmanager.ShuffleVertexManager: Shuffle Vertex Manager: settings
minFrac:0.25 maxFrac:0.75 auto:true desiredTaskIput:256000000 minTasks:0
{code}
was:
When the per-reducer estimate is set to 256Mb and over-all estimate is ~200Mb,
get a divide-by-zero because of integer division in currentParallelism/(200/256)
{code}
2014-05-31 16:17:29,125 FATAL [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.ArithmeticException: / by zero
at
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.determineParallelismAndApply(ShuffleVertexManager.java:392)
at
org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.schedulePendingTasks(ShuffleVertexManager.java:446)
{code}
> Tez Auto Reducer-parallelism throws Divide-by-Zero
> --------------------------------------------------
>
> Key: TEZ-1163
> URL: https://issues.apache.org/jira/browse/TEZ-1163
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Gopal V
> Assignee: Gopal V
> Priority: Minor
>
> When the per-reducer estimate is set to 256Mb and over-all data output is
> ~200Mb, gets a divide-by-zero.
> {code}
> 2014-05-31 16:17:29,125 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
> java.lang.ArithmeticException: / by zero
> at
> org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.determineParallelismAndApply(ShuffleVertexManager.java:392)
> at
> org.apache.tez.dag.library.vertexmanager.ShuffleVertexManager.schedulePendingTasks(ShuffleVertexManager.java:446)
> {code}
> Because of incorrect value set for minTasks
> {code}
> vertexmanager.ShuffleVertexManager: Shuffle Vertex Manager: settings
> minFrac:0.25 maxFrac:0.75 auto:true desiredTaskIput:256000000 minTasks:0
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
