[
https://issues.apache.org/jira/browse/TEZ-394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated TEZ-394:
---------------------------
Attachment: TEZ-394.001.patch
Attaching a patch that reorders the vertices based on maximum distance from
leaf nodes. Nodes that are farther from a leaf will have a higher priority
than nodes that are closer, and nodes at the same distance will break ties
based on number of inputs (fewer inputs is higher priority than more inputs).
This leverages the topology sort implemented in TEZ-31, walking the topology
backwards to compute the distance from the leaf in a single pass. While
working on this I believe I found a bug in the Tarjan strongly connected
algorithm implementation. The implementation uses boxed integers in a way that
implies the value can be updated by callees, but in Java Integer objects are
immutable so any increment done in the callee creates a new object not seen by
the caller. That is fixed in this patch.
In addition I added a utility method to create a stubbed vertex in the unit
test code and updated existing tests to use that utility method.
> Better scheduling for uneven DAGs
> ---------------------------------
>
> Key: TEZ-394
> URL: https://issues.apache.org/jira/browse/TEZ-394
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Rohini Palaniswamy
> Attachments: TEZ-394.001.patch
>
>
> Consider a series of joins or group by on dataset A with few datasets that
> takes 10 hours followed by a final join with a dataset X. The vertex that
> loads dataset X will be one of the top vertexes and initialized early even
> though its output is not consumed till the end after 10 hours.
> 1) Could either use delayed start logic for better resource allocation
> 2) Else if they are started upfront, need to handle failure/recovery cases
> where the nodes which executed the MapTask might have gone down when the
> final join happens.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
