[
https://issues.apache.org/jira/browse/TEZ-800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011422#comment-14011422
]
Hitesh Shah commented on TEZ-800:
---------------------------------
Comments:
{code}
}
-
+
// inform the vertices
{code}
- whitespace added.
{code}
LOG.info("Inferring vertex: " + outVertex.getVertexName() + " to be "
+ + v.getParallelism() + " from 1-1 connection with vertex "
+ + v.getVertexName());
{code}
- log message is not clear. It will show up as "Inferring vertex: v1 to be
10 from 1-1 connection with v2". Maybe added parallelism/ numtasks or something
similar after 10 ? Or "Inferring vertex parallelism for v1 to be ... "
For checkAndInferOneToOneParallelism(), should this be done using a BFS/DFS
approach? The current approach probably needs some comments to clarify how many
loops it will need to converge. Also, there is no handling for vertices that
have multiple inbound edges of different types or even all inbounds being of
type 1:1. I am assuming some of those will be caught by the consistency check
that follows.
In the tests, it will be good to test with re-ordering of the vertices when
they are added to the DAG. i.e. if the DAG is v1->v2->v3, do dag.add(v3);
add(v2); add(v1).
Is there a reason why this was not caught by the AM itself and parallelism set
as needed in the state machines?
> One-one edge with parallelism -1 fails if source vertex parallelism is not -1
> as well
> -------------------------------------------------------------------------------------
>
> Key: TEZ-800
> URL: https://issues.apache.org/jira/browse/TEZ-800
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.3.0
> Reporter: Rohini Palaniswamy
> Assignee: Bikas Saha
> Attachments: TEZ-800.1.patch
>
>
> // no input initializers. At this moment, only other case is 1-1 edge
> // with uninitialized sources
> boolean hasOneToOneUninitedSource = false;
> for (Map.Entry<Vertex, Edge> entry :
> vertex.sourceVertices.entrySet()) {
> if (entry.getValue().getEdgeProperty().getDataMovementType() ==
> DataMovementType.ONE_TO_ONE) {
> if (entry.getKey().getTotalTasks() == -1) {
> hasOneToOneUninitedSource = true;
> break;
> }
> }
> }
> This checks for the source vertex which has the 1-1 edge to also have -1
> parallelism.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
