katty he created TEZ-4512:
-----------------------------
Summary: One-to-One edge Parallelism
Key: TEZ-4512
URL: https://issues.apache.org/jira/browse/TEZ-4512
Project: Apache Tez
Issue Type: Improvement
Affects Versions: 0.10.2
Reporter: katty he
Currently, we check parallelism for One-one edge, if inputVertex's parallelism
doesn't equal outputVertex or -1, tez will report error:" Dag submit failed due
to 1-1 Edge. Destination vertex parallelism must match source vertex. Vertex:
Reducer x does not match vertex: Reducer x", why shouldn't we change the
parallelism with inputVertex and outputVertex instead of reporting error? maybe
the code can be like:
for (Edge e : edges) {
Vertex inputVertex = e.getInputVertex();
Vertex outputVertex = e.getOutputVertex();
if (e.getEdgeProperty().getDataMovementType() ==
DataMovementType.ONE_TO_ONE) {
if (inputVertex.getParallelism() != outputVertex.getParallelism()) {
// both should be equal or equal to -1.
if (outputVertex.getParallelism() != -1) {
int parallelism =
Math.min(inputVertex.getParallelism(),outputVertex.getParallelism());
inputVertex.setParallelism(parallelism);
outputVertex.setParallelism(parallelism);
}
}
}
}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
