Yingda Chen created TEZ-3999:
--------------------------------
Summary: Extend VertexManagerPlugin interface to allow for
relevant events notification
Key: TEZ-3999
URL: https://issues.apache.org/jira/browse/TEZ-3999
Project: Apache Tez
Issue Type: Task
Reporter: Yingda Chen
This is an umbrella task of TEZ-3997
**For concurrent connection, the downstream and upstream vertices would be
running concurrently, and in some cases, they would be scheduled at the same
time as well, such as (sub-graph) gang scheduling. However, *this is not always
true*. In the example in Fig. 1, tasks in PS vertex should be running before
tasks in W vertex should be scheduled. Since otherwise if the resource requests
for PS cannot be fulfilled first, W will be spinning in vain. In other
examples, as long as part of tasks in upstream vertex are running, we can start
scheduling downstream tasks.
In other words, if we put this into the context of existing
interface/implementation of VertexMangerPlugin, we can see strong duality of
“OnSourceTaskRunning” for concurrent connection vs the “OnSourceTaskCompleted”
for (existing) sequential connection. Therefore, we propose an addition of
“_onConcurrentSourceTaskRunning(TaskAttemptIdentifer attempt)_” interface to
the VertexManager Plugin, with default implementation being not supported.
This change will also include the logic to add source task running event and to
send such events to downstream vertices. To reduce unnecessary event traffic,
we will limit the sending of such events to CONCURRENT edge, and when the
ConcurrentSchedulingType is specified to be SOURCE_TASK_STARTED .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
