[
https://issues.apache.org/jira/browse/TEZ-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14194491#comment-14194491
]
Siddharth Seth commented on TEZ-1547:
-------------------------------------
OK. That makes sense. The intent is to send out an event, irrespective of
whether configuration changed at runtime - so that VMs have a reliable
mechanism to monitor, instead of not knowing which ones to monitor for. In
terms of the naming, I'll still maintain that COMPLETE_CONFIGURED is a bit
limiting. RECONFIGURED only makes sense if there's some changes - which isn't
the intent. CONFIGURED would be simpler - at leaves scope for a RECONFIGURED in
the future (with semantic changes).
At the moment, we're effectively trying to use this event for two purposes
(hence the STARTED name suggestion) - one two let VMs know that the TaskSpec
will not change. The other is that tasks have been scheduled for the VM which
is completely configured (which is what will prevent slowdowns, and out of
order scheduling)? Is this interpretation of the intent correct ?
What I'm getting at, is that this event is currently Overloaded, and is being
used in the VertexManagers to mean both Configured and Tasks scheduled.
For the ImmediateStartVertexManager - if a FULLY_CONFIGURED/CONFIGURED event
goes out before the sending vertex schedules it's tasks - this can cause the
vertex linked to the listening VM to schedule tasks ahead of the configured
vertex. (In this case, CONFIGURED is sent out after scheduleTasks - controlled
by Tez itself)
For the ShuffleVertexManager - the event will not go out before the
ShuffleVertexManager indicates that it is fully configured. The responsibility
of scheduling tasks before sending these events ends up falling on the
VertexManager. (In this case, CONFIGURED is sent out after scheduleTasks -
controlled by the user code)
If these two functionalities were split up - between CONFIGURED /
TASKS_SCHEDULED - all the notifications can be sent out by Tez itself, and the
VM code becomes simpler - instead of having to order API invocations correctly.
I'm assuming "vertexReconfigurationPlanned" is supposed to be called only
during VertexManager initialization (initialize() or Constructor). From that
point, it's simple enough to figure out whether the CONFIGURED event needs to
go out immediately on VM initialization (based on parallelism), or later based
on the intent to re-configure.
Thoughts on the current overloading that's happening ? If you agree with this,
lets figure out whether it needs to be resolved right-now or as a follow up.
Rohini, Daniel - thanks for trying this with Pig.
> Make use of state change notifier in VertexManagerPlugins
> ---------------------------------------------------------
>
> Key: TEZ-1547
> URL: https://issues.apache.org/jira/browse/TEZ-1547
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Bikas Saha
> Attachments: TEZ-1547.1.patch, TEZ-1547.3.patch, TEZ-1547.4.patch,
> TEZ-1547.5.patch, TEZ-1547.6.patch, TEZ-1547.7.patch, TEZ-1547.8.patch,
> TEZ-1547.9.patch
>
>
> Instead of the various APIs like onVertexStarted, simple notifications could
> be sent.
> Some existing APIs could end up being deprecated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
