[
https://issues.apache.org/jira/browse/TEZ-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14193527#comment-14193527
]
Bikas Saha commented on TEZ-1547:
---------------------------------
bq. need some info on Pig/Hive VertexManager usage
I already spoke to Gunther and Daniel. VertexManagers set -1 for their cases
and we have verified that distributor case works. Rajesh has tried this patch
with several jobs and Hive seems fine. Daniel is running all Pig tests with
this patch and is going to comment on this jira about the results.
bq. why is it being sent after invoking onVertexStarted
Because the VM can inform us about desire to reconfigure a vertex further at
that point too. So we cannot automatically send the event until then. Vertices
may start but still be reconfigured after that. So its necessary to
differentiate between them. If future changes allow more kinds of
reconfiguration, then FULLY_CONFIGURED needs to continue to support that.
STARTED does not have any definitive meaning. Hence I am in favor of keeping it
as is.
Given the above testing and all the tests in the patch I think this patch
should be fine. This puts some necessary API's in place which would be useful
as more people in the wild write vertex managers.
TEZ-1522 has an approach that may work for now where scheduling the first task
in a vertex indirectly implies that reconfigurations are not going to happen.
It needs to get tested with different complex graphs and scheduling triggers to
make sure we got that coded right and overall is not risk free. Its pluggable
but if there are issues and we plug it out then we open up to the risk of
getting hung again.
Waiting for [~daijy] to report back on the Pig test run which has earlier been
very useful in catching issues in this code path. If he reports no further
issues and there are no further comments then I will commit the last patch.
Thanks!
> Make use of state change notifier in VertexManagerPlugins
> ---------------------------------------------------------
>
> Key: TEZ-1547
> URL: https://issues.apache.org/jira/browse/TEZ-1547
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-1547.1.patch, TEZ-1547.3.patch, TEZ-1547.4.patch,
> TEZ-1547.5.patch, TEZ-1547.6.patch, TEZ-1547.7.patch, TEZ-1547.8.patch,
> TEZ-1547.9.patch
>
>
> Instead of the various APIs like onVertexStarted, simple notifications could
> be sent.
> Some existing APIs could end up being deprecated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
