[
https://issues.apache.org/jira/browse/TEZ-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046245#comment-14046245
]
Siddharth Seth commented on TEZ-1076:
-------------------------------------
Thinking out loud.
The InputInitializer currently only has a single opportunity to send out
events, which is by returning them at the end of the initialize call. This
doubles as a signal to the Vertex to indicate that the InputInitializer is
complete.
[~gopalv] has been asking for the ability to send out events as they're
generated - which isn't possible in the current model.
Additionally, there are initializers which do not need to indicate parallelism,
since that's already set - and just need to distribute events - Vertices likely
need to wait for these events to come through before launching tasks - which is
where an InputInitialized message is useful. In such cases, no event is
generated to set parallelism, and that cannot be used as a signal to get a
vertex started.
Eventually, InputInitializers should be able to signal there completion - i.e
parallelism set, or all events generated etc - likely an API on the context
saying initializerIsDone() - which is a signal to the Vertex to get started.
After this, additional events can continue to be sent - which would mean
VertexManagers get multiple calls from the same initializer to route events.
The API also changes at this point for the InputIniitalizerContext to be used
to send events.
Thoughts ?
That's likely too many changes for this jira. For this one, should be a fairly
straightforward change to add a new event type (RootInputInitializerEvent),
which gets routed like other events. An API on InputInitializer to handle this
event, and exposing Vertex information on the RootInputInitializerContext.
Usage continues to be the same - i.e. the user needs to block on the initialize
call if the Initializer is expecting to get events.
> Allow events to be sent to InputInitializers
> --------------------------------------------
>
> Key: TEZ-1076
> URL: https://issues.apache.org/jira/browse/TEZ-1076
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
>
> Events from tasks / vertices to InputInitializers.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
