[
https://issues.apache.org/jira/browse/TEZ-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119432#comment-14119432
]
Bikas Saha commented on TEZ-1447:
---------------------------------
Interested parties should be able to register(ENUM, List<Vertex>) so that they
can specify that they are only interested on certain notifications and not
every possible notification.
InputInitializerContext already allows querying the number of tasks in a
vertex. There is no need to pass the number of tasks in the event itself. That
info can be queried from the context. After that there is no need for separate
handling of onParallelismUpdated().
Not sure why we need multiple notifiers etc. It was hard to follow the patch
without any design doc or logical outline. So maybe I am missing something.
Does the following approach (similar in part to what is there in the patch)
work?
1) There is a PubSub entity available in appContext. When register(enum,
vertex) is called then it creates a listener entry with key <vertex, enum> in
the PubSub entity.
2) When an interesting state change happens then VertexImpl call
PubSub.notifyChange(this, enum). The translation of vertex state to
notification enum happens in the VertexImpl. The PubSub entity looks up any
listeners based the <vertex, enum> key and calls back on those listeners.
We can potentially have one state change, trigger another state change and
create a cascade. We should consider doing the callback via events instead of
directly calling back in the state machine code on the dispatcher thread. When
the input initializer and vertex manager register themselves to PubSub then
they can pass their vertex names and their references. PubSub can send a state
change update event to the listener by routing it through the listeners vertex.
This is simple and prevents a cascade of callbacks blocking the dispatcher.
> Handle parallelism updates and versioning w/ custom InputInitializerEvents
> --------------------------------------------------------------------------
>
> Key: TEZ-1447
> URL: https://issues.apache.org/jira/browse/TEZ-1447
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Gunther Hagleitner
> Assignee: Siddharth Seth
> Priority: Blocker
> Attachments: TEZ-1447.1.wip.txt
>
>
> I'm trying to do dynamic partition pruning through input initializer events
> in Hive. That means that the initializer of a table scan vertex has to
> receive events from all tasks in another vertex (which contain the pruning
> info) before generating tasks to run.
> The problem with the current API I ran into:
> getNumTasks: I'm currently using a busy loop to wait for the num tasks for a
> vertex to be decided (-1 -> x). There's no way around it, because it's the
> only way to find out what number of events to expect (0 is a valid number of
> tasks - so I can't wait for the first to complete).
> With auto-reducer parallelism I have to employ another busy loop. Because I
> might be initially expecting 10 events, which later get's knocked down to 5.
> Since there's no event associated with this, I have to periodically check
> whether I have enough events.
> Versioning: Events have a version number, but I don't know which task they
> are coming from. Thus I can't de-dup events.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
