[
https://issues.apache.org/jira/browse/TEZ-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14119183#comment-14119183
]
Siddharth Seth commented on TEZ-1447:
-------------------------------------
>From an offline discussion with [~hagleitn] - forcing users to deal with
>different attempts, and status of tasks to process these events can get fairly
>complicated. This would end up having to do similar management to what is done
>in Shuffle - which isn't trivial.
In a majority of cases, events generated by different attempts of the same task
will be identical - in which case just making use of the event generated by the
first successful attempt is adequate. Doing something like this manes that
users don't worry about retries, indices etc - and can just rely on receiving a
set of events which are to be processed once the vertex succeeds.
If different attempts of the same workload generate different events -
processing is likely to be incorrect, since it's very possible for all data to
be processed (VERTEX successful), then a failure and retry - which generates a
different event. The initializer doesn't even run at this point, since it's
already done it's work and is complete. Handling such scenarios, likely
involves re-running the entire initializer and re-starting the vertex which
processed the event from scratch. In situations like this, where data generated
may be different, the best bet is for speculation to be disabled (when it's
supported), and max-attempts to be set to 1.
That said, there may be some other use cases (which don't exist today), which
need to send events in the middle of task execution (rather than after success)
- some kind of sampling from the task for instance. For cases like this, the
more elaborate scheme of sending in Task notifications, events with versions
etc would be required.
For the simpler cases, I'll look at adding an option to send such events on
first successful attempt.
> Handle parallelism updates and versioning w/ custom InputInitializerEvents
> --------------------------------------------------------------------------
>
> Key: TEZ-1447
> URL: https://issues.apache.org/jira/browse/TEZ-1447
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.5.0
> Reporter: Gunther Hagleitner
> Assignee: Siddharth Seth
> Priority: Blocker
>
> I'm trying to do dynamic partition pruning through input initializer events
> in Hive. That means that the initializer of a table scan vertex has to
> receive events from all tasks in another vertex (which contain the pruning
> info) before generating tasks to run.
> The problem with the current API I ran into:
> getNumTasks: I'm currently using a busy loop to wait for the num tasks for a
> vertex to be decided (-1 -> x). There's no way around it, because it's the
> only way to find out what number of events to expect (0 is a valid number of
> tasks - so I can't wait for the first to complete).
> With auto-reducer parallelism I have to employ another busy loop. Because I
> might be initially expecting 10 events, which later get's knocked down to 5.
> Since there's no event associated with this, I have to periodically check
> whether I have enough events.
> Versioning: Events have a version number, but I don't know which task they
> are coming from. Thus I can't de-dup events.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
