[
https://issues.apache.org/jira/browse/TEZ-844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated TEZ-844:
-------------------------------
Attachment: TEZ-844.1.wip.txt
WIP patch, which makes the following changes.
ProcessorContext gets two additional methods (both blocking) which can be used
by Processors to wait for Inputs to complete
waitForAnyInputReady(List<Input>)
waitForAllInputsReady(List<Input>)
Inputs need to implement an additional method - registerInputReadyListener(...)
which is meant for framework use only. While this is visible to Processors,
they are not supposed to use it (similar to the behaviour of Input.close).
Exposing blocking methods on the ProcessorContext makes it easier for
Processors in terms of thread management - and boiler plate code. If a
Processor has advanced requirements, it can always start threads and make
multiple blocking calls to the ProcessorContext.
ReduceProcessor changed to start the sort only after Shuffle completes.
[~bikassaha], [~hitesh] - any inputs on the patch ? Hope to wrap this up
tomorrow.
> Processors should have a mechanism to know when an Input is ready for
> consumption
> ---------------------------------------------------------------------------------
>
> Key: TEZ-844
> URL: https://issues.apache.org/jira/browse/TEZ-844
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Attachments: TEZ-844.1.wip.txt
>
>
> Instead of polling, or blocking on a specific Reader, Processors should be
> able to find out when a specific Input is ready.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
