[
https://issues.apache.org/jira/browse/TEZ-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16632205#comment-16632205
]
Hitesh Sharma edited comment on TEZ-3996 at 9/28/18 7:18 PM:
-------------------------------------------------------------
One thing I forgot to mention is that we are only reordering events sent to
task within a heartbeat only. The happy path cases go like task receiving DMEs
and then some IFEs (typically after some time of receiving the initial DME).
This is fine and the processor can deal with it and in our case we mark the
processor as a non-fatal failure and retry. Upon retry though, the new task
attempt receives the DMEs (old and new version) and IFEs, within the same HB,
and that's why we are looking to order them so that processor can learn about
the IFEs first and ignore the old versioned DMEs.
{quote} Arguably a better fix is to simply not send DMEs to tasks where we know
the input has failed rather than send it and then invalidate it.
{quote}
This will also invalidate the statement that IFE arrive after DME, no? As now
the failed version of DME will never come.
The other potential solution is that we send IFEs after DMEs but DMEs are
sorted in descending order of version. This means that processor receives the
latest version of DMEs first and can ignore the old ones.
was (Author: hrsharma):
One thing I forgot to mention is that we are only looking to events sent to
task within a heartbeat only. The happy path cases go like task receiving DMEs
and then some IFEs (typically after some time of receiving the initial DME).
This is fine and the processor can deal with it and in our case we mark the
processor as a non-fatal failure and retry. Upon retry though, the new task
attempt receives the DMEs (old and new version) and IFEs, within the same HB,
and that's why we are looking to order them so that processor can learn about
the IFEs first and ignore the old versioned DMEs.
{quote}
Arguably a better fix is to simply not send DMEs to tasks where we know the
input has failed rather than send it and then invalidate it.
{quote}
This will also invalidate the statement that IFE arrive after DME, no? As now
the failed version of DME will never come.
The other potential solution is that we send IFEs after DMEs but DMEs are
sorted in descending order of version. This means that processor receives the
latest version of DMEs first and can ignore the old ones.
> Reorder input failed events before data movement events
> -------------------------------------------------------
>
> Key: TEZ-3996
> URL: https://issues.apache.org/jira/browse/TEZ-3996
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Hitesh Sharma
> Priority: Minor
>
> We have a custom processor (AbstractLogicalIOProcessor) that waits for
> DataMovementEvent to arrive and then starts an external process to do some
> work. When a revocation happens then the processor recieves an
> InputFailedEvent, which tells it about the failed input, and we fail the
> processor as it is working on old inputs. When the new inputs are available
> then Tez restarts the processor and sends the InputFailedEvent along with all
> the DataMovementEvent which includes the older versions and the new version
> that was revocated.
> The issue we are seeing is that the events arrive out of order i.e. many
> times we see the older DataMovementEvent first at which our processor thinks
> it is good to start. We then receive the InputFailedEvent and the new version
> of DataMovementEvent, but that's late and the processor fails. This keeps
> repeating on every subsequent task attempt and the task fails.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
