[
https://issues.apache.org/jira/browse/KAFKA-12740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
A. Sophie Blee-Goldman updated KAFKA-12740:
-------------------------------------------
Description:
Building off of that, we can go one step further and avoid duplicate work
within the subtopology itself. Any time a record is partially processed through
a subtopology before hitting an error, all of the processors up to that point
will be applied again when the record is reprocessed. If we can keep track of
how far along the subtopology a record was processed, then we can start
reprocessing it from the last processor it had cleared before hitting an error.
We’ll need to make sure to commit and/or flush everything up to that point in
the subtopology as well.
This proposal is the most likely to benefit from letting a StreamThread recover
after an unexpected exception rather than letting it die and starting up a new
one, as we don’t need to worry about handing anything off from the dying thread
to its replacement.
was:
Building off of that, we can go one step further and avoid duplicate work
within the subtopology itself. Any time a record is partially processed through
a subtopology before hitting an error, all of the processors up to that point
will be applied again when the record is reprocessed. If we can keep track of
how far along the subtopology a record was processed, then we can start
reprocessing it from the last processor it had cleared before hitting an error.
This proposal is the most likely to benefit from letting a StreamThread recover
after an unexpected exception rather than letting it die and starting up a new
one, as we don’t need to worry about handing anything off from the dying thread
to its replacement.
> 3. Resume processing from last-cleared processor after soft crash
> -----------------------------------------------------------------
>
> Key: KAFKA-12740
> URL: https://issues.apache.org/jira/browse/KAFKA-12740
> Project: Kafka
> Issue Type: Sub-task
> Components: streams
> Reporter: A. Sophie Blee-Goldman
> Priority: Major
>
> Building off of that, we can go one step further and avoid duplicate work
> within the subtopology itself. Any time a record is partially processed
> through a subtopology before hitting an error, all of the processors up to
> that point will be applied again when the record is reprocessed. If we can
> keep track of how far along the subtopology a record was processed, then we
> can start reprocessing it from the last processor it had cleared before
> hitting an error. We’ll need to make sure to commit and/or flush everything
> up to that point in the subtopology as well.
> This proposal is the most likely to benefit from letting a StreamThread
> recover after an unexpected exception rather than letting it die and starting
> up a new one, as we don’t need to worry about handing anything off from the
> dying thread to its replacement.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
