[
https://issues.apache.org/jira/browse/KAFKA-13295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17422290#comment-17422290
]
Sagar Rao commented on KAFKA-13295:
-----------------------------------
[~ableegoldman], yes it does. I wanted to ask the same question, do we need to
commit offsets for all the tasks or reduce it somehow and you have answered it.
I have a few of more questions ->
1) [~guozhang] has mentioned that we can invoke commitOffsetOrTransaction in
the onPartitionAssigned method. However, in StreamPartitionRebalanceListener,
there's a comment stating that:
{code:java}
//all task management is already handled by:
//
org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.onAssignment{code}
So, do we need to add the logic in onAssignment? I haven't quite looked at the
flow entirely and hence asking this question
2) How do we handle cases if a TaskCorruptedException or TimeoutException is
thrown during commiting. I see these have been handled in handleRevocation
whereby these are considered dirtyTasks and then closed. So, how do we handle
failure scenarios for offset committing ?
3) Do we also need to write checkpoints post committing?
> Long restoration times for new tasks can lead to transaction timeouts
> ---------------------------------------------------------------------
>
> Key: KAFKA-13295
> URL: https://issues.apache.org/jira/browse/KAFKA-13295
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: A. Sophie Blee-Goldman
> Assignee: Sagar Rao
> Priority: Critical
> Labels: eos
> Fix For: 3.1.0
>
>
> In some EOS applications with relatively long restoration times we've noticed
> a series of ProducerFencedExceptions occurring during/immediately after
> restoration. The broker logs were able to confirm these were due to
> transactions timing out.
> In Streams, it turns out we automatically begin a new txn when calling
> {{send}} (if there isn’t already one in flight). A {{send}} occurs often
> outside a commit during active processing (eg writing to the changelog),
> leaving the txn open until the next commit. And if a StreamThread has been
> actively processing when a rebalance results in a new stateful task without
> revoking any existing tasks, the thread won’t actually commit this open txn
> before it goes back into the restoration phase while it builds up state for
> the new task. So the in-flight transaction is left open during restoration,
> during which the StreamThread only consumes from the changelog without
> committing, leaving it vulnerable to timing out when restoration times exceed
> the configured transaction.timeout.ms for the producer client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
