[
https://issues.apache.org/jira/browse/KAFKA-8737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang resolved KAFKA-8737.
----------------------------------
Assignee: Guozhang Wang (was: Bill Bejeck)
Resolution: Duplicate
> TaskMigrated Exception while rebalancing kafka streams
> ------------------------------------------------------
>
> Key: KAFKA-8737
> URL: https://issues.apache.org/jira/browse/KAFKA-8737
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 1.0.0, 1.0.1
> Environment: 20 partitions
> 1 topic
> 8 Streamer service
> topic-region-1 9 7841726 8236017
> 394291
> streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/
>
> streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer
> topic-region-1 15 7421710 7467666
> 45956
> streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/
>
> streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer
> topic-region-1 19 7737360 8120611
> 383251
> streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/
>
> streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer
> topic-region-1
> Reporter: KUMAR
> Assignee: Guozhang Wang
> Priority: Major
>
> Kafka streams throws following exception while restart of a stream client
> service -
> o.a.k.s.p.internals.StreamThread.? - stream-thread
> [streams-subscriberstopic-region-1-32d968e3-f892-4772-a7a4-6f684d7e43c9-StreamThread-1]
> Detected a task that got migrated to another thread. This implies that this
> thread missed a rebalance and dropped out of the consumer group. Trying to
> rejoin the consumer group now.
> org.apache.kafka.streams.errors.TaskMigratedException: Log end offset of
> topic-region-1-12 should not change while restoring: old end offset 6286727,
> current offset 6380997
>
> Kafka version is 1.0.0 and we have back merged the fix for KIP-6269-
> [https://github.com/apache/kafka/pull/4300/files#|https://github.com/apache/kafka/pull/4300/files]
> However we observe that there seems to be an issue in rebalance when
> "auto.offset.reset" is configured as "latest". Based on log analysis we see
> following behavior -
> # StreamThread starts a restore consumer
> # While Fetching it gets offset out of range
> o.a.k.c.consumer.internals.Fetcher.? - [Consumer
> clientId=streams-subscriberstopic-region-1-11b2d7fb-11ce-4b0b-a40a-388d3c7b6bc9-StreamThread-1-restore-
> consumer, groupId=] Fetch READ_UNCOMMITTED at offset 246431 for partition
> topic-region-1-12 returned fetch data (error=OFFSET_OUT_OF_RANGE,
> highWaterMark=-1, lastStableOffset = -1, logStartOffset = -1,
> abortedTransactions = null, recordsSizeInBytes=0)
> # Fetcher tries to reset the offset
> # While reset the offset it appears it is changing the offset position and
> causing TaskMigrated exception
> Above test repeated with "auto.offset.reset" is configured as "earliest" does
> not throw any TaskMigrated exception as in earliest case we are not reseting
> the restore consumer position.
>
> Please let us know if this is possible and if a fix would be needed for the
> offset reset piece when set to latest.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
