[jira] [Commented] (KAFKA-12279) Kafka 2.7 stream app issue
[ https://issues.apache.org/jira/browse/KAFKA-12279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279938#comment-17279938 ] A. Sophie Blee-Goldman commented on KAFKA-12279: Root cause: the source topic was deleted and recreated with a smaller number of partitions. However, the app was not reset and the local state was not cleared, which left the task directory for 0_13 on disk. Since the new input topics had fewer than 13 partitions, this task was no longer recognized. Since decreasing the partition number on-the-fly without resetting the application is not supported, I'm going to close this as "Not a Bug" > Kafka 2.7 stream app issue > -- > > Key: KAFKA-12279 > URL: https://issues.apache.org/jira/browse/KAFKA-12279 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.7.0 >Reporter: prabhu biradar >Priority: Critical > > After starting the stream application below exception is thrown and threads > stop processing. > 2021-02-02T22:29:10.416::AccountId ::Partition ::Offset > ::[com.xx.xx.xx.xx.xx-client-StreamThread-1]::ERROR::o.a.k.s.processor.internals.StreamThread::stream-thread > [com.xx.xx.xx.xx.xx-client-StreamThread-1] Encountered the following > exception during processing and the thread is going to shut down: > 2021-02-02T22:29:10.416::AccountId ::Partition ::Offset > ::[com.xx.xx.xx.xx.xx-client-StreamThread-1]::ERROR::o.a.k.s.processor.internals.StreamThread::stream-thread > [com.xx.xx.xx.xx.xx-client-StreamThread-1] Encountered the following > exception during processing and the thread is going to shut down: > java.lang.IllegalStateException: Tried to lookup lag for unknown task 0_13 at > org.apache.kafka.streams.processor.internals.assignment.ClientState.lagFor(ClientState.java:318) > at java.util.Comparator.lambda$comparingLong$6043328a$1(Comparator.java:511) > at java.util.Comparator.lambda$thenComparing$36697e65$1(Comparator.java:216) > at java.util.TreeMap.put(TreeMap.java:552) at > java.util.TreeSet.add(TreeSet.java:255) at > java.util.AbstractCollection.addAll(AbstractCollection.java:344) at > java.util.TreeSet.addAll(TreeSet.java:312) at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.getPreviousTasksByLag(StreamsPartitionAssignor.java:1299) > at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assignTasksToThreads(StreamsPartitionAssignor.java:1213) > at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.computeNewAssignment(StreamsPartitionAssignor.java:939) > at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assign(StreamsPartitionAssignor.java:393) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:589) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:689) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordinator.java:111) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:599) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:562) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1164) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1139) > at > org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:206) > at > org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:169) > at > org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:129) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:602) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:412) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1292) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1233) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:801) > at > org.apach
[jira] [Commented] (KAFKA-12279) Kafka 2.7 stream app issue
[ https://issues.apache.org/jira/browse/KAFKA-12279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279097#comment-17279097 ] prabhu biradar commented on KAFKA-12279: Hi Sophie Blee-Goldman, 1) This error occurs on every restart post 2.7 Kafka upgrade. 2) Yes we are running 12 instances (6 ECS TASK and each TASKk runs 2 threads) 3) Below is the detailed logs starting from application restart. Completed 23.2 KiB/23.2 KiB (228.6 KiB/s) with 1 file(s) remainingCompleted 23.2 KiB/23.2 KiB (228.6 KiB/s) with 1 file(s) remainingdownload: s3://x/perf/kafka-client-truststore.jks to ./client.truststore.jksCompleted 256.0 KiB/20.9 MiB (1.9 MiB/s) with 1 file(s) remainingCompleted 512.0 KiB/20.9 MiB (3.7 MiB/s) with 1 file(s) remainingCompleted 768.0 KiB/20.9 MiB (5.4 MiB/s) with 1 file(s) remainingCompleted 1.0 MiB/20.9 MiB (7.2 MiB/s) with 1 file(s) remaining Completed 1.2 MiB/20.9 MiB (8.9 MiB/s) with 1 file(s) remaining Completed 1.5 MiB/20.9 MiB (10.5 MiB/s) with 1 file(s) remaining Completed 1.8 MiB/20.9 MiB (12.1 MiB/s) with 1 file(s) remaining Completed 2.0 MiB/20.9 MiB (13.8 MiB/s) with 1 file(s) remaining Completed 2.2 MiB/20.9 MiB (15.3 MiB/s) with 1 file(s) remaining Completed 2.5 MiB/20.9 MiB (16.9 MiB/s) with 1 file(s) remaining Completed 2.8 MiB/20.9 MiB (18.3 MiB/s) with 1 file(s) remaining Completed 3.0 MiB/20.9 MiB (19.9 MiB/s) with 1 file(s) remaining Completed 3.2 MiB/20.9 MiB (21.4 MiB/s) with 1 file(s) remaining Completed 3.5 MiB/20.9 MiB (22.8 MiB/s) with 1 file(s) remaining Completed 3.8 MiB/20.9 MiB (24.3 MiB/s) with 1 file(s) remaining Completed 4.0 MiB/20.9 MiB (25.8 MiB/s) with 1 file(s) remaining Completed 4.2 MiB/20.9 MiB (27.0 MiB/s) with 1 file(s) remaining Completed 4.5 MiB/20.9 MiB (28.4 MiB/s) with 1 file(s) remaining Completed 4.8 MiB/20.9 MiB (29.9 MiB/s) with 1 file(s) remaining Completed 5.0 MiB/20.9 MiB (31.1 MiB/s) with 1 file(s) remaining Completed 5.2 MiB/20.9 MiB (32.4 MiB/s) with 1 file(s) remaining Completed 5.5 MiB/20.9 MiB (33.7 MiB/s) with 1 file(s) remaining Completed 5.8 MiB/20.9 MiB (34.9 MiB/s) with 1 file(s) remaining Completed 6.0 MiB/20.9 MiB (36.1 MiB/s) with 1 file(s) remaining Completed 6.2 MiB/20.9 MiB (37.4 MiB/s) with 1 file(s) remaining Completed 6.5 MiB/20.9 MiB (38.7 MiB/s) with 1 file(s) remaining Completed 6.8 MiB/20.9 MiB (39.9 MiB/s) with 1 file(s) remaining Completed 7.0 MiB/20.9 MiB (41.1 MiB/s) with 1 file(s) remaining Completed 7.2 MiB/20.9 MiB (42.3 MiB/s) with 1 file(s) remaining Completed 7.5 MiB/20.9 MiB (43.5 MiB/s) with 1 file(s) remaining Completed 7.8 MiB/20.9 MiB (44.6 MiB/s) with 1 file(s) remaining Completed 8.0 MiB/20.9 MiB (45.7 MiB/s) with 1 file(s) remaining Completed 8.2 MiB/20.9 MiB (47.1 MiB/s) with 1 file(s) remaining Completed 8.5 MiB/20.9 MiB (48.1 MiB/s) with 1 file(s) remaining Completed 8.8 MiB/20.9 MiB (49.1 MiB/s) with 1 file(s) remaining Completed 9.0 MiB/20.9 MiB (50.2 MiB/s) with 1 file(s) remaining Completed 9.2 MiB/20.9 MiB (51.5 MiB/s) with 1 file(s) remaining Completed 9.5 MiB/20.9 MiB (52.3 MiB/s) with 1 file(s) remaining Completed 9.8 MiB/20.9 MiB (53.4 MiB/s) with 1 file(s) remaining Completed 10.0 MiB/20.9 MiB (54.5 MiB/s) with 1 file(s) remainingCompleted 10.2 MiB/20.9 MiB (55.3 MiB/s) with 1 file(s) remainingCompleted 10.5 MiB/20.9 MiB (56.3 MiB/s) with 1 file(s) remainingCompleted 10.8 MiB/20.9 MiB (57.4 MiB/s) with 1 file(s) remainingCompleted 11.0 MiB/20.9 MiB (58.3 MiB/s) with 1 file(s) remainingCompleted 11.2 MiB/20.9 MiB (59.3 MiB/s) with 1 file(s) remainingCompleted 11.5 MiB/20.9 MiB (60.2 MiB/s) with 1 file(s) remainingCompleted 11.8 MiB/20.9 MiB (61.1 MiB/s) with 1 file(s) remainingCompleted 12.0 MiB/20.9 MiB (62.0 MiB/s) with 1 file(s) remainingCompleted 12.2 MiB/20.9 MiB (63.0 MiB/s) with 1 file(s) remainingCompleted 12.5 MiB/20.9 MiB (63.9 MiB/s) with 1 file(s) remainingCompleted 12.8 MiB/20.9 MiB (64.7 MiB/s) with 1 file(s) remainingCompleted 13.0 MiB/20.9 MiB (65.7 MiB/s) with 1 file(s) remainingCompleted 13.2 MiB/20.9 MiB (66.5 MiB/s) with 1 file(s) remainingCompleted 13.5 MiB/20.9 MiB (67.4 MiB/s) with 1 file(s) remainingCompleted 13.8 MiB/20.9 MiB (68.3 MiB/s) with 1 file(s) remainingCompleted 14.0 MiB/20.9 MiB (69.2 MiB/s) with 1 file(s) remainingCompleted 14.2 MiB/20.9 MiB (70.0 MiB/s) with 1 file(s) remainingCompleted 14.5 MiB/20.9 MiB (71.0 MiB/s) with 1 file(s) remainingCompleted 14.8 MiB/20.9 MiB (71.7 MiB/s) with 1 file(s) remainingCompleted 15.0 MiB/20.9 MiB (72.5 MiB/s) with 1 file(s) remainingCompleted 15.2 MiB/20.9 MiB (73.5 MiB/s) with 1 file(s) remainingCompleted 15.5 MiB/20.9 MiB (74.2 MiB/s) with 1 file(s) remainingCompleted 15.8 MiB/20.9 MiB (75.1 MiB/s) with 1 file(s) remainingCompleted 16.0 MiB/20.9 MiB (76.1 MiB/s) with 1 file(s) remainingCompleted 16.1 MiB/20.9 MiB (76.0 MiB/s) with 1 file(s) remainingComplet
[jira] [Commented] (KAFKA-12279) Kafka 2.7 stream app issue
[ https://issues.apache.org/jira/browse/KAFKA-12279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277604#comment-17277604 ] A. Sophie Blee-Goldman commented on KAFKA-12279: Hi [~pbiradar], thanks for reporting this. Can you give us some additional context – for example, did this occur after a restart or during your upgrade? Are you running multiple instances? Additional logs, more than anything, would be helpful > Kafka 2.7 stream app issue > -- > > Key: KAFKA-12279 > URL: https://issues.apache.org/jira/browse/KAFKA-12279 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: prabhu biradar >Priority: Major > Fix For: 2.7.0 > > > After starting the stream application below exception is thrown and threads > stop processing. > 2021-02-02T22:29:10.416::AccountId ::Partition ::Offset > ::[com.xx.xx.xx.xx.xx-client-StreamThread-1]::ERROR::o.a.k.s.processor.internals.StreamThread::stream-thread > [com.xx.xx.xx.xx.xx-client-StreamThread-1] Encountered the following > exception during processing and the thread is going to shut down: > 2021-02-02T22:29:10.416::AccountId ::Partition ::Offset > ::[com.xx.xx.xx.xx.xx-client-StreamThread-1]::ERROR::o.a.k.s.processor.internals.StreamThread::stream-thread > [com.xx.xx.xx.xx.xx-client-StreamThread-1] Encountered the following > exception during processing and the thread is going to shut down: > java.lang.IllegalStateException: Tried to lookup lag for unknown task 0_13 at > org.apache.kafka.streams.processor.internals.assignment.ClientState.lagFor(ClientState.java:318) > at java.util.Comparator.lambda$comparingLong$6043328a$1(Comparator.java:511) > at java.util.Comparator.lambda$thenComparing$36697e65$1(Comparator.java:216) > at java.util.TreeMap.put(TreeMap.java:552) at > java.util.TreeSet.add(TreeSet.java:255) at > java.util.AbstractCollection.addAll(AbstractCollection.java:344) at > java.util.TreeSet.addAll(TreeSet.java:312) at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.getPreviousTasksByLag(StreamsPartitionAssignor.java:1299) > at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assignTasksToThreads(StreamsPartitionAssignor.java:1213) > at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.computeNewAssignment(StreamsPartitionAssignor.java:939) > at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assign(StreamsPartitionAssignor.java:393) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:589) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:689) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordinator.java:111) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:599) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:562) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1164) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1139) > at > org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:206) > at > org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:169) > at > org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:129) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:602) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:412) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1292) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1233) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:801) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollPhase(StreamThread.java:763) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:614) > at > org.apache.k
