[ https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16605851#comment-16605851 ]
Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 9/6/18 2:40 PM: --------------------------------------------------------------------------- Problem is that KSTREAM-SOURCE-XXXXXXXXX is mostly KSTREAM-SOURCE-000000000 independently of which process and how much processes are running (or trying to run). How I reproduce error at my side. Let's assume I have low message flow < 100 Msg/sec Msg size ~ 1kB. I am starting app using streaming API. This app reads from 30 topics and send messages to 1 topic. Let's give this app 300MB JVM Heap. It is starting. Cool. At second server I am starting. second instance. The same. It is starting. The other case. Let's assume I have a bit higher message flow > 5000 Msg/sec Msg size ~ 1kB. I am starting app using streaming API. This app reads from 30 topics and send messages to 1 topic. Let's give this app 300 MB JVM Heap. It is not starting, even if memory spec stays that it is enough to calculate 30 sec of messages. 5000 Msg/s ~ 150 000 Mgs/30 sec ~ 150 MB. I am giving to app 2GB Heap. Is starting. Everything between 300 MB and 2 GB leads at some point to yet another mystic crasches. At second server I am starting. second instance. If I am starting it with 300 MB - I got immediately this error. Application tries to starrt, but then I got this error and all affected topics are goig to be dead. If I am giving 1GB, it is better application works some hours, but any minimal peak aroud 5000 Msg/s to e.g. 7000 Msg/s, causes the same. Finally - now - I am starting processes with 5GB. they could work continuously like 2-4 days. I am sorry I have no better description. Once I tried to start TRACE level logs in Kafka, but this is impossible with message flow at 5000 Msg/s. was (Author: habdank): Problem is that KSTREAM-SOURCE-XXXXXXXXX is mostly KSTREAM-SOURCE-000000000 independently of which process and how much processes are running (or trying to run). How I reproduce error at my side. Let's assume I have low message flow < 100 Msg/sec Msg size ~ 1kB. I am starting app using streaming API. This app reads from 30 topics and send messages to 1 topic. Let's give this app 300MB JVM Heap. It is starting. Cool. At second server I am starting. second instance. The same. It is starting. The other case. Let's assume I have a bit higher message flow > 5000 Msg/sec Msg size ~ 1kB. I am starting app using streaming API. This app reads from 30 topics and send messages to 1 topic. Let's give this app 300 MB JVM Heap. It is not starting, even in memory spec stays that it is enough to calculate 30 sec of messages. 5000 Msg/s ~ 150 000 Mgs/30 sec ~ 150 MB. I am giving to app 2GB Heap. Is starting. Everything between 300 MB and 2 GB leads at some point to yet another mystic crasches. At second server I am starting. second instance. If I am starting it with 300 MB - I got immediately this error. Application tries to starrt, but then I got this error and all affected topics are goig to be dead. If I am giving 1GB, it is better application works some hours, but any minimal peak aroud 5000 Msg/s to e.g. 7000 Msg/s, causes the same. Finally - now - I am starting processes with 5GB. they could work continuously like 2-4 days. I am sorry I have no better description. Once I tried to start TRACE level logs in Kafka, but this is impossible with message flow at 5000 Msg/s. > Mystic FATAL error > ------------------ > > Key: KAFKA-7214 > URL: https://issues.apache.org/jira/browse/KAFKA-7214 > Project: Kafka > Issue Type: Bug > Components: streams > Affects Versions: 0.11.0.3, 1.1.1 > Reporter: Seweryn Habdank-Wojewodzki > Priority: Critical > > Dears, > Very often at startup of the streaming application I got exception: > {code} > Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-0000000000, > topic=my_instance_medium_topic, partition=1, offset=198900203; > [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212), > > org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347), > > org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420), > > org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339), > > org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648), > > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513), > > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482), > > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)] > in thread > my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62 > {code} > and then (without shutdown request from my side): > {code} > 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread > [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62] > State transition from PENDING_SHUTDOWN to DEAD. > {code} > What is this? > How to correctly handle it? > Thanks in advance for help. -- This message was sent by Atlassian JIRA (v7.6.3#76005)