[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error
[
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612374#comment-16612374
]
Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 9/12/18 3:45 PM:
[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when
they need to handle the case below?
{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled
exception: Exception caught in process. taskId=0_2,
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2,
offset=1892533025;
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
in thread
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State
transition from RUNNING to PENDING_SHUTDOWN
{code}
* What is wrong there?
* What are the steps to avoid this in the future?
* How to repair the situation?
was (Author: habdank):
[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when
they need to handle the case below?
{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled
exception: Exception caught in process. taskId=0_2,
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2,
offset=1892533025;
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
in thread
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State
transition from RUNNING to PENDING_SHUTDOWN
{code}
What is wrong there? What are the steps to avoid this in the future? How to
repair the situation?
> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn
[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error
[
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612374#comment-16612374
]
Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 9/12/18 3:45 PM:
[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when
they need to handle the case below?
{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled
exception: Exception caught in process. taskId=0_2,
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2,
offset=1892533025;
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
in thread
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State
transition from RUNNING to PENDING_SHUTDOWN
{code}
What is wrong there? What are the steps to avoid this in the future? How to
repair the situation?
was (Author: habdank):
[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when
they need to handle the case below?
{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled
exception: Exception caught in process. taskId=0_2,
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2,
offset=1892533025;
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
in thread
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State
transition from RUNNING to PENDING_SHUTDOWN
{code}
What is wrong there? What are the steps to avoid this in the future?
> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
>
[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error
[
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612374#comment-16612374
]
Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 9/12/18 3:44 PM:
[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when
they need to handle the case below?
{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled
exception: Exception caught in process. taskId=0_2,
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2,
offset=1892533025;
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
in thread
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State
transition from RUNNING to PENDING_SHUTDOWN
{code}
What is wrong there? What are the steps to avoid this in the future?
was (Author: habdank):
[~vvcephei]
Back to the roots. What shall I say to Maintenance and Operations staff, when
they need to handle the case below?
{code}
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51]
Shutdown complete
2018-09-06 10:05:21 [ar313] [FATAL] SingleTopicstreamer:102 - Caught unhandled
exception: Exception caught in process. taskId=0_2,
processor=KSTREAM-SOURCE-00, topic=ar313_medium_topic, partition=2,
offset=1892533025;
[org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:240),
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:94),
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:411),
org.apache.kafka.streams.processor.internals.StreamThread.processAndMaybeCommit(StreamThread.java:922),
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:802),
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:749),
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:719)]
in thread
streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-51
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:200 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
State transition from PENDING_SHUTDOWN to DEAD
2018-09-06 10:05:21 [ar313] [INFO ] StreamThread:1128 - stream-thread
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e-StreamThread-58]
Shutdown complete
2018-09-06 10:05:21 [ar313] [INFO ] KafkaStreams:261 - stream-client
[streamer-ar313-ar313_medium-15864802-2c1b-47e6-90f4-80b8b4fe4c3e] State
transition from RUNNING to PENDING_SHUTDOWN
{code}
What is wrong there?
> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error
[
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607436#comment-16607436
]
John Roesler edited comment on KAFKA-7214 at 9/7/18 6:00 PM:
-
Since you do not have an OutOfMemoryException in your logs, I can only assume
your application did not run out of memory. How can we report "out of memory"
if the application is not actually out of memory?
Your problem seems to be caused by long GC pauses, not running out of memory,
but we cannot confirm this, since you have not reported your GC logs. You can
enable GC logging (the JVM provides an option to do this) to investigate the
problem further if you really wish to run the app in a memory constrained
environment.
Every aspect of the application's runtime performance, including memory, will
be dominated by what exactly your application does and what data it's
processing. There's no fixed amount of "overhead" in Kafka Streams. Two
different topologies will have different amounts of overhead based on the
computations they need to do.
Honestly, I think the procedure you have followed to set your heap size is
perfectly fine. It's very similar to what I would have done. If you really need
to come up with a formal characterization of the memory usage for _your_
topology in terms of throughput, it's something that can only done by you. The
approach I'd recommend is to run with a few different configurations and
analyze the heap dumps at a few different points in the lifecycle.
This is the same procedure you would follow to characterize the required heap
for any Java application, not just Streams.
About this:
> Memory consumption or memory model described in Kafka documentation does not
>fit to reality.
Can you point me to the documentation that's incorrect? We can certainly fix
anything that's documented wrong.
was (Author: vvcephei):
Since you do not have an OutOfMemoryException in your logs, I can only assume
your application did not run out of memory. How can we report "out of memory"
if the application is not actually out of memory?
Your problem seems to be caused by long GC pauses, not running out of memory,
but we cannot confirm this, since you have not reported your GC logs. You can
enable GC logging (the JVM provides an option to do this) to investigate the
problem further if you really wish to run the app in a memory constrained
environment.
Every aspect of the application's runtime performance, including memory, will
be dominated by what exactly your application does and what data it's
processing. There's no fixed amount of "overhead" in Kafka Streams. Two
different topologies will have different amounts of overhead based on the
computations they need to do.
Honestly, I think the procedure you have followed to set your heap size is
perfectly fine. It's very similar to what I would have done. If you really need
to come up with a formal characterization of the memory usage for _your_
topology in terms of throughput, it's something that can only done by you. The
approach I'd recommend is to run with a few different configurations and
analyze the heap dumps at a few different points in the lifecycle.
This is the same procedure you would follow to characterize the required heap
for any Java application, not just Streams.
> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00,
> topic=my_instance_medium_topic, partition=1, offset=198900203;
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
> in thread
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
>
[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error
[
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605851#comment-16605851
]
Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 9/6/18 2:40 PM:
---
Problem is that KSTREAM-SOURCE-X is mostly KSTREAM-SOURCE-0
independently of which process and how much processes are running (or trying to
run).
How I reproduce error at my side. Let's assume I have low message flow < 100
Msg/sec Msg size ~ 1kB.
I am starting app using streaming API. This app reads from 30 topics and send
messages to 1 topic.
Let's give this app 300MB JVM Heap. It is starting. Cool.
At second server I am starting. second instance. The same. It is starting.
The other case. Let's assume I have a bit higher message flow > 5000 Msg/sec
Msg size ~ 1kB.
I am starting app using streaming API. This app reads from 30 topics and send
messages to 1 topic.
Let's give this app 300 MB JVM Heap. It is not starting, even if memory spec
stays that it is enough to calculate 30 sec of messages.
5000 Msg/s ~ 150 000 Mgs/30 sec ~ 150 MB.
I am giving to app 2GB Heap. Is starting. Everything between 300 MB and 2 GB
leads at some point to yet another mystic crasches.
At second server I am starting. second instance. If I am starting it with 300
MB - I got immediately this error. Application tries to starrt, but then I got
this error and all affected topics are goig to be dead. If I am giving 1GB, it
is better application works some hours, but any minimal peak aroud 5000 Msg/s
to e.g. 7000 Msg/s, causes the same. Finally - now - I am starting processes
with 5GB. they could work continuously like 2-4 days.
I am sorry I have no better description.
Once I tried to start TRACE level logs in Kafka, but this is impossible with
message flow at 5000 Msg/s.
was (Author: habdank):
Problem is that KSTREAM-SOURCE-X is mostly KSTREAM-SOURCE-0
independently of which process and how much processes are running (or trying to
run).
How I reproduce error at my side. Let's assume I have low message flow < 100
Msg/sec Msg size ~ 1kB.
I am starting app using streaming API. This app reads from 30 topics and send
messages to 1 topic.
Let's give this app 300MB JVM Heap. It is starting. Cool.
At second server I am starting. second instance. The same. It is starting.
The other case. Let's assume I have a bit higher message flow > 5000 Msg/sec
Msg size ~ 1kB.
I am starting app using streaming API. This app reads from 30 topics and send
messages to 1 topic.
Let's give this app 300 MB JVM Heap. It is not starting, even in memory spec
stays that it is enough to calculate 30 sec of messages.
5000 Msg/s ~ 150 000 Mgs/30 sec ~ 150 MB.
I am giving to app 2GB Heap. Is starting. Everything between 300 MB and 2 GB
leads at some point to yet another mystic crasches.
At second server I am starting. second instance. If I am starting it with 300
MB - I got immediately this error. Application tries to starrt, but then I got
this error and all affected topics are goig to be dead. If I am giving 1GB, it
is better application works some hours, but any minimal peak aroud 5000 Msg/s
to e.g. 7000 Msg/s, causes the same. Finally - now - I am starting processes
with 5GB. they could work continuously like 2-4 days.
I am sorry I have no better description.
Once I tried to start TRACE level logs in Kafka, but this is impossible with
message flow at 5000 Msg/s.
> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00,
> topic=my_instance_medium_topic, partition=1, offset=198900203;
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
> in thread
>
[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error
[
https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598305#comment-16598305
]
Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 8/31/18 6:27 AM:
The keyword in all those errors is: KSTREAM-SOURCE-X
Kafka 1.1.1 makes it even more horrible :-(, but ...
... it seems this is related to memory consumption and perhaps number of
threads used by streaming app.
I had increased JVM heap from 348 MB to 1GB and decreased number of threads
from 16 to 2 and it seems not happened so often.
I will check this further.
But I am going back to my comment from bug report KAFKA-6777. I think (after
code review), there are very many places in code, where potentially OutOfMemory
error ist not handled properly and they could be converted in any kind of
random errors or even completely swallowed giving random behaviour of clients
or servers.
I would expect, that OutOfMemory will lead to fast application crash with clear
infomation, where is the problem.
was (Author: habdank):
Kafka 1.1.1 makes it even more horrible :-(, but ...
... it seems this is related to memory consumption and perhaps number of
threads used by streaming app.
I had increased JVM heap from 348 MB to 1GB and decreased number of threads
from 16 to 2 and it seems not happened so often.
I will check this further.
But I am going back to my comment from bug report KAFKA-6777. I think (after
code review), there are very many places in code, where potentially OutOfMemory
error ist not handled properly and they could be converted in any kind of
random errors or even completely swallowed giving random behaviour of clients
or servers.
I would expect, that OutOfMemory will lead to fast application crash with clear
infomation, where is the problem.
> Mystic FATAL error
> --
>
> Key: KAFKA-7214
> URL: https://issues.apache.org/jira/browse/KAFKA-7214
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 0.11.0.3, 1.1.1
>Reporter: Seweryn Habdank-Wojewodzki
>Priority: Critical
>
> Dears,
> Very often at startup of the streaming application I got exception:
> {code}
> Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-00,
> topic=my_instance_medium_topic, partition=1, offset=198900203;
> [org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:212),
>
> org.apache.kafka.streams.processor.internals.AssignedTasks$2.apply(AssignedTasks.java:347),
>
> org.apache.kafka.streams.processor.internals.AssignedTasks.applyToRunningTasks(AssignedTasks.java:420),
>
> org.apache.kafka.streams.processor.internals.AssignedTasks.process(AssignedTasks.java:339),
>
> org.apache.kafka.streams.processor.internals.StreamThread.processAndPunctuate(StreamThread.java:648),
>
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:513),
>
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:482),
>
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:459)]
> in thread
> my_application-my_instance-my_instance_medium-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62
> {code}
> and then (without shutdown request from my side):
> {code}
> 2018-07-30 07:45:02 [ar313] [INFO ] StreamThread:912 - stream-thread
> [my_application-my_instance-my_instance-72ee1819-edeb-4d85-9d65-f67f7c321618-StreamThread-62]
> State transition from PENDING_SHUTDOWN to DEAD.
> {code}
> What is this?
> How to correctly handle it?
> Thanks in advance for help.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error
[ https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587475#comment-16587475 ] Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 8/21/18 2:11 PM: Hi, I had updated Kafka client to 1.1.1. I have similar situation. max.poll.interval.ms = 10 000 000 ~ 2,7 Hours max.poll.records=500 Usual system message system processing is ~ 5000 Msg/s 2018-08-21 15:59:22 [] [ERROR] StreamTask:550 - task [0_0] Could not close task due to the following error: org.apache.kafka.streams.errors.TaskMigratedException: StreamsTask taskId: 0_0 ProcessorTopology: KSTREAM-SOURCE-00: topics: [my_topic] children: [KSTREAM-FILTER-01] KSTREAM-FILTER-01: children: [KSTREAM-MAP-02] KSTREAM-MAP-02: children: [KSTREAM-SINK-03] KSTREAM-SINK-03: topic: other_topic Partitions [my_topic-0] at org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets (StreamTask.java:380) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.access$000(St reamTask.java:53) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamT ask.java:316) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measu reLatencyNs(StreamsMetricsImpl.java:211) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.commit(Stream Task.java:307) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.suspend(Strea mTask.java:440) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.close(StreamT ask.java:546) [restreamer.jar:?] at org.apache.kafka.streams.processor.internals.AssignedTasks.close(Assi gnedTasks.java:405) [restreamer.jar:?] at org.apache.kafka.streams.processor.internals.TaskManager.shutdown(Tas kManager.java:260) [restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.completeShu tdown(StreamThread.java:) [restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamT hread.java:730) [restreamer.jar:?] Caused by: org.apache.kafka.clients.consumer.CommitFailedException: Commit canno t be completed since the group has already rebalanced and assigned the partition s to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies th at the poll loop is spending too much time message processing. You can address t his either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. Regards, Seweryn. was (Author: habdank): Hi, I had updated Kafka client to 1.1.1. I have similar. max.poll.interval.ms = 10 000 000 ~ 2,7 Hours max.poll.records=500 Usual system message system processing is ~ 5000 Msg/s 2018-08-21 15:59:22 [] [ERROR] StreamTask:550 - task [0_0] Could not close task due to the following error: org.apache.kafka.streams.errors.TaskMigratedException: StreamsTask taskId: 0_0 ProcessorTopology: KSTREAM-SOURCE-00: topics: [my_topic] children: [KSTREAM-FILTER-01] KSTREAM-FILTER-01: children: [KSTREAM-MAP-02] KSTREAM-MAP-02: children: [KSTREAM-SINK-03] KSTREAM-SINK-03: topic: other_topic Partitions [my_topic-0] at org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets (StreamTask.java:380) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.access$000(St reamTask.java:53) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamT ask.java:316) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measu reLatencyNs(StreamsMetricsImpl.java:211) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.commit(Stream Task.java:307) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.suspend(Strea mTask.java:440) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.close(StreamT ask.java:546) [restreamer.jar:?] at org.apache.kafka.streams.processor.internals.AssignedTasks.close(Assi gnedTasks.java:405) [restreamer.jar:?] at
[jira] [Comment Edited] (KAFKA-7214) Mystic FATAL error
[ https://issues.apache.org/jira/browse/KAFKA-7214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587475#comment-16587475 ] Seweryn Habdank-Wojewodzki edited comment on KAFKA-7214 at 8/21/18 2:10 PM: Hi, I had updated Kafka client to 1.1.1. I have similar. max.poll.interval.ms = 10 000 000 ~ 2,7 Hours max.poll.records=500 Usual system message system processing is ~ 5000 Msg/s 2018-08-21 15:59:22 [] [ERROR] StreamTask:550 - task [0_0] Could not close task due to the following error: org.apache.kafka.streams.errors.TaskMigratedException: StreamsTask taskId: 0_0 ProcessorTopology: KSTREAM-SOURCE-00: topics: [my_topic] children: [KSTREAM-FILTER-01] KSTREAM-FILTER-01: children: [KSTREAM-MAP-02] KSTREAM-MAP-02: children: [KSTREAM-SINK-03] KSTREAM-SINK-03: topic: other_topic Partitions [my_topic-0] at org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets (StreamTask.java:380) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.access$000(St reamTask.java:53) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamT ask.java:316) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measu reLatencyNs(StreamsMetricsImpl.java:211) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.commit(Stream Task.java:307) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.suspend(Strea mTask.java:440) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.close(StreamT ask.java:546) [restreamer.jar:?] at org.apache.kafka.streams.processor.internals.AssignedTasks.close(Assi gnedTasks.java:405) [restreamer.jar:?] at org.apache.kafka.streams.processor.internals.TaskManager.shutdown(Tas kManager.java:260) [restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.completeShu tdown(StreamThread.java:) [restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamT hread.java:730) [restreamer.jar:?] Caused by: org.apache.kafka.clients.consumer.CommitFailedException: Commit canno t be completed since the group has already rebalanced and assigned the partition s to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies th at the poll loop is spending too much time message processing. You can address t his either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. Regards, Seweryn. was (Author: habdank): Hi, I had updated Kafka client to 1.1.1. I have similar. max.poll.interval.ms = 10 000 000 ~ 2,7 Hours max.poll.records=500 Usual system message system processing is ~ 5000 Msg/s 2018-08-21 15:59:22 [] [ERROR] StreamTask:550 - task [0_0] Could not close task due to the following error: org.apache.kafka.streams.errors.TaskMigratedException: StreamsTask taskId: 0_0 ProcessorTopology: KSTREAM-SOURCE-00: topics: [my_topic] children: [KSTREAM-FILTER-01] KSTREAM-FILTER-01: children: [KSTREAM-MAP-02] KSTREAM-MAP-02: children: [KSTREAM-SINK-03] KSTREAM-SINK-03: topic: other_topic Partitions [my_topic-0] at org.apache.kafka.streams.processor.internals.StreamTask.commitOffsets (StreamTask.java:380) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.access$000(St reamTask.java:53) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamT ask.java:316) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measu reLatencyNs(StreamsMetricsImpl.java:211) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.commit(Stream Task.java:307) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.suspend(Strea mTask.java:440) ~[restreamer.jar:?] at org.apache.kafka.streams.processor.internals.StreamTask.close(StreamT ask.java:546) [restreamer.jar:?] at org.apache.kafka.streams.processor.internals.AssignedTasks.close(Assi gnedTasks.java:405) [restreamer.jar:?] at
