James Hay created KAFKA-9995: -------------------------------- Summary: IllegalStateException when offsets not found Key: KAFKA-9995 URL: https://issues.apache.org/jira/browse/KAFKA-9995 Project: Kafka Issue Type: Bug Components: streams Affects Versions: 2.4.1 Reporter: James Hay
I have a recently upgraded Kafka Streams 2.4.1 application and we have started seeing the application periodically crash due to the following error: {code:java} 2020-05-14T16:53:03.839Z DEBUG <> [chat-c199e52b-9344-457c-a2ae-04a669a7e0fb-StreamThread-2] o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=chat-c199e52b-9344-457c-a2ae-04a669a7e0fb-StreamThread-2-consumer, groupId=chat] Fetching committed offsets for partitions: [private.chat.endpoint-0, public.chat.message-0] 2020-05-14T16:53:03.841Z INFO <> [chat-c199e52b-9344-457c-a2ae-04a669a7e0fb-StreamThread-2] o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=chat-c199e52b-9344-457c-a2ae-04a669a7e0fb-StreamThread-2-consumer, groupId=chat] Found no committed offset for partition private.chat.endpoint-0 2020-05-14T16:53:03.842Z ERROR <> [chat-c199e52b-9344-457c-a2ae-04a669a7e0fb-StreamThread-2] o.a.k.s.p.internals.StreamThread - stream-thread [chat-c199e52b-9344-457c-a2ae-04a669a7e0fb-StreamThread-2] Encountered the following error during processing: java.lang.IllegalStateException: Offset limit should monotonically increase, but was reduced. New limit: 0. Previous limit: 857859 at org.apache.kafka.streams.processor.internals.StandbyTask.updateOffsetLimits(StandbyTask.java:215) at org.apache.kafka.streams.processor.internals.StandbyTask.update(StandbyTask.java:181) at org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1048) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:825) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:698) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:671) {code} Is this a known issue? What can cause offsets to not be found? Other things to note: * We have TOPOLOGY_OPTIMIZATION set to OPTIMIZE * We are only seeing this on two of our three environments (at the moment). The two environments we are seeing a problem have little traffic and only seems to impact the machines that are on the whole idle. Our prod environment which consumes regular events is showing no signs of having the same problem. * There is some evidence to suggest there is a pattern to the timing of this error. Although not always the case, 24hrs between errors is common. -- This message was sent by Atlassian Jira (v8.3.4#803005)