[jira] [Created] (KAFKA-9995) IllegalStateException when offsets not found
James Hay created KAFKA-9995:
Summary: IllegalStateException when offsets not found
Key: KAFKA-9995
URL: https://issues.apache.org/jira/browse/KAFKA-9995
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.4.1
Reporter: James Hay
I have a recently upgraded Kafka Streams 2.4.1 application and we have started
seeing the application periodically crash due to the following error:
{code:java}
2020-05-14T16:53:03.839Z DEBUG <>
[chat-c199e52b-9344-457c-a2ae-04a669a7e0fb-StreamThread-2]
o.a.k.c.c.i.ConsumerCoordinator - [Consumer
clientId=chat-c199e52b-9344-457c-a2ae-04a669a7e0fb-StreamThread-2-consumer,
groupId=chat] Fetching committed offsets for partitions:
[private.chat.endpoint-0, public.chat.message-0] 2020-05-14T16:53:03.841Z INFO
<> [chat-c199e52b-9344-457c-a2ae-04a669a7e0fb-StreamThread-2]
o.a.k.c.c.i.ConsumerCoordinator - [Consumer
clientId=chat-c199e52b-9344-457c-a2ae-04a669a7e0fb-StreamThread-2-consumer,
groupId=chat] Found no committed offset for partition private.chat.endpoint-0
2020-05-14T16:53:03.842Z ERROR <>
[chat-c199e52b-9344-457c-a2ae-04a669a7e0fb-StreamThread-2]
o.a.k.s.p.internals.StreamThread - stream-thread
[chat-c199e52b-9344-457c-a2ae-04a669a7e0fb-StreamThread-2] Encountered the
following error during processing: java.lang.IllegalStateException: Offset
limit should monotonically increase, but was reduced. New limit: 0. Previous
limit: 857859at
org.apache.kafka.streams.processor.internals.StandbyTask.updateOffsetLimits(StandbyTask.java:215)
at
org.apache.kafka.streams.processor.internals.StandbyTask.update(StandbyTask.java:181)
at
org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1048)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:825)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:698)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:671)
{code}
Is this a known issue? What can cause offsets to not be found?
Other things to note:
* We have TOPOLOGY_OPTIMIZATION set to OPTIMIZE
* We are only seeing this on two of our three environments (at the moment).
The two environments we are seeing a problem have little traffic and only seems
to impact the machines that are on the whole idle. Our prod environment which
consumes regular events is showing no signs of having the same problem.
* There is some evidence to suggest there is a pattern to the timing of this
error. Although not always the case, 24hrs between errors is common.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-7424) State stores restoring from changelog topic not the source topic
[ https://issues.apache.org/jira/browse/KAFKA-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Hay updated KAFKA-7424: - Description: Hi, I've recently attempted to upgrade a streams application form 1.1 to 1.1.1 and I noticed a drop in the number of messages being restored in our state stores. It appears that there is a change in 1.1.1 which causes our state stores to be restored from the changelog topic as opposed to version 1.1 where the stores are restored from the source topic. In our application this causes an issue as we switched to StreamsBuilder from KStreamBuilder in the middle of the applications lifetime and so the changelog doesn't represent a full history of the source topic. Has this switch been introduced intentionally? Is there a way to configure our application to use 1.1.1 and still use the source stream to restore state stores? Any recommendations on getting our changelog in sync with the source? Thanks was: Hi, I've recently attempted to upgrade a streams application form 1.1 to 1.1.1 and I noticed a drop in the number of messages being restored in our state stores. It appears that there is a change in 1.1.1 which causes our state stores to be restored from the changelog topic as opposed to version 1.1 where the stores are restored from the source topic. In our application this causes an issue as we switched to StreamsBuilder from KStreamBuilder in the middle of the applications lifetime and so the changelog doesn't represent a full history of the source topic. Has this switch been introduced intentionally? Is there a way to configure our application to use 1.1.1 and still use the source stream to restore state stores? Any recommendations on getting our changelog in sync with the source? Thanks > State stores restoring from changelog topic not the source topic > > > Key: KAFKA-7424 > URL: https://issues.apache.org/jira/browse/KAFKA-7424 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 1.1.1 >Reporter: James Hay >Priority: Critical > > Hi, > > I've recently attempted to upgrade a streams application form 1.1 to 1.1.1 > and I noticed a drop in the number of messages being restored in our state > stores. > It appears that there is a change in 1.1.1 which causes our state stores to > be restored from the changelog topic as opposed to version 1.1 where the > stores are restored from the source topic. In our application this causes an > issue as we switched to StreamsBuilder from KStreamBuilder in the middle of > the applications lifetime and so the changelog doesn't represent a full > history of the source topic. > Has this switch been introduced intentionally? Is there a way to configure > our application to use 1.1.1 and still use the source stream to restore state > stores? Any recommendations on getting our changelog in sync with the source? > > Thanks > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
