[jira] [Updated] (KAFKA-8922) Failed to get end offsets for topic partitions of global store
[
https://issues.apache.org/jira/browse/KAFKA-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raman Gupta updated KAFKA-8922:
---
Description:
I have a Kafka stream that fails with this error on startup every time:
{code}
org.apache.kafka.streams.errors.StreamsException: Failed to get end offsets for
topic partitions of global store test-uiService-dlq-events-table-store after 0
retry attempts. You can increase the number of retries via configuration
parameter `retries`.
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.register(GlobalStateManagerImpl.java:186)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:101)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:207)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.KeyValueToTimestampedKeyValueByteStoreAdapter.init(KeyValueToTimestampedKeyValueByteStoreAdapter.java:87)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:112)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:123)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread.initialize(GlobalStreamThread.java:345)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:270)
~[kafka-streams-2.3.0.jar:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to get
offsets by times in 30001ms
{code}
The stream was working fine and then this started happening.
The stream now throws this error on every start. I am now going to attempt to
reset the stream and delete its local state.
I hate to say it, but Kafka Streams suck. Its problem after problem.
UPDATE: Some more info: it appears that the brokers may have gotten into some
kind of crazy state, for an unknown reason, and now they are just shrinking and
expanding ISRs repeatedly. Trying to figure out the root cause of this
craziness.
was:
I have a Kafka stream that fails with this error on startup every time:
{code}
org.apache.kafka.streams.errors.StreamsException: Failed to get end offsets for
topic partitions of global store test-uiService-dlq-events-table-store after 0
retry attempts. You can increase the number of retries via configuration
parameter `retries`.
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.register(GlobalStateManagerImpl.java:186)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:101)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:207)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.KeyValueToTimestampedKeyValueByteStoreAdapter.init(KeyValueToTimestampedKeyValueByteStoreAdapter.java:87)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:112)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:123)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.int
[jira] [Updated] (KAFKA-8922) Failed to get end offsets for topic partitions of global store
[
https://issues.apache.org/jira/browse/KAFKA-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raman Gupta updated KAFKA-8922:
---
Description:
I have a Kafka stream that fails with this error on startup every time:
{code}
org.apache.kafka.streams.errors.StreamsException: Failed to get end offsets for
topic partitions of global store test-uiService-dlq-events-table-store after 0
retry attempts. You can increase the number of retries via configuration
parameter `retries`.
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.register(GlobalStateManagerImpl.java:186)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:101)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:207)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.KeyValueToTimestampedKeyValueByteStoreAdapter.init(KeyValueToTimestampedKeyValueByteStoreAdapter.java:87)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:112)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:123)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread.initialize(GlobalStreamThread.java:345)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread.run(GlobalStreamThread.java:270)
~[kafka-streams-2.3.0.jar:?]
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to get
offsets by times in 30001ms
{code}
The stream was working fine and then this started happening.
The stream now throws this error on every start. I am now going to attempt to
reset the stream and delete its local state.
I hate to say it, but Kafka Streams suck. Its problem after problem.
UPDATE: Some more info: it appears that the brokers may have gotten into some
kind of crazy state due to a librdkafka-based client (NodeJS). I see thousands
of logs per minute related to rebalancing that consumer
was:
I have a Kafka stream that fails with this error on startup every time:
{code}
org.apache.kafka.streams.errors.StreamsException: Failed to get end offsets for
topic partitions of global store test-uiService-dlq-events-table-store after 0
retry attempts. You can increase the number of retries via configuration
parameter `retries`.
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.register(GlobalStateManagerImpl.java:186)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.AbstractProcessorContext.register(AbstractProcessorContext.java:101)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.RocksDBStore.init(RocksDBStore.java:207)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.KeyValueToTimestampedKeyValueByteStoreAdapter.init(KeyValueToTimestampedKeyValueByteStoreAdapter.java:87)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.init(CachingKeyValueStore.java:58)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.WrappedStateStore.init(WrappedStateStore.java:48)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.init(MeteredKeyValueStore.java:112)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateManagerImpl.initialize(GlobalStateManagerImpl.java:123)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStateUpdateTask.initialize(GlobalStateUpdateTask.java:61)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread$StateConsumer.initialize(GlobalStreamThread.java:229)
~[kafka-streams-2.3.0.jar:?]
at
org.apache.kafka.streams.processor.internals.GlobalStreamThread.i
