Matthias It happened again yesterday during another rolling update. The first error log I can see on the client side is below. It was there in PENDING_ERROR state for sometime and then went into ERROR state.
Caused by: java.lang.IllegalStateException: KafkaStreams is not running. State is PENDING_ERROR. at org.apache.kafka.streams.KafkaStreams.validateIsRunningOrRebalancing(KafkaStreams.java:381) at org.apache.kafka.streams.KafkaStreams.queryMetadataForKey(KafkaStreams.java:1663) at org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$2(InteractiveQueryService.java:227) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.HashMap$KeySpliterator.tryAdvance(HashMap.java:1728) at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129) at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:647) at org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$3(InteractiveQueryService.ja On Tue, Oct 3, 2023 at 8:50 AM Matthias J. Sax <mj...@apache.org> wrote: > I did mean client side... If KS goes into ERROR state, it should log > the reason. > > If the logs are indeed empty, try to register an > uncaught-exception-handler via > > KafkaStreamssetUncaughtExceptionHandler(...) > > > -Matthias > > On 10/2/23 12:11 PM, Debraj Manna wrote: > > Are you suggesting to check the Kafka broker logs? I do not see any other > > errors logs on the client / application side. > > > > On Fri, 29 Sep, 2023, 22:01 Matthias J. Sax, <mj...@apache.org> wrote: > > > >> In general, Kafka Streams should keep running. > >> > >> Can you inspect the logs to figure out why it's going into ERROR state > >> to begin with? Maybe you need to increase/change some timeouts/retries > >> configs. > >> > >> The stack trace you shared, is a symptom, but not the root cause. > >> > >> -Matthias > >> > >> On 9/21/23 12:56 AM, Debraj Manna wrote: > >>> I am using Kafka broker 2.8.1 (from AWS MSK) with Kafka clients and > Kafka > >>> stream 3.5.1. > >>> > >>> I am observing that whenever some rolling upgrade is done on AWS MSK > our > >>> stream application reaches an error state. I get the below exception on > >>> trying to query the state store > >>> > >>> caused by: java.lang.IllegalStateException: KafkaStreams is not > running. > >>> State is ERROR. > >>> at > >>> > >> > org.apache.kafka.streams.KafkaStreams.validateIsRunningOrRebalancing(KafkaStreams.java:381) > >>> at > >>> > >> > org.apache.kafka.streams.KafkaStreams.queryMetadataForKey(KafkaStreams.java:1663) > >>> at > >>> > >> > org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$2(InteractiveQueryService.java:227) > >>> > >>> Can someone let me know what the recommended way we can keep the stream > >>> application running whenever some rolling upgrade/restart of brokers is > >>> done in the background? > >>> > >> > > >
