Re: Kafka Streams - "state store may have migrated to another instance"
If you run multiple instances of your app you may not be able to access the state store you are trying to access from the instance you are trying from, i.e., it may be on another instance. If streams is in the RUNNING state, this would seem to be the issue. On Wed, 7 Mar 2018 at 15:56 detharon wrote: > I'm afraid that's not what I'm looking for, as I'm just trying to retrieve > the local data, from inside my application (but from outside the stream > topology), and in some cases it becomes impossible. That is, the stream > changes its state from "rebalancing" to "running", but the store is remains > inaccessible. > > I don't want to have access to state stores located on other instances. > > In pseudocode: > streams.start(); <- here's where the stream starts > loopOverLocalData(streams) <- periodical, asynchronous calls to > streams.store, which, if the problem occurs, always result in exceptions > being thrown. I expected to retrieve the local data this way. > > On 7 March 2018 at 16:20, Damian Guy wrote: > > > If you have multiple streams instances then the store might only be > > available on one of the instances. Using `KafkaStreams.store(..)` will > only > > locate stores that are currently accessible by that instance. If you need > > to be able to locate stores on other instances, then you should probably > > have a read of: > > https://kafka.apache.org/10/documentation/streams/ > > developer-guide/interactive-queries.html#querying-remote- > > state-stores-for-the-entire-app > > > > There is also a convenient 3rd party lib that can help you with this: > > https://github.com/lightbend/kafka-streams-query > > > > On Wed, 7 Mar 2018 at 14:07 detharon wrote: > > > > > Hello, I'm experiencing issues accessing the state stores outside the > > Kafka > > > stream. My application queries the state store every n seconds using > the > > > .all() method to retrieve all key value pairs. I know that the state > > store > > > might not be available, so I guard against the > InvalidStateStoreException > > > and in case it's caught, I simply invoke the .store method on my > stream, > > in > > > order to get a new store. The problem is, that for some reason the > store > > > never becomes available. > > > > > > Some facts: > > > - During that time stream processing works correctly, and it > successfully > > > puts and gets data to and from the store. > > > - Stream is in "running" state. I've started logging that because under > > > normal circumstances this exception is being thrown when the stream is > in > > > "rebalancing" phase, but after a while it's gone, as expected, so I can > > > distinguish between these two cases. > > > - It only happens when I run my app in multiple instances. > > > > > > I've set the log level to debug, but I can't see anything suspicious in > > the > > > logs, but maybe there's something particular I should pay attention to? > > > I access the store from inside an Akka actor, to which I pass the > > reference > > > to KafkaStreams object, if that matters. > > > > > > I ran out of ideas what might have caused that behavior, so any help > will > > > be greatly appreciated. > > > > > >
Re: Kafka Streams - "state store may have migrated to another instance"
I'm afraid that's not what I'm looking for, as I'm just trying to retrieve the local data, from inside my application (but from outside the stream topology), and in some cases it becomes impossible. That is, the stream changes its state from "rebalancing" to "running", but the store is remains inaccessible. I don't want to have access to state stores located on other instances. In pseudocode: streams.start(); <- here's where the stream starts loopOverLocalData(streams) <- periodical, asynchronous calls to streams.store, which, if the problem occurs, always result in exceptions being thrown. I expected to retrieve the local data this way. On 7 March 2018 at 16:20, Damian Guy wrote: > If you have multiple streams instances then the store might only be > available on one of the instances. Using `KafkaStreams.store(..)` will only > locate stores that are currently accessible by that instance. If you need > to be able to locate stores on other instances, then you should probably > have a read of: > https://kafka.apache.org/10/documentation/streams/ > developer-guide/interactive-queries.html#querying-remote- > state-stores-for-the-entire-app > > There is also a convenient 3rd party lib that can help you with this: > https://github.com/lightbend/kafka-streams-query > > On Wed, 7 Mar 2018 at 14:07 detharon wrote: > > > Hello, I'm experiencing issues accessing the state stores outside the > Kafka > > stream. My application queries the state store every n seconds using the > > .all() method to retrieve all key value pairs. I know that the state > store > > might not be available, so I guard against the InvalidStateStoreException > > and in case it's caught, I simply invoke the .store method on my stream, > in > > order to get a new store. The problem is, that for some reason the store > > never becomes available. > > > > Some facts: > > - During that time stream processing works correctly, and it successfully > > puts and gets data to and from the store. > > - Stream is in "running" state. I've started logging that because under > > normal circumstances this exception is being thrown when the stream is in > > "rebalancing" phase, but after a while it's gone, as expected, so I can > > distinguish between these two cases. > > - It only happens when I run my app in multiple instances. > > > > I've set the log level to debug, but I can't see anything suspicious in > the > > logs, but maybe there's something particular I should pay attention to? > > I access the store from inside an Akka actor, to which I pass the > reference > > to KafkaStreams object, if that matters. > > > > I ran out of ideas what might have caused that behavior, so any help will > > be greatly appreciated. > > >
Re: Kafka Streams - "state store may have migrated to another instance"
If you have multiple streams instances then the store might only be available on one of the instances. Using `KafkaStreams.store(..)` will only locate stores that are currently accessible by that instance. If you need to be able to locate stores on other instances, then you should probably have a read of: https://kafka.apache.org/10/documentation/streams/developer-guide/interactive-queries.html#querying-remote-state-stores-for-the-entire-app There is also a convenient 3rd party lib that can help you with this: https://github.com/lightbend/kafka-streams-query On Wed, 7 Mar 2018 at 14:07 detharon wrote: > Hello, I'm experiencing issues accessing the state stores outside the Kafka > stream. My application queries the state store every n seconds using the > .all() method to retrieve all key value pairs. I know that the state store > might not be available, so I guard against the InvalidStateStoreException > and in case it's caught, I simply invoke the .store method on my stream, in > order to get a new store. The problem is, that for some reason the store > never becomes available. > > Some facts: > - During that time stream processing works correctly, and it successfully > puts and gets data to and from the store. > - Stream is in "running" state. I've started logging that because under > normal circumstances this exception is being thrown when the stream is in > "rebalancing" phase, but after a while it's gone, as expected, so I can > distinguish between these two cases. > - It only happens when I run my app in multiple instances. > > I've set the log level to debug, but I can't see anything suspicious in the > logs, but maybe there's something particular I should pay attention to? > I access the store from inside an Akka actor, to which I pass the reference > to KafkaStreams object, if that matters. > > I ran out of ideas what might have caused that behavior, so any help will > be greatly appreciated. >
Kafka Streams - "state store may have migrated to another instance"
Hello, I'm experiencing issues accessing the state stores outside the Kafka stream. My application queries the state store every n seconds using the .all() method to retrieve all key value pairs. I know that the state store might not be available, so I guard against the InvalidStateStoreException and in case it's caught, I simply invoke the .store method on my stream, in order to get a new store. The problem is, that for some reason the store never becomes available. Some facts: - During that time stream processing works correctly, and it successfully puts and gets data to and from the store. - Stream is in "running" state. I've started logging that because under normal circumstances this exception is being thrown when the stream is in "rebalancing" phase, but after a while it's gone, as expected, so I can distinguish between these two cases. - It only happens when I run my app in multiple instances. I've set the log level to debug, but I can't see anything suspicious in the logs, but maybe there's something particular I should pay attention to? I access the store from inside an Akka actor, to which I pass the reference to KafkaStreams object, if that matters. I ran out of ideas what might have caused that behavior, so any help will be greatly appreciated.