Re: are kafka state stores global or local?

2020-04-30 Thread Matthias J. Sax
That is correct. For global stores, in fact you cannot put "arbitrary" data into the store. The `Processor` should _only_ take the input data from the topic, and put it into the store without modification. (cf. https://issues.apache.org/jira/browse/KAFKA-7663) If you want to update all copies of

Re: are kafka state stores global or local?

2020-04-30 Thread Pushkar Deole
Thanks Matthias ! my question (So, does it mean that the state store modified locally by each application is replicated to all other applications) was mainly about "global state store", As I understand it from your previous response, the state for global state store would be replicated to all in

Re: are kafka state stores global or local?

2020-04-30 Thread Matthias J. Sax
> Thanks for the information. So, does it mean that the state store modified >> locally by each application is replicated to all other applications? No. As long as the application runs "normally" (ie, without failure), state is only maintained by one instance. In general, only in case of failure,

Re: are kafka state stores global or local?

2020-04-29 Thread Pushkar Deole
Thanks for the information. So, does it mean that the state store modified locally by each application is replicated to all other applications? If yes then does it happen in real time i.e. the state modified by an application is immediately reflected in the state store of the other application? O

Re: are kafka state stores global or local?

2020-04-29 Thread Matthias J. Sax
This might help: https://stackoverflow.com/questions/40274884/is-kafka-stream-statestore-global-over-all-instances-or-just-local/40278168#40278168 Note, that "global stores" are also "local" (ie, on local disk). The difference is really between being sharded or being replicated. -Matthias On 4/

Re: are kafka state stores global or local?

2020-04-29 Thread Bill Bejeck
Hi Pushkar, There is a concept of a global store

are kafka state stores global or local?

2020-04-29 Thread Pushkar Deole
Hi, I am looking for some information as to whether kafka state store KeyValueStore is global or local only. I found that it can be in-memory or made persistent which stores it in local Rocksdb and also logging can be enabled so the state is backed by a topic which allows the state store to be fau