Hi Georg, Great that you could answer your own question and I am glad that I could help.
I was just writing you a similar answer. Yes, the global state store will eventually reflect your write but you do not know when. That is the main issue for your use case. A local state store will immediately contain your previous write, because it is local to your processing. For more information on the global state store see https://kafka.apache.org/25/documentation/streams/developer-guide/dsl-api.html#streams_concepts_globalktable Best, Bruno On Tue, May 19, 2020 at 3:04 PM Schmidt-Dumont Georg (BCI/ESW17) <[email protected]> wrote: > > Hi Bruno, > > I just had a discussion with a colleague of mine regarding this and I wanted > to give you a quick contextual update. With regards to the global state, I > realize that having this state consistent in a distributed system is very > difficult. My expectation was that since it is a global state, Kafka takes > care of the consistency and I can just access the data. I think my > expectation was a bit naïve. The state will probably be eventually > consistent. But this does not fit with what I am trying to do. As you said I > should use a local store. > > With regards to the question in my previous mail with the amount of > partitions. I think I have answered my own question. Ensuring that the > messages have the correct and consistent keys will see to it that all the > data for a specific key ends up in a single partition. It does not mean that > a partition per key is required (which I first thought). > > Thanks again for your help! > > Mit freundlichen Grüßen / Best regards > > Georg Schmidt-Dumont > BCI/ESW17 > Bosch Connected Industry > > Tel. +49 711 811-49893 > > ► Take a look: https://bgn.bosch.com/alias/bci > > > > -----Ursprüngliche Nachricht----- > Von: Bruno Cadonna <[email protected]> > Gesendet: Dienstag, 19. Mai 2020 11:42 > An: Users <[email protected]> > Betreff: Re: Question regarding Kafka Streams Global State Store > > Hi Georg, > > local state stores in Kafka Streams are backed by a Kafka topic by default. > So, if the instance crashes the local state store is restored from the local > state directory. If the local state directory is empty or does not exist the > local state store is restored from the Kafka topic. Local state stores are as > resilient as global state stores. > > As far as I understand, you only look up previous records with the same key. > You do not need to have the global state available at each instance to do > this. Having available all records with the same key is sufficient. If your > input topic are partitioned by key then records with the same key will land > on the same instance. That means, your local state store contains all records > with the same key. > > Best, > Bruno > > On Tue, May 19, 2020 at 11:05 AM Schmidt-Dumont Georg (BCI/ESW17) > <[email protected]> wrote: > > > > Hi Bruno, > > > > Thanks for your quick reply! > > > > I decided to use a global state store for two reasons. If the application > > crashes, the store is populated properly once the reason for the crash has > > been fixed and the app starts again, i.e. I feel that it gives me a certain > > resiliency. Second we will be running multiple instances of the application > > and using a global state store provides the state across all instances. > > > > I am fairly new to Kafka and Kafka Streams, I am very much open to > > suggestions on better ways to handle the flow I need. > > > > Mit freundlichen Grüßen / Best regards > > > > Georg Schmidt-Dumont > > BCI/ESW17 > > Bosch Connected Industry > > > > Tel. +49 711 811-49893 > > > > ► Take a look: https://bgn.bosch.com/alias/bci > > > > > > > > -----Ursprüngliche Nachricht----- > > Von: Bruno Cadonna <[email protected]> > > Gesendet: Dienstag, 19. Mai 2020 10:52 > > An: Users <[email protected]> > > Betreff: Re: Question regarding Kafka Streams Global State Store > > > > Hi Georg, > > > > From your description, I do not see why you need to use a global state > > instead of a local one. Are there any specific reasons for that? With a > > local state store you would have the previous record immediately available. > > > > Best, > > Bruno > > > > On Tue, May 19, 2020 at 10:23 AM Schmidt-Dumont Georg (BCI/ESW17) > > <[email protected]> wrote: > > > > > > Good morning, > > > > > > I have setup a Kafka Streams application with the following logic. The > > > incoming messages are validated and transformed. The transformed messages > > > are then published to a global state store via topic A as well as to an > > > additional topic A for consumption by other applications further down the > > > processing pipeline. > > > > > > As part of the transformation I access the global state store in order to > > > get the values from the previous message and use them in the > > > transformation of the current message. The messages only contain changed > > > values and these changes are merged with the complete data set before > > > being sent on, hence I always hold the latest state in the global store > > > in order to merge it with the incoming changed values. > > > > > > Unfortunately, when I access the store in the transformation I do not get > > > the latest state. The update of the store takes too long so when I access > > > it in the transformation I either get no values or values which do not > > > represent the latest state. > > > > > > The following shows the build-up of my streams app: > > > > > > //setup global state store > > > final KeyValueBytesStoreSupplier storeSupplier = > > > Stores.persistentKeyValueStore( “global-store” ); final > > > StoreBuilder<KeyValueStore<String, JSONObject>> storeBuilder = > > > Stores.keyValueStoreBuilder( storeSupplier, Serdes.String(), new > > > JSONObjectSerde() ); builder.addGlobalStore( storeBuilder, > > > “global-store-topic”, Consumed.with( Serdes.String(), new > > > JSONObjectSerde() ), StoreProcessor::new ); > > > > > > //store processor > > > > > > private KeyValueStore<String, JSONObject> stateStore; > > > > > > @Override > > > public void init( final ProcessorContext context ) { > > > stateStore = (KeyValueStore<String, JSONObject>) > > > context.getStateStore( “global-store” ); } > > > > > > > > > > > > @Override > > > public void process( final String key, final JSONObject state ) { > > > log.info( "Update state store for {}: {}.", key, state ); > > > lastRecentStateStore.put( key, state ); } > > > > > > > > > //streams setup > > > > > > final JSONObjectSerde jsonObjectSerde = new JSONObjectSerde(); > > > > > > final KStream<String, JSONObject> stream = builder.stream( > > > “input-topic”, Consumed.with( Serdes.String(), jsonObjectSerde ) ) > > > > > > .transformValues( ValueTransformer::new ) > > > > > > > > > > > > stream.to( “global-store-topic”, Produced.valueSerde( > > > jsonObjectSerde ) ); > > > > > > stream.to( “output-topic”, Produced.valueSerde( jsonObjectSerde ) ); > > > > > > //global state store access in ValueTransformer > > > > > > JSONObject previousState = Optional.ofNullable( stateStore.get( key ) ) > > > .orElse( new JSONObject() ); > > > > > > > > > I have set the acknowledge property for the producers to “all”. > > > > > > I have tried to disable the caching by setting > > > “cache.max.bytes.buffering” to 0 and by disabling the cache on the store > > > using “.withCachingDisabled()”. I also tried setting the commit interval > > > to 0. All without success. > > > > > > How can I setup a global state which meets the requirements as describe > > > in the scenario above? > > > > > > Thank you! > > > > > > Best regards / Mit freundlichen Grüßen / Üdvözlettel / 致以诚挚的问候 > > > > > > Mr. Georg Schmidt-Dumont > > > Bosch Connected Industry – BCI/ESW17 Robert Bosch GmbH | Postfach 10 > > > 60 50 | 70049 Stuttgart | GERMANY | > > > www.bosch.com<http://www.bosch.com/> > > > Phone +49 711 811-49893 | > > > [email protected]<mailto:[email protected] > > > > > > > > > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000; > > > Aufsichtsratsvorsitzender: Franz Fehrenbach; Geschäftsführung: Dr. > > > Volkmar Denner, Prof. Dr. Stefan Asenkerschbaumer, Dr. Rolf > > > Bulander, Dr. Stefan Hartung, Dr. Markus Heyn, Dr. Dirk Hoheisel, > > > Christoph Kübel, Uwe Raschke, Peter Tyroller > > >
