Fwd: kafka streams with multiple threads and state store

2017-11-12 Thread Ranjit Kumar
Hi All, I am new to kafka. I have some fundamental questions related to choosing architecture for m requirement can please suggest me. I am developing some IOT project, please find the details below. 1. I have moving objects which are continuously sending their position + some other informati

Re: kafka streams with multiple threads and state store

2017-11-10 Thread Damian Guy
Hi Ranjit, it sounds like you might want to use a global table for this. You can use StreamsBuilder#globalTable(String, Materialized) to create the global table. You could do something like: KeyValueBytesStoreSupplier supplier = Stores.inMemoryKeyValueStore("global-store"); Materialized> materiali

Re: kafka streams with multiple threads and state store

2017-11-10 Thread Ranjit Kumar
Hi Guozhang, Thanks for the information. My requirement is some thing like this. 1. i want to read the data from one topic (which is continuously feeding), so i though of using the kafka streams with threads 2. want to store the data in one in memory data base (not the local data store per threa

Re: kafka streams with multiple threads and state store

2017-11-09 Thread Guozhang Wang
Ranjit, Note that the "testStore" instance you are passing is a StateStoreSupplier which will generate a new StateStore instance for each thread's task. If you really want to have all the thread's share the same state store you should implement your own StateStoreSupplier that only return the sam

kafka streams with multiple threads and state store

2017-11-09 Thread Ranjit Kumar
Hi All, I want to use one state store in all my kafka stream threads in my application, how can i do it. 1. i created one topic (name: test2) with 3 partitions . 2. wrote kafka stream with num.stream.threads = 3 in java code 3. using state store (name: count2) in my application. But state store