Thanks for the responses. In the worst case, I might have to keep both rocksdb
for local store and keep an external store like Redis.
-mohan
On 3/13/21, 8:53 PM, "Pushkar Deole" <pdeole2...@gmail.com> wrote:
Another issue with 3rd party state stores could be violation of
exactly-once guarantee provided by kafka streams in the event of a failure
of streams application instance.
Kafka provides exactly once guarantee for consumer -> process -> produce
through transactions and with the use of state store like redis, this
guarantee is weaker
On Sat, Mar 13, 2021 at 5:28 AM Guozhang Wang <wangg...@gmail.com> wrote:
> Hello Mohan,
>
> I think what you had in mind works with Redis, since it is a remote state
> store engine, it does not have the co-partitioning requirements as local
> state stores.
>
> One thing you'd need to tune KS though is that with remote stores, the
> processing latency may be larger, and since Kafka Streams process all
> records of a single partition in order, synchronously, you may need to
tune
> the poll interval configs etc to make sure KS would stay in the consumer
> group and not trigger unnecessary rebalances.
>
> Guozhang
>
> On Thu, Mar 11, 2021 at 6:41 PM Parthasarathy, Mohan <mpart...@hpe.com>
> wrote:
>
> > Hi,
> >
> > I have a use case where messages come in with some key gets assigned
some
> > partition and the state gets created. Later, key changes (but still
> > contains the old key in the message) and gets sent to a different
> > partition. I want to be able to grab the old state using the old key
> before
> > creating the new state on this instance. Redis as a state store makes
it
> > easy to implement this where I can simply do a lookup before creating
the
> > state. I see an implementation here :
> >
>
https://github.com/andreas-schroeder/redisks/tree/master/src/main/java/com/github/andreas_schroeder/redisks
> >
> > Has anyone tried this ? Any caveats.
> >
> > Thanks
> > Mohan
> >
> >
>
> --
> -- Guozhang
>
