Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Thanks John... appreciate your inputs and suggestions. I have been assigned recently to this task (of persisting the cache) and haven't been involved in original design and architecture and agree with all the issues you have highlighted. However, at this point, i don't think the application can be

Re: FAILED tasks in connector status

Here is the log on the app side when the worker tries to start the failing task. { "timestamp": "2020-05-04T23:22:37,759Z", "level": "ERROR", "thread": "pool-7-thread-3", "logger": "org.apache.kafka.connect.runtime.Worker", "timestamp": "2020-05-04T16:22:37,759", "message": "Failed to

FAILED tasks in connector status

Hi community, I have a connect cluster deployed in a 'cloud-like' environment where an instance can die anytime but a new instance gets automatically re-spawn immediately after(within the min...). This obviously leads to an eager rebalance - I am on Kafka 2.1.1 for client and server. What happens

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Thanks John... what parameters would affect the latency in case GlobalKTable will be used and is there any configurations that could be tuned to minimize the latency of sync with input topic? On Mon, May 4, 2020 at 10:20 PM John Roesler wrote: > Hello Pushkar, > > Yes, that’s correct. The operat

Re: Kafka: Messages disappearing from topics, largestTime=0

Here are the startup logs from a deployment where we lost 15 messages in topic-p: https://gist.github.com/josebrandao13/81271140e59e28eda7aaa777d2d3b02c .timeindex files state before the deployment: *Partitions with messages: timestamp mismatch *Partitions without messages: permission denied .tim

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Hello Pushkar, Yes, that’s correct. The operation you describe is currently not supported. If you want to keep the structure you described in place, I’d suggest using an external database for the admin objects. I’ll give another idea below. With your current architecture, I’m a little concern

Re: Kafka: Messages disappearing from topics, largestTime=0

Hi guys, I'm gonna get back to this today, I get mixed feelings regarding the volumes being the cause. This volume switching is around for quite some time, in a lot of clusters, and we only started noticing this problem when we updated some of them. Also, this only happens in *a few* of those .tim

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

If i understand correctly, Kafka is not designed to provide replicated caching mechanism wherein the updates to cache will be synchronous across multiple cache instances. On Sun, May 3, 2020 at 10:49 PM Pushkar Deole wrote: > Thanks John. > > Actually, this is a normal consumer-producer applicat

Kafka consumer

Hey Guys, I am having a topic and in that topic I am having 3000 messages In my springboot application I want to consume the data using @KafkaListener() and also one by one because I need to do some tedious process on that Data it may take some time So within this time I don't need to consume