Re: Kafka: Messages disappearing from topics, largestTime=0

So the logs show a healthy shutdown, so we can eliminate that as an issue. I would look next at the volume management during a rollout based on the other error messages you had earlier about permission denied etc. It's possible there's some journalled but not flushed changes in those time indexes,

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Yes. A `GlobalKTable` uses a global store internally. You can also use `StreamsBuilder.addGlobalStore()` or `Topology.addGlobalStore()` to add a global store "manually". -Matthias On 4/30/20 7:42 AM, Pushkar Deole wrote: > Thanks Matthias. > Can you elaborate on the replicated caching layer

Re: are kafka state stores global or local?

That is correct. For global stores, in fact you cannot put "arbitrary" data into the store. The `Processor` should _only_ take the input data from the topic, and put it into the store without modification. (cf. https://issues.apache.org/jira/browse/KAFKA-7663) If you want to update all copies

Re: Metrics in Kafka Connect

This is what I did inside MM2's Connectors and Tasks. It works pretty well, but I'd certainly prefer if Connect exposed it's own Metrics API. Ryanne On Thu, Apr 30, 2020 at 2:57 PM Gérald Quintana wrote: > Hello, > > We developed a custom Kafka Connect implementation for a specific need, and >

Metrics in Kafka Connect

Hello, We developed a custom Kafka Connect implementation for a specific need, and we would like to monitor its internals (request latency and rate, pool usage). Is it possible to publish custom metrics using the Kafka client metric framework (org.apache.kafka.common.metrics.*) . We would

Re: Kafka: Messages disappearing from topics, largestTime=0

I took a bit because I needed logs of the server shutting down when this occurs. Here they are, I can see some errors: https://gist.github.com/josebrandao13/e8b82469d3e9ad91fbf38cf139b5a726 Regarding systemd, the closest I could find to TimeoutStopSec was DefaultTimeoutStopUSec=1min 30s that

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Thanks Matthias. Can you elaborate on the replicated caching layer part? When you say global stores, do you mean GlobalKTable created from a topic e.g. using StreamsBuilder.globalTable(String topic) method ? On Thu, Apr 30, 2020 at 12:44 PM Matthias J. Sax wrote: > It's not possible to modify

kafka rdd save to hive errer

failed to get records for compacted ...after polling for12 partition 0 offset min=0 max=1427265 get offsetrange 0 until 50 to make rdd

Re: Kafka: Messages disappearing from topics, largestTime=0

I'd also suggest eyeballing your systemd conf to verify that someone hasn't set a very low TimeoutStopSec, or that KillSignal/RestartKillSignal haven't been configured to SIGKILL (confusingly named, imo, as the default for KillSignal is SIGTERM). Also, the Kafka broker logs at shutdown look very

Re: Kafka: Messages disappearing from topics, largestTime=0

Hi, It's quite a complex script generated with ansible where we use a/b deployment and honestly, I don't have full knowledge on it I can share the general guidelines of what is done: > - Any old volumes (from previous releases are removed) (named with suffix > '-old') > - Detach the volumes

Re: How to handle RebalanceInProgressException?

On 29.04.20 09:18, Benoit Delbosc wrote: > Hi > > On 28.04.20 21:05, Guozhang Wang wrote: >> Thanks for the explanation Ben. They are very helpful. >> >> Just to clarify on the context here: >> >> 1) Before Kafka 2.0 the poll(long) call make sure that the rebalance would >> be completed when the

Re: Kafka: Messages disappearing from topics, largestTime=0

Hi, It does look as index corruption... Can you post script that stops kafka? On Wednesday, April 29, 2020, 06:38:18 PM GMT+2, JP MB wrote: > > Can you try using the console consumer to display messages/keys and > timestamps ? > --property print.key=true --property

Re: Kafka Mirror Maker 2

Hi Himanshu Can u pls tell how to use MM2.. I am using Apache Kafka,in this normal mirror maker is only available.. Most of the people saying like to use MM2 but I didn't able to know where to get that MM2. Is it related to Apache or from some other Distributors? Can u pls explain how to

Re: are kafka state stores global or local?

Thanks Matthias ! my question (So, does it mean that the state store modified locally by each application is replicated to all other applications) was mainly about "global state store", As I understand it from your previous response, the state for global state store would be replicated to all

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

It's not possible to modify state store from "outside". If you want to build a "replicated caching layer", you could use global stores and write into the corresponding topics to update all stores. Of course, those updates would be async. -Matthias On 4/29/20 10:52 PM, Pushkar Deole wrote: > Hi

Re: are kafka state stores global or local?

> Thanks for the information. So, does it mean that the state store modified >> locally by each application is replicated to all other applications? No. As long as the application runs "normally" (ie, without failure), state is only maintained by one instance. In general, only in case of failure,