can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Hi All, I am wondering if this is possible: i have been asked to use state stores as a general replicated cache among multiple instances of service instances however the state store is created through streambuilder but is not actually modified through stream processor topology however it is to be

Re: Apache Kafka cluster to cluster

Is there documentation or example for mirror maker 2.0 ? On 4/29/20, 9:04 PM, "Liam Clarke-Hutchinson" wrote: [External] Hi Blake, Replicator is, AFAIK, not FOSS - however, Mirror Maker 2.0, which is built along very similar lines (i.e., on top of Kafka Connect) is, as is

Re: Apache Kafka cluster to cluster

Hi Clark My source cluster is on localhost:9092 My Target cluster was localhost:9091 I am not aware of something like Mirror maker 1 and 2 Because I am using mirror maker which come default from Apache Kafka 2.3.0 On Thu, Apr 30, 2020, 09:43 Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz>

Re: Apache Kafka cluster to cluster

Hi Vishnu, I can see you're using Mirror Maker 1, unlike Mirror Maker 2, this doesn't auto-create topics on the target cluster. Does your target cluster on port 9091 have the same topics created on it that exist on the source cluster on port 9092? Also note that a Kafka consumer defaults to "late

Re: are kafka state stores global or local?

Thanks for the information. So, does it mean that the state store modified locally by each application is replicated to all other applications? If yes then does it happen in real time i.e. the state modified by an application is immediately reflected in the state store of the other application? O

Re: Apache Kafka cluster to cluster

Hi Blake, Replicator is, AFAIK, not FOSS - however, Mirror Maker 2.0, which is built along very similar lines (i.e., on top of Kafka Connect) is, as is Mirror Maker 1.0. On Thu, Apr 30, 2020 at 6:51 AM Blake Miller wrote: > Oh, and it looks like Confluent has released a newer replacement for >

Re: Apache Kafka cluster to cluster

i start the kafka mirror using the below configuration .\bin\windows\kafka-mirror-maker.bat --consumer.config .\config\consumer.properties --producer.config .\config\producer.properties --whitelist=".*" *Consumer.properties:* # format: host1:port1,host2:port2 ... bootstrap.servers=localhost:9092

Re: JDBC source connector to Kafka topic

I am using normal Apache Kafka only not confluent So step by step only I am starting Kafka .\bin\windows\kafka-server-start.bat .\config\server.properties Like this only I am starting and even before shutdown also I need to send configuration details every time through post request to get the da

Re: Kafka Mirror Maker 2

> 1. ... Do I need to manually create a topic TOPIC1 in cluster K2 and this topic will be used for producing messages to when failover happens. Correct. When you fail-over producers from K1 to K2, producers will send to TOPIC1 in K2. You may need to manually create that topic and set up write ACLs

Re: JDBC source connector to Kafka topic

How are you running Kafka? Do you mean when you shut it down you have to reconfigure the connector? -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Wed, 29 Apr 2020 at 20:03, vishnu murali wrote: > Hi guys > I am trying that JDBC source connector to get data fr

Kafka Mirror Maker 2

Hey Team, I've a few doubts regarding how the producers work after failover in Mirror Maker 2 Let's say that we have two clusters K1 and K2 and configured MM2 replication for TOPIC1 (originally created in just K1). We configured the active-active replication: K1->K2.enabled = true K2->K1.enabled

JDBC source connector to Kafka topic

Hi guys I am trying that JDBC source connector to get data from MySQL and send as a data in a topic,so here what I am facing is there is more manual here After starting zookeeper,server, connect-distributed in Apache kafka I need to give Post request every time to the localhost:8083/connectors wi

Re: One cluster topic to another cluster topic

Use mirror maker . On 4/29/20, 11:52 AM, "vishnu murali" wrote: [External] Hi Guys, I am having two separate Kafka cluster running in two independent zookeeper I need to send a set of data from one topic from cluster A to cluster B with the same topic name with all dat

One cluster topic to another cluster topic

Hi Guys, I am having two separate Kafka cluster running in two independent zookeeper I need to send a set of data from one topic from cluster A to cluster B with the same topic name with all data also.. How can I achieve this Done anyone have any idea ??

Re: Apache Kafka cluster to cluster

Thanks Blake..😊😊 More over can we use any connector types for this situation? Like source is a topic and sink is also an another topic Is this possible... On Thu, Apr 30, 2020, 00:19 Blake Miller wrote: > Hi Vishnu, > > Check out MirrorMaker > https://cwiki.apache.org/confluence/pages/viewpag

Re: Apache Kafka cluster to cluster

Oh, and it looks like Confluent has released a newer replacement for MirrorMaker called Replicator https://docs.confluent.io/current/multi-dc-deployments/replicator/migrate-replicator.html On Wed, Apr 29, 2020 at 6:49 PM Blake Miller wrote: > Hi Vishnu, > > Check out MirrorMaker > https://cwi

Re: Apache Kafka cluster to cluster

Hi Vishnu, Check out MirrorMaker https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330 This can do what you want. Note that the offsets are not copied, nor are the message timestamps. HTH On Wed, Apr 29, 2020 at 6:47 PM vishnu murali wrote: > Hi Guys, > > I am having two

Re: are kafka state stores global or local?

This might help: https://stackoverflow.com/questions/40274884/is-kafka-stream-statestore-global-over-all-instances-or-just-local/40278168#40278168 Note, that "global stores" are also "local" (ie, on local disk). The difference is really between being sharded or being replicated. -Matthias On 4/

Apache Kafka cluster to cluster

Hi Guys, I am having two separate Kafka cluster running in two independent zookeeper I need to send a set of data from one topic from cluster A to cluster B with the same topic name with all data also.. How can I achieve this Done anyone have any idea ??

Re: are kafka state stores global or local?

Hi Pushkar, There is a concept of a global store

Re: Kafka: Messages disappearing from topics, largestTime=0

> > Can you try using the console consumer to display messages/keys and > timestamps ? > --property print.key=true --property print.timestamp=true There are a lot off messages so I'm picking an example without and with timeindex entry. All of them have a null key: Offset 57 CreateTime:1588074808

Re: Kstreams Database

Kafka connect is using JDBC... Thus you would just implement a similar solution, which would surely be less efficient :/ https://www.confluent.io/blog/kafka-connect-deep-dive-jdbc-source-connector/ Your problem doesn't seem technical but organizational Le mer. 29 avr. 2020 à 17:47, KhajaAsmath

Kstreams Database

Hi, I have a requirement where I need to read records from the database and load them as JSON into kafka topic. Next step is to handle only updates. I was successful using this for Kafkaconnect for mysql but unfortunately I cannot using kafka connect due to restrictions from database team. Is th

Re: Kafka: Messages disappearing from topics, largestTime=0

Hmm, how are you doing your rolling deploys? I'm wondering if the time indexes are being corrupted by unclean shutdowns. I've been reading code and the only path I could find that led to a largest timestamp of 0 was, as you've discovered, where there was no time index. WRT to the corruption - th

Re: Kafka: Messages disappearing from topics, largestTime=0

Can you try using the console consumer to display messages/keys and timestamps ? --property print.key=true --property print.timestamp=true Le mer. 29 avr. 2020 à 13:23, JP MB a écrit : > The server is in UTC, [2020-04-27 10:36:40,386] was actually my time. On > the server was 9:36. > It doesn't

Re: Kafka: Messages disappearing from topics, largestTime=0

The server is in UTC, [2020-04-27 10:36:40,386] was actually my time. On the server was 9:36. It doesn't look like a timezone problem because it cleans properly other records, exactly 48 hours. Em qua., 29 de abr. de 2020 às 11:26, Goran Sliskovic escreveu: > Hi, > When lastModifiedTime on that

Re: Kafka: Messages disappearing from topics, largestTime=0

We are using the console produce, directly on the machines where we are experiencing the problem. I just inserted 150 messages in a topic and chose the partition with more messages to make this analysis, in this case, partition 15 in broker 1. The log file: > kafka-run-class.sh kafka.tools.DumpLo

Re: Kafka: Messages disappearing from topics, largestTime=0

Hi, When lastModifiedTime on that segment is converted to human readable time: Monday, April 27, 2020 9:14:19 AM UTC In what time zone is the server (IOW: [2020-04-27 10:36:40,386] from the log is in what time zone)?  It looks as largestTime is property of log record and 0 means the log record i

are kafka state stores global or local?

Hi, I am looking for some information as to whether kafka state store KeyValueStore is global or local only. I found that it can be in-memory or made persistent which stores it in local Rocksdb and also logging can be enabled so the state is backed by a topic which allows the state store to be fau

Re: Kafka: Messages disappearing from topics, largestTime=0

Timestamps are stored in the message headers. You can use the tools in kafka/bin to dump those messages timestamps to check their value. My guess is that the producer is setting a specific timestamp that is wrong ? Le mer. 29 avr. 2020 à 10:09, JP MB a écrit : > Hi, > Sorry guys for insisting on

Fwd: Kafka: Messages disappearing from topics, largestTime=0

Hi, Sorry guys for insisting on this but does anyone have an idea of how that largestTime can be 0 ? Regards -- Forwarded message - De: JP MB Date: ter., 28 de abr. de 2020 às 15:36 Subject: Kafka: Messages disappearing from topics, largestTime=0 To: Hi, We have messages disap

Re: How to handle RebalanceInProgressException?

Hi On 28.04.20 21:05, Guozhang Wang wrote: > Thanks for the explanation Ben. They are very helpful. > > Just to clarify on the context here: > > 1) Before Kafka 2.0 the poll(long) call make sure that the rebalance would > be completed when the call returns, no matter how long it takes. Note this