Re: MirrorMaker2 on GKE

data from src topic to target topic On Thu, Feb 23, 2023 at 3:04 PM karan alang wrote: > Logs with the Disconnect Exception (when i set the loglevel to - INFO) : > > > ``` > > [SourceTaskOffsetCommitter-1] > > 2023-02-23 22:54:13,546 INFO [Consumer clientId=consumer-null-

Re: MirrorMaker2 on GKE

n Thu, Feb 23, 2023 at 2:12 PM karan alang wrote: > Hello All, > > Anyone has installed KafkaMirrorMaker2 on GKE ? > Need some help to debug/resolve issues that I've been having. > > I've 2 clusters on GKE with strimzi kafka installed. > cluster1(nossl-e) on use-east1

MirrorMaker2 on GKE

Hello All, Anyone has installed KafkaMirrorMaker2 on GKE ? Need some help to debug/resolve issues that I've been having. I've 2 clusters on GKE with strimzi kafka installed. cluster1(nossl-e) on use-east1, cluster2(nossl-e) on us-west1. Both the clusters are Auto-pilot cluster (mentioning this,

Kafka on GKE - automating distribution of TLS certs to external Kafka clients on certs expiry

Hello All, We have a Strimzi kafka cluster on GKE with TLS implemented, the TLS certs have a default expiry of 1 year i.e. the certs get renewed every year Currently, on expiry - we have to provide the external Kafka clients with new certs, the objective is to automate this process i.e. One shoul

Re: Confluent Schema Registry - auto.register.schemas

hello All - checking to see if anyone has updates on this. tia! On Fri, Nov 11, 2022 at 1:55 PM karan alang wrote: > Hello All, > > ref stackoverflow : > https://stackoverflow.com/questions/74396652/confluent-schema-registry-why-is-auto-register-schemas-not-applicable-at-schem >

Confluent Schema Registry - auto.register.schemas

Hello All, ref stackoverflow : https://stackoverflow.com/questions/74396652/confluent-schema-registry-why-is-auto-register-schemas-not-applicable-at-schem --- I've installed the Schema Registry and want to set the auto.register.schema set to false - so only avro messages conforming to the regist

GCP Dataproc - getting error in importing KafkaProducer

Hello All, I've a GCP Dataproc cluster, and i'm running a Spark StructuredStreaming job on this. I'm trying to use KafkaProducer to push aggregated data into a Kafka topic, however when i import KafkaProducer (from kafka import KafkaProducer), it gives error ``` Traceback (most recent call last

Re: GCP Dataproc - Failed to construct kafka consumer, Failed to load SSL keystore dataproc-versa-sase-p12-1.jks of type JKS

re-checking to see if there is any suggestion on this issue. On Wed, Feb 2, 2022 at 3:36 PM karan alang wrote: > Hello All, > > I'm trying to run a Structured Streaming program on GCP Dataproc, which > accesses the data from Kafka and prints it. > > Access to Kafk

GCP Dataproc - Failed to construct kafka consumer, Failed to load SSL keystore dataproc-versa-sase-p12-1.jks of type JKS

Hello All, I'm trying to run a Structured Streaming program on GCP Dataproc, which accesses the data from Kafka and prints it. Access to Kafka is using SSL, and the truststore and keystore files are stored in buckets. I'm using Google Storage API to access the bucket, and store the file in the cu

multiple tenants accessing Confluent kafka on GCP

Hello All - For multiple external tenants accessing Kafka, is there any issue with providing the same endpoint (ie. set of brokers, ports) to the multiple producers ? what are the best practices wrt multiple tenants producing data on (Confluent) Kafka topics, Kafka being installed on GCP. tia!

what drives deletion of kafka topics

I've Kafka 10. I've a basic question - what determines when the Kafka topic marked for deletion gets deleted ? Today, i marked a topic for deletion, and it got deleted immediately (possibly because the topic was not being used for last few months ?) .. In earlier instances, i'd to wait for some ti

Kafka producer/consumer quotas (Kafka 0.10)

I've Kafka 0.10 .. I've set the following in kafka broker.properties & restarted the brokers - quota.producer.default=10485760 quota.consumer.default=10485760 when i fire the following command, i was expecting to see the quota limits : /bin/kafka-configs --zookeeper localhost:3181 --describe -

Re: left join between PageViews(KStream) & UserProfile (KTable)

ctions for how to > run it. > > It would be great if we had the same setup for the Apache Kafka page view > example, of course. Pull requests are very welcome. :-) > > Hope this helps, > Michael > > > > > > > On Sun, Oct 22, 2017 at 9:30 PM, karan al

left join between PageViews(KStream) & UserProfile (KTable)

Hello all - I'm trying to run the sample PageView Kafka streams example, (ref - https://github.com/apache/kafka/tree/0.11.0/streams/examples/src/main/java/org/apache/kafka/streams/examples/pageview ) and have a question .. There is a leftJoin between PageView (Kstream) and UserProfile(Ktable) as s

Re: Kafka SSL error

the chain parts separately into the keystore will not help. > > In the openssl s_client you can also use the -CAfile option to point it to > the intermediate certificate - that would also wirk but using the whole > chain is probably better. > > Regards > Jakub > > On Thu,

Re: Kafka SSL error

uld be in following order -> 1) Root 2) Intermediate 3) Server On Wed, Sep 20, 2017 at 4:48 PM, karan alang wrote: > Hi Jakub, > > Thanks for the detailed note... > > here is the update -> > > I was able to convert the host.cert.pem to PKCS#12 & import the cert into

Re: Kafka SSL error

at the broker basically > uses the self-signed cert you generated. This is probably the default > subject distinguished name of the self signed certificate. And since it is > self-signed openssl cannot verify its identity (unless you pass it the > public key of this exact self signed

Re: Kafka SSL error

2 key and that can > be converted into keystore. > > BTW: If you run your application with system property “javax.net.debug” set > to “ssl” it will generate a lot of useful debug information which will help > to understand what is going on and fix this. > > Jakub > > On T

Kafka SSL error

Hello All - I was able to set up SSL for the Kafka brokers, using OpenSSL. however, I'm having issues with setting up SSL using the pem file (i.e. SSL certificate - certified by CA, provided by the company) Here is what i've done - created the server/client keystore & truststore files and importe

Re: Kafka Streams - not able to see the text in console consumer

.kafka.common.serialization.LongDeserializer > > > Vito > > > > > > -- > 鄭紹志 Vito Jeng > 亦思科技股份有限公司 研究發展處 > TEL: 03-5630345 Ext.16 > > On Mon, Sep 18, 2017 at 10:58 AM, karan alang > wrote: > > > Hello All > > - i've a basic word count

Kafka Streams - not able to see the text in console consumer

Hello All - i've a basic word count Kafka streams code, which reads data from the input topic, splits the data (per the separator) and outputs the data into the output topic. I've a console producer, which is putting data into the input topic (kstreams3), and a console consumer which is reading th

Kafka SSL - console-consumer on node5 not receiving messages from producer on node4

Hi All - I've Kafka 0.10 .. I've enabled SSL(Non-kerberized) for Kafka Broker on Node 4, and *i'm able to produce/consume messages using console-producer & console-consumer from Node 4.* However, i'm having issues enabling ssl connection between Node 4 & Node 5 & try to consume messages from Node

Re: Confluent Kafka 3.2.2 - rebalancing not happenning

are even across brokers and racks on a per topic and cluster > level while minimising data movement. seems there is a disconnect here, pls let me know if anyone has inputs. On Tue, Aug 1, 2017 at 4:45 PM, karan alang wrote: > Hi All - > i'm trying to rebalance Kafka topi

Confluent Kafka 3.2.2 - rebalancing not happenning

Hi All - i'm trying to rebalance Kafka topic (refer link -> http://docs.confluent.io/current/kafka/rebalancer/rebalancer.html), and somehow the rebalancing is not working. Here is what i'm doing -> - i've 4 Kafka brokers & i've made changes to the server.properties file to enable Confluent Metric

Re: Confluent 3.2.2 - Error in starting broker

HI All - this is fixed, this was due to class path issue. On Thu, Jul 27, 2017 at 3:30 PM, karan alang wrote: > Hi All - i've installed Confluent 3.2.2 & i'm getting error in starting up > the Broker > > Attached the server.properties file, any ideas on what the issue

Confluent 3.2.2 - Error in starting broker

Hi All - i've installed Confluent 3.2.2 & i'm getting error in starting up the Broker Attached the server.properties file, any ideas on what the issue might be ? ./bin/kafka-server-start ./etc/kafka/server.properties > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in

Re: Confluent Control Center UI - not coming up

Hello all- i was able to fix this by modifying the following entry confluent.controlcenter.rest.listeners=http://0.0.0.0:9021 -> confluent.controlcenter.rest.listeners=http://:9021 <http://0.0.0.0:9021/> On Wed, Jul 26, 2017 at 3:35 PM, karan alang wrote: > hi All, > &

Confluent Control Center UI - not coming up

hi All, I've installed Confluent Kafka 3.2.2 & am using the following link to start the services - http://docs.confluent.io/current/control-center/docs/quickstart.html I've started the services shown below, however the Contraol Center UI is not coming up. Any ideas on this ? screen -ls > There

Re: Kafka with SSL enabled - Not able to publish messages (org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms)

Brokers (1001, 1002, 1003) The question is why & what needs to be done to fix this ? On Tue, Jul 25, 2017 at 10:31 PM, Manikumar wrote: > enable debug logs to find out the actual error. > > On Wed, Jul 26, 2017 at 12:49 AM, karan alang > wrote: > > > hi - I'v

Kafka with SSL enabled - Not able to publish messages (org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms)

hi - I've enabled SSL for Kafka & i'm trying to publish messages using console Producer Error is as shown below, any ideas ? > >1. /usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-producer.sh --broker-list >nwk2-bdp-kafka-05.gdcs-qa.apple.com:6668,nwk2-bdp-kafka-04.gdcs-qa. >apple.com:6668

Re: Kafka error after SSL enabled - Bootstrap broker-name :6667 disconnected (org.apache.kafka.clients.NetworkClient)

017296247CC84BB1F7D7842BA9B1 Session-ID-ctx: Master-Key: 795C06945CBD2BABC55A269FF46EAE6848E3834E5EAB54886E10DFD5289498901A5169AFE268872F4B0A3439DA20A378 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1500936569 Timeout : 7200 (sec) Verify return code: 18 (self signed certificate) On Mon, Jul 24, 2017 at 2:08 PM, kar

Re: Kafka error after SSL enabled - Bootstrap broker-name :6667 disconnected (org.apache.kafka.clients.NetworkClient)

.RequestSendThread.brokerReady(ControllerChannelManager.scala:233) > at > kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:182) > at > kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:181) > at kafka.utils.ShutdownableThread.run(Shutdownabl

Kafka error after SSL enabled - Bootstrap broker-name :6667 disconnected (org.apache.kafka.clients.NetworkClient)

Hello - i've enabled SSL for Kafka, and Kafka is starting up fine with SSL enable. However, when i run the Kafka console producer, it is give me error as shown below -> 1. Command : 2. 3. /usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-producer.sh --broker-list nwk2-bdp-kafka-05.gdcs-qa

Re: Kafka-Spark Integration - build failing with sbt

ation, that fixed the issue regds, Karan Alang On Wed, Jul 12, 2017 at 1:01 PM, Jozef.koval wrote: > Hi Karan, > Why do you try to submit class which does not exist? Don't you want to > create it first? > Jozef > > Sent from [ProtonMail](https://protonmail.ch), encrypted

Re: Kafka-Spark Integration - build failing with sbt

he code in Eclipse, i was not able to do that > > eg. > import org.apache.spark.streaming.kafka.KafkaUtils > > gave errors saying KafaUtils was not part of the package. > However, when i used sbt to compile - the compilation went through fine > > So, I assume additional librari

Re: Kafka logs - server.log & controller.log

sorry, missed this email Thanks for the clarification. regds, Karan Alang On Thu, Jun 29, 2017 at 6:52 AM, Tauzell, Dave wrote: > All the brokers write to server.log. The broker that happens to be the > controller will also write to the controller.log file. > > -Dave >

Error in Kafka startup with the JMX exporter - java.lang.IllegalArgumentException: Collector already registered that provides name: jmx_scrape_duration_seconds

Hi All, I'm getting error in starting up Kafka, with the JMX exporter passed as java agent. Error is as shown below, any ideas ? I'm trying to integrate Prometheus to collect Kafka metrics. [root@nwk2-bdp-kafka-06 kafka]# KAFKA_OPTS="$KAFKA_OPTS -javaagent:/usr/hdp/2.5.3.0-37/prometheus/jmx_prom

updating startup script in Apache Amabari

hi All i'm trying to configure/update Kafka startup script, so Prometheus is able to read metrics from Kafka. I need to add the following to the Broker startup script : -javaagent:/path/to/jmx_exporter.jar=$PORT:$CONFIG Where in Apache Ambari can i modify the Kafka Broker startup script ? Pls

Kafka logs - server.log & controller.log

Hi All - in Kafka, i see two different log files - server.log & controller.log files. What specifically gets logged in these log files ? i.e. what are these 2 files meant to log ?

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

ok, thanks ... i'll check that and update in case i see issues. regds, Karan Alang On Fri, Jun 23, 2017 at 12:01 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Karan, > > I think what you are seeing with `--time -1` and '--time -2` confirms that > t

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

te: 06/22/2017 06:43 PM > Subject:Re: Deleting/Purging data from Kafka topics (Kafka 0.10) > > > > Hi Karan, > > I think the issue is in verification step. Because the start and end > offsets are not going to be reset when messages are deleted. > H

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

ike in your example if this broker config > value is much higher, then the broker doesn't delete old logs regular > enough. > > --Vahid > > > > From: karan alang > To: users@kafka.apache.org > Date: 06/22/2017 12:27 PM > Subject:Deleting/Purging da

Re: Kafka 0.10 - kafka console consumer not reading the data in order that it was published

r. > > I hope that helps. > > Thanks, > Subhash > > On Thu, Jun 22, 2017 at 4:37 PM, karan alang > wrote: > > > Hi Subhash, > > > > number of partitions - 3 > > > > On Thu, Jun 22, 2017 at 12:37 PM, Subhash Sriram < > subhash.sri...@gmail.com

Re: Kafka 0.10 - kafka console consumer not reading the data in order that it was published

h Sriram > Sent: Thursday, 22 June, 21:37 > Subject: Re: Kafka 0.10 - kafka console consumer not reading the data in > order that it was published > To: users@kafka.apache.org > > > How many partitions are in your topic? On Thu, Jun 22, 2017 at 3:33 PM, > karan alang wrote: &g

Re: Kafka 0.10 - kafka console consumer not reading the data in order that it was published

Hi Subhash, number of partitions - 3 On Thu, Jun 22, 2017 at 12:37 PM, Subhash Sriram wrote: > How many partitions are in your topic? > > On Thu, Jun 22, 2017 at 3:33 PM, karan alang > wrote: > > > Hi All - > > > > version - kafka 0.10 > > I'm pu

Kafka 0.10 - kafka console consumer not reading the data in order that it was published

Hi All - version - kafka 0.10 I'm publishing data into Kafka topic using command line, and reading the data using kafka console consumer *Publish command ->* $KAFKA_HOME/bin/kafka-verifiable-producer.sh --topic mmtopic1 --max-messages 100 --broker-list localhost:9092,localhost:9093,localhost:909

Deleting/Purging data from Kafka topics (Kafka 0.10)

Hi All - How do i go about deleting data from Kafka Topics ? I've Kafka 0.10 installed. I tried setting the parameter of the topic as shown below -> $KAFKA10_HOME/bin/kafka-topics.sh --zookeeper localhost:2161 --alter --topic mmtopic6 --config retention.ms=1000 I was expecting to have the data p

Re: Kafka MirrorMaker - not replicating messages after being brought up

onfigs > > On Thu, Jun 22, 2017 at 2:00 AM, karan alang > wrote: > > > Hi All, > > > > I've 2 Kafka clusters (Kafka 10) & I'm trying to test the MirrorMaker > > functionality. > > > > Here is what i did : > > > > 1) I hav

Kafka MirrorMaker - not replicating messages after being brought up

Hi All, I've 2 Kafka clusters (Kafka 10) & I'm trying to test the MirrorMaker functionality. Here is what i did : 1) I have identical topics Topic1 on 2 Kafka clusters - Cluster1 & Cluster2 2) On Cluster1, I publish 100 messages on Topic1 3) I've 2 consumers reading messages from the 2 topics o

Re: Kafka MirrorMaker - errors/warning

owever, couple are still pending -> a) producer.type = async was supplied but isn't a known config *How do i ensure the producer is an "async" producer ?* b) queue.time = 100 was supplied but isn't a known config *What needs to be done for this ?* On Tue, Jun 20, 2017 at

New Kafka Producer or the Old One ???

Hello All - I've *Kafka 0.9* & I'm running this command to publish records to Kafka topics - $KAFKA_HOME/bin/kafka-verifiable-producer.sh --topic mmtopic1 --max-messages 500 --broker-list localhost:9092,localhost:9093,localhost:9094,localhost:9095 --producer.config $KAFKA_HOME/config/producer.pro

Kafka MirrorMaker - errors/warning

Hi All - i've setup Kafka MirrorMaker using link - Kafka Mirror Maker best Practices , and am getting the following warnings -> [2017-06-20 12:29:31,381] WARN The configuration queue.time = 100 was suppl

Re: Kafka-Spark Integration - build failing with sbt

through fine So, I assume additional libraries are being downloaded when i provide the appropriate packages in LibraryDependencies ? which ones would have helped compile this ? On Sat, Jun 17, 2017 at 2:52 PM, karan alang wrote: > Hey Jozef, > > Thanks for the quick response .. >

Re: Kafka-Spark Integration - build failing with sbt

Hey Jozef, Thanks for the quick response .. yes, you are right .. spark-sql dependency was missing .. added that & it worked fine. regds, Karan Alang On Sat, Jun 17, 2017 at 2:24 PM, Jozef.koval wrote: > Hey Karan, > I believe you are missing spark-sql dependency. > > Joz

Re: Kafka-Spark Integration - build failing with sbt

Thanks, i was able to get this working. here is what i added in build.sbt file -- scalaVersion := "2.11.7" val sparkVers = "2.1.0" // Base Spark-provided dependencies libraryDependencies ++= Seq( "org

Kafka-Spark Integration - build failing with sbt

I'm trying to compile kafka & Spark Streaming integration code i.e. reading from Kafka using Spark Streaming, and the sbt build is failing with error - [error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found Scala version

Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker

he topic 'newBroker1' exists under this ZK path? > Also, do you still get the rows below if you run the ConsumerOffsetChecker > without providing any topic? > > Thanks. > --Vahid > > > > > From: karan alang > To: users@kafka.apache.org > Date: 06/

Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker

CURRENT OFFSET, LOG END OFFSET, LAG, OWNER any ideas on what needs to be done? On Thu, Jun 15, 2017 at 11:55 AM, karan alang wrote: > Hi All - > I've Kafka 0.9 (going forward will be migrating to Kafka 0.10) & trying to > use the ConsumerOffsetChecker & bin/kafka-consumer-gr

Re: Re-Balancing Kafka topics - Best practices

Thanks Todd.. for the detailed reply. regds, Karan Alang On Tue, Jun 13, 2017 at 3:19 PM, Todd Palino wrote: > A few things here… > > 1) auto.leader.rebalance.enable can have serious performance impacts on > larger clusters. It’s currently in need of some development work to en

kafka-consumer-groups.sh vs ConsumerOffsetChecker

Hi All - I've Kafka 0.9 (going forward will be migrating to Kafka 0.10) & trying to use the ConsumerOffsetChecker & bin/kafka-consumer-groups.sh to check for offsets. I'm seeing different behavior. Here is what i did -> a) When i use ConsumerOffsetChecker $KAFKA_HOME/bin/kafka-run-class.sh kaf

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

ducer is pushing data into the Kafka Topic ? What are the implications of this ? Pls. advise. On Tue, Jun 13, 2017 at 4:29 PM, karan alang wrote: > > Hello, clarifying the question -> > > Within the same Kafka cluster, if i want to add Broker & rebalance topic > (say t

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

, > > On Tue, 13 Jun 2017 at 10:58 pm, karan alang > wrote: > > > Hello All, > > > > I've streaming data flowing into Kafka topics. > > What are the best practices wrt. adding or removing Kafka Brokers while > > data is flowing into Kafka ? > > >

Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

Hello All, I've streaming data flowing into Kafka topics. What are the best practices wrt. adding or removing Kafka Brokers while data is flowing into Kafka ? for eg. Can I add Broker & re-balance the topics while the data flows into Kafka topic ? Pls. advise. Thanks!

Re-Balancing Kafka topics - Best practices

Hi All, Fpr Re-balancing Kafka partitions, we can set property -> *auto.leader.rebalance.enable = true in server.properties file.* *Is that the recommended way or is it better to reBalance the kafka partitions manually ?(using *scripts - *kafka-preferred-replica-election.sh, * *kafka-reassign-

Re: Kafka MirrorMaker error

Thanks, that fixed the issue ! On Thu, Jun 8, 2017 at 11:58 AM, Mostafa Zarifyar wrote: > maybe this will help. > > https://community.hortonworks.com/articles/24599/kafka-mirrormaker.html > > -M > > On Thu, Jun 8, 2017 at 11:42 AM, karan alang > wrote: > > &

Kafka MirrorMaker error

Hi All, I'm trying to transfer data between kafka clusters using Kafka MirrorMaker & running into issues. I've created a consumer.config & producer.config files & using the command shown below. The error indicates - equirement failed: Missing required property 'zookeeper.connect' --