Active <> Active MirrorMaker2 setup via dedicated Kafka Connect cluster

light on how to set up active <> active mm2 via Kafka connect cluster if it is possible. I really appreciate any help you can provide. -- *Sriram G* *Tech*

Kafka server not recognizing consumer thread is down

x27;t reallocate those partitions to other kafka consumers. This results in lag in those partitions which are allocated to consumer threads which are down. Is there anything that I can do to resolve this? Thanks, Sriram

Re: Using Kafka to replace local .net message queues

Hi Akshay, In regards to your 3rd question (and indirectly to your 2nd question), instead of having different consumer groups, why not just multiple consumers in the same group? That would ensure that each consumer only reads from one partition in the topic. You can even assign the partition if yo

Re: Prioritized Topics for Kafka

Use case: we process documents from a variety of sources. We want to process some of these sources in a priority order, but we don’t want to necessarily finish all the higher priority sources before going to lower priority because the volume of higher priority sources can be extremely high. We hav

Re: SQL Server CDC

You can read more here: https://docs.confluent.io/current/connect/connect-jdbc/docs/source_connector.html Hope that helps. Thanks, Subhash Sent from my iPhone > On Jun 19, 2018, at 11:12 AM, Pranav Shah wrote: > > Hello, > > Is there connector available for Kafka SQL Server CDC? > > Thanks

Re: SQL Server CDC

Hi Pranav, Yes, there is a JDBC source connector that you can use to achieve CDC from SQL into Kafka using connect. Thanks, Subhash Sent from my iPhone > On Jun 19, 2018, at 11:12 AM, Pranav Shah wrote: > > Hello, > > Is there connector available for Kafka SQL Server CDC? > > Thanks, > Pr

Re: one topic, one partition, many consumer instances

duplicates. Hope that helps. Thanks, Subhash On Tue, Feb 13, 2018 at 10:44 AM, Xavier Noria wrote: > On Tue, Feb 13, 2018 at 2:59 PM, Subhash Sriram > wrote: > > Hey Xavier, > > > > Within a consumer group, you can only have as many consumers as you have > > parti

Re: one topic, one partition, many consumer instances

Hey Xavier, Within a consumer group, you can only have as many consumers as you have partitions in a topic. If you start more consumers than partitions within the same group, they will just be idle. Thanks, Subhash Sent from my iPhone > On Feb 13, 2018, at 8:30 AM, Xavier Noria wrote: > > C

Re: monitor consumer offset lag script/code

Hi Sunil, Burrow might be of interest to you: https://github.com/linkedin/Burrow Hope that helps. Thanks, Subhash Sent from my iPhone > On Jan 29, 2018, at 7:40 PM, Sunil Parmar wrote: > > We're using 0.9 ( CDH ) and consumer offsets are stored within Kafka. What > is the preferred way to g

Re: Training and Cert course recommendation

Hi Brian, Maybe this will be of help: https://www.confluent.io/certification/ Thanks, Subhash Sent from my iPhone > On Jan 19, 2018, at 1:15 PM, brian spallholtz > wrote: > > I am a SA/SE on several kafka clusters, I was wondering if there was a > training and certification program or trac

Re: Authorization in Kafka

Hi, I am not an expert, but from looking at the ACL documentation, you can't control read authorization at the partition level, only at the topic level. If it is possible to control access at the partition level, maybe you could have a dedicated partition for each customerID? Thanks, Subhash

Re: Kafka Monitoring

Hi Irtiza, Have you looked at jmxtrans? It has multiple output writers for the metrics and one of them is the KeyOutWriter which just writes to disk. https://github.com/jmxtrans/jmxtrans/wiki Hope that helps! Thanks, Subhash Sent from my iPhone > On Dec 6, 2017, at 5:36 AM, Irtiza Ali wrote

Re: [ANNOUCE] Apache Kafka 0.11.0.1 Released

Thanks for driving the release Damian. > On Sep 13, 2017, at 1:18 PM, Guozhang Wang wrote: > > Thanks for driving this Damian! > > > Guozhang > >> On Wed, Sep 13, 2017 at 4:36 AM, Damian Guy wrote: >> >> The Apache Kafka community is pleased to announce the release for Apache >> Kafka 0.11.

Re: Querying consumer groups programmatically (from Golang)

Hi Jens, Have you looked at Burrow? https://github.com/linkedin/Burrow/blob/master/README.md Thanks, Subhash Sent from my iPhone > On Aug 12, 2017, at 8:55 AM, Jens Rantil wrote: > > Hi, > > I am one of the maintainers of prometheus-kafka-consumer-group-exporter[1], > which exports consumer

Re: Kafka 0.10 - kafka console consumer not reading the data in order that it was published

st by publishing to a single partition topic. When you consume, you should see that it is all in order. I hope that helps. Thanks, Subhash On Thu, Jun 22, 2017 at 4:37 PM, karan alang wrote: > Hi Subhash, > > number of partitions - 3 > > On Thu, Jun 22, 2017 at 12:37 PM, Subha

Re: Kafka 0.10 - kafka console consumer not reading the data in order that it was published

How many partitions are in your topic? On Thu, Jun 22, 2017 at 3:33 PM, karan alang wrote: > Hi All - > > version - kafka 0.10 > I'm publishing data into Kafka topic using command line, > and reading the data using kafka console consumer > > *Publish command ->* > > $KAFKA_HOME/bin/kafka-verifia

Re: [ANNOUNCE] New committer: Damian Guy

Congrats Damian! Thanks for all your contributions. On Fri, Jun 9, 2017 at 2:52 PM, Martin Gainty wrote: > congratulations damian! > > > Martin > > > > From: Gwen Shapira > Sent: Friday, June 9, 2017 4:55 PM > To: users@kafka.apache.org > Cc: d...@kafka.apache.o

Re: [VOTE] KIP-156 Add option "dry run" to Streams application reset tool

+1 On Wed, May 10, 2017 at 9:45 AM, Neha Narkhede wrote: > +1 > > On Wed, May 10, 2017 at 12:32 PM Gwen Shapira wrote: > > > +1. Also not sure that adding a parameter to a CLI requires a KIP. It > seems > > excessive. > > > > > > On Tue, May 9, 2017 at 7:57 PM Jay Kreps wrote: > > > > > +1 > >

Re: Kafka 0.10.1.0 - Question about kafka-consumer-groups.sh

output, but it should get fresh > offsets (with `--describe` for example), since the old offsets were > removed once it became inactive. > > --Vahid > > > > > From: Subhash Sriram > To: users@kafka.apache.org > Date: 05/05/2017 02:38 PM > Subject:Re

Re: Kafka 0.10.1.0 - Question about kafka-consumer-groups.sh

et and it > will not be listed in the consumer group command output. > > But the consumer group in your case should be alive, since it did not > become inactive. > > Did the command use to list the group in the output before? > > --Vahid > > > > > From: Subhas

Kafka 0.10.1.0 - Question about kafka-consumer-groups.sh

Hey everyone, I am a little bit confused about how the kafka-consumer-groups.sh/ ConsumerGroupCommand works, and was hoping someone could shed some light on this for me. We are running Kafka 0.10.1.0, and using the new Consumer API with the Confluent.Kafka C# library (v0.9.5) that uses librdkafka

Re: [ANNOUNCE] New committer: Rajini Sivaram

Congrats Rajini! On Mon, Apr 24, 2017 at 2:21 PM, Onur Karaman wrote: > Congrats! > > On Mon, Apr 24, 2017 at 2:20 PM, Guozhang Wang wrote: > > > Congrats Rajini! > > > > Guozhang > > > > On Mon, Apr 24, 2017 at 2:08 PM, Vahid S Hashemian < > > vahidhashem...@us.ibm.com> wrote: > > > > > Great

How to get metrics from Kafka?

Any help would be highly appreciated. Regards, Sriram S

Kafka MTLS Support?

Team, Would like to know if MTLS is supported as a security protocol, for inter-broker communications and clients to broker communications? Could someone throw some light? Regards, Sriram S

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

StreamsBuilder would be my vote. > On Mar 13, 2017, at 9:42 PM, Jay Kreps wrote: > > Hey Matthias, > > Make sense, I'm more advocating for removing the word topology than any > particular new replacement. > > -Jay > > On Mon, Mar 13, 2017 at 12:30 PM, Matthias J. Sax > wrote: > >> Jay, >>

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

Thanks Ewen for driving this. On Wed, Feb 22, 2017 at 12:40 AM, Guozhang Wang wrote: > Thanks Ewen for driving the release! > > Guozhang > > On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava > > wrote: > > > The Apache Kafka community is pleased to announce the release for Apache > > Kafk

Re: [ANNOUNCE] New committer: Grant Henke

Congratulations Grant! On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shapira wrote: > The PMC for Apache Kafka has invited Grant Henke to join as a > committer and we are pleased to announce that he has accepted! > > Grant contributed 88 patches, 90 code reviews, countless great > comments on discussio

Re: [VOTE] KIP-106 - Default unclean.leader.election.enabled True => False

+1 On Wed, Jan 11, 2017 at 11:10 AM, Ismael Juma wrote: > Thanks for raising this, +1. > > Ismael > > On Wed, Jan 11, 2017 at 6:56 PM, Ben Stopford wrote: > > > Looks like there was a good consensus on the discuss thread for KIP-106 > so > > lets move to a vote. > > > > Please chime in if you w

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

Congratulations! On Mon, Oct 31, 2016 at 12:23 PM, Ismael Juma wrote: > Congratulations Becket. :) > > Ismael > > On 31 Oct 2016 1:44 pm, "Joel Koshy" wrote: > > > The PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a > > committer and we are pleased to announce that he has ac

Re: [VOTE] Add REST Server to Apache Kafka

-1 for all the reasons that have been described before. This does not need to be part of the core project. On Tue, Oct 25, 2016 at 3:25 PM, Suresh Srinivas wrote: > +1. > > This is an http access to core Kafka. This is very much needed as part of > Apache Kafka under ASF governance model. This

Re: [ANNOUNCE] New committer: Jason Gustafson

Congratulations Jason! On Tue, Sep 6, 2016 at 3:40 PM, Vahid S Hashemian wrote: > Congratulations Jason on this very well deserved recognition. > > --Vahid > > > > From: Neha Narkhede > To: "d...@kafka.apache.org" , > "users@kafka.apache.org" > Cc: "priv...@kafka.apache.org" > Date:

Re: [ANNOUNCE] New committer: Ismael Juma

Congratulations! On Tue, Apr 26, 2016 at 6:57 AM, Gwen Shapira wrote: > Congratulations, very well deserved. > On Apr 25, 2016 10:53 PM, "Neha Narkhede" wrote: > > > The PMC for Apache Kafka has invited Ismael Juma to join as a committer > and > > we are pleased to announce that he has accepted

Re: [DISCUSSION] adding the serializer api back to the new java producer

, one can simply use a ByteArraySerializer. On 12/5/14 5:00 PM, "Jay Kreps" wrote: >Hey Sriram, > >Thanks! I think this is a very helpful summary. > >Let me try to address your point about passing in the serde at send time. > >I think the first objection is

Re: [DISCUSSION] adding the serializer api back to the new java producer

This thread has diverged multiple times now and it would be worth summarizing them. There seems to be the following points of discussion - 1. Can we keep the serialization semantics outside the Producer interface and have simple bytes in / bytes out for the interface (This is what we have today)

Re: 0.8.1 stability

Auto rebalance is already turned off in 0.8.1. On 3/18/14 5:56 PM, "Neha Narkhede" wrote: >Thanks for giving the 0.8.1 release a spin! A few people have reported >bugs >with delete topic and >also the automatic leader >rebalancing

Re: Config for new clients (and server)

+1 on Jun's suggestion. On 2/10/14 2:01 PM, "Jun Rao" wrote: >I actually prefer to see those at INFO level. The reason is that the >config >system in an application can be complex. Some configs can be overridden in >different layers and it may not be easy to determine what the final >binding >va

Apache Samza talk in LA on Feb 18

in LA who are interested in Samza in the meetup. Also, don't forget to register on the meetup link below. http://www.meetup.com/LA-HUG/events/160628632/ Sriram

Flush producer queue

I am using kafka 0.7.2. I want to make sure that the producer queue is flushed before I can call a close on the producer. Is there a way to achieve this? Thanks, Sriram

Re: unclean shutdown

Is this a bug? Can an unclean shutdown leave the index in a corrupt state that the broker cannot start (expected state)? On 10/14/13 4:52 PM, "Neha Narkhede" wrote: > It is possible that the unclean broker shutdown left the index in an >inconsistent state. You can delete the corrupted index file

Re: Question about auto-rebalancing

We already have a JIRA for auto rebalance. I would be working on this soon. KAFKA-930 On 10/11/13 5:39 PM, "Guozhang Wang" wrote: >Hello Siyuan, > >For the automatic leader re-election, yes we are considering to make it >work. Could you file a

Re: Strategies for auto generating broker ID

uster to dynamically rebalance load, if new >nodes are added to the cluster (or to at least prefer assigning new >partitions to brokers which have more space available). I expect this >will >be something to prioritize in the future versions as well. > >Jason > > >On Wed, Oct

Re: Strategies for auto generating broker ID

I agree that we need a unique id and have something independent of the machine. I am not sure you want a dependency on ZK to generate the unique id though. There are other ways to generate an unique id (Example - UUID). In case there was a collision (highly unlikely), the node creation in ZK will a

Re: Patch for mmap + windows

I did take a look at KAFKA-1008 a while back and added some comments. On 9/9/13 3:52 PM, "Jay Kreps" wrote: >Cool can we get a reviewer for KAFKA-1008 then? I can take on the other >issue for the checkpoint files. > >-Jay > > >On Mon, Sep 9, 2013 at 3:16 PM, Neha Narkhede >wrote: > >> +1 for win

Re: problems reasignating partitions

I have few questions before I can help you with your issue - 1. Which version/build of Kafka are you using? 2. Is the current status that you provided below before or after issuing the reassign replica command? 3. If the answer to 2 above is "after issuing the command", how long was it after which

Re: compression performance

We need to first decide on the right behavior before optimizing on the implementation. Few key goals that I would put forward are - 1. Decoupling compression codec of the producer and the log 2. Ensuring message validity by the server on receiving bytes. This is done by the iterator today and thi

Re: Stale TopicMetadata

We need to improve how the metadata caching works in kafka. Currently, we have multiple places where we send the updated metadata to the individual brokers from the controller when the state of the metadata changes. This is hard to track. What we need to implement is to let the metadata structure i

Re: Does ShutdownBroker returns meaningful return code

One thing to note is that we do support controlled shutdown as part of the regular shutdown hook in the broker. The wiki was not very clear w.r.t this and I have updated it to convey this. You can turn on controlled shutdown by setting "controlled.shutdown.enable" to true in kafka config. This will

Re: Error when processing messages in Windows

As far as I am aware it is not possible to resize mapped buffer without unmapping in Windows. W.r.t Java the bug here gives more context on why it does not support synchronous unmap function. http://bugs.sun.com/view_bug.do?bug_id=4724038 On 7/9/13 9:54 PM, "Jay Kreps" wrote: >The problem app

Re: site updates

ndle requests.") >> > If we did it this way we could actually have a dumpConfigs method that >> > would print out the up-to-date table of configs so we could more >> > easily keep the docs in sync. >> > >> > For the time bein

Re: site updates

On 6/28/13 2:48 PM, "Sriram Subramanian" wrote: >1. I have moved the FAQ to a wiki. I have separated the sections into >producer, consumers and broker related questions. I would still need to >add replication FAQ. The main FAQ will now link to this. Let me know if >you gu

Re: site updates

7:17 PM, "Sriram Subramanian" wrote: >Looks much better. > >1. We need to update FAQ for 0.8 >2. We should probably have a separate section for implementation. >3. The migration tool explanation seems to be hard to get to. > > > >On 6/27/13 5:40 PM, "Jay

Re: site updates

Looks much better. 1. We need to update FAQ for 0.8 2. We should probably have a separate section for implementation. 3. The migration tool explanation seems to be hard to get to. On 6/27/13 5:40 PM, "Jay Kreps" wrote: >Hey Folks, > >I did a pass on the website. Changes: >1. Rewrote the 0.8 q

Re: batch sending in sync mode

The messages will be grouped by their destination broker and further grouped by topic/partition. The send then happens to each broker with a list of topic/partitions and messages for them and waits for an acknowledgement from each broker. This happens sequentially. So, the messages are acknowledged

Re: preferred leader election...

Hi Jason, A rolling bounce will create an imbalance in the number of leaders distribution between the brokers and this is not ideal. We do plan to have the preferred leader election tool integrated into kafka that periodically balances the leader count across the brokers in the cluster. For now yo

Re: produce request failed: due to Leader not local for partition

Hey Jason, The producer on failure initiates a metadata request to refresh its state and should issue subsequent requests to the new leader. The errors that you see should only happen once per topic partition per producer. Let me know if this is not what you see. On the producer end you should see