[DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig
Hi there, Resending as probably missed earlier to grab your attention. Regards, Umesh -- Forwarded message - From: UMESH CHAUDHARY Date: Mon, 3 Jul 2017 at 11:04 Subject: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig To: d...@kafka.apache.org Hello All, I have added a KIP recently to deprecate and remove internal converter configs in WorkerConfig.java class because these have ultimately just caused a lot more trouble and confusion than it is worth. Please find the KIP here <https://cwiki.apache.org/confluence/display/KAFKA/KIP-174+-+Deprecate+and+remove+internal+converter+configs+in+WorkerConfig> and the related JIRA here <https://issues.apache.org/jira/browse/KAFKA-5540>. Appreciate your review and comments. Regards, Umesh
Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig
Hi Ewen, Thanks for your comments. 1) Yes, there are some test and java classes which refer these configs, so I will include them as well in "public interface" section of KIP. What should be our approach to deal with the classes and tests which use these configs: we need to change them to use JsonConverter when we plan for removal of these configs right? 2) I believe we can target the deprecation in 1.0.0 release as it is planned in October 2017 and then removal in next major release. Let me know your thoughts as we don't have any information for next major release (next to 1.0.0) yet. 3) Thats a good point and mentioned JIRA can help us to validate the usage of any other converters. I will list this down in the KIP. Let me know if you have some additional thoughts on this. Regards, Umesh On Wed, 26 Jul 2017 at 09:27 Ewen Cheslack-Postava wrote: > Umesh, > > Thanks for the KIP. Straightforward and I think it's a good change. > Unfortunately it is hard to tell how many people it would affect since we > can't tell how many people have adjusted that config, but I think this is > the right thing to do long term. > > A couple of quick things that might be helpful to refine: > > * Note that there are also some references in the example configs that we > should remove. > * It's nice to be explicit about when the removal is planned. This lets us > set expectations with users for timeframe (especially now that we have time > based releases), allows us to give info about the removal timeframe in log > error messages, and lets us file a JIRA against that release so we remember > to follow up. Given the update to 1.0.0 for the next release, we may also > need to adjust how we deal with deprecations/removal if we don't want to > have to wait all the way until 2.0 to remove (though it is unclear how > exactly we will be handling version bumps from now on). > * Migration path -- I think this is the major missing gap in the KIP. Do we > need a migration path? If not, presumably it is because people aren't using > any other converters in practice. Do we have some way of validating this ( > https://issues.apache.org/jira/browse/KAFKA-3988 might be pretty > convincing > evidence)? If there are some users using other converters, how would they > migrate to newer versions which would no longer support that? > > -Ewen > > > On Fri, Jul 14, 2017 at 2:37 AM, UMESH CHAUDHARY > wrote: > > > Hi there, > > Resending as probably missed earlier to grab your attention. > > > > Regards, > > Umesh > > > > -- Forwarded message - > > From: UMESH CHAUDHARY > > Date: Mon, 3 Jul 2017 at 11:04 > > Subject: [DISCUSS] KIP-174 - Deprecate and remove internal converter > > configs in WorkerConfig > > To: d...@kafka.apache.org > > > > > > Hello All, > > I have added a KIP recently to deprecate and remove internal converter > > configs in WorkerConfig.java class because these have ultimately just > > caused a lot more trouble and confusion than it is worth. > > > > Please find the KIP here > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 174+-+Deprecate+and+remove+internal+converter+configs+in+WorkerConfig> > > and > > the related JIRA here <https://issues.apache.org/jira/browse/KAFKA-5540 > >. > > > > Appreciate your review and comments. > > > > Regards, > > Umesh > > >
Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig
Hi Ewen, Sorry, I am bit late in responding this. Thanks for your inputs and I've updated the KIP by adding more details to it. Regards, Umesh On Mon, 31 Jul 2017 at 21:51 Ewen Cheslack-Postava wrote: > On Sun, Jul 30, 2017 at 10:21 PM, UMESH CHAUDHARY > wrote: > >> Hi Ewen, >> Thanks for your comments. >> >> 1) Yes, there are some test and java classes which refer these configs, >> so I will include them as well in "public interface" section of KIP. What >> should be our approach to deal with the classes and tests which use these >> configs: we need to change them to use JsonConverter when we plan for >> removal of these configs right? >> > > I actually meant the references in config/connect-standalone.properties > and config/connect-distributed.properties > > >> 2) I believe we can target the deprecation in 1.0.0 release as it is >> planned in October 2017 and then removal in next major release. Let me >> know your thoughts as we don't have any information for next major release >> (next to 1.0.0) yet. >> > > That sounds fine. Tough to say at this point what our approach to major > version bumps will be since the approach to version numbering is changing a > bit. > > >> 3) Thats a good point and mentioned JIRA can help us to validate the >> usage of any other converters. I will list this down in the KIP. >> >> Let me know if you have some additional thoughts on this. >> >> Regards, >> Umesh >> >> >> >> On Wed, 26 Jul 2017 at 09:27 Ewen Cheslack-Postava >> wrote: >> >>> Umesh, >>> >>> Thanks for the KIP. Straightforward and I think it's a good change. >>> Unfortunately it is hard to tell how many people it would affect since we >>> can't tell how many people have adjusted that config, but I think this is >>> the right thing to do long term. >>> >>> A couple of quick things that might be helpful to refine: >>> >>> * Note that there are also some references in the example configs that we >>> should remove. >>> * It's nice to be explicit about when the removal is planned. This lets >>> us >>> set expectations with users for timeframe (especially now that we have >>> time >>> based releases), allows us to give info about the removal timeframe in >>> log >>> error messages, and lets us file a JIRA against that release so we >>> remember >>> to follow up. Given the update to 1.0.0 for the next release, we may also >>> need to adjust how we deal with deprecations/removal if we don't want to >>> have to wait all the way until 2.0 to remove (though it is unclear how >>> exactly we will be handling version bumps from now on). >>> * Migration path -- I think this is the major missing gap in the KIP. Do >>> we >>> need a migration path? If not, presumably it is because people aren't >>> using >>> any other converters in practice. Do we have some way of validating this >>> ( >>> https://issues.apache.org/jira/browse/KAFKA-3988 might be pretty >>> convincing >>> evidence)? If there are some users using other converters, how would they >>> migrate to newer versions which would no longer support that? >>> >>> -Ewen >>> >>> >>> On Fri, Jul 14, 2017 at 2:37 AM, UMESH CHAUDHARY >>> wrote: >>> >>> > Hi there, >>> > Resending as probably missed earlier to grab your attention. >>> > >>> > Regards, >>> > Umesh >>> > >>> > -- Forwarded message - >>> > From: UMESH CHAUDHARY >>> > Date: Mon, 3 Jul 2017 at 11:04 >>> > Subject: [DISCUSS] KIP-174 - Deprecate and remove internal converter >>> > configs in WorkerConfig >>> > To: d...@kafka.apache.org >>> > >>> > >>> > Hello All, >>> > I have added a KIP recently to deprecate and remove internal converter >>> > configs in WorkerConfig.java class because these have ultimately just >>> > caused a lot more trouble and confusion than it is worth. >>> > >>> > Please find the KIP here >>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>> > 174+-+Deprecate+and+remove+internal+converter+configs+in+WorkerConfig> >>> > and >>> > the related JIRA here < >>> https://issues.apache.org/jira/browse/KAFKA-5540>. >>> > >>> > Appreciate your review and comments. >>> > >>> > Regards, >>> > Umesh >>> > >>> >>
Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig
Thanks Ewen, I just edited the KIP to reflect the changes. Regards, Umesh On Wed, 9 Aug 2017 at 11:00 Ewen Cheslack-Postava wrote: > Great, looking good. I'd probably be a bit more concrete about the > Proposed Changes (e.g., "will log an warning if the config is specified" > and "since the JsonConverter is the default, the configs will be removed > immediately from the example worker configuration files"). > > Other than that this LGTM and I'll be happy to get rid of those settings! > > -Ewen > > On Tue, Aug 8, 2017 at 2:54 AM, UMESH CHAUDHARY > wrote: > >> Hi Ewen, >> Sorry, I am bit late in responding this. >> >> Thanks for your inputs and I've updated the KIP by adding more details to >> it. >> >> Regards, >> Umesh >> >> On Mon, 31 Jul 2017 at 21:51 Ewen Cheslack-Postava >> wrote: >> >>> On Sun, Jul 30, 2017 at 10:21 PM, UMESH CHAUDHARY >>> wrote: >>> >>>> Hi Ewen, >>>> Thanks for your comments. >>>> >>>> 1) Yes, there are some test and java classes which refer these configs, >>>> so I will include them as well in "public interface" section of KIP. What >>>> should be our approach to deal with the classes and tests which use these >>>> configs: we need to change them to use JsonConverter when we plan for >>>> removal of these configs right? >>>> >>> >>> I actually meant the references in config/connect-standalone.properties >>> and config/connect-distributed.properties >>> >>> >>>> 2) I believe we can target the deprecation in 1.0.0 release as it is >>>> planned in October 2017 and then removal in next major release. Let me >>>> know your thoughts as we don't have any information for next major release >>>> (next to 1.0.0) yet. >>>> >>> >>> That sounds fine. Tough to say at this point what our approach to major >>> version bumps will be since the approach to version numbering is changing a >>> bit. >>> >>> >>>> 3) Thats a good point and mentioned JIRA can help us to validate the >>>> usage of any other converters. I will list this down in the KIP. >>>> >>>> Let me know if you have some additional thoughts on this. >>>> >>>> Regards, >>>> Umesh >>>> >>>> >>>> >>>> On Wed, 26 Jul 2017 at 09:27 Ewen Cheslack-Postava >>>> wrote: >>>> >>>>> Umesh, >>>>> >>>>> Thanks for the KIP. Straightforward and I think it's a good change. >>>>> Unfortunately it is hard to tell how many people it would affect since >>>>> we >>>>> can't tell how many people have adjusted that config, but I think this >>>>> is >>>>> the right thing to do long term. >>>>> >>>>> A couple of quick things that might be helpful to refine: >>>>> >>>>> * Note that there are also some references in the example configs that >>>>> we >>>>> should remove. >>>>> * It's nice to be explicit about when the removal is planned. This >>>>> lets us >>>>> set expectations with users for timeframe (especially now that we have >>>>> time >>>>> based releases), allows us to give info about the removal timeframe in >>>>> log >>>>> error messages, and lets us file a JIRA against that release so we >>>>> remember >>>>> to follow up. Given the update to 1.0.0 for the next release, we may >>>>> also >>>>> need to adjust how we deal with deprecations/removal if we don't want >>>>> to >>>>> have to wait all the way until 2.0 to remove (though it is unclear how >>>>> exactly we will be handling version bumps from now on). >>>>> * Migration path -- I think this is the major missing gap in the KIP. >>>>> Do we >>>>> need a migration path? If not, presumably it is because people aren't >>>>> using >>>>> any other converters in practice. Do we have some way of validating >>>>> this ( >>>>> https://issues.apache.org/jira/browse/KAFKA-3988 might be pretty >>>>> convincing >>>>> evidence)? If there are some users using other converters, how would >>>>> they >>>>> migrate to newer versions which would no longer support that? >>>>> >>>>> -Ewen >>>>> >>>>> >>>>> On Fri, Jul 14, 2017 at 2:37 AM, UMESH CHAUDHARY >>>>> wrote: >>>>> >>>>> > Hi there, >>>>> > Resending as probably missed earlier to grab your attention. >>>>> > >>>>> > Regards, >>>>> > Umesh >>>>> > >>>>> > -- Forwarded message - >>>>> > From: UMESH CHAUDHARY >>>>> > Date: Mon, 3 Jul 2017 at 11:04 >>>>> > Subject: [DISCUSS] KIP-174 - Deprecate and remove internal converter >>>>> > configs in WorkerConfig >>>>> > To: d...@kafka.apache.org >>>>> > >>>>> > >>>>> > Hello All, >>>>> > I have added a KIP recently to deprecate and remove internal >>>>> converter >>>>> > configs in WorkerConfig.java class because these have ultimately just >>>>> > caused a lot more trouble and confusion than it is worth. >>>>> > >>>>> > Please find the KIP here >>>>> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>> > >>>>> 174+-+Deprecate+and+remove+internal+converter+configs+in+WorkerConfig> >>>>> > and >>>>> > the related JIRA here < >>>>> https://issues.apache.org/jira/browse/KAFKA-5540>. >>>>> > >>>>> > Appreciate your review and comments. >>>>> > >>>>> > Regards, >>>>> > Umesh >>>>> > >>>>> >>>> >
Re: Consumer stops consuming after one of the broker stopped
Are you seeing any errors in consumer when your broker gets down? May be any debug messages would be helpful. After killing the broker do you notice any leadership change for those topic's partitions using [kafka-topics.sh --zookeepr <> --describe] ? On Fri, 11 Aug 2017 at 06:23 Rakesh Singh wrote: > I have installed kafka_2.11-0.11.0.0 and using java to create producer and > consumer > > I have 3 brokers server running on the same unix machine . > > I created a topic with 3 partition with a replication factor of 2. > > I started a very simple producer with key/value as auto incrementing > integers. > > I have one consumer running and it consumes data appropriately from all > the partitions. > > Now I kill one of the broker server manually using kill command. At this > point the consumer stops consuming but the producer is able to send > messages. > > When I run the consumer-group command I get the below error: > > *Error: Executing consumer group command failed due to The consumer group > command timed out while waiting for group to initialize:* > > If I start the broker server again then the consumer starts consuming > message from the point the broke was stopped. > > Any help is appreciated! > > Thanks > > Rakesh > > > > > -- > Thanks & Regards > Rakesh Singh > 732-829-7244 <073282%2097244> >
Re: [ANNOUNCE] Apache Kafka 1.0.0 Released
ed streaming platform with four four core > APIs: > > > > ** The Producer API allows an application to publish a stream records to > > one > > or more Kafka topics. > > > > ** The Consumer API allows an application to subscribe to one or more > > topics > > and process the stream of records produced to them. > > > > ** The Streams API allows an application to act as a stream processor, > > consuming > > an input stream from one or more topics and producing an output stream to > > one or more output topics, effectively transforming the input streams to > > output streams. > > > > ** The Connector API allows building and running reusable producers or > > consumers > > that connect Kafka topics to existing applications or data systems. For > > example, a connector to a relational database might capture every change > to > > a table.three key capabilities: > > > > > > With these APIs, Kafka can be used for two broad classes of application: > > > > ** Building real-time streaming data pipelines that reliably get data > > between > > systems or applications. > > > > ** Building real-time streaming applications that transform or react > > to the streams > > of data. > > > > > > Apache Kafka is in use at large and small companies worldwide, including > > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, > > Target, The New York Times, Uber, Yelp, and Zalando, among others. > > > > > > A big thank you for the following 108 contributors to this release! > > > > Abhishek Mendhekar, Xi Hu, Andras Beni, Andrey Dyachkov, Andy Chambers, > > Apurva Mehta, Armin Braun, Attila Kreiner, Balint Molnar, Bart De Vylder, > > Ben Stopford, Bharat Viswanadham, Bill Bejeck, Boyang Chen, Bryan > Baugher, > > Colin P. Mccabe, Koen De Groote, Dale Peakall, Damian Guy, Dana Powers, > > Dejan Stojadinović, Derrick Or, Dong Lin, Zhendong Liu, Dustin Cote, > > Edoardo Comar, Eno Thereska, Erik Kringen, Erkan Unal, Evgeny > Veretennikov, > > Ewen Cheslack-Postava, Florian Hussonnois, Janek P, Gregor Uhlenheuer, > > Guozhang Wang, Gwen Shapira, Hamidreza Afzali, Hao Chen, Jiefang He, > Holden > > Karau, Hooman Broujerdi, Hugo Louro, Ismael Juma, Jacek Laskowski, Jakub > > Scholz, James Cheng, James Chien, Jan Burkhardt, Jason Gustafson, Jeff > > Chao, Jeff Klukas, Jeff Widman, Jeremy Custenborder, Jeyhun Karimov, > > Jiangjie Qin, Joel Dice, Joel Hamill, Jorge Quilcate Otoya, Kamal C, > Kelvin > > Rutt, Kevin Lu, Kevin Sweeney, Konstantine Karantasis, Perry Lee, Magnus > > Edenhill, Manikumar Reddy, Manikumar Reddy O, Manjula Kumar, Mariam John, > > Mario Molina, Matthias J. Sax, Max Zheng, Michael Andre Pearce, Michael > > André Pearce, Michael G. Noll, Michal Borowiecki, Mickael Maison, Nick > > Pillitteri, Oleg Prozorov, Onur Karaman, Paolo Patierno, Pranav Maniar, > > Qihuang Zheng, Radai Rosenblatt, Alex Radzish, Rajini Sivaram, Randall > > Hauch, Richard Yu, Robin Moffatt, Sean McCauliff, Sebastian Gavril, Siva > > Santhalingam, Soenke Liebau, Stephane Maarek, Stephane Roset, Ted Yu, > > Thibaud Chardonnens, Tom Bentley, Tommy Becker, Umesh Chaudhary, Vahid > > Hashemian, Vladimír Kleštinec, Xavier Léauté, Xianyang Liu, Xin Li, > Linhua > > Xin > > > > > > We welcome your help and feedback. For more information on how to report > > problems, and to get involved, visit the project website at > > http://kafka.apache.org/ > > > > > > > > > > Thanks, > > Guozhang Wang > > >
Re: [ANNOUNCE] New committer: Onur Karaman
Congratulations Onur! On Tue, 7 Nov 2017 at 21:44 Jun Rao wrote: > Affan, > > All known problems in the controller are described in the doc linked from > https://issues.apache.org/jira/browse/KAFKA-5027. > > Thanks, > > Jun > > On Mon, Nov 6, 2017 at 11:00 PM, Affan Syed wrote: > > > Congrats Onur, > > > > Can you also share the document where all known problems are listed; I am > > assuming these bugs are still valid for the current stable release. > > > > Affan > > > > - Affan > > > > On Mon, Nov 6, 2017 at 10:24 PM, Jun Rao wrote: > > > > > Hi, everyone, > > > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer > Onur > > > Karaman. > > > > > > Onur's most significant work is the improvement of Kafka controller, > > which > > > is the brain of a Kafka cluster. Over time, we have accumulated quite a > > few > > > correctness and performance issues in the controller. There have been > > > attempts to fix controller issues in isolation, which would make the > code > > > base more complicated without a clear path of solving all problems. > Onur > > is > > > the one who took a holistic approach, by first documenting all known > > > issues, writing down a new design, coming up with a plan to deliver the > > > changes in phases and executing on it. At this point, Onur has > completed > > > the two most important phases: making the controller single threaded > and > > > changing the controller to use the async ZK api. The former fixed > > multiple > > > deadlocks and race conditions. The latter significantly improved the > > > performance when there are many partitions. Experimental results show > > that > > > Onur's work reduced the controlled shutdown time by a factor of 100 > times > > > and the controller failover time by a factor of 3 times. > > > > > > Congratulations, Onur! > > > > > > Thanks, > > > > > > Jun (on behalf of the Apache Kafka PMC) > > > > > >
List all Kafka consumers
Hi Folks, Was searching for some command to list all connected(live) producers to Kafka brokers. Is there any such tool to get the live producers and consumers? Regards, Umesh
Question regarding functionality of MirrorMaker
Hey Folks, I was trying to understand the behavior of MirrorMaker but looks like I am missing something here. Please see the steps which I performed : 1) I configured MM on source Kafka cluster 2) Created a topic and pushed some data in it using console producer. 3) My understanding is that MM would start mirroring the data (which is there in the topic) based on "offsetCommitIntervalMs" and it would be there in destination cluster. https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/tools/MirrorMaker.scala#L503 4) But when I list the topics on destination, I cant see the topic which I recently created on source. 5) I tried to check the offset of "mirrormaker_group" for that topic (on source cluster) using kafka.admin.ConsumerGroupCommand, I see the offsets for that topic as "unknown". 6) But when I start console consumer for that topic on source or destination (auto creation of topic is true), I see that all data in being mirrored via MM and kafka.admin.ConsumerGroupCommand tells the right offsets this time. Is this expected behavior of MM or did I mess up with some configuration? Regards, Umesh
Re: Question regarding functionality of MirrorMaker
Hello Mate, Thanks for your detailed response and it surely helps. WhiteList is the required config for MM from 0.9.0 onwards. And you are correct that --new-consumer requires --bootstrap-servers rather than --zookeeper . However, did you notice that MM picks the topics which are present at the time of its startup and mirrors the data. When you add some new topics after its startup it doesn't pick it automatically? Regards, Umesh Chaudhary On Thu, 25 Aug 2016 at 19:23 cs user wrote: > Hi Umesh, > > I am new to kafka as well, and configuring the MirrorMaker. I got mine > working in the following way. > > I run the mirror maker instance on the mirror cluster, as in where you want > to mirror the topics to, although I'm not sure it matters. > > I use the following options when starting my service (systemd file): > > KAFKA_RUN="/opt/kafka/bin/kafka-run-class.sh" > KAFKA_ARGS="kafka.tools.MirrorMaker" > KAFKA_CONFIG="--new.consumer --offset.commit.interval.ms=5000 > --consumer.config /opt/kafka/config/consumer-mirror1.properties > --producer.config /opt/kafka/config/producer.properties --whitelist=\".*\"" > > Without the --new.consumer parameter, the --consumer.config and > producer.config files need to contain the zookeeper config for relevant > clusters. When using the --new.consumer switch this is no longer needed (as > I understand it). > > The consumer config points at my source cluster, the producer config points > locally, to my mirror cluster. I think it's also important to configure the > whitelist to tell it which topics you want to mirror, in my case I mirror > all of them with a wildcard. > > Not much config in the consumer.config and producer.config files apart from > the bootstrap.servers list, pointing at the relevant cluster. I have 3 > brokers in my mirror cluster and each one of them runs the same mirror > maker service so one will take over if another one fails. > > I hope someone will correct me if I am wrong about anything, and hopefully > this will help! > > Cheers! > > > > > > > > On Thu, Aug 25, 2016 at 9:36 AM, UMESH CHAUDHARY > wrote: > > > Hey Folks, > > I was trying to understand the behavior of MirrorMaker but looks like I > am > > missing something here. Please see the steps which I performed : > > > > 1) I configured MM on source Kafka cluster > > 2) Created a topic and pushed some data in it using console producer. > > 3) My understanding is that MM would start mirroring the data (which is > > there in the topic) based on "offsetCommitIntervalMs" and it would be > there > > in destination cluster. > > > > https://github.com/apache/kafka/blob/0.9.0/core/src/ > > main/scala/kafka/tools/MirrorMaker.scala#L503 > > > > 4) But when I list the topics on destination, I cant see the topic which > I > > recently created on source. > > 5) I tried to check the offset of "mirrormaker_group" for that topic (on > > source cluster) using kafka.admin.ConsumerGroupCommand, I see the offsets > > for that topic as "unknown". > > 6) But when I start console consumer for that topic on source or > > destination (auto creation of topic is true), I see that all data in > being > > mirrored via MM and kafka.admin.ConsumerGroupCommand tells the right > > offsets this time. > > > > Is this expected behavior of MM or did I mess up with some configuration? > > > > Regards, > > Umesh > > >
Re: Question regarding functionality of MirrorMaker
Hello cs, Apologies for delayed response. I found one topic in my Kafka env which has no leaders and no replicas. That was pretty weird and I am not sure what caused this. Because of this topic MirrorMaker was hanging and printing messages like "No leader found for topic ..". Due to this hang MM was not able to replicate other topics. When I deleted that zombie topic, MM worked as expected. Regards, Umesh Chaudhary On Fri, 26 Aug 2016 at 12:40 cs user wrote: > Hi Umesh, > > I haven't had that problem, it seems to work fine for me. The only issue I > found, which kind of makes sense, it that it doesn't mirror existing topics > immediately, only when messages are first set to the topic after mirror > maker connects. It doesn't start from the first offset available, only the > current one. > > However once you start sending messages it seems to subscribe to them fine > and they get created on the mirror maker cluster, same for new topics which > are created on the source cluster, they seem to come over fine. > > Only thing I can think of is that you have disabled auto topic creation on > the mirror maker cluster so that mirror maker is unable to create them > automatically? But then it wouldn't be able to create the existing topics > either so that doesn't make sense. > > Are there any error messages in your mirror maker logs or on the mirror > maker cluster which point to what the issue might be? > > Other than the boostrap servers, my producer settings look as follows: > > producer.type=async > compression.codec=0 > serializer.class=kafka.serializer.DefaultEncoder > max.message.size=1000 > queue.time=1000 > queue.enqueueTimeout.ms=-1 > > > > Cheers! > > > > > On Fri, Aug 26, 2016 at 6:08 AM, UMESH CHAUDHARY > wrote: > > > Hello Mate, > > Thanks for your detailed response and it surely helps. > > > > WhiteList is the required config for MM from 0.9.0 onwards. And you are > > correct that --new-consumer requires --bootstrap-servers rather than > > --zookeeper . > > > > However, did you notice that MM picks the topics which are present at the > > time of its startup and mirrors the data. When you add some new topics > > after its startup it doesn't pick it automatically? > > > > Regards, > > Umesh Chaudhary > > > > On Thu, 25 Aug 2016 at 19:23 cs user wrote: > > > > > Hi Umesh, > > > > > > I am new to kafka as well, and configuring the MirrorMaker. I got mine > > > working in the following way. > > > > > > I run the mirror maker instance on the mirror cluster, as in where you > > want > > > to mirror the topics to, although I'm not sure it matters. > > > > > > I use the following options when starting my service (systemd file): > > > > > > KAFKA_RUN="/opt/kafka/bin/kafka-run-class.sh" > > > KAFKA_ARGS="kafka.tools.MirrorMaker" > > > KAFKA_CONFIG="--new.consumer --offset.commit.interval.ms=5000 > > > --consumer.config /opt/kafka/config/consumer-mirror1.properties > > > --producer.config /opt/kafka/config/producer.properties > > --whitelist=\".*\"" > > > > > > Without the --new.consumer parameter, the --consumer.config and > > > producer.config files need to contain the zookeeper config for relevant > > > clusters. When using the --new.consumer switch this is no longer needed > > (as > > > I understand it). > > > > > > The consumer config points at my source cluster, the producer config > > points > > > locally, to my mirror cluster. I think it's also important to configure > > the > > > whitelist to tell it which topics you want to mirror, in my case I > mirror > > > all of them with a wildcard. > > > > > > Not much config in the consumer.config and producer.config files apart > > from > > > the bootstrap.servers list, pointing at the relevant cluster. I have 3 > > > brokers in my mirror cluster and each one of them runs the same mirror > > > maker service so one will take over if another one fails. > > > > > > I hope someone will correct me if I am wrong about anything, and > > hopefully > > > this will help! > > > > > > Cheers! > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 25, 2016 at 9:36 AM, UMESH CHAUDHARY > > > wrote: > > > > > > > Hey Folks, > > > > I was trying to understand the behavior of MirrorMaker but looks
Re: which port should I use 9091 or 9092 or 2181 to send messages through kafka when using a client Library?
No that is not required, when you use new consumer API. You have to specify bootstrap.servers, which will have 9092 (for PLAINTEXT usually ). In old consumer API you need zookeeper server which points on 2181. On Thu, 15 Sep 2016 at 17:03 kant kodali wrote: > I haven't changed anything from > https://github.com/apache/kafka/blob/trunk/config/server.properties > and it looks like it is pointing to zookeeper. > Question: > Does producer client need to point 9092 and Consumer need to point 2181? > is that > the standard? Why not both point to the same thing? > > > > > > > On Thu, Sep 15, 2016 4:24 AM, Ali Akhtar ali.rac...@gmail.com > wrote: > Examine server.properties and see which port you're using in there > > > > > On Thu, Sep 15, 2016 at 3:52 PM, kant kodali wrote: > > > > > > which port should I use 9091 or 9092 or 2181 to send messages through > kafka > > > when using a client Library? > > > I start kafka as follows: > > > sudo bin/zookeeper-server-start.sh config/zookeeper.propertiessudo > > > ./bin/kafka-server-start.sh config/server.properties > > > > > > and I dont see any process running on 9091 or 9092 however lot of client > > > library > > > examples have a consumer client pointing to 9092. for example here > > > https://github.com/apache/kafka/blob/trunk/examples/src/main > > > /java/kafka/examples/Producer.java#L34 > > > shouldn't both producer and consumer point to zookeeper port 2181? which > I > > > am > > > assuming will do the lookup? > > > Thanks,Kant
Re: Apache Kafka integration using Apache Camel
Did you test that kafka console consumer is displaying the produced
message?
On Fri, Jan 6, 2017 at 9:18 AM, Gupta, Swati wrote:
> Hello All,
>
>
>
> I am trying to create a Consumer using Apache Camel for a topic in Apache
> Kafka.
> I am using Camel 2.17.0 and Kafka 0.10 and JDK 1.8.
> I have attached a file, KafkaCamelTestConsumer.java which is a standalone
> application trying to read from a topic “test1”created in Apache Kafka
> I am producing messages from the console and also was successful to
> produce messages using a Camel program in the topic "test1", but not able
> to consume messages. Ideally, it should get printed, but nothing seems to
> happen. The log says that the route has started but does not process any
> message.
>
> Please help to confirm if there is anything wrong with the below syntax:
>
> from(*"kafka:localhost:9092?topic=test1&groupId=testingGroupNew&autoOffsetReset=earliest"
> *+
>
> *"&consumersCount=1&keyDeserializer=org.apache.kafka.common.serialization.StringDeserializer&"
> *+
> *"valueDeserializer=org.apache.kafka.common.serialization.StringDeserializer"
> *+
> *"&autoCommitIntervalMs=1000&sessionTimeoutMs=3&autoCommitEnable=true"*
> ).split()
> .body()
> .process(*new *Processor() {
> @Override
> *public void *process(Exchange exchange)
> *throws *Exception {
> String messageKey = *""*;
> *if *(exchange.getIn() != *null*) {
> Message message = exchange.getIn();
> Integer partitionId = (Integer) message
> .getHeader(KafkaConstants.*PARTITION*
> );
> String topicName = (String) message
> .getHeader(KafkaConstants.*TOPIC*);
> *if *(message.getHeader(KafkaConstants.*KEY*)
> != *null*)
> messageKey = (String) message
> .getHeader(KafkaConstants.*KEY*);
> Object data = message.getBody();
>
>
> System.*out*.println(
> *"topicName :: " *+ topicName +
> *" partitionId :: " *+ partitionId +
> *" messageKey :: " *+ messageKey +
> *" message :: " *+ data + *"**\n**"*);
> }
> }
> }).to(
> *"file://C:/swati/?fileName=MyOutputFile.txt&charset=utf-8"*);
> }
> });
>
>
>
> I have also tried with the basic parameters as below and it still fails to
> read messages.
>
> from(
> *"kafka:localhost:9092?topic=test1&groupId=testingGroupNew&autoOffsetReset=earliest")*
>
> Any help on this will be greatly appreciated.
>
> Thanks in advance
>
>
>
> Thanks & Regards
>
> Swati
>
> --
> This e-mail and any attachments to it (the "Communication") is, unless
> otherwise stated, confidential, may contain copyright material and is for
> the use only of the intended recipient. If you receive the Communication in
> error, please notify the sender immediately by return e-mail, delete the
> Communication and the return e-mail, and do not read, copy, retransmit or
> otherwise deal with it. Any views expressed in the Communication are those
> of the individual sender only, unless expressly stated to be those of
> Australia and New Zealand Banking Group Limited ABN 11 005 357 522, or any
> of its related entities including ANZ Bank New Zealand Limited (together
> "ANZ"). ANZ does not accept liability in connection with the integrity of
> or errors in the Communication, computer virus, data corruption,
> interference or delay arising from or in respect of the Communication.
>
>
Re: Kafka isync replica
Do you find any messages on broker 3 w.r.t. Topic: XYZ Partition 82? Also broker 3 is in isr for other partitions (at least 83,84) so I don't see any broker issue in this. On Thu, 14 Dec 2017 at 01:23 Tarun Garg wrote: > Hi, > > I have a Kafka cluster and it is running from a long time, just today I > realize that some of the topic partition are not in good state, there Isr > is been reduced to 2 from 3 even though the node of the lost replica is > working > > Like > Topic: XYZ Partition 82 Leader: 1 Replicas:1,2,3 Isr:1,2 <- problem > Topic: XYZ Partition 83 Leader: 2 Replicas:2,3,4 Isr:3,2,4 > Topic: XYZ Partition 84 Leader: 3 Replicas:3,4,5 Isr:3,4,5 > > How can I fix bring replica 3 in isr for partition 82. > > Any help is appreciated. > > Thanks > Tarun
Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig
Thanks Ewen, and apologies as I missed this too. I'll start vote for this soon and proceed for the next steps. On Fri, 5 Jan 2018 at 09:03 Ewen Cheslack-Postava wrote: > Sorry I lost track of this thread. If things are in good shape we should > probably vote on this and get the deprecation commit through. It seems like > a good idea as this has been confusing to users from day one. > > -Ewen > > On Wed, Aug 9, 2017 at 5:18 AM, UMESH CHAUDHARY > wrote: > >> Thanks Ewen, >> I just edited the KIP to reflect the changes. >> >> Regards, >> Umesh >> >> On Wed, 9 Aug 2017 at 11:00 Ewen Cheslack-Postava >> wrote: >> >>> Great, looking good. I'd probably be a bit more concrete about the >>> Proposed Changes (e.g., "will log an warning if the config is specified" >>> and "since the JsonConverter is the default, the configs will be removed >>> immediately from the example worker configuration files"). >>> >>> Other than that this LGTM and I'll be happy to get rid of those settings! >>> >>> -Ewen >>> >>> On Tue, Aug 8, 2017 at 2:54 AM, UMESH CHAUDHARY >>> wrote: >>> >>>> Hi Ewen, >>>> Sorry, I am bit late in responding this. >>>> >>>> Thanks for your inputs and I've updated the KIP by adding more details >>>> to it. >>>> >>>> Regards, >>>> Umesh >>>> >>>> On Mon, 31 Jul 2017 at 21:51 Ewen Cheslack-Postava >>>> wrote: >>>> >>>>> On Sun, Jul 30, 2017 at 10:21 PM, UMESH CHAUDHARY >>>> > wrote: >>>>> >>>>>> Hi Ewen, >>>>>> Thanks for your comments. >>>>>> >>>>>> 1) Yes, there are some test and java classes which refer these >>>>>> configs, so I will include them as well in "public interface" section of >>>>>> KIP. What should be our approach to deal with the classes and tests which >>>>>> use these configs: we need to change them to use JsonConverter when >>>>>> we plan for removal of these configs right? >>>>>> >>>>> >>>>> I actually meant the references in >>>>> config/connect-standalone.properties and >>>>> config/connect-distributed.properties >>>>> >>>>> >>>>>> 2) I believe we can target the deprecation in 1.0.0 release as it is >>>>>> planned in October 2017 and then removal in next major release. Let >>>>>> me know your thoughts as we don't have any information for next major >>>>>> release (next to 1.0.0) yet. >>>>>> >>>>> >>>>> That sounds fine. Tough to say at this point what our approach to >>>>> major version bumps will be since the approach to version numbering is >>>>> changing a bit. >>>>> >>>>> >>>>>> 3) Thats a good point and mentioned JIRA can help us to validate the >>>>>> usage of any other converters. I will list this down in the KIP. >>>>>> >>>>>> Let me know if you have some additional thoughts on this. >>>>>> >>>>>> Regards, >>>>>> Umesh >>>>>> >>>>>> >>>>>> >>>>>> On Wed, 26 Jul 2017 at 09:27 Ewen Cheslack-Postava >>>>>> wrote: >>>>>> >>>>>>> Umesh, >>>>>>> >>>>>>> Thanks for the KIP. Straightforward and I think it's a good change. >>>>>>> Unfortunately it is hard to tell how many people it would affect >>>>>>> since we >>>>>>> can't tell how many people have adjusted that config, but I think >>>>>>> this is >>>>>>> the right thing to do long term. >>>>>>> >>>>>>> A couple of quick things that might be helpful to refine: >>>>>>> >>>>>>> * Note that there are also some references in the example configs >>>>>>> that we >>>>>>> should remove. >>>>>>> * It's nice to be explicit about when the removal is planned. This >>>>>>> lets us >>>>>>> set expectations with users for timeframe (especially now that we >>>>>>> have time &g
Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram
Congratulations Rajini ! On Fri, 19 Jan 2018 at 07:11 Jun Rao wrote: > Congratulations, Rajini ! > > Jun > > On Wed, Jan 17, 2018 at 10:48 AM, Gwen Shapira wrote: > > > Dear Kafka Developers, Users and Fans, > > > > Rajini Sivaram became a committer in April 2017. Since then, she > remained > > active in the community and contributed major patches, reviews and KIP > > discussions. I am glad to announce that Rajini is now a member of the > > Apache Kafka PMC. > > > > Congratulations, Rajini and looking forward to your future contributions. > > > > Gwen, on behalf of Apache Kafka PMC > > >
Re: Setting topic's offset from the shell
You might want to set group.id config in kafka-console-consumer (or in any other consumer) to the value which you haven't used before. This will replay all available messages in the topic from start if you use --from-beginning in console consumer. On Wed, 28 Feb 2018 at 14:19 Zoran wrote: > Hi, > > > If I have a topic that has been fully read by consumers, how to set the > offset from the shell to some previous value in order to reread again > several messages? > > > Regards. > >
