[jira] [Commented] (KAFKA-826) Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x
[ https://issues.apache.org/jira/browse/KAFKA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13622062#comment-13622062 ] Maxime Brugidou commented on KAFKA-826: --- Yes there has been some work and it breaks when I try to use 3.0.0-BETA1 from 0.8 branch. We need to have a MetricsRegistry (static or injected) and use the new register() method to add gauges/metrics. Also the core namespace has been removed > Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x > - > > Key: KAFKA-826 > URL: https://issues.apache.org/jira/browse/KAFKA-826 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Neha Narkhede >Assignee: Dragos Manolescu >Priority: Blocker > Labels: build, kafka-0.8, metrics > Attachments: kafka-fix-for-826.patch, kafka-fix-for-826-take2.patch > > > In order to mavenize Kafka 0.8, we have to depend on metrics 2.2.0 since > metrics 3.x is a huge change as well as not an officially supported release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (KAFKA-826) Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x
[ https://issues.apache.org/jira/browse/KAFKA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13622062#comment-13622062 ] Maxime Brugidou edited comment on KAFKA-826 at 4/4/13 11:48 AM: Yes there has been some work and it breaks when I try to use 3.0.0-BETA1 from 0.8 branch. We need to have a MetricsRegistry (static or injected) and use the new register() method to add gauges/metrics. Also the core namespace has been removed Anyone working on a patch to get the BETA1 ? (and remove the core/lib/metrics*.jar files? ) By the way I don't think we use metrics-annotation, just metrics-core right? was (Author: brugidou): Yes there has been some work and it breaks when I try to use 3.0.0-BETA1 from 0.8 branch. We need to have a MetricsRegistry (static or injected) and use the new register() method to add gauges/metrics. Also the core namespace has been removed > Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x > - > > Key: KAFKA-826 > URL: https://issues.apache.org/jira/browse/KAFKA-826 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Neha Narkhede >Assignee: Dragos Manolescu >Priority: Blocker > Labels: build, kafka-0.8, metrics > Attachments: kafka-fix-for-826.patch, kafka-fix-for-826-take2.patch > > > In order to mavenize Kafka 0.8, we have to depend on metrics 2.2.0 since > metrics 3.x is a huge change as well as not an officially supported release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-845) Re-implement shallow iteration on 0.8.1
Maxime Brugidou created KAFKA-845: - Summary: Re-implement shallow iteration on 0.8.1 Key: KAFKA-845 URL: https://issues.apache.org/jira/browse/KAFKA-845 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.1 Reporter: Maxime Brugidou After KAFKA-732 we decided to not support the shallow iteration feature to speed up the 0.8 release. This severely impacts the performance for heavy mirroring between clusters. The MirrorMaker needs to decompress/recompress the data which actually triple the CPU compression cost in the data flow (initial compression + 2x compression for mirroring). It would be great to discuss a solution to re-implement the feature correctly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-743) PreferredReplicaLeaderElectionCommand has command line error
[ https://issues.apache.org/jira/browse/KAFKA-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566574#comment-13566574 ] Maxime Brugidou commented on KAFKA-743: --- Same thing for CheckReassignmentStatus and ReassignPartitionsCommand > PreferredReplicaLeaderElectionCommand has command line error > > > Key: KAFKA-743 > URL: https://issues.apache.org/jira/browse/KAFKA-743 > Project: Kafka > Issue Type: Bug > Components: tools >Affects Versions: 0.8 >Reporter: Jun Rao >Priority: Blocker > Attachments: kafka-743.patch > > > bin/kafka-preferred-replica-election.sh > Exception in thread "main" joptsimple.IllegalOptionSpecificationException: ' > ' is not a legal option character > at > joptsimple.ParserRules.ensureLegalOptionCharacter(ParserRules.java:81) > at joptsimple.ParserRules.ensureLegalOption(ParserRules.java:71) > at joptsimple.ParserRules.ensureLegalOptions(ParserRules.java:76) > at joptsimple.OptionParser.acceptsAll(OptionParser.java:309) > at joptsimple.OptionParser.accepts(OptionParser.java:271) > at > kafka.admin.PreferredReplicaLeaderElectionCommand$.main(PreferredReplicaLeaderElectionCommand.scala:29) > at > kafka.admin.PreferredReplicaLeaderElectionCommand.main(PreferredReplicaLeaderElectionCommand.scala) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-732) MirrorMaker with shallow.iterator.enable=true produces unreadble messages
Maxime Brugidou created KAFKA-732: - Summary: MirrorMaker with shallow.iterator.enable=true produces unreadble messages Key: KAFKA-732 URL: https://issues.apache.org/jira/browse/KAFKA-732 Project: Kafka Issue Type: Bug Components: core, producer Affects Versions: 0.8 Reporter: Maxime Brugidou Assignee: Jun Rao Priority: Blocker Trying to use MirrorMaker between two 0.8 clusters When using shallow.iterator.enable=true on the consumer side, the performance gain is big (when incoming messages are compressed) and the producer does not complain but write the messages uncompressed without the compression flag. If you try: - enable compression on the producer, it obviously makes things worse since the data get double-compressed (the wiki warns about this) - disable compression and the compressed messages are written in bulk in an uncompressed message, thus making it unreadable. If I follow correctly the current state of code from MirrorMaker to the produce request, there is no way for the producer to know whether the message is deep or not. So I wonder how it worked on 0.7? Here is the code as i read it (correct me if i'm wrong): 1. MirrorMakerThread.run(): create KeyedMessage[Array[Byte],Array[Byte]](topic, message) 2. Producer.send() -> DefaultEventHandler.handle() 3. DefaultEventHandler.serialize(): use DefaultEncoder for the message (does nothing) 4. DefaultEventHandler.dispatchSerializedData(): 4.1 DefaultEventHandler.partitionAndCollate(): group messages by broker/partition/topic 4.2 DefaultEventHandler.dispatchSerializeData(): cycle through each broker 4.3 DefaultEventHandler.groupMessagesToSet(): Create a ByteBufferMessageSet for each partition/topic grouping all the messages together, and compressing them if needed 4.4 DefaultEventHandler.send(): send the ByteBufferMessageSets for this broker in one ProduceRequest The gist is that in DEH.groupMessagesToSet(), you don't know wether the raw message in KeyedMessage.message is shallow or not. So I think I missed something... Also it doesn't seem possible to send batch of deep messages in one ProduceRequest. I would love to provide a patch (or if you tell me that i'm doing it wrong, it's even better), since I can easily test it on my test clusters but I will need guidance here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-139) cross-compile multiple Scala versions and upgrade to SBT 0.12.1
[ https://issues.apache.org/jira/browse/KAFKA-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560520#comment-13560520 ] Maxime Brugidou commented on KAFKA-139: --- I checked out 0.8 this morning and this breaks the Mavenization from KAFKA-133 The dependencies for com.yammer.metrics is nested in the resulting pom.xml, which is not valid. > cross-compile multiple Scala versions and upgrade to SBT 0.12.1 > --- > > Key: KAFKA-139 > URL: https://issues.apache.org/jira/browse/KAFKA-139 > Project: Kafka > Issue Type: Improvement > Components: packaging >Affects Versions: 0.8 >Reporter: Chris Burroughs > Labels: build > Fix For: 0.8 > > Attachments: kafka-sbt0-11-3-0.8.patch, kafka-sbt0-11-3-0.8-v2.patch, > kafka-sbt0-11-3-0.8-v3.patch, kafka-sbt0-11-3-0.8-v4.patch, > kafka-sbt0-11-3-0.8-v5-smeder.patch, kafka-sbt0-11-3-0.8-v6-smeder.patch, > kafka-sbt0-11-3.patch > > > Since scala does not maintain binary compatibly between versions, > organizations tend to have to move all of there code at the same time. It > would thus be very helpful if we could cross build multiple scala versions. > http://code.google.com/p/simple-build-tool/wiki/CrossBuild > Unclear if this would require KAFKA-134 or just work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-693: -- Attachment: KAFKA-693-v3.patch Added v3 with your remarks > Consumer rebalance fails if no leader available for a partition and stops all > fetchers > -- > > Key: KAFKA-693 > URL: https://issues.apache.org/jira/browse/KAFKA-693 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou >Assignee: Maxime Brugidou > Attachments: KAFKA-693.patch, KAFKA-693-v2.patch, KAFKA-693-v3.patch, > mirror_debug.log, mirror.log > > > I am currently experiencing this with the MirrorMaker but I assume it happens > for any rebalance. The symptoms are: > I have replication factor of 1 > 1. If i start the MirrorMaker (bin/kafka-run-class.sh kafka.tools.MirrorMaker > --consumer.config mirror-consumer.properties --producer.config > mirror-producer.properties --blacklist 'xdummyx' --num.streams=1 > --num.producers=1) with a broker down > 1.1 I set the refresh.leader.backoff.ms to 60 (10min) so that the > ConsumerFetcherManager doesn't retry to often to get the unavailable > partitions > 1.2 The rebalance starts at the init step and fails: Exception in thread > "main" kafka.common.ConsumerRebalanceFailedException: > KafkaMirror_mirror-01-1357893495345-fac86b15 can't rebalance after 4 retries > 1.3 After the exception, everything stops (fetchers and queues) > 1.4 I attached the full logs (info & debug) for this case > 2. If i start the MirrorMaker with all the brokers up and then kill a broker > 2.1 The first rebalance is successful > 2.2 The consumer will handle correctly the broker down and stop the > associated ConsumerFetcherThread > 2.3 The refresh.leader.backoff.ms to 60 works correctly > 2.4 If something triggers a rebalance (new topic, partition reassignment...), > then we go back to 1., the rebalance fails and stops everything. > I think the desired behavior is to consumer whatever is available, and try > later at some intervals. I would be glad to help on that issue although the > Consumer code seems a little tough to get on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554852#comment-13554852 ] Maxime Brugidou commented on KAFKA-693: --- 10. Created PartitionTopicInfo.InvalidOffset 11. In ConsumerFetcherManager.doWork(), I believe that addFetcher() is called before the partition is removed from noLeaderPartitionSet, if an exception is caught the partition will still be in the noLeaderPartitionSet, so I didn't change anything 12. done 13. done > Consumer rebalance fails if no leader available for a partition and stops all > fetchers > -- > > Key: KAFKA-693 > URL: https://issues.apache.org/jira/browse/KAFKA-693 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou >Assignee: Maxime Brugidou > Attachments: KAFKA-693.patch, KAFKA-693-v2.patch, mirror_debug.log, > mirror.log > > > I am currently experiencing this with the MirrorMaker but I assume it happens > for any rebalance. The symptoms are: > I have replication factor of 1 > 1. If i start the MirrorMaker (bin/kafka-run-class.sh kafka.tools.MirrorMaker > --consumer.config mirror-consumer.properties --producer.config > mirror-producer.properties --blacklist 'xdummyx' --num.streams=1 > --num.producers=1) with a broker down > 1.1 I set the refresh.leader.backoff.ms to 60 (10min) so that the > ConsumerFetcherManager doesn't retry to often to get the unavailable > partitions > 1.2 The rebalance starts at the init step and fails: Exception in thread > "main" kafka.common.ConsumerRebalanceFailedException: > KafkaMirror_mirror-01-1357893495345-fac86b15 can't rebalance after 4 retries > 1.3 After the exception, everything stops (fetchers and queues) > 1.4 I attached the full logs (info & debug) for this case > 2. If i start the MirrorMaker with all the brokers up and then kill a broker > 2.1 The first rebalance is successful > 2.2 The consumer will handle correctly the broker down and stop the > associated ConsumerFetcherThread > 2.3 The refresh.leader.backoff.ms to 60 works correctly > 2.4 If something triggers a rebalance (new topic, partition reassignment...), > then we go back to 1., the rebalance fails and stops everything. > I think the desired behavior is to consumer whatever is available, and try > later at some intervals. I would be glad to help on that issue although the > Consumer code seems a little tough to get on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-693: -- Attachment: KAFKA-693-v2.patch > Consumer rebalance fails if no leader available for a partition and stops all > fetchers > -- > > Key: KAFKA-693 > URL: https://issues.apache.org/jira/browse/KAFKA-693 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou >Assignee: Maxime Brugidou > Attachments: KAFKA-693.patch, KAFKA-693-v2.patch, mirror_debug.log, > mirror.log > > > I am currently experiencing this with the MirrorMaker but I assume it happens > for any rebalance. The symptoms are: > I have replication factor of 1 > 1. If i start the MirrorMaker (bin/kafka-run-class.sh kafka.tools.MirrorMaker > --consumer.config mirror-consumer.properties --producer.config > mirror-producer.properties --blacklist 'xdummyx' --num.streams=1 > --num.producers=1) with a broker down > 1.1 I set the refresh.leader.backoff.ms to 60 (10min) so that the > ConsumerFetcherManager doesn't retry to often to get the unavailable > partitions > 1.2 The rebalance starts at the init step and fails: Exception in thread > "main" kafka.common.ConsumerRebalanceFailedException: > KafkaMirror_mirror-01-1357893495345-fac86b15 can't rebalance after 4 retries > 1.3 After the exception, everything stops (fetchers and queues) > 1.4 I attached the full logs (info & debug) for this case > 2. If i start the MirrorMaker with all the brokers up and then kill a broker > 2.1 The first rebalance is successful > 2.2 The consumer will handle correctly the broker down and stop the > associated ConsumerFetcherThread > 2.3 The refresh.leader.backoff.ms to 60 works correctly > 2.4 If something triggers a rebalance (new topic, partition reassignment...), > then we go back to 1., the rebalance fails and stops everything. > I think the desired behavior is to consumer whatever is available, and try > later at some intervals. I would be glad to help on that issue although the > Consumer code seems a little tough to get on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554458#comment-13554458 ] Maxime Brugidou commented on KAFKA-347: --- Agreed. Except you also have to think about people not using keyed partitioning. It would be very convenient to add (maybe not remove) partitions as your topic gets larger and you increase your cluster size. And that could be done online since the number of partitions in this case is only used for scaling and distribution. (and I think the auto partition balancing is off-topic and I should probably open another JIRA) > change number of partitions of a topic online > - > > Key: KAFKA-347 > URL: https://issues.apache.org/jira/browse/KAFKA-347 > Project: Kafka > Issue Type: Sub-task > Components: core >Affects Versions: 0.8 >Reporter: Jun Rao > Labels: features > > We will need an admin tool to change the number of partitions of a topic > online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-347) change number of partitions of a topic online
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13553873#comment-13553873 ] Maxime Brugidou commented on KAFKA-347: --- It's sort of out of scope but I think that using some sort of consistent hashing for partition assignment could really help here. The process could be that if you activate "automatic partition assignment", you don't need to send the ReassignPartition admin command to manage partitions but you over-partition and spread partitions for each topics over brokers using consistent hashing. This would: - Minimize the number of partitions moved when you add/remove brokers - Partitions would stick to a broker when some partitions are added/deleted - Could be run on a regular basis by brokers or other means, and would be perfectly consistent This would greatly improve operations > change number of partitions of a topic online > - > > Key: KAFKA-347 > URL: https://issues.apache.org/jira/browse/KAFKA-347 > Project: Kafka > Issue Type: Sub-task > Components: core >Affects Versions: 0.8 >Reporter: Jun Rao > Labels: features > > We will need an admin tool to change the number of partitions of a topic > online. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-155) Support graceful Decommissioning of Broker
[ https://issues.apache.org/jira/browse/KAFKA-155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou resolved KAFKA-155. --- Resolution: Fixed Fix Version/s: 0.8 > Support graceful Decommissioning of Broker > -- > > Key: KAFKA-155 > URL: https://issues.apache.org/jira/browse/KAFKA-155 > Project: Kafka > Issue Type: Improvement >Reporter: Sharad Agarwal > Fix For: 0.8 > > > There should be a graceful way of decommissioning the broker so that there is > absolutely 0 data loss. Decommissioning is not necessarily related to > replication (Kafka-50). > There should be a way to get the broker out of the cluster only from the > produce side. Consumers should be able to continue keep pulling data. When > the administrator is sure that all data has been consumed by consumers, > broker node can be removed permanently. > Same would be useful for rolling upgrades without any message loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-155) Support graceful Decommissioning of Broker
[ https://issues.apache.org/jira/browse/KAFKA-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13553867#comment-13553867 ] Maxime Brugidou commented on KAFKA-155: --- I think this is done in 0.8 branch with the ShutdownBroker admin command (which goes through JMX command KafkController.shutdownBroker()) > Support graceful Decommissioning of Broker > -- > > Key: KAFKA-155 > URL: https://issues.apache.org/jira/browse/KAFKA-155 > Project: Kafka > Issue Type: Improvement >Reporter: Sharad Agarwal > > There should be a graceful way of decommissioning the broker so that there is > absolutely 0 data loss. Decommissioning is not necessarily related to > replication (Kafka-50). > There should be a way to get the broker out of the cluster only from the > produce side. Consumers should be able to continue keep pulling data. When > the administrator is sure that all data has been consumed by consumers, > broker node can be removed permanently. > Same would be useful for rolling upgrades without any message loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-693: -- Assignee: Maxime Brugidou > Consumer rebalance fails if no leader available for a partition and stops all > fetchers > -- > > Key: KAFKA-693 > URL: https://issues.apache.org/jira/browse/KAFKA-693 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou >Assignee: Maxime Brugidou > Attachments: KAFKA-693.patch, mirror_debug.log, mirror.log > > > I am currently experiencing this with the MirrorMaker but I assume it happens > for any rebalance. The symptoms are: > I have replication factor of 1 > 1. If i start the MirrorMaker (bin/kafka-run-class.sh kafka.tools.MirrorMaker > --consumer.config mirror-consumer.properties --producer.config > mirror-producer.properties --blacklist 'xdummyx' --num.streams=1 > --num.producers=1) with a broker down > 1.1 I set the refresh.leader.backoff.ms to 60 (10min) so that the > ConsumerFetcherManager doesn't retry to often to get the unavailable > partitions > 1.2 The rebalance starts at the init step and fails: Exception in thread > "main" kafka.common.ConsumerRebalanceFailedException: > KafkaMirror_mirror-01-1357893495345-fac86b15 can't rebalance after 4 retries > 1.3 After the exception, everything stops (fetchers and queues) > 1.4 I attached the full logs (info & debug) for this case > 2. If i start the MirrorMaker with all the brokers up and then kill a broker > 2.1 The first rebalance is successful > 2.2 The consumer will handle correctly the broker down and stop the > associated ConsumerFetcherThread > 2.3 The refresh.leader.backoff.ms to 60 works correctly > 2.4 If something triggers a rebalance (new topic, partition reassignment...), > then we go back to 1., the rebalance fails and stops everything. > I think the desired behavior is to consumer whatever is available, and try > later at some intervals. I would be glad to help on that issue although the > Consumer code seems a little tough to get on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-693: -- Status: Patch Available (was: Open) > Consumer rebalance fails if no leader available for a partition and stops all > fetchers > -- > > Key: KAFKA-693 > URL: https://issues.apache.org/jira/browse/KAFKA-693 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Attachments: KAFKA-693.patch, mirror_debug.log, mirror.log > > > I am currently experiencing this with the MirrorMaker but I assume it happens > for any rebalance. The symptoms are: > I have replication factor of 1 > 1. If i start the MirrorMaker (bin/kafka-run-class.sh kafka.tools.MirrorMaker > --consumer.config mirror-consumer.properties --producer.config > mirror-producer.properties --blacklist 'xdummyx' --num.streams=1 > --num.producers=1) with a broker down > 1.1 I set the refresh.leader.backoff.ms to 60 (10min) so that the > ConsumerFetcherManager doesn't retry to often to get the unavailable > partitions > 1.2 The rebalance starts at the init step and fails: Exception in thread > "main" kafka.common.ConsumerRebalanceFailedException: > KafkaMirror_mirror-01-1357893495345-fac86b15 can't rebalance after 4 retries > 1.3 After the exception, everything stops (fetchers and queues) > 1.4 I attached the full logs (info & debug) for this case > 2. If i start the MirrorMaker with all the brokers up and then kill a broker > 2.1 The first rebalance is successful > 2.2 The consumer will handle correctly the broker down and stop the > associated ConsumerFetcherThread > 2.3 The refresh.leader.backoff.ms to 60 works correctly > 2.4 If something triggers a rebalance (new topic, partition reassignment...), > then we go back to 1., the rebalance fails and stops everything. > I think the desired behavior is to consumer whatever is available, and try > later at some intervals. I would be glad to help on that issue although the > Consumer code seems a little tough to get on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13553751#comment-13553751 ] Maxime Brugidou commented on KAFKA-693: --- Here is a patch: 1. AbstractFetchThread.addPartition(): call handleOffsetOutOfRange if initialOffset < 0 2. I didnt touch ConsumerFetcherManager.doWork() since addFetcher() is called for partitions with leaders only (which is why 3 is unnecessary). 3. ConsumerFetcherThrad.handleOffsetOutOfRange: check partitionErrorAndOffset.error and throw appropriate exception (which should have been done anyway, I don't think this is necessary for the patch) 3.1 Note: this should probably be done in the ReplicaFetcherThread too? 4. ZookeeperConsumerConnector.ZkRebalanceListener: Do not compute leaderIdForPartitionMap in rebalance() and set PartitionTopicInfo offsets to -1 if not in Zk (new consumer) 5. PartitionTopicInfo: removed brokerId 6. Fixed tests for compilation (I am having a hard time running tests since ./sbt test does not seem to work for me very well) 7. Should we increase the default refresh.leader.backoff.ms ? It's tradeoff between being able to pick fast a new leader to consume (useful when replication is on) and not flooding the broker when there is no leader (or replication is off). 200ms is very short, but something hybrid like "try 5 times at 200ms backoff, then every 5min" would get all use cases. I am running this on test clusters with a mirrormaker andthe error that I had in my initial test case (in the description) does not occur anymore. > Consumer rebalance fails if no leader available for a partition and stops all > fetchers > -- > > Key: KAFKA-693 > URL: https://issues.apache.org/jira/browse/KAFKA-693 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Attachments: KAFKA-693.patch, mirror_debug.log, mirror.log > > > I am currently experiencing this with the MirrorMaker but I assume it happens > for any rebalance. The symptoms are: > I have replication factor of 1 > 1. If i start the MirrorMaker (bin/kafka-run-class.sh kafka.tools.MirrorMaker > --consumer.config mirror-consumer.properties --producer.config > mirror-producer.properties --blacklist 'xdummyx' --num.streams=1 > --num.producers=1) with a broker down > 1.1 I set the refresh.leader.backoff.ms to 60 (10min) so that the > ConsumerFetcherManager doesn't retry to often to get the unavailable > partitions > 1.2 The rebalance starts at the init step and fails: Exception in thread > "main" kafka.common.ConsumerRebalanceFailedException: > KafkaMirror_mirror-01-1357893495345-fac86b15 can't rebalance after 4 retries > 1.3 After the exception, everything stops (fetchers and queues) > 1.4 I attached the full logs (info & debug) for this case > 2. If i start the MirrorMaker with all the brokers up and then kill a broker > 2.1 The first rebalance is successful > 2.2 The consumer will handle correctly the broker down and stop the > associated ConsumerFetcherThread > 2.3 The refresh.leader.backoff.ms to 60 works correctly > 2.4 If something triggers a rebalance (new topic, partition reassignment...), > then we go back to 1., the rebalance fails and stops everything. > I think the desired behavior is to consumer whatever is available, and try > later at some intervals. I would be glad to help on that issue although the > Consumer code seems a little tough to get on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-693: -- Attachment: KAFKA-693.patch > Consumer rebalance fails if no leader available for a partition and stops all > fetchers > -- > > Key: KAFKA-693 > URL: https://issues.apache.org/jira/browse/KAFKA-693 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Attachments: KAFKA-693.patch, mirror_debug.log, mirror.log > > > I am currently experiencing this with the MirrorMaker but I assume it happens > for any rebalance. The symptoms are: > I have replication factor of 1 > 1. If i start the MirrorMaker (bin/kafka-run-class.sh kafka.tools.MirrorMaker > --consumer.config mirror-consumer.properties --producer.config > mirror-producer.properties --blacklist 'xdummyx' --num.streams=1 > --num.producers=1) with a broker down > 1.1 I set the refresh.leader.backoff.ms to 60 (10min) so that the > ConsumerFetcherManager doesn't retry to often to get the unavailable > partitions > 1.2 The rebalance starts at the init step and fails: Exception in thread > "main" kafka.common.ConsumerRebalanceFailedException: > KafkaMirror_mirror-01-1357893495345-fac86b15 can't rebalance after 4 retries > 1.3 After the exception, everything stops (fetchers and queues) > 1.4 I attached the full logs (info & debug) for this case > 2. If i start the MirrorMaker with all the brokers up and then kill a broker > 2.1 The first rebalance is successful > 2.2 The consumer will handle correctly the broker down and stop the > associated ConsumerFetcherThread > 2.3 The refresh.leader.backoff.ms to 60 works correctly > 2.4 If something triggers a rebalance (new topic, partition reassignment...), > then we go back to 1., the rebalance fails and stops everything. > I think the desired behavior is to consumer whatever is available, and try > later at some intervals. I would be glad to help on that issue although the > Consumer code seems a little tough to get on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13552951#comment-13552951 ] Maxime Brugidou commented on KAFKA-693: --- Looks good, It should work but I still have a pain point about PartitionTopicInfo that uses AtomicLong to track consume/fetch offsets. Using Option[AtomicLong] looks strange, because I have to change the 2 counters and make them variables... And it's probably not thread safe at all so I would need some sort of lock to "initialize" the counters. > Consumer rebalance fails if no leader available for a partition and stops all > fetchers > -- > > Key: KAFKA-693 > URL: https://issues.apache.org/jira/browse/KAFKA-693 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Attachments: mirror_debug.log, mirror.log > > > I am currently experiencing this with the MirrorMaker but I assume it happens > for any rebalance. The symptoms are: > I have replication factor of 1 > 1. If i start the MirrorMaker (bin/kafka-run-class.sh kafka.tools.MirrorMaker > --consumer.config mirror-consumer.properties --producer.config > mirror-producer.properties --blacklist 'xdummyx' --num.streams=1 > --num.producers=1) with a broker down > 1.1 I set the refresh.leader.backoff.ms to 60 (10min) so that the > ConsumerFetcherManager doesn't retry to often to get the unavailable > partitions > 1.2 The rebalance starts at the init step and fails: Exception in thread > "main" kafka.common.ConsumerRebalanceFailedException: > KafkaMirror_mirror-01-1357893495345-fac86b15 can't rebalance after 4 retries > 1.3 After the exception, everything stops (fetchers and queues) > 1.4 I attached the full logs (info & debug) for this case > 2. If i start the MirrorMaker with all the brokers up and then kill a broker > 2.1 The first rebalance is successful > 2.2 The consumer will handle correctly the broker down and stop the > associated ConsumerFetcherThread > 2.3 The refresh.leader.backoff.ms to 60 works correctly > 2.4 If something triggers a rebalance (new topic, partition reassignment...), > then we go back to 1., the rebalance fails and stops everything. > I think the desired behavior is to consumer whatever is available, and try > later at some intervals. I would be glad to help on that issue although the > Consumer code seems a little tough to get on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-701) ConsoleProducer does not exit correctly and fix some config properties following KAFKA-648
Maxime Brugidou created KAFKA-701: - Summary: ConsoleProducer does not exit correctly and fix some config properties following KAFKA-648 Key: KAFKA-701 URL: https://issues.apache.org/jira/browse/KAFKA-701 Project: Kafka Issue Type: Bug Components: config, core Affects Versions: 0.8 Reporter: Maxime Brugidou Priority: Minor Attachments: KAFKA-701.patch Just added a proper try/catch around the ConsoleProducer so that when an exception is thrown, the system exits (with error code 1) In addition, KAFKA-648 broker some configs like request.enqueue.timeout.ms and zk.connection.timeout.ms that I fixed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-701) ConsoleProducer does not exit correctly and fix some config properties following KAFKA-648
[ https://issues.apache.org/jira/browse/KAFKA-701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-701: -- Status: Patch Available (was: Open) > ConsoleProducer does not exit correctly and fix some config properties > following KAFKA-648 > -- > > Key: KAFKA-701 > URL: https://issues.apache.org/jira/browse/KAFKA-701 > Project: Kafka > Issue Type: Bug > Components: config, core >Affects Versions: 0.8 >Reporter: Maxime Brugidou >Priority: Minor > Attachments: KAFKA-701.patch > > > Just added a proper try/catch around the ConsoleProducer so that when an > exception is thrown, the system exits (with error code 1) > In addition, KAFKA-648 broker some configs like request.enqueue.timeout.ms > and zk.connection.timeout.ms that I fixed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-701) ConsoleProducer does not exit correctly and fix some config properties following KAFKA-648
[ https://issues.apache.org/jira/browse/KAFKA-701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-701: -- Attachment: KAFKA-701.patch > ConsoleProducer does not exit correctly and fix some config properties > following KAFKA-648 > -- > > Key: KAFKA-701 > URL: https://issues.apache.org/jira/browse/KAFKA-701 > Project: Kafka > Issue Type: Bug > Components: config, core >Affects Versions: 0.8 >Reporter: Maxime Brugidou >Priority: Minor > Attachments: KAFKA-701.patch > > > Just added a proper try/catch around the ConsoleProducer so that when an > exception is thrown, the system exits (with error code 1) > In addition, KAFKA-648 broker some configs like request.enqueue.timeout.ms > and zk.connection.timeout.ms that I fixed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13552781#comment-13552781 ] Maxime Brugidou commented on KAFKA-693: --- I looked up the code in details and I am stuck because during the rebalance() operation, the ZookeeperConsumerConnector's topicRegistry is updated with some PartitionTopicInfo that needs to store the consumerOffset and fetchOffset. During addPartitionTopicInfo(), the consumer offset is read from Zookeeper, however it needs to be initialized if no offsetString is available on Zookeeper (first time starting a consumer), and we need to access the broker/leader to get the start offset (using SimpleConsumer.earliestOrLatestOffset() in addPartitionTopicInfo()). I digged a bit and we could probably initialize the offset later in the ConsumerFetcherManager? I could help with a patch if i get general directions because i'm not 100% familiar with the codebase yet. > Consumer rebalance fails if no leader available for a partition and stops all > fetchers > -- > > Key: KAFKA-693 > URL: https://issues.apache.org/jira/browse/KAFKA-693 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Attachments: mirror_debug.log, mirror.log > > > I am currently experiencing this with the MirrorMaker but I assume it happens > for any rebalance. The symptoms are: > I have replication factor of 1 > 1. If i start the MirrorMaker (bin/kafka-run-class.sh kafka.tools.MirrorMaker > --consumer.config mirror-consumer.properties --producer.config > mirror-producer.properties --blacklist 'xdummyx' --num.streams=1 > --num.producers=1) with a broker down > 1.1 I set the refresh.leader.backoff.ms to 60 (10min) so that the > ConsumerFetcherManager doesn't retry to often to get the unavailable > partitions > 1.2 The rebalance starts at the init step and fails: Exception in thread > "main" kafka.common.ConsumerRebalanceFailedException: > KafkaMirror_mirror-01-1357893495345-fac86b15 can't rebalance after 4 retries > 1.3 After the exception, everything stops (fetchers and queues) > 1.4 I attached the full logs (info & debug) for this case > 2. If i start the MirrorMaker with all the brokers up and then kill a broker > 2.1 The first rebalance is successful > 2.2 The consumer will handle correctly the broker down and stop the > associated ConsumerFetcherThread > 2.3 The refresh.leader.backoff.ms to 60 works correctly > 2.4 If something triggers a rebalance (new topic, partition reassignment...), > then we go back to 1., the rebalance fails and stops everything. > I think the desired behavior is to consumer whatever is available, and try > later at some intervals. I would be glad to help on that issue although the > Consumer code seems a little tough to get on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-691) Fault tolerance broken with replication factor 1
[ https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13552759#comment-13552759 ] Maxime Brugidou commented on KAFKA-691: --- So I wait for your feedback first, but I guess that increasing the time out is good enough, although it's 1500ms by default which is very short. > Fault tolerance broken with replication factor 1 > > > Key: KAFKA-691 > URL: https://issues.apache.org/jira/browse/KAFKA-691 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Jay Kreps >Assignee: Maxime Brugidou > Fix For: 0.8 > > Attachments: KAFKA-691-v1.patch, KAFKA-691-v2.patch > > > In 0.7 if a partition was down we would just send the message elsewhere. This > meant that the partitioning was really more of a "stickiness" then a hard > guarantee. This made it impossible to depend on it for partitioned, stateful > processing. > In 0.8 when running with replication this should not be a problem generally > as the partitions are now highly available and fail over to other replicas. > However in the case of replication factor = 1 no longer really works for most > cases as now a dead broker will give errors for that broker. > I am not sure of the best fix. Intuitively I think this is something that > should be handled by the Partitioner interface. However currently the > partitioner has no knowledge of which nodes are available. So you could use a > random partitioner, but that would keep going back to the down node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-691) Fault tolerance broken with replication factor 1
[ https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13551978#comment-13551978 ] Maxime Brugidou commented on KAFKA-691: --- Should i make another patch? I'll try on Monday. 1. It would probably require yet another config variable like "producer.metadata.request.batch.size" or something like that. 2. Should it be batched for every updateInfo() or just during the metadata refresh? It could help if we do the former because failing messages from many different topics could probably never go through if the metadata request timeouts. 3. Isn'it getting a little convoluted? Maybe i am missing something but the producer side is getting trickier. 4. Please note that I also opened KAFKA-693 about the consumer side. And I'd love to submit a patch but the rebalance logic seems complex so I'd prefer to have some insights first before going in the wrong direction. > Fault tolerance broken with replication factor 1 > > > Key: KAFKA-691 > URL: https://issues.apache.org/jira/browse/KAFKA-691 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Jay Kreps >Assignee: Maxime Brugidou > Fix For: 0.8 > > Attachments: KAFKA-691-v1.patch, KAFKA-691-v2.patch > > > In 0.7 if a partition was down we would just send the message elsewhere. This > meant that the partitioning was really more of a "stickiness" then a hard > guarantee. This made it impossible to depend on it for partitioned, stateful > processing. > In 0.8 when running with replication this should not be a problem generally > as the partitions are now highly available and fail over to other replicas. > However in the case of replication factor = 1 no longer really works for most > cases as now a dead broker will give errors for that broker. > I am not sure of the best fix. Intuitively I think this is something that > should be handled by the Partitioner interface. However currently the > partitioner has no knowledge of which nodes are available. So you could use a > random partitioner, but that would keep going back to the down node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-693: -- Attachment: mirror_debug.log mirror.log > Consumer rebalance fails if no leader available for a partition and stops all > fetchers > -- > > Key: KAFKA-693 > URL: https://issues.apache.org/jira/browse/KAFKA-693 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Attachments: mirror_debug.log, mirror.log > > > I am currently experiencing this with the MirrorMaker but I assume it happens > for any rebalance. The symptoms are: > I have replication factor of 1 > 1. If i start the MirrorMaker (bin/kafka-run-class.sh kafka.tools.MirrorMaker > --consumer.config mirror-consumer.properties --producer.config > mirror-producer.properties --blacklist 'xdummyx' --num.streams=1 > --num.producers=1) with a broker down > 1.1 I set the refresh.leader.backoff.ms to 60 (10min) so that the > ConsumerFetcherManager doesn't retry to often to get the unavailable > partitions > 1.2 The rebalance starts at the init step and fails: Exception in thread > "main" kafka.common.ConsumerRebalanceFailedException: > KafkaMirror_mirror-01-1357893495345-fac86b15 can't rebalance after 4 retries > 1.3 After the exception, everything stops (fetchers and queues) > 1.4 I attached the full logs (info & debug) for this case > 2. If i start the MirrorMaker with all the brokers up and then kill a broker > 2.1 The first rebalance is successful > 2.2 The consumer will handle correctly the broker down and stop the > associated ConsumerFetcherThread > 2.3 The refresh.leader.backoff.ms to 60 works correctly > 2.4 If something triggers a rebalance (new topic, partition reassignment...), > then we go back to 1., the rebalance fails and stops everything. > I think the desired behavior is to consumer whatever is available, and try > later at some intervals. I would be glad to help on that issue although the > Consumer code seems a little tough to get on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers
Maxime Brugidou created KAFKA-693: - Summary: Consumer rebalance fails if no leader available for a partition and stops all fetchers Key: KAFKA-693 URL: https://issues.apache.org/jira/browse/KAFKA-693 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8 Reporter: Maxime Brugidou Attachments: mirror_debug.log, mirror.log I am currently experiencing this with the MirrorMaker but I assume it happens for any rebalance. The symptoms are: I have replication factor of 1 1. If i start the MirrorMaker (bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config mirror-consumer.properties --producer.config mirror-producer.properties --blacklist 'xdummyx' --num.streams=1 --num.producers=1) with a broker down 1.1 I set the refresh.leader.backoff.ms to 60 (10min) so that the ConsumerFetcherManager doesn't retry to often to get the unavailable partitions 1.2 The rebalance starts at the init step and fails: Exception in thread "main" kafka.common.ConsumerRebalanceFailedException: KafkaMirror_mirror-01-1357893495345-fac86b15 can't rebalance after 4 retries 1.3 After the exception, everything stops (fetchers and queues) 1.4 I attached the full logs (info & debug) for this case 2. If i start the MirrorMaker with all the brokers up and then kill a broker 2.1 The first rebalance is successful 2.2 The consumer will handle correctly the broker down and stop the associated ConsumerFetcherThread 2.3 The refresh.leader.backoff.ms to 60 works correctly 2.4 If something triggers a rebalance (new topic, partition reassignment...), then we go back to 1., the rebalance fails and stops everything. I think the desired behavior is to consumer whatever is available, and try later at some intervals. I would be glad to help on that issue although the Consumer code seems a little tough to get on. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-691) Fault tolerance broken with replication factor 1
[ https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13550338#comment-13550338 ] Maxime Brugidou commented on KAFKA-691: --- Thanks for committing the patch. 3.1 Are you sure that the rebalance doesn't require all partitions to have a leader? My experience earlier today was that the rebalance would fail and throw ConsumerRebalanceFailedException after having stopped all fetchers and cleared all queues. If you are sure then i'll try to reproduce the behavior I encountered, and maybe open a separate JIRA? > Fault tolerance broken with replication factor 1 > > > Key: KAFKA-691 > URL: https://issues.apache.org/jira/browse/KAFKA-691 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Jay Kreps >Assignee: Maxime Brugidou > Fix For: 0.8 > > Attachments: KAFKA-691-v1.patch, KAFKA-691-v2.patch > > > In 0.7 if a partition was down we would just send the message elsewhere. This > meant that the partitioning was really more of a "stickiness" then a hard > guarantee. This made it impossible to depend on it for partitioned, stateful > processing. > In 0.8 when running with replication this should not be a problem generally > as the partitions are now highly available and fail over to other replicas. > However in the case of replication factor = 1 no longer really works for most > cases as now a dead broker will give errors for that broker. > I am not sure of the best fix. Intuitively I think this is something that > should be handled by the Partitioner interface. However currently the > partitioner has no knowledge of which nodes are available. So you could use a > random partitioner, but that would keep going back to the down node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-691) Fault tolerance broken with replication factor 1
[ https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-691: -- Attachment: KAFKA-691-v2.patch Thanks for your feedback, I updated it (v2) according to your notes (1. and 2.). for 3. I believe you are right, except that: 3.1 It seems (correct me if i'm wrong) that a rebalance happen at the consumer initialization, so that means a consumer can't start if a broker is down 3.2 Can a rebalance be triggered when a partition is added or moved? Having a broker down shouldn't prevent me from reassigning partitions or adding partitions. > Fault tolerance broken with replication factor 1 > > > Key: KAFKA-691 > URL: https://issues.apache.org/jira/browse/KAFKA-691 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Jay Kreps > Attachments: KAFKA-691-v1.patch, KAFKA-691-v2.patch > > > In 0.7 if a partition was down we would just send the message elsewhere. This > meant that the partitioning was really more of a "stickiness" then a hard > guarantee. This made it impossible to depend on it for partitioned, stateful > processing. > In 0.8 when running with replication this should not be a problem generally > as the partitions are now highly available and fail over to other replicas. > However in the case of replication factor = 1 no longer really works for most > cases as now a dead broker will give errors for that broker. > I am not sure of the best fix. Intuitively I think this is something that > should be handled by the Partitioner interface. However currently the > partitioner has no knowledge of which nodes are available. So you could use a > random partitioner, but that would keep going back to the down node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-691) Fault tolerance broken with replication factor 1
[
https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Brugidou updated KAFKA-691:
--
Attachment: KAFKA-691-v1.patch
Here is a first draft (v1) patch.
1. Added the consumer property "producer.metadata.refresh.interval.ms" defaults
to 60 (10min)
2. The metadata is refreshed every 10min (only if a message is sent), and the
set of topics to refresh is tracked in the topicMetadataToRefresh Set (cleared
after every refresh) - I think the added value of refreshing regardless of
partition availability is to detect new partitions
3. The good news is that I didn't touch the Partitioner API, I only changed the
code to use available partitions if the key is null (as suggested by Jun), it
will also throw a UnknownTopicOrPartitionException("No leader for any
partition") if no partition is available at all
Let me know what you think about this patch. I ran a producer with that code
successfully and tested with a broker down.
I now have some concerns about the consumer: the refresh.leader.backoff.ms
config could help me (if i increase it to say, 10min) BUT the rebalance fails
in any case since there is no leader for some partitions
I don't have a good workaround yet for that, any help/suggestion appreciated.
> Fault tolerance broken with replication factor 1
>
>
> Key: KAFKA-691
> URL: https://issues.apache.org/jira/browse/KAFKA-691
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Jay Kreps
> Attachments: KAFKA-691-v1.patch
>
>
> In 0.7 if a partition was down we would just send the message elsewhere. This
> meant that the partitioning was really more of a "stickiness" then a hard
> guarantee. This made it impossible to depend on it for partitioned, stateful
> processing.
> In 0.8 when running with replication this should not be a problem generally
> as the partitions are now highly available and fail over to other replicas.
> However in the case of replication factor = 1 no longer really works for most
> cases as now a dead broker will give errors for that broker.
> I am not sure of the best fix. Intuitively I think this is something that
> should be handled by the Partitioner interface. However currently the
> partitioner has no knowledge of which nodes are available. So you could use a
> random partitioner, but that would keep going back to the down node.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-691) Fault tolerance broken with replication factor 1
[ https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13549105#comment-13549105 ] Maxime Brugidou commented on KAFKA-691: --- I agree with Jun solution, this would solve 3 (1 and 2 can be done manualy already -- just send a ReassignPartition command when you add a broker) I could probably implement this very quickly, I'm just not sure of how you get the availability of a partition, but i'll try to figure it out and submit a first patch tomorrow. > Fault tolerance broken with replication factor 1 > > > Key: KAFKA-691 > URL: https://issues.apache.org/jira/browse/KAFKA-691 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Jay Kreps > > In 0.7 if a partition was down we would just send the message elsewhere. This > meant that the partitioning was really more of a "stickiness" then a hard > guarantee. This made it impossible to depend on it for partitioned, stateful > processing. > In 0.8 when running with replication this should not be a problem generally > as the partitions are now highly available and fail over to other replicas. > However in the case of replication factor = 1 no longer really works for most > cases as now a dead broker will give errors for that broker. > I am not sure of the best fix. Intuitively I think this is something that > should be handled by the Partitioner interface. However currently the > partitioner has no knowledge of which nodes are available. So you could use a > random partitioner, but that would keep going back to the down node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-691) Fault tolerance broken with replication factor 1
[ https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548667#comment-13548667 ] Maxime Brugidou commented on KAFKA-691: --- I think the work-around is not really acceptable for me since it will consume 3x the resources (because replication of 3 is the minimum acceptable) and it will still make the cluster less available anyway (unless i have only 3 brokers). The thing is that 0.7 was making the cluster 100% available (for my use case, accepting data loss) as long a single broker was alive. A way to handle this would be to: 1. Have a lot of partitions per topic (more than the # of brokers) 2. Have something that rebalances the partitions and make sure a broker has a at least a partition for each topic (to make every topic "available") 3. Have a setting in the consumer/producer that say "I don't care about partitioning, just produce/consume wherever you can" > Fault tolerance broken with replication factor 1 > > > Key: KAFKA-691 > URL: https://issues.apache.org/jira/browse/KAFKA-691 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8 >Reporter: Jay Kreps > > In 0.7 if a partition was down we would just send the message elsewhere. This > meant that the partitioning was really more of a "stickiness" then a hard > guarantee. This made it impossible to depend on it for partitioned, stateful > processing. > In 0.8 when running with replication this should not be a problem generally > as the partitions are now highly available and fail over to other replicas. > However in the case of replication factor = 1 no longer really works for most > cases as now a dead broker will give errors for that broker. > I am not sure of the best fix. Intuitively I think this is something that > should be handled by the Partitioner interface. However currently the > partitioner has no knowledge of which nodes are available. So you could use a > random partitioner, but that would keep going back to the down node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-634) ConsoleProducer compresses messages and ignores the --compress flag
[ https://issues.apache.org/jira/browse/KAFKA-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548441#comment-13548441 ] Maxime Brugidou commented on KAFKA-634: --- KAFKA-506 fixed this (commit f64fd3dcbaace1dba7bbd72398bb3e7d28b41d61 in the 0.8 branch) This will be fixed in 0.8 I guess > ConsoleProducer compresses messages and ignores the --compress flag > --- > > Key: KAFKA-634 > URL: https://issues.apache.org/jira/browse/KAFKA-634 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.7 >Reporter: Anentropic > Labels: console, producer > > I am using the kafka-producer-shell.sh script without the --compress option > however my messages seem to be gzipped > the docs say compression is off by default: > http://incubator.apache.org/kafka/configuration.html > The only producer.properties file I can find is at: > /home/ubuntu/kafka-0.7.2-incubating-src/config/producer.properties > In there is: > compression.codec=0 > My process looks like: > root 1748 1746 0 Nov19 ?00:02:37 java -Xmx512M -server > -Dlog4j.configuration=file:/usr/local/bin/kafka/../config/log4j.properties > -Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false -cp > :/usr/local/bin/kafka/../project/boot/scala-2.8.0/lib/scala-compiler.jar:/usr/local/bin/kafka/../project/boot/scala-2.8.0/lib/scala-library.jar:/usr/local/bin/kafka/../core/target/scala_2.8.0/kafka-0.7.2.jar:/usr/local/bin/kafka/../core/lib/*.jar:/usr/local/bin/kafka/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar > kafka.producer.ConsoleProducer --topic logtail --zookeeper x.x.x.x:2181 > But the messages come out gobbledegook unless I use a client that understands > compressed messages, and in that client it identifies the bit as set to 1, > gzip compression. > Jun Rao jun...@gmail.com via incubator.apache.org > Nov 26 (1 day ago) > to kafka-users > This seems to be a bug in ConsoleProducer. It also compresses messages and > ignores the --compress flag. Could you file a jira? > Thanks, > Jun -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-634) ConsoleProducer compresses messages and ignores the --compress flag
[ https://issues.apache.org/jira/browse/KAFKA-634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou resolved KAFKA-634. --- Resolution: Fixed Fix Version/s: 0.8 > ConsoleProducer compresses messages and ignores the --compress flag > --- > > Key: KAFKA-634 > URL: https://issues.apache.org/jira/browse/KAFKA-634 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.7 >Reporter: Anentropic > Labels: console, producer > Fix For: 0.8 > > > I am using the kafka-producer-shell.sh script without the --compress option > however my messages seem to be gzipped > the docs say compression is off by default: > http://incubator.apache.org/kafka/configuration.html > The only producer.properties file I can find is at: > /home/ubuntu/kafka-0.7.2-incubating-src/config/producer.properties > In there is: > compression.codec=0 > My process looks like: > root 1748 1746 0 Nov19 ?00:02:37 java -Xmx512M -server > -Dlog4j.configuration=file:/usr/local/bin/kafka/../config/log4j.properties > -Dcom.sun.management.jmxremote > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.ssl=false -cp > :/usr/local/bin/kafka/../project/boot/scala-2.8.0/lib/scala-compiler.jar:/usr/local/bin/kafka/../project/boot/scala-2.8.0/lib/scala-library.jar:/usr/local/bin/kafka/../core/target/scala_2.8.0/kafka-0.7.2.jar:/usr/local/bin/kafka/../core/lib/*.jar:/usr/local/bin/kafka/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar > kafka.producer.ConsoleProducer --topic logtail --zookeeper x.x.x.x:2181 > But the messages come out gobbledegook unless I use a client that understands > compressed messages, and in that client it identifies the bit as set to 1, > gzip compression. > Jun Rao jun...@gmail.com via incubator.apache.org > Nov 26 (1 day ago) > to kafka-users > This seems to be a bug in ConsoleProducer. It also compresses messages and > ignores the --compress flag. Could you file a jira? > Thanks, > Jun -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-690) TopicMetadataRequest throws exception when no topics are specified
[ https://issues.apache.org/jira/browse/KAFKA-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548436#comment-13548436 ] Maxime Brugidou commented on KAFKA-690: --- this would resolve KAFKA-653 > TopicMetadataRequest throws exception when no topics are specified > -- > > Key: KAFKA-690 > URL: https://issues.apache.org/jira/browse/KAFKA-690 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: David Arthur > Fix For: 0.8 > > Attachments: KAFKA-690.patch > > > If no topics are sent in a TopicMetadataRequest, `readFrom` throws an > exception when trying to get the the head of the topic list for a debug > statement. > java.util.NoSuchElementException: head of empty list > at scala.collection.immutable.Nil$.head(List.scala:386) > at scala.collection.immutable.Nil$.head(List.scala:383) > at > kafka.api.TopicMetadataRequest$$anonfun$readFrom$2.apply(TopicMetadataRequest.scala:43) > at > kafka.api.TopicMetadataRequest$$anonfun$readFrom$2.apply(TopicMetadataRequest.scala:43) > at kafka.utils.Logging$class.debug(Logging.scala:51) > at kafka.api.TopicMetadataRequest$.debug(TopicMetadataRequest.scala:25) > at > kafka.api.TopicMetadataRequest$.readFrom(TopicMetadataRequest.scala:43) > at kafka.api.RequestKeys$$anonfun$4.apply(RequestKeys.scala:37) > at kafka.api.RequestKeys$$anonfun$4.apply(RequestKeys.scala:37) > at kafka.network.RequestChannel$Request.(RequestChannel.scala:47) > at kafka.network.Processor.read(SocketServer.scala:320) > at kafka.network.Processor.run(SocketServer.scala:231) > at java.lang.Thread.run(Thread.java:680) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-281) support multiple root log directories
[
https://issues.apache.org/jira/browse/KAFKA-281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13546858#comment-13546858
]
Maxime Brugidou commented on KAFKA-281:
---
I guess this was done on 0.8 as part of KAFKA-188
> support multiple root log directories
> -
>
> Key: KAFKA-281
> URL: https://issues.apache.org/jira/browse/KAFKA-281
> Project: Kafka
> Issue Type: Improvement
> Components: core
>Reporter: Jun Rao
>
> Currently, the log layout is {log.dir}/topicname-partitionid and one can only
> specify 1 {log.dir}. This limits the # of topics we can have per broker. We
> can potentially support multiple directories for {log.dir} and just assign
> topics using hashing or round-robin.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-684) ConsoleProducer does not have the queue-size option
[ https://issues.apache.org/jira/browse/KAFKA-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-684: -- Attachment: KAFKA-684-3.patch While I'm at it, I added this small feature: Exit at end of input stream (so you can do echo "test" | ./kafka-console-producer.sh or ./kafka-console-producer.sh < test without stopping the producer manually > ConsoleProducer does not have the queue-size option > --- > > Key: KAFKA-684 > URL: https://issues.apache.org/jira/browse/KAFKA-684 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Fix For: 0.8 > > Attachments: KAFKA-684-2.patch, KAFKA-684-3.patch, KAFKA-684.patch > > > When using the kafka ConsoleProducer (from script kafka-console-producer.sh), > you cannot set the queue.size, which gets very annoying when you want to > produce quickly a lot of messages. You definitely need to increase the > queue.size (or decrease the send timeout). > Here is a simple patch to add the option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-684) ConsoleProducer does not have the queue-size option
[ https://issues.apache.org/jira/browse/KAFKA-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-684: -- Attachment: KAFKA-684-2.patch I added: queue.enqueueTimeout.ms producer.request.required.acks producer.request.timeout.ms > ConsoleProducer does not have the queue-size option > --- > > Key: KAFKA-684 > URL: https://issues.apache.org/jira/browse/KAFKA-684 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Fix For: 0.8 > > Attachments: KAFKA-684-2.patch, KAFKA-684.patch > > > When using the kafka ConsoleProducer (from script kafka-console-producer.sh), > you cannot set the queue.size, which gets very annoying when you want to > produce quickly a lot of messages. You definitely need to increase the > queue.size (or decrease the send timeout). > Here is a simple patch to add the option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-685) ConsoleOffsetChecker does not work with 0.8
[ https://issues.apache.org/jira/browse/KAFKA-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-685: -- Attachment: KAFKA-685-2.patch New patch, updated the regex to ignore JMX port > ConsoleOffsetChecker does not work with 0.8 > --- > > Key: KAFKA-685 > URL: https://issues.apache.org/jira/browse/KAFKA-685 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Attachments: KAFKA-685-2.patch, KAFKA-685.patch > > > The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very > useful when used with the MirrorMaker. > Here is a patch to make it work with some cosmetic changes: > * script-friendly formatting (one line per partition) > * offsets do not correspond to bytes anymore (so the lag is in number of > messages, not GiB) > * --broker-info optional option to print the broker list at the end (like the > previous version) > Example: > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror > --zkconnect zoo.example.org:2181 > Group Topic Pid Offset logSize Lag > Owner > KafkaMirror test 0 215385 215385 0 > Some(KafkaMirror_broker01-1379350-71cf9117-0) > KafkaMirror test 1 683564 683564 0 > Some(KafkaMirror_broker03-1379351-71cf9117-0) > KafkaMirror test2 0 176943 176943 0 > Some(KafkaMirror_broker05-1379353-71cf91 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-685) ConsumerOffsetChecker does not work with 0.8
[ https://issues.apache.org/jira/browse/KAFKA-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-685: -- Description: The ConsumerOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker. Here is a patch to make it work with some cosmetic changes: * script-friendly formatting (one line per partition) * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB) * --broker-info optional option to print the broker list at the end (like the previous version) Example: bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181 Group Topic Pid Offset logSize Lag Owner KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0) KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0) KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91 was: The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker. Here is a patch to make it work with some cosmetic changes: * script-friendly formatting (one line per partition) * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB) * --broker-info optional option to print the broker list at the end (like the previous version) Example: bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181 Group Topic Pid Offset logSize Lag Owner KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0) KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0) KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91 > ConsumerOffsetChecker does not work with 0.8 > > > Key: KAFKA-685 > URL: https://issues.apache.org/jira/browse/KAFKA-685 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Attachments: KAFKA-685-2.patch, KAFKA-685.patch > > > The ConsumerOffsetChecker does not work anymore with 0.8, this tool is very > useful when used with the MirrorMaker. > Here is a patch to make it work with some cosmetic changes: > * script-friendly formatting (one line per partition) > * offsets do not correspond to bytes anymore (so the lag is in number of > messages, not GiB) > * --broker-info optional option to print the broker list at the end (like the > previous version) > Example: > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror > --zkconnect zoo.example.org:2181 > Group Topic Pid Offset logSize Lag > Owner > KafkaMirror test 0 215385 215385 0 > Some(KafkaMirror_broker01-1379350-71cf9117-0) > KafkaMirror test 1 683564 683564 0 > Some(KafkaMirror_broker03-1379351-71cf9117-0) > KafkaMirror test2 0 176943 176943 0 > Some(KafkaMirror_broker05-1379353-71cf91 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-685) ConsumerOffsetChecker does not work with 0.8
[ https://issues.apache.org/jira/browse/KAFKA-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-685: -- Summary: ConsumerOffsetChecker does not work with 0.8 (was: ConsoleOffsetChecker does not work with 0.8) > ConsumerOffsetChecker does not work with 0.8 > > > Key: KAFKA-685 > URL: https://issues.apache.org/jira/browse/KAFKA-685 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Attachments: KAFKA-685-2.patch, KAFKA-685.patch > > > The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very > useful when used with the MirrorMaker. > Here is a patch to make it work with some cosmetic changes: > * script-friendly formatting (one line per partition) > * offsets do not correspond to bytes anymore (so the lag is in number of > messages, not GiB) > * --broker-info optional option to print the broker list at the end (like the > previous version) > Example: > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror > --zkconnect zoo.example.org:2181 > Group Topic Pid Offset logSize Lag > Owner > KafkaMirror test 0 215385 215385 0 > Some(KafkaMirror_broker01-1379350-71cf9117-0) > KafkaMirror test 1 683564 683564 0 > Some(KafkaMirror_broker03-1379351-71cf9117-0) > KafkaMirror test2 0 176943 176943 0 > Some(KafkaMirror_broker05-1379353-71cf91 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-685) ConsoleOffsetChecker does not work with 0.8
[ https://issues.apache.org/jira/browse/KAFKA-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-685: -- Status: Patch Available (was: Open) > ConsoleOffsetChecker does not work with 0.8 > --- > > Key: KAFKA-685 > URL: https://issues.apache.org/jira/browse/KAFKA-685 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Attachments: KAFKA-685.patch > > > The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very > useful when used with the MirrorMaker. > Here is a patch to make it work with some cosmetic changes: > * script-friendly formatting (one line per partition) > * offsets do not correspond to bytes anymore (so the lag is in number of > messages, not GiB) > * --broker-info optional option to print the broker list at the end (like the > previous version) > Example: > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror > --zkconnect zoo.example.org:2181 > Group Topic Pid Offset logSize Lag > Owner > KafkaMirror test 0 215385 215385 0 > Some(KafkaMirror_broker01-1379350-71cf9117-0) > KafkaMirror test 1 683564 683564 0 > Some(KafkaMirror_broker03-1379351-71cf9117-0) > KafkaMirror test2 0 176943 176943 0 > Some(KafkaMirror_broker05-1379353-71cf91 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-685) ConsoleOffsetChecker does not work with 0.8
[ https://issues.apache.org/jira/browse/KAFKA-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-685: -- Attachment: KAFKA-685.patch > ConsoleOffsetChecker does not work with 0.8 > --- > > Key: KAFKA-685 > URL: https://issues.apache.org/jira/browse/KAFKA-685 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Attachments: KAFKA-685.patch > > > The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very > useful when used with the MirrorMaker. > Here is a patch to make it work with some cosmetic changes: > * script-friendly formatting (one line per partition) > * offsets do not correspond to bytes anymore (so the lag is in number of > messages, not GiB) > * --broker-info optional option to print the broker list at the end (like the > previous version) > Example: > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror > --zkconnect zoo.example.org:2181 > Group Topic Pid Offset logSize Lag > Owner > KafkaMirror test 0 215385 215385 0 > Some(KafkaMirror_broker01-1379350-71cf9117-0) > KafkaMirror test 1 683564 683564 0 > Some(KafkaMirror_broker03-1379351-71cf9117-0) > KafkaMirror test2 0 176943 176943 0 > Some(KafkaMirror_broker05-1379353-71cf91 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-685) ConsoleOffsetChecker does not work with 0.8
[
https://issues.apache.org/jira/browse/KAFKA-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Brugidou updated KAFKA-685:
--
Description:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
{{test}}
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
}}
was:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
{{
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
}}
> ConsoleOffsetChecker does not work with 0.8
> ---
>
> Key: KAFKA-685
> URL: https://issues.apache.org/jira/browse/KAFKA-685
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.8
>Reporter: Maxime Brugidou
> Attachments: KAFKA-685.patch
>
>
> The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
> useful when used with the MirrorMaker.
> Here is a patch to make it work with some cosmetic changes:
> * script-friendly formatting (one line per partition)
> * offsets do not correspond to bytes anymore (so the lag is in number of
> messages, not GiB)
> * --broker-info optional option to print the broker list at the end (like the
> previous version)
> Example:
> {{test}}
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
> --zkconnect zoo.example.org:2181
> Group Topic Pid Offset logSize Lag
> Owner
> KafkaMirror test 0 215385 215385 0
> Some(KafkaMirror_broker01-1379350-71cf9117-0)
> KafkaMirror test 1 683564 683564 0
> Some(KafkaMirror_broker03-1379351-71cf9117-0)
> KafkaMirror test2 0 176943 176943 0
> Some(KafkaMirror_broker05-1379353-71cf91
> }}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-685) ConsoleOffsetChecker does not work with 0.8
[
https://issues.apache.org/jira/browse/KAFKA-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Brugidou updated KAFKA-685:
--
Description:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
was:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
{{test}}
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
}}
> ConsoleOffsetChecker does not work with 0.8
> ---
>
> Key: KAFKA-685
> URL: https://issues.apache.org/jira/browse/KAFKA-685
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.8
>Reporter: Maxime Brugidou
> Attachments: KAFKA-685.patch
>
>
> The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
> useful when used with the MirrorMaker.
> Here is a patch to make it work with some cosmetic changes:
> * script-friendly formatting (one line per partition)
> * offsets do not correspond to bytes anymore (so the lag is in number of
> messages, not GiB)
> * --broker-info optional option to print the broker list at the end (like the
> previous version)
> Example:
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
> --zkconnect zoo.example.org:2181
> Group Topic Pid Offset logSize Lag
> Owner
> KafkaMirror test 0 215385 215385 0
> Some(KafkaMirror_broker01-1379350-71cf9117-0)
> KafkaMirror test 1 683564 683564 0
> Some(KafkaMirror_broker03-1379351-71cf9117-0)
> KafkaMirror test2 0 176943 176943 0
> Some(KafkaMirror_broker05-1379353-71cf91
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-685) ConsoleOffsetChecker does not work with 0.8
[
https://issues.apache.org/jira/browse/KAFKA-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Brugidou updated KAFKA-685:
--
Description:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
{{
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
}}
was:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
{noformat}
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
{noformat}
> ConsoleOffsetChecker does not work with 0.8
> ---
>
> Key: KAFKA-685
> URL: https://issues.apache.org/jira/browse/KAFKA-685
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.8
>Reporter: Maxime Brugidou
> Attachments: KAFKA-685.patch
>
>
> The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
> useful when used with the MirrorMaker.
> Here is a patch to make it work with some cosmetic changes:
> * script-friendly formatting (one line per partition)
> * offsets do not correspond to bytes anymore (so the lag is in number of
> messages, not GiB)
> * --broker-info optional option to print the broker list at the end (like the
> previous version)
> Example:
> {{
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
> --zkconnect zoo.example.org:2181
> Group Topic Pid Offset logSize Lag
> Owner
> KafkaMirror test 0 215385 215385 0
> Some(KafkaMirror_broker01-1379350-71cf9117-0)
> KafkaMirror test 1 683564 683564 0
> Some(KafkaMirror_broker03-1379351-71cf9117-0)
> KafkaMirror test2 0 176943 176943 0
> Some(KafkaMirror_broker05-1379353-71cf91
> }}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-685) ConsoleOffsetChecker does not work with 0.8
[
https://issues.apache.org/jira/browse/KAFKA-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Brugidou updated KAFKA-685:
--
Description:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
{code:bash}
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
{code}
was:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
{code}
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
{code}
> ConsoleOffsetChecker does not work with 0.8
> ---
>
> Key: KAFKA-685
> URL: https://issues.apache.org/jira/browse/KAFKA-685
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.8
>Reporter: Maxime Brugidou
>
> The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
> useful when used with the MirrorMaker.
> Here is a patch to make it work with some cosmetic changes:
> * script-friendly formatting (one line per partition)
> * offsets do not correspond to bytes anymore (so the lag is in number of
> messages, not GiB)
> * --broker-info optional option to print the broker list at the end (like the
> previous version)
> Example:
> {code:bash}
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
> --zkconnect zoo.example.org:2181
> Group Topic Pid Offset logSize Lag
> Owner
> KafkaMirror test 0 215385 215385 0
> Some(KafkaMirror_broker01-1379350-71cf9117-0)
> KafkaMirror test 1 683564 683564 0
> Some(KafkaMirror_broker03-1379351-71cf9117-0)
> KafkaMirror test2 0 176943 176943 0
> Some(KafkaMirror_broker05-1379353-71cf91
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-685) ConsoleOffsetChecker does not work with 0.8
[
https://issues.apache.org/jira/browse/KAFKA-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Brugidou updated KAFKA-685:
--
Description:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
{noformat}
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
{noformat}
was:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
{code}
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
{code}
> ConsoleOffsetChecker does not work with 0.8
> ---
>
> Key: KAFKA-685
> URL: https://issues.apache.org/jira/browse/KAFKA-685
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.8
>Reporter: Maxime Brugidou
>
> The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
> useful when used with the MirrorMaker.
> Here is a patch to make it work with some cosmetic changes:
> * script-friendly formatting (one line per partition)
> * offsets do not correspond to bytes anymore (so the lag is in number of
> messages, not GiB)
> * --broker-info optional option to print the broker list at the end (like the
> previous version)
> Example:
> {noformat}
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
> --zkconnect zoo.example.org:2181
> Group Topic Pid Offset logSize Lag
> Owner
> KafkaMirror test 0 215385 215385 0
> Some(KafkaMirror_broker01-1379350-71cf9117-0)
> KafkaMirror test 1 683564 683564 0
> Some(KafkaMirror_broker03-1379351-71cf9117-0)
> KafkaMirror test2 0 176943 176943 0
> Some(KafkaMirror_broker05-1379353-71cf91
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-685) ConsoleOffsetChecker does not work with 0.8
[
https://issues.apache.org/jira/browse/KAFKA-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Brugidou updated KAFKA-685:
--
Description:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
{code}
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
{code}
was:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
{code:bash}
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
{code}
> ConsoleOffsetChecker does not work with 0.8
> ---
>
> Key: KAFKA-685
> URL: https://issues.apache.org/jira/browse/KAFKA-685
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.8
>Reporter: Maxime Brugidou
>
> The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
> useful when used with the MirrorMaker.
> Here is a patch to make it work with some cosmetic changes:
> * script-friendly formatting (one line per partition)
> * offsets do not correspond to bytes anymore (so the lag is in number of
> messages, not GiB)
> * --broker-info optional option to print the broker list at the end (like the
> previous version)
> Example:
> {code}
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
> --zkconnect zoo.example.org:2181
> Group Topic Pid Offset logSize Lag
> Owner
> KafkaMirror test 0 215385 215385 0
> Some(KafkaMirror_broker01-1379350-71cf9117-0)
> KafkaMirror test 1 683564 683564 0
> Some(KafkaMirror_broker03-1379351-71cf9117-0)
> KafkaMirror test2 0 176943 176943 0
> Some(KafkaMirror_broker05-1379353-71cf91
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-685) ConsoleOffsetChecker does not work with 0.8
Maxime Brugidou created KAFKA-685: - Summary: ConsoleOffsetChecker does not work with 0.8 Key: KAFKA-685 URL: https://issues.apache.org/jira/browse/KAFKA-685 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8 Reporter: Maxime Brugidou The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very useful when used with the MirrorMaker. Here is a patch to make it work with some cosmetic changes: * script-friendly formatting (one line per partition) * offsets do not correspond to bytes anymore (so the lag is in number of messages, not GiB) * --broker-info optional option to print the broker list at the end (like the previous version) Example: bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect zoo.example.org:2181 Group Topic Pid Offset logSize Lag Owner KafkaMirror test 0 215385 215385 0 Some(KafkaMirror_broker01-1379350-71cf9117-0) KafkaMirror test 1 683564 683564 0 Some(KafkaMirror_broker03-1379351-71cf9117-0) KafkaMirror test2 0 176943 176943 0 Some(KafkaMirror_broker05-1379353-71cf91 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-685) ConsoleOffsetChecker does not work with 0.8
[
https://issues.apache.org/jira/browse/KAFKA-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Brugidou updated KAFKA-685:
--
Description:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
{code}
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
{code}
was:
The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
useful when used with the MirrorMaker.
Here is a patch to make it work with some cosmetic changes:
* script-friendly formatting (one line per partition)
* offsets do not correspond to bytes anymore (so the lag is in number of
messages, not GiB)
* --broker-info optional option to print the broker list at the end (like the
previous version)
Example:
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
--zkconnect zoo.example.org:2181
Group Topic Pid Offset logSize Lag
Owner
KafkaMirror test 0 215385 215385 0
Some(KafkaMirror_broker01-1379350-71cf9117-0)
KafkaMirror test 1 683564 683564 0
Some(KafkaMirror_broker03-1379351-71cf9117-0)
KafkaMirror test2 0 176943 176943 0
Some(KafkaMirror_broker05-1379353-71cf91
> ConsoleOffsetChecker does not work with 0.8
> ---
>
> Key: KAFKA-685
> URL: https://issues.apache.org/jira/browse/KAFKA-685
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 0.8
>Reporter: Maxime Brugidou
>
> The ConsoleOffsetChecker does not work anymore with 0.8, this tool is very
> useful when used with the MirrorMaker.
> Here is a patch to make it work with some cosmetic changes:
> * script-friendly formatting (one line per partition)
> * offsets do not correspond to bytes anymore (so the lag is in number of
> messages, not GiB)
> * --broker-info optional option to print the broker list at the end (like the
> previous version)
> Example:
> {code}
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror
> --zkconnect zoo.example.org:2181
> Group Topic Pid Offset logSize Lag
> Owner
> KafkaMirror test 0 215385 215385 0
> Some(KafkaMirror_broker01-1379350-71cf9117-0)
> KafkaMirror test 1 683564 683564 0
> Some(KafkaMirror_broker03-1379351-71cf9117-0)
> KafkaMirror test2 0 176943 176943 0
> Some(KafkaMirror_broker05-1379353-71cf91
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-684) ConsoleProducer does not have the queue-size option
[ https://issues.apache.org/jira/browse/KAFKA-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-684: -- Attachment: KAFKA-684.patch > ConsoleProducer does not have the queue-size option > --- > > Key: KAFKA-684 > URL: https://issues.apache.org/jira/browse/KAFKA-684 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Fix For: 0.8 > > Attachments: KAFKA-684.patch > > > When using the kafka ConsoleProducer (from script kafka-console-producer.sh), > you cannot set the queue.size, which gets very annoying when you want to > produce quickly a lot of messages. You definitely need to increase the > queue.size (or decrease the send timeout). > Here is a simple patch to add the option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (KAFKA-684) ConsoleProducer does not have the queue-size option
Maxime Brugidou created KAFKA-684: - Summary: ConsoleProducer does not have the queue-size option Key: KAFKA-684 URL: https://issues.apache.org/jira/browse/KAFKA-684 Project: Kafka Issue Type: Bug Components: core Reporter: Maxime Brugidou When using the kafka ConsoleProducer (from script kafka-console-producer.sh), you cannot set the queue.size, which gets very annoying when you want to produce quickly a lot of messages. You definitely need to increase the queue.size (or decrease the send timeout). Here is a simple patch to add the option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-684) ConsoleProducer does not have the queue-size option
[ https://issues.apache.org/jira/browse/KAFKA-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-684: -- Fix Version/s: 0.8 Affects Version/s: 0.8 Status: Patch Available (was: Open) > ConsoleProducer does not have the queue-size option > --- > > Key: KAFKA-684 > URL: https://issues.apache.org/jira/browse/KAFKA-684 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.8 >Reporter: Maxime Brugidou > Fix For: 0.8 > > > When using the kafka ConsoleProducer (from script kafka-console-producer.sh), > you cannot set the queue.size, which gets very annoying when you want to > produce quickly a lot of messages. You definitely need to increase the > queue.size (or decrease the send timeout). > Here is a simple patch to add the option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-133) Publish kafka jar to a public maven repository
[
https://issues.apache.org/jira/browse/KAFKA-133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Brugidou updated KAFKA-133:
--
Attachment: KAFKA-133.patch
Taking Otis notes into account, Here is the simplest patch I could make, that
applies cleanly on 0.8:
Here are the changes:
* Version is 0.8.0-SNAPSHOT until 0.8.0 becomes stable
* You need to change the publish URL and credentials location
* Use com.101tec.zkclient 0.2
* Use com.yammer.metrics.metrics-* 3.0.0-SNAPSHOT (I had to use snapshot
because there is no 3.0.0 stable)
* Use org.slf4j.slf4j-simple 1.6.4 ("latest.release" was used)
To publish:
* ./sbt publish-local && ./sbt publish
This will create a pom for kafka that depends on all the subprojects. The
"main" project that you generally want to use is core-kafka_2.8.0 like this:
kafka
core-kafka_2.8.0
0.8.0-SNAPSHOT
> Publish kafka jar to a public maven repository
> --
>
> Key: KAFKA-133
> URL: https://issues.apache.org/jira/browse/KAFKA-133
> Project: Kafka
> Issue Type: Improvement
>Affects Versions: 0.6, 0.8
>Reporter: Neha Narkhede
> Labels: patch
> Fix For: 0.8
>
> Attachments: KAFKA-133.patch, pom.xml
>
>
> The released kafka jar must be download manually and then deploy to a private
> repository before they can be used by a developer using maven2.
> Similar to other Apache projects, it will be nice to have a way to publish
> Kafka releases to a public maven repo.
> In the past, we gave it a try using sbt publish to Sonatype Nexus maven repo,
> but ran into some authentication problems. It will be good to revisit this
> and get it resolved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
