Re: High system.io.await on Kafka brokers?
Sounds like you're reaching the limits of what your disks will do either on reads or writes. Debug it as you would any other disk based app, https://haydenjames.io/linux-server-performance-disk-io-slowing-application/ might help. On Tue, 22 Jan 2019 at 09:19, wenxing zheng wrote: > Dear all, > > We got a kafka cluster with 5 nodes, and from the metrics of datadog, we > found that regularly the elapse for sending to kafka was more than 200ms, > and there was a peek on the system.io.await. > > Please help to advice what would be the problem and any hints? > [image: image.png] > > Kind regards, Wenxing >
Re: MockConsumer class for Python?
Why not just mock out the Kafka client in your tests and have it call a
function which yields a kafka message every call?
```
def consumer():
for _ in range(99):
yield KafkaMessage('key', 'value')mock_consumer =
mocker.patch.object(foo, 'consumer', consumer())
```
Is there any specific feature you're after?
Re: Broker won't exit...
Have you tried a normal kill (sigterm) against the java process? __ Sam Pegler PRODUCTION ENGINEER T. +44(0) 07 562 867 486 <http://www.infectiousmedia.com/> 3-7 Herbal Hill / London / EC1R 5EJ www.infectiousmedia.com This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately, and do not disclose the contents to another person, use it for any purpose, or store, or copy the information in any medium. Please also destroy and delete the message from your computer. On 9 January 2018 at 22:44, Skip Montanaro <skip.montan...@gmail.com> wrote: > I only discovered the kafka-server-stop.sh script a couple days ago. I > can't seem to make it do its thing (the corresponding zookeeper stop > script seems to work just fine). All consumers have been stopped. Lsof > still shows the Kafka broker process listening on its port. The last > connection left the CLOSE_WAIT state several minutes ago. Gstack shows > 169 threads, most in pthread_cond_wait(), a handful in other wait-like > functions (sem_wait, pthread_join, pthread_cond_timedwait, poll, > epoll_wait). I'm running 2.11-1.0.0 on a Red Hat 6 server. > > What does it take to get a broker to exit (short of kill -9)? > > Thx, > > Skip Montanaro >
Re: How to choose which topic to replicate in MirrorMaker?
Use a whitelist with one topic. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330#Kafkamirroring(MirrorMaker)-Whitelistorblacklist __ Sam Pegler PRODUCTION ENGINEER T. +44(0) 07 562 867 486 <http://www.infectiousmedia.com/> 3-7 Herbal Hill / London / EC1R 5EJ www.infectiousmedia.com This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately, and do not disclose the contents to another person, use it for any purpose, or store, or copy the information in any medium. Please also destroy and delete the message from your computer. On 30 October 2017 at 15:54, Alexandr Porunov <alexandr.poru...@gmail.com> wrote: > Hello, > > Is it possible to choose which topic to replicate in MirrorMaker (to not > mirror the whole cluster but the part of it)? > Is it possible to choose a partition in a topic to replicate? > > Best regards, > Alexandr >
Re: Consumer vs replication traffic priority
Have you checked the documentation on quotas? https://kafka.apache.org/documentation/#design_quotas This should enable you to do what you're after. __ Sam Pegler PRODUCTION ENGINEER T. +44(0) 07 562 867 486 <http://www.infectiousmedia.com/> 3-7 Herbal Hill / London / EC1R 5EJ www.infectiousmedia.com This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately, and do not disclose the contents to another person, use it for any purpose, or store, or copy the information in any medium. Please also destroy and delete the message from your computer. On 10 July 2017 at 19:27, Łukasz Mierzwa <l.mier...@gmail.com> wrote: > Hi, > > looking at docs I see that Kafka seems to support throttling of > consumer/replication traffic, but I can't find anything that would suggest > you can prioritize one traffic type over another. > The problem: if at some point consumers starts to be lagging they will > start consuming messages as fast as they can, if there's enough lag to > recover from and if there's enough affected consumers that traffic can > easily saturate network on the leader, which can in turn affect replicas of > the partitions on that leader. > Is there a way to avoid replicas falling out of sync with the leader under > such scenario? Is there a way to prefer replication traffic over consumer > traffic? Or is throttling the only way to achieve this? Throttling solves > this, but it requires setting limits that can change over time, so it's bit > more manual and requires more maintenance than priority based solution. > > -- > Łukasz Mierzwa >
Re: Brokers is down by “java.io.IOException: Too many open files”
If you're using a systemd based OS you'll actually need to set it in the unit file. LimitNOFILE=10 https://kafka.apache.org/documentation/#upgrade_10_1_breaking contains some changes re file handles as well. __ Sam Pegler PRODUCTION ENGINEER T. +44(0) 07 562 867 486 <http://www.infectiousmedia.com/> 3-7 Herbal Hill / London / EC1R 5EJ www.infectiousmedia.com This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately, and do not disclose the contents to another person, use it for any purpose, or store, or copy the information in any medium. Please also destroy and delete the message from your computer. On 13 May 2017 at 02:57, Caleb Welton <ca...@autonomic.ai> wrote: > You need to up your OS open file limits, something like this should work: > > # /etc/security/limits.conf > * - nofile 65536 > > > > > On Fri, May 12, 2017 at 6:34 PM, Yang Cui <y...@freewheel.tv> wrote: > > > Our Kafka cluster is broken down by the problem “java.io.IOException: > Too > > many open files” three times in 3 weeks. > > > > We encounter these problem on both 0.9.0.1 and 0.10.2.1 version. > > > > The error is like: > > > > java.io.IOException: Too many open files > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > > at sun.nio.ch.ServerSocketChannelImpl.accept( > > ServerSocketChannelImpl.java:422) > > at sun.nio.ch.ServerSocketChannelImpl.accept( > > ServerSocketChannelImpl.java:250) > > at kafka.network.Acceptor.accept(SocketServer.scala:340) > > at kafka.network.Acceptor.run(SocketServer.scala:283) > > at java.lang.Thread.run(Thread.java:745) > > > > Is someone encounter the similar problem? > > > > > > >
Re: checking consumer lag on KStreams app?
Is the following PR present https://github.com/apache/kafka/pull/1336? __ Sam Pegler PRODUCTION ENGINEER T. +44(0) 07 562 867 486 <http://www.infectiousmedia.com/> 3-7 Herbal Hill / London / EC1R 5EJ www.infectiousmedia.com This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately, and do not disclose the contents to another person, use it for any purpose, or store, or copy the information in any medium. Please also destroy and delete the message from your computer. On 13 December 2016 at 10:03, Sachin Mittal <sjmit...@gmail.com> wrote: > If this is a bug then it is not fixed because I just build kafka from > source and it gave me the reported error. > > On Tue, Dec 13, 2016 at 3:24 PM, Sam Pegler <sam.pegler@infectiousmedia. > com> > wrote: > > > >You can only check the offsets when there is an active member of the > > consumer group. > > > > This was a bug [1] thats been fixed. Thanks to Vincent Dautremont for > > pointing this out to me a while ago. > > > > http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/% > > 3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail. > > gmail.com%3E > > > > __ > > > > Sam Pegler > > > > PRODUCTION ENGINEER > > > > T. +44(0) 07 562 867 486 > > > > <http://www.infectiousmedia.com/> > > 3-7 Herbal Hill / London / EC1R 5EJ > > www.infectiousmedia.com > > > > This email and any attachments are confidential and may also be > privileged. > > If you > > are not the intended recipient, please notify the sender immediately, and > > do not > > disclose the contents to another person, use it for any purpose, or > store, > > or copy > > the information in any medium. Please also destroy and delete the message > > from > > your computer. > > > > > > On 13 December 2016 at 09:39, Damian Guy <damian@gmail.com> wrote: > > > > > Hi Sachin > > > > > > That is correct. You can only check the offsets when there is an active > > > member of the consumer group. In this case that would mean that you > have > > at > > > least one instance of your streams application running. > > > > > > Thanks, > > > Damian > > > > > > On Tue, 13 Dec 2016 at 06:58 Sachin Mittal <sjmit...@gmail.com> wrote: > > > > > > > Hi, > > > > I used the following command > > > > bin\windows\kafka-consumer-groups.bat --bootstrap-server > > localhost:9092 > > > > --describe --group test > > > > and I get the following output > > > > > > > > Note: This will only show information about consumers that use the > Java > > > > consumer API (non-ZooKeeper-based consumers). > > > > > > > > Error: Consumer group 'test' has no active members. > > > > > > > > What does this mean. > > > > > > > > It means I can check the offset of consumer only when streams > > applictaion > > > > "test" is running. > > > > > > > > Thanks > > > > Sachin > > > > > > > > > > > > On Mon, Dec 12, 2016 at 8:33 PM, Damian Guy <damian@gmail.com> > > > wrote: > > > > > > > > > Hi Sachin, > > > > > > > > > > You should use the kafka-consumer-groups.sh command. The > > > > > ConsumerOffsetChecker is deprecated and is only for the old > consumer. > > > > > > > > > > Thanks, > > > > > Damian > > > > > > > > > > On Mon, 12 Dec 2016 at 14:32 Sachin Mittal <sjmit...@gmail.com> > > wrote: > > > > > > > > > > > Hi, > > > > > > I have a streams application running with application id test. > > > > > > When I try to check consumer lag like you suggested I get the > > > following > > > > > > issue: > > > > > > > > > > > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker > > --zookeeper > > > > > > localhost:2181 --group test > > > > > > [2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is > > > > > > deprecated and will be dropped in releases following 0.9.0. Use > > > > > > ConsumerGroupCommand instead. (kafka.tools. > ConsumerOffsetChecker$) > > > > > > SLF4J:
Re: checking consumer lag on KStreams app?
>You can only check the offsets when there is an active member of the consumer group. This was a bug [1] thats been fixed. Thanks to Vincent Dautremont for pointing this out to me a while ago. http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/%3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail.gmail.com%3E __ Sam Pegler PRODUCTION ENGINEER T. +44(0) 07 562 867 486 <http://www.infectiousmedia.com/> 3-7 Herbal Hill / London / EC1R 5EJ www.infectiousmedia.com This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately, and do not disclose the contents to another person, use it for any purpose, or store, or copy the information in any medium. Please also destroy and delete the message from your computer. On 13 December 2016 at 09:39, Damian Guy <damian@gmail.com> wrote: > Hi Sachin > > That is correct. You can only check the offsets when there is an active > member of the consumer group. In this case that would mean that you have at > least one instance of your streams application running. > > Thanks, > Damian > > On Tue, 13 Dec 2016 at 06:58 Sachin Mittal <sjmit...@gmail.com> wrote: > > > Hi, > > I used the following command > > bin\windows\kafka-consumer-groups.bat --bootstrap-server localhost:9092 > > --describe --group test > > and I get the following output > > > > Note: This will only show information about consumers that use the Java > > consumer API (non-ZooKeeper-based consumers). > > > > Error: Consumer group 'test' has no active members. > > > > What does this mean. > > > > It means I can check the offset of consumer only when streams applictaion > > "test" is running. > > > > Thanks > > Sachin > > > > > > On Mon, Dec 12, 2016 at 8:33 PM, Damian Guy <damian@gmail.com> > wrote: > > > > > Hi Sachin, > > > > > > You should use the kafka-consumer-groups.sh command. The > > > ConsumerOffsetChecker is deprecated and is only for the old consumer. > > > > > > Thanks, > > > Damian > > > > > > On Mon, 12 Dec 2016 at 14:32 Sachin Mittal <sjmit...@gmail.com> wrote: > > > > > > > Hi, > > > > I have a streams application running with application id test. > > > > When I try to check consumer lag like you suggested I get the > following > > > > issue: > > > > > > > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper > > > > localhost:2181 --group test > > > > [2016-12-12 10:26:01,348] WARN WARNING: ConsumerOffsetChecker is > > > > deprecated and will be dropped in releases following 0.9.0. Use > > > > ConsumerGroupCommand instead. (kafka.tools.ConsumerOffsetChecker$) > > > > SLF4J: Class path contains multiple SLF4J bindings. > > > > SLF4J: Found binding in > > > > > > > > [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/ > > > libs/logback-classic-1.0.3.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > > > > SLF4J: Found binding in > > > > > > > > [jar:file:/home/testuser/kafka/kafka_2.10-0.10.0.1/ > > > libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > > > > explanation. > > > > SLF4J: Actual binding is of type > > > > [ch.qos.logback.classic.selector.DefaultContextSelector] > > > > Exiting due to: org.apache.zookeeper.KeeperException$ > NoNodeException: > > > > KeeperErrorCode = NoNode for /consumers/test/owners. > > > > > > > > Please let me know where I may be going wrong. > > > > I have the kafka logs set in folder > > > > /data01/testuser/kafka-logs > > > > > > > > Under kafka-logs I see many folders with name something like > > > > consumer_offsets_* > > > > > > > > I have the stream dir set in folder > > > > /data01/testuser/kafka-streams/test > > > > > > > > Thanks > > > > Sachin > > > > > > > > > > > > On Sun, Dec 11, 2016 at 2:19 AM, Matthias J. Sax < > > matth...@confluent.io> > > > > wrote: > > > > > > > > > It's basically just a consumer as any other. The application.id is > > > used > > > > > as consumer group.id. > > > > > > > > > > So just use the available tools you do use to check consumer lag. > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > On 12/9/16 5:49 PM, Jon Yeargers wrote: > > > > > > How would this be done? > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Kafka 0.10 Monitoring tool
Have you thought of just wrapping the provided Java classes (kafka-consumer-groups.sh) with the programming language of your choice? For example... /opt/kafka/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server eu1-kb-1:9092 --describe --group foo GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER foo click 9 71971 71971 0 sarama_/10.240.0.46 foo click 10 71699 71700 1 sarama_/10.240.0.46 foo click 11 72052 72052 0 sarama_/10.240.0.46 Can be quite easily parsed to provide whatever output you require. __ Sam Pegler PRODUCTION ENGINEER T. +44(0) 07 562 867 486 <http://www.infectiousmedia.com/> 3-7 Herbal Hill / London / EC1R 5EJ www.infectiousmedia.com This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately, and do not disclose the contents to another person, use it for any purpose, or store, or copy the information in any medium. Please also destroy and delete the message from your computer. On 15 November 2016 at 14:28, Ghosh, Achintya (Contractor) < achintya_gh...@comcast.com> wrote: > Thank you Otis for your reply. > > Kafka Manger does not work during the high load, it shows the timeout and > Burrow and KafkaOffsetMonitor does not return the group names properly even > during the load. > > SPM is not an open source, so do you have anything opensource that works > in Kafka 0.10 version? > > Thanks > Achintya > > -Original Message- > From: Otis Gospodnetić [mailto:otis.gospodne...@gmail.com] > Sent: Monday, November 14, 2016 9:25 PM > To: users@kafka.apache.org > Cc: d...@kafka.apache.org > Subject: Re: Kafka 0.10 Monitoring tool > > Hi, > > Why are these tools not working perfectly for you? > Does it *have to* be open-source? If not, Sematext SPM collects a lot of > Kafka metrics, with consumer lag being one of them -- > https://sematext.com/blog/2016/06/07/kafka-consumer-lag- > offsets-monitoring/ > > Otis > -- > Monitoring - Log Management - Alerting - Anomaly Detection Solr & > Elasticsearch Consulting Support Training - http://sematext.com/ > > > On Mon, Nov 14, 2016 at 5:16 PM, Ghosh, Achintya (Contractor) < > achintya_gh...@comcast.com> wrote: > > > Hi there, > > What is the best open source tool for Kafka monitoring mainly to check > > the offset lag. We tried the following tools: > > > > > > 1. Burrow > > > > 2. KafkaOffsetMonitor > > > > 3. Prometheus and Grafana > > > > 4. Kafka Manager > > > > But nothing is working perfectly. Please help us on this. > > > > Thanks > > Achintya > > > > >
Re: Lost offsets after migration to Kafka brokers v0.10.0
clj-kafka uses the old consumer API's and offset storage in ZK. If I were you I'd migrate to https://github.com/weftio/gregor which wraps the new consumer API and stores offsets in Kafka. I'm going to assume you didn't migrate ZK state based off this? __ On 16 August 2016 at 12:15, Javier Holguera < javier.holgu...@fundingcircle.com> wrote: > Hi, > > Yesterday my company completed a “successful” migration from Kafka brokers > v0.9.0.1 to Kafka 0.10.0. > > However the migration can’t be considered completely successfully because > we accidentally lost our offsets. Fortunately our apps are designed to be > able to replay from the beginning on the topic without much problem, but > it’s something we weren’t expecting and I would like to understand what we > did wrong to let his happen. > > Our apps use kafka client v0.8.2.1 wrapped around latest version of > clj-kafka. We are using its functionality to commit offsets (like here: > https://github.com/pingles/clj-kafka/blob/master/src/clj_ > kafka/offset.clj#L70) > using OffsetCommitRequest. > > Any help would be welcomed. > > Thanks! > > -- > Javier Holguera > Sent with Airmail >
Re: Lots of messages when starting Kafka
>If you kill Kafka with SIGTERM, then naturally it will leave behind corrupt index files. It will safely rebuild them. Don't worry. I think you mean SIGKILL. SIGTERM should be handled normally an won't result in corrupted index files at startup. __ Sam Pegler Production Engineer T. +44(0) 07 562 867 486 <http://www.infectiousmedia.com/> 3-7 Herbal Hill / London / EC1R 5EJ www.infectiousmedia.com This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately, and do not disclose the contents to another person, use it for any purpose, or store, or copy the information in any medium. Please also destroy and delete the message from your computer. On 6 July 2016 at 14:54, Peter Davis <davi...@gmail.com> wrote: > If you kill Kafka with SIGTERM, then naturally it will leave behind corrupt > index files. It will safely rebuild them. Don't worry. > > The Group Metadata Manager messages are normal -- that's just Kafka > starting up. > > Perhaps the corrupted index log messages could be less scary, though. This > is a very common question from new users... > > -Peter > > On Wed, Jul 6, 2016 at 06:50 Fabian Kochem < > fabian.koc...@concepts-and-training.de> wrote: > > > Hi, > > > > this is my first time using a mailing list so please forgive me if I'm > > doing something wrong. > > > > When I start Kafka, I get a ton of messages like this: > > > > WARN Found a corrupted index file, > > /kafka/logs/__consumer_offsets-48/.index, deleting > > and rebuilding index... (kafka.log.Log) > > INFO Recovering unflushed segment 0 in log __consumer_offsets-48. > > (kafka.log.Log) > > INFO Completed load of log __consumer_offsets-48 with log end offset 0 > > (kafka.log.Log) > > WARN Found a corrupted index file, > > /kafka/logs/__consumer_offsets-19/.index, deleting > > and rebuilding index... (kafka.log.Log) > > INFO Recovering unflushed segment 0 in log __consumer_offsets-19. > > (kafka.log.Log) > > INFO Completed load of log __consumer_offsets-19 with log end offset 0 > > (kafka.log.Log) > > WARN Found a corrupted index file, > > /kafka/logs/__consumer_offsets-3/.index, deleting > > and rebuilding index... (kafka.log.Log) > > INFO Recovering unflushed segment 0 in log __consumer_offsets-3. > > (kafka.log.Log) > > INFO Completed load of log __consumer_offsets-3 with log end offset 0 > > (kafka.log.Log) > > > > And after that, there's an equal amount of messages which look like this: > > > > INFO [Group Metadata Manager on Broker 1001]: Loading offsets and group > > metadata from [__consumer_offsets,22] > > (kafka.coordinator.GroupMetadataManager) > > INFO [Group Metadata Manager on Broker 1001]: Finished loading offsets > > from [__consumer_offsets,22] in 11 milliseconds. > > (kafka.coordinator.GroupMetadataManager) > > INFO [Group Metadata Manager on Broker 1001]: Loading offsets and group > > metadata from [__consumer_offsets,25] > > (kafka.coordinator.GroupMetadataManager) > > INFO [Group Metadata Manager on Broker 1001]: Finished loading offsets > > from [__consumer_offsets,25] in 4 milliseconds. > > (kafka.coordinator.GroupMetadataManager) > > INFO [Group Metadata Manager on Broker 1001]: Loading offsets and group > > metadata from [__consumer_offsets,28] > > (kafka.coordinator.GroupMetadataManager) > > INFO [Group Metadata Manager on Broker 1001]: Finished loading offsets > > from [__consumer_offsets,28] in 2 milliseconds. > > (kafka.coordinator.GroupMetadataManager) > > > > Did I misconfigure something? > > > > I only have one topic with a few consumers, so this seems a bit much. I > > shutdown Kafka using SIGTERM. > > > > Thanks in advance, > > Fabian Kochem > > >
Re: Kafka LTS release
I would assume (maybe incorrectly) that users who were after a LTS style release would instead be going for one of the commercial versions. Clouderas for example is https://cloudera.com/products/apache-hadoop/apache-kafka.html, they'll then manage patches and provide support for you? Sam Pegler Site Reliability Engineer T. +44(0) 07 562 867 486 [image: Infectious Media] 3-7 Herbal Hill / London / EC1R 5EJ www.infectiousmedia.com [image: Infectious Media] <http://www.infectiousmedia.com/> [image: Facebook] <http://www.facebook.com/infectiousmedia> [image: Twitter] <https://twitter.com/infectiousmedia> [image: LinkedIn] <http://www.linkedin.com/company/infectious-media-ltd> [image: Youtube] <http://www.youtube.com/user/InfectiousMediaLtd> This email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately, and do not disclose the contents to another person, use it for any purpose, or store, or copy the information in any medium. Please also destroy and delete the message from your computer. On 21 March 2016 at 12:51, Achanta Vamsi Subhash <achanta.va...@flipkart.com > wrote: > Gerard, > > I think many people use Kafka just like any other stable software. The > producer and consumer apis are mostly fixed now and many companies across > the world are using it on production for critical use-cases. I think it is > already *expected *to work as per the theory and any bugs need to be > patched. As there is no one patching the older releases and the companies > refusing to upgrade due to the way enterprises work, can we somehow start > towards an LTS release by treating 0.10.0.0 as the LTS release to start > with? > > On Mon, Mar 21, 2016 at 4:49 PM, Gerard Klijs <gerard.kl...@dizzit.com> > wrote: > > > I think Kafka at the moment is not mature enough to support a LTS > release. > > I think it will take a lot of effort to 'guarantee' a back-port will be > > more safe to use in production then the new release. For example, when > you > > will manage the release of 0.9.0.2, with the fixes from 0.10.0.0, you > need > > to make sure all the 0.9.0.1 clients still work with it, and you don't > > introduce new bugs by the partial merge. > > I do think once there will be a 1.0.0.0 release it would be great to > have a > > lts release. > > > > On Mon, Mar 21, 2016 at 11:54 AM Achanta Vamsi Subhash < > > achanta.va...@flipkart.com> wrote: > > > > > *bump* > > > > > > Any opinions on this? > > > > > > On Mon, Mar 14, 2016 at 4:37 PM, Achanta Vamsi Subhash < > > > achanta.va...@flipkart.com> wrote: > > > > > > > Hi all, > > > > > > > > We find that there are many releases of Kafka and not all the bugs > are > > > > back ported to the older releases. Can we have a LTS (Long Term > > Support) > > > > release which can be supported for 2 years with all the bugs > > back-ported? > > > > > > > > This will be very helpful as during the last 2-3 releases, we often > > have > > > > the cases where the api of producers/consumers changes and the bugs > are > > > > only fixed in the newer components. Also, many people who are on the > > > older > > > > versions treat the latest release of that series is the stable one > and > > > end > > > > up with bugs in production. > > > > > > > > I can volunteer for the release management of the LTS release but > as a > > > > community, can we follow the rigour of back-porting the bug-fixes to > > the > > > > LTS branch? > > > > > > > > -- > > > > Regards > > > > Vamsi Subhash > > > > > > > > > > > > > > > > -- > > > Regards > > > Vamsi Subhash > > > > > > > > > -- > Regards > Vamsi Subhash >
Re: best python library to use?
kafka-python (https://github.com/dpkp/kafka-python) has also just merged performance improvements to the consumer in https://github.com/dpkp/kafka-python/issues/290 which should see a pretty decent boost in throughput. We were somewhat put off by the poor performance in earlier versions, I imagine many people would have been in the same position so it's worth revisiting. Sam Pegler WEBOPS ENGINEER T. +44(0) 07 562 867 486 [image: Infectious Media]3-7 Herbal Hill / London / EC1R 5EJwww.infectiousmedia.com [image: Infectious Media] <http://www.infectiousmedia.com/>[image: Facebook] <http://www.facebook.com/infectiousmedia>[image: Twitter] <https://twitter.com/infectiousmedia>[image: LinkedIn] <http://www.linkedin.com/company/infectious-media-ltd>[image: Youtube] <http://www.youtube.com/user/InfectiousMediaLtd> This email and any attachments are confidential and may also be privileged. If youare not the intended recipient, please notify the sender immediately, and do notdisclose the contents to another person, use it for any purpose, or store, or copythe information in any medium. Please also destroy and delete the message fromyour computer. On 11 January 2016 at 16:28, Andrew Otto <o...@wikimedia.org> wrote: > pykafka’s balanced consumer is very useful. pykafka also has Python > bindings to the librdkafka C library that you can optionally enable, which > might get you some speed boosts. > > python-kafka (oh, I just saw this 0.9x version, hm!) was better at > producing than pykafka for us, so we am currently using pykafka for > consumption, and python-kafka for production. python-kafka allows you to > produce to multiple topics using the same client instance. (pykafka may > support this soon: https://github.com/Parsely/pykafka/issues/354) > > > > On Sat, Jan 9, 2016 at 10:04 AM, Dana Powers <dana.pow...@gmail.com> > wrote: > > > pykafka uses a custom zookeeper implementation for consumer groups. > > kafka-python uses the 0.9.0.0 server apis to accomplish the same. > > > > -Dana > > On Jan 8, 2016 18:32, "chengxin Cai" <ia...@outlook.com> wrote: > > > > > Hi > > > > > > I heard that Pykakfa can create a balanced consumer. > > > > > > And there should be no other big difference. > > > > > > > > > Best Regards > > > > > > > 在 2016年1月9日，08:58，Dana Powers <dana.pow...@rd.io> 写道： > > > > > > > > Hi Doug, > > > > > > > > The differences are fairly subtle. kafka-python is a community-backed > > > > project that aims to be consistent w/ the official java client; > pykafka > > > is > > > > sponsored by parse.ly and aims to provide a pythonic interface. > > > whichever > > > > you go with, I would love to hear your specific feedback on > > kafka-python. > > > > > > > > -Dana (kafka-python maintainer) > > > > > > > >> On Fri, Jan 8, 2016 at 4:32 PM, Doug Tomm <dct...@gmail.com> wrote: > > > >> > > > >> we're using kafka-python, weighing pykafka, and wondering if there's > > > >> another that is bettor to use. does confluent endorse or recommend > a > > > >> particular python package (psorry for the alliteration)? > > > >> > > > >> doug > > > >> > > > >> > > > > > >
