Re: [ANNOUNCE] New committer: Matthias J. Sax

Congratulations Matthias. Very well deserved! On Fri, Jan 12, 2018 at 7:11 PM Manikumar wrote: > Congrats. well deserved. > > On Sat, Jan 13, 2018 at 8:37 AM, Martin Gainty > wrote: > > > Willkommen Matthias! > > Martin- > >

Re: [ANNOUNCE] New committer: Damian Guy

Well deserved. Congratulations Damian! On Fri, Jun 9, 2017 at 1:34 PM Guozhang Wang wrote: > Hello all, > > > The PMC of Apache Kafka is pleased to announce that we have invited Damian > Guy as a committer to the project. > > Damian has made tremendous contributions to

Re: [VOTE] KIP-156 Add option "dry run" to Streams application reset tool

+1 On Wed, May 10, 2017 at 12:32 PM Gwen Shapira wrote: > +1. Also not sure that adding a parameter to a CLI requires a KIP. It seems > excessive. > > > On Tue, May 9, 2017 at 7:57 PM Jay Kreps wrote: > > > +1 > > On Tue, May 9, 2017 at 3:41 PM BigData dev

Re: [VOTE] 0.10.2.1 RC1

+1 (binding). Verified signatures, ran quickstart and tests. On Fri, Apr 14, 2017 at 3:45 PM Gwen Shapira wrote: > Verified my own signatures, ran quickstart and created few Connectors. > > +1 (binding) > > > On Wed, Apr 12, 2017 at 5:25 PM, Gwen Shapira

Re: [ANNOUNCE] New committer: Grant Henke

Congratulations, Grant. Well deserved! On Thu, Jan 12, 2017 at 7:51 AM Grant Henke wrote: > Thanks everyone! > > On Thu, Jan 12, 2017 at 2:58 AM, Damian Guy wrote: > > > Congratulations! > > > > On Thu, 12 Jan 2017 at 03:35 Jun Rao

Re: [kafka-clients] [VOTE] 0.10.1.0 RC3

+1 (binding) Verified quick start and artifacts. On Mon, Oct 17, 2016 at 10:39 PM Dana Powers wrote: > +1 -- passes kafka-python integration tests > > On Mon, Oct 17, 2016 at 1:28 PM, Jun Rao wrote: > > Thanks for preparing the release. Verified quick

[ANNOUNCE] New committer: Jason Gustafson

The PMC for Apache Kafka has invited Jason Gustafson to join as a committer and we are pleased to announce that he has accepted! Jason has contributed numerous patches to a wide range of areas, notably within the new consumer and the Kafka Connect layers. He has displayed great taste and

Re: [VOTE] 0.10.0.1 RC2

+1 (binding) On Fri, Aug 5, 2016 at 12:29 PM, Grant Henke wrote: > +1 (non-binding) > > On Fri, Aug 5, 2016 at 2:04 PM, Dana Powers wrote: > > > passed kafka-python integration tests, +1 > > > > -Dana > > > > > > On Fri, Aug 5, 2016 at 9:35 AM, Tom

[ANNOUNCE] New committer: Ismael Juma

The PMC for Apache Kafka has invited Ismael Juma to join as a committer and we are pleased to announce that he has accepted! Ismael has contributed 121 commits to a wide range of areas, notably within the security and the network layer. His

Re: 0.9.0.1 RC1

+1 (binding). Verified source and binary artifacts, ran ./gradlew testAll, quick start on source artifact and Scala 2.11 binary artifact. On Mon, Feb 15, 2016 at 7:43 PM, Ewen Cheslack-Postava wrote: > Yeah, I saw > > kafka.network.SocketServerTest > tooBigRequestIsRejected

Re: 0.9.0.0 RC4

+1 (binding). Verified source and binary artifacts, ran unit tests. On Mon, Nov 23, 2015 at 9:32 AM, Jun Rao wrote: > I updated the release notes. Since this doesn't affect the release > artifacts to be voted upon, we don't have to do another RC. > > Please vote by 6pm PT

Re: [DISCUSSION] Kafka 0.8.2.2 release?

How about looking at the scope for the 0.8.3 release first before we cut yet another point release off of 0.8.2.2? Each release includes some overhead and if there is a larger release in the works, it might be worth working on getting that. My take is that the 2 things the community has been

Help Us Nominate Apache Kafka for a 2015 Bossie (Best of OSS) Award - Due June 30th

Hello Kafka community members, We appreciate your use and support of Kafka and all the feedback you’ve provided to us along the way. If all is still going well with Kafka and you’re realizing great value from it, we’d like your support in nominating Kafka for a 2015 InfoWorld Bossie award, which

Re: [ANNOUNCEMENT] Apache Kafka 0.8.2.1 Released

Thanks for driving this Jun and everyone for the contributions! On Wed, Mar 11, 2015 at 12:01 PM, Jun Rao jun...@apache.org wrote: The Apache Kafka community is pleased to announce the release for Apache Kafka 0.8.2.1. The 0.8.2.1 release fixes 4 critical issues in 0.8.2.0. All of the

Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 2

+1. Verified quick start, unit tests. On Tue, Mar 3, 2015 at 12:09 PM, Joe Stein joe.st...@stealth.ly wrote: Ok, lets fix the transient test failure on trunk agreed not a blocker. +1 quick start passed, verified artifacts, updates in scala

Re: Kafka Poll: Version You Use?

Thanks for running the poll and sharing the results! On Wed, Mar 4, 2015 at 8:34 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, You can see the number of voters in the poll itself (view poll results link in the poll widget). Audience details unknown, but the poll was posted on:

Re: Trying to get kafka data to Hadoop

Thanks Jagat for the callout! Confluent Platform 1.0 http://confluent.io/product/ includes Camus and we were happy to address any questions in our community mailing list confluent-platf...@googlegroups.com. On Wed, Mar 4, 2015 at 8:41 PM, max square max2subscr...@gmail.com wrote: Thunder,

Announcing the Confluent Platform built on Apache Kafka

Folks, We, at Confluent http://confluent.io, are excited to announce the release of Confluent Platform 1.0 built around Apache Kafka - http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0/ We also published a detailed two-part guide on how you can put Kafka to use in your

Re: About Symantec's encryption-thru-Kafka proof of concept

Great. Thanks for sharing! On Thu, Feb 19, 2015 at 8:51 PM, Jim Hoagland jim_hoagl...@symantec.com wrote: Hi Folks, At the recent Kafka Meetup in Mountain View there was interest expressed about the encryption through Kafka proof of concept that Symantec did a few months ago, so I have

Re: Kafka Leader Rebalance

Are you are on 0.8.1.1 or higher and are you using the kafka-topics tool to see which leaders are on which brokers? Did you also wait for more than 15 seconds to see if the leader election takes place? Are there any errors in the controller log? On Sat, Feb 14, 2015 at 9:35 AM, nitin sharma

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 3

Great! Thanks Jun for helping with the release and everyone involved for your contributions. On Mon, Feb 2, 2015 at 1:32 PM, Joe Stein joe.st...@stealth.ly wrote: Huzzah! Thanks Jun for preparing the release candidates and getting this out to the community. - Joe Stein On Mon, Feb 2,

Re: [VOTE] 0.8.2.0 Candidate 3

+1 (binding). Verified quickstart and unit tests ran ok. On Sun, Feb 1, 2015 at 9:11 AM, Steven Wu stevenz...@gmail.com wrote: In Netflix, we have been using route53 DNS name as bootstrap servers in AWS env. Basically, when a kafka broker start, we add it to route53 DNS name for the cluster.

Re: [kafka-clients] Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

+1 (binding) Verified keys, quick start, unit tests. On Sat, Jan 24, 2015 at 4:26 PM, Joe Stein joe.st...@stealth.ly wrote: That makes sense, thanks! On Sat, Jan 24, 2015 at 7:00 PM, Jay Kreps jay.kr...@gmail.com wrote: But I think the flaw in trying to guess what kind of serializer they

Re: Release Date for Kafka 0.8.2

You can track the blockers here https://issues.apache.org/jira/browse/KAFKA-1841?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.8.2%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Blocker%20ORDER%20BY%20key%20DESC. We are waiting on follow up patches for 2 JIRAs which are

Re: NPE in debug logging statement in kafka new producer

0.8.2 official release will have generic Producer interface? On Sun, Dec 28, 2014 at 11:57 AM, Neha Narkhede neha.narkh...@gmail.com wrote: This is a bug. Would you mind filing a JIRA? Also feel free to upload a patch. On Sat, Dec 27, 2014 at 7:25 PM, Bae, Jae Hyeon metac...@gmail.com

Re: Kafka 0.8.2 release - before Santa Claus?

the votes and a stable version. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop / On Dec 1, 2014 7:56 PM, Neha

Re: leader and isr were not set when create the topic

It seems that somehow the follower lost its highwatermark checkpoint file. Can you share the steps to reproduce this along with the Kafka versions you are using? On Mon, Dec 22, 2014 at 4:17 PM, Sa Li sal...@gmail.com wrote: I have three nodes: 100, 101, and 102 When I restart all of them,

Re: NPE in debug logging statement in kafka new producer

This is a bug. Would you mind filing a JIRA? Also feel free to upload a patch. On Sat, Dec 27, 2014 at 7:25 PM, Bae, Jae Hyeon metac...@gmail.com wrote: Hi While I am testing kafka java producer, I saw the following NPE SLF4J: Failed toString() invocation on an object of type

Re: Kafka consumer session timeouts

Terry, The zookeeper client used by the high level Kafka consumer has a separate thread that does the heartbeat in the background. So even if it takes long to process the message, it should not make the consumer's session to time out or make the consumer rebalance. You may be running into long GC

Re: leader and isr were not set when create the topic

There is possibly some error in your broker logs. Can you check if you see any and send it around? On Mon, Dec 22, 2014 at 2:46 PM, Sa Li sal...@gmail.com wrote: Hi, All I created a topic with 3 replications and 6 partitions, but when I check this topic, seems there is no leader and isr were

Re: Rebalance not happening even after increasing max retries causing conflict in ZK

, Nov 10, 2014 at 8:46 AM, Neha Narkhede neha.narkh...@gmail.com wrote: A rebalance should trigger on all consumers when you add a new consumer to the group. If you don't see the zookeeper watch fire, the consumer may have somehow lost the watch. We have seen this behavior on older zk versions

Re: can't produce message in kafka production

The producer is complaining that it's socket channel is already closed. Which makes me think it was closed due to some error that is not present in your logs. I'd enable DEBUG and see if that shows the cause. On Thu, Dec 18, 2014 at 4:13 PM, Gwen Shapira gshap...@cloudera.com wrote: Perhaps you

Re: Kafka Topic/Partition Reassignment issue

The reassignment tool outputs the original assignment before executing the next one. If you have that saved, you can initiate another assignment to make it go to its initial state. That is probably a safer way to fix the reassignment. On Wed, Dec 17, 2014 at 3:12 PM, Salman Ahmed

Re: Built-in replication for broker

Replication is available in Kafka 0.8.0 onwards. On Tue, Dec 16, 2014 at 4:31 PM, Haoming Zhang haoming.zh...@outlook.com wrote: Dear developers, In the paper Kafka: A Distributed Messaging System for Log Processing, Jay Kreps, Neha Narkhede, Jun Rao from LinkedIn, at NetDB workshop 2011

Re: getOffsetsBefore and latest

what about getOffsetsBefore using kafka.api.OffsetRequest.LatestTime? am i safe to assume this returns me truly the most recent offset + 1 for each partition? That's right. On Mon, Dec 15, 2014 at 7:18 AM, Koert Kuipers ko...@tresata.com wrote: i read in several places that

Re: kafka producer consumer problem

Hi, For the producer, we have moved to a new producer API (under org.apache.kafka.clients.producer.KafkaProducer). Please feel free to give that a spin and report any issues that you see. I think the consumer issue you reported is being discussed in another thread and is fixed in 0.8.2. Can you

Re: kafka async producer takes a lot of cpu

dropping so many messages. On Sun, Dec 14, 2014 at 4:58 PM, Neha Narkhede n...@confluent.io wrote: Thanks for reporting the issue, Rajiv. Since we are actively phasing out the old client, it will be very helpful to know what the behavior on the new client is. On Fri, Dec 12, 2014 at 8:12

Re: Number of Consumers Connected

In addition to Gwen's suggestion, we actually don't have jmx metrics that give you a list of actively consuming processes. On Mon, Dec 15, 2014 at 12:59 PM, Gwen Shapira gshap...@cloudera.com wrote: Currently you can find the number of consumer groups through ZooKeeper: connect to ZK and run

Re: How to Setup MirrorMaker in Generalized way

[ *kill -9* ]. Is there any solution to avoid this duplicated entry in target cluster? I am using Kafka *8.1.1.* On Mon, Dec 8, 2014 at 11:17 PM, Neha Narkhede n...@confluent.io wrote: Hi Madhukar, From the same documentation link you referred to - The source and destination clusters

Re: Reading only the latest message

, kafka.api.OffsetRequest.*CurrentVersion*(), clientName); OffsetResponse response = consumer.getOffsetsBefore(request); *if*(readOffset != 0) readOffset--; else handle this case… *From:* Neha Narkhede [mailto:n...@confluent.io] *Sent:* Monday, December 08, 2014 12:43 PM

Re: Topic config reset on alter

Thanks for reporting the bug. Would you mind filing a JIRA https://issues.apache.org/jira/browse/KAFKA/?selectedTab=com.atlassian.jira.jira-projects-plugin:issues-panel ? On Tue, Dec 9, 2014 at 12:46 PM, Andrew Jorgensen ajorgen...@twitter.com.invalid wrote: I am using kafka 8.1.1 If you run

Re: Increased CPU usage with 0.8.2-beta

mathias.soederb...@gmail.com wrote: Hi Neha, Yeah sure. I'm not familiar with hprof, so any particular options I should include or just run with defaults? Best regards, Mathias On Mon Dec 08 2014 at 7:41:32 PM Neha Narkhede n...@confluent.io wrote: Thanks for reporting the issue. Would

Re: How to raise a question in forum

You can post it here :-) On Tue, Dec 9, 2014 at 1:26 PM, nitin sharma kumarsharma.ni...@gmail.com wrote: HI, I need some help related to Kafka Consumer Lag monitoring.. Kindly let me know how can i post my query to the forum. Regards, Nitin Kumar Sharma. -- Thanks, Neha

Re: Reading only the latest message

that will be published. If I fetch at that offset I will block until a new message is published to that partition. I am actually trying to read the contents of the latest message in the partition, and based on info in the message resubscribe to the data source. -Original Message- From: Neha

Re: How to Setup MirrorMaker in Generalized way

Hi Madhukar, From the same documentation link you referred to - The source and destination clusters are completely independent entities: they can have different numbers of partitions and the offsets will not be the same. For this reason the mirror cluster is not really intended as a

Re: Increased CPU usage with 0.8.2-beta

Thanks for reporting the issue. Would you mind running hprof and sending the output? On Mon, Dec 8, 2014 at 1:25 AM, Mathias Söderberg mathias.soederb...@gmail.com wrote: Good day, I upgraded a Kafka cluster from v0.8.1.1 to v0.8.2-beta and noticed that the CPU usage on the broker machines

Re: kafka consumer to write into DB

a hash map in memory, load data into memory and bulk copy to DB instead of insert into DB line by line. Does it mean I need to ack each message while load to memory? thanks On Thu, Dec 4, 2014 at 1:21 PM, Neha Narkhede n...@confluent.io wrote: This is specific

Re: Can Mirroring Preserve Every Topic's Partition?

Going back to your previous requirement of ensuring that the data in the target cluster is in the same order as the source cluster, all you need is to specify a key with every record in your data. The mirror maker and its producer takes care of placing all the data for a particular key in the same

Re: Reading only the latest message

You can use the getOffsetsBefore() API and specify -1L to get the offset of the last committed message (at the time of the request) for that partition. On Fri, Dec 5, 2014 at 12:42 PM, Orelowitz, David david.orelow...@baml.com wrote: What is the best mechanism to retrieve the latest message

Re: Broker don't get back when killed and restarted

Have you tried using the latest stable version of Kafka (0.8.1.1) with controlled shutdown? On Fri, Dec 5, 2014 at 2:39 PM, Haeley Yao hae...@quantifind.com wrote: Hi, Kafka group We try to improve the fault-tolerance of kafka cluster. We setup 4 nodes kafka cluster and 3 nodes zookeeper

Re: Can Mirroring Preserve Every Topic's Partition?

As Guozhang mentioned, that will allow exact partitioning but note that this still doesn't ensure identical partitions since your target cluster can have duplicates depending on whether the producer retried sending data or not. On Wed, Nov 26, 2014 at 10:33 AM, Guozhang Wang wangg...@gmail.com

Re: kafka consumer to write into DB

This is specific for pentaho but may be useful - https://github.com/RuckusWirelessIL/pentaho-kafka-consumer On Thu, Dec 4, 2014 at 12:58 PM, Sa Li sal...@gmail.com wrote: Hello, all I never developed a kafka consumer, I want to be able to make an advanced kafka consumer in java to consume

Re: Questions about new consumer API

1. In this doc it says kafka consumer will automatically do load balance. Is it based on throughtput or same as what we have now balance the cardinality among all consumers in same ConsumerGroup? In a real case different partitions could have different peak time. Load balancing is still based on

Re: Questions about new consumer API

under group.id, dose it mean in one group, there should be at most one subscriber for each topic partition? Best, Siyuan On Tue, Dec 2, 2014 at 12:55 PM, Neha Narkhede neha.narkh...@gmail.com wrote: 1. In this doc it says kafka consumer will automatically do load balance. Is it based

Re: Best practice for upgrading Kafka cluster from 0.8.1 to 0.8.1.1

Will doing one broker at a time by brining the broker down, updating the code, and restarting it be sufficient? Yes this should work for the upgrade. On Mon, Dec 1, 2014 at 10:23 PM, Yu Yang yuyan...@gmail.com wrote: Hi, We have a kafka cluster that runs Kafka 0.8.1 that we are considering

Re: Kafka 0.8.2 release - before Santa Claus?

+1 for doing a 0.8.2 final before the December break. On Mon, Dec 1, 2014 at 8:40 AM, Jun Rao jun...@gmail.com wrote: We are currently discussing a last-minute API change to the new java producer. We have also accumulated a few more 0.8.2 blockers.

Re: logging agent based on fuse and kafka: first release

Great. Thanks for sharing. I added it to our ecosystem https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem wiki. On Tue, Nov 25, 2014 at 9:58 AM, yazgoo yaz...@gmail.com wrote: Hi, First I'd like to thank kafka developers for writing kafka. This is an announcement for the first

Re: Broker keeps rebalancing

timeout entries (these are not ERROR entries). Guozhang Guozhang On Wed, Nov 12, 2014 at 5:31 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Does this help? https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog ? On Wed, Nov 12

Re: Add partitions with replica assignment in same command

When we first add partitions, will it change the assignment of replicas for existing partitions? Nope. It should not touch the existing partitions. Also, will there be any issues executing the second reassignment command which will change the assignment again for the new partitions added? No.

Re: Broker keeps rebalancing

Does this help? https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog ? On Wed, Nov 12, 2014 at 3:53 PM, Chen Wang chen.apache.s...@gmail.com wrote: Hi there, My kafka client is reading a 3 partition topic from kafka with 3 threads distributed on

Re: expanding cluster and reassigning parititions without restarting producer

a trigger fired when partitions are added or does the producer refresh some cache every some given time period? Shlomi On Tue, Nov 11, 2014 at 4:25 AM, Neha Narkhede neha.narkh...@gmail.com wrote: How can I auto refresh keyed producers to use new partitions as these partitions are added

Re: expanding cluster and reassigning parititions without restarting producer

this situation in small scale... maybe 3 brokers, one topic with one partition and then add partitions. we'll see. thanks for clarifying. Oh, Good luck with Confluent!! :) On Mon, Nov 10, 2014 at 4:17 AM, Neha Narkhede neha.narkh...@gmail.com wrote: The producer might get

Re: expanding cluster and reassigning parititions without restarting producer

The producer might get an error code if the leader of the partitions being reassigned also changes. However it should retry and succeed. Do you see a behavior that suggests otherwise? On Sat, Nov 8, 2014 at 11:45 PM, Shlomi Hazan shl...@viber.com wrote: Hi All, I recently had an issue

Re: Interrupting controlled shutdown breaks Kafka cluster

We fixed a couple issues related to automatic leader balancing and controlled shutdown. Would you mind trying out 0.8.2-beta? On Fri, Nov 7, 2014 at 11:52 AM, Solon Gordon so...@knewton.com wrote: We're using 0.8.1.1 with auto.leader.rebalance.enable=true. On Fri, Nov 7, 2014 at 2:35 PM,

Re: Rebalance not happening even after increasing max retries causing conflict in ZK

the ephemeral node which was the case last time. Just to note that the ephemeral node conflict which we were seeing earlier also appeared after rebalance failed. My hunch is that fixing the rebalance failure will fix that issue as well. -Thanks, Mohit On Mon, Oct 20, 2014 at 7:48 PM, Neha

Re: Strategies for high-concurrency consumers

Jack, Zookeeper is likely the bottleneck if rebalancing takes a very long time. As Jay said, this will be addressed in the consumer rewrite planned for 0.9. Few more workarounds that were tried at LinkedIn - 1) To deploy Zookeeper on SSDs and 2) Turning sync on every write off

Re: High CPU usage of Crc32 on Kafka broker

Allen, Apache mailing lists don't allow attachments. Could you please link to a pastebin or something? Thanks, Neha On Thu, Nov 6, 2014 at 12:02 PM, Allen Wang aw...@netflix.com.invalid wrote: After digging more into the stack trace got from flight recorder (which is attached), it seems that

Re: nulls found in topic, created by recovery?

IIRC, the bug that introduced the nulls was related to compressed data. Is this topic compressed? Did you try to run a consumer through the topic's data or alternately the DumpLogSegments tool? On Thu, Nov 6, 2014 at 12:56 PM, Neil Harkins nhark...@gmail.com wrote: Hi all. I saw something weird

Re: Interaction of retention settings for broker and topic plus partitions

To clarify though, is it correct that a per topic limit will always override the default limit of the same type? (e.g. a large per-topic retention hours vs. a small default retention hours)? That's correct. On Thu, Nov 6, 2014 at 9:34 AM, Jason Rosenberg j...@squareup.com wrote: Jun, To

Re: Kafka Release timelines

0.8.2 should be available in a month. Though 0.9 might take a couple more months and there is a good chance that it will not be this year. Thanks, Neha On Thu, Nov 6, 2014 at 3:01 AM, dinesh kumar dinesh...@gmail.com wrote: Hi, I found the future release plan wiki here

Re: Consumer lag keep increasing

Chen, Consumers lag either due to an I/O or network bottleneck or due to slow processing of messages by the user. To confirm that you are not hitting the latter issue, you can run a console consumer on the same data and observe the throughput that it provides and it's lag. Thanks, Neha On Wed,

Re: Kafka Cluster disaster decovery

A common solution for disaster recovery is to mirror the Kafka cluster into another one deployed in a separate data center. The mirroring is not synchronous so there might be some message loss when you lose the entire cluster in some disaster. Thanks, Neha On Mon, Nov 3, 2014 at 7:43 AM,

Re: corrupt message

This may be due to a bug in the client. Non-java Kafka clients are maintained by the individual client owners. You might want to ping the owner of your library directly. On Mon, Nov 3, 2014 at 7:21 AM, Fredrik S Loekke f...@lindcapital.com wrote: When running a C# producer against a kafka

Re: Dynamically adding Kafka brokers

I agree that KAFKA-1070 would be great to get in. I especially felt the need for something like this while using a few other systems that automated the port, id etc to give a good OOTB experience. Sorry, I lost track of the review. Will do so in the next few days. Thanks, Neha On Mon, Nov 3,

Re: [ANNOUNCEMENT] Apache Kafka 0.8.2-beta Released

Also the 0.9 consumer javadoc is here - http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/org/apache/kafka/clients/consumer/KafkaConsumer.html Thanks, Neha On Thu, Oct 30, 2014 at 5:00 AM, Joe Stein joe.st...@stealth.ly wrote: Hey, yeah! For the new producer

Re: Consumer cannot find leader after leadership changes on broker side

Which Kafka version are you on for the consumer? We recently fixed a NPE related to ZkClient, could you give 0.8.2-beta a try on your consumer and retry the test? On Wed, Oct 29, 2014 at 10:34 AM, Allen Wang allenxw...@gmail.com wrote: After executing PreferredReplicaLeaderElectionCommand on

Re: High Level Consumer Iterator IllegalStateException Issue

and this seems to cause some issue with java.lang. IllegalStateException: Iterator is in failed state*. Please let me know your finding and recommendation. Thanks, Bhavesh On Mon, Oct 27, 2014 at 6:24 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Sometime it give following exception

Re: High Level Consumer Iterator IllegalStateException Issue

) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) Thanks, Bhavesh On Sun, Oct 26, 2014 at 3:14 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Can you provide the steps to reproduce this issue? On Fri, Oct 24, 2014 at 6:11 PM, Bhavesh Mistry

Re: High Level Consumer Iterator IllegalStateException Issue

Can you provide the steps to reproduce this issue? On Fri, Oct 24, 2014 at 6:11 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: I am using one from the Kafka Trunk branch. Thanks, Bhavesh On Fri, Oct 24, 2014 at 5:24 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Which version

Re: kafka.common.ConsumerRebalanceFailedException

Can you see if the workarounds mentioned here https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped,why? for failed rebalance attempts is useful? On Fri, Oct 24, 2014 at 11:22 AM, Mohit Anchlia mohitanch...@gmail.com wrote: I am seeing following exception, don't

Re: High Level Consumer Iterator IllegalStateException Issue

Which version of Kafka are you using on the consumer? On Fri, Oct 24, 2014 at 4:14 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: HI Kafka Community , I am using kafka trunk source code and I get following exception. What could cause the iterator to have FAILED state. Please let me

Re: Example of using simple consumer to fetch from multiple partitions of a topic

Is there a better way to do this where I can build one big requests with broker - partition mappings and call a consumer.fetch() with one giant request? Unfortunately, not until the 0.9 consumer is released. Until then, you can take a look at ZookeeperConsumerConnector on how it manages multiple

Re: Reusable consumer across consumer groups

to reproduce https://issues.apache.org/jira/browse/KAFKA-1716 It would be better if one could reuse a (pool of) Consumer instances, and through a read operation parameter specify for which consumer group should read be performed. Kind regards, Stevo Slavic. On Tue, Oct 14, 2014 at 6:17 PM, Neha

Re: Kafka 0.9

The new consumer development hasn't started yet. But we have a very detailed design doc https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design and JIRA https://issues.apache.org/jira/browse/KAFKA-1326 plan, if you'd like to contribute. The new client will be protocol

Re: 0.8.1.2

: at the time I thought it was a good idea but if I understand correctly what Jun is saying is that 0.8.1.2 will not happen. I assume Jun sees 0.8.2 coming soon enough to remove any added value from 0.8.1.2. Shlomi On Wed, Oct 22, 2014 at 5:21 AM, Neha Narkhede neha.narkh...@gmail.com wrote: Shlomi

Re: frequent periods of ~1500 replicas not in sync

Neil, We fixed a bug related to the BadVersion problem in 0.8.1.1. Would you mind repeating your test on 0.8.1.1 and if you can still reproduce this issue, then send around the thread dump and attach the logs to KAFKA-1407? Thanks, Neha On Tue, Oct 21, 2014 at 11:56 AM, Neil Harkins

Re: Errors after reboot on single node setup

Can you provide steps to reproduce this? I'm not sure I understand how you run into this. It does look like a bug. On Wed, Oct 22, 2014 at 9:55 AM, Ciprian Hacman ciprian.hac...@sematext.com wrote: Hi, First of all, I am new to Kafka and more of a user than a developer. I will try to

Re: How many partition can one single machine handle in Kafka?

In my experience, RAID 10 doesn't really provide value in the presence of replication. When a disk fails, the RAID resync process is so I/O intensive that it renders the broker useless until it completes. When this happens, you actually have to take the broker out of rotation and move the leaders

Re: Sending Same Message to Two Topics on Same Broker Cluster

/clients/producer/ProducerRecord.html Thanks for your quick response and I appreciate your help. Thanks, Bhavesh On Mon, Oct 20, 2014 at 9:10 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Not really. You need producers to send data to Kafka. On Mon, Oct 20, 2014 at 9:05 PM, Bhavesh

Re: Rebalance not happening even after increasing max retries causing conflict in ZK

Mohit, I wonder if it is related to https://issues.apache.org/jira/browse/KAFKA-1585. When zookeeper expires a session, it doesn't delete the ephemeral nodes immediately. So if you end up trying to recreate ephemeral nodes quickly, it could either be in the valid latest session or from the

Re: taking broker down and returning it does not restore cluster state (nor rebalance)

Did you ensure that your replication factor was set higher than 1? If so, things should recover automatically after adding the killed broker back into the cluster. On Mon, Oct 20, 2014 at 1:32 AM, Shlomi Hazan shl...@viber.com wrote: Hi, Running some tests on 0811 and wanted to see what

Re: How to produce and consume events in 2 DCs?

Another way to set up this kind of mirroring is by deploying 2 clusters in each DC - a local Kafka cluster and an aggregate Kafka cluster. The mirror maker copies data from both the DC's local clusters into the aggregate clusters. So if you want access to a topic with data from both DC's, you

Re: log.cleanup.interval.mins still valid for 0.8.1?

Which example are you referring to? On Mon, Oct 20, 2014 at 7:47 AM, Libo Yu yu_l...@hotmail.com wrote: Hi all, This config property does not appear in the table of broker config properties. But it appears in the example on the Web page. So I wonder if this is still a valid config property

Re: MBeans, dashes, underscores, and KAFKA-1481

of kafka.server:type=BrokerTopicMetrics,name=topic-1-BytesInPerSec we will have kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=topic-1 Thanks, Jun On Thu, Oct 9, 2014 at 11:12 AM, Neha Narkhede neha.narkh...@gmail.com wrote: I am going to vote for 1482 to be included in 0.8.2

Re: 0.8.x = 0.8.2 upgrade - live seamless?

Yes, you should be able to upgrade seamlessly. On Wed, Oct 15, 2014 at 10:07 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Hi, Some of our SPM users who are eager to monitor their Kafka 0.8.x clusters with SPM are asking us whether the upgrade to 0.8.2 from 0.8.1 will be seamless.

Re: Kafka/Zookeeper deployment Questions

In other words, if I change the number of partitions, can I restart the brokers one at a time so that I can continue processing data? Changing the # of partitions is an online operation and doesn't require restarting the brokers. However, any other configuration (with the exception of a few

Re: Broker brought down and under replicated partitions

, Oct 15, 2014 at 6:46 AM, Neha Narkhede neha.narkh...@gmail.com wrote: Which version of Kafka are you using? The current stable one is 0.8.1.1 On Tue, Oct 14, 2014 at 5:51 PM, Jean-Pascal Billaud j...@tellapart.com wrote: Hey Neha, so I removed another broker like 30mn ago

Re: read N items from topic

Josh, The consumer's API doesn't allow you to specify N messages, but you can invoke iter.next() as Gwen suggested and count the messages. Note that the iterator can block if you have less than N messages so you will have to careful design around it. The new consumer's API provides a non blocking

Re: [DISCUSS] Release 0.8.2-beta before 0.8.2?

people want to include any JIRAs (other than the ones mentioned above) in 0.8.2-beta? If so, it will be great to know now so it will allow us to move forward with the beta release quickly. Thanks, Neha On Wed, Oct 15, 2014 at 4:46 PM, Neha Narkhede neha.narkh...@gmail.com wrote: Hi, We have

Re: [Kafka-users] Producer not distributing across all partitions

A topic.metadata.refresh.interval.ms of 10 mins means that the producer will take 10 mins to detect new partitions. So newly added or reassigned partitions might not get data for 10 mins. In general, if you're still at prototyping stages, I'd recommend using the new producer available on kafka

  1   2   3   4   5   6   7   >