Re: [kafka-clients] [VOTE] 0.8.2.0 Candidate 2

All links are pointing to https://people.apache.org/~junrao/kafka-0.8.2.0-candidate1/. They should be https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/ right? On Tue, Jan 20, 2015 at 8:32 AM, Jun Rao wrote: > This is the second candidate for release of Apache Kafka 0.8.2.0. There > ha

Re: How to handle broker disk failure

Is it possible to continue to server topics from the remaining disks while waiting for a replacement disk or will the broker exit/stop working. (we would like to be able to replace disks in a relaxed manner since we have the datacenter colocated and we don't have permanent staff there since there i

Isr difference between Metadata Response vs /kafka-topics.sh --describe

We are running an external (like in non supported) C++ client library agains 0.8.2-rc2 and see differences in the Isr vector in Metadata Response compared to what ./kafka-topics.sh --describe returns. We have a triple replicated topic that is not updated during the test. kafka-topics.sh returns

Receive Facebook real time updates into Kafka

im pretty new to kafka and i'm wondering if i'm heading in the right direction. I want to recive Facebook real time update [facebook subscriptions][1] into kafka . To get the data back from Facebook you need to provide URL that you will be receiving the data back. So i figured out the the bes

Re: Isr difference between Metadata Response vs /kafka-topics.sh --describe

Sounds like you are bumping into this https://issues.apache.org/jira/browse/KAFKA-1367 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: How to handle broker disk failure

same situation with us. we run jbod and actually dont replace the failed data disks at all. we simply keep boxes running until non-failed drives falls below some threshold. so our procedure with kafka would be: 1) ideally kafka server simply survives failed disk and keeps going, and fixes itself wi

Re: Isr difference between Metadata Response vs /kafka-topics.sh --describe

thanks, svante 2015-01-21 16:30 GMT+01:00 Joe Stein : > Sounds like you are bumping into this > https://issues.apache.org/jira/browse/KAFKA-1367 > >

Re: How to handle broker disk failure

Yes, what you can do is to exclude the bad disk from log.dirs and then restart the broker. The missing data will be automatically copies over. This is likely cheaper than reassigning partitions. Thanks, Jun On Wed, Jan 21, 2015 at 7:49 AM, Koert Kuipers wrote: > same situation with us. we run

Re: [kafka-clients] [VOTE] 0.8.2.0 Candidate 2

Yes, will send out a new email with the correct links. Thanks, Jun On Wed, Jan 21, 2015 at 3:12 AM, Manikumar Reddy wrote: > All links are pointing to > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate1/. > They should be https://people.apache.org/~junrao/kafka-0.8.2.0-candidate2/ > r

Inter Mirror maker processes offset sync

Hi all, I have two mirror maker processes running on two different machines fetching messages from same topic from one data center to another data center. These two processes are assigned to the same consumer group. If I want no data loss or data duplication even when one of the mirror maker proce

Re: [kafka-clients] [VOTE] 0.8.2.0 Candidate 2

Also Maven artifacts link is not correct On Wed, Jan 21, 2015 at 9:50 PM, Jun Rao wrote: > Yes, will send out a new email with the correct links. > > Thanks, > > Jun > > On Wed, Jan 21, 2015 at 3:12 AM, Manikumar Reddy > wrote: > >> All links are pointing to >> https://people.apache.org/~junrao

Re: [kafka-clients] [VOTE] 0.8.2.0 Candidate 2

Is it? You just need to navigate into org, then apache, then kafka, etc. Thanks, Jun On Wed, Jan 21, 2015 at 8:28 AM, Manikumar Reddy wrote: > Also Maven artifacts link is not correct > > On Wed, Jan 21, 2015 at 9:50 PM, Jun Rao wrote: > >> Yes, will send out a new email with the correct link

[VOTE] 0.8.2.0 Candidate 2 (with the correct links)

This is the second candidate for release of Apache Kafka 0.8.2.0. There has been some changes since the 0.8.2 beta release, especially in the new java producer api and jmx mbean names. It would be great if people can test this out thoroughly. Release Notes for the 0.8.2.0 release https://people.ap

Re: [kafka-clients] [VOTE] 0.8.2.0 Candidate 2

Ok, got it. Link is different from Release Candidate 1. On Wed, Jan 21, 2015 at 10:01 PM, Jun Rao wrote: > Is it? You just need to navigate into org, then apache, then kafka, etc. > > Thanks, > > Jun > > On Wed, Jan 21, 2015 at 8:28 AM, Manikumar Reddy > wrote: > >> Also Maven artifacts link i

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

For the maven artifacts listed above, there doesn't seem to be any distinction between rc1 or rc2, so is it assumed that this is release2 here?: https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.11/0.8.2.0/ Thanks! Jason On Wed, Jan 21, 2015 at 8:28 AM, Jun Rao wrot

Re: Inter Mirror maker processes offset sync

Hi Tao, You may want to take a look at the current progress we are making at MM: https://issues.apache.org/jira/browse/KAFKA-1650 which makes a zero data-loss MM, and: https://issues.apache.org/jira/browse/KAFKA-1839 attempts to remove possible duplicates at MM. Guozhang On Wed, Jan 21, 2015

Re: Leadership rebalance causing drop of incoming messages

After a closer look to other metrics and broker logs, we found that the instability of the broker cluster might have been caused by a snappy un-compression error. In our case, the consumer and producer happens to be the same application so restarting the client made the recovery of the broker clust

Re: [VOTE] 0.8.2.0 Candidate 2 (with the correct links)

That's right. What's in maven staging is always for the latest RC being voted. The stuff in maven staging will be promoted to maven central once the vote passes. Thanks, Jun On Wed, Jan 21, 2015 at 10:35 AM, Jason Rosenberg wrote: > For the maven artifacts listed above, there doesn't seem to b

Kafka Producers and Virtual Kafka Endpoint

Hello Kafka Users I am new to Kafka and am evaluating it for using it for our high throughput messaging needs. I tested my producers using two different approaches. (a) metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092 And (b) metadata.broker.list=vip:9092 Where vip is an ELB (elastic loa

Re: Kafka Producers and Virtual Kafka Endpoint

<< My goal is to be able to hide the underlying hosts from the producer and be able to use a virtual endpoint (like the vip). That is a typical approach and good to be done if you can for the reasons you brought up. The first call of a producer is to "bootstrap" itself with the list of kafka broke

Re: Updated Logstash-Kafka plugin

Please go ahead, Jun. Thanks! Joe Lawson From: Jun Rao Sent: Tuesday, January 20, 2015 8:39 PM To: users@kafka.apache.org Subject: Re: Updated Logstash-Kafka plugin Joe, Thanks for sharing. Do you want to add a link to our wiki ( https://cwiki.apache.o

Partition reassignment doesn't work on few partitions

Hello, I have a 6 node kafka cluster (0.8.0) where partition reassignment doesn't seem to work on a few partitions. This happens within the same, as well as across other topics. Following is the behavior observed : 1. For a successful reassignment, the kafka-reassign-partitions.sh returns success

Re: Leadership rebalance causing drop of incoming messages

> instability of the broker cluster might have been caused by a snappy > un-compression error. In our case, the consumer and producer happens to be > the same application so restarting the client made the recovery of the > ... > The un-compression error is likely to be caused by incompatible snappy

Re: Receive Facebook real time updates into Kafka

FB will require specific endpoints to POST data to (and it looks like it also requires a GET endpoint that it can query to validate your service), and the data won't be in the format expected by the Dropwizard Kafka API. The amount of work required in implementing a simple REST API to do this yours

Re: Leadership rebalance causing drop of incoming messages

> > It is unclear to me why restarting clients would fix the compatibility > issue. Or do you mean you bounced in the right version of the snappy > jar? > Also, what version of the broker are you on? We restarted the client with the compression turned off and that fixed the problem. We are using

Re: Partition reassignment doesn't work on few partitions

Any error in the controller and state-change log? Also, you may want to upgrade to 0.8.1, which fixed some reassignment issues. Thanks, Jun On Wed, Jan 21, 2015 at 12:38 PM, Raghu Udiyar wrote: > Hello, > > I have a 6 node kafka cluster (0.8.0) where partition reassignment doesn't > seem to wo

Re: How to setup inter DC replication in Kafka 0.8.1.1

Hi Madhukar, We are currently working on improving MM to avoid data loss / duplicates, details can be found in KAFKA-1650 / 1839 / 1840. For your concern, by saying "MM killed" do you mean it is shutdown or hard-failed / crashed that can cause duplicates? Guozhang On Mon, Jan 19, 2015 at 10:51

using the new logstash-kafka plugin

We are starting to use the new logstash-kafka plugin, and I am wondering if it is possible to read multiple topics? Or do you need to create separate logstashes for each topic to parse? We are consuming multi-line logs from a service running on a bunch of different hosts, so we address that by cre

Re: Partition reassignment doesn't work on few partitions

No errors in the state-change log or the controller. Its as if the controller never got the request for that partition. Regarding the upgrade, we did upgrade one of the nodes, and initiate the replication. Here, the controller at 0.8.0 and this node at 0.8.1.1. In this case, when we initiated the

Re: How to setup inter DC replication in Kafka 0.8.1.1

Thanks Guozhang for your reply. Checked the details as mentioned above. Issue KAFKA-1650 has been fixed but it will come with 0.8.3. :-( By "MM Killed", it is hard-failed(kill -9 )/crashed due to system reboot or any other reason. In that case some of the messages get duplicated. Can you please