Re: Facing issues with Kafka 0.8.1.1 and kafka-reassign-partitions.sh

[Jun Rao]

The failure could mean that the reassignment is still in progress. If you
have lots of data, it may take some time to move the data to new brokers.
You could observe the max lag in each broker to see how far behind new
replicas are (see http://kafka.apache.org/documentation.html#monitoring).

Thanks,

Jun


On Mon, Jul 7, 2014 at 4:42 PM, Florian Dambrine  wrote:

> When I run the tool with the --verify option it says failed for the some
> partitions.
>
> The problem is I do not know if it is a zookeeper issue or if the tool
> really failed.
>
> I faced one time the zookeeper issue (
> https://issues.apache.org/jira/browse/KAFKA-1382) and by killing the
> responsible Kafka the partition switched from failed to completed
> successfully.
>
> What should I do when the Kafka tool says that it failed to move the
> partition?
>
>
>
>
> On Mon, Jul 7, 2014 at 4:33 PM, Clark Haskins
>  > wrote:
>
> > How does it get stuck?
> >
> > -Clark
> >
> > Clark Elliott Haskins III
> > LinkedIn DDS Site Reliability Engineer
> > Kafka, Zookeeper, Samza SRE
> > Mobile: 505.385.1484
> > BlueJeans: https://www.bluejeans.com/chaskins
> >
> >
> > chask...@linkedin.com
> > https://www.linkedin.com/in/clarkhaskins
> > There is no place like 127.0.0.1
> >
> >
> >
> >
> > On 7/7/14, 3:49 PM, "Florian Dambrine"  wrote:
> >
> > >Hi,
> > >
> > >I am trying to add new brokers to an existing 8 nodes Kafka cluster. We
> > >have around 10 topics and the number of partition is set to 50. In order
> > >to
> > >test the reassgin-partitions scripts, I tried on a sandbox cluster the
> > >following steps.
> > >
> > >I developed a script which is able to parse the reassignment partition
> > >plan
> > >given by the Kafka tool in smaller pieces (reassigning maximum 10
> > >partitions at a time).
> > >
> > >Unfortunately I faced some issues with the tool that sometimes get stuck
> > >on
> > >one partition. In this case I have to kill and restart the three Kafkas
> on
> > >which the partition has been relocated to unlock the process (One kafka
> at
> > >a time).
> > >
> > >Moreover, I have also faced these two issues that are already on Jira:
> > >
> > >https://issues.apache.org/jira/browse/KAFKA-1382
> > >https://issues.apache.org/jira/browse/KAFKA-1479
> > >
> > >We really need to add new nodes to our Kafka cluster, does anybody have
> > >already rebalance a Kafka 0.8.1.1? What could you advise me?
> > >
> > >Thanks, and feel free to ask me if you need more details.
> > >
> > >
> > >
> > >--
> > >*Florian Dambrine*  |  Intern, Big Data
> > >*GumGum*   |  *Ads that stick*
> > >209-797-3994  |  flor...@gumgum.com
> >
> >
>
>
> --
> *Florian Dambrine*  |  Intern, Big Data
> *GumGum*   |  *Ads that stick*
> 209-797-3994  |  flor...@gumgum.com
>

Re: kafka.common.FailedToSendMessageException: Failed to send messages after 1 tries.

[Jun Rao]

Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan'tmyconsumers/producersconnecttothebrokers
?

Thanks,

Jun


On Mon, Jul 7, 2014 at 9:18 AM, Kalpa 1977  wrote:

> hi all,
>I am using kafka 0-8-1.
>
> I have created a simple topic test no partition and no replication,
> producer and consumer both works fine running in the same system,
> However, If i call from the remote system as per the below mentioned
> command.
>
> ./bin/kafka-producer-perf-test.sh --broker-list *172.16.229.229*:9092
> --topic test --messages 1 --producer-num-retries 1
>
> it gives error saying.
>
> ERROR Producer connection to ubuntu.01-229:9092 unsuccessful
> (kafka.producer.SyncProducer)
> java.net.ConnectException: Connection refused
> at sun.nio.ch.Net.connect0(Native Method)
>
> In the broker list, I am specifying the IP (*172.16.229.229*), but, why it
> is trying to connect using a host name of itself, and which is not
> correct.I can ping remote ip, and also telnet on port 9092.
>
> Please can anybody help me how to make it to connect to the remote ip.
>
> thanks,
> Kalpa.
>

Re: Largest known Kafka deployment?

[Todd Palino]

Most of the testing predates me, however from archaeological expeditions
into old test servers, we’ve used JBOD configurations at some point in the
past for at least some purposes. I assume that RAID 10 was chosen
specifically for the redundancy (previously, deployments and maintenance
had been much more difficult than it is for us now), combined with the
performance aspects of going with RAID 10 when it comes to reads.

Given the progress we’ve made with automated remediation systems, it would
probably be worth another try to see if we can automate the process and
get better performance. The RAID controller itself is a continual source
of pain (a single reset can throw a cluster into chaos for a bit).

-Todd


On 7/7/14, 8:43 PM, "Bert Corderman"  wrote:

>Thanks for the updated deck.  I had not seen that one yet.  I noticed in
>the preso you are running RAID10 in prod.  Any thoughts of going JBOD?  In
>our testing we saw significant performance improvements.  This of course
>comes with trade off of manual steps if brokers fail.
>
>Bert
>
>On Monday, July 7, 2014, Todd Palino  wrote:
>
>> You¹re out of date, Jun. We¹re up to 20 now :)
>>
>> Our ops presentation on Kafka is a little more up to date on numbers:
>> http://www.slideshare.net/ToddPalino/enterprise-kafka-kafka-as-a-service
>>
>> -Todd
>>
>>
>> On 7/7/14, 7:21 AM, "Jun Rao" > wrote:
>>
>> >LinkedIn's largest Kafka cluster has 16 nodes now. More detailed info
>>can
>> >be found in
>> >
>> 
>>http://www.slideshare.net/Hadoop_Summit/building-a-realtime-data-pipeline
>>-
>> >apache-kafka-at-linkedin?from_search=5
>> >
>> >Thanks,
>> >
>> >Jun
>> >
>> >
>> >On Mon, Jul 7, 2014 at 3:33 AM, Ersin Er > > wrote:
>> >
>> >> Hi,
>> >>
>> >> LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are
>> >>larger
>> >> deployments than LinkedIn's. What's are the largest Kafka deployments
>> >>you
>> >> know of? Any public performance and scalability data published for
>>such
>> >> clusters?
>> >>
>> >> Any pointers would be interesting and helpful.
>> >>
>> >> Regards,
>> >>
>> >> --
>> >> Ersin Er
>> >>
>>
>>

Re: Largest known Kafka deployment?

[Bert Corderman]

Thanks for the updated deck.  I had not seen that one yet.  I noticed in
the preso you are running RAID10 in prod.  Any thoughts of going JBOD?  In
our testing we saw significant performance improvements.  This of course
comes with trade off of manual steps if brokers fail.

Bert

On Monday, July 7, 2014, Todd Palino  wrote:

> You¹re out of date, Jun. We¹re up to 20 now :)
>
> Our ops presentation on Kafka is a little more up to date on numbers:
> http://www.slideshare.net/ToddPalino/enterprise-kafka-kafka-as-a-service
>
> -Todd
>
>
> On 7/7/14, 7:21 AM, "Jun Rao" > wrote:
>
> >LinkedIn's largest Kafka cluster has 16 nodes now. More detailed info can
> >be found in
> >
> http://www.slideshare.net/Hadoop_Summit/building-a-realtime-data-pipeline-
> >apache-kafka-at-linkedin?from_search=5
> >
> >Thanks,
> >
> >Jun
> >
> >
> >On Mon, Jul 7, 2014 at 3:33 AM, Ersin Er  > wrote:
> >
> >> Hi,
> >>
> >> LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are
> >>larger
> >> deployments than LinkedIn's. What's are the largest Kafka deployments
> >>you
> >> know of? Any public performance and scalability data published for such
> >> clusters?
> >>
> >> Any pointers would be interesting and helpful.
> >>
> >> Regards,
> >>
> >> --
> >> Ersin Er
> >>
>
>

RE: How recover leader when broker restart

[chenlax]

use preferred tool can rebalance leadership,but if the isr are null then the 
leader is only -1,how i can recover the leader.


Thanks，
Lax


> Date: Mon, 7 Jul 2014 08:06:16 -0700
> Subject: Re: How recover leader when broker restart
> From: wangg...@gmail.com
> To: users@kafka.apache.org
> 
> You can use the preferred leader election tool to move the leadership.
> 
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool
> 
> Guozhang
> 
> 
> On Mon, Jul 7, 2014 at 7:56 AM, 鞠大升  wrote:
> 
> > you can use the preferred leader election tool to reset leaders to
> > preferred replicas.
> > 2014年7月7日 PM10:37于 "François Langelier" 写道：
> >
> > > AFAIK, the simplest way will be to shutdown your 2 others brokers after
> > you
> > > restarted your broker 1, which will force your topics to have your
> > broker 1
> > > as leader since it's the only one available, and then restart your
> > brokers
> > > 2 and 3
> > >
> > > But I can't really see why you want your leaders on broker 1...
> > >
> > >
> > >
> > > François Langelier
> > > Étudiant en génie Logiciel - École de Technologie Supérieure
> > > 
> > > Capitaine Club Capra 
> > > VP-Communication - CS Games  2014
> > > Jeux de Génie  2011 à 2014
> > > Argentier Fraternité du Piranha 
> > > 2012-2014
> > > Comité Organisateur Olympiades ÉTS 2012
> > > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> > >
> > >
> > > On 7 July 2014 05:59, 陈翔  wrote:
> > >
> > > > i have 3 broker,when i restart a broker 1,then 1 can not as leader.i
> > want
> > > > to know how i can recover broker 1 as a leader.
> > > >
> > > > thanks,
> > > > lax
> > > >
> > >
> >
> 
> 
> 
> -- 
> -- Guozhang
  

Re: status of 0.8.2

[Jun Rao]

Two biggest features in 0.8.2 are Kafka-based offset management and the new
producer. We are in the final stage of testing them. We also haven't fully
tested the delete topic feature. So, we are probably 4-6 weeks away from
releasing 0.8.2.

For kafka-1180, the patch hasn't been applied yet and we will need a patch
for trunk.

Thanks,

Jun


On Mon, Jul 7, 2014 at 7:31 AM, Jason Rosenberg  wrote:

> What's the status for an 0.8.2 release?  We are currently using 0.8.0, and
> would like to upgrade to take advantage of some of the per-topic retention
> options available now in 0.8.1.
>
> However, we'd also like to take advantage of some fixes coming in 0.8.2
> (e.g. deleting topics).
>
> Also, we have been using a patch for (
> https://issues.apache.org/jira/browse/KAFKA-1180) applied to 0.8.0.  This
> is marked as scheduled for 0.8.2, with a patch available, but I'm not sure
> if this has been committed and applied to the 0.8.2 branch yet.
>
> Thanks,
>
> Jason
>

Re: Facing issues with Kafka 0.8.1.1 and kafka-reassign-partitions.sh

[Florian Dambrine]

When I run the tool with the --verify option it says failed for the some
partitions.

The problem is I do not know if it is a zookeeper issue or if the tool
really failed.

I faced one time the zookeeper issue (
https://issues.apache.org/jira/browse/KAFKA-1382) and by killing the
responsible Kafka the partition switched from failed to completed
successfully.

What should I do when the Kafka tool says that it failed to move the
partition?




On Mon, Jul 7, 2014 at 4:33 PM, Clark Haskins  wrote:

> How does it get stuck?
>
> -Clark
>
> Clark Elliott Haskins III
> LinkedIn DDS Site Reliability Engineer
> Kafka, Zookeeper, Samza SRE
> Mobile: 505.385.1484
> BlueJeans: https://www.bluejeans.com/chaskins
>
>
> chask...@linkedin.com
> https://www.linkedin.com/in/clarkhaskins
> There is no place like 127.0.0.1
>
>
>
>
> On 7/7/14, 3:49 PM, "Florian Dambrine"  wrote:
>
> >Hi,
> >
> >I am trying to add new brokers to an existing 8 nodes Kafka cluster. We
> >have around 10 topics and the number of partition is set to 50. In order
> >to
> >test the reassgin-partitions scripts, I tried on a sandbox cluster the
> >following steps.
> >
> >I developed a script which is able to parse the reassignment partition
> >plan
> >given by the Kafka tool in smaller pieces (reassigning maximum 10
> >partitions at a time).
> >
> >Unfortunately I faced some issues with the tool that sometimes get stuck
> >on
> >one partition. In this case I have to kill and restart the three Kafkas on
> >which the partition has been relocated to unlock the process (One kafka at
> >a time).
> >
> >Moreover, I have also faced these two issues that are already on Jira:
> >
> >https://issues.apache.org/jira/browse/KAFKA-1382
> >https://issues.apache.org/jira/browse/KAFKA-1479
> >
> >We really need to add new nodes to our Kafka cluster, does anybody have
> >already rebalance a Kafka 0.8.1.1? What could you advise me?
> >
> >Thanks, and feel free to ask me if you need more details.
> >
> >
> >
> >--
> >*Florian Dambrine*  |  Intern, Big Data
> >*GumGum*   |  *Ads that stick*
> >209-797-3994  |  flor...@gumgum.com
>
>


-- 
*Florian Dambrine*  |  Intern, Big Data
*GumGum*   |  *Ads that stick*
209-797-3994  |  flor...@gumgum.com

Re: Facing issues with Kafka 0.8.1.1 and kafka-reassign-partitions.sh

[Clark Haskins]

How does it get stuck?

-Clark

Clark Elliott Haskins III
LinkedIn DDS Site Reliability Engineer
Kafka, Zookeeper, Samza SRE
Mobile: 505.385.1484
BlueJeans: https://www.bluejeans.com/chaskins


chask...@linkedin.com
https://www.linkedin.com/in/clarkhaskins
There is no place like 127.0.0.1




On 7/7/14, 3:49 PM, "Florian Dambrine"  wrote:

>Hi,
>
>I am trying to add new brokers to an existing 8 nodes Kafka cluster. We
>have around 10 topics and the number of partition is set to 50. In order
>to
>test the reassgin-partitions scripts, I tried on a sandbox cluster the
>following steps.
>
>I developed a script which is able to parse the reassignment partition
>plan
>given by the Kafka tool in smaller pieces (reassigning maximum 10
>partitions at a time).
>
>Unfortunately I faced some issues with the tool that sometimes get stuck
>on
>one partition. In this case I have to kill and restart the three Kafkas on
>which the partition has been relocated to unlock the process (One kafka at
>a time).
>
>Moreover, I have also faced these two issues that are already on Jira:
>
>https://issues.apache.org/jira/browse/KAFKA-1382
>https://issues.apache.org/jira/browse/KAFKA-1479
>
>We really need to add new nodes to our Kafka cluster, does anybody have
>already rebalance a Kafka 0.8.1.1? What could you advise me?
>
>Thanks, and feel free to ask me if you need more details.
>
>
>
>-- 
>*Florian Dambrine*  |  Intern, Big Data
>*GumGum*   |  *Ads that stick*
>209-797-3994  |  flor...@gumgum.com

Re: kafka 0.8.1.1 log.retention.minutes NOT being honored

[Virendra Pratap Singh]

By setting this property
log.retention.mins=10
in the server.properties file, which is passed as argument when starting
the broker.

Virendra

On 7/7/14, 3:31 PM, "Guozhang Wang"  wrote:

>How do you set the retention.minutes property? Is it through zk-based
>topics tool?
>
>Guozhang
>
>
>On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0 and
>> the other 0.8.1.1. Should the retention of topics for partitions
>> owned/replicated by the broker running 0.8.1.1 not enforce the server
>> properties settings as defined for that server.
>>
>> So this brings an interesting question, in case of heterogeneous
>> environment (as is in my case, which system parameters will take
>> preference/precedence).
>>
>> Virendra
>>
>> On 6/30/14, 9:19 AM, "Guozhang Wang"  wrote:
>>
>> >The retention.minute property is only introduced in 0.8.1:
>> >
>> >https://issues.apache.org/jira/browse/KAFKA-918
>> >
>> >if you are running 0.8.0 then it will not be recognized.
>> >
>> >Guozhang
>> >
>> >
>> >
>> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
>> >vpsi...@yahoo-inc.com.invalid> wrote:
>> >
>> >> Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
>> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release link.
>> >> Directly using the tar ball, no local build used).
>> >>
>> >> I have set the log.retention.minutes=10. However the broker is not
>> >> honoring the setting. I see its not cleaning the log.dir at all.
>> >>
>> >> However when I set the log.retention.hours=1, then it starts cleaning
>> >>the
>> >> log.
>> >>
>> >> When I have the log.retention.minutes set in the server.properties
>>then
>> >>I
>> >> see this logged in server.log:
>> >>
>> >> Š..
>> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
>> >>valid
>> >> (kafka.utils.VerifiableProperties)
>> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
>> >>valid
>> >> (kafka.utils.VerifiableProperties)
>> >> ŠŠ
>> >>
>> >>
>> >> I have set these properties too:
>> >>
>> >> log.cleaner.enable=true
>> >> log.cleanup.policy=delete
>> >>
>> >>
>> >> But I see similar warning logged for these properties too.
>> >>
>> >> Regards,
>> >> Virendra
>> >>
>> >>
>> >
>> >
>> >--
>> >-- Guozhang
>>
>>
>
>
>-- 
>-- Guozhang

Facing issues with Kafka 0.8.1.1 and kafka-reassign-partitions.sh

[Florian Dambrine]

Hi,

I am trying to add new brokers to an existing 8 nodes Kafka cluster. We
have around 10 topics and the number of partition is set to 50. In order to
test the reassgin-partitions scripts, I tried on a sandbox cluster the
following steps.

I developed a script which is able to parse the reassignment partition plan
given by the Kafka tool in smaller pieces (reassigning maximum 10
partitions at a time).

Unfortunately I faced some issues with the tool that sometimes get stuck on
one partition. In this case I have to kill and restart the three Kafkas on
which the partition has been relocated to unlock the process (One kafka at
a time).

Moreover, I have also faced these two issues that are already on Jira:

https://issues.apache.org/jira/browse/KAFKA-1382
https://issues.apache.org/jira/browse/KAFKA-1479

We really need to add new nodes to our Kafka cluster, does anybody have
already rebalance a Kafka 0.8.1.1? What could you advise me?

Thanks, and feel free to ask me if you need more details.



-- 
*Florian Dambrine*  |  Intern, Big Data
*GumGum*   |  *Ads that stick*
209-797-3994  |  flor...@gumgum.com

Re: kafka 0.8.1.1 log.retention.minutes NOT being honored

[Guozhang Wang]

How do you set the retention.minutes property? Is it through zk-based
topics tool?

Guozhang


On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh <
vpsi...@yahoo-inc.com.invalid> wrote:

> I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0 and
> the other 0.8.1.1. Should the retention of topics for partitions
> owned/replicated by the broker running 0.8.1.1 not enforce the server
> properties settings as defined for that server.
>
> So this brings an interesting question, in case of heterogeneous
> environment (as is in my case, which system parameters will take
> preference/precedence).
>
> Virendra
>
> On 6/30/14, 9:19 AM, "Guozhang Wang"  wrote:
>
> >The retention.minute property is only introduced in 0.8.1:
> >
> >https://issues.apache.org/jira/browse/KAFKA-918
> >
> >if you are running 0.8.0 then it will not be recognized.
> >
> >Guozhang
> >
> >
> >
> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
> >vpsi...@yahoo-inc.com.invalid> wrote:
> >
> >> Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release link.
> >> Directly using the tar ball, no local build used).
> >>
> >> I have set the log.retention.minutes=10. However the broker is not
> >> honoring the setting. I see its not cleaning the log.dir at all.
> >>
> >> However when I set the log.retention.hours=1, then it starts cleaning
> >>the
> >> log.
> >>
> >> When I have the log.retention.minutes set in the server.properties then
> >>I
> >> see this logged in server.log:
> >>
> >> Š..
> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
> >>valid
> >> (kafka.utils.VerifiableProperties)
> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
> >>valid
> >> (kafka.utils.VerifiableProperties)
> >> ŠŠ
> >>
> >>
> >> I have set these properties too:
> >>
> >> log.cleaner.enable=true
> >> log.cleanup.policy=delete
> >>
> >>
> >> But I see similar warning logged for these properties too.
> >>
> >> Regards,
> >> Virendra
> >>
> >>
> >
> >
> >--
> >-- Guozhang
>
>


-- 
-- Guozhang

Re: kafka 0.8.1.1 log.retention.minutes NOT being honored

[Virendra Pratap Singh]

I am running a mixed cluster as I mentioned earlier. 1 broker 0.8.0 and
the other 0.8.1.1. Should the retention of topics for partitions
owned/replicated by the broker running 0.8.1.1 not enforce the server
properties settings as defined for that server.

So this brings an interesting question, in case of heterogeneous
environment (as is in my case, which system parameters will take
preference/precedence).

Virendra

On 6/30/14, 9:19 AM, "Guozhang Wang"  wrote:

>The retention.minute property is only introduced in 0.8.1:
>
>https://issues.apache.org/jira/browse/KAFKA-918
>
>if you are running 0.8.0 then it will not be recognized.
>
>Guozhang
>
>
>
>On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> Running a mixed 2 broker cluster. Mixed as in one of the broker1 is
>> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release link.
>> Directly using the tar ball, no local build used).
>>
>> I have set the log.retention.minutes=10. However the broker is not
>> honoring the setting. I see its not cleaning the log.dir at all.
>>
>> However when I set the log.retention.hours=1, then it starts cleaning
>>the
>> log.
>>
>> When I have the log.retention.minutes set in the server.properties then
>>I
>> see this logged in server.log:
>>
>> Š..
>> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
>>valid
>> (kafka.utils.VerifiableProperties)
>> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes is not
>>valid
>> (kafka.utils.VerifiableProperties)
>> ŠŠ
>>
>>
>> I have set these properties too:
>>
>> log.cleaner.enable=true
>> log.cleanup.policy=delete
>>
>>
>> But I see similar warning logged for these properties too.
>>
>> Regards,
>> Virendra
>>
>>
>
>
>-- 
>-- Guozhang

Re: Monitoring Producers at Large Scale

[Otis Gospodnetic]

Hi,

I'm late to the thread... but that "...we intercept log4j..." caught my
attention.  Why intercept, especially if it's causing trouble?

Could you use log4j syslog appender and get logs routed to wherever you
want them via syslog, for example?
Or you can have syslog tail log4j log files (e.g. rsyslog has "imfile" you
can use for tailing).

We use our own Logsene  for Kafka and all
other logs and SPM  for Kafka and all other
metrics we monitor.

Oh, actually, this may help you:
https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Sending+Events+to+Logsene
(ignore the Logsene-specific parts --- there is plenty of general info,
configs, etc. for log handling)

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/


On Thu, Jun 26, 2014 at 3:09 PM, Bhavesh Mistry 
wrote:

> Hi All,
>
> Thanks for all your responses.
>
>
>
> JMX metrics are there and we do pull the metrics, but I would like to
> capture the logs from Kafka lib as well especially WARN, FATAL and ERROR
> etc to debug the issue.
>
>
>
> To do this, we intercept Log4j logging and send it to Kafka Log Topics, but
> I realize that under heavy Kafka Lib error/warn/  it will create a deadlock
> between Producer Send thread  (Logging Kafka log topic queue...)
>
>
>
> *public* *class* KafkaLog4jAppender *extends* AppenderSkeleton {
>
>
>
> Producer  producer..
>
> *protected* *void* append(LoggingEvent event) {
>
>
> if(event.getLoggerName().startsWith("kafka")){
>
>  if(event is WARN, FATAL and ERROR){
>
> producer.send(event.getRenderedMessage())
>
> }
>
> }
>
>
>
> }
>
>
> Other option is to log Kafka Logs into disk and transport logs via
> separate process
> to Kafka Topic and transport via https://github.com/harelba/tail2kafka to
> topic...
>
>
> We use Kafka for Log transportation and we want to debug/trouble shoot
> issue via logs or create alerts/etc
>
>
> Thanks,
>
>
> Bhavesh
>
>
>
>
> On Wed, Jun 25, 2014 at 10:49 AM, Neha Narkhede 
> wrote:
>
> > We monitor producers or for that matter any process/service using JMX
> > metrics. Every server and service in LinkedIn sends metrics in a Kafka
> > message to a metrics Kafka cluster. We have subscribers that connect to
> the
> > metrics cluster to index that data in RRDs.
> >
> > Our aim is to expose all important metrics through JMX. We are doing that
> > for the new producer under org.apache.kafka.clients.producer. Feel free
> to
> > take a look at that and give feedback.
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Jun 24, 2014 at 7:59 PM, Darion Yaphet 
> > wrote:
> >
> > > Sorry I want to  know  you want to monitor kafka producers or kafka
> > brokers
> > > and zookeepers ?
> > > It's seems you will want to monitor monitor Exceptions eg Leader Not
> > Found,
> > > Queue is full, resend fail  etc  are kafka cluster
> > >
> > >
> > > 2014-06-25 8:20 GMT+08:00 Bhavesh Mistry :
> > >
> > > > We use Kafka as Transport Layer to transport application logs.  How
> do
> > we
> > > > monitor Producers at large scales about 6000 boxes x 4 topic per box
> so
> > > > roughly 24000 producers (spread across multiple data center.. we have
> > > > brokers per DC).  We do the monitoring based on logs.  I have tried
> > > > intercepting logs via Log4J custom implementation which only
> intercept
> > > WARN
> > > > and ERROR and FATAL events  org.apache.log4j.AppenderSkeleton append
> > > method
> > > > which send its logs to brokers (This is working but after load
> testing
> > it
> > > > is causing deadlock some times between ProducerSendThread and
> > Producer).
> > > >
> > > > I know there are JMX monitoring MBeans available which we can pull
> the
> > > > data, but I would like to monitor Exceptions eg Leader Not Found,
> Queue
> > > is
> > > > full, resend fail etc in Kafka Library.
> > > >
> > > > How does LinkedIn monitor the Producers ?
> > > >
> > > > Thanks,
> > > >
> > > > Bhavesh
> > > >
> > >
> > >
> > >
> > > --
> > >
> > >
> > > long is the way and hard  that out of Hell leads up to light
> > >
> >
>

Re: quick question about new consumer api

[Guozhang Wang]

We plan to have a working prototype ready end of September.

Guozhang


On Mon, Jul 7, 2014 at 11:05 AM, Jason Rosenberg  wrote:

> Great, that's reassuring!
>
> What's the time frame for having a more or less stable version to try out?
>
> Jason
>
>
> On Mon, Jul 7, 2014 at 12:59 PM, Guozhang Wang  wrote:
>
> > I see your point now. The old consumer does have a hard-coded
> > "round-robin-per-topic" logic which have this issue. In the new consumer,
> > we will make the assignment logic customizable so that people can specify
> > different rebalance algorithms they like.
> >
> > Also I will soon send out a new consumer design summary email for more
> > comments. Feel free to give us more thoughts you have about the new
> > consumer design.
> >
> > Guozhang
> >
> >
> > On Mon, Jul 7, 2014 at 8:44 AM, Jason Rosenberg 
> wrote:
> >
> > > Guozhang,
> > >
> > > I'm not suggesting we parallelize within a partition
> > >
> > > The problem with the current high-level consumer is, if you use a regex
> > to
> > > select multiple topics, and then have multiple consumers in the same
> > group,
> > > usually the first consumer will 'own' all the topics, and no amount of
> > > sub-sequent rebalancing will allow other consumers in the group to own
> > some
> > > of the topics.  Re-balancing does allow other consumers to own multiple
> > > partitions, but if a topic has only 1 partition, only the first
> consumer
> > to
> > > initialize will get all the work.
> > >
> > > So, I'm wondering if the new api will be better about re-balancing the
> > work
> > > at the partition level, and not the topic level, as such.
> > >
> > > Jason
> > >
> > >
> > > On Mon, Jul 7, 2014 at 11:17 AM, Guozhang Wang 
> > wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > In the new design the consumption is still at the per-partition
> > > > granularity. The main rationale of doing this is ordering: Within a
> > > > partition we want to preserve the ordering such that message B
> produced
> > > > after message A will also be consumed and processed after message A.
> > And
> > > > producers can use keys to make sure messages with the same ordering
> > group
> > > > will be in the same partition. To do this we have to make one
> partition
> > > > only being consumed by a single client at a time. On the other hand,
> > when
> > > > one wants to add the number of consumers beyond the number of
> > partitions,
> > > > he can always use the topic tool to dynamically add more partitions
> to
> > > the
> > > > topic.
> > > >
> > > > Do you have a specific scenario in mind that would require
> > > single-partition
> > > > topics?
> > > >
> > > > Guozhang
> > > >
> > > >
> > > >
> > > > On Mon, Jul 7, 2014 at 7:43 AM, Jason Rosenberg 
> > > wrote:
> > > >
> > > > > I've been looking at the new consumer api outlined here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
> > > > >
> > > > > One issue in the current high-level consumer, is that it does not
> do
> > a
> > > > good
> > > > > job of distributing a set of topics between multiple consumers,
> > unless
> > > > each
> > > > > topic has multiple partitions.  This has always seemed strange to
> me,
> > > > since
> > > > > at the end of the day, even for single partition topics, the basic
> > unit
> > > > of
> > > > > consumption is still at the partition level (so you'd expect
> > > rebalancing
> > > > to
> > > > > try to evenly distribute partitions (regardless of the topic)).
> > > > >
> > > > > It's not clearly spelled out in the new consumer api wiki, so I'll
> > just
> > > > > ask, will this issue be addressed in the new api?  I think I've
> asked
> > > > this
> > > > > before, but I wanted to go check again, and am not seeing this
> > > explicitly
> > > > > addressed in the design.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Jason
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
-- Guozhang

Re: quick question about new consumer api

[Jason Rosenberg]

Great, that's reassuring!

What's the time frame for having a more or less stable version to try out?

Jason


On Mon, Jul 7, 2014 at 12:59 PM, Guozhang Wang  wrote:

> I see your point now. The old consumer does have a hard-coded
> "round-robin-per-topic" logic which have this issue. In the new consumer,
> we will make the assignment logic customizable so that people can specify
> different rebalance algorithms they like.
>
> Also I will soon send out a new consumer design summary email for more
> comments. Feel free to give us more thoughts you have about the new
> consumer design.
>
> Guozhang
>
>
> On Mon, Jul 7, 2014 at 8:44 AM, Jason Rosenberg  wrote:
>
> > Guozhang,
> >
> > I'm not suggesting we parallelize within a partition
> >
> > The problem with the current high-level consumer is, if you use a regex
> to
> > select multiple topics, and then have multiple consumers in the same
> group,
> > usually the first consumer will 'own' all the topics, and no amount of
> > sub-sequent rebalancing will allow other consumers in the group to own
> some
> > of the topics.  Re-balancing does allow other consumers to own multiple
> > partitions, but if a topic has only 1 partition, only the first consumer
> to
> > initialize will get all the work.
> >
> > So, I'm wondering if the new api will be better about re-balancing the
> work
> > at the partition level, and not the topic level, as such.
> >
> > Jason
> >
> >
> > On Mon, Jul 7, 2014 at 11:17 AM, Guozhang Wang 
> wrote:
> >
> > > Hi Jason,
> > >
> > > In the new design the consumption is still at the per-partition
> > > granularity. The main rationale of doing this is ordering: Within a
> > > partition we want to preserve the ordering such that message B produced
> > > after message A will also be consumed and processed after message A.
> And
> > > producers can use keys to make sure messages with the same ordering
> group
> > > will be in the same partition. To do this we have to make one partition
> > > only being consumed by a single client at a time. On the other hand,
> when
> > > one wants to add the number of consumers beyond the number of
> partitions,
> > > he can always use the topic tool to dynamically add more partitions to
> > the
> > > topic.
> > >
> > > Do you have a specific scenario in mind that would require
> > single-partition
> > > topics?
> > >
> > > Guozhang
> > >
> > >
> > >
> > > On Mon, Jul 7, 2014 at 7:43 AM, Jason Rosenberg 
> > wrote:
> > >
> > > > I've been looking at the new consumer api outlined here:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
> > > >
> > > > One issue in the current high-level consumer, is that it does not do
> a
> > > good
> > > > job of distributing a set of topics between multiple consumers,
> unless
> > > each
> > > > topic has multiple partitions.  This has always seemed strange to me,
> > > since
> > > > at the end of the day, even for single partition topics, the basic
> unit
> > > of
> > > > consumption is still at the partition level (so you'd expect
> > rebalancing
> > > to
> > > > try to evenly distribute partitions (regardless of the topic)).
> > > >
> > > > It's not clearly spelled out in the new consumer api wiki, so I'll
> just
> > > > ask, will this issue be addressed in the new api?  I think I've asked
> > > this
> > > > before, but I wanted to go check again, and am not seeing this
> > explicitly
> > > > addressed in the design.
> > > >
> > > > Thanks
> > > >
> > > > Jason
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: Largest known Kafka deployment?

[Todd Palino]

You¹re out of date, Jun. We¹re up to 20 now :)

Our ops presentation on Kafka is a little more up to date on numbers:
http://www.slideshare.net/ToddPalino/enterprise-kafka-kafka-as-a-service

-Todd


On 7/7/14, 7:21 AM, "Jun Rao"  wrote:

>LinkedIn's largest Kafka cluster has 16 nodes now. More detailed info can
>be found in
>http://www.slideshare.net/Hadoop_Summit/building-a-realtime-data-pipeline-
>apache-kafka-at-linkedin?from_search=5
>
>Thanks,
>
>Jun
>
>
>On Mon, Jul 7, 2014 at 3:33 AM, Ersin Er  wrote:
>
>> Hi,
>>
>> LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are
>>larger
>> deployments than LinkedIn's. What's are the largest Kafka deployments
>>you
>> know of? Any public performance and scalability data published for such
>> clusters?
>>
>> Any pointers would be interesting and helpful.
>>
>> Regards,
>>
>> --
>> Ersin Er
>>

Re: quick question about new consumer api

[Guozhang Wang]

I see your point now. The old consumer does have a hard-coded
"round-robin-per-topic" logic which have this issue. In the new consumer,
we will make the assignment logic customizable so that people can specify
different rebalance algorithms they like.

Also I will soon send out a new consumer design summary email for more
comments. Feel free to give us more thoughts you have about the new
consumer design.

Guozhang


On Mon, Jul 7, 2014 at 8:44 AM, Jason Rosenberg  wrote:

> Guozhang,
>
> I'm not suggesting we parallelize within a partition
>
> The problem with the current high-level consumer is, if you use a regex to
> select multiple topics, and then have multiple consumers in the same group,
> usually the first consumer will 'own' all the topics, and no amount of
> sub-sequent rebalancing will allow other consumers in the group to own some
> of the topics.  Re-balancing does allow other consumers to own multiple
> partitions, but if a topic has only 1 partition, only the first consumer to
> initialize will get all the work.
>
> So, I'm wondering if the new api will be better about re-balancing the work
> at the partition level, and not the topic level, as such.
>
> Jason
>
>
> On Mon, Jul 7, 2014 at 11:17 AM, Guozhang Wang  wrote:
>
> > Hi Jason,
> >
> > In the new design the consumption is still at the per-partition
> > granularity. The main rationale of doing this is ordering: Within a
> > partition we want to preserve the ordering such that message B produced
> > after message A will also be consumed and processed after message A. And
> > producers can use keys to make sure messages with the same ordering group
> > will be in the same partition. To do this we have to make one partition
> > only being consumed by a single client at a time. On the other hand, when
> > one wants to add the number of consumers beyond the number of partitions,
> > he can always use the topic tool to dynamically add more partitions to
> the
> > topic.
> >
> > Do you have a specific scenario in mind that would require
> single-partition
> > topics?
> >
> > Guozhang
> >
> >
> >
> > On Mon, Jul 7, 2014 at 7:43 AM, Jason Rosenberg 
> wrote:
> >
> > > I've been looking at the new consumer api outlined here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
> > >
> > > One issue in the current high-level consumer, is that it does not do a
> > good
> > > job of distributing a set of topics between multiple consumers, unless
> > each
> > > topic has multiple partitions.  This has always seemed strange to me,
> > since
> > > at the end of the day, even for single partition topics, the basic unit
> > of
> > > consumption is still at the partition level (so you'd expect
> rebalancing
> > to
> > > try to evenly distribute partitions (regardless of the topic)).
> > >
> > > It's not clearly spelled out in the new consumer api wiki, so I'll just
> > > ask, will this issue be addressed in the new api?  I think I've asked
> > this
> > > before, but I wanted to go check again, and am not seeing this
> explicitly
> > > addressed in the design.
> > >
> > > Thanks
> > >
> > > Jason
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
-- Guozhang

kafka.common.FailedToSendMessageException: Failed to send messages after 1 tries.

[Kalpa 1977]

hi all,
   I am using kafka 0-8-1.

I have created a simple topic test no partition and no replication,
producer and consumer both works fine running in the same system,
However, If i call from the remote system as per the below mentioned
command.

./bin/kafka-producer-perf-test.sh --broker-list *172.16.229.229*:9092
--topic test --messages 1 --producer-num-retries 1

it gives error saying.

ERROR Producer connection to ubuntu.01-229:9092 unsuccessful
(kafka.producer.SyncProducer)
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)

In the broker list, I am specifying the IP (*172.16.229.229*), but, why it
is trying to connect using a host name of itself, and which is not
correct.I can ping remote ip, and also telnet on port 9092.

Please can anybody help me how to make it to connect to the remote ip.

thanks,
Kalpa.

Re: quick question about new consumer api

[Jason Rosenberg]

Guozhang,

I'm not suggesting we parallelize within a partition

The problem with the current high-level consumer is, if you use a regex to
select multiple topics, and then have multiple consumers in the same group,
usually the first consumer will 'own' all the topics, and no amount of
sub-sequent rebalancing will allow other consumers in the group to own some
of the topics.  Re-balancing does allow other consumers to own multiple
partitions, but if a topic has only 1 partition, only the first consumer to
initialize will get all the work.

So, I'm wondering if the new api will be better about re-balancing the work
at the partition level, and not the topic level, as such.

Jason


On Mon, Jul 7, 2014 at 11:17 AM, Guozhang Wang  wrote:

> Hi Jason,
>
> In the new design the consumption is still at the per-partition
> granularity. The main rationale of doing this is ordering: Within a
> partition we want to preserve the ordering such that message B produced
> after message A will also be consumed and processed after message A. And
> producers can use keys to make sure messages with the same ordering group
> will be in the same partition. To do this we have to make one partition
> only being consumed by a single client at a time. On the other hand, when
> one wants to add the number of consumers beyond the number of partitions,
> he can always use the topic tool to dynamically add more partitions to the
> topic.
>
> Do you have a specific scenario in mind that would require single-partition
> topics?
>
> Guozhang
>
>
>
> On Mon, Jul 7, 2014 at 7:43 AM, Jason Rosenberg  wrote:
>
> > I've been looking at the new consumer api outlined here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
> >
> > One issue in the current high-level consumer, is that it does not do a
> good
> > job of distributing a set of topics between multiple consumers, unless
> each
> > topic has multiple partitions.  This has always seemed strange to me,
> since
> > at the end of the day, even for single partition topics, the basic unit
> of
> > consumption is still at the partition level (so you'd expect rebalancing
> to
> > try to evenly distribute partitions (regardless of the topic)).
> >
> > It's not clearly spelled out in the new consumer api wiki, so I'll just
> > ask, will this issue be addressed in the new api?  I think I've asked
> this
> > before, but I wanted to go check again, and am not seeing this explicitly
> > addressed in the design.
> >
> > Thanks
> >
> > Jason
> >
>
>
>
> --
> -- Guozhang
>

Re: quick question about new consumer api

[Guozhang Wang]

Hi Jason,

In the new design the consumption is still at the per-partition
granularity. The main rationale of doing this is ordering: Within a
partition we want to preserve the ordering such that message B produced
after message A will also be consumed and processed after message A. And
producers can use keys to make sure messages with the same ordering group
will be in the same partition. To do this we have to make one partition
only being consumed by a single client at a time. On the other hand, when
one wants to add the number of consumers beyond the number of partitions,
he can always use the topic tool to dynamically add more partitions to the
topic.

Do you have a specific scenario in mind that would require single-partition
topics?

Guozhang



On Mon, Jul 7, 2014 at 7:43 AM, Jason Rosenberg  wrote:

> I've been looking at the new consumer api outlined here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
>
> One issue in the current high-level consumer, is that it does not do a good
> job of distributing a set of topics between multiple consumers, unless each
> topic has multiple partitions.  This has always seemed strange to me, since
> at the end of the day, even for single partition topics, the basic unit of
> consumption is still at the partition level (so you'd expect rebalancing to
> try to evenly distribute partitions (regardless of the topic)).
>
> It's not clearly spelled out in the new consumer api wiki, so I'll just
> ask, will this issue be addressed in the new api?  I think I've asked this
> before, but I wanted to go check again, and am not seeing this explicitly
> addressed in the design.
>
> Thanks
>
> Jason
>



-- 
-- Guozhang

Re: Kafka 0.8.x failover with multiple data centers

[Guozhang Wang]

Hello Janos,

The approach we took at LinkedIn is the first option, i.e. using different
clusters at different DC, and mirroring data asynchronously. For the offset
inconsistency issue, our applications usually use the offset request with
the timestamp when primary DC was down and conservatively get an older
offset to start with, and dedup messages at the application level.

Guozhang


On Mon, Jul 7, 2014 at 7:54 AM, Janos Mucza  wrote:

> Dear Kafka Users,
>
> I would like to use Kafka 0.8.x in a multi-cluster environment so that
> when my primary cluster fails, producers and consumers could switch to the
> secondary cluster. Clusters would be hosted in different data centers.
>
> A possibility would be mirroring topics (similar to Kafka 0.7.x mirror
> maker). The issue with this is consumer offset management, since a mirrored
> message will probably have different offset than the source message.
>
> Running a single Kafka cluster with nodes in both data centers raises the
> question of how to ensure a message was persisted by at least one broker in
> each data center. Even with all in sync replicas ACK requested, the
> producer can't be sure what brokers persisted a message because in sync
> replicas might change dynamically.
>
> Could you please share your experience about running Kafka 0.8.x
> cluster(s) on multiple data centers?
>
> Thank you very much.
>
> Best regards,
> Janos
>



-- 
-- Guozhang

Re: How recover leader when broker restart

[Guozhang Wang]

You can use the preferred leader election tool to move the leadership.

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool

Guozhang


On Mon, Jul 7, 2014 at 7:56 AM, 鞠大升  wrote:

> you can use the preferred leader election tool to reset leaders to
> preferred replicas.
> 2014年7月7日 PM10:37于 "François Langelier" 写道：
>
> > AFAIK, the simplest way will be to shutdown your 2 others brokers after
> you
> > restarted your broker 1, which will force your topics to have your
> broker 1
> > as leader since it's the only one available, and then restart your
> brokers
> > 2 and 3
> >
> > But I can't really see why you want your leaders on broker 1...
> >
> >
> >
> > François Langelier
> > Étudiant en génie Logiciel - École de Technologie Supérieure
> > 
> > Capitaine Club Capra 
> > VP-Communication - CS Games  2014
> > Jeux de Génie  2011 à 2014
> > Argentier Fraternité du Piranha 
> > 2012-2014
> > Comité Organisateur Olympiades ÉTS 2012
> > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
> >
> >
> > On 7 July 2014 05:59, 陈翔  wrote:
> >
> > > i have 3 broker,when i restart a broker 1,then 1 can not as leader.i
> want
> > > to know how i can recover broker 1 as a leader.
> > >
> > > thanks,
> > > lax
> > >
> >
>



-- 
-- Guozhang

Re: How recover leader when broker restart

[鞠大升]

you can use the preferred leader election tool to reset leaders to
preferred replicas.
2014年7月7日 PM10:37于 "François Langelier" 写道：

> AFAIK, the simplest way will be to shutdown your 2 others brokers after you
> restarted your broker 1, which will force your topics to have your broker 1
> as leader since it's the only one available, and then restart your brokers
> 2 and 3
>
> But I can't really see why you want your leaders on broker 1...
>
>
>
> François Langelier
> Étudiant en génie Logiciel - École de Technologie Supérieure
> 
> Capitaine Club Capra 
> VP-Communication - CS Games  2014
> Jeux de Génie  2011 à 2014
> Argentier Fraternité du Piranha 
> 2012-2014
> Comité Organisateur Olympiades ÉTS 2012
> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>
>
> On 7 July 2014 05:59, 陈翔  wrote:
>
> > i have 3 broker,when i restart a broker 1,then 1 can not as leader.i want
> > to know how i can recover broker 1 as a leader.
> >
> > thanks,
> > lax
> >
>

Kafka 0.8.x failover with multiple data centers

[Janos Mucza]

Dear Kafka Users,

I would like to use Kafka 0.8.x in a multi-cluster environment so that when my 
primary cluster fails, producers and consumers could switch to the secondary 
cluster. Clusters would be hosted in different data centers.

A possibility would be mirroring topics (similar to Kafka 0.7.x mirror maker). 
The issue with this is consumer offset management, since a mirrored message 
will probably have different offset than the source message.

Running a single Kafka cluster with nodes in both data centers raises the 
question of how to ensure a message was persisted by at least one broker in 
each data center. Even with all in sync replicas ACK requested, the producer 
can't be sure what brokers persisted a message because in sync replicas might 
change dynamically.

Could you please share your experience about running Kafka 0.8.x cluster(s) on 
multiple data centers?

Thank you very much.

Best regards,
Janos

quick question about new consumer api

[Jason Rosenberg]

I've been looking at the new consumer api outlined here:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design

One issue in the current high-level consumer, is that it does not do a good
job of distributing a set of topics between multiple consumers, unless each
topic has multiple partitions.  This has always seemed strange to me, since
at the end of the day, even for single partition topics, the basic unit of
consumption is still at the partition level (so you'd expect rebalancing to
try to evenly distribute partitions (regardless of the topic)).

It's not clearly spelled out in the new consumer api wiki, so I'll just
ask, will this issue be addressed in the new api?  I think I've asked this
before, but I wanted to go check again, and am not seeing this explicitly
addressed in the design.

Thanks

Jason

Re: How recover leader when broker restart

[François Langelier]

AFAIK, the simplest way will be to shutdown your 2 others brokers after you
restarted your broker 1, which will force your topics to have your broker 1
as leader since it's the only one available, and then restart your brokers
2 and 3

But I can't really see why you want your leaders on broker 1...



François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure

Capitaine Club Capra 
VP-Communication - CS Games  2014
Jeux de Génie  2011 à 2014
Argentier Fraternité du Piranha  2012-2014
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior


On 7 July 2014 05:59, 陈翔  wrote:

> i have 3 broker,when i restart a broker 1,then 1 can not as leader.i want
> to know how i can recover broker 1 as a leader.
>
> thanks,
> lax
>

status of 0.8.2

[Jason Rosenberg]

What's the status for an 0.8.2 release?  We are currently using 0.8.0, and
would like to upgrade to take advantage of some of the per-topic retention
options available now in 0.8.1.

However, we'd also like to take advantage of some fixes coming in 0.8.2
(e.g. deleting topics).

Also, we have been using a patch for (
https://issues.apache.org/jira/browse/KAFKA-1180) applied to 0.8.0.  This
is marked as scheduled for 0.8.2, with a patch available, but I'm not sure
if this has been committed and applied to the 0.8.2 branch yet.

Thanks,

Jason

Re: Largest known Kafka deployment?

[Jun Rao]

LinkedIn's largest Kafka cluster has 16 nodes now. More detailed info can
be found in
http://www.slideshare.net/Hadoop_Summit/building-a-realtime-data-pipeline-apache-kafka-at-linkedin?from_search=5

Thanks,

Jun


On Mon, Jul 7, 2014 at 3:33 AM, Ersin Er  wrote:

> Hi,
>
> LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are larger
> deployments than LinkedIn's. What's are the largest Kafka deployments you
> know of? Any public performance and scalability data published for such
> clusters?
>
> Any pointers would be interesting and helpful.
>
> Regards,
>
> --
> Ersin Er
>

How recover leader when broker restart

[陈翔]

i have 3 broker,when i restart a broker 1,then 1 can not as leader.i want to 
know how i can recover broker 1 as a leader.

thanks,
lax
  

Re: Largest known Kafka deployment?

[Ersin Er]

I mean brokers particularly but others are also welcome.
On Jul 7, 2014 3:36 PM, "Otis Gospodnetic" 
wrote:

> Hi,
>
> I think it depends on what you mean by largest?  Most brokers? Producers?
>  Consumers?  Messages?  Bytes?
>
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Mon, Jul 7, 2014 at 6:33 AM, Ersin Er  wrote:
>
> > Hi,
> >
> > LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are larger
> > deployments than LinkedIn's. What's are the largest Kafka deployments you
> > know of? Any public performance and scalability data published for such
> > clusters?
> >
> > Any pointers would be interesting and helpful.
> >
> > Regards,
> >
> > --
> > Ersin Er
> >
>

Re: Largest known Kafka deployment?

[Otis Gospodnetic]

Hi,

I think it depends on what you mean by largest?  Most brokers? Producers?
 Consumers?  Messages?  Bytes?

Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/


On Mon, Jul 7, 2014 at 6:33 AM, Ersin Er  wrote:

> Hi,
>
> LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are larger
> deployments than LinkedIn's. What's are the largest Kafka deployments you
> know of? Any public performance and scalability data published for such
> clusters?
>
> Any pointers would be interesting and helpful.
>
> Regards,
>
> --
> Ersin Er
>

Largest known Kafka deployment?

[Ersin Er]

Hi,

LinkedIn has 8 node Kafka clusters AFAIK, right? I guess there are larger
deployments than LinkedIn's. What's are the largest Kafka deployments you
know of? Any public performance and scalability data published for such
clusters?

Any pointers would be interesting and helpful.

Regards,

-- 
Ersin Er