kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.

[David Li]

Hi I have a very simple code to just send out one message, the topic is
created automatically, but the message just cannot be sent out. I also
tried to change the configuration, the result is still the same. Sorry to
bother you all with this silly question.

For your information, the kafka server is running on a docker container,
which is run in a ubuntu server vm. The test class is run from IntelliJ
IDEA on the host, which is a amc os x.

public static void main(String[] args) {
String topic = "test3";

Properties props = new Properties();
props.put("serializer.class", "kafka.serializer.StringEncoder");
props.put("metadata.broker.list", "192.168.144.10:29092");
//props.put("retry.backoff.ms", "1000");
//props.put("message.send.max.retries", "10");
//props.put("topic.metadata.refresh.interval.ms", "0");

Producer producer = new Producer(new ProducerConfig(props));

int messageNo = 1;
String messageStr = new String("Message_" + messageNo);
producer.send(new KeyedMessage(topic, messageStr));
}

Re: Checkpointing with custom metadata

[Gwen Shapira]

If it was my architecture, I'd consider publishing a "control message"
either to same topic or separate saying "done with topic A".
The "thing" that needs to continue copying A also needs to know where to
continue, so presumably if the "done with A" message is missing, it will
read from sorted-topic and find out where it was at. So even if we
accidentally dropped the "done" message, it will waste a bit of time, but
won't corrupt anything.

This is one of the use-cases that transactional producers would be so cool
:)

I can see how adding metadata to commit message can emulate some
light-weight transactions, but I'd be concerned that this capability can
get abused...

P.S
Thanks. I like my new address :)

On Mon, Aug 3, 2015 at 6:46 PM, James Cheng  wrote:

> Nice new email address, Gwen. :)
>
> On Aug 3, 2015, at 3:17 PM, Gwen Shapira  wrote:
>
> > You are correct. You can see that ZookeeperConsumerConnector is hardcoded
> > with null metadata.
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala#L310
> >
> > More interesting, it looks like the Metadata is not exposed in the new
> > KafkaConsumer either.
> >
> > Mind sharing what did you plan to use it for? this will help us figure
> out
> > how to expose it :)
> >
>
> I'm juggling around a couple designs on what I'm trying to do, so it may
> turn out that I actually don't need it.
>
> My generic answer is, I have some application state related to the
> processing of a topic, and I wanted to store it somewhere persistent that
> will survive across process death and disk failure. So storing it with the
> offset seemed like a nice solution. I could alternatively store it in a
> standalone kafka topic instead.
>
> The more detailed answer: I'm doing a time-based merge sort of 3 topics
> (A, B, and C) and outputtiing the results into a new output topic (let's
> call it "sorted-topic"). Except that during the initial creation of
> "sorted-topic", and I want all of topic A to be output to sorted-topic
> first, and then followed by an on-going merge sort of topics B, C, and any
> updates to A that come along.
>
> I was trying to handle the situation of what happens if I crash when
> initially copying topic A into sorted-topic. And I was thinking that I
> could save some metadata in my topic A checkpoint that says "still doing
> initial copy of A". So that way, when I start up next time, I would know to
> continue copying topic A to the output. Once I have finished copying all of
> A to sorted-topic, that I would store "finished doing initial copy of A"
> into my checkpoint, and that upon restart, I would check that and know to
> start doing the merge sort of A B C.
>
> I have a couple other designs that seem cleaner, tho, so I might not
> actually need it.
>
> -James
>
> > Gwen
> >
> >
> > On Mon, Aug 3, 2015 at 1:52 PM, James Cheng  wrote:
> >
> >> According to
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommitRequest
> ,
> >> we can store custom metadata with our checkpoints. It looks like the
> high
> >> level consumer does not support committing offsets with metadata, and
> that
> >> in order to checkpoint with custom metadata, we have to issue the
> >> OffsetCommitRequest ourselves. Is that correct?
> >>
> >> Thanks,
> >> -James
> >>
> >>
>
>

Re: Checkpointing with custom metadata

[James Cheng]

Nice new email address, Gwen. :)

On Aug 3, 2015, at 3:17 PM, Gwen Shapira  wrote:

> You are correct. You can see that ZookeeperConsumerConnector is hardcoded
> with null metadata.
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala#L310
> 
> More interesting, it looks like the Metadata is not exposed in the new
> KafkaConsumer either.
> 
> Mind sharing what did you plan to use it for? this will help us figure out
> how to expose it :)
> 

I'm juggling around a couple designs on what I'm trying to do, so it may turn 
out that I actually don't need it.

My generic answer is, I have some application state related to the processing 
of a topic, and I wanted to store it somewhere persistent that will survive 
across process death and disk failure. So storing it with the offset seemed 
like a nice solution. I could alternatively store it in a standalone kafka 
topic instead.

The more detailed answer: I'm doing a time-based merge sort of 3 topics (A, B, 
and C) and outputtiing the results into a new output topic (let's call it 
"sorted-topic"). Except that during the initial creation of "sorted-topic", and 
I want all of topic A to be output to sorted-topic first, and then followed by 
an on-going merge sort of topics B, C, and any updates to A that come along.

I was trying to handle the situation of what happens if I crash when initially 
copying topic A into sorted-topic. And I was thinking that I could save some 
metadata in my topic A checkpoint that says "still doing initial copy of A". So 
that way, when I start up next time, I would know to continue copying topic A 
to the output. Once I have finished copying all of A to sorted-topic, that I 
would store "finished doing initial copy of A" into my checkpoint, and that 
upon restart, I would check that and know to start doing the merge sort of A B 
C.

I have a couple other designs that seem cleaner, tho, so I might not actually 
need it.

-James

> Gwen
> 
> 
> On Mon, Aug 3, 2015 at 1:52 PM, James Cheng  wrote:
> 
>> According to
>> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommitRequest,
>> we can store custom metadata with our checkpoints. It looks like the high
>> level consumer does not support committing offsets with metadata, and that
>> in order to checkpoint with custom metadata, we have to issue the
>> OffsetCommitRequest ourselves. Is that correct?
>> 
>> Thanks,
>> -James
>> 
>> 

Re: Checkpointing with custom metadata

[Gwen Shapira]

You are correct. You can see that ZookeeperConsumerConnector is hardcoded
with null metadata.
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala#L310

More interesting, it looks like the Metadata is not exposed in the new
KafkaConsumer either.

Mind sharing what did you plan to use it for? this will help us figure out
how to expose it :)

Gwen


On Mon, Aug 3, 2015 at 1:52 PM, James Cheng  wrote:

> According to
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommitRequest,
> we can store custom metadata with our checkpoints. It looks like the high
> level consumer does not support committing offsets with metadata, and that
> in order to checkpoint with custom metadata, we have to issue the
> OffsetCommitRequest ourselves. Is that correct?
>
> Thanks,
> -James
>
>

Re: 0.9.0 release

[Gwen Shapira]

Yep. Fixed the JIRA.

Thanks :)

On Mon, Aug 3, 2015 at 2:01 PM, Shrikant Patel  wrote:

> I think the answer is yes, I was confused because on Jira it say 0.9.0.
>
> __
> Shrikant Patel  |  817.246.6760  |  ext. 4302
> Enterprise Architecture Team
> PDX-NHIN-Rx.com
>
> -Original Message-
> From: Shrikant Patel [mailto:spa...@pdxinc.com]
> Sent: Monday, August 03, 2015 3:55 PM
> To: users@kafka.apache.org
> Subject: RE: 0.9.0 release
>
> I'm looking at https://issues.apache.org/jira/browse/KAFKA-1684. Will
> this be part of 0.8.3??
>
> We are in healthcare domain, its company security requirement to have
> anything on wire to be encrypt. So producers to broker and broker to
> consumer needs to be encrypted.
>
>
> __
> Shrikant Patel  |  817.246.6760  |  ext. 4302 Enterprise Architecture
> Team PDX-NHIN-Rx.com
>
> -Original Message-
> From: Gwen Shapira [mailto:g...@confluent.io]
> Sent: Monday, August 03, 2015 3:23 PM
> To: users@kafka.apache.org
> Subject: Re: 0.9.0 release
>
> According to the plan, never :)
>
> Is there a specific feature you are looking forward to? I think the most
> exciting features are planned for 0.8.3 - which is targeted for Oct.
>
> On Mon, Aug 3, 2015 at 1:14 PM, Shrikant Patel  wrote:
>
> > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
> >
> > Any idea when 0.9.0 will be released?
> >
> > Thanks,
> > Shri
> >
> >
> > 
> > This message and its contents (to include attachments) are the
> > property of National Health Systems, Inc. and may contain confidential
> > and proprietary information. This email and any files transmitted with
> > it are intended solely for the use of the individual or entity to whom
> they are addressed.
> > You are hereby notified that any unauthorized disclosure, copying, or
> > distribution of this message, or the taking of any unauthorized action
> > based on information contained herein is strictly prohibited.
> > Unauthorized use of information contained herein may subject you to
> > civil and criminal prosecution and penalties. If you are not the
> > intended recipient, you should delete this message immediately and
> > notify the sender immediately by telephone or by replying to this
> transmission.
> >
>
> This message and its contents (to include attachments) are the property of
> National Health Systems, Inc. and may contain confidential and proprietary
> information. This email and any files transmitted with it are intended
> solely for the use of the individual or entity to whom they are addressed.
> You are hereby notified that any unauthorized disclosure, copying, or
> distribution of this message, or the taking of any unauthorized action
> based on information contained herein is strictly prohibited. Unauthorized
> use of information contained herein may subject you to civil and criminal
> prosecution and penalties. If you are not the intended recipient, you
> should delete this message immediately and notify the sender immediately by
> telephone or by replying to this transmission.
>
> This message and its contents (to include attachments) are the property of
> National Health Systems, Inc. and may contain confidential and proprietary
> information. This email and any files transmitted with it are intended
> solely for the use of the individual or entity to whom they are addressed.
> You are hereby notified that any unauthorized disclosure, copying, or
> distribution of this message, or the taking of any unauthorized action
> based on information contained herein is strictly prohibited. Unauthorized
> use of information contained herein may subject you to civil and criminal
> prosecution and penalties. If you are not the intended recipient, you
> should delete this message immediately and notify the sender immediately by
> telephone or by replying to this transmission.
>

RE: 0.9.0 release

[Shrikant Patel]

I think the answer is yes, I was confused because on Jira it say 0.9.0.

__
Shrikant Patel  |  817.246.6760  |  ext. 4302
Enterprise Architecture Team
PDX-NHIN-Rx.com

-Original Message-
From: Shrikant Patel [mailto:spa...@pdxinc.com]
Sent: Monday, August 03, 2015 3:55 PM
To: users@kafka.apache.org
Subject: RE: 0.9.0 release

I'm looking at https://issues.apache.org/jira/browse/KAFKA-1684. Will this be 
part of 0.8.3??

We are in healthcare domain, its company security requirement to have anything 
on wire to be encrypt. So producers to broker and broker to consumer needs to 
be encrypted.


__
Shrikant Patel  |  817.246.6760  |  ext. 4302 Enterprise Architecture Team 
PDX-NHIN-Rx.com

-Original Message-
From: Gwen Shapira [mailto:g...@confluent.io]
Sent: Monday, August 03, 2015 3:23 PM
To: users@kafka.apache.org
Subject: Re: 0.9.0 release

According to the plan, never :)

Is there a specific feature you are looking forward to? I think the most 
exciting features are planned for 0.8.3 - which is targeted for Oct.

On Mon, Aug 3, 2015 at 1:14 PM, Shrikant Patel  wrote:

> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
>
> Any idea when 0.9.0 will be released?
>
> Thanks,
> Shri
>
>
> 
> This message and its contents (to include attachments) are the
> property of National Health Systems, Inc. and may contain confidential
> and proprietary information. This email and any files transmitted with
> it are intended solely for the use of the individual or entity to whom they 
> are addressed.
> You are hereby notified that any unauthorized disclosure, copying, or
> distribution of this message, or the taking of any unauthorized action
> based on information contained herein is strictly prohibited.
> Unauthorized use of information contained herein may subject you to
> civil and criminal prosecution and penalties. If you are not the
> intended recipient, you should delete this message immediately and
> notify the sender immediately by telephone or by replying to this 
> transmission.
>

This message and its contents (to include attachments) are the property of 
National Health Systems, Inc. and may contain confidential and proprietary 
information. This email and any files transmitted with it are intended solely 
for the use of the individual or entity to whom they are addressed. You are 
hereby notified that any unauthorized disclosure, copying, or distribution of 
this message, or the taking of any unauthorized action based on information 
contained herein is strictly prohibited. Unauthorized use of information 
contained herein may subject you to civil and criminal prosecution and 
penalties. If you are not the intended recipient, you should delete this 
message immediately and notify the sender immediately by telephone or by 
replying to this transmission.

This message and its contents (to include attachments) are the property of 
National Health Systems, Inc. and may contain confidential and proprietary 
information. This email and any files transmitted with it are intended solely 
for the use of the individual or entity to whom they are addressed. You are 
hereby notified that any unauthorized disclosure, copying, or distribution of 
this message, or the taking of any unauthorized action based on information 
contained herein is strictly prohibited. Unauthorized use of information 
contained herein may subject you to civil and criminal prosecution and 
penalties. If you are not the intended recipient, you should delete this 
message immediately and notify the sender immediately by telephone or by 
replying to this transmission.

RE: 0.9.0 release

[Shrikant Patel]

I'm looking at https://issues.apache.org/jira/browse/KAFKA-1684. Will this be 
part of 0.8.3??

We are in healthcare domain, its company security requirement to have anything 
on wire to be encrypt. So producers to broker and broker to consumer needs to 
be encrypted.


__
Shrikant Patel  |  817.246.6760  |  ext. 4302
Enterprise Architecture Team
PDX-NHIN-Rx.com

-Original Message-
From: Gwen Shapira [mailto:g...@confluent.io]
Sent: Monday, August 03, 2015 3:23 PM
To: users@kafka.apache.org
Subject: Re: 0.9.0 release

According to the plan, never :)

Is there a specific feature you are looking forward to? I think the most 
exciting features are planned for 0.8.3 - which is targeted for Oct.

On Mon, Aug 3, 2015 at 1:14 PM, Shrikant Patel  wrote:

> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
>
> Any idea when 0.9.0 will be released?
>
> Thanks,
> Shri
>
>
> 
> This message and its contents (to include attachments) are the
> property of National Health Systems, Inc. and may contain confidential
> and proprietary information. This email and any files transmitted with
> it are intended solely for the use of the individual or entity to whom they 
> are addressed.
> You are hereby notified that any unauthorized disclosure, copying, or
> distribution of this message, or the taking of any unauthorized action
> based on information contained herein is strictly prohibited.
> Unauthorized use of information contained herein may subject you to
> civil and criminal prosecution and penalties. If you are not the
> intended recipient, you should delete this message immediately and
> notify the sender immediately by telephone or by replying to this 
> transmission.
>

This message and its contents (to include attachments) are the property of 
National Health Systems, Inc. and may contain confidential and proprietary 
information. This email and any files transmitted with it are intended solely 
for the use of the individual or entity to whom they are addressed. You are 
hereby notified that any unauthorized disclosure, copying, or distribution of 
this message, or the taking of any unauthorized action based on information 
contained herein is strictly prohibited. Unauthorized use of information 
contained herein may subject you to civil and criminal prosecution and 
penalties. If you are not the intended recipient, you should delete this 
message immediately and notify the sender immediately by telephone or by 
replying to this transmission.

Checkpointing with custom metadata

[James Cheng]

According to 
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommitRequest,
 we can store custom metadata with our checkpoints. It looks like the high 
level consumer does not support committing offsets with metadata, and that in 
order to checkpoint with custom metadata, we have to issue the 
OffsetCommitRequest ourselves. Is that correct?

Thanks,
-James

Re: 0.9.0 release

[Gwen Shapira]

According to the plan, never :)

Is there a specific feature you are looking forward to? I think the most
exciting features are planned for 0.8.3 - which is targeted for Oct.

On Mon, Aug 3, 2015 at 1:14 PM, Shrikant Patel  wrote:

> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
>
> Any idea when 0.9.0 will be released?
>
> Thanks,
> Shri
>
>
> 
> This message and its contents (to include attachments) are the property of
> National Health Systems, Inc. and may contain confidential and proprietary
> information. This email and any files transmitted with it are intended
> solely for the use of the individual or entity to whom they are addressed.
> You are hereby notified that any unauthorized disclosure, copying, or
> distribution of this message, or the taking of any unauthorized action
> based on information contained herein is strictly prohibited. Unauthorized
> use of information contained herein may subject you to civil and criminal
> prosecution and penalties. If you are not the intended recipient, you
> should delete this message immediately and notify the sender immediately by
> telephone or by replying to this transmission.
>

0.9.0 release

[Shrikant Patel]

https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan

Any idea when 0.9.0 will be released?

Thanks,
Shri



This message and its contents (to include attachments) are the property of 
National Health Systems, Inc. and may contain confidential and proprietary 
information. This email and any files transmitted with it are intended solely 
for the use of the individual or entity to whom they are addressed. You are 
hereby notified that any unauthorized disclosure, copying, or distribution of 
this message, or the taking of any unauthorized action based on information 
contained herein is strictly prohibited. Unauthorized use of information 
contained herein may subject you to civil and criminal prosecution and 
penalties. If you are not the intended recipient, you should delete this 
message immediately and notify the sender immediately by telephone or by 
replying to this transmission.

Re: Consumer limit for pub-sub mode

[Sharninder]

I don't know of any limits as such but I don't think your problem is suitable 
for Kafka. The third point especially wouldn't work with Kafka. Using Kafka, 
only one consumer will get a message out of the 30k



> On 03-Aug-2015, at 10:39 am, Vaibhav Kirte  wrote:
> 
> Hi,
> 
> I need to know how many consumers can subscribe to a single topic ( with
> one partition ).
> 
> I have a requirement such that,
>1. The producer will post to 1 topic having 1 partition.
>1. *20,000-30,000 consumers *should to be able to consume messages.
>2. All of the consumers should receive all messages that are produced.
> 
> will this be possible using kafka ?
> is there a limit on number of consumers ?
> what will be the number of machines that I will need to satisfy the
> requirements ?
> 
> -- 
> Regards,
> Vaibhav Kirte

Re: Consumer limit for pub-sub mode

[Gwen Shapira]

I don't know a specific limit for number of consumers, perhaps someone will
have better idea.

Do note that with a single topic and a single partition, you can't really
scale by adding more machines - the way Kafka is currently designed, all
consumers will read from the one machine that has the leader for that
partition.

Can you share a bit more about why you have a requirement for 30K consumers
with one partition? What's the specific use-case?

Gwen

On Sun, Aug 2, 2015 at 10:09 PM, Vaibhav Kirte 
wrote:

> Hi,
>
> I need to know how many consumers can subscribe to a single topic ( with
> one partition ).
>
> I have a requirement such that,
> 1. The producer will post to 1 topic having 1 partition.
> 1. *20,000-30,000 consumers *should to be able to consume messages.
> 2. All of the consumers should receive all messages that are produced.
>
> will this be possible using kafka ?
> is there a limit on number of consumers ?
> what will be the number of machines that I will need to satisfy the
> requirements ?
>
> --
> Regards,
> Vaibhav Kirte
>

Re: Consumer limit for pub-sub mode

[Jason Gustafson]

Hey Valibhav,

With only one partition, all of the consumers will end up hitting a single
broker (since partitions cannot be split). Whether it is possible to get
that number of consumers on a single broker may depend on the message load
through the topic. I think there has been some interest in allowing
consumers to read from partition replicas and that would help, but I'm not
sure if there is anything solid. If the single broker is not enough to
handle the load, then you might also consider replicating the topic to
several other topics in the same cluster (e.g. using mirror maker), and
that would allow you to distribute the consumer load across those other
topics. Others might have better ideas.

-Jason

On Sun, Aug 2, 2015 at 10:09 PM, Vaibhav Kirte 
wrote:

> Hi,
>
> I need to know how many consumers can subscribe to a single topic ( with
> one partition ).
>
> I have a requirement such that,
> 1. The producer will post to 1 topic having 1 partition.
> 1. *20,000-30,000 consumers *should to be able to consume messages.
> 2. All of the consumers should receive all messages that are produced.
>
> will this be possible using kafka ?
> is there a limit on number of consumers ?
> what will be the number of machines that I will need to satisfy the
> requirements ?
>
> --
> Regards,
> Vaibhav Kirte
>

Re: new consumer api?

[Jun Rao]

Jalpesh,

We are still iterating on the new consumer a bit and are waiting for some
of the security jiras to be committed. So now, we are shooting for
releasing 0.8.3 in Oct (just updated
https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan).

Thanks,

Jun

On Mon, Aug 3, 2015 at 8:41 AM, Jalpesh Patadia <
jalpesh.pata...@clickbank.com> wrote:

> Hello guys,
>
> A while ago i read that the new consumer api was going to be released
> sometime in July as part of the 0.8.3/0.9 release.
> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
>
>
> Do we have an update when we think that can happen?
>
>
> Thanks,
>
> Jalpesh
>
>
> -- PRIVILEGED AND CONFIDENTIAL This transmission may contain privileged,
> proprietary or confidential information. If you are not the intended
> recipient, you are instructed not to review this transmission. If you are
> not the intended recipient, please notify the sender that you received this
> message and delete this transmission from your system.
>

Re: 0.8.3 ETA?

[Jun Rao]

Hi, Stevo,

Yes, we are still iterating on the new consumer a bit and are waiting for
some of the security jiras to be committed. So now, we are shooting for
releasing 0.8.3 in Oct (just updated
https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan).

As we are getting closer, we will clean up the 0.8.3 jiras and push
non-critical ones to future releases.

Thanks,

Jun

On Mon, Aug 3, 2015 at 5:52 AM, Stevo Slavić  wrote:

> Hello Apache Kafka community,
>
> If I recall well, two weeks ago it was mentioned in a discussion that Kafka
> 0.8.3 might be released in a month time.
>
> Is this still Kafka dev team goal, in few weeks time to have Kafka 0.8.3
> released? Or is more (re)work (e.g. more new consumer API changes) planned
> for 0.8.3 than already in JIRA, which would further delay 0.8.3 release?
>
> Btw, Kafka JIRA has quite a lot unresolved tickets targeting 0.8.3 as fix
> version (see here
> <
> https://issues.apache.org/jira/browse/KAFKA-1853?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%200.8.3%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20due%20ASC%2C%20priority%20DESC%2C%20created%20ASC
> >
> complete list).
>
> Kind regards,
> Stevo Slavic.
>

Consumer limit for pub-sub mode

[Vaibhav Kirte]

Hi,

I need to know how many consumers can subscribe to a single topic ( with
one partition ).

I have a requirement such that,
1. The producer will post to 1 topic having 1 partition.
1. *20,000-30,000 consumers *should to be able to consume messages.
2. All of the consumers should receive all messages that are produced.

will this be possible using kafka ?
is there a limit on number of consumers ?
what will be the number of machines that I will need to satisfy the
requirements ?

-- 
Regards,
Vaibhav Kirte

Lead Broker from kafka.message.MessageAndMetadata

[Sreenivasulu Nallapati]

Hello,

Is there a way that we can find the lead broker
from kafka.message.MessageAndMetadata class?

My use case is simple, I have topic and partition and wanted to find out
the lead broker for that partition.

Please provide your insights


Thanks
Sreeni

new consumer api?

[Jalpesh Patadia]

Hello guys,

A while ago i read that the new consumer api was going to be released
sometime in July as part of the 0.8.3/0.9 release.
https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan


Do we have an update when we think that can happen?


Thanks,

Jalpesh


-- PRIVILEGED AND CONFIDENTIAL This transmission may contain privileged, 
proprietary or confidential information. If you are not the intended recipient, 
you are instructed not to review this transmission. If you are not the intended 
recipient, please notify the sender that you received this message and delete 
this transmission from your system.

0.8.3 ETA?

[Stevo Slavić]

Hello Apache Kafka community,

If I recall well, two weeks ago it was mentioned in a discussion that Kafka
0.8.3 might be released in a month time.

Is this still Kafka dev team goal, in few weeks time to have Kafka 0.8.3
released? Or is more (re)work (e.g. more new consumer API changes) planned
for 0.8.3 than already in JIRA, which would further delay 0.8.3 release?

Btw, Kafka JIRA has quite a lot unresolved tickets targeting 0.8.3 as fix
version (see here

complete list).

Kind regards,
Stevo Slavic.

Kafka Zookeeper Issues

[Wollert, Fabian]

hi everyone,

we are trying to deploy Kafka 0.8.2.1 and Zookeeper on AWS using
Cloudformation, ASG's and other Services. For Zookeeper we are using
Netflix' Exhibitor (V 1.5.5) to ensure failover stability.

What we are observing right now is that after some days our Brokers are not
registered anymore in the "/brokers/ids" path in Zookeeper. I was trying to
see when they get deleted to check the logs, but the ZK Transaction logs
only shows the create stmt, no deletes or something (though deletes are
written down there). Can someone explain me how the mechanism works with
registering and deregistering in Zookeeper or point me to a doc or even
source code, where this happens? Or some one has even some idea what
happens there.

Any experience on what to take care of deploying kafka on AWS (or generally
a cloud env) would be also helpful.

Cheers

-- 
*Fabian Wollert*
Business Intelligence

*POSTAL ADDRESS*
Zalando SE
11501 Berlin

*OFFICE*
Zalando SE
Mollstraße 1
10178 Berlin
Germany

Phone: +49 30 20968 1819
Fax:   +49 30 27594 693
E-Mail: fabian.woll...@zalando.de
Web: www.zalando.de
Jobs: jobs.zalando.de

Zalando SE, Tamara-Danz-Straße 1, 10243 Berlin
Company registration: Amtsgericht Charlottenburg, HRB 158855 B
Tax ID: 29/560/00596 * VAT registration number: DE 260543043
Management Board: Robert Gentz, David Schneider, Rubin Ritter
Chairperson of the Supervisory Board: Cristina Stenbeck
Registered office: Berlinn