Re: New to Apache Kafka

[Li Tao]

As a cluster, each none in the cluster should know each other to function
properly. For a Cassandra cluster(I don't know ActiveMQ's mechanism), as an
example, it has its own
protocal to communicate with each other to know their condition.

For Kafka, each node stays independantly, they use zookeeper to know the
condition of the whole cluster. Each node communicates with zookeeper.

On Wed, Sep 16, 2015 at 7:24 AM, Ashish Shenoy 
wrote:

> Zookeeper is a distributed coordination service. Kafka uses Zookeeper for
> various things like leader election, storing consumer-partition offsets
> etc.
>
> More information on each service is available at
> http://kafka.apache.org/documentation.html and
> https://zookeeper.apache.org/
> I highly recommend reading them for more information.
>
> TLDR:
> http://stackoverflow.com/questions/23751708/kafka-is-zookeeper-a-must
>
> Thanks,
> Ashish
>
> On Tue, Sep 15, 2015 at 1:45 PM, Joe San  wrote:
>
> > Hi Apache Kafka,
> >
> > I'm evaluating Apache Kafka for one of the projects that I'm into. I have
> > used ActiveMQ in the past which makes using Kafka pretty straightforward.
> > One thing that I do not understand is the need for Zookeeper?
> >
> > I understand what Zookeeper is, but I fail to understand what purpose it
> > solves with Apache Kafka. Is it needed for the Apache Kafka cluster
> > management? Could you please elaborate a bit on this?
> >
> > Regards,
> > Joe
> >
>

Re: kafka consumer thread killed - no more message consumption

[Li Tao]

Hi, I think there is no difference between shutting down a consumer or
killing a consumer.
For the whole system, it only means a consumer has left for some reason
which is not interested.

So if you kill a consumer, some consumer in the same consumer group should
take over and consume messages.
Correct me if I am wrong.

On Tue, Sep 15, 2015 at 7:46 PM, Jürgen Ziegler  wrote:

> Hi all,
>
> I'm playing around with the kafka high level java api.
>
> If I have multiple consumers in a group, consuming the same topic with a
> single partition, only one consumer will receive messages, as is expected.
> When shutting down the consumer, another consumer will automatically
> consume the messages.
> If the picked consumer thread dies (killed, not shutdown!), is it the
> expected behavior that no other consumer will consume the leftover
> messages?
> That's at least what I'm experiencing, which seems weird to me.
>
> Thanks!
>

Re: Failed to Send Message After 3 Tries

[Helleren, Erik]

So, this is expected behavior on the producer when its unable to
communicate with the kafka broker that is the leader for the message is
being sent.  

First, if the design of your app allows, try to migrate to the new
producer API release is 0.8.1  It is fully asynchronous, and provides
callbacks on completion or error cases.  The documentation makes a clear
case for its use. (Easier and more efficient)  And since your server is
already on 0.8.2, you should run the client jar for 0.8.2 as well.

If the design does not, the app may either manually catch and retry the
message, or you can increase the number of times the kafka library will
try to resend using the producer property.

There is no way to prevent retries, its up to the app to handle them
properly at the risk of message loss, retransmission, or hung processes.
-Erik


On 9/15/15, 1:17 PM, "Leng, Hao"  wrote:

>Hi there,
>
>We had error logs for three messages failed to produce to Kafka during
>last week. All three are failed on the same day within one hour range. We
>checked Kafka logs (server.log and statechange.log) but found no abnormal
>behaviors.
>
>The exception is :
>kafka.common.FailedToSendMessageException: Failed to send
>messages after 3 tries.
>at kafka.producer.async.DefaultEventHandler.handle(Unknown Source)
>at kafka.producer.Producer.send(Unknown Source)
>at kafka.javaapi.producer.Producer.send(Unknown Source)
>
>The Kafka server we use is 0.8.2-beta and our kafka jar is
>kafka_2.10.0-0.8.0-beta1.jar. On producer properties, we use
>producer.type = sync and request.required.acks = 1. The rest are default
>values. There is no activity needed and the system becomes normal again.
>
>Can you please suggest what may cause this issue and how we can avoid it?
>Thank you and have a good day.
>
>Sincerely,
>Hao Leng
>

Dead letter queue support in Kafka

[Alexander Pakulov]

Hi everyone,

Since Kafka doesn’t have a dead-letter queue support built in - I’m looking for 
advice and best approaches to handle bad messages or cases when system is going 
crazy, once you receive an exception it basically means you’re blocking the 
whole kaka-stream from consuming other messages in the partition.

There are couple of ideas that potentially could address the problem:
1) Wrap message processing logging into try-catch block -> log the error if 
there is an exception and proceed to the next messages. Which basically mean 
just swallow the message.
2) Dead-letter queue could be implemented on top of Kafka itself, the same 
try-catch block -> if there is an exception you forward broken message to 
dead-letter topic and consume it later once all issues are resolved.

Any feedback and suggestions are highly welcomed.

Thank you!
-- 
Alexander Pakulov

Dead letter queue support in Kafka

[Alexander Pakulov]

Hi everyone,

Since Kafka doesn’t have a dead-letter queue support built in - I’m looking for 
advice and best approaches to handle bad messages or cases when system is going 
crazy, once you receive an exception it basically means you’re blocking the 
whole kaka-stream from consuming other messages in the partition.

There are couple of ideas that potentially could address the problem:
1) Wrap message processing logging into try-catch block -> log the error if 
there is an exception and proceed to the next messages. Which basically mean 
just swallow the message.
2) Dead-letter queue could be implemented on top of Kafka itself, the same 
try-catch block -> if there is an exception you forward broken message to 
dead-letter topic and consume it later once all issues are resolved.

Any feedback and suggestions are highly welcomed.

Thank you!
-- 
Alexander Pakulov

Re: New to Apache Kafka

[Ashish Shenoy]

Zookeeper is a distributed coordination service. Kafka uses Zookeeper for
various things like leader election, storing consumer-partition offsets etc.

More information on each service is available at
http://kafka.apache.org/documentation.html and https://zookeeper.apache.org/
I highly recommend reading them for more information.

TLDR: http://stackoverflow.com/questions/23751708/kafka-is-zookeeper-a-must

Thanks,
Ashish

On Tue, Sep 15, 2015 at 1:45 PM, Joe San  wrote:

> Hi Apache Kafka,
>
> I'm evaluating Apache Kafka for one of the projects that I'm into. I have
> used ActiveMQ in the past which makes using Kafka pretty straightforward.
> One thing that I do not understand is the need for Zookeeper?
>
> I understand what Zookeeper is, but I fail to understand what purpose it
> solves with Apache Kafka. Is it needed for the Apache Kafka cluster
> management? Could you please elaborate a bit on this?
>
> Regards,
> Joe
>

Re: 0.9.0.0 remaining jiras

[Jason Rosenberg]

I'd be interested to see:

https://issues.apache.org/jira/browse/KAFKA-2434  (has patch available, we
will be using 'old' consumer for some time)
https://issues.apache.org/jira/browse/KAFKA-2125  (seems rather serious,
unclear if no longer relevant with new code?)
https://issues.apache.org/jira/browse/KAFKA-  (seems important to fix)

Jason

On Tue, Sep 15, 2015 at 3:28 PM, Jason Rosenberg  wrote:

> Yep,
>
> It looks like this was only communicated originally to the dev list (and
> not the users list), so it wasn't obvious to all!
>
> Thanks,
>
> Jason
>
> On Mon, Sep 14, 2015 at 12:43 AM, Stevo Slavić  wrote:
>
>> Hello Jason,
>>
>> Maybe this answers your question:
>>
>> http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAFc58G-UScVKrSF1kdsowQ8Y96OAaZEdiZsk40G8fwf7iToFaw%40mail.gmail.com%3E
>>
>> Kind regards,
>> Stevo Slavic.
>>
>>
>> On Mon, Sep 14, 2015 at 8:56 AM, Jason Rosenberg 
>> wrote:
>>
>> > Hi Jun,
>> >
>> > Can you clarify, will there not be a 0.8.3.0 (and instead we move
>> straight
>> > to 0.9.0.0)?
>> >
>> > Also, can you outline the man new features/updates for 0.9.0.0?
>> >
>> > Thanks,
>> >
>> > Jason
>> >
>> > On Sat, Sep 12, 2015 at 12:40 PM, Jun Rao  wrote:
>> >
>> > > The following is a candidate list of jiras that we want to complete in
>> > the
>> > > upcoming release (0.9.0.0). Our goal is to finish at least all the
>> > blockers
>> > > and as many as the non-blockers possible in that list.
>> > >
>> > >
>> > >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20fixVersion%20%3D%200.9.0.0
>> > >
>> > > Anything should be added/removed from this list?
>> > >
>> > > We are shooting to cut an 0.9.0.0 release branch in early October.
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> >
>>
>
>

Failed to Send Message After 3 Tries

[Leng, Hao]

Hi there,

We had error logs for three messages failed to produce to Kafka during last 
week. All three are failed on the same day within one hour range. We checked 
Kafka logs (server.log and statechange.log) but found no abnormal behaviors.

The exception is :
kafka.common.FailedToSendMessageException: Failed to send messages 
after 3 tries.
at kafka.producer.async.DefaultEventHandler.handle(Unknown Source)
at kafka.producer.Producer.send(Unknown Source)
at kafka.javaapi.producer.Producer.send(Unknown Source)

The Kafka server we use is 0.8.2-beta and our kafka jar is 
kafka_2.10.0-0.8.0-beta1.jar. On producer properties, we use producer.type = 
sync and request.required.acks = 1. The rest are default values. There is no 
activity needed and the system becomes normal again.

Can you please suggest what may cause this issue and how we can avoid it? Thank 
you and have a good day.

Sincerely,
Hao Leng

New to Apache Kafka

[Joe San]

Hi Apache Kafka,

I'm evaluating Apache Kafka for one of the projects that I'm into. I have
used ActiveMQ in the past which makes using Kafka pretty straightforward.
One thing that I do not understand is the need for Zookeeper?

I understand what Zookeeper is, but I fail to understand what purpose it
solves with Apache Kafka. Is it needed for the Apache Kafka cluster
management? Could you please elaborate a bit on this?

Regards,
Joe

Create topic fails

[Chittaranjan Hota]

Hi,
We run a 10 node cluster in production with 5 zk nodes.
The cluster is operating fine without issues with other topics(we have
close to 10 topics).

However, when we try to create new topic, it doesnt go through
successfully. Have tried it couple of times with same result.
Topic shows as created, but the ISR is empty and Leader is none.

We are using 0.8.2.1 in production.

[kafka@kafka10a bin]$ ./kafka-topics.sh --create -topic 
--partitions 10 --replication-factor 2 --config retention.ms=1800
--zookeeper zkhost:2181

Created topic "".

[kafka@kafka10a bin]$ ./kafka-topics.sh --describe
-topic  --zookeeper zkhost:2181

Topic:  PartitionCount:10 ReplicationFactor:2 Configs:
retention.ms=1800

Topic:  Partition: 0 Leader: none Replicas: 1,2 Isr:

Topic:  Partition: 1 Leader: none Replicas: 2,3 Isr:

Topic:  Partition: 2 Leader: none Replicas: 3,4 Isr:

Topic:  Partition: 3 Leader: none Replicas: 4,5 Isr:

Topic:  Partition: 4 Leader: none Replicas: 5,6 Isr:

Topic:  Partition: 5 Leader: none Replicas: 6,7 Isr:

Topic:  Partition: 6 Leader: none Replicas: 7,8 Isr:

Topic:  Partition: 7 Leader: none Replicas: 8,9 Isr:

Topic:  Partition: 8 Leader: none Replicas: 9,1 Isr:

Topic:  Partition: 9 Leader: none Replicas: 1,3 Isr:

Any inputs would be very helpful.
Disk is just 20% full and there is a lot of available FDs.

Thanks !

Re: 0.9.0.0 remaining jiras

[Jason Rosenberg]

Yep,

It looks like this was only communicated originally to the dev list (and
not the users list), so it wasn't obvious to all!

Thanks,

Jason

On Mon, Sep 14, 2015 at 12:43 AM, Stevo Slavić  wrote:

> Hello Jason,
>
> Maybe this answers your question:
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201509.mbox/%3CCAFc58G-UScVKrSF1kdsowQ8Y96OAaZEdiZsk40G8fwf7iToFaw%40mail.gmail.com%3E
>
> Kind regards,
> Stevo Slavic.
>
>
> On Mon, Sep 14, 2015 at 8:56 AM, Jason Rosenberg  wrote:
>
> > Hi Jun,
> >
> > Can you clarify, will there not be a 0.8.3.0 (and instead we move
> straight
> > to 0.9.0.0)?
> >
> > Also, can you outline the man new features/updates for 0.9.0.0?
> >
> > Thanks,
> >
> > Jason
> >
> > On Sat, Sep 12, 2015 at 12:40 PM, Jun Rao  wrote:
> >
> > > The following is a candidate list of jiras that we want to complete in
> > the
> > > upcoming release (0.9.0.0). Our goal is to finish at least all the
> > blockers
> > > and as many as the non-blockers possible in that list.
> > >
> > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20fixVersion%20%3D%200.9.0.0
> > >
> > > Anything should be added/removed from this list?
> > >
> > > We are shooting to cut an 0.9.0.0 release branch in early October.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> >
>

Re: Question on Kafka Replication

[Todd Palino]

I put an answer to this on Stack Overflow. Basically, that's not how RF
works for Kafka. It's not a guarantee, it's just how the partitions are
created, and how it is reported when something is down (under replicated
partitions). While there is an option to do auto leader rebalancing,
there's no equivalent option for auto partition migration. It's a fairly
resource intensive task, moving all that data around. If you want to move
replicas around when something is down, you have to do it manually.

That said, it may be interesting to consider.

-Todd


On Tue, Sep 15, 2015 at 7:47 AM, Dhyan  wrote:

> Hi All,
>
> Below is my partition information for the topic **xx_json_topic** .This is
> a Kafka cluster with three nodes .
>
> All nodes up :
>
> Topic: xx_json_topicPartitionCount:4
> ReplicationFactor:2Configs:
> Topic: xx_json_topicPartition: 0Leader: 1   Replicas:
> 3,1   Isr: 3,1
> Topic: xx_json_topicPartition: 1Leader: 2   Replicas:
> 1,2   Isr: 2,1
> Topic: xx_json_topicPartition: 2Leader: 2   Replicas:
> 2,3   Isr: 2,3
> Topic: xx_json_topicPartition: 3Leader: 3   Replicas:
> 3,2   Isr: 2,3
>
> At this point..  if i bring down the node "node-1" ..It looks like below :
>
> Topic: xx_json_topicPartitionCount:4
> ReplicationFactor:2Configs:
> Topic: xx_json_topicPartition: 0Leader: 3   Replicas:
> 3,1   Isr: 3
> Topic: xx_json_topicPartition: 1Leader: 2   Replicas:
> 1,2   Isr: 2
> Topic: xx_json_topicPartition: 2Leader: 2   Replicas:
> 2,3   Isr: 2,3
> Topic: xx_json_topicPartition: 3Leader: 3   Replicas:
> 3,2   Isr: 2,3
>
> My question is ..if kafka knows that the node-1 is down and it needs to
> maintain the replication factor-2 ,wouldn't it make node 3 a replica for
> partition-1 && node-2 a replica for partition-0  then make node-3 and
> node-2 part of their Isr ?
>
> Or you think Kafka doesn't promise that...
> If replication factor is 2 ..It doesn't mean that data will be available
> in atleast 2 nodes at all time(---like consistency level in Cassandra) .
>
> I also have this question posted below :
>
> http://stackoverflow.com/questions/32588784/would-kafka-create-a-new-follower-if-one-of-its-replica-is-down-to-keep-up-with
> <
> http://stackoverflow.com/questions/32588784/would-kafka-create-a-new-follower-if-one-of-its-replica-is-down-to-keep-up-with
> >
>
> —Dhyan

Re: port already in use error when trying to add topic

[allen chan]

Moving the export statement to kafka-server.start.sh fixed the issue. I was
able to start kafka with JMX monitoring and run kafka-topics.sh.

Thanks Lance.



On Mon, Sep 14, 2015 at 6:43 PM, Lance Laursen 
wrote:

> This is not a bug. The java process spawned by kafka-topics.sh is trying to
> bind to 9998 upon start. The java process spawned by kafka-server-start.sh
> already owns that port. It's doing this because both of these scripts use
> kafka-run-class.sh and that is where you defined your 'export JMX_PORT'.
>
> Put your export statement into kafka-server-start.sh instead and run
> kafka-topics.sh using a separate terminal or user account. Also, google
> search "linux environment variables." You could also just run
> kafka-topics.sh from a separate host, such as your workstation, so long as
> it can see zookeeper:2181.
>
> On Mon, Sep 14, 2015 at 3:52 PM, allen chan 
> wrote:
>
> > After completely disabling JMX settings, i was able to create topics.
> Seems
> > like there is an issue with using JMX with the product. Should i create
> > bug?
> >
> > On Sun, Sep 13, 2015 at 9:07 PM, allen chan <
> allen.michael.c...@gmail.com>
> > wrote:
> >
> > > Changing the port to 9998 did not help. Still the same error occurred
> > >
> > > On Sat, Sep 12, 2015 at 12:27 AM, Foo Lim  wrote:
> > >
> > >> Try throwing
> > >>
> > >> JMX_PORT=9998
> > >>
> > >> In front of the command. Anything other than 9994
> > >>
> > >> Foo
> > >>
> > >> On Friday, September 11, 2015, allen chan <
> allen.michael.c...@gmail.com
> > >
> > >> wrote:
> > >>
> > >> > Hi all,
> > >> >
> > >> > First time testing kafka with brand new cluster.
> > >> >
> > >> > Running into an issue that i do not understand.
> > >> >
> > >> > Server started up fine but I get error when trying to create a
> topic.
> > >> >
> > >> > *[achan@server1 ~]$ ps -ef | grep -i kafka*
> > >> > *root  6507 1  0 15:42 ?00:00:00 sudo
> > >> > /opt/kafka_2.10-0.8.2.1/bin/kafka-server-start.sh
> > >> > /opt/kafka_2.10-0.8.2.1/config/server.properties*
> > >> > *root  6508  6507  0 15:42 ?00:00:36 java -Xmx1G -Xms1G
> > >> -server
> > >> > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> -XX:+CMSClassUnloadingEnabled
> > >> > -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC
> > >> > -Djava.awt.headless=true
> > >> > -Xloggc:/opt/kafka_2.10-0.8.2.1/bin/../logs/kafkaServer-gc.log
> > >> -verbose:gc
> > >> > -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
> > >> > -Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=server1
> > >> > -Dcom.sun.management.jmxremote.authenticate=false
> > >> > -Dcom.sun.management.jmxremote.ssl=false
> > >> > -Dcom.sun.management.jmxremote.port=9994
> > >> > -Dkafka.logs.dir=/opt/kafka_2.10-0.8.2.1/bin/../logs
> > >> >
> > >> >
> > >>
> >
> -Dlog4j.configuration=file:/opt/kafka_2.10-0.8.2.1/bin/../config/log4j.properties
> > >> > -cp
> > >> >
> > >> >
> > >>
> >
> :/opt/kafka_2.10-0.8.2.1/bin/../core/build/dependant-libs-2.10.4*/*.jar:/opt/kafka_2.10-0.8.2.1/bin/../examples/build/libs//kafka-examples*.jar:/opt/kafka_2.10-0.8.2.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/opt/kafka_2.10-0.8.2.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/opt/kafka_2.10-0.8.2.1/bin/../clients/build/libs/kafka-clients*.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/jopt-simple-3.2.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-javadoc.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-scaladoc.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-sources.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-test.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka-clients-0.8.2.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/log4j-1.2.16.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/lz4-1.2.0.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/scala-library-2.10.4.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/slf4j-api-1.7.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/slf4j-log4j12-1.6.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/snappy-java-1.1.1.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/zkclient-0.3.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/zookeeper-3.4.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../core/build/libs/kafka_2.10*.jar
> > >> > kafka.Kafka /opt/kafka_2.10-0.8.2.1/config/server.properties*
> > >> >
> > >> >
> > >> > *[achan@server1 ~]$ sudo
> /opt/kafka_2.10-0.8.2.1/bin/kafka-topics.sh
> > >> > --create –zookeeper server1:2181 –partition 3 –replica 0 –topic
> > >> test.logs*
> > >> > *[sudo] password for achan:*
> > >> > *Error: Exception thrown by the agent :
> > java.rmi.server.ExportException:
> > >> > Port already in use: 9994; nested exception is:*
> > >> > * java.net.BindException: Address already in use*
> > >> >
> > >> > I have pretty much the manila kakfa-run-class.sh except i added the
> > >> > following
> > >> > *export JMX_PORT=9994* (near the top of th

Durability and Integrity for long standing key-compacted log.

[Bart Wyatt]

Hello,

We have a set of processing jobs (in samza) using key compacted Kafka logs as a 
durable Key-Value store.  Recently, after some network troubles that resulted 
in various parts of the infrastructure rebooting, we discovered that a key that 
we expected to be "alive" was compacted out of the log.

Because of the nature of the outage and our current level of logging, it is 
impossible to know whether the application level was at fault and send an 
erroneous tombstone to Kafka or if Kafka's cleaner was at fault however, it got 
me thinking whether it was good practice to use Kafka as a long term backing 
for a Key Value store.

Are there best practices concerning data loss and integrity when expecting 
certain messages to live "forever" and never be reaped/compacted?  It seems 
like the basic log abstraction can assume that messages only have to live for 
their contracted amount of time/space however, with the key compacted logs this 
can be defeated perpetually.

FWIW, we are deployed on top of ZFS in mirrored mode.

-Bart



This e-mail may contain CONFIDENTIAL AND PROPRIETARY INFORMATION and/or 
PRIVILEGED AND CONFIDENTIAL COMMUNICATION intended solely for the recipient 
and, therefore, may not be retransmitted to any party outside of the 
recipient's organization without the prior written consent of the sender. If 
you have received this e-mail in error please notify the sender immediately by 
telephone or reply e-mail and destroy the original message without making a 
copy. Deep Silver, Inc. accepts no liability for any losses or damages 
resulting from infected e-mail transmissions and viruses in e-mail attachments.

Question on Kafka Replication

[Dhyan]

Hi All,

Below is my partition information for the topic **xx_json_topic** .This is a 
Kafka cluster with three nodes .

All nodes up :

Topic: xx_json_topicPartitionCount:4ReplicationFactor:2 
   Configs:
Topic: xx_json_topicPartition: 0Leader: 1   Replicas: 3,1   
Isr: 3,1
Topic: xx_json_topicPartition: 1Leader: 2   Replicas: 1,2   
Isr: 2,1
Topic: xx_json_topicPartition: 2Leader: 2   Replicas: 2,3   
Isr: 2,3
Topic: xx_json_topicPartition: 3Leader: 3   Replicas: 3,2   
Isr: 2,3

At this point..  if i bring down the node "node-1" ..It looks like below :  

Topic: xx_json_topicPartitionCount:4ReplicationFactor:2 
   Configs:
Topic: xx_json_topicPartition: 0Leader: 3   Replicas: 3,1   
Isr: 3
Topic: xx_json_topicPartition: 1Leader: 2   Replicas: 1,2   
Isr: 2
Topic: xx_json_topicPartition: 2Leader: 2   Replicas: 2,3   
Isr: 2,3
Topic: xx_json_topicPartition: 3Leader: 3   Replicas: 3,2   
Isr: 2,3

My question is ..if kafka knows that the node-1 is down and it needs to 
maintain the replication factor-2 ,wouldn't it make node 3 a replica for 
partition-1 && node-2 a replica for partition-0  then make node-3 and node-2 
part of their Isr ?

Or you think Kafka doesn't promise that... 
If replication factor is 2 ..It doesn't mean that data will be available in 
atleast 2 nodes at all time(---like consistency level in Cassandra) .

I also have this question posted below :
http://stackoverflow.com/questions/32588784/would-kafka-create-a-new-follower-if-one-of-its-replica-is-down-to-keep-up-with
 


—Dhyan

Re: New Consumer & committed offsets

[Stevo Slavić]

Hello Damian,

Yes, there's a +1 difference. See related discussion
http://mail-archives.apache.org/mod_mbox/kafka-users/201507.mbox/%3CCAOeJiJh2SMzVn23JsoWiNk3sfsw82Jr_-kRLcNRd-oZ7pR1yWg%40mail.gmail.com%3E

Kind regards,
Stevo Slavic.

On Tue, Sep 15, 2015 at 3:56 PM, Damian Guy  wrote:

> I turned off compression and still get duplicates, but only 1 from each
> topic.
> Should the initial fetch offset for a partition be committed offset +1 ?
>
> Thanks,
> Damian
>
> On 15 September 2015 at 14:07, Damian Guy  wrote:
>
> > Hi,
> >
> > I've been trying out the new consumer and have noticed that i get
> > duplicate messages when i stop the consumer and then restart (different
> > processes, same consumer group).
> >
> > I consume all of the messages on the topic and commit the offsets for
> each
> > partition and stop the consumer. On the next run i expect to get 0
> > messages, however i get a batch of records from each partition - in this
> > case works out 1020 messages. Run it again and i get the same batch of
> > records.
> >
> > My logging shows that i've received messages with offsets lower than were
> > previously committed.
> >
> > committed:{damian_test_one-2=137669}
> > committed:{damian_test_one-0=139161}
> > committed:{damian_test_one-1=137663}
> >
> > min offsets received: {damian_test_one-0=138824,
> damian_test_one-1=137321,
> > damian_test_one-2=137331}
> >
> > I've debugged the initial fetch requests for offsets and the offsets
> match
> > up with what has been committed. Is this expected behaviour? Something to
> > do with batching of compression of message sets?
> >
> > TIA,
> > Damian
> >
>

Re: New Consumer & committed offsets

[Damian Guy]

I turned off compression and still get duplicates, but only 1 from each
topic.
Should the initial fetch offset for a partition be committed offset +1 ?

Thanks,
Damian

On 15 September 2015 at 14:07, Damian Guy  wrote:

> Hi,
>
> I've been trying out the new consumer and have noticed that i get
> duplicate messages when i stop the consumer and then restart (different
> processes, same consumer group).
>
> I consume all of the messages on the topic and commit the offsets for each
> partition and stop the consumer. On the next run i expect to get 0
> messages, however i get a batch of records from each partition - in this
> case works out 1020 messages. Run it again and i get the same batch of
> records.
>
> My logging shows that i've received messages with offsets lower than were
> previously committed.
>
> committed:{damian_test_one-2=137669}
> committed:{damian_test_one-0=139161}
> committed:{damian_test_one-1=137663}
>
> min offsets received: {damian_test_one-0=138824, damian_test_one-1=137321,
> damian_test_one-2=137331}
>
> I've debugged the initial fetch requests for offsets and the offsets match
> up with what has been committed. Is this expected behaviour? Something to
> do with batching of compression of message sets?
>
> TIA,
> Damian
>

Re: Jumbled up ISR

[Grant Henke]

The first replica in the ISR is the preferred replica, but is not required
to be the leader at all times. If you execute a preferred leader election,
or enable auto.leader.rebalance.enable, then replica 4 will become the
leader again.

More can be read here:

   - http://kafka.apache.org/documentation.html#basic_ops_leader_balancing
   -
   
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.PreferredReplicaLeaderElectionTool


Thank you,
Grant

On Tue, Sep 15, 2015 at 5:26 AM, Prabhjot Bharaj 
wrote:

> Hi,
>
> Topic:part_1_repl_3_3 PartitionCount:1 ReplicationFactor:3 Configs:
>
> Topic: part_1_repl_3_3 Partition: 0 Leader: 3 Replicas: 3,4,5 Isr: 4,3,5
>
>
> I see that the replicas are 3,4,5 but, the ISR is 4,3,5.
>
>
> I have this doubt:-
>
> When the leader is 3, can the ISR be 4, 3, 5 ?
>
> Does the ISR got to have the leader as the first replica ??
>
>
> Regards,
>
> Prabhjot
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

JMX metrices are not updated after deleting topics

[shanthu Venkatesan]

Hi all,

I have a cluster with 3 brokers. I've created a topic "test" with 3
partitions and replication factor 3. I produced 2 messages to
"test-2". Then I checked JMX metrices (LogEndOffset) which showed
2 for "test-2".

Now I deleted "test", the logs related to "test" are deleted in both
kafka and zookeeper but the JMX metrices were not updated. It still
showed the topic "test" and LogEndOffset for "test-2" was 2.

Again I created a topic with same name "test" with 5 partitions and
replication factor 3. I produced 1 messages to "test-1". When I
checked JMX metrices, the partitions were increased from 3 to 5 but
the LogEndOffset was not updated (ie) it was 2 for "test-2" and 0
for "test-1". Now I deleted this topic.

Again I created "test" with 3 partitions and replication factor 3. JMX
metrices showed 5 partitions for "test"(not updated as 3).

The changes are reflected only when I restarted all the brokers one by
one. Is there a solution to this where JMX metrices will be updated
without restarting the server.

Thanks in advance!

Regards,
Santhakumari.

New Consumer & committed offsets

[Damian Guy]

Hi,

I've been trying out the new consumer and have noticed that i get duplicate
messages when i stop the consumer and then restart (different processes,
same consumer group).

I consume all of the messages on the topic and commit the offsets for each
partition and stop the consumer. On the next run i expect to get 0
messages, however i get a batch of records from each partition - in this
case works out 1020 messages. Run it again and i get the same batch of
records.

My logging shows that i've received messages with offsets lower than were
previously committed.

committed:{damian_test_one-2=137669}
committed:{damian_test_one-0=139161}
committed:{damian_test_one-1=137663}

min offsets received: {damian_test_one-0=138824, damian_test_one-1=137321,
damian_test_one-2=137331}

I've debugged the initial fetch requests for offsets and the offsets match
up with what has been committed. Is this expected behaviour? Something to
do with batching of compression of message sets?

TIA,
Damian

Re: Unclean leader election docs outdated

[Stevo Slavić]

Created https://issues.apache.org/jira/browse/KAFKA-2551

On Mon, Sep 14, 2015 at 7:22 PM, Guozhang Wang  wrote:

> Yes you are right. Could you file a JIRA to edit the documents?
>
> Guozhang
>
> On Fri, Sep 11, 2015 at 4:41 PM, Stevo Slavić  wrote:
>
> > That sentence is in both
> > https://svn.apache.org/repos/asf/kafka/site/083/design.html and
> > https://svn.apache.org/repos/asf/kafka/site/082/design.html near the end
> > of
> > "Unclean leader election: What if they all die?" section. Next one,
> > "Availability and Durability Guarantees", mentions ability to disable
> > unclean leader election, so likely just this one reference needs to be
> > updated.
> >
> > On Sat, Sep 12, 2015 at 1:05 AM, Guozhang Wang 
> wrote:
> >
> > > Hi Stevo,
> > >
> > > Could you point me to the link of the docs?
> > >
> > > Guozhang
> > >
> > > On Fri, Sep 11, 2015 at 5:47 AM, Stevo Slavić 
> wrote:
> > >
> > > > Hello Apache Kafka community,
> > > >
> > > > Current unclean leader election docs state:
> > > > "In the future, we would like to make this configurable to better
> > support
> > > > use cases where downtime is preferable to inconsistency. "
> > > >
> > > > If I'm not mistaken, since 0.8.2, unclean leader election strategy
> > > (whether
> > > > to allow it or not) is already configurable via
> > > > unclean.leader.election.enable broker config property.
> > > >
> > > > Kind regards,
> > > > Stevo Slavic.
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

kafka consumer thread killed - no more message consumption

[Jürgen Ziegler]

Hi all,

I'm playing around with the kafka high level java api.

If I have multiple consumers in a group, consuming the same topic with a
single partition, only one consumer will receive messages, as is expected.
When shutting down the consumer, another consumer will automatically
consume the messages.
If the picked consumer thread dies (killed, not shutdown!), is it the
expected behavior that no other consumer will consume the leftover messages?
That's at least what I'm experiencing, which seems weird to me.

Thanks!

Jumbled up ISR

[Prabhjot Bharaj]

Hi,

Topic:part_1_repl_3_3 PartitionCount:1 ReplicationFactor:3 Configs:

Topic: part_1_repl_3_3 Partition: 0 Leader: 3 Replicas: 3,4,5 Isr: 4,3,5


I see that the replicas are 3,4,5 but, the ISR is 4,3,5.


I have this doubt:-

When the leader is 3, can the ISR be 4, 3, 5 ?

Does the ISR got to have the leader as the first replica ??


Regards,

Prabhjot