Re: FW: Mirrormaker stops consuming

Debbie, In our case, the issue was with custom SSL implementation and we need to fix a bug related to buffer over flow handling. This may not be applicable to you, if you are using open source kafka. The best way to troubleshoot this problem would be to take couple of thread dumps with few second

Does Kafka recover all data if node is reimaged

I was wondering if a kafka broker node get reimaged and all data is wiped off, Will kafka recover all data on node from replication? -- Thanks, Raja.

Re: Painfully slow kafka recovery / cluster breaking

ka, >>> and it should possibly get into sync faster without a steady stream of new >>> messages. >>> Kafka itself is working fine during this on the other hand, "just" >>> missing ISR, hence redundancy. If I stop another broker (clean!) though, it >>>

Re: Painfully slow kafka recovery

We are seeing same behavior in 5 broker cluster when losing one broker. In our case, we are losing broker as well as kafka data dir. Jörg Wagner, Are you losing just broker or kafka data dir as well? Gwen, We have also observed that latency of messages arriving at consumers goes up by 10x when

Re: java.lang.OutOfMemoryError: Java heap space

Hi Sourabh, We have seen this error, if kafka broker was running with SSL on Consumer is trying to consumer in plaintext mode. Are you using high level consumer or SimpleConsumer..? If you using using SimpleConsumer, pull latest code from my repo

Re: How to read in batch using HighLevel Consumer?

Here is an example on what sharninder suggested http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/ Thanks, Raja. On Tue, Aug 4, 2015 at 12:01 PM, Sharninder wrote: > You can't. Kafka is essentially a queue, so you always read messages one > by one. What you can d

Reimaging zookeeper host

We are running 3-node zookeeper cluster and we need to re-image (re-install os) on zookeeper host. Is it ok to lose zookeeper dataDir during upgrade or should back up zookeeper dataDir and restore when zookeeper comes backup online? Will kafka and consumers work fine if we bring up zookeeper with e

Re: how to modify offsets stored in Kafka in 0.8.2.1 version?

Hi Marina, Check slide 32 in this presentation . Hope this helps. Thanks, Raja. On Fri, Jun 19, 2015 at 9:43 AM, Marina wrote: > Thanks, Stevo, for the quick reply, > Yes, I understand how to do this programmatically - but I would

Re: Mirrormaker stops consuming

ou can also hook up jvisualvm > or yourkit to see which threads are active and it may well be that > producer in a tight loop on the writeCompletely. Just to confirm you > did not see this issue before upgrading? > > Joel > > On Fri, May 22, 2015 at 11:35:19AM -0400, Rajasek

Mirrormaker stops consuming

We recently upgraded to kafka 0.8.2.1 and found issues with mirrormaker that randomly stops consuming. We had to restart the mirrormaker process to resolve the problem. This problem has occurred several times in past two weeks. Here is what I found in analysis: When this problem happens: Mirrorm

Re: Kafka consumer offset checker hangs indefinitely

Hi Meghana, We also faced similar issue and found that it returned ConsumerCoordinatorNotAvailableCode always for one broker (server id 3) and leader for all partitions of __consumer_offsets topic is same broker id 3. So wiped off kafka data dir on that broker and restarted it. After that Consume

Re: How to set console consumer group ID

Yes, you pass any consumer property including group.id by having them in property file and passing path to it using --consumer.config of consumer consumer. Thanks, Raja. On Wed, Apr 22, 2015 at 1:45 AM, Lukáš Havrlant wrote: > Hi, > is it possible to set group ID for console consumer on command

Re: [ANNOUNCEMENT] Apache Kafka 0.8.2.0 Released

YaY!. Thanks to Jun and everybody who contributed to this release. We have been waiting for this release for a while. Thanks, Rajasekar Elango (Salesforce.com). On Tue, Feb 3, 2015 at 8:37 PM, Jun Rao wrote: > The Apache Kafka community is pleased to announce the release for Apache >

Re: kafka monitoring

Hi Sa Li, You need to set environment variable $JMX_PORT to enable jmx while starting kafka. See to kafka-run-class.sh on how it is used. Then you can connect to : using Jconsole. Thanks, Raja. On Thu, Jan 8, 2015 at 2:08 PM, Sa Li wrote: > Hello, All > > I understand many of you are using jm

Re: kafka monitoring system

Hi Sa Li, You can also try jmxtrans + graphite (for charting). jmxtrans has graphite output adapter out of the box. Regards, Raja. On Mon, Dec 22, 2014 at 10:39 PM, YuanJia Li wrote: > Hi Sa Li, > You can try to use jmxtrans+opentsdb to monitor kafka. Jmxtrans is > collecting data with JMX and

Re: Announcing Confluent

Congrats. Wish you all the very best and success. Thanks, Raja. On Thu, Nov 6, 2014 at 1:36 PM, Niek Sanders wrote: > Congrats! > > On Thu, Nov 6, 2014 at 10:28 AM, Jay Kreps wrote: > > Hey all, > > > > I’m happy to announce that Jun Rao, Neha Narkhede and I are creating a > > company around K

Re: MBeans, dashes, underscores, and KAFKA-1481

+1 on getting rid of quotes in jmx mbeans. Thanks, Raja. On Fri, Oct 17, 2014 at 4:48 PM, Neha Narkhede wrote: > +1 on getting rid of the quotes. > > On Fri, Oct 17, 2014 at 12:31 PM, Magnus Spångdal < > magnus.spang...@deltaprojects.com> wrote: > > > +1 to get rid of quotes, thanks! > > > > >

Re: [DISCUSS] Kafka Security Specific Features

tter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > / > > > On Wed, Jul 30, 2014 at 7:10 PM, Calvin Lei wrote: > > > yeah i just saw that. Looking forward to the prod release of 0.8.2 > > > > > >

Re: [DISCUSS] Kafka Security Specific Features

ucer.send(SyncProducer.scala:117) > > > > > > > > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) > > > > > > > > ... 12 more > > > > [2014-07-18 13:12:45,337] WARN Fetching topic metadata with > co

Re: KAFKA-1477 (authentication layer) and 0.8.2

Yes we are very much interested in getting this code merged to trunk. I can also do testing once it's available on trunk. Thanks, Raja. On Fri, Jul 25, 2014 at 12:11 PM, Joe Stein wrote: > Hi Chris, glad to hear that even more folks are going to (want to) use the > feature. I didn't author th

Re: [DISCUSS] Kafka Security Specific Features

ng.scala:106) > > at kafka.utils.Utils$.swallowError(Utils.scala:45) > > at > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67) > > at > > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) > > at > > ka

Re: [DISCUSS] Kafka Security Specific Features

Pramod, I presented secure kafka configuration and usage at last meet up. So hope this video recording would help. You can skip to about 59 min to jump to security talk. Thanks, Raja. On Wed, Jul 16, 2014 at 5:57 PM, Pramod Deshmukh wrote: > Hello Joe

Re: [DISCUSS] Kafka Security Specific Features

Hi Jay, Thanks for putting together a spec for security. Joe, Looks "Securing zookeeper.." part has been deleted from assumptions section. communication with zookeeper need to be secured as well to make entire kafka cluster secure. It may or may not require changes to kafka. But it's good to hav

Re: Spring integration?

Hi Michael, We are using spring integration kafkain production and have been working fine. We also contributed some features (for eg: support for topic filter

Re: Kafka and authentication

Hi Vijay, We implemented mutual ssl authentication in kafka for our internal use and we have plans to it contributed back to community. But we implemented SSL over older snapshot of version of kafka 0.8 release. We have been busy with other projects and haven't got chance to merge our ssl changes

Re: Spring Integration Kafka support

Hi Preetham, We are able to successfully use spring-integration-kafka with non-zero broker ids without any issues. Could you provide more details on what exactly problem/error you are getting.? Also, You can try this examples under samples

Re: Producer SSL?

Hi Jonathan We forked kafka to add SSL feature. It not part of kafka official release Sent from my iPhone On Nov 15, 2013, at 12:32 PM, Jonathan Hodges wrote: > Hi, > > While searching the user group messages I found the following thread - > http://grokbase.com/t/kafka/users/138vqq1x07/getting

Mirrormaker consumer looping to offset out of range and reset offset errors

We are seeing that mirrormaker consumer started looping through offset out of range and reset offset errors for some of partitions (2 out of 8 partitions). The consumerOffsetChecker reported very high Lag for these 2 partitions. Looks like this problem has started after a consumer rebalance. Here i

Re: JMX connections timing out in EC2 in Kafka 0.8

May be this helps. See last post on http://qnalist.com/questions/4522179/jmx Thanks, Raja. On Thu, Oct 3, 2013 at 11:17 AM, Aniket Bhatnagar < aniket.bhatna...@gmail.com> wrote: > I don't think this issue is related to Kafka at all but I just wanted to > try my luck in this user mailing list j

Re: Metadata API returns localhost.localdomain for one of the brokers in EC2

Hi Aniket, We had same issue it turns out that we need to make sure ip to hostname mapping should be correctly configured in /etc/hosts file. For eg: If you had something like 127.0.0.1 localhost localhost as first line in /etc/hosts file, you will get his error. To fix we need to ad

Re: Kafka consumer - Mbean for max lag

uot;([-.\w]+)-MaxLag",type="ConsumerFetcherManager". > Updated the website to reflect that. > > Thanks, > Neha > > > On Mon, Sep 23, 2013 at 12:48 PM, Rajasekar Elango > wrote: > > > In kafka documentation for > > monitoring<http://kafka.apache.org/docume

Kafka consumer - Mbean for max lag

In kafka documentation for monitoring. I see "we should be looking at max in messages among all partitions.". All I can see is mbeans *kafka.server".FetcherLagMetrics.*ConsumerFetcherThread* and it's value is pretty much 0. Is this the correct

Re: Leader doesn't get assigned for new topics

cation+tools#Replicationtools-7.StateChangeLogMergerTool > . > > Let me know if you run into any issues while using it. > > Thanks, > Neha > > > On Tue, Sep 17, 2013 at 12:27 PM, Rajasekar Elango > wrote: > > > Neha/Jun, > > > > The same proble

Re: Leader doesn't get assigned for new topics

der replicated (we have replication factor set to 3). Any clues on what could be issue and how can we get it back to working? Thanks, Raja. On Fri, Sep 13, 2013 at 6:26 PM, Neha Narkhede wrote: > Ah ok. Thanks for sharing that. > > > > On Fri, Sep 13, 2013 at 2:50 PM, Rajasekar Ela

Re: Leader doesn't get assigned for new topics

zookeepers in two different cluster. Thanks, Raja. On Fri, Sep 13, 2013 at 1:04 PM, Neha Narkhede wrote: > Just curious to know, what was the misconfiguration? > > > On Fri, Sep 13, 2013 at 10:02 AM, Rajasekar Elango > wrote: > > > Thanks Neha and Jun, It turned out to be m

Re: Leader doesn't get assigned for new topics

> Thanks, > > Jun > > > On Thu, Sep 12, 2013 at 4:56 PM, Rajasekar Elango >wrote: > > > We are seeing a problem that we we try to send messages to new topic it > > fails kafka.common.LeaderNotAvailableException. But usually this problem > > will be transient and

Leader doesn't get assigned for new topics

We are seeing a problem that we we try to send messages to new topic it fails kafka.common.LeaderNotAvailableException. But usually this problem will be transient and if we re-send messages to same topic will work. But now we tried rending message to same topic several time, but still fails with sa

Re: Mirror maker doesn't replicate new topics

offset.reset = smallest or largest? If it is > largest it could be due to this bug: > > https://issues.apache.org/jira/browse/KAFKA-1006 > > Guozhang > > > > On Tue, Sep 10, 2013 at 10:09 AM, Rajasekar Elango > wrote: > > > Hi Guozhang , > > >

Re: Mirror maker doesn't replicate new topics

r it only > shows this log for new topic? > > Guozhang > > > On Tue, Sep 10, 2013 at 7:47 AM, Rajasekar Elango >wrote: > > > Thanks Guozhang, > > > > 1, 2, 3 all are true. We are using default value 200 for > batch.num.messages > > and 5000ms queu

Re: Mirror maker doesn't replicate new topics

ce the consumer successfully > recognize all the partitions and start consuming. > > If both 2) and 3) is true, I would wonder if the batch size of the mirror > maker producer is large and hence will not send until enough messages are > accumulated at the producer queue. > > Guozhang

Re: Mirror maker doesn't replicate new topics

; On 9/5/13 8:38 PM, "Guozhang Wang" wrote: > > >Could you let me know the process of reproducing this issue? > > > >Guozhang > > > > > >On Thu, Sep 5, 2013 at 5:04 PM, Rajasekar Elango > >wrote: > > > >> Yes guozhang

Re: Mirror maker doesn't replicate new topics

Yes guozhang Sent from my iPhone On Sep 5, 2013, at 7:53 PM, Guozhang Wang wrote: > Hi Rajasekar, > > Is auto.create.topics.enable set to true in your target cluster? > > Guozhang > > > On Thu, Sep 5, 2013 at 4:39 PM, Rajasekar Elango > wrote: > >> We havi

Mirror maker doesn't replicate new topics

We having issues that mirormaker not longer replicate newly created topics. It continues to replicate data for existing topics and but new topics doesn't get created on target cluster. ConsumerOffsetTracker shows that new topics are successfully consumed and Lag is 0. But those topics doesn't get c

Re: Kafka Monitoring

Thanks a lot Jun. This is very helpful. Thanks, Raja. On Thu, Sep 5, 2013 at 1:12 AM, Jun Rao wrote: > Updated the doc at http://kafka.apache.org/documentation.html#monitoring > > Hopefully that answers your questions. > > Thanks, > > Jun > > > On Tue, Sep 3, 2013 at 11:16 PM, Vadim Keylis >

Re: Number of file handles increases indefinitely in producer if broker host is unresolvable

I will > take a closer look at this today. Do you mind filing a bug? > > Thanks, > Neha > On Sep 4, 2013 7:23 AM, "Rajasekar Elango" wrote: > > > I can easily reproduce this with console producer, If I run console > > producer with right hostname and if broker

Re: Kafka-0.8.0-beta1-src Has ObjectName starting with Double Quotes

We have the same problem, it doesn't work with restful JMX console jiminix. Is it possible to change kafka to expose mbeans without quotes? Thanks, Raja. On Wed, Sep 4, 2013 at 10:30 AM, Neha Narkhede wrote: > I had the same problem and could not use jmxterm t

Re: Number of file handles increases indefinitely in producer if broker host is unresolvable

, Sep 3, 2013 at 3:28 PM, Rajasekar Elango >wrote: > > > We found a issue that if broker host is un resolvable, the number of file > > handle keep increasing for every message we produce and eventually it > uses > > up all available files handles in operating system.

Number of file handles increases indefinitely in producer if broker host is unresolvable

We found a issue that if broker host is un resolvable, the number of file handle keep increasing for every message we produce and eventually it uses up all available files handles in operating system. If broker itself is not running and broker host name is resolvable, open file handles count stays

Re: Mirrormaker stopped consuming

hread dump and point us to it? Meanwhile, you might want > to fix the GC pauses. > > Thanks, > Neha > > > On Tue, Sep 3, 2013 at 8:59 AM, Rajasekar Elango >wrote: > > > We found that mirrormaker stopped consuming and producing over the week > end > > (09/01)

Mirrormaker stopped consuming

We found that mirrormaker stopped consuming and producing over the week end (09/01). Just seeing "Client session timed out" messages in mirrormaker log. I restarted to it today 09/03 to resume processing. Here is the logs line in reverse order. 2013-09-03 14:20:40,918 [mirrormakerProd_ops-mmrs1-1

Re: kafka produce API batching

gt; > Thanks, > > Jun > > > On Fri, Aug 30, 2013 at 10:28 AM, Rajasekar Elango > wrote: > > > Kafka producer API supports sending a single message and as well sending > > list of messages with two different methods. I believe irrespective of > > e

kafka produce API batching

Kafka producer API supports sending a single message and as well sending list of messages with two different methods. I believe irrespective of either of send method used, producer internally batches batch.num.messages and sends them in bulk. Is there any advantage in performance of using send(List

Re: Num of streams for consumers using TopicFilter.

a, we can do option 2. Thanks, Raja. On Fri, Aug 30, 2013 at 10:34 AM, Jun Rao wrote: > Right, but if you set #partitions in each topic to 16, you can use a total > of 16 streams. > > Thanks, > > Jun > > > On Thu, Aug 29, 2013 at 9:08 PM, Rajasekar Elango >wrote: &

Re: Num of streams for consumers using TopicFilter.

r. I was suggesting that > you use 16 partitions. Then you can use option 1) with 8 streams in each > consumer. > > Thanks, > > Jun > > > On Thu, Aug 29, 2013 at 8:51 PM, Rajasekar Elango >wrote: > > > Hi Jun, > > > > If you read my previous posts, b

Re: Num of streams for consumers using TopicFilter.

: > You can always use more partitions to get more parallelism in the > consumers. > > Thanks, > > Jun > > > On Thu, Aug 29, 2013 at 12:44 PM, Rajasekar Elango > wrote: > > > So what is best way to load balance multiple consumers consuming from > topic > >

Re: Securing kafka

* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > / > > > On Thu, Aug 29, 2013 at 11:22 PM,

Re: Securing kafka

We have made changes to kafka code to support certificate based mutual SSL authentication. So the clients and broker will exchange trusted certificates for successful communication. This provides both authentication and ssl encryption. Planning to contribute that code back to kafka soon. Thanks, R

Re: Num of streams for consumers using TopicFilter.

16 partitions, the remaining 8 threads will not do any > work. This is not ideal and we want to look into a better rebalancing > algorithm. Though it is a big change and we prefer doing it as part of the > consumer client rewrite. > > Thanks, > Neha > > > On Thu, Aug 29, 201

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

Created JIRA <https://issues.apache.org/jira/browse/KAFKA-1035> and attached patch to it. Please review. On Wed, Aug 28, 2013 at 1:11 PM, Guozhang Wang wrote: > I think this patch can be made in trunk. You can mark it as 0.8.1 > > Guozhang > > On Wednesday, August 28, 2

Num of streams for consumers using TopicFilter.

So my understanding is num of active streams that a consumer can utilize is number of partitions in topic. This is fine if we consumer from specific topic. But if we consumer from TopicFilter, I thought consumer should able to utilize (number of topics that match filter * number of partitions in to

Re: Changing the number of partitions after a topic is created

this tool can only increase the number of partitions for a > topic, there is no tool to reduce the number of partitions yet. > > Thanks, > Neha > > > On Wed, Aug 28, 2013 at 9:54 AM, Rajasekar Elango >wrote: > > > Hi Neha, > > > > If we set auto.create.to

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

s, > Neha > > > On Wed, Aug 28, 2013 at 8:36 AM, Rajasekar Elango >wrote: > > > Guozhang , > > > > *The documentation says I need to work off of trunk. Can you confirm If I > > should be working in trunk or different branch.* > > * > > *

Re: Changing the number of partitions after a topic is created

. On Wed, Aug 28, 2013 at 12:38 PM, Neha Narkhede wrote: > Hi Rajasekar, > > I'm not sure I fully understood your question. Are you asking about a tool > that can increase the number of partitions for a topic? > > Thanks, > Neha > > > On Wed, Aug 28, 2013 a

Re: Changing the number of partitions after a topic is created

Hi Jun, It's been some time since you last post, is this patch available now. Also we are doing following manual steps to update existing topic to use new partitions. 1) stop all zookeepers 2) stop all kafka brokers 3) clean data dir or zookeepers and kafka 4) start zookeepers and kafka. This in

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

che.org/confluence/display/KAFKA/FAQ > > Guozhang > > > On Tue, Aug 27, 2013 at 2:38 PM, Rajasekar Elango >wrote: > > > Thanks Guozhang, Changing max retry to 5 worked. Since I am changing > > console producer code, I can also submit patch adding both > > message.send.ma

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

cer does not have the > message.send.max.retries options yet. Could you file a JIRA for this and I > will followup to add this option? As for now you can hard modify the > default value from 3 to a larger number. > > Guozhang > > > On Tue, Aug 27, 2013 at 12:37 PM, Rajasekar

Re: Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

roducer gives up > too > > soon before the leader info ever propagated to producers, hence to > > producers also. Could you try to increase producer.num.retries and see if > > the producer can eventually succeed in re-trying? > > > > Guozhang > > > &

Getting LeaderNotAvailableException in console producer after increasing partitions from 4 to 16.

Hello everyone, We recently increased number of partitions from 4 to 16 and after that console producer mostly fails with LeaderNotAvailableException and exits after 3 tries: Here is last few lines of console producer log: No partition metadata for topic test-41 due to kafka.common.LeaderNotAvai

Re: Differences in size of data replicated by mirror maker

e compression rate could be 25/910. > > Guozhang > > > On Thu, Aug 22, 2013 at 3:48 PM, Rajasekar Elango >wrote: > > > Yes, both source and target clusters have 5 brokers in cluster. > > > > Sent from my iPhone > > > > On Aug 22, 2013, at 6:11 PM,

Fwd: Tuning mirror maker performance

) run more instances of producers. Metadata refresh interval is configurable. It's mainly for the producer to pick up newly available partitions. Thanks, Jun On Thu, Aug 22, 2013 at 1:44 PM, Rajasekar Elango wrote: > I am trying to tune mirrormaker configurations based on th

Re: Differences in size of data replicated by mirror maker

Yes, both source and target clusters have 5 brokers in cluster. Sent from my iPhone On Aug 22, 2013, at 6:11 PM, Guozhang Wang wrote: > Hello Rajasekar, > > Are the size of the source cluster and target cluster the same? > > Guozhang > > > On Thu, Aug 22, 2013 at 2

Differences in size of data replicated by mirror maker

Hi, We are using mirrormaker to replicate data between two kafka clusters. I am seeing huge difference in size of log in data dir between the broker in source cluster vs broker in destination cluster: For eg: Size of ~/data/Topic-0/ is about 910 G in source broker, but only its only 25G in destin

Re: Tuning mirror maker performance

traffic increases.? Thanks for help, Thanks, Raja. On Thu, Aug 22, 2013 at 4:44 PM, Rajasekar Elango wrote: > I am trying to tune mirrormaker configurations based on this doc > <https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker)#Kafka

Tuning mirror maker performance

I am trying to tune mirrormaker configurations based on this doc and would like know your recommendations. Our configuration: We are doing int