Re: Kafka Authenication and Monitoring Tools

Hi Joseph, For the monitoring piece there is SPM - see http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/. It will monitor your ZooKeeper, too. Kafka uses ZK, so you may want to keep an eye on it as well. Otis -- Performance Monitoring * Log Analytics * Searc

Re: Persist Queue On HDFS

Nice idea, but different sort of animal. Going to HDFS is different. It requires aggregation of traffic, so there is the whole offset commit strategy concern. When pulling traffic for per message work, we commit after every pull, so exactly once. The tradeoff with aggregation is whether to a

Re: Persist Queue On HDFS

The api in HDFS is quite different from what's in a regular POSIX file system. Thanks, Jun On Tue, Jan 14, 2014 at 1:16 PM, Blender Bl wrote: > > Hi, > > > My team trying to implement lambda architecture. > We need to stream all our new data though Kafka to storm, and HDFS. > > > As i see it

Re: purging a topic

You would need to first stop the consumer, update the offset in ZK and then restart the consumer. Also, have you looked at the tool ImportZkOffsets? Thanks, Jun On Tue, Jan 14, 2014 at 12:38 PM, Yu, Libo wrote: > Hi folks, > > I am writing a tool to "purge" the pending topics for a user. Assu

Re: Killing broker leader

Hi Francois, just a quick question, when you set the number of retries to 10, does its log still have "ailed to send messages after 10 tries." entry? On Tue, Jan 14, 2014 at 11:41 AM, Francois Langelier < francois.langel...@mate1inc.com> wrote: > Of course! As soon as it will close, i'll try! >

Fwd: Upcoming Apache Samza talk in Seattle, January 22

FYI... -Jay -- Forwarded message -- From: Jakob Homan Date: Tue, Jan 14, 2014 at 2:09 PM Subject: Upcoming Apache Samza talk in Seattle, January 22 To: d...@samza.incubator.apache.org, users@kafka.apache.org Hey all- Anyone in the Seattle area interested in Apache Samza shou

Re: Persist Queue On HDFS

There is also hadoop contrib producer and consumer https://github.com/apache/kafka/tree/0.8/contrib for hdfs /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Persist Queue On HDFS

Hi, My team trying to implement lambda architecture. We need to stream all our new data though Kafka to storm, and HDFS. As i see it were are two options: Using Camus - not very efficent Streaming via Storm - not very efficent Is it possible to persist the queue's files over the HDFS (with s

Upcoming Apache Samza talk in Seattle, January 22

Hey all- Anyone in the Seattle area interested in Apache Samza should come to Bellevue next week for the Seattle Scalability Meetup. I'll be presenting an overview of the project, how it works with Kafka (among other friends) and use cases at LinkedIn. Hope to see you there. Be sure to regist

Re: Kafka Authenication and Monitoring Tools

> https://cwiki.apache.org/confluence/display/KAFKA/Security Interesting. I'm curious what people are doing in the interim. We'd been chewing on options for secure communication and mutual authentication like stunnel. Actually imposing discretionary controls on the activity of clients using

Re: purging a topic

Hello Libo, This should work, but I am more curious about your exact usage scenarios. Would setting the default offset reset to largest be appropriate for you? Guozhang On Tue, Jan 14, 2014 at 12:38 PM, Yu, Libo wrote: > Hi folks, > > I am writing a tool to "purge" the pending topics for a us

purging a topic

Hi folks, I am writing a tool to "purge" the pending topics for a user. Assume the user has never consumed this topic previously. If I create all the nodes on the path /consumers/[myuser]/offsets/[mytopic]/[partition] and put the maximum available offset to the node, is that enough to let the con

Re: Killing broker leader

Of course! As soon as it will close, i'll try! On Tue, Jan 14, 2014 at 2:29 PM, Guozhang Wang wrote: > Hanish and Francois, > > The current patch of 1193 still have something missing and I am currently > working on it to be closed soon. Could you retry the scenario after it is > checked in? > >

Re: Killing broker leader

Hanish and Francois, The current patch of 1193 still have something missing and I am currently working on it to be closed soon. Could you retry the scenario after it is checked in? Guozhang On Tue, Jan 14, 2014 at 8:03 AM, Francois Langelier < francois.langel...@mate1inc.com> wrote: > @Guozhan

Re: Kafka Authenication and Monitoring Tools

https://cwiki.apache.org/confluence/display/KAFKA/Security https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing https://cwiki.apache.org/confluence/display/KAFKA/System+Tools /*** Joe Stein Founder, Principal Consultant Big Data Open So

Kafka Authenication and Monitoring Tools

Hi, Which security implementation is imposed by Kafka? What type of authentication is required by Kafka? Which tools would best monitor the performance of Kafka clusters? Joseph Simmons Accenture LLC TGP-CMT Analyst San Francisco, CA B: 415.537.4703 C: 951.941.1125 _

Re: Need help for kafka on windows

Hi Sripada, Unfortunately I can't provide a code fix, but it's an easy fix actually. Basically the path that is trying to look for kafka-run-class.bat is wrong as it expects that file to be from the current window's folder. You can either cd into the parent and run it or fix the script. Tim O

Re: Need help for kafka on windows

I think your bug was reported here, but there is not enough details so maybe i'm wrong https://issues.apache.org/jira/browse/KAFKA-1195 If this is your bug, the good new is you are not alone on the other hand, the bug priority is minor and there is no fix yet... On Tue, Jan 14, 2014 at 6:15 AM

Re: Killing broker leader

@Guozhang Wang: I set the max retries to 10 and the backoof at 1000 ms but the bug still there and some messages don't reach my consumers... @Hanish : Yes, it looks like we have the same issue! On Mon, Jan 13, 2014 at 9:43 PM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > I am not

Re: Kafka Consumer connection issue

Does that happen in the middle of a consumer rebalance? If so, that could be normal since on every rebalance, we will interrupt existing socket connections. The question is weather rebalances complete in the end. If there are too many rebalances, take a look at https://cwiki.apache.org/confluence/d

Need help for kafka on windows

Hi All, I am new to apache kafka. Can you please any one help me to setup apache kafka (kafka_2.8.0-0.8.0). I have download and try to run batch file under bin/windows folder I am getting below error. "'kafka-run-class.bat' is not recognized as an internal or external command,operable progr

Kafka Consumer connection issue

Hi, I keep getting the folllowing error in Kafka. 2014-01-14 10:46:55:073 SyncProducer [ERROR] Producer connection to :9092 unsuccessful java.nio.channels.ClosedByInterruptException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)

Re: v0.8.0 JMX Beans

JMX on 0.8 is different than 0.7 Check out also https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-JMXTool to go along with the doc explanation. Some sample maybe could get posted somewhere for more clarity but it is a lot there per the docs. Always good to see it liv

Re: v0.8.0 JMX Beans

Thanks Joe, I've had a look through the metrics in the link you sent, but I can't find the specific metrics that I listed below (NumFetchRequests, TotalFetchRequestMs, MaxFetchRequestMs, NumProduceRequests, TotalProduceRequestMs, MaxProduceRequestMs, TotalBytesRead and TotalBytesWritten) or equival

Re: v0.8.0 JMX Beans

http://kafka.apache.org/documentation.html#monitoring /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop / On Jan 14, 2014, at 6:5

v0.8.0 JMX Beans

Hi, I'm upgrading from version 0.7.2-incubating to 0.8.0, and I've noticed that the JMX beans have changed considerably. I'm trying to find equivalent values to the following (all were previously on kafka.SocketServerStats): NumFetchRequests, TotalFetchRequestMs, MaxFetchRequestMs, NumProduceReque

Re: Consumers can't connect while broker is under load

12 zookeepers seems like a lot..and you should always, by default, prefer an odd number of zookeepers. Consumers negotiate with each other for partition ownership, via zookeeper. Jason On Fri, Jan 10, 2014 at 9:20 PM, Guozhang Wang wrote: > Can you post the consumer logs for the long-ti

Re: custom kafka consumer - strangeness

Hi, thanks I will do this. On Tue, Jan 14, 2014 at 9:51 AM, Joe Stein wrote: > I Gerrit, do you have a ticket already for this issue? Is it possible to > attach code that reproduces it? Would be great if you can run it against a > Kafka VM you can grab one from this project for 0.8.0 > http

Re: custom kafka consumer - strangeness

I Gerrit, do you have a ticket already for this issue? Is it possible to attach code that reproduces it? Would be great if you can run it against a Kafka VM you can grab one from this project for 0.8.0 https://github.com/stealthly/scala-kafka to launch a Kafka VM and add whatever you need to it t

Re: custom kafka consumer - strangeness

Yes, I'm using my own client following: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol Everything works except for this weirdness. On Tue, Jan 14, 2014 at 5:50 AM, Jun Rao wrote: > So, you implemented your own consumer client using netty? > > Thanks, > > Jun >