Re: delete topic ?

2014-08-06 Thread Timothy Chen
Hi Gwen, That is a very confusing error message for sure, feel free to file a jira for both the experience cases. But in general how delete topic works is that it creates a entry in the delete_topic zk path, and the leader has a delete topic thread that watches that path and starts the topic dele

Re: consumer rebalance weirdness

2014-08-06 Thread Clark Haskins
Is your application possibly timing out its zookeeper connection during consumption while doing its processing, thus triggering the rebalance? -Clark On 8/6/14, 11:18 PM, "Jason Rosenberg" wrote: >We've noticed that some of our consumers are more likely to repeatedly >trigger rebalancing when t

Re: delete topic ?

2014-08-06 Thread Gwen Shapira
Hi Timothy, While we are on the subject, few questions/comments (based on the trunk implementation of delete topic command): * After deleting a topic, I still see it when listing topics. Is the expected behavior? Should it disappear after some time? * When does the actual deletion gets triggered?

consumer rebalance weirdness

2014-08-06 Thread Jason Rosenberg
We've noticed that some of our consumers are more likely to repeatedly trigger rebalancing when the app is consuming messages more slowly (e.g. persisting data to back-end systems, etc.). If on the other hand we 'fast-forward' the consumer (which essentially means we tell it to consume but do noth

Re: delete topic ?

2014-08-06 Thread Timothy Chen
Is this the latest master? I've added the delete option in trunk, but it's not in any release yet. We used to have the delete option flag but I believe we removed it that's why the documentation difference. Tim On Wed, Aug 6, 2014 at 10:53 PM, Shlomi Hazan wrote: > if the answer is pointing out

Re: delete topic ?

2014-08-06 Thread Shlomi Hazan
if the answer is pointing out the 'chroot', as a word, it makes no difference. the result is the same: kafka/bin/kafka-topics.sh --zookeeper localhost:2181/chroot --delete --topic topic-3 gives the same: "Command must include exactly one action: --list, --describe, --create or --alter..." or sh

Re: Graceful shutdown without using jmx

2014-08-06 Thread Daniel Compton
Hi Jongyoul If you did want to set the JMX Port on startup, you could do this by setting the JMX_PORT environment variable when starting the server. Here is how we do it. We've put this into it's own script file. env JMX_PORT=8110 $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.pr

Re: Issue with kafka-server-start.sh

2014-08-06 Thread Gwen Shapira
For completeness, you are probably running into: https://issues.apache.org/jira/browse/KAFKA-1451 On Wed, Aug 6, 2014 at 2:26 PM, Gwen Shapira wrote: > These are info messages, not necessarily errors. > > Do they occur in a never ending loop? Do you also see errors? > Do you ever get a successf

Re: Issue with kafka-server-start.sh

2014-08-06 Thread Gwen Shapira
These are info messages, not necessarily errors. Do they occur in a never ending loop? Do you also see errors? Do you ever get a successful broker registration message in the log? (Registered broker 1 at path) Gwen On Wed, Aug 6, 2014 at 1:53 PM, Shikha Srivastava wrote: > Hi, > > I am work

Issue with kafka-server-start.sh

2014-08-06 Thread Shikha Srivastava
Hi, I am working with kafka_2.8.0-0.8.1.1. It used to work fine but since this morning when I try to start the kafka-server after starting zookeeper, I am getting this error- INFO I wrote this conflicted ephemeral node [{"version":1,"brokerid":0,"timestamp":"1407357554030"}] at /controller a whi

Re: trying to tune kafka's internal logging - need help...

2014-08-06 Thread Harsha
Hi Sholmi, kafka-run-class.sh uses config/tools-log4.properties. The config it has writes to console log4j.appender.stdout=org.apache.log4j.ConsoleAppender If you want to save the log output from kafka-run-class change the above appender to use DailyRollingFileAppender. for ex

trying to tune kafka's internal logging - need help...

2014-08-06 Thread Shlomi Hazan
Hi, I am trying to get rid of the log files written under “$base_dir/logs”, folder create by line 26 at “bin/kafka-run-class.sh”. I use an EC2 machine with small primary disk and it blows away on occasions when writing to these logs is excessive, and I bumped into a few already (from Jira it look

Re: Kafka 0.8 automatically changing leadership

2014-08-06 Thread Marcos Juarez
Thank you Joel, will go thorough those docs and make sure our settings are appropriate on these instances. Marcos Juarez On Tue, Aug 5, 2014 at 5:58 PM, Joel Koshy wrote: > The session expirations (in the log you pasted) lead to the broker > losing its registration from zookeeper (which trigge

Re: kafka.SocketServerStats write rate metrics seem wrong

2014-08-06 Thread Jun Rao
TotalBytesWritten is probably for sending the responses. You probably should look at TotalBytesRead for incoming data rate. Thanks, Jun On Tue, Aug 5, 2014 at 12:23 PM, Neil Harkins wrote: > Hi. > I'm seeing some odd numbers from kafka.SocketServerStats. > > Ideally, I'd like to have stats br