Re: kafka application logdir

2014-11-17 Thread Jimmy John
at 11:29 AM, Harsha ka...@harsha.io wrote: you can configure it under /opt/kafka/config/log4j.properties and look for kafka.log.dir On Mon, Nov 17, 2014, at 11:11 AM, Jimmy John wrote: How do I configure the application kafka log dir? Right now the default is /var/log/upstart/kafka.log . I

Re: kafka application logdir

2014-11-17 Thread Jimmy John
Looks like there is already a tkt for this: https://issues.apache.org/jira/browse/KAFKA-1204 thx jim On Mon, Nov 17, 2014 at 1:59 PM, Jimmy John jj...@livefyre.com wrote: I tried that but it did not work. Dug a little deeper and saw this line in bin/kafka-run-class.sh: KAFKA_LOG4J_OPTS

Re: powered by kafka

2014-11-08 Thread Jimmy John
Livefyre (http://web.livefyre.com/) uses kafka for the real time notifications, analytics pipeline and as the primary mechanism for general pub/sub. thx... jim On Sat, Nov 8, 2014 at 7:41 AM, Gwen Shapira gshap...@cloudera.com wrote: Done! Thank you for using Kafka and letting us know :)

Re: OffsetOutOfRange errors

2014-11-07 Thread Jimmy John
to, e.g. the head of the log. How frequent do your clients read / write the offsets in ZK? Guozhang On Thu, Nov 6, 2014 at 6:23 PM, Jimmy John jj...@livefyre.com wrote: Hello, I understand what this error means, just not sure why I keep running into it after 24-48 hrs of running fine

Re: OffsetOutOfRange Error

2014-11-06 Thread Jimmy John
I dug deeper and saw this during normal operation: In the kafka broker log: [2014-11-03 21:39:25,658] ERROR [KafkaApi-8] Error when processing fetch request for partition [activity.stream,5] offset 7475239 from consumer with correlation id 69 (kafka.server.KafkaApis)

OffsetOutOfRange errors

2014-11-06 Thread Jimmy John
Hello, I understand what this error means, just not sure why I keep running into it after 24-48 hrs of running fine consuming 300 messages / second. What happens when a kafka log rolls over and some old records are aged out? I mean what happens to the offsets? We are using a python client

OffsetOutOfRange Error

2014-11-04 Thread Jimmy John
Hello, We are using kafka version 0.8.1 and the python kafka client. Everything has been working fine and suddenly this morning I saw a OffsetOutOfRange on one of the partitions. (We have 20 partitions in our kafka cluster) We fixed it by seeking to the head offset and restarting the app.