Kafka 0.8 jmxtrans + puppet

Hi, I just got jmxtrans set up with Kafka 0.8 over at Wikimedia. We're using puppet, and the https://github.com/wikimedia/puppet-kafka module to install and set up Kafka. This module now comes with a kafka::server::jmxtrans class which will automatically set up jmxtrans with some nice Kafka

Re: Kafka 0.8 jmxtrans + puppet

Cool, thanks for sharing this! -Neha On Nov 6, 2013 6:16 AM, Andrew Otto o...@wikimedia.org wrote: Hi, I just got jmxtrans set up with Kafka 0.8 over at Wikimedia. We're using puppet, and the https://github.com/wikimedia/puppet-kafka module to install and set up Kafka. This module now

Kafka IPv6

Hm, Does Kafka support IPv6? I'm trying it now, but I'm just using the console-producer, which seems to not be able to read in --broker-list with IPv6 addresses. It looks like it is interpreting the colons in the address as the addy:port separator. -Andrew

Re: Kafka 0.8 jmxtrans + puppet

We will be using Kafka to transport access logs from frontend cache servers to HDFS. But! We aren't doing this in production yet, so I wouldn't quite yet feel comfortable saying we are powered by Kafka. I hope to get this into production more in a few weeks*, and at that time we'll write a

Re: Kafka IPv6

Yes, that seems like a problem for IPv6. Could you file a jira? In the meantime, could you try using the hostname instead of the ip address? Thanks, Jun On Wed, Nov 6, 2013 at 7:43 AM, Andrew Otto o...@wikimedia.org wrote: Hm, Does Kafka support IPv6? I'm trying it now, but I'm just using

Logging of errors

Hello, How can I have the brokers log errors to a file? Do I just have to configure something like log4j or is something else used? Thanks. Casey

Re: Logging of errors

Yes, configure the kafka/config/log4j.properties that ships with Kafka. Thanks, Neha On Wed, Nov 6, 2013 at 8:48 AM, Sybrandy, Casey casey.sybra...@six3systems.com wrote: Hello, How can I have the brokers log errors to a file? Do I just have to configure something like log4j or is

Re: Logging of errors

Hi Casey, Did you want to route all the error log entries to one file and the others to another file? Guozhang On Wed, Nov 6, 2013 at 9:07 AM, Neha Narkhede neha.narkh...@gmail.comwrote: Yes, configure the kafka/config/log4j.properties that ships with Kafka. Thanks, Neha On Wed, Nov 6,

RE: Logging of errors

One file is good for now. I just didn't find any documentation on this, so I figured I'd ask. Guess I should have just looked at the config directory. -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: Wednesday, November 06, 2013 12:16 PM To:

Re: Kafka IPv6

Can do and done! https://issues.apache.org/jira/browse/KAFKA-1123 On Nov 6, 2013, at 11:04 AM, Jun Rao jun...@gmail.com wrote: Yes, that seems like a problem for IPv6. Could you file a jira? In the meantime, could you try using the hostname instead of the ip address? Thanks, Jun

Re: JMXTrans not sending kafka 0.8 metrics to Ganglia

typeNames is required in settings section for beanName to be included in the metric name when jmxtrans sends it out. Here's an example: outputWriters : [ { @class : com.googlecode.jmxtrans.model.output.GangliaWriter, settings : { groupName : kafka, port :

Re: Topic creation on restart

Ok, I updated https://issues.apache.org/jira/browse/KAFKA- with this scenario also. Jason On Tue, Nov 5, 2013 at 9:11 PM, Neha Narkhede neha.narkh...@gmail.comwrote: . Could a fetch request from a consumer cause a Topic creation request (seems implausible). Yes, that seems like a

Re: Message Width

Each message size has a default limit of around 1 MB then, and multiple messages are batched together, presumably, up to some other default batch limit. Has anyone needed to increase the message limit to greater than 1 MB? If so, to what limit and were there any issues? JP On Nov 5, 2013,

Re: kafka and log retention policy when disk is full

So, I've hit this issue with kafka - when disk is full kafka stops with exception FATAL [KafkaApi-1] Halting due to unrecoverable I/O error while handling produce request: (kafka.server.KafkaApis) kafka.common.KafkaStorageException: I/O exception in append to log 'perf1-2' I think it will be

Re: Commit Offset per topic

This question seems to come up often - added this to the FAQ. https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Idon%27twantmyconsumer%27soffsetstobecommittedautomatically.CanImanuallymanagemyconsumer%27soffsets%3F On Tue, Nov 05, 2013 at 04:02:09PM +, Roman Garcia wrote: Thanks

Re: kafka and log retention policy when disk is full

We don't have this global setting right now - so the guideline is to capacity plan to set your log retention settings and set alerts on disk space accordingly. In order to recover you should be able to change your retention settings, but the log cleaner is not called on start up so that won't

Re: Kafka 0.8.0 in maven

After about 5 days of relentless head pounding, and twittling about everything under the sun, I figured it out. If you read to the bottom of this page: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example You find: consumerProps.put(auto.offset.reset, smallest); Now I can

Re: Kafka 0.8.0 in maven

Do you need to use that configuration to get the tests (as currently checked in) to pass ?I did not find i needed that particular knob (although it is a good one to know about). and Sorry about your suffering... I can sympathize ! - cb On Wed, Nov 6, 2013 at 7:38 PM, Edward Capriolo

Re: Kafka 0.8.0 in maven

One thing I noticed about your code. I thought in kafka 0.8.0 topics are not created automatically on first message. I do not see anywhere in your code which creates the topics. I am creating the topic as part of my tests. Before I was not setting that property, and getting some of the messages.