jruby-kafka 1.4.0 released

Hi everyone, I updated jruby-kafka to version 1.4.0 which uses Kafka 0.8.2.1 and has a new interface for the latest KafkaProducer. I also recently updated the packaging of the gemset to include the jar files needed to run everything so it's easier to use. Check it out https://github.com/joeki

KafkaProducer javadoc Valid configuration strings link correct?

Hi everyone, I was reviewing the javadocs for the 082 and 083 Kafka Producer (http://kafka.apache.org/082/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html and http://kafka.apache.org/083/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html)

Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 2

Congratulations everyone! From: Joe Stein Sent: Tuesday, March 10, 2015 11:12 AM To: Jun Rao Cc: d...@kafka.apache.org; users@kafka.apache.org; kafka-clie...@googlegroups.com Subject: Re: [kafka-clients] Re: [VOTE] 0.8.2.1 Candidate 2 Thanks Jun for gett

Re: If you run Kafka in AWS or Docker, how do you persist data?

Side question: why run kafka on docker for aws? Is the docker config being used for configuration management? Are there more systems running on the instance other than kafka? Sent by Outlook for Android On Sun, Mar 1, 2015 at 1:10 PM -0800, "Ewen Cheslack-Postav

Re: Announcing the Confluent Platform built on Apache Kafka

This is really awesome stuff. It's great to see y'all growing! Thank you and congratulations! From: Neha Narkhede Sent: Wednesday, February 25, 2015 1:31 PM To: users@kafka.apache.org; d...@kafka.apache.org Subject: Announcing the Confluent Platform bui

Re: Kafka High Level Consumer

Doh that was probably my bad Pranay! A misinterpretation of some old consumer code. btw, jruby-kafka is now at 1.1.1 with proper support for deleting the offset, setting the auto_offset_reset and whitelist/blacklist topics. It's packed up in a nice gem file that includes all Kafka and log4j p

Re: using the new logstash-kafka plugin

to make updates to the plugin outside of kafka? On Thu Jan 22 2015 at 9:31:26 AM Joseph Lawson wrote: > Scott you will have to do just one topic per input right now but multiple > topics per group, whitelisting and blacklisting just got merged into > jruby-kafka and I'm working the

Re: using the new logstash-kafka plugin

Scott you will have to do just one topic per input right now but multiple topics per group, whitelisting and blacklisting just got merged into jruby-kafka and I'm working them up the chain to my logstash-kafka and then pass it to the logstash-input/output/-kafka plugin.

Re: Updated Logstash-Kafka plugin

://cwiki.apache.org/confluence/display/KAFKA/Ecosystem)? Jun On Tue, Jan 20, 2015 at 7:59 AM, Joseph Lawson wrote: > Hello Everyone, > > > As we wait for the 1.5 release of Logstash which will include the Kafka > plugin by default I've put some work into the logstash-kafka plugin which >

Updated Logstash-Kafka plugin

Hello Everyone, As we wait for the 1.5 release of Logstash which will include the Kafka plugin by default I've put some work into the logstash-kafka plugin which makes it much easier to install and also enables logging on from the kafka library when you use --debug or --verbose toggles with lo

Re: kafka cluster on aws

We have a separate daemon process that assigns EIPs to servers when they startup in an autoscaling group based off of an autoscaling message. So for a cluster of 3 we have 3 EIPs. Then we inject the EIPs into startup script for Kafka which checks to see if it has one of the EIPs and assigns its

Kafka getting some love from Logstash and Elasticsearch

?Kafka made some headlines with Logstash announcing their latest version beta (1.5) which includes by default a Kafka input and output plugin. Good stuff. http://www.elasticsearch.org/blog/logstash-1-5-0-beta1-released/?

Re: Is AWS Kinesis Kafka?

deal with setting up and maintaining a > Kafka cluster. And even then it's only applicable if you're use case > fits inside the retention and message size limitations. > > Best, > Niek > > > On Thu, Nov 13, 2014 at 2:32 PM, Joseph Lawson > wrote: > > Oh man they look similar. Any comments? >

Re: Is AWS Kinesis Kafka?

: users@kafka.apache.org Subject: Re: Is AWS Kinesis Kafka? I've wondered that about Azure Event Hubs as well. They both use a different consumer offset tracking mechanism than the one in 0.8 for their higher level consumers. Christian On Thu, Nov 13, 2014 at 2:32 PM, Joseph Lawson wrote:

Is AWS Kinesis Kafka?

Oh man they look similar. Any comments?

Re: java api code and javadoc

You probably have to build your own right now. Check out https://github.com/apache/kafka#building-javadocs-and-scaladocs From: 4mayank <4may...@gmail.com> Sent: Wednesday, October 15, 2014 11:38 AM To: users@kafka.apache.org Subject: java api code and java

Re: Log 4j Conversion Pattern not working kafka.producer.KafkaLog4jAppender in Kafka.0.8.1.1

://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> / On Mon, Sep 22, 2014 at 1:56 PM, Joseph Lawson wrote: > I don't think the log4j appender supports patterns :( > > On Sep 22, 2014 1:14 PM, Lakshmanan Muthuram

Re: Log 4j Conversion Pattern not working kafka.producer.KafkaLog4jAppender in Kafka.0.8.1.1

I don't think the log4j appender supports patterns :( On Sep 22, 2014 1:14 PM, Lakshmanan Muthuraman wrote: Hello , I am trying to use KafkaLog4jAppender to write to kafka and also a file appender to write to a file. The conversion pattern works for the file, but for the kafka appender it is not

RE: Apache webserver access logs + Kafka producer

:9092" You can also pipe a custom log into it as well :) ________ From: Joseph Lawson Sent: Thursday, August 07, 2014 5:35 PM To: users@kafka.apache.org; Philip O'Toole Subject: RE: Apache webserver access logs + Kafka producer Check out my logst

RE: Apache webserver access logs + Kafka producer

Check out my logstash-kafka project: https://github.com/joekiller/logstash-kafka I believe the plugin will be merged into logstash itself soon but for now you can make it yourself. I would suggest making your apache format in json in your apache config and then stream the data through the logs

Informal Kafka logstash user survey

Hi everyone, The maintainers of logstash is considering adding my Kafka plugin (https://github.com/joekiller/logstash-kafka ) to the core of logstash.  They are asking at the following link for some +1s from users. Please feel free to chime in. https://groups.google.com/forum/m/#!topic/logstash

jruby-kafka and logstash-kafka nearly feature complete

Hi everyone, I have updated my jruby-kafka library to support pretty much all the default consumer and producer classes to support all the options included in the docs. Check it out, give some feedback: https://github.com/joekiller/jruby-kafka This library is primarily supporting my logstash

Re: New Producer Public API

My 2 cents: Getting the broker metadata via active brokers is the way to go. It allows one to dynamically rebalance or introduce a whole new set of servers into a cluster just by adding them to the cluster and migrating partitions. We use this to periodically introduce newer Kafka cluster cloud

Re: Apache Kafka HTTP Producer & Consumer

JavaScript logs to kafka anyone :) On Jan 16, 2014 4:50 PM, Joe Stein wrote: http://github.com/stealthly/dropwizard-kafka-http is an Apache Kafka HTTP Endpoint http://allthingshadoop.com/2014/01/16/apache-kafka-http-producer-and-consumerfor producing and consuming messages for topics. We have mo

Jruby kafka 0.8.0 producer and consumer

Hi everyone, I wanted to let you all know that my jruby-kafka library now has a working producer. The consumer has been working quite well for some time but now the producer is functional as well!  This is a simple jruby wrapper around the Scala jars so it functions very much like the binary ve

Re: kafka + storm

If your task is completely serial you can still use Kafka and storm but you are limited in the number of workers that can apply whatever data transformation you need serialized. Basically you can use one consumer reading one Kafka partition and one storm worker being fed by the one Kafka consume

RE: kafka-reassign-partitions.sh --status-check-json-file not working

acter. From: Joseph Lawson Sent: Tuesday, November 05, 2013 8:32 AM To: users@kafka.apache.org Subject: Re: kafka-reassign-partitions.sh --status-check-json-file not working Guozhang, I'll try to come up with some steps to repicate today. I didn't notice anything obvious in the logs. Jo

Re: kafka-reassign-partitions.sh --status-check-json-file not working

two topics originally located? Guozhang On Mon, Nov 4, 2013 at 9:47 AM, Joseph Lawson wrote: > Hi everyone, > > I'm using Kafka 0.8.0 from the git repository. > > I'm trying the following commands: > > bin/kafka-reassign-partitions.sh --topics-to-move-json-file >

kafka-reassign-partitions.sh --status-check-json-file not working

Hi everyone, I'm using Kafka 0.8.0 from the git repository. I'm trying the following commands: bin/kafka-reassign-partitions.sh --topics-to-move-json-file topics-to-move.json --zookeeper zk-qa.us-e.roomkey.net:2181/kafka_stage/kafka_qa --broker-list "1" --execute where topics-to-move.json is:

Re: Recovering High Level Consumer from exceptions

ed to make sure you don't kill the consumer thread if it hits an application error. Thanks, Jun On Mon, Oct 28, 2013 at 5:21 PM, Joseph Lawson wrote: > Hi everyone, > > how should a high level consumer implementation handle consumer > exceptions. Using the high level consumer wik

Recovering High Level Consumer from exceptions

Hi everyone, how should a high level consumer implementation handle consumer exceptions. Using the high level consumer wiki example, the primary thread calls an example.run() and then waits for a thread.sleep and then shuts down. If each ConsumerTest is a thread, what is the best way for the

Re: Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine

Sent from my Droid Charge on Verizon 4G LTE Joseph Lawson wrote: it may be that the broker is reporting its internal ip address to zookeeper which then may interfere with your ability to submit the the broker with a producer that isn't on AWS's network. you can make the broker report the

Re: Hi, Can anyone tell me why I cannot produce any message to remote kafka server from my local machine

it may be that the broker is reporting its internal ip address to zookeeper which then may interfere with your ability to submit the the broker with a producer that isn't on AWS's network. you can make the broker report the public dns name instead of the internal host name which i believe it re

RE: introducing jruby-kafka (Kafka 0.8beta1) and Kafka logstash input

maintaining this, would you like to add this to https://cwiki.apache.org/confluence/display/KAFKA/Clients ? Thanks, Neha On Mon, Oct 7, 2013 at 8:10 AM, Joseph Lawson wrote: > Hi everyone, > > > I wrapped kafka 0.8beta1 client into a jruby class: > https://github.com/joekiller/jruby-k

introducing jruby-kafka (Kafka 0.8beta1) and Kafka logstash input

Hi everyone, I wrapped kafka 0.8beta1 client into a jruby class: https://github.com/joekiller/jruby-kafka and then a wrote an input for logstash: https://github.com/joekiller/logstash/tree/kafka8 It's a little rough around the edges still but it does work fine. I'll be enhancing both the logs