Re: Using the 0.8.2.2 KafkaConsumer Client

2015-10-30 Thread Pratapi Hemant Patel
Actually in 0.8.2.2 only kafkaproducer is fully implemented not Kafkaconsumer. Here is the implementation of kafkaconsumer poll method in 0.8.2.2. @Override public Map> poll(long timeout) { // TODO Auto-generated method stub return null; } KafkaConsumer will be releas

Topic per entity

2015-10-30 Thread Alex Buchanan
Hey Kafka community. I'm researching possible architecture for a distributed data processing system. In this system, there's a close relationship between a specific dataset and the processing code. The user might upload a few datasets and write code to run analysis on that data. In other words, fr

Re: One more Kafka Meetup hosted by LinkedIn in 201[5] (this time in San Francisco) - does anyone want to talk?

2015-10-30 Thread Ed Yakabosky
d'oh! I definitely meant 2015 everyone :-). Thanks for catching that! On Fri, Oct 30, 2015 at 6:25 PM, Andy Chambers wrote: > Wow you guys plan a long time ahead ;-) > > On Fri, Oct 30, 2015 at 12:49 PM, Ed Yakabosky > wrote: > > > Hi all, > > > > LinkedIn is hoping to host one more Apache

Re: One more Kafka Meetup hosted by LinkedIn in 2016 (this time in San Francisco) - does anyone want to talk?

2015-10-30 Thread Andy Chambers
Wow you guys plan a long time ahead ;-) On Fri, Oct 30, 2015 at 12:49 PM, Ed Yakabosky wrote: > Hi all, > > LinkedIn is hoping to host one more Apache Kafka meetup this year on > November 18 in our San Francisco office. We're working on building the > agenda now. Does anyone want to talk? Ple

Re: Using the 0.8.2.2 KafkaConsumer Client

2015-10-30 Thread Stu Smith
Note that I did work around this issue, by including the entire Kafka: org.apache.kafka kafka_2.10 0.8.2.2 dependency, and using the legacy Consumer API, instead of the kafka-clients dependency: org.apache.kafka kafka-clients 0.8.2.2 Listed in the documentation: htt

Using the 0.8.2.2 KafkaConsumer Client

2015-10-30 Thread Stu Smith
Hello! I'm running into trouble using the latest Kafka client. 0.8.2.2 appears to be listed as a stable release on Maven Central: http://mvnrepository.com/artifact/org.apache.kafka/kafka-clients And it only includes the: org.apache.kafka.clients.consumer.KafkaConsumer client All the other C

question about async publisher blocking when broker is down.

2015-10-30 Thread Ilya Goberman
I am new to kafka and apologize if this is already answered. I am testing a simple async publisher behavior when broker is down. I use kafka version 8.2.2. I have set up "queue.buffering.max.messages" to 200 and "queue.enqueue.timeout.ms" set to -1. My understanding is that if "queue.enqueue.t

Re: secure communication between producer/consumer to brokers

2015-10-30 Thread Ismael Juma
SSL will be supported in 0.9.0.0, so please use trunk of you'd like to test it in the meantime. Ismael On 28 Oct 2015 12:09, "Pratapi Hemant Patel" wrote: > Hi all, > I followed the wiki > https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka > for secure communication betwee

One more Kafka Meetup hosted by LinkedIn in 2016 (this time in San Francisco) - does anyone want to talk?

2015-10-30 Thread Ed Yakabosky
Hi all, LinkedIn is hoping to host one more Apache Kafka meetup this year on November 18 in our San Francisco office. We're working on building the agenda now. Does anyone want to talk? Please send me (and Clark) a private email with a short description of what you would be talking about if int

brokers skew % - not sure I know what this means

2015-10-30 Thread Jennifer Fountain
I am seeing a brokers skew % at 25 for some of my topics. However, I cannot find any information on what that exactly means (google-fu is failing me). Thanks for any info! -- Jennifer Fountain DevOPS

New producer and storing offsets in Kafka - previously committed offsets fetched as uncommitted

2015-10-30 Thread Stevo Slavić
Hello Apache Kafka community, I'm trying to use new producer, from kafka-clients 0.8.2.2, together with simple consumer to fetch and commit offsets stored in Kafka, and I'm seeing strange behavior - a committed offset/message gets read multiple times, offset fetch requests do not always see commit

Best practices for producer

2015-10-30 Thread Gmail - Jai
Hello, I am working/learning on kafka for producing messages and had a little doubt about kafka producer configuration. Currently I am using kafka 0.8.0 version and using zk.connect property which is not specified in documentation for .8 version but still works on this version. There is a prope