Re: build error

I too would like to understand the command you used to build and what directory. That aside it is likely best to test using the release candidate artifacts that are described in Jun's email here: http://search-hadoop.com/m/uyzND1bBATE1M3hVn/candidate=+VOTE+0+9+0+0+Candiate+2 This gives you all

issue with javaapi consumer

Hi all Iam currently using kafka 0.8.2.1 consumer which has kafka java api consumer but I had to upgrade to kafka 0.8.2.3 and when I add the jars for kafka 0.8.2.3 my consumer code doesn't compile. consumerConnector = ConsumerConfig

Is a Kafka 0.9 broker supposed to connect to itself?

I would think not I'm bringing up a new 0.9 cluster and i'm getting the below Exception (and the same thing on all nodes) - the IP address is the IP for the host the broker is running on. I think DNS is a bit stuffed on these machines and maybe that is the cause, but... any ideas? [2015-11-17

Re: build error

I got main class not found error. So I installed gradle 2.5 and run gradle build (not the wrapper) On Mon, Nov 16, 2015 at 10:17 PM, Guozhang Wang wrote: > Did you just use "./gradlew build" in root directory? > > Guozhang > > On Mon, Nov 16, 2015 at 6:41 PM,

Re: build error

Glad you resolved it. Note that we are currently on gradle 2.8 and likely 2.9 soon feel free to update whenever makes sense for you. Ewen recently updated the developer setup page here to show how to use the wrapper effectively

Re: build error

And I couldn't find wrapper jar files under the gradle folder https://github.com/apache/kafka/tree/0.9.0/gradle On Mon, Nov 16, 2015 at 10:17 PM, Guozhang Wang wrote: > Did you just use "./gradlew build" in root directory? > > Guozhang > > On Mon, Nov 16, 2015 at 6:41 PM,

Re: issue with javaapi consumer

Are you sure you mean 0.8.2.3? The latest version available is 0.8.2.2, and it is completely compatible. http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.kafka%22%20a%3A%22kafka_2.10%22 If you still have an issue, what error are you seeing? Thanks, Grant On Tue, Nov 17, 2015 at 5:47

Re: Questions about new consumer API

Thanks Guozhang, Maybe I should give a few words about what I'm going to achieve with new API Currently, I'm building a new kafka connector for Apache Apex( http://apex.incubator.apache.org/) using 0.9.0 API Apex support dynamic partition, so in the old version, We manage all the consumer

Re: Questions about new consumer API

Hi Siyuan, Your understanding about assign/subscribe is correct. We think of topic subscription as enabling automatic assignment as opposed to doing manual assignment through assign(). We don't currently them to be mixed. Can you elaborate on your findings with respect to using one thread per

Re: Custom metadata in message header

Hey Lesong, Unfortunately not. Any metadata you would like to include must be placed in the message payload. This means you must consume each message and process it in order to determine whether it is relevant to your current task. You can achieve some data locality by using keyed messages and

Re: Questions about new consumer API

By efficiency, I mean maximize throughput while minimize resources on both broker sides and consumer sides. One example is if you have over 200 partitions on 10 brokers and you can start 5 consumer processes to consume data, if each one is single-thread and you do round-robin to distribute the

RE: where do I get the Kafka classes

Learning build tools is going to take a bit. For the time being, I'd really just like to get the jars and add them manually. I'm not sure anybody has showed me where to get the jars yet. B. > From: ghe...@cloudera.com > Date: Tue, 17 Nov 2015 08:25:11 -0600 > Subject: Re: where do I get the

questions on number of partitions

We have a cluster of 32 nodes cluster, with a few topics. When we set 100 partitions for each topic, the overall CPU usage is 30%, and when we increase to 400 partitions, the CPU goes up over 65% - 70% sometimes. The # of producers are not changed. I understand that more partitions will drive up

Re: Optimal number of partitions for topic

Hi Carles Just check in, did you do the benchmarks eventually on your end? Curious to know the data points. Chen On Thu, May 21, 2015 at 2:47 PM, Carles Sistare < carles.sist...@googlemail.com> wrote: > Thanks a lot guys for your answers, > We’ll be doing some benchmarks comparing different

Re: questions on number of partitions

BTW, we are running Kafka 0.8.2.2. On Tue, Nov 17, 2015 at 3:48 PM, Chen Song wrote: > We have a cluster of 32 nodes cluster, with a few topics. When we set 100 > partitions for each topic, the overall CPU usage is 30%, and when we > increase to 400 partitions, the CPU

Custom metadata in message header

Hey all, I am new to Kafka. Is there a way to provide a custom metadata in the message header? Thanks, Lesong

Re: Is a Kafka 0.9 broker supposed to connect to itself?

There is inter-broker communication. It seems that the broker got a request more than the default allowed size (~10MB). How many topic/partitions do you have on this cluster? Do you have clients running on the broker host? Thanks, Jun On Tue, Nov 17, 2015 at 4:10 AM, Damian Guy

Re: Questions about new consumer API

Thanks for the explanation. Certainly you'd use less connections with this approach, but it might be worthwhile to do some performance analysis to see whether there is much difference in throughput (I'd be interested in seeing these results myself). Another approach that might be interesting would

Re: [VOTE] 0.9.0.0 Candiate 2

We hit a blocker (KAFKA-2841) in RC2. It's being addressed now. Will roll out RC3 once it's fixed. Thanks, Jun On Fri, Nov 13, 2015 at 6:28 PM, Jun Rao wrote: > This is the second candidate for release of Apache Kafka 0.9.0.0. This a > major release that includes (1)

[VOTE] 0.9.0.0 Candiate 3

This is the third candidate for release of Apache Kafka 0.9.0.0. This a major release that includes (1) authentication (through SSL and SASL) and authorization, (2) a new java consumer, (3) a Kafka connect framework for data ingestion and egression, and (4) quotas. Since this is a major release,

Subscription Request

Hi I have recently started using kafka and I want to subscribe to mailing list . Thanks Saurabh

Re: Subscription Request

Hi Saurabh, Sure thing. You can follow the steps here: http://kafka.apache.org/contact.html Regards, Prabhjot On Nov 18, 2015 11:32 AM, "saurabh pratap singh" wrote: > Hi > > I have recently started using kafka and I want to subscribe to mailing list > . > > > Thanks >

Re: where do I get the Kafka classes

BTW - you could download the jars directly from maven central if you want, for example: http://search.maven.org/#search|ga|1|a%3A%22kafka-clients%22 http://search.maven.org/#search|ga|1|a%3A%22spark-streaming-kafka-assembly_2.11%22%20AND%20v%3A%221.5.2%22 Ofir On Tue, Nov 17, 2015 at 9:36 PM,