Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

Hey Harsha, A few comments: Can you finish up the KIP there are some unfinished sentences and odd whitespace things going on. Here are the questions I think we should consider: 1. Do we need this at all given that we have the partition argument in ProducerRecord which gives full control? I

[ANN] Bottled Water: PostgreSQL to Kafka replication

Hi Kafka users, I'd like to announce a new open source project, called Bottled Water, for getting data from PostgreSQL into Kafka: http://blog.confluent.io/2015/04/23/bottled-water-real-time-integration-of-postgresql-and-kafka/ https://github.com/confluentinc/bottledwater-pg/ Bottled Water

Re: How to set console consumer group ID

Thank you, Raja! 2015-04-23 0:34 GMT+02:00 Rajasekar Elango rela...@salesforce.com: Yes, you pass any consumer property including group.id by having them in property file and passing path to it using --consumer.config of consumer consumer. Thanks, Raja. On Wed, Apr 22, 2015 at 1:45 AM,

Horizontal scaling a topic

I have a cluster of 3 nodes and I've created a topic with some number of partitions and some number of replicas, let's say 10 and 2, respectively. Later, after I've got my 3 nodes fairly consumed with data in the 10 partitions, I want to add 2 more nodes to the mix to help balance out the

Re: Fetch Request Purgatory and Mirrormaker

This is still occurring for us. In addition, it has started occurring on one of the six nodes in the healthy cluster, for no reason we have been able to determine. We're willing to put in some serious time to help debug/solve this, but we need *some* hint as to where to start. I understand that

Re: Horizontal scaling a topic

For Kafka 0.8.x[.x], refer to https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool . 2015-04-23 23:20 GMT+03:00 Corey Nolet cjno...@gmail.com: I have a cluster of 3 nodes and I've created a topic with some number of partitions and some

Kafka client - 0.9

Hi, I'm looking for the 0.9 client release plan. https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design Is there a planned date for this release? Thanks, Bharath

Remote kafka - Connection refused.

Dear all, I am trying to connect my python consumer to a remote kafka server. But in kafka/conn.py#reinit, the socket call socket.create_connection throws connection refused error. to confirm, it is not a firewall issue, i tried connecting to other servers (in the same machine as kafka server)

NoClassDefFoundError at producer end

Hi, we are getting NoClassDefFoundError at our producer end. We are using only syn producer. we have 3 data producer systems whose using same kafka.javaapi.producer.Producer instance.Bellow is the maven dependency dependency

kafka user group in los angeles

Hi, Sorry this isn't directly a kafka question, but I was wondering if there are andy Kafka user groups in (or in near driving range of) Los Angeles.  Looking through meetup.com and the usual web search engines hasn't brought me much outside of the LA Hadoop user group and I was hoping for

Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

Hi Jay,          Sorry about the KIP formatting . I fixed those in the KIP. 2. We certainly need to add both the serialized and unserialized form for  the key as both are useful.  I added those to the interface. 3. Do we need to add the value? I suspect people will have uses for  computing

Re: [KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

Hey Sriharsha, Great, thanks! For 4: Yeah the use case for init and close is making use of any kind of metadata. An example of this would be if you are trying to do range partitioning you need to map lexicographic ranges to numeric partitions. You might do this by adding a new property to the

[KIP-DISCUSSION] KIP-22 Expose a Partitioner interface in the new producer

Hi, Here is the KIP for adding a partitioner interface for producer. https://cwiki.apache.org/confluence/display/KAFKA/KIP-+22+-+Expose+a+Partitioner+interface+in+the+new+producer There is one open question about how interface should look like. Please take a look and let me know if you

Re: Kafka client - 0.9

Thanks Gwen. I'm specifically looking for the consumer rewrite API ( org.apache.kafka.clients.consumer.KafkaConsumer). Based on the wiki, this feature is available only in 0.9. The specific use case is that, I wanted to use the high level consumer but with the ability to rollback the offset in

Re: Remote kafka - Connection refused.

Perhaps this will help: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan'tmyconsumers/producersconnecttothebrokers? On Thu, Apr 23, 2015 at 3:24 PM, madhavan kumar madhavan020...@gmail.com wrote: Dear all, I am trying to connect my python consumer to a remote kafka server.

Re: NoClassDefFoundError at producer end

Since it is a runtime error, Maven dependency is less relevant than what you have in your class path (unless you built a shaded uber-jar). You'll need Scala runtime and zkclient jar in the classpath, can you check that you have those around? On Thu, Apr 23, 2015 at 6:15 AM, abdul hameed pathan

Re: Kafka client - 0.9

We don't normally plan dates for releases, when we are done with features we want in the next release and happy with quality, we release. Many Apache communities are like that. If you need firmer roadmaps and specific release dates, there are few vendors selling Kafka distributions and support.