Re: Kafka on yarn

Hi Gwen Your recommendations in the field to partition off non-cluster nodes and reserve them for kafka brokers totally make sense given current YARN limitations. I'm familiar with the llama hacks - effectively reserving containers with dummy processes that just sit there and then running the '

Re: Kafka on yarn

till there. -Jay On Wed, Jul 23, 2014 at 4:44 PM, Kam Kasravi wrote: > Thanks Joe for the input related to Mesos as well as acknowledging the need > for YARN to support this type of cluster allocation - long running services > with node locality priority. > > Thanks Jay - That&#

Re: Kafka on yarn

and secondarily new functionality in HDFS that depending on the use-case can be very interesting with in-memory files.   -Steve On Wed, Jul 23, 2014 at 4:44 PM, Kam Kasravi wrote: Thanks Joe for the input related to Mesos as well as acknowledging the need for YARN to support this type of cluster alloc

Re: Kafka on yarn

inal data, but this isn't strictly necessary, it is just a good optimization. As long as you run with replication you can restart a broker elsewhere with no data, and it will restore it's state off the other replicas. -Jay On Wed, Jul 23, 2014 at 3:47 PM, Kam Kasravi wrote: >

Re: Kafka on yarn

Hi  Kafka-on-yarn requires YARN to consistently allocate a kafka broker at a particular resource since the broker needs to always use its local data. YARN doesn't do this well, unless you provide (override) the default scheduler (CapacityScheduler or FairScheduler). SequenceIO did something alo

Re: Make kafka storage engine pluggable and provide a HDFS plugin?

Hi Hangjun I've explored deploying kafka on yarn and current YARN does not support long running services with locality constraints. Deploying kafka producers / consumers (not brokers) is supported in the apache incubator samza project. Background on YARN limitations can be found here: YARN-371,

Re: consumer partition rebalancing

. Perhaps this starts to get into Samza territory. On Sep 11, 2013, at 1:14 PM, Joel Koshy wrote: > Correct - but since you wanted sticky allocation rebalancing wouldn't > really be necessary. > > Thanks, > > Joel > > On Wed, Sep 11, 2013 at 10:08 AM, Kam Kasravi w

Re: consumer partition rebalancing

quot;users@kafka.apache.org" ; Kam Kasravi Sent: Tuesday, September 10, 2013 4:36 PM Subject: Re: consumer partition rebalancing >        * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation >within a topic and 2) theconstraintthat a single consumer C(i) within a >co

consumer partition rebalancing

I wasreviewingtheconsumer partition rebalancingalgorithm and had a fewrelatedquestions  * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a topic and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be the only reader of that partition:

Re: Kafka/Hadoop consumers and producers

Thanks - I'll ask on bigtop regarding the .deb requirement - it seems they don't abide by this. I may merge a bit of your work into bigtop-989 if that's ok with you. I do know the bigtop folks  would like to see sbt support. From: Andrew Otto

Re: Kafka/Hadoop consumers and producers

Thanks Andrew - I like the shell wrapper - very clean and simple.  What installs all the kafka dependencies under /usr/share/java? From: Andrew Otto To: Kam Kasravi Cc: "d...@kafka.apache.org" ; Ken Goodhope ; Andrew Psaltis ; "dib

Re: Kafka/Hadoop consumers and producers

Thanks Andrew. I upgrade it to use the high level consumer. From: Andrew Psaltis To: "users@kafka.apache.org" ; Kam Kasravi ; "d...@kafka.apache.org" ; Ken Goodhope Cc: Andrew Psaltis ; "dibyendu.bhattacha...@pearson.com"

Re: Kafka/Hadoop consumers and producers

I would like to do this refactoring since I did a high level consumer a while ago.  A few weeks ago I had opened KAFKA-949 Kafka on Yarn which I was also hoping to add to contribute. It's almost done. KAFKA-949 is paired with BIGTOP-989 which adds kafka 0.8 to the bigtop distribution. KAFKA-949