Make kafka storage engine pluggable and provide a HDFS plugin?

Hi there, I recently started to use Kafka for our data analysis pipeline and it works very well. One problem to us so far is expanding our cluster when we need more storage space. Kafka provides some scripts for helping do this but the process wasn't smooth. To make it work perfectly, seems Kafk

Re: Make kafka storage engine pluggable and provide a HDFS plugin?

Take a look at Camus François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure Capitaine Club Capra VP-Communication - CS Games 2014 Jeux de Génie

Re: Make kafka storage engine pluggable and provide a HDFS plugin?

Thanks Jun and Francois. We used Kafka 0.8.0 previously. We got some weird error when expanding cluster and it couldn't be finished. Now we use 0.8.1.1, I would have a try on cluster expansion sometime. I read the discussion on that jira issue and I agree with points raised there. HDFS was also i

Re: Make kafka storage engine pluggable and provide a HDFS plugin?

Hangjun, Does having Kafka in Yarn would be a big architectural change from where it is now? From what I have seen on most typical setup you want machines optimized for Kafka, not just it on top of hdfs. -Steve On Tue, May 20, 2014 at 8:37 PM, Hangjun Ye wrote: > Thanks Jun and Francois. > >

Re: Make kafka storage engine pluggable and provide a HDFS plugin?

Hi Steve, Yes, what I want is that Kafka doesn't have to care about machines physically (as an option). Best, Hangjun 2014-05-21 11:46 GMT+08:00 Steve Morin : > Hangjun, > Does having Kafka in Yarn would be a big architectural change from where > it is now? From what I have seen on most typi

Re: Make kafka storage engine pluggable and provide a HDFS plugin?

Hi Hangjun I've explored deploying kafka on yarn and current YARN does not support long running services with locality constraints. Deploying kafka producers / consumers (not brokers) is supported in the apache incubator samza project. Background on YARN limitations can be found here: YARN-371,