May be you can try Apache NiFi its quicker as well. Give a try good luck
-----Original Message----- From: R P [mailto:hadoo...@outlook.com] Sent: 11 February 2016 16:09 To: users@kafka.apache.org Subject: Re: What is the best way to write Kafka data into HDFS? Hello Steve, Thanks for the suggestion. Looks like this Git repo is not updated for more than 10 months. Is this project still supported? Where can I find current usage and performance metrics ? Thanks, R P ________________________________________ From: steve.mo...@gmail.com <steve.mo...@gmail.com> on behalf of Steve Morin <st...@stevemorin.com> Sent: Wednesday, February 10, 2016 6:36 PM To: users@kafka.apache.org Subject: Re: What is the best way to write Kafka data into HDFS? R P, happy to walk you through https://github.com/DemandCube/Scribengin if your interested On Wed, Feb 10, 2016 at 5:09 PM, R P <hadoo...@outlook.com> wrote: > Hello All, > New Kafka user here. What is the best way to write Kafka data into HDFS? > I have looked into following options and found that Flume is quickest > and easiest to setup. > > 1. Flume > 2. KaBoom > 3. Kafka Hadoop Loader > 4. Camus -> Gobblin > > Although Flume can result into small file problems when your data is > partitioned and some partitions generate sporadic data. > > What are some best practices and options to write data from Kafka to HDFS? > > Thanks, > R P > > > > -- *Steve Morin | Managing Partner - CTO* *Nvent* O 800-407-1156 ext 803 <800-407-1156;803> | M 347-453-5579 smo...@nventdata.com <smo...@nventdata.com> *Enabling the Data Driven Enterprise* *(Ask us how we can setup scalable open source realtime billion+ event/data collection/analytics infrastructure in weeks)* Service Areas: Management & Strategy Consulting | Data Engineering | Data Science & Visualization BigData Technologies: Hadoop & Ecosystem | NoSql| Hbase | Cassandra | Storm | Spark | Kafka | Mesos | Docker | & More Industries: IoT | Advertising | Retail | Manufacturing | TV & Cable | Energy | Oil & Gas | Insurance | Finance | Telecom
