Take a look at secor: https://github.com/pinterest/secor
Secor is a no-frill kafka->HDFS/Ingesting tool, doesn't depend on any underlying systems such as Hadoop, it only uses Kafka high level consumer to balance the work loads. Very easy to understand and manage. It's probably the 2nd most popular kafka/HDFS ingestion tool (behind camus). Lots of web companies use this to do the kafka data ingestion (Pinterest/Uber/AirBnb). On Thu, Oct 22, 2015 at 3:56 AM, Adrian Woodhead <awoodh...@hotels.com> wrote: > Hello all, > > We're looking at options for getting data from Kafka onto HDFS and Camus > looks like the natural choice for this. It's also evident that LinkedIn who > originally created Camus are taking things in a different direction and are > advising people to use their Gobblin ETL framework instead. We feel that > Gobblin is overkill for many simple use cases and Camus seems a much > simpler and better fit. The problem now is that with LinkedIn apparently > withdrawing official support for it it appears that any changes to Camus > are being managed by various forks of it and it looks like everyone is > building and using their own versions. Wouldn't it be better for a > community to form around one official fork so development efforts can be > focused on this? Any thoughts on this? > > Thanks, > > Adrian > >
