You should do this as a consumer (i.e. "archiveDataConsumer") Take a look at the AWS section of the eco system https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem (e.g. https://github.com/pinterest/secor ).
Also the tools is a good place to check out https://cwiki.apache.org/confluence/display/KAFKA/System+Tools (e.g. https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-MirrorMaker ). If there isn't a consumer you need you could write one (most often what folks do) or google and maybe find it and let the community know. Thanks! /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Mon, Jun 16, 2014 at 6:02 AM, Anatoly Deyneka <adeyn...@gmail.com> wrote: > Hi all, > > I'm looking for the way of archiving data. > The data is hot for few days in our system. > After that it can rarely be used. Speed is not so important for archive. > > Lets say we have kafka cluster and storage system. > It would be great if kafka supported moving data to storage system instead > of eviction and end user could specify what storage system is used(dynamo, > s3, hadoop, etc...). > Is it possible to implement? > > What other solutions you can advice? > > Regards, > Anatoly >
