If you use camus to backup you could use something like this? https://groups.google.com/forum/#!searchin/camus_etl/sumac/camus_etl/hIi1cbOCFvU/grqd7UHLoFUJ
Please note I haven't used this, but remember seeing it ages ago. On Wed, Mar 16, 2016 at 2:24 PM, Giidox <aki-xodiig-...@marmelandia.com> wrote: > > I'm not sure what HDFS backups would bring > > I’m not sure how realistic the threat is, but I was thinking a case in > which bug in Kafka corrupts the log files. I would personally sleep better > knowing there is a Kafka-independent backup of all data. > > > how would you recover from e.g. all Kafka nodes blowing up if you only > have an HDFS backup > > I’m not sure which problem you see. I am thinking a backup which has a > copy of every message and its key, by partition, in offset order. The > backup would be appended with new messages as fast as they appear in Kafka. > It should be possible to recreate a Kafka cluster by creating the topics > and producing messages from the backup in original order into the empty > Kafka cluster. In a system in which consumers maintain their own offsets, > consumers should in theory see no change after the reconstruction > (consumers should be shut down while the recreation is ongoing). -- This email, including attachments, is private and confidential. If you have received this email in error please notify the sender and delete it from your system. Emails are not secure and may contain viruses. No liability can be accepted for viruses that might be transferred by this email or any attachment. Any unauthorised copying of this message or unauthorised distribution and publication of the information contained herein are prohibited. 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB. Registered in England and Wales. Registered No. 04843573.
