Re: Kafka Backup Strategy

2016-12-22 Thread Stephane Maarek
Thanks Andrew for the detailed response We’re having a replication factor of 3 so we’re safe there. What do you recommend for min.insync.replicas, acks and log flush internal? I was worried about region failures or someone that goes in and deletes our instances and associated volumes (that’s a

Re: Kafka Backup Strategy

2016-12-21 Thread Andrew Clarkson
Hi Stephane, I say this not to be condescending in any way, but simple replication *might* cover your needs. This will cover most node failures (causing unclean shutdown) like disk or power failure. This assumes that one of the replicas of your data survives (see the configs min.insync.replicas,

Kafka Backup Strategy

2016-12-21 Thread Stephane Maarek
Hi, I have Kafka running on EC2 in AWS. I would like to backup my data volumes daily in order to recover to a point in time in case of a disaster. One thing I’m worried about is that if I do an EBS snapshot while Kafka is running, it seems a Kafka that recovers on it will have to deal with