-----Original Message----- From: Gerard Klijs [mailto:gerard.kl...@dizzit.com] Sent: Wednesday, May 11, 2016 3:00 AM To: users@kafka.apache.org Subject: Re: Backing up Kafka data and using it later?
You could create a docker image with a kafka installation, and start a mirror maker in it, you could set the retention time for it to infinite, and mount the data volume. With the data you could always restart the docker, en mirror it to somewhere else. Not sure that's what you want, but it's an option to save data for use some other place/time. On Wed, May 11, 2016 at 12:33 AM Alex Loddengaard <a...@confluent.io> wrote: > You may find this interesting, although I don't believe it's exactly > what you're looking for: > > https://github.com/pinterest/secor > > I'm not sure how stable and commonly used it is. > > Additionally, I see a lot of users use MirrorMaker for a "backup," > where MirrorMaker copies all topics from one Kafka cluster to another "backup" > cluster. I put "backup" in quotes because this architecture doesn't > support snapshotting like a traditional backup would. I realize this > doesn't address your specific use case, but thought you may find it > interesting regardless. > > Sorry I'm a little late to the thread, too. > > Alex > > On Thu, May 5, 2016 at 7:05 AM, Rad Gruchalski <ra...@gruchalski.com> > wrote: > > > John, > > > > I’m not as expert expert in Kafka but I would assume so. > > > > > > > > > > > > > > > > > > > > > > Best regards, > > Radek Gruchalski > > ra...@gruchalski.com (mailto:ra...@gruchalski.com) (mailto: > > ra...@gruchalski.com) > > de.linkedin.com/in/radgruchalski/ ( > > http://de.linkedin.com/in/radgruchalski/) > > > > Confidentiality: > > This communication is intended for the above-named person and may be > > confidential and/or legally privileged. > > If it has come to you in error you must take no action based on it, > > nor must you copy or show it to anyone; please delete/destroy and > > inform the sender immediately. > > > > > > > > On Thursday, 5 May 2016 at 01:46, John Bickerstaff wrote: > > > > > Thanks - does that mean that the only way to safely back up Kafka > > > is to have replication? > > > > > > (I have done this partially - I can get the entire topic on the > > > command line, after completely recreating the server, but my code > > > that is > > intended > > > to do the same thing just hangs) > > > > > > On Wed, May 4, 2016 at 3:18 PM, Rad Gruchalski > > > <ra...@gruchalski.com > > (mailto:ra...@gruchalski.com)> wrote: > > > > > > > John, > > > > > > > > I believe you mean something along the lines of: > > > > http://markmail.org/message/f7xb5okr3ujkplk4 > > > > I don’t think something like this has been done. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > Radek Gruchalski > > > > ra...@gruchalski.com (mailto:ra...@gruchalski.com) (mailto: > > > > ra...@gruchalski.com (mailto:ra...@gruchalski.com)) > > > > de.linkedin.com/in/radgruchalski/ ( > > http://de.linkedin.com/in/radgruchalski/) ( > > > > http://de.linkedin.com/in/radgruchalski/) > > > > > > > > Confidentiality: > > > > This communication is intended for the above-named person and > > > > may be confidential and/or legally privileged. > > > > If it has come to you in error you must take no action based on > > > > it, > nor > > > > must you copy or show it to anyone; please delete/destroy and > > > > inform > > the > > > > sender immediately. > > > > > > > > > > > > > > > > On Wednesday, 4 May 2016 at 23:04, John Bickerstaff wrote: > > > > > > > > > Hi, > > > > > > > > > > I have what is probably an edge use case. I'd like to back up > > > > > a > > single > > > > > Kafka instance such that I can recreate a new server, drop > > > > > Kafka > in, > > drop > > > > > the data in, start Kafka -- and have all my data ready to go > > > > > again > > for > > > > > consumers. > > > > > > > > > > Is such a thing done? Does anyone have any experience trying this? > > > > > > > > > > I have, and I've run into some problems which suggest there's > > > > > a > > setting > > > > or > > > > > some other thing I'm unaware of... > > > > > > > > > > If you like, don't think of it as a backup problem so much as > > > > > a > > "cloning" > > > > > problem. I want to clone a new Kafka machine without actually > > cloning it > > > > > > > > > > > > > - > > > > > I.E. the data is somewhere else (log and index files) although > > Zookeeper > > > > > > > > is > > > > > up and running just fine. > > > > > > > > > > Thanks > > > > >
