OK, thanks again Jon for the detailed answers. I feel I'm ready to go. I'll use a holding disk and run archive dumps for offset storage.
-M On Wed, Mar 5, 2014 at 12:49 AM, Jon LaBadie <j...@jgcomp.com> wrote: > On Tue, Mar 04, 2014 at 05:16:05PM -0500, Michael Stauffer wrote: > > On Fri, Feb 28, 2014 at 1:07 AM, Jon LaBadie <j...@jgcomp.com> wrote: > > > > > On Thu, Feb 27, 2014 at 07:19:24PM -0500, Michael Stauffer wrote: > > > > Amanda 3.3.4 > > > > > > > > Hi, > > > > > > > > Another long post from me - thanks to anyone who has time to read it. > > > > > > > > I've been reading various docs and posts about dumpcycle, > runspercycle, > > > > tapecycle, runtapes online, but still can't figure out how I should > set > > > > things up for my needs. > > > > > > > > I've got: > > > > - ca. 30TB of data to backup > > > > - I'll try to split this into 500GB or 1TB DLE's, but some will be > much > > > > easier to leave as 2-4TB DLE's > > > > - I'm fine with a level 0 only every 30 days, especially considering > a > > > full > > > > backup will take 8-10 days. > > > > - I've got 1.5TB tapes that hold about 2TB of data with hardware > > > > compression from testing > > > > - my changer holds 35 tapes > > > > > > First question, will you be using more than 35 tapes. I.e. will you > > > periodically pull some recently used and replace with less recently > > > used tapes? If not, I think you are short on tapes. A full dump will > > > take 15 tapes. You really want a MINIMUM of two full dumps. I prefer > > > more. So 2x15 is two full dumps leaving only 5 tapes for incrementals. > > > I don't think that will be enough. > > > > > > > I was thinking of doing a periodic (at the start and then every 3 months) > > level 0 archive dump to a different set of tapes (probably four sets to > > retain 1 year's worth of dumps). Then I thought the library in the > changer > > would be fine if it held less than two level 0 dumps at any time. I'd > > rather just go switch tapes once every 3 months than more often, and have > > offset archive too. Does that seem reasonable? > > > I wouldn't be concerned about how many currently in the library. > I'd consider how many total I have. Anyone who has administered > tape backups for any significant time has a war story. Tapes that > seemed to write correctly but a month later were not readable. > Damage, physical and environmental. My own involves a large > magnet that I did not realize was right next to my tape storage. > > If you only have one level 0 and it goes bad, you have no backup! > > > > > > > > I'd like to have the changer always hold a level 0 dump and then the > set > > > of > > > > subseuqent incrementals. So it should take about 15-20 tapes for a > level > > > 0 > > > > of all DLE's, and then the incrementals over a month should easily > fit > > > > within 2-4 tapes, judging from experience here. When the next level 0 > > > dump > > > > starts, I'd like amanda to use as many of the remaining 10 or so > tapes > > > > before overwriting tapes from the old level 0 dump (overwriting only > > > tapes > > > > of DLE's that have just had a new level 0, of course). (I will > > > periodically > > > > do a level 0 dump to a different tape set for offset archiving) > > > > > > > Do you plan to let amanda do the scheduling? Or are you going to force > > > her kicking and screaming into the traditional schedule of one monster > > > full dump followed by all incrementals. Then another monster. Blech. > > > > > > I'm fine with Amanda's scheduling. When I do my first round of amdump's > > though, it will be effectively a monster dump until all DLE's have a > level > > 0. Then I presume amanda can even things out using her own scheduling? > > > > > You can ease amanda into your dumpcycle. Add a couple of DLEs with each > dump run until they are all added. Avoids the initial monster dump. > > > > > > How do I setup dumpcycle, runspercycle, tapecycle, runtapes to > achieve > > > this? > > > > > > > > >From the docs, it seems I'd want: > > > > dumpcycle 30 days > > > > runspercycle 15 #15, for running amdump every other day > > > > runtapes 2 # to allow for DLE's that can get up to 4TB > > > > tapecycle 34 # at least (runspercycle + 1) * runtapes - per docs > > > > suggestion > > > > # and leave one extra as a spare > > > > > > > > Is this right? > > > > > > > > Does this mean that when I run amdump, it will at most write two > > > > tapes-worth of DLE's, and then stop? Then the next run will pick up > from > > > > there? I think so, but would like to make sure. I'm used to the > manual > > > > paradigm of "run a full backup" and then do incrementals. But this > seems > > > > that it will level out to be the same in the end as that? > > > > > > > > HOWEVER, I'd rather have runtapes at 3 or 4 to minimize tape waste > and > > > make > > > > it less critical to have evenly-sized DLE's, which will be difficult > to > > > > maintain. But if runtapes is 3, the recommended value of tapecycle > would > > > be > > > > >= 48, more than my # of tapes. But in practice, 35 should still > plenty > > > of > > > > tapes to do what I want without overwriting level 0's prematurely. It > > > seems > > > > like tapecycle minimum should be more like '# of tapes per full > backup + > > > # > > > > of tapes for incrementals over dumpcycle + 2 * runtapes', plus one > or two > > > > as a buffer. > > > > > > The following sentence shows you are still thinking the "non-amanda > way. > > > > > > > The formula in the docs of "(runspercycle + 1) * > runtapes" > > > > plays it very safe when you consider many incremental dumps will go > to > > > > holding disk and be collected onto one tape periodically. > > > > > > There are not going to be a bunch of incrementls to collect onto one > > > tape. Each amdump run will be a mix of level 0's for some DLEs and > > > incrementals for the others. > > > > > > With 15 runspercycle you will AVERAGE 2TB of level 0 plus ??GB of > > > incrementals. But remember you said some DLEs will be 4 or more TB. > > > When they get level 0's you'll need more than 2 tapes. > > > > > > Can I be sure that there will always be some level 0's in a run of > amdump? > > I guess because there's ~30TB of data and 15 runs, so that pretty much > > guarantees a level 0 every run? If not, or if they're << tape-size, I'd > > like them to go to holding disk so as not to greatly underutilize a tape. > > I'm going to try to have my largest DLE < 2TB - there will be 3 or 4 of > > those probably - for simplicity's sake. > > Depends on the number and size of your DLEs and the dumpcycle. > > > However it's not clear, but it seems like when using a holding disk the > > whole DLE is written to holding disk before streaming to tape, which > seems > > inefficient time-wise when the DLE is large enough that's going to get > > streamed to tape anyway. > > Most experience the opposite. Taping is considerably faster than > dumping. So by dumping directly to tape you are streaming to tape > less than optimally. This results in the tape stopping, backing > up so that it can get up to speed at the point it needs to start > writing again. > > When you are dumping direct to tape, only one DLE is dumping. If > they are going to holding disk, several DLEs can be dumping at the > same time. > > > I guess I have to work params something like this: > > > > # to have amanda start writing when 10% of volume capacity is written > to > > holding disk, and > > # flush-threshold-scheduled is satisfied > > flush-threshold-dumped 10 > > > > # have amanda write to tape if the holding-disk storage plus planned > data > > exceeds this > > # % of tape capacity > > flush-threshold-scheduled 90 > > > > # taperflush param not needed if above two params are used > > #taperflush 90 > > > > Do I understand right that these above settings will tell amanda to start > > streaming to tape relatively early in a dump process if it knows it's > going > > to end up with >= 90% of volume capacity on the holding disk? I'm > thinking > > this should make the process more time efficient since it won't write > > everything to disk before streaming to tape? > > Haven't played with these params so have no guidance to offer. > > jl > -- > Jon H. LaBadie j...@jgcomp.com > 11226 South Shore Rd. (703) 787-0688 (H) > Reston, VA 20190 (609) 477-8330 (C) >
