[bareos-users] Always Incremental: Copy Job for the longterm pool

['Toni Burger' via bareos-users]

Hi again, 

I setup a Always Incremental bareos configuration, like described hier: 
https://docs.bareos.org/master/TasksAndConcepts/AlwaysIncrementalBackupScheme.html

Most of it is working correct, wihtout the last step. 
What i want to achieve: 

1) Incremental Backups from my clients -> go to AI-Pool [OK]
2) Consolidate the incrementals every ~month -> go to AI-Consolidated-Pool 
[OK]
3) Make a VirtualFullBackup every 3 Month -> go to AI-Longterm-Pool [OK]
4) Copy all Jobs from AI-Longterm-Pool to AI-Longterm-Extern-Pool, manually 
triggerd [not working]

Steps 1-3 are ok. I got a VirtualFullBackup in the AI-Longterm pool: 

Choose a query (1-21): 14
Enter Volume name: AI-Longterm-0011
+---+-+-+--+---+-+-++
| jobid | name| starttime   | type | level | files   | 
gb  | status |
+---+-+-+--+---+-+-++
| 4 | VirtualLongTerm | 2020-08-06 22:27:05 | A| F | 247,188 | 
417.346 | T  |
+---+-+-+--+---+-+-++
*


This is the pool definition for my AI-Longterm-Pool: 

Pool {
  Name = AI-Longterm
  Pool Type = Backup
  Next Pool = AI-Longterm-Extern
  Recycle = yes   # Bareos can automatically recycle 
Volumes
  Auto Prune = yes# Prune expired volumes
  Volume Retention = 7 months # How long should jobs be kept?
  Maximum Volume Bytes = 50G  # Limit Volume size to something 
reasonable
  Label Format = "AI-Longterm-"
  Volume Use Duration = 23h
  Storage = File
}

And this my AI-Longertm-Extern-Pool: 

Pool {
  Name = AI-Longterm-Extern
  Pool Type = Backup
  Recycle = yes   # Bacula can automatically recycle 
Volumes
  AutoPrune = yes # Prune expired volumes
  Volume Retention = 2 years  # 2 years fullbacks are stored / 
fullbackup is do>
  Maximum Volume Bytes = 50G  # Limit Volume size to something 
reasonable
  Volume Use Duration = 23h
  Storage = ExternFile
  Label Format = AI-Longterm-Extern-
}

And here is the copy job i definied: 

Job {
  Name = "LongtermCopyToExtern"
  Type = Copy
  Selection Type = PoolUncopiedJobs
  Messages = Standard
  Pool = AI-Longterm
  Level = Full
  Full Backup Pool = AI-Longterm
  Write Bootstrap = 
"/media/bareosExternBackup/bareos/bootstrap/LongtermCopyToExtern-%n.bsr"
}


Sadly, if i run this one, no jobs are found to be copied: 

07-Aug 15:18 storageSrv-dir JobId 11: No JobIds found to copy.
07-Aug 15:18 storageSrv-dir JobId 11: Bareos storageSrv-dir 19.2.7 
(16Apr20):
  Build OS:   Linux-3.10.0-1062.18.1.el7.x86_64 debian Debian 
GNU/Linux 10 (buster)
  Current JobId:  11
  Current Job:LongtermCopyToExtern.2020-08-07_15.18.54_04
  Catalog:"MyCatalog" (From Client resource)
  Start time: 07-Aug-2020 15:18:56
  End time:   07-Aug-2020 15:18:56
  Elapsed time:   0 secs
  Priority:   10
  Bareos binary info: bareos.org build: Get official binaries and 
vendor support on bareos.com
  Termination:Copying -- no files to copy


Maybe, it's because the jobs in AI-Longterm are of the type: Archive? How 
to select them? 

Or what else I'm missing? 

Thanks a lot. 
Toni

-- 
You received this message because you are subscribed to the Google Groups 
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to bareos-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/bareos-users/9df55c7a-a353-4910-b4ba-a62d6e4b3ec3n%40googlegroups.com.

Re: [bareos-users] Backup strategy for large dataset

[Brock Palen]

Yeah I don’t know any backup system that works the way you laid it out here.  
All I know of have to eventually roll up the backups into a consolidated 
version.  This is done two ways.

1. Periodic Full backup  (Classic method, pulling a full new copy of the data 
from the backup target, does not read back the old Full in the backup system)
2. Always Incremental  (new method where it’s done on the backup server, made 
possible by accurate jobs, reads the old Full and applies incrementals like 
patch diffs overtop creating a single new job)

Think about it, if you didn’t merge your full + incrementals over time, to do a 
restore in 1 year you would need your full tapes, which have not been checked 
in say 1 year,  plus 365 incremental jobs, applying one on top of the other 
till you get the current state.

The documentation for the Always Incremental has some really good plots showing 
the behavior.
https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html

What this means though is you have to have enough capacity to have 
Old Full + Incrementals + New Full

Thus you always need capacity of 2x + Incrementals the data volume you are 
backing up.  



Now if you do option 1.  you have a big load on your network and systems you 
are taking Fulls say 1x/month,  but you can in theory eject your volumes used 
for your Full backup, because you won’t read them back except on restore  (and 
periodic testing that restores work).  

You can also use classic Copy jobs without the wonky behavior I mentioned 
before because you have no Consolidate jobs.
https://docs.bareos.org/TasksAndConcepts/MigrationAndCopy.html#

Bareos doesn’t like having more than one Copy of a job in my experense, so in 
your case where you want 3 copies of your data,  I would do:
Primary Job to Disk
Copy Jobs to Tape  (Full and Incrementals)
Periodic VirtualFull Job to Tape  (Different Pool than above)

You can then eject the tapes safely.   In this setup if you have a Full + 
Incrementals on disk always for quick access, you in theory can run your Copy 
of your Full and your VirtualFull archive job, out of sync and you could get 
away with your single loader.

You will though still need 2x the disk as you will have that old full + new 
full overlapping for some period of time.  You could have your disk volumes 
expire before your tape volumes, but again you have a gap that for a restore 
you would need to go get your tapes from off site load them in the loader etc.  
before you could do a restore of any file that only resdies on those tape 
volumes.


Ah Dicom,  there is a project to replace our Pax at my day job, it’s no small 
job.



If you go with Option #2,   you likely will have VirtualFulls created more 
ofthen and thus burn through a lot more WORM media, Though you could put off 
setting Always Incremental Max Full Age  to 1 Month or something and other 
values so you have 1 month of incrementals, and once a month roll them up into 
a new full.

Doing this though with WORM you will have piles and piles of media.   For 2 off 
sites copies of 1 Full a month and 5 Tapes/35TByte,  you are looking at 120 
Tapes/year.

Double check,  can you take an archive snapshot once a month that’s WORM?   You 
could do that with the Archive job with the VirtualFull, set retention time, 
and it’s a frozen snapshot of how everything looked at that time.  But do your 
Backup Planning using traditional media you can recycle as jobs expire and are 
replaced with new ones.


Your idea of the second drive with the daily incremental, would probably work, 
then do the Copy job to the pool in the loader. 

Still though you need to sort out how your going to get your periodic fresh 
“Here if a full of the current state of my system”


You could do the wonky setup I do,  where I always consolidate back to disk, 
Migrate to tape, and expire on disk quickly to free space.  Your  Full in case 
#1  would read all the data form your servers to disk,  in Consolidation in #2, 
 you would read from WORM in the loader to disk, but then when you migrate from 
Disk to Tape the new full you won’t have enough space in your loader so you 
will have to eject the tapes and load new ones.   That may be fine for you.


Brock Palen
1 (989) 277-6075
bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting



> On Aug 7, 2020, at 3:58 AM, a.br...@mail.de  wrote:
> 
> Thanks a lot, Brock, for your comprehensive post and also to the others. I 
> haven't fully worked through your example cases yet, but it  will certainly 
> help me to get my head around it all. Maybe it helps if I provide a few more 
> details about how the data/images are organized:
> 
> I run a Linux based virtualization cluster on RAID6-hosts with Windows VMs. 
> The images are organised in windows-folders of 2TB size each, like 
> "E:\img\img01\" to currently "E:\img\img17\". 
> Once a folder is full, it's contents will never change again. They're like 
> archives 

Re: Re: [bareos-users] Re: installation of bareos fails

['Jon SCHEWE' via bareos-users]

Tilman,

Note that I specifically used the 16.04 repository because it still has 32-bit 
packages. These are working for me on my 18.04 system.


Jon Schewe

Research Scientist

C: +1 612.263.2718

O: +1 952.545.5720

jon.sch...@rtx.com

Raytheon Technologies

Raytheon Intelligence & Space

5775 Wayzata Blvd. Suite 630

St. Louis Park, MN 55416


RTX.com | 
LinkedIn | 
Twitter | 
Instagram


From: 'Tilman Glotzner' via bareos-users 
Sent: Thursday, August 6, 2020 18:26
To: bareos-users 
Subject: [External] Re: [bareos-users] Re: installation of bareos fails

Dear Jon

thank you for you reply. I have include the repository for 18.04 in the apt 
package system, i.e.I added
deb 
http://download.bareos.org/bareos/release/latest/xUbuntu_18.04
 /
to /etc/apt/sources.list.d/bareos.list

An apt search seems to return only 64 bit package. So I assume that the 
repository for 18.04 does not host any 32bit packages.
root:/usr/local/sbin# apt search bareos-director
..
bareos-director/unknown 19.2.7-2 amd64
  Backup Archiving Recovery Open Sourced - director daemon

bareos-director-python-plugin/unknown 19.2.7-2 amd64
  Backup Archiving Recovery Open Sourced - director Python plugin



In the meanwhile, I have pursued a different path and compiled bareos from 
source. After reimplementing a bug fix ( 
0001220) that apparently is not 
included in the source release but only  in a later version and recompiling it, 
if works.

Thanks again

Tilman


--
You received this message because you are subscribed to the Google Groups 
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
bareos-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/bareos-users/8e393960-eab6-4563-9fae-74ade0d58672o%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to bareos-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/bareos-users/SN5P110MB0527973F9948677757BCAA18ED490%40SN5P110MB0527.NAMP110.PROD.OUTLOOK.COM.

Re: [bareos-users] Re: installation of bareos fails

[Andreas Rogge]

Hi Tilman,

Am 07.08.20 um 01:26 schrieb 'Tilman Glotzner' via bareos-users:
> An apt search seems to return only 64 bit package. So I assume that the
> repository for 18.04 does not host any 32bit packages.
That's correct. We decided not to build i386 packages for Ubuntu 18.04
anymore. The last Intel CPUs that didn't support x86_64 were End-of-Life
in 2006 and we had the impression that 12 years was enough time to
assume people have finally migrated.

> In the meanwhile, I have pursued a different path and compiled bareos
> from source. After reimplementing a bug fix ( 0001220
> ) that apparently is not
> included in the source release but only  in a later version and
> recompiling it, if works.What sources were you building? For Bareos 19.2 that 
> patch is included
in the sources for 19.2.7 and later:

$ git tag --contains 60e7731
Release/19.2.7
Release/19.2.8

Best Regards,
Andreas

-- 
Andreas Rogge andreas.ro...@bareos.com
  Bareos GmbH & Co. KG  Phone: +49 221-630693-86
  http://www.bareos.com

  Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
  Komplementär: Bareos Verwaltungs-GmbH
  Geschäftsführer: S. Dühr, M. Außendorf, J. Steffens, Philipp Storz

-- 
You received this message because you are subscribed to the Google Groups 
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to bareos-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/bareos-users/ba97e4aa-f084-0001-3537-1cea240e60bb%40bareos.com.


signature.asc
Description: OpenPGP digital signature

[bareos-users] Re: How to configure a copy job for different source pools

['Toni Burger' via bareos-users]

Hi, 
i think this post isn't relevant anymore. I had a look at the "Always 
incremental feature" ... which fits more to my backup strategy. 
Also one question came up with this configuraiton, but I will start a new 
thread about that. 

Thanks. 
Toni
Toni Burger schrieb am Montag, 3. August 2020 um 14:51:56 UTC+2:

> Hi, 
> I try to understand the copy-job feature. I've a relativ simple setup. I 
> use three pools. One for full-backups, one for differential-backups and one 
> for incremental backups. These are configured like this: 
>
> Pool {
>   Name = Full-Pool
>   Pool Type = Backup
>   Next Pool = Extern-Full-Pool
>   Recycle = yes   
>   AutoPrune = yes 
>   Volume Retention = 6 months 
>   Maximum Volume Bytes = 50G  
>   Maximum Volumes = 35
>   Storage = File
>   Label Format = Full-
> }
>
> Pool {
>   Name = Diff-Pool
>   Pool Type = Backup
>   Next Pool = Extern-Diff-Pool
>   Recycle = yes   
>   AutoPrune = yes 
>   Volume Retention = 4 month  
>   Storage = File
>   Maximum Volume Bytes = 50G  
>   Maximum Volumes = 2 
>   Storage = File
>   Label Format = Diff-
> }
>
>
> Pool {
>   Name = Inc-Pool
>   Pool Type = Backup
>   Next Pool = Extern-Inc-Pool
>   Recycle = yes   
>   AutoPrune = yes 
>   Volume Retention = 20days   
>   Storage = File
>   Maximum Volume Bytes = 50G  
>   Maximum Volumes = 3 
>   Storage = File
>   Label Format = Inc-
> }
>
>
> Now, I want to define a copy job, which copies all uncopied jobs in the 
> pools to the corresponding pools (you can see the Next Pool confiugration 
> above) on an external usb drive. Therefore I setup my new external pools: 
>
> Pool {
>   Name = Extern-Full-Pool
>   Pool Type = Backup
>   Recycle = yes   
>   AutoPrune = yes 
>   Volume Retention = 2 years  
>   Maximum Volume Bytes = 50G 
>   Maximum Volumes = 35   
>   Storage = ExternFile
>   Label Format = Extern-Full-
> }
>
> Pool {
>   Name = Extern-Diff-Pool
>   Pool Type = Backup
>   Recycle = yes   
>   AutoPrune = yes 
>   Volume Retention = 4 month  
>   Maximum Volume Bytes = 50G  
>   Maximum Volumes = 2 
>   Storage = ExternFile
>   Label Format = Extern-Diff-
> }
>
>
> Pool {
>   Name = Extern-Inc-Pool
>   Pool Type = Backup
>   Recycle = yes   
>   AutoPrune = yes 
>   Volume Retention = 20days   
>   Maximum Volume Bytes = 50G  
>   Maximum Volumes = 3 
>   Storage = ExternFile
>   Label Format = Extern-Inc-
> }
>
>
> My problem is now to configure the copyJob :). My first approach was this 
> one: 
>
> Job {
>   Name = "CopyToExtern"
>   Type = Copy
>   Selection Type = PoolUncopiedJobs
>   Messages = Standard
>   Pool = Full-Pool
>
>   Full  Backup Pool = Full-Pool
>   Differential Backup Pool = Diff-Pool
>   Incremental Backup Pool = Inc-Pool
> }
>
>
> I think i got almost there. If I run this job, I get a copy of my 
> Full-Pool in my External-Full-Pool. But how to trigger the Inc and Diff 
> Pool copy action? 
> Do I need to define 3 different copy jobs, which targets every pool? I 
> tried to modify the Pool Parameter in this job, but this doesn't find jobs 
> to copy. 
>
> Thanks and best regards
> Toni
>

-- 
You received this message because you are subscribed to the Google Groups 
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to bareos-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/bareos-users/7864c772-385c-4f21-9509-4779dfd56cd8n%40googlegroups.com.

Re: [bareos-users] Backup strategy for large dataset

[a.br...@mail.de]

 Thanks a lot, Brock, for your comprehensive post and also to the others. I 
haven't fully worked through your example cases yet, but it  will certainly 
help me to get my head around it all. Maybe it helps if I provide a few 
more details about how the data/images are organized:

I run a Linux based virtualization cluster on RAID6-hosts with Windows VMs. 
The images are organised in windows-folders of 2TB size each, like 
"E:\img\img01\" to currently "E:\img\img17\". 
Once a folder is full, it's contents will never change again. They're like 
archives that will be read from but no more written to. 

So I thought I'd proceed like this:
1. Backup "img01" to "img17" to tape, store the tapes offsite. 
2. Do this a second time and store the tapes offsite, seperate from the 
first generation.
3. Do this a third time to disk, for quick access if needed.
4. Make sure the catalog of at least 1. and 2. is in a very safe place.
5. Develop a daily backup strategy - starting with "img18".

As for  (1.) - (3.) I have created seperate Full-jobs for each 
imgXX-folder. (1.) has already completed successfully, (2.) is currently in 
progress.
I thought that once (1.) and (2.) are completed successfully I'm safe what 
"img01-17" is concerned and never have to consider these folders for backup 
again. Right or am I missing something?

What I'd like to discuss here is (5.) - under consideration of a few 
parameters:
- the daily increment of image data is roughly 50 GB. BTW: The images 
(DICOM, JPEG2000) don't compress at all :). 
- for legal reasons we have to store the images on WORM-media. So I need a 
daily job that writes to tape.
- the doctors want best possible protection against fire, supernova, 
Borg-attack etc. They want a daily tape change routine with the latest 
WORM-tape taken offsite. 

For the daily tape change I could buy another LTO drive. I can also expand 
my backup-server to fit above (3.) and the daily increment.

So, here's what I thought I need to develop:
- Backup the daily increment to disk.
- Backup the same daily increment to a tape (in a new 1-slot drive) that is 
part of a "daily change" pool of tapes (MON-SAT or so...)
- Append the same daily increment to another WORM tape in the 8-slot 
loader. Once the tape is full, take it offsite and add a fresh tape in the 
loader.

If that strategy doesn't sound to weird I need to transfer this into a 
working bareos config.
Sorry if it all sounds confusing but for me it's still really, really 
complex.

Thanks
Andreas



spadaj...@gmail.com schrieb am Freitag, 7. August 2020 um 09:56:48 UTC+2:

> As I wrote earlier, this looks more like archiving plan, not a backup one 
> (or a combination of backup and archiving). But more to the point - in case 
> of backups you have to have a verification plan and periodical restore 
> tests. In case of archiving you need to have a verification plan (i.e. 
> every half a year you read each archive unit, check whether it's readable 
> and if its checksum is correct; in case of more critical data you might 
> want to even have some kind of functional tests like trying to read the 
> data into appropriate software and check whether the data is still 
> parseable and loadable; If any copy fails you'd want to re-create it from 
> other existing copies).
> On 07.08.2020 09:49, a.br...@mail.de wrote:
>
> Thanks a lot, Brock, for your comprehensive post and also to the others. I 
> haven't fully worked through your example cases yet, but it  will certainly 
> help me to get my head around it all. Maybe it helps if I provide a few 
> more details about how the data/images are organized:
>
> I run a Linux based virtualization cluster on RAID6-hosts with Windows 
> VMs. 
> The images are organised in windows-folders of 2TB size each, like 
> "E:\img\img01\" to currently "E:\img\img17\". 
> Once a folder is full, it's contents will never change again. They're like 
> archives that will be read from but no more written to. 
>
> So I thought I'd proceed like this:
> 1. Backup "img01" to "img17" to tape, store the tapes offsite. 
> 2. Do this a second time and store the tapes offsite, seperate from the 
> first generation.
> 3. Do this a third time to disk, for quick access if needed.
> 4. Make sure the catalog of at least 1. and 2. is in a very safe place.
> 5. Develop a daily backup strategy - starting with "img18".
>
> As for  (1.) - (3.) I have created seperate Full-jobs for each 
> imgXX-folder. (1.) has already completed successfully, (2.) is currently in 
> progress.
> I thought that once (1.) and (2.) are completed successfully I'm safe what 
> "img01-17" is concerned and never have to consider these folders for backup 
> again. Right or am I missing something?
>
> What I'd like to discuss here is (5.) - under consideration of a few 
> parameters:
> - the daily increment of image data is roughly 50 GB. BTW: The images 
> (DICOM, JPEG2000) don't compress at all :). 
> - for legal reasons we have to store the images on 

Re: [bareos-users] Backup strategy for large dataset

[Spadajspadaj]

As I wrote earlier, this looks more like archiving plan, not a backup 
one (or a combination of backup and archiving). But more to the point - 
in case of backups you have to have a verification plan and periodical 
restore tests. In case of archiving you need to have a verification plan 
(i.e. every half a year you read each archive unit, check whether it's 
readable and if its checksum is correct; in case of more critical data 
you might want to even have some kind of functional tests like trying to 
read the data into appropriate software and check whether the data is 
still parseable and loadable; If any copy fails you'd want to re-create 
it from other existing copies).


On 07.08.2020 09:49, a.br...@mail.de wrote:
Thanks a lot, Brock, for your comprehensive post and also to the 
others. I haven't fully worked through your example cases yet, but it  
will certainly help me to get my head around it all. Maybe it helps if 
I provide a few more details about how the data/images are organized:


I run a Linux based virtualization cluster on RAID6-hosts with Windows 
VMs.
The images are organised in windows-folders of 2TB size each, like 
"E:\img\img01\" to currently "E:\img\img17\".
Once a folder is full, it's contents will never change again. They're 
like archives that will be read from but no more written to.


So I thought I'd proceed like this:
1. Backup "img01" to "img17" to tape, store the tapes offsite.
2. Do this a second time and store the tapes offsite, seperate from 
the first generation.

3. Do this a third time to disk, for quick access if needed.
4. Make sure the catalog of at least 1. and 2. is in a very safe place.
5. Develop a daily backup strategy - starting with "img18".

As for  (1.) - (3.) I have created seperate Full-jobs for each 
imgXX-folder. (1.) has already completed successfully, (2.) is 
currently in progress.
I thought that once (1.) and (2.) are completed successfully I'm safe 
what "img01-17" is concerned and never have to consider these folders 
for backup again. Right or am I missing something?


What I'd like to discuss here is (5.) - under consideration of a few 
parameters:
- the daily increment of image data is roughly 50 GB. BTW: The images 
(DICOM, JPEG2000) don't compress at all :).
- for legal reasons we have to store the images on WORM-media. So I 
need a daily job that writes to tape.
- the doctors want best possible protection against fire, supernova, 
Borg-attack etc. They want a daily tape change routine with the latest 
WORM-tape taken offsite.


For the daily tape change I could buy another LTO drive. I can also 
expand my backup-server to fit above (3.) and the daily increment.


So, here's what I thought I need to develop:
- Backup the daily increment to disk.
- Backup the same daily increment to a WORM tape (in a new 1-slot 
drive) that is part of a "daily change" pool of tapes (MON-SAT or so...)
- Append the same daily increment to another WORM tape in the 8-slot 
loader. Once the tape is full, take it offsite and add a fresh tape in 
the loader.


If that strategy doesn't sound to weird I need to transfer this into a 
working bareos config.
Sorry if it all sounds confusing but for me it's still really, really 
complex.


Thanks
Andreas

bro...@mlds-networks.com schrieb am Mittwoch, 5. August 2020 um 
20:21:10 UTC+2:


You will have some complexity with the size of your data and the
size of your loader. Unless your data compresses really well.
Does it have more than one tape drive? Your total loader capacity
is 48 TBytes raw, and you need 2x your full size to do
Consolidations or new Fulls or you have gaps in your protection.

If I’m reading this right you want an off site copy.

If that’s correct I would go about this two different ways,

* Get a much bigger loader with 2 drives
or
* Expand backups server raid6 to have Full + growth*Growth Window
capacity

I would then use migrate+Archive jobs to make my off site and copy
to tape.

In the first case you can avoid the extra migrate, just do an
archive to a pool of tapes you eject.

Workflow case 1 Bigger Tape loader 2 or more tape drives.
* Spool to Disk
* AI-Consolidated Pool Tape
* AI-Incremental Pool Disk
* Offsite Pool Tape

Fulls and Consolidations backups go to AI-Consolidated Tape pool,
Your daily go to disk until they are consolidated into tape.

To create your off sites you can use a copy job of whatever full
you want.

https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#copy-jobs


I personally for offsite to avoid issues with Always Incremental
jobs use an Archive job

https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#virtual-full-jobs


This avoids the offsite tapes being upgraded to the primary copy
when the Consolidate job prunes the older jobs.

To do a VirtualFull archive job like this though you need enough

Re: [bareos-users] Backup strategy for large dataset

[a.br...@mail.de]

Thanks a lot, Brock, for your comprehensive post and also to the others. I 
haven't fully worked through your example cases yet, but it  will certainly 
help me to get my head around it all. Maybe it helps if I provide a few 
more details about how the data/images are organized:

I run a Linux based virtualization cluster on RAID6-hosts with Windows VMs. 
The images are organised in windows-folders of 2TB size each, like 
"E:\img\img01\" to currently "E:\img\img17\". 
Once a folder is full, it's contents will never change again. They're like 
archives that will be read from but no more written to. 

So I thought I'd proceed like this:
1. Backup "img01" to "img17" to tape, store the tapes offsite. 
2. Do this a second time and store the tapes offsite, seperate from the 
first generation.
3. Do this a third time to disk, for quick access if needed.
4. Make sure the catalog of at least 1. and 2. is in a very safe place.
5. Develop a daily backup strategy - starting with "img18".

As for  (1.) - (3.) I have created seperate Full-jobs for each 
imgXX-folder. (1.) has already completed successfully, (2.) is currently in 
progress.
I thought that once (1.) and (2.) are completed successfully I'm safe what 
"img01-17" is concerned and never have to consider these folders for backup 
again. Right or am I missing something?

What I'd like to discuss here is (5.) - under consideration of a few 
parameters:
- the daily increment of image data is roughly 50 GB. BTW: The images 
(DICOM, JPEG2000) don't compress at all :). 
- for legal reasons we have to store the images on WORM-media. So I need a 
daily job that writes to tape.
- the doctors want best possible protection against fire, supernova, 
Borg-attack etc. They want a daily tape change routine with the latest 
WORM-tape taken offsite. 

For the daily tape change I could buy another LTO drive. I can also expand 
my backup-server to fit above (3.) and the daily increment.

So, here's what I thought I need to develop:
- Backup the daily increment to disk.
- Backup the same daily increment to a WORM tape (in a new 1-slot drive) 
that is part of a "daily change" pool of tapes (MON-SAT or so...)
- Append the same daily increment to another WORM tape in the 8-slot 
loader. Once the tape is full, take it offsite and add a fresh tape in the 
loader.

If that strategy doesn't sound to weird I need to transfer this into a 
working bareos config.
Sorry if it all sounds confusing but for me it's still really, really 
complex.

Thanks
Andreas

bro...@mlds-networks.com schrieb am Mittwoch, 5. August 2020 um 20:21:10 
UTC+2:

> You will have some complexity with the size of your data and the size of 
> your loader. Unless your data compresses really well.
> Does it have more than one tape drive? Your total loader capacity is 48 
> TBytes raw, and you need 2x your full size to do Consolidations or new 
> Fulls or you have gaps in your protection.
>
> If I’m reading this right you want an off site copy.
>
> If that’s correct I would go about this two different ways, 
>
> * Get a much bigger loader with 2 drives
> or
> * Expand backups server raid6 to have Full + growth*Growth Window capacity 
>
> I would then use migrate+Archive jobs to make my off site and copy to tape.
>
> In the first case you can avoid the extra migrate, just do an archive to a 
> pool of tapes you eject.
>
> Workflow case 1 Bigger Tape loader 2 or more tape drives.
> * Spool to Disk
> * AI-Consolidated Pool Tape
> * AI-Incremental Pool Disk
> * Offsite Pool Tape
>
> Fulls and Consolidations backups go to AI-Consolidated Tape pool,
> Your daily go to disk until they are consolidated into tape.
>
> To create your off sites you can use a copy job of whatever full you want.
>
> https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#copy-jobs
>
> I personally for offsite to avoid issues with Always Incremental jobs use 
> an Archive job
>
> https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#virtual-full-jobs
>
> This avoids the offsite tapes being upgraded to the primary copy when the 
> Consolidate job prunes the older jobs.
>
> To do a VirtualFull archive job like this though you need enough tape for 
> 2x the data otherwise your swapping tapes every 5.5 hours.
> I would then use a script called at the end of the archive job to eject 
> the volumes and make volumestatus=Used from that Offsite pool. Load new 
> tapes label and put back in Offsite for the next archive.
>
> To do AI jobs this way and the VirtualFull archive job you need 2 tape 
> drives to read from one pool/tape and write to the other. It will be fast 
> as you get LTO speeds. No second Full backup.
>
>
>
> Workflow case 2 Bigger Disk in Backup Server
> * Spool if you want to avoid failed jobs cluttering disk volumes, maybe 
> skip.
> * AI-Consolidated Pool Disk
> * AI-Incremental Pool Disk
> * Offsite Pool Tape
>
> This would work similar, but your Fulls and Consolidations all happen on 
> Disk. This