Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Rao, Uthra R. (GSFC-672.0)[ADNET SYSTEMS INC] via Bacula-users]

Thank you all for your response to my question. So, it looks like there is no 
easy way one can set-up bacula to backup simultaneously to the tape and to the 
cloud.

UR.


From: Davide Franco 
Sent: Thursday, January 30, 2020 12:41 AM
To: Rao, Uthra R. (GSFC-672.0)[ADNET SYSTEMS INC] 
Cc: bacula-users@lists.sourceforge.net
Subject: [EXTERNAL] Re: [Bacula-users] bacula cloud backup

Hello,

IMHO, I’m not sure you can perform simultaneous backup jobs to both tape and S3 
(please let me know if I’m wrong).

I’d suggest maybe to use copy jobs like this

1. Backup to tapes
2. Copy job to S3 bucket based on previous backup on tapes.

I don’t have huge experience with copy jobs, hopefully someone reading this can 
give more hints.

Best,

Davide

On Tue, 28 Jan 2020 at 20:18, Rao, Uthra R. (GSFC-672.0)[ADNET SYSTEMS INC] via 
Bacula-users 
mailto:bacula-users@lists.sourceforge.net>> 
wrote:
I have recently upgraded my bacula server to version 9.4.4 and wanted to test 
the cloud backup feature. Currently we are backing up to LTO7 tapes. Is it 
possible to set-up bacula to backup both to the tape and to the cloud 
simultaneously?
If so I would appreciate some help with that.

Thank you.
UR.
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Phil Stracchino]

On 2020-01-30 10:12, Rao, Uthra R. (GSFC-672.0)[ADNET SYSTEMS INC] via
Bacula-users wrote:
> Thank you all for your response to my question. So, it looks like there
> is no easy way one can set-up bacula to backup simultaneously to the
> tape and to the cloud.


There is not a way to stream the same backup job to two different
storage devices simultaneously, no.  That question has come up before.
But as Davide suggests, you could copy the job(s) afterwards.  I'm doing
something along these lines; my backups go to disk, but Full backups
then get post-copied to removable disk sets.



-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Heitor Faria]

Hello Uthra, 

You have currently 3 options: 

1. Copy Jobs (already suggested). Ref.: [ 
https://www.bacula.org/9.4.x-manuals/en/main/Migration_Copy.html | 
https://www.bacula.org/9.4.x-manuals/en/main/Migration_Copy.html ] ; 
2. Two simultaneous similar Jobs, using different Pools and Storage (one for 
tape and other for cloud). I think it is better to have different Job names for 
this option. E.g. host-backup-tape, host-backup-cloud. Same data will be read 
from the client twice, but is still do-able. 
3. You can have one Job configuration that spawns others for different storages 
(Clone Jobs), that will run in a serial fashion. Same data will be read from 
the client twice, but is still do-able. 

"Run = job-name 
The Run directive (not to be confused with the Run option in a Schedule) allows 
you to start other jobs or to clone jobs. By using the cloning keywords (see 
below), you can backup the same data (or almost the same data) to two or more 
drives at the same time. The job-name is normally the same name as the current 
Job resource (thus creating a clone). However, it may be any Job name, so one 
job may start other related jobs. 

The part after the equal sign must be enclosed in double quotes, and can 
contain any string or set of options (overrides) that you can specify when 
entering the Run command from the console. For example storage=DDS-4  In 
addition, there are two special keywords that permit you to clone the current 
job. They are level=%l and since=%s. The %l in the level keyword permits 
entering the actual level of the current job and the %s in the since keyword 
permits putting the same time for comparison as used on the current job. Note, 
in the case of the since keyword, the %s must be enclosed in double quotes, and 
thus they must be preceded by a backslash since they are already inside quotes. 
For example: 

run = "Nightly-backup level=%l since=\"%s\" storage=DDS-4" 

A cloned job will not start additional clones, so it is not possible to 
recurse. 

Please note that all cloned jobs, as specified in the Run directives are 
submitted for running before the original job is run (while it is being 
initialized). This means that any clone job will actually start before the 
original job, and may even block the original job from starting until the 
original job finishes unless you allow multiple simultaneous jobs. Even if you 
set a lower priority on the clone job, if no other jobs are running, it will 
start before the original job. 

If you are trying to prioritize jobs by using the clone feature (Run 
directive), you will find it much easier to do using a RunScript resource, or a 
RunBeforeJob directive." - Ref.: [ 
https://www.bacula.org/9.4.x-manuals/en/main/Configuring_Director.html#SECTION00203
 | 
https://www.bacula.org/9.4.x-manuals/en/main/Configuring_Director.html#SECTION00203
 ] 

If NASA needs help I'm Florida and could drop by to assist you. =) 

Regards, 

> From: "bacula-users" 
> To: "Davide Franco, dev" 
> Cc: "bacula-users" 
> Sent: Thursday, January 30, 2020 10:12:51 AM
> Subject: Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

> Thank you all for your response to my question. So, it looks like there is no
> easy way one can set-up bacula to backup simultaneously to the tape and to the
> cloud.

> UR.

> From: Davide Franco 
> Sent: Thursday, January 30, 2020 12:41 AM
> To: Rao, Uthra R. (GSFC-672.0)[ADNET SYSTEMS INC] 
> Cc: bacula-users@lists.sourceforge.net
> Subject: [EXTERNAL] Re: [Bacula-users] bacula cloud backup

> Hello,

> IMHO, I’m not sure you can perform simultaneous backup jobs to both tape and 
> S3
> (please let me know if I’m wrong).

> I’d suggest maybe to use copy jobs like this

> 1. Backup to tapes

> 2. Copy job to S3 bucket based on previous backup on tapes.

> I don’t have huge experience with copy jobs, hopefully someone reading this 
> can
> give more hints.

> Best,

> Davide

> On Tue, 28 Jan 2020 at 20:18, Rao, Uthra R. (GSFC-672.0)[ADNET SYSTEMS INC] 
> via
> Bacula-users < [ mailto:bacula-users@lists.sourceforge.net |
> bacula-users@lists.sourceforge.net ] > wrote:

>> I have recently upgraded my bacula server to version 9.4.4 and wanted to test
>> the cloud backup feature. Currently we are backing up to LTO7 tapes. Is it
>> possible to set-up bacula to backup both to the tape and to the cloud
>> simultaneously?

>> If so I would appreciate some help with that.

>> Thank you.

>> UR.

>> ___
>> Bacula-users mailing list
>> [ mailto:Bacula-users@lists.sourceforge.net | 
>> Bacula-users@lists.sourceforge.net
>> ]
>> [
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourcef

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[dmaziuk via Bacula-users]

On 1/30/2020 9:20 AM, Phil Stracchino wrote:


There is not a way to stream the same backup job to two different
storage devices simultaneously, no.  That question has come up before.
But as Davide suggests, you could copy the job(s) afterwards.  I'm doing
something along these lines; my backups go to disk, but Full backups
then get post-copied to removable disk sets.


And when the disk decides to go titsup in the middle of a job, you lose 
everything since the last full.


I have a post-script that rsyncs the current "magazine" to a spare disk, 
this way I only lose the last backup.


Amanda had "RAIT" since forever.

Dima


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Phil Stracchino]

On 2020-01-30 11:29, dmaziuk via Bacula-users wrote:
> On 1/30/2020 9:20 AM, Phil Stracchino wrote:
> 
>> There is not a way to stream the same backup job to two different
>> storage devices simultaneously, no.  That question has come up before.
>> But as Davide suggests, you could copy the job(s) afterwards.  I'm doing
>> something along these lines; my backups go to disk, but Full backups
>> then get post-copied to removable disk sets.
> 
> And when the disk decides to go titsup in the middle of a job, you lose 
> everything since the last full.

Yes, I consider that an acceptable risk in my environment because if
something drastic enough happens that I simultaneously lose the highly
redundant NAS storage AND all of the clients, it probably means that my
entire infrastructure and possibly the house itself is gone.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Radosław Korzeniewski]

Hello,

czw., 30 sty 2020 o 17:30 dmaziuk via Bacula-users <
bacula-users@lists.sourceforge.net> napisał(a):

> Amanda had "RAIT" since forever.
>

If you would be so kind and have an idea how it could be implemented in
bacula and so kind to describe it on this list it would be implemented by
developers.

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[dmaziuk via Bacula-users]

On 1/31/2020 3:33 AM, Radosław Korzeniewski wrote:



Amanda had "RAIT" since forever.



If you would be so kind and have an idea how it could be implemented in
bacula and so kind to describe it on this list it would be implemented by
developers.


I don't have any idea how it could be implemented *in bacula* since I 
never looked at any bacula code.


In general sending the same stream of bytes to two different places 
instead of one is not exactly rocket science. I guess it and can be 
implemented in several places, e.g. special storage inside SD, or two 
different SDs.


I would probably send to two SDs at once because it allows for more 
flexible setups: the 2nd SD can write to "the cloud", it can be on the 
off-site box, etc. Whereas amanda-style RAIT device only protects 
against single disk/tape failure.


Dima



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Josh Fisher]

On 1/31/2020 10:25 AM, dmaziuk via Bacula-users wrote:

On 1/31/2020 3:33 AM, Radosław Korzeniewski wrote:



Amanda had "RAIT" since forever.



If you would be so kind and have an idea how it could be implemented in
bacula and so kind to describe it on this list it would be 
implemented by

developers.


I don't have any idea how it could be implemented *in bacula* since I 
never looked at any bacula code.


In general sending the same stream of bytes to two different places 
instead of one is not exactly rocket science. I guess it and can be 
implemented in several places, e.g. special storage inside SD, or two 
different SDs.



Actually, it is pretty tricky from user space and why the need for the 
md kernel module in Linux. For true RAIT, like for RAID, it should be a 
device driver. Most do not need RAIT, but I know some (Oak Ridge 
National Lab, for one) use RAIT systems both for redundancy and 
striping. Similarly, DRBD does this for block devices located on 
separate machines, a network RAID-1, and again requires kernel-space 
drivers. So implementing this in user space is still just making a copy, 
because there would be no way to make a write atomic. And in that case, 
what is wrong with using Copy jobs?





I would probably send to two SDs at once because it allows for more 
flexible setups: the 2nd SD can write to "the cloud", it can be on the 
off-site box, etc. Whereas amanda-style RAIT device only protects 
against single disk/tape failure.


Dima



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Dimitri Maziuk via Bacula-users]

On 1/31/20 10:36 AM, Josh Fisher wrote:
> 
> Actually, it is pretty tricky from user space and why the need for the
> md kernel module in Linux. For true RAIT, like for RAID, it should be a
> device driver.

Yes, iff you need true RAIT. If all you want is to write to 2 file
descriptors instead of one, I'll need to dig up my Stevens and do some
light reading to refresh my memory and see if a simple dup() will
suffice. And no, that's not the same as a copy job: one is parallel, the
other: serial.

I expect the fun bit is fitting it into bacula's universe of tapes and
volumes and catalogs. Like if you want to restore, how do you chose the
copy to restore from.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Kern Sibbald]

Hello,

If I am not mistaken, the RAIT code in Amanda was taken from code 
written long ago by CERN.   That code was mainly for speed rather than 
redundancy.


Writing to multiple simultaneous devices has always been on the projects 
list for Bacula, however, until recently it has never been a priority 
since there are alternatives such as Copy jobs. Admittedly Copying is 
not as good as writing two simultaneous streams for certain applications.


As the multiple simultaneous writing code is now more important, I hope 
to see it implemented in the not too distant future.


Best regards,
Kern

On 1/31/20 4:25 PM, dmaziuk via Bacula-users wrote:

On 1/31/2020 3:33 AM, Radosław Korzeniewski wrote:



Amanda had "RAIT" since forever.



If you would be so kind and have an idea how it could be implemented in
bacula and so kind to describe it on this list it would be 
implemented by

developers.


I don't have any idea how it could be implemented *in bacula* since I 
never looked at any bacula code.


In general sending the same stream of bytes to two different places 
instead of one is not exactly rocket science. I guess it and can be 
implemented in several places, e.g. special storage inside SD, or two 
different SDs.


I would probably send to two SDs at once because it allows for more 
flexible setups: the 2nd SD can write to "the cloud", it can be on the 
off-site box, etc. Whereas amanda-style RAIT device only protects 
against single disk/tape failure.


Dima



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Radosław Korzeniewski]

Hello,

pt., 31 sty 2020 o 16:27 dmaziuk via Bacula-users <
bacula-users@lists.sourceforge.net> napisał(a):

> On 1/31/2020 3:33 AM, Radosław Korzeniewski wrote:
> >
> >> Amanda had "RAIT" since forever.
> >>
> >
> > If you would be so kind and have an idea how it could be implemented in
> > bacula and so kind to describe it on this list it would be implemented by
> > developers.
>
> I don't have any idea how it could be implemented *in bacula* since I
> never looked at any bacula code.
>

I do not want you to know anything about Bacula source code when I asked
about functionality description. It is a pure architecture description
which developers could implement.


> In general sending the same stream of bytes to two different places
> instead of one is not exactly rocket science.


Well, I do not want to dive deeper into discussion how rocket science or
not it is...


> I guess it and can be
> implemented in several places, e.g. special storage inside SD, or two
> different SDs.
>

This is the exact discussion I could make.
What is better - two distinct SD which do not know anything about each
other and FD just sends two concurrent data streams to both (or any
multiple number of SD when we implement chunking or erasure code) or
special kind of SD which handles redundancy internally syncing data to
other SD synchronously.


>
> I would probably send to two SDs at once because it allows for more
> flexible setups:


And by default lower the backup speed twice.


> the 2nd SD can write to "the cloud", it can be on the
> off-site box, etc. Whereas amanda-style RAIT device only protects
> against single disk/tape failure.
>

So, why not to design a special kind of SD which will get data from FD and
then save it on different backend SD synchronously defining full and
flexible RAIT or EC solution?

This is a kind of help I want to get from community as you always complain
about decisions which developers take himself.

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Heitor Faria]

Hello Radoslaw,

Perhaps you should consider a total FD revamp that allows:

- Parallel Client disk reading streams, for faster backups even for a single SD 
device writing;
- FD multi-SD writing capability, with redundant streams or even just fail-over 
between SDs.

Regards,
--
MSc Heitor Faria
CEO Bacula LatAm
mobile1: + 1 909 655-8971
mobile2: + 55 61 98268-4220

América Latina
[ http://bacula.lat/]

 Original Message 
From: Radosław Korzeniewski 
Sent: Monday, February 10, 2020 07:19 AM
To: dmaziuk 
Subject: Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup
CC: bacula-users 

>Hello,
>
>pt., 31 sty 2020 o 16:27 dmaziuk via Bacula-users <
>bacula-users@lists.sourceforge.net> napisał(a):
>
>> On 1/31/2020 3:33 AM, Radosław Korzeniewski wrote:
>> >
>> >> Amanda had "RAIT" since forever.
>> >>
>> >
>> > If you would be so kind and have an idea how it could be implemented in
>> > bacula and so kind to describe it on this list it would be implemented by
>> > developers.
>>
>> I don't have any idea how it could be implemented *in bacula* since I
>> never looked at any bacula code.
>>
>
>I do not want you to know anything about Bacula source code when I asked
>about functionality description. It is a pure architecture description
>which developers could implement.
>
>
>> In general sending the same stream of bytes to two different places
>> instead of one is not exactly rocket science.
>
>
>Well, I do not want to dive deeper into discussion how rocket science or
>not it is...
>
>
>> I guess it and can be
>> implemented in several places, e.g. special storage inside SD, or two
>> different SDs.
>>
>
>This is the exact discussion I could make.
>What is better - two distinct SD which do not know anything about each
>other and FD just sends two concurrent data streams to both (or any
>multiple number of SD when we implement chunking or erasure code) or
>special kind of SD which handles redundancy internally syncing data to
>other SD synchronously.
>
>
>>
>> I would probably send to two SDs at once because it allows for more
>> flexible setups:
>
>
>And by default lower the backup speed twice.
>
>
>> the 2nd SD can write to "the cloud", it can be on the
>> off-site box, etc. Whereas amanda-style RAIT device only protects
>> against single disk/tape failure.
>>
>
>So, why not to design a special kind of SD which will get data from FD and
>then save it on different backend SD synchronously defining full and
>flexible RAIT or EC solution?
>
>This is a kind of help I want to get from community as you always complain
>about decisions which developers take himself.
>
>best regards
>-- 
>Radosław Korzeniewski
>rados...@korzeniewski.net
>
>
>___
>Bacula-users mailing list
>Bacula-users@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/bacula-users
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Josh Fisher]

On 2/10/2020 5:12 AM, Radosław Korzeniewski wrote:

Hello,

pt., 31 sty 2020 o 16:27 dmaziuk via Bacula-users 
> napisał(a):

...

the 2nd SD can write to "the cloud", it can be on the
off-site box, etc. Whereas amanda-style RAIT device only protects
against single disk/tape failure.


So, why not to design a special kind of SD which will get data from FD 
and then save it on different backend SD synchronously defining full 
and flexible RAIT or EC solution?


This is a kind of help I want to get from community as you always 
complain about decisions which developers take himself.



I, for one, would prefer it be implemented in the SD and not in the 
client. This allows a single interface for the client, while allowing 
the SD to communicate with SDs on other networks that are perhaps not 
accessible to the client. Also, this allows adding multiple device 
capability without having to update all clients.


What about a SD writing to multiple local 'Device' objects? And segueing 
into a related subject, why is a job locked into a particular Device at 
job startup? Rather than a one-time allocation of device and then one or 
more subsequent allocations of volume(s) as separate operations, why not 
allocate device and volume together as a pair in a single atomic 
operation? This would allow a job to potentially change devices when a 
different volume was needed and would be beneficial for autochangers 
that have more than one drive, including virtual disk autochangers.


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Dimitri Maziuk via Bacula-users]

On 2/10/20 11:31 AM, Josh Fisher wrote:

> What about a SD writing to multiple local 'Device' objects? And segueing
> into a related subject, why is a job locked into a particular Device at
> job startup? Rather than a one-time allocation of device and then one or
> more subsequent allocations of volume(s) as separate operations, why not
> allocate device and volume together as a pair in a single atomic
> operation? This would allow a job to potentially change devices when a
> different volume was needed and would be beneficial for autochangers
> that have more than one drive, including virtual disk autochangers.

Why is media type tied to a device? I never understood this: I get that
you may not want to stick an lto-1 tape into an lto-4 drive, but those
are device *types* -- nothing's stopping me from having two lto-1 drives
in my jukebox and have two devices capable of using lto-1 media.

(Or is it no longer the case)
-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Dimitri Maziuk via Bacula-users]

On 2/10/20 4:12 AM, Radosław Korzeniewski wrote:

> So, why not to design a special kind of SD which will get data from FD and
> then save it on different backend SD synchronously defining full and
> flexible RAIT or EC solution?

How should I know why you need a "specail" SD instead of a "regular" one?

> This is a kind of help I want to get from community as you always complain
> about decisions which developers take himself.

I personally only care about failure of my HDD "magazine" being written
to. SD writing to two file descriptors instead of one is all *I* want.

I don't even care for having the 2nd copy in the catalog: I just want
the same exact volume file on two physical drives. Once full, I'll put
one of them on the shelf and wipe out the other, to be reused w/ the
next magazine.

Somebody else may want the copy to go to "the cloud" instead, that may
or may not require a different approach.
-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Radosław Korzeniewski]

Hello,

pon., 10 lut 2020 o 23:01 Dimitri Maziuk via Bacula-users <
bacula-users@lists.sourceforge.net> napisał(a):

> On 2/10/20 4:12 AM, Radosław Korzeniewski wrote:
>
> > So, why not to design a special kind of SD which will get data from FD
> and
> > then save it on different backend SD synchronously defining full and
> > flexible RAIT or EC solution?
>
> How should I know why you need a "specail" SD instead of a "regular" one?
>

Because a community _always_ know better then developers. :) Just read some
hot threads on this group.


>
> > This is a kind of help I want to get from community as you always
> complain
> > about decisions which developers take himself.
>
> I personally only care about failure of my HDD "magazine" being written
> to. SD writing to two file descriptors instead of one is all *I* want.
>
>
This is a small tip of the ideberg! and the devil is in the details -
always! I.e. how you want a single SD disconnection and reconnection to be
handled? Do you want to fail the whole job? or continue on remaining one?
what about a disconnected SD and its partially written volume on
reconnection? do you want SD reconnection in this case? do you want a
single or multiple jobs (in case if simple job mirroring) stored in catalog
db? do you want to fall back to other volume/job during restore errors?
etc. etc. etc. all of these requirements you do not want to discuss are the
important part of the functionality. I can implement a functionality which
community reject. This is not what I want.


> I don't even care for having the 2nd copy in the catalog: I just want
> the same exact volume file on two physical drives.


Then use low-level block replication (storage array or storage cluster -
sds or even os-level like drbd). It is fast and proven to be extremely
reliable as Enterprise class solutions. But we are discussing about
application-level replication or RAIT like solution. This is a very
different story.


> Somebody else may want the copy to go to "the cloud" instead, that may
> or may not require a different approach.
>

Sure, so I want to discuss that will be the better or the best approach in
this case.

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Radosław Korzeniewski]

Hello,

pon., 10 lut 2020 o 11:37 Heitor Faria  napisał(a):

> Hello Radoslaw,
>
> Perhaps you should consider a total FD revamp that allows:
>
> - Parallel Client disk reading streams, for faster backups even for a
> single SD device writing;
>
This a functionality which is led by Kern, Eric and others so I do not want
to duplicate their efforts.

> - FD multi-SD writing capability, with redundant streams or even just
> fail-over between SDs.
>
This is the exact functionality we are discussing here. I know all these
fancy words, but the devil is in the details.
And I want to discuss these details - architecture details, not the code
itself. I.e. what should be the procedure to select required volumes in
Bacula in this multi SD case? How we should handle single SD disconnection
and reconnection? etc. etc. there are literally tons of high level
questions which community should answer.

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Dimitri Maziuk via Bacula-users]

On 2/11/20 10:59 AM, Radosław Korzeniewski wrote:

> Because a community _always_ know better then developers. :) Just read some
> hot threads on this group.

BS. This is IT 101: users don't know what they want.

>>
> This is a small tip of the ideberg! and the devil is in the details -
> always! I.e. how you want a single SD disconnection and reconnection to be
> handled? Do you want to fail the whole job? or continue on remaining one?
...

My point exactly.

Now that I figured out how to keep zfs snaphots forever, mounted and
browsable, on the backup box and garbage-collect them on the clients
too, I am very rapidly losing interest in the "tape backup" paradigm
anyway. So, like, whatever...

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Josh Fisher]

On 2/11/2020 1:17 PM, Dimitri Maziuk via Bacula-users wrote:

On 2/11/20 10:59 AM, Radosław Korzeniewski wrote:
...

This is a small tip of the ideberg! and the devil is in the details -
always! I.e. how you want a single SD disconnection and reconnection to be
handled? Do you want to fail the whole job? or continue on remaining one?

...

My point exactly.

Now that I figured out how to keep zfs snaphots forever, mounted and
browsable, on the backup box and garbage-collect them on the clients
too, I am very rapidly losing interest in the "tape backup" paradigm
anyway. So, like, whatever...



One problem I have with that is, from the zxfer man page,

"Note also that transferring between different operating systems and 
even different versions of the same operating system (e.g. FreeBSD 8.1 
and 8.2) has not been tested and will probably fail or give unexpected 
results."


That could make upgrading OS on the involved boxes a potential problem 
and seems to imply all client boxes/VMs must use the same OS version at 
all times.


Also, how do you backup the backup box itself?

___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[dmaziuk via Bacula-users]

On 2/12/2020 9:54 AM, Josh Fisher wrote:

That could make upgrading OS on the involved boxes a potential problem 
and seems to imply all client boxes/VMs must use the same OS version at 
all times.


Yeah, but rolling upgrades are a PITA anyway so what else's new. Also, 
linux-to-linux you're pretty much on the same zfs version on all boxen, 
and so far I have not seen any problems between e.g. centos 7/zfs 0.8.1 
and centos 6/zfs 0.8.3.



Also, how do you backup the backup box itself?


I don't. ;)

Yes, it you want all your change history in a tape archive in a vault 
somewhere far away, that would be a problem because snapshots are not 
visible to your tape backup software. But then you may want to consider 
putting your stuff on GitHub: it's better at tracking version history 
than incremental tapes, easier to restore, and might actually cost less 
than the tapes and the vault and the monkey to cart the tapes there and 
back.


Dima




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[William Muriithi]

Hi Dima,

>> Also, how do you backup the backup box itself?
> 
> I don't. ;)
> 
> Yes, it you want all your change history in a tape archive in a vault 
> somewhere far away, that would be a problem because snapshots are not visible 
> to your tape backup software. But then you may want to consider putting your 
> stuff on GitHub: it's better at tracking version history than incremental 
> tapes, easier to restore, and might actually cost less than the tapes and the 
> vault and the monkey to cart the tapes there and back.

I really wish you aren't that dismissive.  He had raised a valid gap in your 
risk analysis, and honestly understandable.  The way you respond may make 
someone reading this thread down the line to miss the risk involved, and only 
learn the hard way.  Backup is an exercise of balancing the risk with expenses.

A better way would have been to say, yes, I understand I am exposed if someone 
was to get in my network and delete both the data  and the backup, but I accept 
and plans to live with that risk..

Regards,
William




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[dmaziuk via Bacula-users]

On 2/12/2020 11:05 AM, William Muriithi wrote:


A better way would have been to say, yes, I understand I am exposed if someone 
was to get in my network and delete both the data  and the backup, but I accept 
and plans to live with that risk..


If somebody gets in my network and "securely wipes" all servers 
including the co-lo'ed backup box, I plan to have other problems.


Dima


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] [EXTERNAL] Re: bacula cloud backup

[Radosław Korzeniewski]

Hello,

wt., 11 lut 2020 o 19:19 Dimitri Maziuk via Bacula-users <
bacula-users@lists.sourceforge.net> napisał(a):

> On 2/11/20 10:59 AM, Radosław Korzeniewski wrote:
>
> > Because a community _always_ know better then developers. :) Just read
> some
> > hot threads on this group.
>
> BS. This is IT 101: users don't know what they want.
>

Exclude Bacula Community users. :)

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users