Re: [Bacula-users] Bacula driver misconceptions

[Radosław Korzeniewski]

Hello,

2017-08-23 17:06 GMT+02:00 Dimitri Maziuk :

> On 2017-08-22 21:26, Heitor Faria wrote:
>
> It usually cost a new tape library, a new set of tapes and coffee.
>> It is not that hard to migrate legacy backups if you use a backup
>> rotation strategy such as GFS and works in a common company with 1-5
>> maximum montly/yearly retention backups.
>>
>
> I repeat, how many times have *you* done it, how many tapes, and how much
> did it cost you? In hardware, your time, and energy bills?
>
>
I did it once. Does it counts?

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Radosław Korzeniewski]

Hello,

2017-08-22 17:01 GMT+02:00 Dimitri Maziuk :

>
> However if you want to backup to removable disks and put them on the shelf
> when full, that's where things get less than perfect fast.
>
>
And make it behave like a tape when you complain it is not a tape. :)

P.S. Please excuse my joke above. I can't stop to write it.

best regards
-- 
Radosław Korzeniewski
rados...@korzeniewski.net
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Dan Langille]

> On Aug 23, 2017, at 11:27 AM, Phil Stracchino  wrote:
> 
> On 08/23/17 11:06, Dimitri Maziuk wrote:
>> On 2017-08-22 21:26, Heitor Faria wrote:
>> 
>>> It usually cost a new tape library, a new set of tapes and coffee.
>>> It is not that hard to migrate legacy backups if you use a backup rotation 
>>> strategy such as GFS and works in a common company with 1-5 maximum 
>>> montly/yearly retention backups.
>> 
>> I repeat, how many times have *you* done it, how many tapes, and how 
>> much did it cost you? In hardware, your time, and energy bills?
> 
> 
> I don't have the luxury of a controlled environment for my backup
> system.  One of the things I've learned as a result is that I greatly
> extend the life of my tape drive by *only powering it on* when I'm
> preparing to run monthly full backups.  (This also, practically
> speaking, rules out internal drives, period.)  If you want to get a long
> service life out of a modern tape drive you practically need to have a
> clean-room for it.  Requiring a controlled environment to get a useful
> service life is another hidden cost factor in LTO tape, and the higher
> the track densities go, the more sensitive the drives are going to be to
> environmental factors.

FWIW, I do the same, mostly for noise and power considerations. The
tape libraries sit about three feet from my desk.

Mine is for home use. Not a common situation.

> I'm REALLY hoping that by the time I need to upgrade from my LTO4 drive,
> there is a better and more robust alternative available.


I'm hoping my LTO4 lasts at least 3 more years.

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Dimitri Maziuk]

On 2017-08-25 06:52, Josh Fisher wrote:


 1 "copy" disk, and 10 "magazines" 
-- or 5 raid=1 "magazines". It's certainly doable but I'll have to 
swap the disks twice as often


I don't quite understand. It will take twice the disk space whether 
using RAID1 or rsync.


There is only one copy disk. I assume that once a "magazine" disk if 
full it's not going to fail while it's sitting there idle. (That was 
proven wrong, by the way, once or twice over the years.)


The copy disk keeps a copy of the "currently active" magazine, it's only 
using "+1" space. If you only have one magazine drive that's twice, I 
have 10.


Dima

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Josh Fisher]

On 8/24/2017 10:49 AM, Dimitri Maziuk wrote:

On 2017-08-24 09:24, Josh Fisher wrote:

As for writing the same job to two disks simultaneously, I don't 
think that will be very easy to accomplish. It would, I believe, 
require a new Device type driver in Bacula that atomically reserves 
two Device resources at a time. Why not use Copy jobs? Or pair the 
removable drives into RAID1 arrays?


Amanda had RAIT option since forever: you just tell it to use 2 tape 
devices.


It is very possible. I just think the devs feel it is very low priority 
due to md raid.




Copy jobs aren't quite for that, and  with, say, 2U 12-3.5" bay 
chassis I'm looking at a spool ssd, 1 "copy" disk, and 10 "magazines" 
-- or 5 raid=1 "magazines". It's certainly doable but I'll have to 
swap the disks twice as often, that is less than perfect. I might 
actually do it in the next iteration, but the truth is our hdds don't 
fail that often so at this point it ain't really broken, just suboptimal.


I don't quite understand. It will take twice the disk space whether 
using RAID1 or rsync. I can see why a Copy job may not be wanted, since 
it would additionally increase the catalog size and use a different volume.




Someday I'll read up on fuse, too, I wonder how much trickery one can 
program into that.


Dima

-- 


Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Dimitri Maziuk]

On 2017-08-24 09:24, Josh Fisher wrote:

As for writing the same job to two disks simultaneously, I don't think 
that will be very easy to accomplish. It would, I believe, require a new 
Device type driver in Bacula that atomically reserves two Device 
resources at a time. Why not use Copy jobs? Or pair the removable drives 
into RAID1 arrays?


Amanda had RAIT option since forever: you just tell it to use 2 tape 
devices.


Copy jobs aren't quite for that, and  with, say, 2U 12-3.5" bay chassis 
I'm looking at a spool ssd, 1 "copy" disk, and 10 "magazines" -- or 5 
raid=1 "magazines". It's certainly doable but I'll have to swap the 
disks twice as often, that is less than perfect. I might actually do it 
in the next iteration, but the truth is our hdds don't fail that often 
so at this point it ain't really broken, just suboptimal.


Someday I'll read up on fuse, too, I wonder how much trickery one can 
program into that.


Dima

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Josh Fisher]

On 8/22/2017 7:13 PM, Dimitri Maziuk wrote:

On 08/22/2017 05:07 PM, Dan Langille wrote:

On Aug 22, 2017, at 11:01 AM, Dimitri Maziuk  wrote:

However if you want to backup to removable disks and put them on the shelf when 
full, that's where things get less than perfect fast.

Is this a common use case?  I'm not challenging your approach. I am seeking 
knowledge about how Bacula gets used.

I don't know but there is RDX so there's gotta be some demand.

On the flip side, we went from AIT to HDD and never even touched LTO. I
mean, they only guarantee backwards compatibility to 2 generations and
have version creep worse than iPhones. Who in their right mind would use
LTO for their long-term storage, and we have plenty of hard drives
laying around.


This is why I wrote vchanger in the first place. I do not know how many 
use it. It averages around 70 downloads per month.


I tend to use portable 2.5 inch USB 3 drives, as opposed to SAS/SATA, 
for a number of reasons. Portable drives are designed to be moved around 
and are more durable and shock resistant/ The USB connector is designed 
for repetitive use, whereas hot-swap bays are not. I can see why the 
SAS/SATA might be chosen for speed, although the USB 3 drives are fast 
enough (around 100 MB/s) for the small businesses that I am involved 
with that have < 50 jobs per day.


I looked into RDX, but do not really see the advantage over USB portable 
drives. Perhaps I am missing something, like extended shelf life, etc., 
but they are quite expensive.


As for writing the same job to two disks simultaneously, I don't think 
that will be very easy to accomplish. It would, I believe, require a new 
Device type driver in Bacula that atomically reserves two Device 
resources at a time. Why not use Copy jobs? Or pair the removable drives 
into RAID1 arrays?






--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[muhasin]

Hi,

BTW, Is there any way to stop rescheduling a incomplete job . 

OUTPUT


  SD termination status:  Incomplete job
  Termination:Backup failed -- incomplete

Time: 2017-08-24 19:22:57
 LogText: bacula9.server.com-dir JobId 781: Rescheduled Job 
compute.2017-08-24_19.22.24_08 at 24-Aug-2017 19:22 to re-run in 1800 seconds 
(24-Aug-2017 19:52).

Time: 2017-08-24 19:22:57
 LogText: bacula9.server.com-dir JobId 781: Job compute.2017-08-24_19.22.24_08 
waiting 1799 seconds for scheduled start time.

Time: 2017-08-24 19:52:58
 LogText: bacula9.server.com-dir JobId 781: Restart Incomplete Backup JobId 
781, Job=compute.2017-08-24_19.22.24_08

- Original Message -
From: "Dimitri Maziuk" 
To: "bacula-users" 
Sent: Wednesday, August 23, 2017 9:15:09 PM
Subject: Re: [Bacula-users] Bacula driver misconceptions

On 2017-08-23 10:27, Phil Stracchino wrote:

> If you want to get a long
> service life out of a modern tape drive you practically need to have a
> clean-room for it.

We have a Sun Blade 150 complete with 2 AIT jukeboxes (1 and 3 IIRC), 
working install of Legato Networker, and a couple of sets backup system 
& catalog drives dd'ed and tested working. And a few hundred AIT tapes.

We also have couple of DLT drives sitting on the shelf, and I'm pretty 
sure their plastic parts will turn into dust when turned on, and so will 
the rubber ones. Unless they ooze out as black goo. So I don't even care 
what sort of unobtainable scsi connector they have. There's a ton of DLT 
tapes sitting around and then some assorted others, like a loose DDS or 
two, that I don't even know why we have it.

Anyone tells you you "just" keep and old drive around and "just" migrate 
your archive to newer media is IMO full of it.

Dima

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
-- 
Muhasin C.M 
Systems Engineer R & D 
Assistanz Networks Pvt LTD

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Dimitri Maziuk]

On 2017-08-23 10:27, Phil Stracchino wrote:


If you want to get a long
service life out of a modern tape drive you practically need to have a
clean-room for it.


We have a Sun Blade 150 complete with 2 AIT jukeboxes (1 and 3 IIRC), 
working install of Legato Networker, and a couple of sets backup system 
& catalog drives dd'ed and tested working. And a few hundred AIT tapes.


We also have couple of DLT drives sitting on the shelf, and I'm pretty 
sure their plastic parts will turn into dust when turned on, and so will 
the rubber ones. Unless they ooze out as black goo. So I don't even care 
what sort of unobtainable scsi connector they have. There's a ton of DLT 
tapes sitting around and then some assorted others, like a loose DDS or 
two, that I don't even know why we have it.


Anyone tells you you "just" keep and old drive around and "just" migrate 
your archive to newer media is IMO full of it.


Dima

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Phil Stracchino]

On 08/23/17 11:06, Dimitri Maziuk wrote:
> On 2017-08-22 21:26, Heitor Faria wrote:
> 
>> It usually cost a new tape library, a new set of tapes and coffee.
>> It is not that hard to migrate legacy backups if you use a backup rotation 
>> strategy such as GFS and works in a common company with 1-5 maximum 
>> montly/yearly retention backups.
> 
> I repeat, how many times have *you* done it, how many tapes, and how 
> much did it cost you? In hardware, your time, and energy bills?


I don't have the luxury of a controlled environment for my backup
system.  One of the things I've learned as a result is that I greatly
extend the life of my tape drive by *only powering it on* when I'm
preparing to run monthly full backups.  (This also, practically
speaking, rules out internal drives, period.)  If you want to get a long
service life out of a modern tape drive you practically need to have a
clean-room for it.  Requiring a controlled environment to get a useful
service life is another hidden cost factor in LTO tape, and the higher
the track densities go, the more sensitive the drives are going to be to
environmental factors.

I'm REALLY hoping that by the time I need to upgrade from my LTO4 drive,
there is a better and more robust alternative available.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Dimitri Maziuk]

On 2017-08-22 21:26, Heitor Faria wrote:


It usually cost a new tape library, a new set of tapes and coffee.
It is not that hard to migrate legacy backups if you use a backup rotation 
strategy such as GFS and works in a common company with 1-5 maximum 
montly/yearly retention backups.


I repeat, how many times have *you* done it, how many tapes, and how 
much did it cost you? In hardware, your time, and energy bills?


Dima

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[muhasin]

Hi,

Another approach, instead of using volume-per-job, would be to allow
multiple jobs per volume but with a volume use duration, and have
run-after-job add the volume to a list of volumes to be replicated, that
list acting as a queue for a separate tool which dequeues and replicates
each volume ONLY when it has been marked USED.


How to achieve this without performance issues.

- Original Message -
From: "Phil Stracchino" 
To: "bacula-users" 
Sent: Wednesday, August 23, 2017 12:12:43 AM
Subject: Re: [Bacula-users] Bacula driver misconceptions

On 08/22/17 14:27, Dimitri Maziuk wrote:
> On 08/22/2017 12:27 PM, Phil Stracchino wrote:
> 
>>> So I've a post-backup script that figures out which of the vchanger
>>> "magazines" is the "current" one and rsyncs it to anther drive. That way
>>> if a drive dies during write, I lose at most the latest backup.
>>
>> That sounds like a pretty good idea.  After-the-fact redundancy.
> 
> It isn't: what you want is to write to two disks at once. But if the
> software wasn't designed for that, that's problems all the way to the
> catalog.

Oh sure, simultaneous two-volume write would arguably be the best
solution.  (Though there are viewpoints from which it's not.  What if
the job fails at 90% complete?  You've wasted all that duplicate
writing.)  However, making a redundant copy of the volume in the
background *as soon as the job completes* is not a bad next-best, and
could arguably actually be more efficient if the copy is fired off in
the background as a run-after-job if AND ONLY if the job successfully
completes.

> The other problem is that rsyncing an umpteen-TB drive can take a
> non-trivial amount of time and ram.

I would argue that if you're doing a full checksum rsync of a
multi-terabyte drive, you're probably going about the task wrong.  If I
were to implement a scheme like this, I'd do volume-per-job, I'd use a
run-after-job that appended the volume name to a list of volumes to be
copied, and I wouldn't use rsync at all.  When you already know you want
to copy an entire volume, rsync is an inefficient way to do it.

Another approach, instead of using volume-per-job, would be to allow
multiple jobs per volume but with a volume use duration, and have
run-after-job add the volume to a list of volumes to be replicated, that
list acting as a queue for a separate tool which dequeues and replicates
each volume ONLY when it has been marked USED.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
-- 
Muhasin C.M 
Systems Engineer R & D 
Assistanz Networks Pvt LTD

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Heitor Faria]

Hello, Dimitri/Phil,

>>> They have the largest life expectancy (30 years) under ideal conditions, 
>>> they
>>> are very inexpensive and scalable.
>>> I don't see why not to use them.
>> 
>> How many times have you migrated your complete LTO archives and how much
>> did it cost you?

It usually cost a new tape library, a new set of tapes and coffee.
It is not that hard to migrate legacy backups if you use a backup rotation 
strategy such as GFS and works in a common company with 1-5 maximum 
montly/yearly retention backups. 

>> http://spectrum.ieee.org/computing/it/the-lost-picture-show-hollywood-archivists-cant-outpace-obsolescence

Dimitri: this is the worse case scenario and the author does not provide a 
better solution/says it does not exist. This argument is invalid.
If you are worried about compatibility amazon.com still sells brand new LTO1 
tape drives .
In fact you can still buy DLT compatible drives .

> This is why I'm using a 2 (soon to be 3) generations old tape drive, and
> why I can't afford an autochanger.

Phil: the price difference between a standalone and a tape library is not that 
large. 
LTO7 TL .
LTO7 Standalone Drive .
If I had a short on budget Small Business I would buy an used/refurbished one. 
Today there are plenty of people capable of fixing this kind of equipament, 
even in Brazil.

> --
>  Phil Stracchino
>  Babylon Communications
>  ph...@caerllewys.net
>  p...@co.ordinate.org
>  Landline: +1.603.293.8485
>  Mobile:   +1.603.998.6958
> 

Regards,
-- 
=== 
Heitor Medrado de Faria | CEO Bacula do Brasil | Visto EB-1 | LPIC-III | EMC 
05-001 | ITIL-F 
• Não seja tarifado pelo tamanho dos seus backups, conheça o Bacula Enterprise: 
http://www.bacula.com.br/enterprise/ 
• Ministro treinamento e implementação in-company do Bacula Community: 
http://www.bacula.com.br/in-company/ 
+55 61 98268-4220 | www.bacula.com.br 
 
Indicamos também as capacitações complementares: 
• Shell básico e Programação em Shell com Julio Neves. 
• Zabbix com Adail Host. 


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Phil Stracchino]

On 08/22/17 20:20, Dimitri Maziuk wrote:
> On 08/22/2017 06:52 PM, Heitor Faria wrote:
> 
>> They have the largest life expectancy (30 years) under ideal conditions, 
>> they are very inexpensive and scalable.
>> I don't see why not to use them.
> 
> How many times have you migrated your complete LTO archives and how much
> did it cost you?
> 
> http://spectrum.ieee.org/computing/it/the-lost-picture-show-hollywood-archivists-cant-outpace-obsolescence


This is why I'm using a 2 (soon to be 3) generations old tape drive, and
why I can't afford an autochanger.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958



signature.asc
Description: OpenPGP digital signature
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Dimitri Maziuk]

On 08/22/2017 06:52 PM, Heitor Faria wrote:

> They have the largest life expectancy (30 years) under ideal conditions, they 
> are very inexpensive and scalable.
> I don't see why not to use them.

How many times have you migrated your complete LTO archives and how much
did it cost you?

http://spectrum.ieee.org/computing/it/the-lost-picture-show-hollywood-archivists-cant-outpace-obsolescence

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Heitor Faria]

Hello, People,

>>> However if you want to backup to removable disks and put them on the shelf 
>>> when
>>> full, that's where things get less than perfect fast.
>> 
>> Is this a common use case?  I'm not challenging your approach. I am seeking
>> knowledge about how Bacula gets used.
> 
> I don't know but there is RDX so there's gotta be some demand.

- RDX is very "niche" according to Google Trends 
.
- Even a manufacturer put it on the SMB niche position 
.
- I only saw two in my life (Brazil). Configured as VTL, since I read 
(somewhere) that sustained continuous r/w was faster than having a filesystem 
(this is in an old thread here). I think this is why most backup software 
writes in a sequential fashion (just as tapes).

> On the flip side, we went from AIT to HDD and never even touched LTO. I
> mean, they only guarantee backwards compatibility to 2 generations and
> have version creep worse than iPhones.

Every new LTO generation doubles capacity. I think is a nice pace and allows 
the user to plan migrations.
You always have Bacula copy jobs to upgrade tapes.

> Who in their right mind would use
> LTO for their long-term storage, and we have plenty of hard drives
> laying around.

They have the largest life expectancy (30 years) under ideal conditions, they 
are very inexpensive and scalable.
I don't see why not to use them.

> --
> Dimitri Maziuk
> Programmer/sysadmin
> BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu

Regards,
-- 
=== 
Heitor Medrado de Faria | EB-1 Visa | LPIC-III | ITIL-F | EMC 05-001| Bacula 
Systems Certified Administrator II 
• Bacula Enterprise does not charge per backup size and has plugins for many 
applications. In the USA: http://bacula.us/ 
• Do you need Bacula training? http://bacula.us/video-classes/ 
+55 61 98268-4220 | http://bacula.us 
===

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Dimitri Maziuk]

On 08/22/2017 05:07 PM, Dan Langille wrote:
>> On Aug 22, 2017, at 11:01 AM, Dimitri Maziuk  wrote:
>>
>> However if you want to backup to removable disks and put them on the shelf 
>> when full, that's where things get less than perfect fast.
> 
> Is this a common use case?  I'm not challenging your approach. I am seeking 
> knowledge about how Bacula gets used.

I don't know but there is RDX so there's gotta be some demand.

On the flip side, we went from AIT to HDD and never even touched LTO. I
mean, they only guarantee backwards compatibility to 2 generations and
have version creep worse than iPhones. Who in their right mind would use
LTO for their long-term storage, and we have plenty of hard drives
laying around.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Dan Langille]

> On Aug 22, 2017, at 11:01 AM, Dimitri Maziuk  wrote:
> 
> On 2017-08-22 03:10, Kern Sibbald wrote:
>> Dimitri,
>> I have heard the same complaint from you a number of times: "Bacula sucks at 
>> disk as tapes".  It is my view that your statement is simply not true.
> 
> I think at the very basic level, a disk is a device that doesn't quack like a 
> tape and doesn't walk like a tape. What this means is if your software is 
> built on the tape paradigm, it can call a disk "a tape", and it will suck 
> because it actually is not a tape. It does not matter to me what API the code 
> calls to do addressing: the point is that a disk has multiple files on it 
> while a tape has bytes until EOT.
> 
> Don't get me wrong, this is not specific to bacula: every tape backup 
> software has the same conceptual limitation and has to employ some 
> workarounds for backups to disk. But this is the bacula list, we're not 
> talking about every software here.
> 
> Bacula, in fact, is much better than some, if you can fit all your backups on 
> a single filesystem: bacula's built-in disk backup setup is a breeze.
> 
> However if you want to backup to removable disks and put them on the shelf 
> when full, that's where things get less than perfect fast.

Is this a common use case?  I'm not challenging your approach. I am seeking 
knowledge about how Bacula gets used.

-- 
Dan Langille - BSDCan / PGCon
d...@langille.org




--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Dimitri Maziuk]

On 08/22/2017 01:42 PM, Phil Stracchino wrote:

> I would argue that if you're doing a full checksum rsync of a
> multi-terabyte drive, you're probably going about the task wrong.  If I
> were to implement a scheme like this, I'd do volume-per-job, I'd use a
> run-after-job that appended the volume name to a list of volumes to be
> copied, and I wouldn't use rsync at all.  When you already know you want
> to copy an entire volume, rsync is an inefficient way to do it.

Yes, maybe I need to revisit that script: rsync seemed simple and easy
at the time, but that was before I ran into bacula's tape selection
algorithm. As soon as I get a fresh supply of round tuits...

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Phil Stracchino]

On 08/22/17 14:27, Dimitri Maziuk wrote:
> On 08/22/2017 12:27 PM, Phil Stracchino wrote:
> 
>>> So I've a post-backup script that figures out which of the vchanger
>>> "magazines" is the "current" one and rsyncs it to anther drive. That way
>>> if a drive dies during write, I lose at most the latest backup.
>>
>> That sounds like a pretty good idea.  After-the-fact redundancy.
> 
> It isn't: what you want is to write to two disks at once. But if the
> software wasn't designed for that, that's problems all the way to the
> catalog.

Oh sure, simultaneous two-volume write would arguably be the best
solution.  (Though there are viewpoints from which it's not.  What if
the job fails at 90% complete?  You've wasted all that duplicate
writing.)  However, making a redundant copy of the volume in the
background *as soon as the job completes* is not a bad next-best, and
could arguably actually be more efficient if the copy is fired off in
the background as a run-after-job if AND ONLY if the job successfully
completes.

> The other problem is that rsyncing an umpteen-TB drive can take a
> non-trivial amount of time and ram.

I would argue that if you're doing a full checksum rsync of a
multi-terabyte drive, you're probably going about the task wrong.  If I
were to implement a scheme like this, I'd do volume-per-job, I'd use a
run-after-job that appended the volume name to a list of volumes to be
copied, and I wouldn't use rsync at all.  When you already know you want
to copy an entire volume, rsync is an inefficient way to do it.

Another approach, instead of using volume-per-job, would be to allow
multiple jobs per volume but with a volume use duration, and have
run-after-job add the volume to a list of volumes to be replicated, that
list acting as a queue for a separate tool which dequeues and replicates
each volume ONLY when it has been marked USED.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958



signature.asc
Description: OpenPGP digital signature
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Dimitri Maziuk]

On 08/22/2017 12:27 PM, Phil Stracchino wrote:

>> So I've a post-backup script that figures out which of the vchanger
>> "magazines" is the "current" one and rsyncs it to anther drive. That way
>> if a drive dies during write, I lose at most the latest backup.
> 
> That sounds like a pretty good idea.  After-the-fact redundancy.

It isn't: what you want is to write to two disks at once. But if the
software wasn't designed for that, that's problems all the way to the
catalog.

The other problem is that rsyncing an umpteen-TB drive can take a
non-trivial amount of time and ram.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Phil Stracchino]

On 08/22/17 13:10, Dimitri Maziuk wrote:
> 
> We're using plain SATA drives in hot-swap bays. With vchanger they work
> just fine for backup -- we haven't tried restoring anything from an
> already removed disk yet -- but they do die on you even without dropping
> 'em.

> So I've a post-backup script that figures out which of the vchanger
> "magazines" is the "current" one and rsyncs it to anther drive. That way
> if a drive dies during write, I lose at most the latest backup.

That sounds like a pretty good idea.  After-the-fact redundancy.

> There is a number of obvious problems with that, and here's a
> non-obvious one: instead of filling them up strictly sequentially, every
> once in a while bacula randomly decides to write a volume in some other
> magazine. At which point my clever script works exactly as designed...

I can see how that would be a problem.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958



signature.asc
Description: OpenPGP digital signature
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Dimitri Maziuk]

On 08/22/2017 11:06 AM, Phil Stracchino wrote:

> Ah, well now you're talking about a different issue.  Because that's not
> really a virtual-tape-on-disk problem.  That's now a removable-storage
> problem.

Well, a tape is both. You can put a disk in a drive, fill it up with
bytes, write an EOT in the last block and call it a tape. It's a bit
inconvenient when its an 8TB "tape".

> I actually have not tried the current generation of removable disk
> technology (Tandberg RDX or similar), but I'd like to, budget
> permitting, because I'm getting awfully tired of LTO tape.  I expect it
> to become more and more feasible and cost-effective as the cost of flash
> storage continues to drop.  (I've never been much of a fan of the idea
> of removable spinning-rust as backup media, because you only have to
> drop it once...)

We're using plain SATA drives in hot-swap bays. With vchanger they work
just fine for backup -- we haven't tried restoring anything from an
already removed disk yet -- but they do die on you even without dropping
'em.

So I've a post-backup script that figures out which of the vchanger
"magazines" is the "current" one and rsyncs it to anther drive. That way
if a drive dies during write, I lose at most the latest backup.

There is a number of obvious problems with that, and here's a
non-obvious one: instead of filling them up strictly sequentially, every
once in a while bacula randomly decides to write a volume in some other
magazine. At which point my clever script works exactly as designed...

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu



signature.asc
Description: OpenPGP digital signature
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Phil Stracchino]

On 08/22/17 11:01, Dimitri Maziuk wrote:
> Bacula, in fact, is much better than some, if you can fit all your 
> backups on a single filesystem: bacula's built-in disk backup setup is a 
> breeze.
> 
> However if you want to backup to removable disks and put them on the 
> shelf when full, that's where things get less than perfect fast.


Ah, well now you're talking about a different issue.  Because that's not
really a virtual-tape-on-disk problem.  That's now a removable-storage
problem.

I actually have not tried the current generation of removable disk
technology (Tandberg RDX or similar), but I'd like to, budget
permitting, because I'm getting awfully tired of LTO tape.  I expect it
to become more and more feasible and cost-effective as the cost of flash
storage continues to drop.  (I've never been much of a fan of the idea
of removable spinning-rust as backup media, because you only have to
drop it once...)


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Dimitri Maziuk]

On 2017-08-22 03:10, Kern Sibbald wrote:

Dimitri,

I have heard the same complaint from you a number of times: "Bacula 
sucks at disk as tapes".  It is my view that your statement is simply 
not true.


I think at the very basic level, a disk is a device that doesn't quack 
like a tape and doesn't walk like a tape. What this means is if your 
software is built on the tape paradigm, it can call a disk "a tape", and 
it will suck because it actually is not a tape. It does not matter to me 
what API the code calls to do addressing: the point is that a disk has 
multiple files on it while a tape has bytes until EOT.


Don't get me wrong, this is not specific to bacula: every tape backup 
software has the same conceptual limitation and has to employ some 
workarounds for backups to disk. But this is the bacula list, we're not 
talking about every software here.


Bacula, in fact, is much better than some, if you can fit all your 
backups on a single filesystem: bacula's built-in disk backup setup is a 
breeze.


However if you want to backup to removable disks and put them on the 
shelf when full, that's where things get less than perfect fast.


Dima

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Bacula driver misconceptions

[Kern Sibbald]

  
  
Hello,
  
  Bacula Enterprise can do what you want, but it accomplishes it in
  a different way.  For Bacula Enterprise, there is software that
  allows MSPs/ISPs do multi-tenant implementations with Bacula
  ensuring that users (each tenant) can access only his/her own
  data.
  
  If you specifically authorize me I will give your email to a
  Bacula Enterprise salesman (I need to know where you are based),
  or you can contact Bacula Systems directly by going to 
  www.baculasystems.com and asking for a quote.
  
  Best regards,
  Kern
  
  On 22/08/2017 12:16, muha...@assistanz.com wrote:


  Hi,

Can we achieve this separate pool configuration, or multiple
device storage in  BACULA ENTERPRISE without any performance
issues.

I want maximum throughput may be for a 3000 vm's and each vm
should have their backup separated. They must be able to see
their individual backups and do what ever they want.

Regards,
Muhasin


From: "Kern Sibbald"
  
  To: "Dimitri Maziuk" ,
  "bacula-users" 
  Sent: Tuesday, August 22, 2017 1:40:46 PM
  Subject: [Bacula-users] Bacula driver misconceptions



  Dimitri,

I have heard the same complaint from you a number of times:
"Bacula sucks at disk as tapes".  It is my view that your
statement is simply not true.  Bacula handles tapes, disk,
fifo, cloud, aligned volumes, deduplication, and many other
device types.  The original code was largely monolithic with
tests that did different things depending on the device. 
However, for quite some time now Bacula has been split into
separate drivers for each device type, which means that it
is not at all correct to say that Bacula treats tapes and
disk the same.  They are both handled very differently
including the device addressing -- that is each device has
very different ways of tracking device addresses and the OS
APIs that it uses to address those devices.

You can keep thinking as you wish, but it worries me that
your (what I consider incorrect) concept of Bacula drivers
may confuse other users.  There are certain common factors
between each of the end devices mentioned above that Bacula
handles, but even a cursory look at the source code for
Bacula 9.0.x will prove that the details of the device are
handled very differently.

Best regards,
Kern

On 21/08/2017 19:27, Dimitri Maziuk wrote:
  
  
On 08/21/2017 11:49 AM, Phil Stracchino wrote:



  I can immediately cite one practical *disadvantage*:  you're going to be
doing a fandango on disk.  With 500 volumes open, you may well spend
more time seeking than actually writing.


Well, I'd also want to look at the elevator scheduler and the actual
hardware too...

The problem as I see it, is bacula sucks at "disks as tapes". It doesn't
work without vchanger, and it can't auto-label volumes in the vchanger.
(Inability to write two copies at once to guard against disk failure is
icing on the cake.)

If you back up to a single filesystem, you can spool jobs in parallel to
fast drive and have them despooling to volumes sequentially to spinning
rust with very little practical speed penalty. Provided your disk
subsystem is faster than the network, of course. Adding one volume per
job/client constraint and $(JobId) to volume label shouldn't be too hard.

If you want to use removable drives as "magazines" with one drive per
client... I'd be tempted to take a close look at BackupPC or some such
before I'd commit to bacula.





--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot



___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

  
  
  
  
--
  Check out the vibrant tech community on one of the world's
  most
  engaging tech sites, Slashdot.org! http://sdm.link/slashdot
  __

Re: [Bacula-users] Bacula driver misconceptions

[muhasin]

Hi, 

Can we achieve this separate pool configuration, or multiple device storage in 
BACULA ENTERPRISE without any performance issues. 

I want maximum throughput may be for a 3000 vm's and each vm should have their 
backup separated. They must be able to see their individual backups and do what 
ever they want. 

Regards, 
Muhasin 


From: "Kern Sibbald"  
To: "Dimitri Maziuk" , "bacula-users" 
 
Sent: Tuesday, August 22, 2017 1:40:46 PM 
Subject: [Bacula-users] Bacula driver misconceptions 

Dimitri, 

I have heard the same complaint from you a number of times: "Bacula sucks at 
disk as tapes". It is my view that your statement is simply not true. Bacula 
handles tapes, disk, fifo, cloud, aligned volumes, deduplication, and many 
other device types. The original code was largely monolithic with tests that 
did different things depending on the device. However, for quite some time now 
Bacula has been split into separate drivers for each device type, which means 
that it is not at all correct to say that Bacula treats tapes and disk the 
same. They are both handled very differently including the device addressing -- 
that is each device has very different ways of tracking device addresses and 
the OS APIs that it uses to address those devices. 

You can keep thinking as you wish, but it worries me that your (what I consider 
incorrect) concept of Bacula drivers may confuse other users. There are certain 
common factors between each of the end devices mentioned above that Bacula 
handles, but even a cursory look at the source code for Bacula 9.0.x will prove 
that the details of the device are handled very differently. 

Best regards, 
Kern 

On 21/08/2017 19:27, Dimitri Maziuk wrote: 



On 08/21/2017 11:49 AM, Phil Stracchino wrote: 

BQ_BEGIN

I can immediately cite one practical *disadvantage*:  you're going to be
doing a fandango on disk.  With 500 volumes open, you may well spend
more time seeking than actually writing. 



Well, I'd also want to look at the elevator scheduler and the actual
hardware too...

The problem as I see it, is bacula sucks at "disks as tapes". It doesn't
work without vchanger, and it can't auto-label volumes in the vchanger.
(Inability to write two copies at once to guard against disk failure is
icing on the cake.)

If you back up to a single filesystem, you can spool jobs in parallel to
fast drive and have them despooling to volumes sequentially to spinning
rust with very little practical speed penalty. Provided your disk
subsystem is faster than the network, of course. Adding one volume per
job/client constraint and $(JobId) to volume label shouldn't be too hard.

If you want to use removable drives as "magazines" with one drive per
client... I'd be tempted to take a close look at BackupPC or some such
before I'd commit to bacula. 


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! [ http://sdm.link/slashdot | 
http://sdm.link/slashdot ] 


___
Bacula-users mailing list [ mailto:Bacula-users@lists.sourceforge.net | 
Bacula-users@lists.sourceforge.net ] [ 
https://lists.sourceforge.net/lists/listinfo/bacula-users | 
https://lists.sourceforge.net/lists/listinfo/bacula-users ] 

BQ_END




-- 
Check out the vibrant tech community on one of the world's most 
engaging tech sites, Slashdot.org! http://sdm.link/slashdot 
___ 
Bacula-users mailing list 
Bacula-users@lists.sourceforge.net 
https://lists.sourceforge.net/lists/listinfo/bacula-users 

-- 
Muhasin C.M 
Systems Engineer R & D 
Assistanz Networks Pvt LTD 

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Bacula driver misconceptions

[Kern Sibbald]

  
  
Dimitri,
  
  I have heard the same complaint from you a number of times:
  "Bacula sucks at disk as tapes".  It is my view that your
  statement is simply not true.  Bacula handles tapes, disk, fifo,
  cloud, aligned volumes, deduplication, and many other device
  types.  The original code was largely monolithic with tests that
  did different things depending on the device.  However, for quite
  some time now Bacula has been split into separate drivers for each
  device type, which means that it is not at all correct to say that
  Bacula treats tapes and disk the same.  They are both handled very
  differently including the device addressing -- that is each device
  has very different ways of tracking device addresses and the OS
  APIs that it uses to address those devices.
  
  You can keep thinking as you wish, but it worries me that your
  (what I consider incorrect) concept of Bacula drivers may confuse
  other users.  There are certain common factors between each of the
  end devices mentioned above that Bacula handles, but even a
  cursory look at the source code for Bacula 9.0.x will prove that
  the details of the device are handled very differently.
  
  Best regards,
  Kern
  
  On 21/08/2017 19:27, Dimitri Maziuk wrote:


  On 08/21/2017 11:49 AM, Phil Stracchino wrote:


  
I can immediately cite one practical *disadvantage*:  you're going to be
doing a fandango on disk.  With 500 volumes open, you may well spend
more time seeking than actually writing.

  
  
Well, I'd also want to look at the elevator scheduler and the actual
hardware too...

The problem as I see it, is bacula sucks at "disks as tapes". It doesn't
work without vchanger, and it can't auto-label volumes in the vchanger.
(Inability to write two copies at once to guard against disk failure is
icing on the cake.)

If you back up to a single filesystem, you can spool jobs in parallel to
fast drive and have them despooling to volumes sequentially to spinning
rust with very little practical speed penalty. Provided your disk
subsystem is faster than the network, of course. Adding one volume per
job/client constraint and $(JobId) to volume label shouldn't be too hard.

If you want to use removable drives as "magazines" with one drive per
client... I'd be tempted to take a close look at BackupPC or some such
before I'd commit to bacula.


  
  
  
  --
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
  
  
  
  ___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users




  


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users