Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Phil Stracchino]

On 03/20/16 14:43, Dimitri Maziuk wrote:
> On 2016-03-19 14:41, Phil Stracchino wrote:
> 
>> What's the best way to handle a removable-cartridge-drive technology
>> like RDX in Bacula -- use the virtual changer...?
> 
> "Without vchanger, Bacula has no right to claim it supports disk-based 
> backup". Vchanger is the only way to fly.
> 
> LTO-6: ~$11/TB
> Seagate 8TB "archive" SATA drive: ~$28/TB
> RDX: ~$100/TB
> 
> The drive: $2K for LTO vs $200 for RDX vs $20 for SATA.

And frankly, unless you have a cleanroom to put it in, it's replacing
all those $2K drives that sours you on LTO.  They don't last long
outside of a controlled environment.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: 603.293.8485

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Dimitri Maziuk]

On 2016-03-19 14:41, Phil Stracchino wrote:

> What's the best way to handle a removable-cartridge-drive technology
> like RDX in Bacula -- use the virtual changer...?

"Without vchanger, Bacula has no right to claim it supports disk-based 
backup". Vchanger is the only way to fly.

LTO-6: ~$11/TB
Seagate 8TB "archive" SATA drive: ~$28/TB
RDX: ~$100/TB

The drive: $2K for LTO vs $200 for RDX vs $20 for SATA.

Dima


--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Josh Fisher]

On 3/19/2016 3:41 PM, Phil Stracchino wrote:
> On 03/19/16 10:56, Josh Fisher wrote:
>> On 3/17/2016 8:48 AM, Alan Brown wrote:
>>> . What's killed all these "smaller"
>>> formats is cheap(ish) HDD/SSDs, cloud storage and the likes of Netflix.
>>> That's despite even BDXL 120GB not being large enough capacity to hold a
>>> complete 4k video title.
>>>
>> RDX is a good choice for "smaller" format, although smaller is relative.
>> The bottom line is that a USB3 RDX drive and 6 2 TB cartridges is about
>> the same cost as a single LTO-6 drive and 4 2.5 TB cartridges. If media
>> needs in the long term will stay below 12 - 16 TB, then RDX is the
>> simpler, and IMO better choice. Above that, LTO-6 wins out due to much
>> lower media cost.
>>
>> For the backup window factor, LTO-6 wins every time. However, RDX
>> performance is on par with LTO-4, so for many, if not most, small
>> businesses, it meets their needs.
> I'm still surprised that cloud storage is even considered by anyone but
> single users with a single PC.  And even there I'm surprised it's viable
> given how poor most US "broadband" service is.
>
> What do you mean about the backup window, though?

The amount of time available to run backups, or that the backups must 
run within a certain time frame such that the speed of the backup 
devices becomes an important factor.

>
> What's the best way to handle a removable-cartridge-drive technology
> like RDX in Bacula -- use the virtual changer...?

Apart from hardware encryption and robustness, RDX is just normal HDD to 
the system, so virtual changer or vchanger. The built-in virtual changer 
allows auto-labeling volumes, while vchanger allows multiple 
simultaneously attached cartridges to act as a single autolchanger and 
supports automounting and autorun of 'update slots' on cartridge changes.



--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Cejka Rudolf]

Alan Brown wrote (2016/03/17):
> Caveat: BDXL is up to 120GB per disc (quad layer) and  It _may_ be worth 
> investigating this format for backups, but bacula doesn't play nicely 
> with optical media.
> 
> HVD development (6TB per disc) was abandoned in 2008. Ritek demonstrated 
> 250GB BDXL discs (10 layer) 8 years ago but they've never been marketed. 
> Ditto with Pioneer's 400GB BDXL format and the "1TB Blueray" disk is now 
> 4 years past proposed launch date. What's killed all these "smaller" 
> formats is cheap(ish) HDD/SSDs, cloud storage and the likes of Netflix. 
> That's despite even BDXL 120GB not being large enough capacity to hold a 
> complete 4k video title.

I think that two main killers for all these optical things for data
storage are unbelief in reliability (or simply unreliability) and
slooowness.

-- 
Rudolf Cejka  http://www.fit.vutbr.cz/~cejkar
Brno University of Technology, Faculty of Information Technology
Bozetechova 2, 612 66  Brno, Czech Republic

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Kern Sibbald]

On 03/20/2016 05:41 AM, Phil Stracchino wrote:
> On 03/19/16 10:56, Josh Fisher wrote:
>> On 3/17/2016 8:48 AM, Alan Brown wrote:
>>> . What's killed all these "smaller"
>>> formats is cheap(ish) HDD/SSDs, cloud storage and the likes of Netflix.
>>> That's despite even BDXL 120GB not being large enough capacity to hold a
>>> complete 4k video title.
>>>
>>
>> RDX is a good choice for "smaller" format, although smaller is relative.
>> The bottom line is that a USB3 RDX drive and 6 2 TB cartridges is about
>> the same cost as a single LTO-6 drive and 4 2.5 TB cartridges. If media
>> needs in the long term will stay below 12 - 16 TB, then RDX is the
>> simpler, and IMO better choice. Above that, LTO-6 wins out due to much
>> lower media cost.
>>
>> For the backup window factor, LTO-6 wins every time. However, RDX
>> performance is on par with LTO-4, so for many, if not most, small
>> businesses, it meets their needs.
>
> I'm still surprised that cloud storage is even considered by anyone but
> single users with a single PC.  And even there I'm surprised it's viable
> given how poor most US "broadband" service is.

Hello Phil,

I agree with your statement above. However, there is a big demand for 
Cloud backup, so one of my next projects will be to add S3 support to 
Bacula.  A certain number of large enterprises are adding local private 
cloud installations where bandwidth is not an issue, so even for large 
enterprises, having an S3 option is becoming an important issue.

Best regards,
Kern

>
> What do you mean about the backup window, though?
>
> What's the best way to handle a removable-cartridge-drive technology
> like RDX in Bacula -- use the virtual changer...?
>
>

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Phil Stracchino]

On 03/19/16 10:56, Josh Fisher wrote:
> On 3/17/2016 8:48 AM, Alan Brown wrote:
>> . What's killed all these "smaller"
>> formats is cheap(ish) HDD/SSDs, cloud storage and the likes of Netflix.
>> That's despite even BDXL 120GB not being large enough capacity to hold a
>> complete 4k video title.
>>
> 
> RDX is a good choice for "smaller" format, although smaller is relative. 
> The bottom line is that a USB3 RDX drive and 6 2 TB cartridges is about 
> the same cost as a single LTO-6 drive and 4 2.5 TB cartridges. If media 
> needs in the long term will stay below 12 - 16 TB, then RDX is the 
> simpler, and IMO better choice. Above that, LTO-6 wins out due to much 
> lower media cost.
> 
> For the backup window factor, LTO-6 wins every time. However, RDX 
> performance is on par with LTO-4, so for many, if not most, small 
> businesses, it meets their needs.

I'm still surprised that cloud storage is even considered by anyone but
single users with a single PC.  And even there I'm surprised it's viable
given how poor most US "broadband" service is.

What do you mean about the backup window, though?

What's the best way to handle a removable-cartridge-drive technology
like RDX in Bacula -- use the virtual changer...?


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: 603.293.8485

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Josh Fisher]

On 3/17/2016 8:48 AM, Alan Brown wrote:
> . What's killed all these "smaller"
> formats is cheap(ish) HDD/SSDs, cloud storage and the likes of Netflix.
> That's despite even BDXL 120GB not being large enough capacity to hold a
> complete 4k video title.
>

RDX is a good choice for "smaller" format, although smaller is relative. 
The bottom line is that a USB3 RDX drive and 6 2 TB cartridges is about 
the same cost as a single LTO-6 drive and 4 2.5 TB cartridges. If media 
needs in the long term will stay below 12 - 16 TB, then RDX is the 
simpler, and IMO better choice. Above that, LTO-6 wins out due to much 
lower media cost.

For the backup window factor, LTO-6 wins every time. However, RDX 
performance is on par with LTO-4, so for many, if not most, small 
businesses, it meets their needs.

Because of RDX (or even simple USB HDD), I see very little reason for 
Bacula to worry about supporting optical drives. They are slow, prone to 
errors, and at this point simply not adequate, even for small business 
or personal use.


--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Alan Brown]

On 12/03/16 00:14, Heitor Faria wrote:
>
>> SSD is the only way to fly. After having tested with a PCIe NVMe drive, I'd 
>> say
>> that's preferred, but a _fast_ SATA2/3 or SAS2 drive will work too (The old
>> spool was a stripe of Intel SLC SSDs, the new one is a DC3700 card)
> I never got this spooling / disk backup fetish. I mean: it keeps data 
> interleaving to happen, but at what cost?
> With SSD you can have a ridiculous hight throughput, but you still need to 
> wait backup data being copied to tapes / definitive slow disk. Unless you 
> have a really short backup window at client size, it is useless.

If you use tape, you need spooling on (fast) SSD. If you don't need to 
use tape then spooling is superfluous.

Because of the cost of drives, tape is only worth using once you pass 
about 1-200TB or so needing to be backed up (or several hundred small 
systems) and at that point you're going to want multiple simultaneous 
backups anyway.

Don't bother with DAT for backups. It's not reliable due to the narrow 
tape format. Any minor defect in the substrate translates to a major 
loss and in any case it's too small to be worth the hassle in comparison 
to using solid state storage.

Apart from DAT and  LTO/3592/T1D every other tape format is now 
abandonware so there's not much point in persisting with them and even 
IBM is looking at dropping 3592 in favour of LTO. I wouldn't be at all 
surprised to find Oracle announcing that T1 format is end of life 
soonish too. The tape market is too small to support 3 competing half 
inch formats even if 3592 and T1D are twice the capacity of LTO7 
(the cost of cartridges is a lot more than twice as much as LTO7, as are 
the drives, so there's no financial incentive to use them in new 
installations.)

Caveat: BDXL is up to 120GB per disc (quad layer) and  It _may_ be worth 
investigating this format for backups, but bacula doesn't play nicely 
with optical media.

HVD development (6TB per disc) was abandoned in 2008. Ritek demonstrated 
250GB BDXL discs (10 layer) 8 years ago but they've never been marketed. 
Ditto with Pioneer's 400GB BDXL format and the "1TB Blueray" disk is now 
4 years past proposed launch date. What's killed all these "smaller" 
formats is cheap(ish) HDD/SSDs, cloud storage and the likes of Netflix. 
That's despite even BDXL 120GB not being large enough capacity to hold a 
complete 4k video title.





--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Alan Brown]

On 13/03/16 21:48, Dan Langille wrote:
>> As well as increasing max file size you need to boost the tape buffer size 
>> from the 64kB default. I use 2MB
> This is a hardware setting?

No, it's a bacula-sd setting

> I tried Minimum block size & Maximum block size on my tape drive, but need to 
> try it again.
>

My settings:

   Spool Directory = /var/bacula/spool/TAPE
   Maximum File Size = 16G
   Maximum Network Buffer Size = 262144
   Maximum block size = 2M
   Maximum Spool Size = 220G
   Maximum Job Spool Size = 32G


Don't touch minimum block size. It's fine as it is.

2MB is the maximum allowable in bacula, however you probably won't see 
much speed improvement past 500kB unless there's compressible data  in 
the train (it's faster, but only a little. I use it because I have tight 
backup windows and a lot of data to deal with)

FWIW Most LTO tape drives have 8MB(IBM) or 16MB(HP) write buffers.




--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Cejka Rudolf]

Dan Langille wrote (2016/03/13):
> > As well as increasing max file size you need to boost the tape buffer size 
> > from the 64kB default. I use 2MB
> 
> This is a hardware setting?

Hello, it is software settings:

* Increasing max file size is "Maximum File Size". I use 16 GB for
  LTO-5 and for LTO-4 it should be similar. Every file mark means delay
  of several seconds (3-5?), so you can count, how big slow down for
  each file size it is in percents. On the other side, too big file size
  means slower seeks.

* Tape buffer size: It is "Maximum Block Size". It seems that in Linux
  it is a must to increase it. However, in FreeBSD, I have no problems
  with default 65536 bytes, where I have seen speeds over 250 MB/s with
  well compressable datas and without notable system load, so I did not
  increase it yet. For check, how big block you can use, try mt status -v.
  It is smaller of maxio and cpi_maxio. Without kernel patching, you could
  certainly use 131072 bytes. However, maxio could be increased easily
  (MAXPHYS kernel option) and cpi_maxio depends on driver, how well the
  value is obtained. Sometimes it is just a constant, which is known that
  it could be increased. But, I still think that it is not needed so much
  in FreeBSD.

-- 
Rudolf Cejka  http://www.fit.vutbr.cz/~cejkar
Brno University of Technology, Faculty of Information Technology
Bozetechova 2, 612 66  Brno, Czech Republic

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Dan Langille]

> On Mar 11, 2016, at 6:56 PM, Alan Brown  wrote:
> 
> On 11/03/16 20:14, Simon Templar wrote:
>> In my case using spooling didn’t prevent shoe-shining; it just introduced 
>> long pauses while data was spooled. I think all this means is that I can 
>> read from my data sources faster than my tape can write.
> 
> Unless you are using DAT, do not use mechanical drives for spooling - they 
> can't keep up with the tape drive unless you're using one that's dedicated 
> and only spooling/despooling for a single job (LTO1-2-3, incompressible data) 
> or can't keep up at all (As above with any form of compressible data, or 
> LTO4,5,6,7)
> 
> SSD is the only way to fly. After having tested with a PCIe NVMe drive, I'd 
> say that's preferred, but a _fast_ SATA2/3 or SAS2 drive will work too (The 
> old spool was a stripe of Intel SLC SSDs, the new one is a DC3700 card)
> 
> Spooling really comes into its own when you're running multiple jobs. Whilst 
> one job is despooling, others can be spooling. The interleaving effect means 
> all your jobs complete in a faster period of time.

Perhaps I should start testing with multiple concurrent copy-disk-to-tape jobs.

> As well as increasing max file size you need to boost the tape buffer size 
> from the 64kB default. I use 2MB

This is a hardware setting?

I tried Minimum block size & Maximum block size on my tape drive, but need to 
try it again.

--
Dan Langille - BSDCan / PGCon
d...@langille.org






signature.asc
Description: Message signed with OpenPGP using GPGMail
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Dan Langille]

> On Mar 11, 2016, at 7:14 PM, Heitor Faria  wrote:
> 
> On 11/03/16 20:14, Simon Templar wrote:
> In my case using spooling didn’t prevent shoe-shining; it just introduced 
> long pauses while data was spooled. I think all this means is that I can read 
> from my data sources faster than my tape can write.
> 
> Unless you are using DAT, do not use mechanical drives for spooling - they 
> can't keep up with the tape drive unless you're using one that's dedicated 
> and only spooling/despooling for a single job (LTO1-2-3, incompressible data) 
> or can't keep up at all (As above with any form of compressible data, or 
> LTO4,5,6,7)
> Hello Alan: I have the same perception.
> SSD is the only way to fly. After having tested with a PCIe NVMe drive, I'd 
> say that's preferred, but a _fast_ SATA2/3 or SAS2 drive will work too (The 
> old spool was a stripe of Intel SLC SSDs, the new one is a DC3700 card)
> I never got this spooling / disk backup fetish. I mean: it keeps data 
> interleaving to happen, but at what cost?
> With SSD you can have a ridiculous hight throughput, but you still need to 
> wait backup data being copied to tapes / definitive slow disk. Unless you 
> have a really short backup window at client size, it is useless.


I backup to disk, then copy to tape later.  This way I have backups on disk and 
tape. Restoring from disk is faster than restoring from tape. It is quite 
useful.

--
Dan Langille - BSDCan / PGCon
d...@langille.org






signature.asc
Description: Message signed with OpenPGP using GPGMail
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Heitor Faria]

> On 11/03/16 20:14, Simon Templar wrote:

>> In my case using spooling didn’t prevent shoe-shining; it just introduced 
>> long
>> pauses while data was spooled. I think all this means is that I can read from
>> my data sources faster than my tape can write.
> Unless you are using DAT, do not use mechanical drives for spooling - they 
> can't
> keep up with the tape drive unless you're using one that's dedicated and only
> spooling/despooling for a single job (LTO1-2-3, incompressible data) or can't
> keep up at all (As above with any form of compressible data, or LTO4,5,6,7)
Hello Alan: I have the same perception. 

> SSD is the only way to fly. After having tested with a PCIe NVMe drive, I'd 
> say
> that's preferred, but a _fast_ SATA2/3 or SAS2 drive will work too (The old
> spool was a stripe of Intel SLC SSDs, the new one is a DC3700 card)
I never got this spooling / disk backup fetish. I mean: it keeps data 
interleaving to happen, but at what cost? 
With SSD you can have a ridiculous hight throughput, but you still need to wait 
backup data being copied to tapes / definitive slow disk. Unless you have a 
really short backup window at client size, it is useless. 

> Spooling really comes into its own when you're running multiple jobs. Whilst 
> one
> job is despooling, others can be spooling. The interleaving effect means all
> your jobs complete in a faster period of time.
Again I agree with you here. You will tend to have more optimum spooling usage 
in this scenario if you have enough spooling size. 

Regards, 
-- 
=== 
Heitor Medrado de Faria - LPIC-III | ITIL-F | Bacula Systems Certified 
Administrator II 
Próximas aulas telepresencial ao-vivo - 15 de fevereiro: 
http://www.bacula.com.br/agenda/ 
Ministro treinamento e implementação in-company Bacula: 
http://www.bacula.com.br/in-company/ 
Ou assista minhas videoaulas on-line: 
http://www.bacula.com.br/treinamento-bacula-ed/ 
61 8268-4220 
Site: www.bacula.com.br | Facebook: heitor.faria 
 
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Alan Brown]

On 11/03/16 20:14, Simon Templar wrote:
> In my case using spooling didn’t prevent shoe-shining; it just
> introduced long pauses while data was spooled. I think all this means
> is that I can read from my data sources faster than my tape can write.

Unless you are using DAT, do not use mechanical drives for spooling -
they can't keep up with the tape drive unless you're using one that's
dedicated and only spooling/despooling for a single job (LTO1-2-3,
incompressible data) or can't keep up at all (As above with any form of
compressible data, or LTO4,5,6,7)

SSD is the only way to fly. After having tested with a PCIe NVMe drive,
I'd say that's preferred, but a _fast_ SATA2/3 or SAS2 drive will work
too (The old spool was a stripe of Intel SLC SSDs, the new one is a
DC3700 card)

Spooling really comes into its own when you're running multiple jobs.
Whilst one job is despooling, others can be spooling. The interleaving
effect means all your jobs complete in a faster period of time.

As well as increasing max file size you need to boost the tape buffer
size from the 64kB default. I use 2MB


>
> So far the only change I made to help with shoe-shining was to set Max
> File Size to a large number (mine is now set to 20g, after first
> trying 3gb then 5gb). This one change alone is probably responsible
> for most of the performance increase that I’ve been able to achieve
> thus far. I’d like to test and tune more, but I’m still wrestling with
> things like tape mount timeouts (no 3rd shift operators), job run time
> timeouts, etc…
>
> -Simon
>
>
>> On Mar 10, 2016, at 11:09 PM, Kern Sibbald > > wrote:
>>
>> I have not tried this, but one thing that may help a lot is to turn on 
>> data spooling for the tape device. This will probably not speed up the 
>> process but should prevent that tape shoe-shine (start and stopping).
>
>
>
> --
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140
>
>
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users



--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Dan Langille]

> On Mar 11, 2016, at 3:14 PM, Simon Templar  wrote:

>> On Mar 10, 2016, at 11:09 PM, Kern Sibbald  wrote:
>> 
>> I have not tried this, but one thing that may help a lot is to turn on
>> data spooling for the tape device. This will probably not speed up the
>> process but should prevent that tape shoe-shine (start and stopping).
> 
> 
> In my case using spooling didn’t prevent shoe-shining; it just introduced 
> long pauses while data was spooled. I think all this means is that I can read 
> from my data sources faster than my tape can write.
> 
> So far the only change I made to help with shoe-shining was to set Max File 
> Size to a large number (mine is now set to 20g, after first trying 3gb then 
> 5gb). This one change alone is probably responsible for most of the 
> performance increase that I’ve been able to achieve thus far. I’d like to 
> test and tune more, but I’m still wrestling with things like tape mount 
> timeouts (no 3rd shift operators), job run time timeouts, etc…

I have this set to about 375GB

  Maximum Spool Size  = 375809638400
  Maximum Job Spool Size  = 375809638400
--
Dan Langille - BSDCan / PGCon
d...@langille.org






signature.asc
Description: Message signed with OpenPGP using GPGMail
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Simon Templar]

In my case using spooling didn’t prevent shoe-shining; it just introduced long 
pauses while data was spooled. I think all this means is that I can read from 
my data sources faster than my tape can write.

So far the only change I made to help with shoe-shining was to set Max File 
Size to a large number (mine is now set to 20g, after first trying 3gb then 
5gb). This one change alone is probably responsible for most of the performance 
increase that I’ve been able to achieve thus far. I’d like to test and tune 
more, but I’m still wrestling with things like tape mount timeouts (no 3rd 
shift operators), job run time timeouts, etc…

-Simon


> On Mar 10, 2016, at 11:09 PM, Kern Sibbald  wrote:
> 
> I have not tried this, but one thing that may help a lot is to turn on 
> data spooling for the tape device. This will probably not speed up the 
> process but should prevent that tape shoe-shine (start and stopping).

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Dan Langille]

--
Dan Langille - BSDCan / PGCon
d...@langille.org




> On Mar 10, 2016, at 11:09 PM, Kern Sibbald  wrote:
> 
> Hello Dan,
> 
> Copying from disk to tape with Bacula's current algorithm is virtually 
> guaranteed to be slower than using tar.  This is for several reasons:
> 
> 1. Bacula currently is single threaded and reads a block from disk then stops 
> to write the output block to tape.
> 
> 2. After reading the block from disk, Bacula checks the checksum (CPU 
> intensive).
> 
> 3. After the checksum is verified, Bacula reads each record from the block 
> and then packs the records into a new block for writing.
> 
> 4. Once the output block is full, Bacula computes a checksum for the block, 
> writes it and waits for the I/O to be complete.
> 
> So, as you can see, it is a very expensive process, but it is the only way to 
> guarantee that the whole prior backup is properly copied.
> 
> I have not tried this, but one thing that may help a lot is to turn on data 
> spooling for the tape device. This will probably not speed up the process but 
> should prevent that tape shoe-shine (start and stopping).
> 
> At some later time, Bacula will have multiple threads -- one that reads and 
> one that writes, and this could drastically improve performance.

I tried data spooling.

Original job, no spooling: 12752.6 KB/s

First, I tried spooling to local HHD: 11131.7 KB/s

Then I tried spooling to local SSD: 11542.1 KB/s

I have updated the gist with the bconsole 
output:https://gist.github.com/dlangille/2341a6da8f9ee836270c 

> 
> Best regards
> Kern
> 
> On 03/11/2016 08:44 AM, Dan Langille wrote:
>>> On Mar 9, 2016, at 6:52 PM, Heitor Faria  wrote:
>>> 
>>> I have a copy to tape job which copies from disk to tape using Bacula 7.4.0 
>>> and PostgreSQL 9.4 on FreeBSD 10.2
>>> 
>>> Everything is within one SD
>>> 
>>> Full details at https://gist.github.com/dlangille/2341a6da8f9ee836270c
>>> 
>>> The job summary:
>>> 
>>>  Start time: 09-Mar-2016 19:41:54
>>>  End time:   09-Mar-2016 20:51:02
>>>  Elapsed time:   1 hour 9 mins 8 secs
>>>  Priority:   410
>>>  SD Files Written:   1
>>>  SD Bytes Written:   52,897,660,928 (52.89 GB)
>>>  Rate:   12752.6 KB/s
>>> 
>>> If I tar the volumes directly to tape, it takes only 16 minutes.
>>> 
>>> 
>>> $ time sudo tar -cf /dev/nsa1 IncrAuto-4525 IncrAuto-4320 IncrAuto-4324 
>>> IncrAuto-4321 \
 IncrAuto-4055 IncrAuto-4322 IncrAuto-4319 IncrAuto-3969 IncrAuto-3972 \
 IncrAuto-3973 IncrAuto-3971 IncrAuto-4058
>>> 
>>> real15m47.508s
>>> user0m5.844s
>>> sys 1m51.834s
>>> 
>>> Spooling attributes is trivial:
>>> 
>>> 09-Mar 20:51 crey-sd JobId 232945: Sending spooled attrs to the Director. 
>>> Despooling 321 bytes ...
>>> 09-Mar 20:51 bacula-dir JobId 232944: Bacula bacula-dir 7.4.0 (16Jan16):
>>> 
>>> I am not sure where to look to figure this out.
>>> Hello, Dan: maybe there is nothing to figure it out. Packing volumes with 
>>> tar directly to tapes makes you unable to restore a single file or even a 
>>> single job (with bootstrap) from an entire tape. I think it's a trade-off.
>> 
>> I have no desire to use tar.
>> 
>> I have a desire to effectively use an LTO tape drive.
>> 
>> --
>> Dan Langille - BSDCan / PGCon
>> d...@langille.org
>> 
>> 
>> 
>> 
>> 
>> 
>> --
>> Transform Data into Opportunity.
>> Accelerate data analysis in your applications with
>> Intel Data Analytics Acceleration Library.
>> Click to learn more.
>> http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140
>> 
>> 
>> 
>> ___
>> Bacula-users mailing list
>> Bacula-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>> 



signature.asc
Description: Message signed with OpenPGP using GPGMail
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Kern Sibbald]

Hello Dan,

Copying from disk to tape with Bacula's current algorithm is virtually 
guaranteed to be slower than using tar.  This is for several reasons:

1. Bacula currently is single threaded and reads a block from disk then 
stops to write the output block to tape.

2. After reading the block from disk, Bacula checks the checksum (CPU 
intensive).

3. After the checksum is verified, Bacula reads each record from the 
block and then packs the records into a new block for writing.

4. Once the output block is full, Bacula computes a checksum for the 
block, writes it and waits for the I/O to be complete.

So, as you can see, it is a very expensive process, but it is the only 
way to guarantee that the whole prior backup is properly copied.

I have not tried this, but one thing that may help a lot is to turn on 
data spooling for the tape device. This will probably not speed up the 
process but should prevent that tape shoe-shine (start and stopping).

At some later time, Bacula will have multiple threads -- one that reads 
and one that writes, and this could drastically improve performance.

Best regards
Kern

On 03/11/2016 08:44 AM, Dan Langille wrote:
>> On Mar 9, 2016, at 6:52 PM, Heitor Faria  wrote:
>>
>> I have a copy to tape job which copies from disk to tape using Bacula 7.4.0 
>> and PostgreSQL 9.4 on FreeBSD 10.2
>>
>> Everything is within one SD
>>
>> Full details at https://gist.github.com/dlangille/2341a6da8f9ee836270c
>>
>> The job summary:
>>
>>   Start time: 09-Mar-2016 19:41:54
>>   End time:   09-Mar-2016 20:51:02
>>   Elapsed time:   1 hour 9 mins 8 secs
>>   Priority:   410
>>   SD Files Written:   1
>>   SD Bytes Written:   52,897,660,928 (52.89 GB)
>>   Rate:   12752.6 KB/s
>>
>> If I tar the volumes directly to tape, it takes only 16 minutes.
>>
>>
>> $ time sudo tar -cf /dev/nsa1 IncrAuto-4525 IncrAuto-4320 IncrAuto-4324 
>> IncrAuto-4321 \
>>> IncrAuto-4055 IncrAuto-4322 IncrAuto-4319 IncrAuto-3969 IncrAuto-3972 \
>>> IncrAuto-3973 IncrAuto-3971 IncrAuto-4058
>>
>> real15m47.508s
>> user0m5.844s
>> sys 1m51.834s
>>
>> Spooling attributes is trivial:
>>
>> 09-Mar 20:51 crey-sd JobId 232945: Sending spooled attrs to the Director. 
>> Despooling 321 bytes ...
>> 09-Mar 20:51 bacula-dir JobId 232944: Bacula bacula-dir 7.4.0 (16Jan16):
>>
>> I am not sure where to look to figure this out.
>> Hello, Dan: maybe there is nothing to figure it out. Packing volumes with 
>> tar directly to tapes makes you unable to restore a single file or even a 
>> single job (with bootstrap) from an entire tape. I think it's a trade-off.
>
> I have no desire to use tar.
>
> I have a desire to effectively use an LTO tape drive.
>
> --
> Dan Langille - BSDCan / PGCon
> d...@langille.org
>
>
>
>
>
>
> --
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140
>
>
>
> ___
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Dan Langille]

> On Mar 10, 2016, at 6:35 PM, compdoc  wrote:
> 
>> Given LTO-4 can do 120MB/s, yeah, 94 is good enough
> 
> Be sure to disable compression. I think I have it disabled two places, on Dir 
> and well as on the SD.

Side note: I see compression mentioned at 
http://www.bacula.org/7.4.x-manuals/en/main/Configuring_Director.html 
 but not 
at http://www.bacula.org/7.4.x-manuals/en/main/Storage_Daemon_Configuratio.html 


http://www.bacula.org/7.4.x-manuals/en/main/New_Features_in_5_2_13.html#SECTION00654000
 

 mentions 'Storage resource within the Director's configuration' though.

> I built a low-power, mini-itx system to house an LTO-4 drive. The controller 
> is an LSI LSI20320IE Ultra320. All backup data is over the lan.
> 
> This is my last Windows backup. No spooling:
> 
> 
>  Elapsed time:   6 hours 13 mins 54 secs
>  Priority:   10
>  FD Files Written:   218,515
>  SD Files Written:   218,515
>  FD Bytes Written:   610,852,830,755 (610.8 GB)
>  SD Bytes Written:   610,902,766,901 (610.9 GB)
>  Rate:   27228.9 KB/s
>  Software Compression:   None
>  Snapshot/VSS:   yes
>  Encryption: no

That's 27 MB/s.

Checking for compression:

on bacula server:
$ sudo grep -ri compression /usr/local/etc/bacula/
$

on bacula sd:

$ sudo grep -ri compression /usr/local/etc/bacula/
$

No compression specified.

Thank you.

--
Dan Langille - BSDCan / PGCon
d...@langille.org






signature.asc
Description: Message signed with OpenPGP using GPGMail
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[compdoc]

> Given LTO-4 can do 120MB/s, yeah, 94 is good enough

Be sure to disable compression. I think I have it disabled two places, on Dir 
and well as on the SD. 

I built a low-power, mini-itx system to house an LTO-4 drive. The controller is 
an LSI LSI20320IE Ultra320. All backup data is over the lan. 

This is my last Windows backup. No spooling:


  Elapsed time:   6 hours 13 mins 54 secs
  Priority:   10
  FD Files Written:   218,515
  SD Files Written:   218,515
  FD Bytes Written:   610,852,830,755 (610.8 GB)
  SD Bytes Written:   610,902,766,901 (610.9 GB)
  Rate:   27228.9 KB/s
  Software Compression:   None
  Snapshot/VSS:   yes
  Encryption: no




--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Dan Langille]

> On Mar 9, 2016, at 6:52 PM, Heitor Faria  wrote:
> 
> I have a copy to tape job which copies from disk to tape using Bacula 7.4.0 
> and PostgreSQL 9.4 on FreeBSD 10.2
> 
> Everything is within one SD
> 
> Full details at https://gist.github.com/dlangille/2341a6da8f9ee836270c
> 
> The job summary:
> 
>  Start time: 09-Mar-2016 19:41:54
>  End time:   09-Mar-2016 20:51:02
>  Elapsed time:   1 hour 9 mins 8 secs
>  Priority:   410
>  SD Files Written:   1
>  SD Bytes Written:   52,897,660,928 (52.89 GB)
>  Rate:   12752.6 KB/s
> 
> If I tar the volumes directly to tape, it takes only 16 minutes.
> 
> 
> $ time sudo tar -cf /dev/nsa1 IncrAuto-4525 IncrAuto-4320 IncrAuto-4324 
> IncrAuto-4321 \
> > IncrAuto-4055 IncrAuto-4322 IncrAuto-4319 IncrAuto-3969 IncrAuto-3972 \
> > IncrAuto-3973 IncrAuto-3971 IncrAuto-4058
> 
> real15m47.508s
> user0m5.844s
> sys 1m51.834s
> 
> Spooling attributes is trivial:
> 
> 09-Mar 20:51 crey-sd JobId 232945: Sending spooled attrs to the Director. 
> Despooling 321 bytes ...
> 09-Mar 20:51 bacula-dir JobId 232944: Bacula bacula-dir 7.4.0 (16Jan16):
> 
> I am not sure where to look to figure this out.
> Hello, Dan: maybe there is nothing to figure it out. Packing volumes with tar 
> directly to tapes makes you unable to restore a single file or even a single 
> job (with bootstrap) from an entire tape. I think it's a trade-off.

I have no desire to use tar.

I have a desire to effectively use an LTO tape drive.

--
Dan Langille - BSDCan / PGCon
d...@langille.org






signature.asc
Description: Message signed with OpenPGP using GPGMail
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Dan Langille]

> On Mar 10, 2016, at 10:10 AM, Simon Templar  wrote:
> 
> Dan,
> 
> I’m new to bacula, and arguably not very smart, but I’ve been struggling with 
> tape drive performance pretty much since the moment I got the configurations 
> to a functional state so I’ll share my learnings thus far.
> 
> Can you hear your tape drive? If so, do you hear lots of stops and starts 
> when bacula is running the job? When mine was writing I’d hear lots of stops 
> and starts (shoe-shining, I believe it’s called) followed by long pauses of 
> silence.

I did not, but I know the sound you mean.

> 
> I have a single LTO6 drive, and so far I’ve made 2 changes to my setup that 
> brought my average write speed from less than 60 MB/s to between 130-160 MB/s.
> 
> First, the long pauses with no activity were found to be related to spooling. 
> I started by spooling to an idle set of 6 sata drives in a raidz, then later 
> switched to an ssd, and with either the behavior was that the tape drive 
> would be completely idle while the data was spooled, then the data was 
> written to the drive, then when it all was written the tape drive was 
> completely idle again while data was spooled. Needless to say this added a 
> long time to the backups for our platform. Our data ultimately resides on 
> another platform (a freenas) and the goal is to replicate it to our backup 
> server, then write to tape for archival purposes. It turns out that I can 
> write the 15tb directly from the nfs mount to tape faster than I can spool it 
> from local disk to ssd then write to tape. I’m sure something’s wrong with my 
> setup, but if you’re spooling it’s worth taking a look to see if you have 
> long tape drive idle times while the data is spooled. I would have guessed 
> that the spool directory should be kept as full as possible WHILE the spooled 
> data is written to tape, but that’s not the behavior I’m seeing.

There is no spooling involved in this job.  It is direct from local HDD to the 
tape drive.  Given it takes over an hour to copy to tape, I am sure stop/start 
is involved.

> 
> The second thing I did was add:
> Maximum File Size = 20GB
> 
> to my bacula-sd.conf. I probably need to tune this further, and it might make 
> restores slower (at least for individual files?) but it drastically cut down 
> on my drive starts and stops. Together, the maximum file size and elimination 
> of spooling more than doubled my average write speed in large backups.
> 
> -Simon
> 
> P.S. Is your tar write speed as fast as you’d expect for your LTO drive?

I did some math based on the values at 
https://gist.github.com/dlangille/2341a6da8f9ee836270c

27703979 KB
287 s
94.3 MBps

Given LTO-4 can do 120MB/s, yeah, 94 is good enough: 
https://en.wikipedia.org/wiki/Linear_Tape-Open#Generations


> 
> 
>> On Mar 9, 2016, at 6:14 PM, Dan Langille  wrote:
>> 
>> I have a copy to tape job which copies from disk to tape using Bacula 7.4.0 
>> and PostgreSQL 9.4 on FreeBSD 10.2
>> 
>> Everything is within one SD
>> 
>> Full details at https://gist.github.com/dlangille/2341a6da8f9ee836270c
>> 
>> The job summary:
>> 
>>  Start time: 09-Mar-2016 19:41:54
>>  End time:   09-Mar-2016 20:51:02
>>  Elapsed time:   1 hour 9 mins 8 secs
>>  Priority:   410
>>  SD Files Written:   1
>>  SD Bytes Written:   52,897,660,928 (52.89 GB)
>>  Rate:   12752.6 KB/s
>> 
>> If I tar the volumes directly to tape, it takes only 16 minutes.
>> 
>> 
>> $ time sudo tar -cf /dev/nsa1 IncrAuto-4525 IncrAuto-4320 IncrAuto-4324 
>> IncrAuto-4321 \
>> > IncrAuto-4055 IncrAuto-4322 IncrAuto-4319 IncrAuto-3969 IncrAuto-3972 \
>> > IncrAuto-3973 IncrAuto-3971 IncrAuto-4058
>> 
>> real15m47.508s
>> user0m5.844s
>> sys 1m51.834s
>> 
>> Spooling attributes is trivial:
>> 
>> 09-Mar 20:51 crey-sd JobId 232945: Sending spooled attrs to the Director. 
>> Despooling 321 bytes ...
>> 09-Mar 20:51 bacula-dir JobId 232944: Bacula bacula-dir 7.4.0 (16Jan16):
>> 
>> I am not sure where to look to figure this out.
>> 
>> Full job output:
>> 
>> 
>> 09-Mar 19:41 bacula-dir JobId 232944: Warning: FileSet MD5 digest not found.
>> 09-Mar 19:41 bacula-dir JobId 232944: The following 1 JobId was chosen to be 
>> copied: 232778
>> 09-Mar 19:41 bacula-dir JobId 232944: Copying using JobId=232778 
>> Job=BackupCatalog.2016-03-08_03.05.16_52
>> 09-Mar 19:41 bacula-dir JobId 232944: Bootstrap records written to 
>> /usr/local/bacula/working/bacula-dir.restore.107.bsr
>> 09-Mar 19:41 bacula-dir JobId 232944: Start Copying JobId 232944, 
>> Job=CopyToTape-Full-Just-One-tape-01b.2016-03-09_19.41.51_42
>> 09-Mar 19:41 bacula-dir JobId 232944: Using Device "vDrive-0" to read.
>> 09-Mar 19:41 bacula-dir JobId 232945: Using Device "LTO_0" to write.
>> 09-Mar 19:41 crey-sd JobId 232944: Ready to read from volume "IncrAuto-4525" 
>> on file device "vDrive-0" (/usr/local/bacula/volumes).
>> 09-Mar 19:41 crey-sd JobId 232944: Forward spacing Volume "

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Dan Langille]

> On Mar 10, 2016, at 2:19 PM, Josh Fisher  wrote:
> 
> 
> Was job 232778 the only job on these inc volumes?

Yes, and no.  On 4 of the 12 volumes, part of the job was at the end of each 
volume.

I have indicated yes/no for exclusive use of the volume, and the time Bacula 
spends reading from that Volume

These URLs pull details from my Catalog:

https://services.unixathome.org/bacula/jobs.php?name=IncrAuto-4525&action=listing
 

 no 6m
https://services.unixathome.org/bacula/jobs.php?name=IncrAuto-4320&action=listing
 

 no 2m
https://services.unixathome.org/bacula/jobs.php?name=IncrAuto-4324&action=listing
 

 no 2m
https://services.unixathome.org/bacula/jobs.php?name=IncrAuto-4321&action=listing
 

 yes 8m
https://services.unixathome.org/bacula/jobs.php?name=IncrAuto-4055&action=listing
 

 yes 7m
https://services.unixathome.org/bacula/jobs.php?name=IncrAuto-4322&action=listing
 

 yes 7m
https://services.unixathome.org/bacula/jobs.php?name=IncrAuto-4319&action=listing
 

 yes 7m
https://services.unixathome.org/bacula/jobs.php?name=IncrAuto-3969&action=listing
 

 yes 7m
https://services.unixathome.org/bacula/jobs.php?name=IncrAuto-3972&action=listing
 

 yes 7m
https://services.unixathome.org/bacula/jobs.php?name=IncrAuto-3973&action=listing
 

 yes 6m
https://services.unixathome.org/bacula/jobs.php?name=IncrAuto-3971&action=listing
 

 yes 7m
https://services.unixathome.org/bacula/jobs.php?name=IncrAuto-4058&action=listing
 

 no 3m

total: 69 minutes

At first, I was confused as to how there were multiple jobs on a given Volume, 
but then I realized: this job was just appending to volumes not yet at Max 
Volume Size.

> More specifically, were multiple concurrent jobs writing interleaved blocks 
> to these inc volumes at the same time job 232778 was? A copy job is 
> essentially the same thing as a restore and could be slowed by interleaving. 
> Bacula has to step through the file, finding the correct blocks to add to the 
> write buffer, whereas tar simply reads a block, writes a block. It is the 
> ideal situation for tar, and tar would not likely be so quick if it had to 
> write all of the individual files, rather than a single large volume file.

I think there is no interleaving at all. Each device has maximum of one current 
job.

Have a look at https://gist.github.com/dlangille/2341a6da8f9ee836270c 
 where I have:


Autochanger {
  Name = VirtualDisk

  Changer Device  = /dev/null
  Changer Command = /dev/null

  Device  = vDrive-0, vDrive-1, vDrive-2, vDrive-3, vDrive-4
}

Each of these Devices then specifies: Maximum Concurrent Jobs = 1

--
Dan Langille - BSDCan / PGCon
d...@langille.org




> 
> 
> On 3/9/2016 6:14 PM, Dan Langille wrote:
>> I have a copy to tape job which copies from disk to tape using Bacula 7.4.0 
>> and PostgreSQL 9.4 on FreeBSD 10.2
>> 
>> Everything is within one SD
>> 
>> Full details at https://gist.github.com/dlangille/2341a6da8f9ee836270c 
>> 
>> 
>> The job summary:
>> 
>>  Start time: 09-Mar-2016 19:41:54
>>  End time:   09-Mar-2016 20:51:02
>>  Elapsed time:   1 hour 9 mins 8 secs
>>  Priority:   410
>>  SD Files Written:   1
>>  SD Bytes Written:   52,897,660,928 (52.89 GB)
>>  Rate:   12752.6 KB/s
>> 
>> If I tar the volumes directly to tape, it takes only 16 minutes.
>> 
>> 
>> $ time sudo tar -cf /dev/nsa1 IncrAuto-4525 IncrAuto-4320 IncrAuto-4324 
>> IncrAuto-4321 \
>> > IncrAuto-4055 IncrAuto-4322 IncrAuto-4319 IncrAuto-3969 IncrAuto-3972 \
>> > IncrAuto-3973 IncrAuto-3971 IncrAuto-4058
>> 
>> real15m47.508s
>> user0m5.844s
>> sys 1m51.834s
>> 
>> Spooling attributes is trivial:
>> 
>> 09-Mar 20:51 crey-sd JobId 232945: Sending spooled attrs to the Director. 
>> Despooling 321 bytes ...
>> 09-Mar 20:51 bacula-dir JobId 232944: Bacula bacula-dir 7.4.0 (16Jan16):
>> 
>> I am not sure 

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Josh Fisher]

Was job 232778 the only job on these inc volumes? More specifically, 
were multiple concurrent jobs writing interleaved blocks to these inc 
volumes at the same time job 232778 was? A copy job is essentially the 
same thing as a restore and could be slowed by interleaving. Bacula has 
to step through the file, finding the correct blocks to add to the write 
buffer, whereas tar simply reads a block, writes a block. It is the 
ideal situation for tar, and tar would not likely be so quick if it had 
to write all of the individual files, rather than a single large volume 
file.



On 3/9/2016 6:14 PM, Dan Langille wrote:
I have a copy to tape job which copies from disk to tape using Bacula 
7.4.0 and PostgreSQL 9.4 on FreeBSD 10.2


Everything is within one SD

Full details at https://gist.github.com/dlangille/2341a6da8f9ee836270c

The job summary:

 Start time: 09-Mar-2016 19:41:54
 End time:   09-Mar-2016 20:51:02
 Elapsed time:   1 hour 9 mins 8 secs
 Priority:   410
 SD Files Written:   1
 SD Bytes Written:   52,897,660,928 (52.89 GB)
 Rate:   12752.6 KB/s

If I tar the volumes directly to tape, it takes only 16 minutes.


$ time sudo tar -cf /dev/nsa1 IncrAuto-4525 IncrAuto-4320 
IncrAuto-4324 IncrAuto-4321 \

> IncrAuto-4055 IncrAuto-4322 IncrAuto-4319 IncrAuto-3969 IncrAuto-3972 \
> IncrAuto-3973 IncrAuto-3971 IncrAuto-4058

real15m47.508s
user0m5.844s
sys 1m51.834s

Spooling attributes is trivial:

09-Mar 20:51 crey-sd JobId 232945: Sending spooled attrs to the 
Director. Despooling 321 bytes ...

09-Mar 20:51 bacula-dir JobId 232944: Bacula bacula-dir 7.4.0 (16Jan16):

I am not sure where to look to figure this out.

Full job output:


09-Mar 19:41 bacula-dir JobId 232944: Warning: FileSet MD5 digest not 
found.
09-Mar 19:41 bacula-dir JobId 232944: The following 1 JobId was chosen 
to be copied: 232778
09-Mar 19:41 bacula-dir JobId 232944: Copying using JobId=232778 
Job=BackupCatalog.2016-03-08_03.05.16_52
09-Mar 19:41 bacula-dir JobId 232944: Bootstrap records written to 
/usr/local/bacula/working/bacula-dir.restore.107.bsr
09-Mar 19:41 bacula-dir JobId 232944: Start Copying JobId 232944, 
Job=CopyToTape-Full-Just-One-tape-01b.2016-03-09_19.41.51_42

09-Mar 19:41 bacula-dir JobId 232944: Using Device "vDrive-0" to read.
09-Mar 19:41 bacula-dir JobId 232945: Using Device "LTO_0" to write.
09-Mar 19:41 crey-sd JobId 232944: Ready to read from volume 
"IncrAuto-4525" on file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 19:41 crey-sd JobId 232944: Forward spacing Volume 
"IncrAuto-4525" to file:block 0:1207281748.
09-Mar 19:47 crey-sd JobId 232944: End of Volume at file 1 on device 
"vDrive-0" (/usr/local/bacula/volumes), Volume "IncrAuto-4525"
09-Mar 19:47 crey-sd JobId 232944: Ready to read from volume 
"IncrAuto-4320" on file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 19:47 crey-sd JobId 232944: Forward spacing Volume 
"IncrAuto-4320" to file:block 0:3844390440.
09-Mar 19:49 crey-sd JobId 232944: End of Volume at file 1 on device 
"vDrive-0" (/usr/local/bacula/volumes), Volume "IncrAuto-4320"
09-Mar 19:49 crey-sd JobId 232944: Ready to read from volume 
"IncrAuto-4324" on file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 19:49 crey-sd JobId 232944: Forward spacing Volume 
"IncrAuto-4324" to file:block 0:3867148676.
09-Mar 19:51 crey-sd JobId 232944: End of Volume at file 1 on device 
"vDrive-0" (/usr/local/bacula/volumes), Volume "IncrAuto-4324"
09-Mar 19:51 crey-sd JobId 232944: Ready to read from volume 
"IncrAuto-4321" on file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 19:51 crey-sd JobId 232944: Forward spacing Volume 
"IncrAuto-4321" to file:block 0:214.
09-Mar 19:59 crey-sd JobId 232944: End of Volume at file 1 on device 
"vDrive-0" (/usr/local/bacula/volumes), Volume "IncrAuto-4321"
09-Mar 19:59 crey-sd JobId 232944: Ready to read from volume 
"IncrAuto-4055" on file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 19:59 crey-sd JobId 232944: Forward spacing Volume 
"IncrAuto-4055" to file:block 0:214.
09-Mar 20:06 crey-sd JobId 232944: End of Volume at file 1 on device 
"vDrive-0" (/usr/local/bacula/volumes), Volume "IncrAuto-4055"
09-Mar 20:06 crey-sd JobId 232944: Ready to read from volume 
"IncrAuto-4322" on file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 20:06 crey-sd JobId 232944: Forward spacing Volume 
"IncrAuto-4322" to file:block 0:214.
09-Mar 20:13 crey-sd JobId 232944: End of Volume at file 1 on device 
"vDrive-0" (/usr/local/bacula/volumes), Volume "IncrAuto-4322"
09-Mar 20:13 crey-sd JobId 232944: Ready to read from volume 
"IncrAuto-4319" on file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 20:13 crey-sd JobId 232944: Forward spacing Volume 
"IncrAuto-4319" to file:block 0:214.
09-Mar 20:20 crey-sd JobId 232944: End of Volume at file 1 on device 
"vDrive-0" (/usr/local/bacula/volumes), Volume "IncrAuto-4319"
09-Mar 20:20 crey-sd JobI

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Simon Templar]

Dan,

I’m new to bacula, and arguably not very smart, but I’ve been struggling with 
tape drive performance pretty much since the moment I got the configurations to 
a functional state so I’ll share my learnings thus far.

Can you hear your tape drive? If so, do you hear lots of stops and starts when 
bacula is running the job? When mine was writing I’d hear lots of stops and 
starts (shoe-shining, I believe it’s called) followed by long pauses of silence.

I have a single LTO6 drive, and so far I’ve made 2 changes to my setup that 
brought my average write speed from less than 60 MB/s to between 130-160 MB/s.

First, the long pauses with no activity were found to be related to spooling. I 
started by spooling to an idle set of 6 sata drives in a raidz, then later 
switched to an ssd, and with either the behavior was that the tape drive would 
be completely idle while the data was spooled, then the data was written to the 
drive, then when it all was written the tape drive was completely idle again 
while data was spooled. Needless to say this added a long time to the backups 
for our platform. Our data ultimately resides on another platform (a freenas) 
and the goal is to replicate it to our backup server, then write to tape for 
archival purposes. It turns out that I can write the 15tb directly from the nfs 
mount to tape faster than I can spool it from local disk to ssd then write to 
tape. I’m sure something’s wrong with my setup, but if you’re spooling it’s 
worth taking a look to see if you have long tape drive idle times while the 
data is spooled. I would have guessed that the spool directory should be kept 
as full as possible WHILE the spooled data is written to tape, but that’s not 
the behavior I’m seeing.

The second thing I did was add:
Maximum File Size = 20GB

to my bacula-sd.conf. I probably need to tune this further, and it might make 
restores slower (at least for individual files?) but it drastically cut down on 
my drive starts and stops. Together, the maximum file size and elimination of 
spooling more than doubled my average write speed in large backups.

-Simon

P.S. Is your tar write speed as fast as you’d expect for your LTO drive?


> On Mar 9, 2016, at 6:14 PM, Dan Langille  wrote:
> 
> I have a copy to tape job which copies from disk to tape using Bacula 7.4.0 
> and PostgreSQL 9.4 on FreeBSD 10.2
> 
> Everything is within one SD
> 
> Full details at https://gist.github.com/dlangille/2341a6da8f9ee836270c 
> 
> 
> The job summary:
> 
>  Start time: 09-Mar-2016 19:41:54
>  End time:   09-Mar-2016 20:51:02
>  Elapsed time:   1 hour 9 mins 8 secs
>  Priority:   410
>  SD Files Written:   1
>  SD Bytes Written:   52,897,660,928 (52.89 GB)
>  Rate:   12752.6 KB/s
> 
> If I tar the volumes directly to tape, it takes only 16 minutes.
> 
> 
> $ time sudo tar -cf /dev/nsa1 IncrAuto-4525 IncrAuto-4320 IncrAuto-4324 
> IncrAuto-4321 \
> > IncrAuto-4055 IncrAuto-4322 IncrAuto-4319 IncrAuto-3969 IncrAuto-3972 \
> > IncrAuto-3973 IncrAuto-3971 IncrAuto-4058
> 
> real15m47.508s
> user0m5.844s
> sys 1m51.834s
> 
> Spooling attributes is trivial:
> 
> 09-Mar 20:51 crey-sd JobId 232945: Sending spooled attrs to the Director. 
> Despooling 321 bytes ...
> 09-Mar 20:51 bacula-dir JobId 232944: Bacula bacula-dir 7.4.0 (16Jan16):
> 
> I am not sure where to look to figure this out.
> 
> Full job output:
> 
> 
> 09-Mar 19:41 bacula-dir JobId 232944: Warning: FileSet MD5 digest not found.
> 09-Mar 19:41 bacula-dir JobId 232944: The following 1 JobId was chosen to be 
> copied: 232778
> 09-Mar 19:41 bacula-dir JobId 232944: Copying using JobId=232778 
> Job=BackupCatalog.2016-03-08_03.05.16_52
> 09-Mar 19:41 bacula-dir JobId 232944: Bootstrap records written to 
> /usr/local/bacula/working/bacula-dir.restore.107.bsr
> 09-Mar 19:41 bacula-dir JobId 232944: Start Copying JobId 232944, 
> Job=CopyToTape-Full-Just-One-tape-01b.2016-03-09_19.41.51_42
> 09-Mar 19:41 bacula-dir JobId 232944: Using Device "vDrive-0" to read.
> 09-Mar 19:41 bacula-dir JobId 232945: Using Device "LTO_0" to write.
> 09-Mar 19:41 crey-sd JobId 232944: Ready to read from volume "IncrAuto-4525" 
> on file device "vDrive-0" (/usr/local/bacula/volumes).
> 09-Mar 19:41 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-4525" to 
> file:block 0:1207281748.
> 09-Mar 19:47 crey-sd JobId 232944: End of Volume at file 1 on device 
> "vDrive-0" (/usr/local/bacula/volumes), Volume "IncrAuto-4525"
> 09-Mar 19:47 crey-sd JobId 232944: Ready to read from volume "IncrAuto-4320" 
> on file device "vDrive-0" (/usr/local/bacula/volumes).
> 09-Mar 19:47 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-4320" to 
> file:block 0:3844390440.
> 09-Mar 19:49 crey-sd JobId 232944: End of Volume at file 1 on device 
> "vDrive-0" (/usr/local/bacula/volumes), Volume "IncrAuto-4320"
> 09-Mar 19:49 crey-sd JobId 232944:

Re: [Bacula-users] Copy disk to tape is 4x slower than tar

[Heitor Faria]

> I have a copy to tape job which copies from disk to tape using Bacula 7.4.0 
> and
> PostgreSQL 9.4 on FreeBSD 10.2

> Everything is within one SD

> Full details at https://gist.github.com/dlangille/2341a6da8f9ee836270c

> The job summary:

> Start time: 09-Mar-2016 19:41:54
> End time: 09-Mar-2016 20:51:02
> Elapsed time: 1 hour 9 mins 8 secs
> Priority: 410
> SD Files Written: 1
> SD Bytes Written: 52,897,660,928 (52.89 GB)
> Rate: 12752.6 KB/s

> If I tar the volumes directly to tape, it takes only 16 minutes.

> $ time sudo tar -cf /dev/nsa1 IncrAuto-4525 IncrAuto-4320 IncrAuto-4324
> IncrAuto-4321 \
> > IncrAuto-4055 IncrAuto-4322 IncrAuto-4319 IncrAuto-3969 IncrAuto-3972 \
> > IncrAuto-3973 IncrAuto-3971 IncrAuto-4058

> real 15m47.508s
> user 0m5.844s
> sys 1m51.834s

> Spooling attributes is trivial:

> 09-Mar 20:51 crey-sd JobId 232945: Sending spooled attrs to the Director.
> Despooling 321 bytes ...
> 09-Mar 20:51 bacula-dir JobId 232944: Bacula bacula-dir 7.4.0 (16Jan16):

> I am not sure where to look to figure this out.

Hello, Dan: maybe there is nothing to figure it out. Packing volumes with tar 
directly to tapes makes you unable to restore a single file or even a single 
job (with bootstrap) from an entire tape. I think it's a trade-off. 

> --
> Dan Langille - BSDCan / PGCon
> d...@langille.org

Regards, 
-- 
=== 
Heitor Medrado de Faria - LPIC-III | ITIL-F | Bacula Systems Certified 
Administrator II 
Próximas aulas telepresencial ao-vivo - 15 de fevereiro: 
http://www.bacula.com.br/agenda/ 
Ministro treinamento e implementação in-company Bacula: 
http://www.bacula.com.br/in-company/ 
Ou assista minhas videoaulas on-line: 
http://www.bacula.com.br/treinamento-bacula-ed/ 
61 8268-4220 
Site: www.bacula.com.br | Facebook: heitor.faria 
 
--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785111&iu=/4140___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Copy disk to tape is 4x slower than tar

[Dan Langille]

I have a copy to tape job which copies from disk to tape using Bacula 7.4.0 and 
PostgreSQL 9.4 on FreeBSD 10.2

Everything is within one SD

Full details at https://gist.github.com/dlangille/2341a6da8f9ee836270c 


The job summary:

 Start time: 09-Mar-2016 19:41:54
 End time:   09-Mar-2016 20:51:02
 Elapsed time:   1 hour 9 mins 8 secs
 Priority:   410
 SD Files Written:   1
 SD Bytes Written:   52,897,660,928 (52.89 GB)
 Rate:   12752.6 KB/s

If I tar the volumes directly to tape, it takes only 16 minutes.


$ time sudo tar -cf /dev/nsa1 IncrAuto-4525 IncrAuto-4320 IncrAuto-4324 
IncrAuto-4321 \
> IncrAuto-4055 IncrAuto-4322 IncrAuto-4319 IncrAuto-3969 IncrAuto-3972 \
> IncrAuto-3973 IncrAuto-3971 IncrAuto-4058

real15m47.508s
user0m5.844s
sys 1m51.834s

Spooling attributes is trivial:

09-Mar 20:51 crey-sd JobId 232945: Sending spooled attrs to the Director. 
Despooling 321 bytes ...
09-Mar 20:51 bacula-dir JobId 232944: Bacula bacula-dir 7.4.0 (16Jan16):

I am not sure where to look to figure this out.

Full job output:


09-Mar 19:41 bacula-dir JobId 232944: Warning: FileSet MD5 digest not found.
09-Mar 19:41 bacula-dir JobId 232944: The following 1 JobId was chosen to be 
copied: 232778
09-Mar 19:41 bacula-dir JobId 232944: Copying using JobId=232778 
Job=BackupCatalog.2016-03-08_03.05.16_52
09-Mar 19:41 bacula-dir JobId 232944: Bootstrap records written to 
/usr/local/bacula/working/bacula-dir.restore.107.bsr
09-Mar 19:41 bacula-dir JobId 232944: Start Copying JobId 232944, 
Job=CopyToTape-Full-Just-One-tape-01b.2016-03-09_19.41.51_42
09-Mar 19:41 bacula-dir JobId 232944: Using Device "vDrive-0" to read.
09-Mar 19:41 bacula-dir JobId 232945: Using Device "LTO_0" to write.
09-Mar 19:41 crey-sd JobId 232944: Ready to read from volume "IncrAuto-4525" on 
file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 19:41 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-4525" to 
file:block 0:1207281748.
09-Mar 19:47 crey-sd JobId 232944: End of Volume at file 1 on device "vDrive-0" 
(/usr/local/bacula/volumes), Volume "IncrAuto-4525"
09-Mar 19:47 crey-sd JobId 232944: Ready to read from volume "IncrAuto-4320" on 
file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 19:47 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-4320" to 
file:block 0:3844390440.
09-Mar 19:49 crey-sd JobId 232944: End of Volume at file 1 on device "vDrive-0" 
(/usr/local/bacula/volumes), Volume "IncrAuto-4320"
09-Mar 19:49 crey-sd JobId 232944: Ready to read from volume "IncrAuto-4324" on 
file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 19:49 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-4324" to 
file:block 0:3867148676.
09-Mar 19:51 crey-sd JobId 232944: End of Volume at file 1 on device "vDrive-0" 
(/usr/local/bacula/volumes), Volume "IncrAuto-4324"
09-Mar 19:51 crey-sd JobId 232944: Ready to read from volume "IncrAuto-4321" on 
file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 19:51 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-4321" to 
file:block 0:214.
09-Mar 19:59 crey-sd JobId 232944: End of Volume at file 1 on device "vDrive-0" 
(/usr/local/bacula/volumes), Volume "IncrAuto-4321"
09-Mar 19:59 crey-sd JobId 232944: Ready to read from volume "IncrAuto-4055" on 
file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 19:59 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-4055" to 
file:block 0:214.
09-Mar 20:06 crey-sd JobId 232944: End of Volume at file 1 on device "vDrive-0" 
(/usr/local/bacula/volumes), Volume "IncrAuto-4055"
09-Mar 20:06 crey-sd JobId 232944: Ready to read from volume "IncrAuto-4322" on 
file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 20:06 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-4322" to 
file:block 0:214.
09-Mar 20:13 crey-sd JobId 232944: End of Volume at file 1 on device "vDrive-0" 
(/usr/local/bacula/volumes), Volume "IncrAuto-4322"
09-Mar 20:13 crey-sd JobId 232944: Ready to read from volume "IncrAuto-4319" on 
file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 20:13 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-4319" to 
file:block 0:214.
09-Mar 20:20 crey-sd JobId 232944: End of Volume at file 1 on device "vDrive-0" 
(/usr/local/bacula/volumes), Volume "IncrAuto-4319"
09-Mar 20:20 crey-sd JobId 232944: Ready to read from volume "IncrAuto-3969" on 
file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 20:20 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-3969" to 
file:block 0:218.
09-Mar 20:27 crey-sd JobId 232944: End of Volume at file 1 on device "vDrive-0" 
(/usr/local/bacula/volumes), Volume "IncrAuto-3969"
09-Mar 20:27 crey-sd JobId 232944: Ready to read from volume "IncrAuto-3972" on 
file device "vDrive-0" (/usr/local/bacula/volumes).
09-Mar 20:27 crey-sd JobId 232944: Forward spacing Volume "IncrAuto-3972" to 
file:block 0:218.
09-Mar 2