Re: [Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

Mandi! Josh Fisher via Bacula-users
  In chel di` si favelave...

> Another way that might be better for your case is to leave 
> MaximumSpoolSize = 0 (unlimited) and specify a MaximumJobSpoolSize in 
> the Device resource instead. The difference is that when one job reaches 
> the MaximumJobSpoolSize it will begin writing its spool file to tape, 
> but it will not cause the other job(s) to pause until they also reach 
> the MaximumJobSpoolSize. Then by starting one job a few minutes after 
> the other it should be possible to mostly avoid both jobs pausing 
> (de-spooling) at the same time.

Bingo! Thanks!

-- 




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Josh Fisher via Bacula-users]

On 6/24/24 11:04, Marco Gaiarin wrote:

Mandi! Josh Fisher via Bacula-users
   In chel di` si favelave...


Except when the MaximumSpoolSize for the Device resource is reached or
the Spool Directory becomes full. When there is no more storage space
for data spool files, all jobs writing to that device are paused and the
spool files for each of those jobs are written to the device, one at a
time. I'm not sure if every job's spool file is written to tape at that
time, or if the forced despooling stops once some threshold for
sufficient space has been freed up. So, more accurately, other jobs
continue spooling IFF there is sufficient space in the spool directory.

Ok, doing some more test in real condition (and still arguing on why logs
row are not ordered by date...) i've splitted a job:

...


Sicerily, i hoped there's some more 'smart' way to manage despooling in
bacula; the only way to boost performance seems to me tune MaximumSpoolSize
so tapes it takes more or less the same time to despool of the other
tasks(to) to spool, so dead time interleaved can be minimized.

Seems an hard task. ;-)



When you set MaximumSpoolSize in the Device resource, that sets the 
maximum storage available for all spool files, not for each spool file. 
When that is reached, it means there is no more space for any job's 
spool file, so they all must pause and write their spool files to tape.


Another way that might be better for your case is to leave 
MaximumSpoolSize = 0 (unlimited) and specify a MaximumJobSpoolSize in 
the Device resource instead. The difference is that when one job reaches 
the MaximumJobSpoolSize it will begin writing its spool file to tape, 
but it will not cause the other job(s) to pause until they also reach 
the MaximumJobSpoolSize. Then by starting one job a few minutes after 
the other it should be possible to mostly avoid both jobs pausing 
(de-spooling) at the same time.


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

Mandi! Josh Fisher via Bacula-users
  In chel di` si favelave...

> Except when the MaximumSpoolSize for the Device resource is reached or 
> the Spool Directory becomes full. When there is no more storage space 
> for data spool files, all jobs writing to that device are paused and the 
> spool files for each of those jobs are written to the device, one at a 
> time. I'm not sure if every job's spool file is written to tape at that 
> time, or if the forced despooling stops once some threshold for 
> sufficient space has been freed up. So, more accurately, other jobs 
> continue spooling IFF there is sufficient space in the spool directory.

Ok, doing some more test in real condition (and still arguing on why logs
row are not ordered by date...) i've splitted a job:

 root@lnfbacula:~# cat /var/log/bacula/bacula.log | grep '2251[67]' | sort
 21-Jun 20:00 cnpve3-sd JobId 22516: Volume "AAJ664L9" previously written, 
moving to end of data.
 21-Jun 20:00 lnfbacula-dir JobId 22516: Using Device "LTO9Storage0" to write.
 21-Jun 20:00 lnfbacula-dir JobId 22517: Start Backup JobId 22517, 
Job=VEN-CN-CNPVE3Sedi.2024-06-21_20.00.00_32
 21-Jun 20:00 lnfbacula-dir JobId 22517: Using Device "LTO9Storage0" to write.
 21-Jun 20:02 cnpve3-sd JobId 22516: Ready to append to end of Volume 
"AAJ664L9" at file=8.
 21-Jun 20:02 cnpve3-sd JobId 22516: Spooling data ...
 21-Jun 20:02 cnpve3-sd JobId 22517: Spooling data ...

Clearly job use the same schedule, so start at the same time.

 21-Jun 20:48 cnpve3-sd JobId 22516: User specified Device spool size reached: 
DevSpoolSize=257,699,937,020 MaxDevSpoolSize=257,698,037,760
 21-Jun 20:48 cnpve3-sd JobId 22516: Writing spooled data to Volume. Despooling 
144,990,358,762 bytes ...
 21-Jun 20:48 cnpve3-sd JobId 22517: User specified Device spool size reached: 
DevSpoolSize=257,699,937,020 MaxDevSpoolSize=257,698,037,760
 21-Jun 20:48 cnpve3-sd JobId 22517: Writing spooled data to Volume. Despooling 
112,709,578,258 bytes ...

OK; MaximumSpoolSize reached, both jobs stop.

 21-Jun 20:56 cnpve3-sd JobId 22517: Despooling elapsed time = 00:08:50, 
Transfer rate = 212.6 M Bytes/second
 21-Jun 20:56 cnpve3-sd JobId 22517: Spooling data again ...

One of the job start and complete spooling; when end the spooling, start
immediately to despool again.

 21-Jun 21:08 cnpve3-sd JobId 22516: Despooling elapsed time = 00:11:59, 
Transfer rate = 201.6 M Bytes/second
 21-Jun 21:08 cnpve3-sd JobId 22516: Spooling data again ...

The other, after the former end despooling, start the despooling.

So, for job 22517 the spooling start at 20:56, while for job 22516 at 21:08;
so for 12 minutes i've effectively TWO jobs running, one despooling and one
spooling...

 21-Jun 22:05 cnpve3-sd JobId 22516: User specified Device spool size reached: 
DevSpoolSize=257,698,889,378 MaxDevSpoolSize=257,698,037,760
 21-Jun 22:05 cnpve3-sd JobId 22516: Writing spooled data to Volume. Despooling 
128,134,307,647 bytes ...
 21-Jun 22:05 cnpve3-sd JobId 22517: User specified Device spool size reached: 
DevSpoolSize=257,699,937,966 MaxDevSpoolSize=257,698,037,760
 21-Jun 22:05 cnpve3-sd JobId 22517: Writing spooled data to Volume. Despooling 
129,565,630,319 bytes ...
 21-Jun 22:14 cnpve3-sd JobId 22516: Despooling elapsed time = 00:09:27, 
Transfer rate = 225.9 M Bytes/second
 21-Jun 22:14 cnpve3-sd JobId 22516: Spooling data again ...
 21-Jun 22:24 cnpve3-sd JobId 22517: Despooling elapsed time = 00:10:08, 
Transfer rate = 213.1 M Bytes/second
 21-Jun 22:24 cnpve3-sd JobId 22517: Spooling data again ...

 21-Jun 23:07 cnpve3-sd JobId 22516: User specified Device spool size reached: 
DevSpoolSize=257,698,889,378 MaxDevSpoolSize=257,698,037,760
 21-Jun 23:07 cnpve3-sd JobId 22516: Writing spooled data to Volume. Despooling 
161,555,952,974 bytes ...
 21-Jun 23:07 cnpve3-sd JobId 22517: User specified Device spool size reached: 
DevSpoolSize=257,699,937,966 MaxDevSpoolSize=257,698,037,760
 21-Jun 23:07 cnpve3-sd JobId 22517: Writing spooled data to Volume. Despooling 
96,143,984,992 bytes ...
 21-Jun 23:17 cnpve3-sd JobId 22516: Despooling elapsed time = 00:09:50, 
Transfer rate = 273.8 M Bytes/second
 21-Jun 23:17 cnpve3-sd JobId 22516: Spooling data again ...
 21-Jun 23:28 cnpve3-sd JobId 22517: Despooling elapsed time = 00:11:13, 
Transfer rate = 142.8 M Bytes/second
 21-Jun 23:28 cnpve3-sd JobId 22517: Spooling data again ...

the same schema repeated some times.

at some time one of the two jobs end, and so the other continue.

 22-Jun 18:51 cnpve3-sd JobId 22516: User specified Device spool size reached: 
DevSpoolSize=257,698,373,454 MaxDevSpoolSize=257,698,037,760
 22-Jun 18:51 cnpve3-sd JobId 22516: Writing spooled data to Volume. Despooling 
114,582,356,345 bytes ...
 22-Jun 18:58 cnpve3-sd JobId 22517: Despooling elapsed time = 00:09:51, 
Transfer rate = 242.1 M Bytes/second
 22-Jun 18:58 cnpve3-sd JobId 22517: Elapsed time=22:56:52, Transfer rate=26.32 
M Bytes/second
 22-Jun 18:58 cnpve3-sd JobId 22517: 

Re: [Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

Mandi! Bill Arlofski via Bacula-users
  In chel di` si favelave...

>> But, now, a question: this mean that in spool data get interleaved too? How
>> they are interleaved? File by file? Block by block? What block size?
> No. When you have jobs running, take a look into the SpoolDirectory. You will 
> see a 'data' *.spool file and an 'attr' *.sppol 
> file for each job running.

>> There's no way to 'violate' this behaviour, right?! A single SD process
>> cannot spool and despool at the same time?
> An SD can be spooling multiple jobs wile *one* and only one Job spool file is 
> despooling to one drive.
> Add another drive and and the same is still true, but the SD can now be 
> despooling two jobs at the same time while other jobs 
> are spooling, and so on as you add drives.

Reading:
https://www.bacula.org/11.0.x-manuals/en/main/Data_Spooling.html

I read (on top):

While the spooled data is being written to the tape, the despooling 
process has exclusive use of the tape. This means that you can spool multiple 
simultaneous jobs to disk, then have them very efficiently despooled one at a 
time without having the data blocks from several jobs intermingled, thus 
substantially improving the time needed to restore files. While despooling, all 
jobs spooling continue running. 

and (on bottom):

It is probably best to provide as large a spool file as possible to 
avoid repeatedly spooling/despooling. Also, while a job is despooling to tape, 
the File daemon must wait (i.e. spooling stops for the job while it is 
despooling).
If you are running multiple simultaneous jobs, Bacula will continue 
spooling other jobs while one is despooling to tape, provided there is 
sufficient spool file space. 

So, trying to understand your answer:

1) bacula spool different jobs in different files on spool dir

2) when 'Maximum Spool Size' reached, bacula write to tape different
 spoolfiles one at a time, not 'mixing' them.

3) when bacula are despooling JobA, spooling of other JobX files continue;
 only JobA stop spooling files.

So seems to me that (clearly on a decent figure...) it is better to split
jobs then not...


Right?

-- 




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Josh Fisher via Bacula-users]

On 6/20/24 18:58, Bill Arlofski via Bacula-users wrote:

On 6/20/24 8:58 AM, Marco Gaiarin wrote:

Once that is hit, the spoofles are written to tape, during which active
jobs have to wait because the spool is full.

There's no way to 'violate' this behaviour, right?! A single SD process
cannot spool and despool at the same time?


An SD can be spooling multiple jobs wile *one* and only one Job spool 
file is despooling to one drive.


Add another drive and and the same is still true, but the SD can now 
be despooling two jobs at the same time while other jobs are spooling, 
and so on as you add drives.




Except when the MaximumSpoolSize for the Device resource is reached or 
the Spool Directory becomes full. When there is no more storage space 
for data spool files, all jobs writing to that device are paused and the 
spool files for each of those jobs are written to the device, one at a 
time. I'm not sure if every job's spool file is written to tape at that 
time, or if the forced despooling stops once some threshold for 
sufficient space has been freed up. So, more accurately, other jobs 
continue spooling IFF there is sufficient space in the spool directory.




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Bill Arlofski via Bacula-users]

On 6/20/24 8:58 AM, Marco Gaiarin wrote:


But, now, a question: this mean that in spool data get interleaved too? How
they are interleaved? File by file? Block by block? What block size?


No. When you have jobs running, take a look into the SpoolDirectory. You will see a 'data' *.spool file and an 'attr' *.sppol 
file for each job running.




Once that is hit, the spoofles are written to tape, during which active
jobs have to wait because the spool is full.


There's no way to 'violate' this behaviour, right?! A single SD process
cannot spool and despool at the same time?


An SD can be spooling multiple jobs wile *one* and only one Job spool file is 
despooling to one drive.

Add another drive and and the same is still true, but the SD can now be despooling two jobs at the same time while other jobs 
are spooling, and so on as you add drives.



Hope this helps,
Bill

--
Bill Arlofski
w...@protonmail.com



signature.asc
Description: OpenPGP digital signature
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

Mandi! Gary R. Schmidt
  In chel di` si favelave...

>> job involved (in the same pool, i think) start to write down spool to tape.
> MaximumSpoolSize is the total space used in the spool area, by all jobs.

After posting, i've looked more carefully at the log and understod that.
Sorry for the misunderstanding.

This als make sense, because 'MaximumSpoolSize' is an SD parameters, and so
cannot be 'job dependant'...


But, now, a question: this mean that in spool data get interleaved too? How
they are interleaved? File by file? Block by block? What block size?

What defined in 'Maximum File Size'? Seems not to me...


> Once that is hit, the spoofles are written to tape, during which active 
> jobs have to wait because the spool is full.

There's no way to 'violate' this behaviour, right?! A single SD process
cannot spool and despool at the same time?

This seems a bit strange to me...


Thanks.

-- 




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Gary R. Schmidt]

On 17/06/2024 17:45, Marco Gaiarin wrote:
[SNIP]
> > So, literaly, if one of the job fill the 'MaximumSpoolSize' buffer, 
*ALL*

job involved (in the same pool, i think) start to write down spool to tape.


MaximumSpoolSize is the total space used in the spool area, by all jobs.

Once that is hit, the spoofles are written to tape, during which active 
jobs have to wait because the spool is full.



But this does not interleave nothing! There's some way to have job 22200
spooling while job 22201 are despooling? What are the gain of having
synchronized despooling?
Sounds good, maybe you should write it, or fund someone to write it.


Cheers,
GaryB-)


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

Mandi! Bill Arlofski via Bacula-users
  In chel di` si favelave...

> With DataSpooling enabled in all jobs, the only "interleaving" that you will 
> have on your tapes is one big block of Job 1's 
> de-spooled data, then maybe another Job 1 block, or a Job 2 block, or a Job 3 
> block, and so on, depending on which Job's 
> DataSpool file reached the defined maximum Job Spool size at particular times 
> throughout the backup window, or when one hits 
> the total MaximumSpool size and begins de-spooling.

OK, now with log in hand i can understood better what you are meaning:

 root@lnfbacula:~# cat /var/log/bacula/bacula.log | grep '2220[01]' | grep 
Despooling | sort 
 14-Jun 22:01 bpbkplom-sd JobId 22200: Writing spooled data to Volume. 
Despooling 37,410,474,034 bytes ...
 14-Jun 22:01 bpbkplom-sd JobId 22201: Writing spooled data to Volume. 
Despooling 220,289,463,963 bytes ...
 14-Jun 22:26 bpbkplom-sd JobId 22201: Despooling elapsed time = 00:25:42, 
Transfer rate = 142.8 M Bytes/second
 14-Jun 22:30 bpbkplom-sd JobId 22200: Despooling elapsed time = 00:03:55, 
Transfer rate = 159.1 M Bytes/second
 15-Jun 00:40 bpbkplom-sd JobId 22200: Writing spooled data to Volume. 
Despooling 32,627,864,192 bytes ...
 15-Jun 00:40 bpbkplom-sd JobId 22201: Writing spooled data to Volume. 
Despooling 225,072,073,743 bytes ...
 15-Jun 00:42 bpbkplom-sd JobId 22200: Despooling elapsed time = 00:02:23, 
Transfer rate = 228.1 M Bytes/second
 15-Jun 00:55 bpbkplom-sd JobId 22201: Despooling elapsed time = 00:13:24, 
Transfer rate = 279.9 M Bytes/second
 15-Jun 01:34 bpbkplom-sd JobId 22200: Writing spooled data to Volume. 
Despooling 31,032,961,822 bytes ...
 15-Jun 01:34 bpbkplom-sd JobId 22201: Writing spooled data to Volume. 
Despooling 226,666,976,157 bytes ...
 15-Jun 01:49 bpbkplom-sd JobId 22201: Despooling elapsed time = 00:14:21, 
Transfer rate = 263.2 M Bytes/second
 15-Jun 01:51 bpbkplom-sd JobId 22200: Despooling elapsed time = 00:02:42, 
Transfer rate = 191.5 M Bytes/second

So, literaly, if one of the job fill the 'MaximumSpoolSize' buffer, *ALL*
job involved (in the same pool, i think) start to write down spool to tape.

But this does not interleave nothing! There's some way to have job 22200
spooling while job 22201 are despooling? What are the gain of having
synchronized despooling?


Also, a little curiosity... if i check the logs, they are not 'ordered by
date'; job start:

 root@lnfbacula:~# cat /var/log/bacula/bacula.log | grep '2220[01]'
 14-Jun 20:00 lnfbacula-dir JobId 22200: Start Backup JobId 22200, 
Job=LOM-BP-BPBKPLOM.2024-06-14_20.00.00_55
 14-Jun 20:00 lnfbacula-dir JobId 22200: Using Device "LTO9Storage0" to write.
 14-Jun 20:00 bpbkplom-sd JobId 22200: Volume "AAJ618L9" previously written, 
moving to end of data.
 14-Jun 20:01 bpbkplom-sd JobId 22200: Ready to append to end of Volume 
"AAJ618L9" at file=58.
 14-Jun 20:01 bpbkplom-sd JobId 22200: Spooling data ...
 14-Jun 22:01 bpbkplom-sd JobId 22200: User specified Device spool size 
reached: DevSpoolSize=257,699,937,997 MaxDevSpoolSize=257,698,037,760
 14-Jun 22:01 bpbkplom-sd JobId 22200: Writing spooled data to Volume. 
Despooling 37,410,474,034 bytes ...
 14-Jun 22:30 bpbkplom-sd JobId 22200: Despooling elapsed time = 00:03:55, 
Transfer rate = 159.1 M Bytes/second
 14-Jun 22:30 bpbkplom-sd JobId 22200: Spooling data again ...
 [...]

after some time:

 15-Jun 08:53 bpbkplom-sd JobId 22200: Despooling elaps14-Jun 20:00 
lnfbacula-dir JobId 22201: Start Backup JobId 22201, 
Job=LOM-BP-BPBKPLOMSEDI.2024-06-14_20.00.00_56
 14-Jun 20:00 lnfbacula-dir JobId 22201: Using Device "LTO9Storage0" to write.
 14-Jun 20:01 bpbkplom-sd JobId 22201: Spooling data ...
 14-Jun 22:01 bpbkplom-sd JobId 22201: User specified Device spool size 
reached: DevSpoolSize=257,698,889,409 MaxDevSpoolSize=257,698,037,760
 14-Jun 22:01 bpbkplom-sd JobId 22201: Writing spooled data to Volume. 
Despooling 220,289,463,963 bytes ...
 14-Jun 22:26 bpbkplom-sd JobId 22201: Despooling elapsed time = 00:25:42, 
Transfer rate = 142.8 M Bytes/second
 14-Jun 22:26 bpbkplom-sd JobId 22201: Spooling data again ...

but the dates in logs goes back.


Boh, i'm only curious...

-- 




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

Mandi! Bill Arlofski via Bacula-users
  In chel di` si favelave...

> Hope this helps!

Thanks to all for the hints and the explainings; bacula is really a bad
beast... there's ever room for improvement! ;-)

-- 




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Josh Fisher via Bacula-users]

On 6/13/24 08:13, Gary R. Schmidt wrote:

On 13/06/2024 20:12, Stefan G. Weichinger wrote:


interested as well, I need to speedup my weekly/monthly FULL runs 
(with LTO6, though: way slower anyway).


Shouldn't the file daemon do multiple jobs in parallel?

To tape you can only write ONE stream of data.

To the spooling disk there could be more than one stream.


Yes, that seems wrong:
$ grep Concurrent *.conf
bacula-dir.conf:  Maximum Concurrent Jobs = 50
bacula-dir.conf:  Maximum Concurrent Jobs = 50
bacula-fd.conf:  Maximum Concurrent Jobs = 50
bacula-sd.conf:  Maximum Concurrent Jobs = 50


Sorry, I still don't understand what to adjust ;-)

that interleaving to tape sounds dangerous to me.


That's how Bacula works - and has since day one.

We've been using it like that since 2009, starting with an LTO-4 
autoloader, currently using an LTO-6, and I'm about to start agitating 
to upgrade to LTO-9.


Interleaving is not really an issue when data spooling is enabled. Data 
is despooled to tape one job at a time. Only when the spool size is too 
small will there be any interleaving. Even then, the interleaving will 
be a whole bunch of one job's blocks followed by a whole bunch of 
another. I't not a problem, and with sufficient disk space for the 
spool, it doesn't even happen.




What I want to have: the fd(s) should be able to dump backups to the 
spooling directory WHILE in parallel the sd spools previous backup 
jobs from spooling directory to tape (assuming I have only one tape 
drive, which is the case)


Bacula does not work that way.  No doubt if you tried really hard with 
priority and concurrency and pools you could maybe make it work like 
that, but just RTFM and use it as designed.


Why not? According to 
https://www.bacula.org/15.0.x-manuals/en/main/Data_Spooling.html it 
works exactly that way already. Most importantly, concurrent jobs 
continue to spool while one job is despooling to tape. Only one job is 
ever despooling at a given time.


On the other hand, the job that is despooling has exclusive control of 
the tape drive. On the last despool for a job (there ma be more than one 
if the job data exceeds the maximum spool size), the job has to also 
despool the job's spooled attributes to the catalog database before it 
releases the tape drive. Thus, even when other concurrent jobs are 
waiting to be despooled, the tape drive will be idle (or at least 
writing from its internal buffer) while the database writes occur. This 
is one of the reasons that database performance is so important in 
Bacula. I believe that the attributes are despooled before releasing the 
tape drive in order to ensure that despooling of both data and 
attributes is an atomic operation at job completion, probably to avoid 
race conditions.






-> parallelizing things more
It all seems quite parallel to me.


Cheers,
    Gary    B-)


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Gary R. Schmidt]

On 13/06/2024 20:12, Stefan G. Weichinger wrote:


interested as well, I need to speedup my weekly/monthly FULL runs 
(with LTO6, though: way slower anyway).


Shouldn't the file daemon do multiple jobs in parallel?

To tape you can only write ONE stream of data.

To the spooling disk there could be more than one stream.


Yes, that seems wrong:
$ grep Concurrent *.conf
bacula-dir.conf:  Maximum Concurrent Jobs = 50
bacula-dir.conf:  Maximum Concurrent Jobs = 50
bacula-fd.conf:  Maximum Concurrent Jobs = 50
bacula-sd.conf:  Maximum Concurrent Jobs = 50


Sorry, I still don't understand what to adjust ;-)

that interleaving to tape sounds dangerous to me.


That's how Bacula works - and has since day one.

We've been using it like that since 2009, starting with an LTO-4 
autoloader, currently using an LTO-6, and I'm about to start agitating 
to upgrade to LTO-9.


What I want to have: the fd(s) should be able to dump backups to the 
spooling directory WHILE in parallel the sd spools previous backup jobs 
from spooling directory to tape (assuming I have only one tape drive, 
which is the case)


Bacula does not work that way.  No doubt if you tried really hard with 
priority and concurrency and pools you could maybe make it work like 
that, but just RTFM and use it as designed.



-> parallelizing things more
It all seems quite parallel to me.


Cheers,
GaryB-)


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Stefan G. Weichinger]

interested as well, I need to speedup my weekly/monthly FULL runs 
(with LTO6, though: way slower anyway).


Shouldn't the file daemon do multiple jobs in parallel?

To tape you can only write ONE stream of data.

To the spooling disk there could be more than one stream.


Yes, that seems wrong:
$ grep Concurrent *.conf
bacula-dir.conf:  Maximum Concurrent Jobs = 50
bacula-dir.conf:  Maximum Concurrent Jobs = 50
bacula-fd.conf:  Maximum Concurrent Jobs = 50
bacula-sd.conf:  Maximum Concurrent Jobs = 50


Sorry, I still don't understand what to adjust ;-)

that interleaving to tape sounds dangerous to me.

What I want to have: the fd(s) should be able to dump backups to the 
spooling directory WHILE in parallel the sd spools previous backup jobs 
from spooling directory to tape (assuming I have only one tape drive, 
which is the case)


-> parallelizing things more

thanks!





___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Bill Arlofski via Bacula-users]

On 6/11/24 10:45 AM, Marco Gaiarin wrote:


Sorry, i really don't understand and i need feedback...

I've read many time that tapes are handled better as they are, sequential
media; so they need on storage:

Maximum Concurrent Jobs = 1


Hello Marco,

If you are using DataSpooling for all of your jobs, this setting is somewhat redundant because Bacula will de-spool exactly 
one Job's Data Spool file at a time.


With DataSpooling enabled in all jobs, the only "interleaving" that you will have on your tapes is one big block of Job 1's 
de-spooled data, then maybe another Job 1 block, or a Job 2 block, or a Job 3 block, and so on, depending on which Job's 
DataSpool file reached the defined maximum Job Spool size at particular times throughout the backup window, or when one hits 
the total MaximumSpool size and begins de-spooling.


If, on the other hand, you have many clients and enough network bandwidth, you can disable Data Spooling, and increase the 
Ta

pe Drive's MaximumConcurrentJobs setting and Bacula will stream and interleave 
the data from all the concurrently running jobs.

But, you can probably never really guarantee that all jobs will be streaming enough data concurrently to saturate the link to 
the Tape Drive, so using DataSpooling to *fast*, local, SSD, flash, or NVMe etc drives is probably a better and more 
consistent solution.



Hope this helps!
Bill

--
Bill Arlofski
w...@protonmail.com



signature.asc
Description: OpenPGP digital signature
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

>> Not for a single job. When the storage daemon is writing a job's spooled 
>> data to tape, the client must wait. However, if multiple jobs are 
>> running in parallel, then the other jobs will continue to spool their 
>> data while one job is despooling to tape.
> 
> I come back on this. I've setup on director's storage Autochanger{}
> configuration:
> Maximum Concurrent Jobs = 5
> but i've set on storage configuration, in the Device{} stanza:
> Maximum Concurrent Jobs = 1
> as suggested for tapes; clearly i've only one device/tape. I've defined two
> jobs, but the first run, the second:
> is waiting on Storage "BPBKPLOMAutochanger"
> And don't start to spool data, wait there indefinetively.

Sorry, i really don't understand and i need feedback...

I've read many time that tapes are handled better as they are, sequential
media; so they need on storage:

Maximum Concurrent Jobs = 1

but in this way, if i need spooling (and *I* need spooling for tapes!) i can
run on tape jobs only sequentially: eg:
  - spooling job1 -> despooling job1 (repeated many times)
  - spooling job2 -> despooling job2 (repeated many times)
  - ...

A bright suggestion is to define different jobs and so 'interleave' that,
so when job1 is despooling, job2 can spool data.

But if set on storage 'Maximum Concurrent Jobs = 1', the storage do exactly
one job at a time. job2 wait the end of job1. Period.


I can set 'Maximum Concurrent Jobs > 1'? What i can expect from the SD? Will
despool job1 and then despool job2, or i can expect to have also the
despooling of job1 and job2 'interleaved', that i suppose is a bad thing?


Sorry, thanks.

-- 




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Gary R. Schmidt]

On 11/06/2024 18:56, Stefan G. Weichinger wrote:

Am 06.06.24 um 15:57 schrieb Marco Gaiarin:

Mandi! Josh Fisher via Bacula-users
   In chel di` si favelave...


Not for a single job. When the storage daemon is writing a job's spooled
data to tape, the client must wait. However, if multiple jobs are
running in parallel, then the other jobs will continue to spool their
data while one job is despooling to tape.


I come back on this. I've setup on director's storage Autochanger{}
configuration:

Maximum Concurrent Jobs = 5

but i've set on storage configuration, in the Device{} stanza:

Maximum Concurrent Jobs = 1

as suggested for tapes; clearly i've only one device/tape. I've 
defined two

jobs, but the first run, the second:

is waiting on Storage "BPBKPLOMAutochanger"

And don't start to spool data, wait there indefinetively.


What i'm missing?! Thanks.


interested as well, I need to speedup my weekly/monthly FULL runs (with 
LTO6, though: way slower anyway).


Shouldn't the file daemon do multiple jobs in parallel?

To tape you can only write ONE stream of data.

To the spooling disk there could be more than one stream.


Yes, that seems wrong:
$ grep Concurrent *.conf
bacula-dir.conf:  Maximum Concurrent Jobs = 50
bacula-dir.conf:  Maximum Concurrent Jobs = 50
bacula-fd.conf:  Maximum Concurrent Jobs = 50
bacula-sd.conf:  Maximum Concurrent Jobs = 50

Cheers,
GaryB-)


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Stefan G. Weichinger]

Am 06.06.24 um 15:57 schrieb Marco Gaiarin:

Mandi! Josh Fisher via Bacula-users
   In chel di` si favelave...


Not for a single job. When the storage daemon is writing a job's spooled
data to tape, the client must wait. However, if multiple jobs are
running in parallel, then the other jobs will continue to spool their
data while one job is despooling to tape.


I come back on this. I've setup on director's storage Autochanger{}
configuration:

Maximum Concurrent Jobs = 5

but i've set on storage configuration, in the Device{} stanza:

Maximum Concurrent Jobs = 1

as suggested for tapes; clearly i've only one device/tape. I've defined two
jobs, but the first run, the second:

is waiting on Storage "BPBKPLOMAutochanger"

And don't start to spool data, wait there indefinetively.


What i'm missing?! Thanks.


interested as well, I need to speedup my weekly/monthly FULL runs (with 
LTO6, though: way slower anyway).


Shouldn't the file daemon do multiple jobs in parallel?

To tape you can only write ONE stream of data.

To the spooling disk there could be more than one stream.





___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

Mandi! Josh Fisher via Bacula-users
  In chel di` si favelave...

> Not for a single job. When the storage daemon is writing a job's spooled 
> data to tape, the client must wait. However, if multiple jobs are 
> running in parallel, then the other jobs will continue to spool their 
> data while one job is despooling to tape.

I come back on this. I've setup on director's storage Autochanger{}
configuration:

Maximum Concurrent Jobs = 5

but i've set on storage configuration, in the Device{} stanza:

Maximum Concurrent Jobs = 1

as suggested for tapes; clearly i've only one device/tape. I've defined two
jobs, but the first run, the second:

is waiting on Storage "BPBKPLOMAutochanger"

And don't start to spool data, wait there indefinetively.


What i'm missing?! Thanks.

-- 
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''  http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
  http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

Mandi! Josh Fisher via Bacula-users
  In chel di` si favelave...

> Not for a single job. When the storage daemon is writing a job's spooled 
> data to tape, the client must wait. However, if multiple jobs are 
> running in parallel, then the other jobs will continue to spool their 
> data while one job is despooling to tape.

OK. It is right. Better use more jobs then one... right...


> 300 MB/s. Even then, i am skeptical that LTO9 is possible without spooling.

For what i'm learning by study and tests, i confirm that.


> It depends greatly on the client, the network, and the type of job. Only 

Sorry, forgot to specify. No network, all local. ;-)

-- 
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''  http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
  http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)





___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

Mandi! Gary R. Schmidt
  In chel di` si favelave...

> And a sensible amount of RAM - millions of files on ZFS should not be a 
> problem - unless you're doing it on a system with 32G of RAM or the like.

 root@bpbkplom:~# free -h
   totalusedfree  shared  buff/cache   available
 Mem:  125Gi61Gi60Gi   739Mi   2.9Gi62Gi
 Swap:0B  0B  0B

;-)


> Oh, and is it Solaris, BSD, or Linux?

Linux/Debian Buster (PVE6).

-- 
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''  http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
  http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)





___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

Mandi! Heitor Faria
  In chel di` si favelave...

> Is the ZFS local?

Yep.


> Does it have ZFS compression or dedup enabled?

Damn. Dedup no, but compression IS enabled... right! Never minded about
that... I've created a different mountpoint with compression disabled, i'll
provide feedback.

Thanks!


> I wouldn't use those options for data spooling. You have to also consider the 
>number of disks: 
>https://arstechnica.com/gadgets/2020/05/zfs-versus-raid-eight-ironwolf-disks-two-filesystems-one-winner/

I'll read on. Thanks for the link!

-- 
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''  http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
  http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)





___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

> Damn. Dedup no, but compression IS enabled... right! Never minded about
> that... I've created a different mountpoint with compression disabled, i'll
> provide feedback.

OK, as supposed with ZFS compression disabled provide some performance
improvement, but little one, nothing dramatic.


Still testing...

-- 
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''  http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
  http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Gary R. Schmidt]

On 20/05/2024 21:25, Heitor Faria wrote:

Hello Marco,

 > Anyway, i'm hit another trouble. Seems that creating the spool file 
took an

insane amount of time: source to backup are complex dirs, with millions of
files. Filesystem is ZFS.

Is the ZFS local? Does it have ZFS compression or dedup enabled? I 
wouldn't use those options for data spooling. You have to also consider 
the number of disks: 
https://arstechnica.com/gadgets/2020/05/zfs-versus-raid-eight-ironwolf-disks-two-filesystems-one-winner/ 


You can test Disks reading and writing capacity with hdparm, dd/iostat. 
https://www.bacula.lat/benchmarking-disks-reading-and-writing-capacity/?lang=en 


And network with iperf: 
https://www.bacula.lat/testing-bacula-machines-network-capacity-iperf/?lang=en 


I wouldn't also use Bacula compression software compression for tapes.

 > How can i do to improve the spooling performance? What factors impact 
more?


Probably better network and better/more disks, such as Enterprise Grade 
NVMe/SSDs.


And a sensible amount of RAM - millions of files on ZFS should not be a 
problem - unless you're doing it on a system with 32G of RAM or the like.


Oh, and is it Solaris, BSD, or Linux?

Cheers,
GaryB-)


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Heitor Faria]

Hello Marco,



> Anyway, i'm hit another trouble. Seems that creating the spool file took an
insane amount of time: source to backup are complex dirs, with millions of
files. Filesystem is ZFS.



Is the ZFS local? Does it have ZFS compression or dedup enabled? I wouldn't use 
those options for data spooling. You have to also consider the number of disks: 
https://arstechnica.com/gadgets/2020/05/zfs-versus-raid-eight-ironwolf-disks-two-filesystems-one-winner/

You can test Disks reading and writing capacity with hdparm, dd/iostat. 
https://www.bacula.lat/benchmarking-disks-reading-and-writing-capacity/?lang=en

And network with iperf: 
https://www.bacula.lat/testing-bacula-machines-network-capacity-iperf/?lang=en

I wouldn't also use Bacula compression software compression for tapes.



> How can i do to improve the spooling performance? What factors impact more?



Probably better network and better/more disks, such as Enterprise Grade 
NVMe/SSDs.



Rgds. 


MSc,MBA Heitor Faria (Miami/USA)
Bacula LATAM CIO

mobile1: + 1 909 655-8971
mobile2: + 55 61 98268-4220



bacula.lat | bacula.com.br


___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

Re: [Bacula-users] Again LTO9 and performances...

[Josh Fisher via Bacula-users]

On 5/17/24 06:29, Marco Gaiarin wrote:

I'm still fiddling with LTO9 and backup performances; finally i've managed
to test a shiny new server with an LTO9 tape (library indeed, but...) and i
can reach with 'btape test' 300MB/s, that is pretty cool, even if IBM
specification say that the tape could perform 400 MB/s.

Also, following suggestion, i'm using spooling to prevent tape to spin
up but this clearly 'doubles' the backup time... there's some way to
do spooling in parallel? EG, while creating the next spool file, bacula
write to tape the current one?


Not for a single job. When the storage daemon is writing a job's spooled 
data to tape, the client must wait. However, if multiple jobs are 
running in parallel, then the other jobs will continue to spool their 
data while one job is despooling to tape.


It is not clear that spooling doubles the backup time. That would only 
be true if the client is able to supply data at 300 MB/s, or if multiple 
clients running in parallel can supply a cumulative stream of data at 
300 MB/s. Even then, i am skeptical that LTO9 is possible without spooling.





Anyway, i'm hit another trouble. Seems that creating the spool file took an
insane amount of time: source to backup are complex dirs, with millions of
files. Filesystem is ZFS.

'insane amount' means that having a despool performance of 300MB/s, i'm a
overral backup performance of 40MB/s...


It depends greatly on the client, the network, and the type of job. Only 
full jobs will supply a somewhat continuous stream of data. Client 
performance is also a factor. Is the client busy while being backed up? 
How busy is the network?


At the end of data despooling, the attributes are despooled and written 
to the database, so that is also a part of the overall backup 
performance. Check the performance of the database writes.





How can i do to improve the spooling performance? What factors impact more?


Thanks.





___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Bacula-users] Again LTO9 and performances...

[Marco Gaiarin]

I'm still fiddling with LTO9 and backup performances; finally i've managed
to test a shiny new server with an LTO9 tape (library indeed, but...) and i
can reach with 'btape test' 300MB/s, that is pretty cool, even if IBM
specification say that the tape could perform 400 MB/s.

Also, following suggestion, i'm using spooling to prevent tape to spin
up but this clearly 'doubles' the backup time... there's some way to
do spooling in parallel? EG, while creating the next spool file, bacula
write to tape the current one?


Anyway, i'm hit another trouble. Seems that creating the spool file took an
insane amount of time: source to backup are complex dirs, with millions of
files. Filesystem is ZFS.

'insane amount' means that having a despool performance of 300MB/s, i'm a
overral backup performance of 40MB/s...


How can i do to improve the spooling performance? What factors impact more?


Thanks.

-- 
dott. Marco Gaiarin GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''  http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
  http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)




___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users