Thanks Heitor,
I have changed configuration back to normal volumes (write to file and
not to folders with fileparts). I have limited max jobs parameter to 1.
I'm still puzzled with volume reuse:
a) I have limited maximum volumes to 6 on my clients weekly pool (dc1)
b) The label command was triggered a month ago (I'm not sure why). At
that time I have created another volume (dc1_weekly).
c) Today I'm trying to create a full backup to weekly pool but I'm again
getting error: "Cannot find any appendable volumes."
d) Label command works fine on other pools (I can create backup of dc1
client in monthly pool, but there is no problem with maximum volumes limit)
e) I have simulated this on another client (jhost05) with the same
configuration. It has created 7th volume (not sure why - I have limited
max volume to 6). At next backup it triggered label command (not sure
why - I expect that mediaid 475 is suitable to write to)
I have attached relevant part of configuration bellow.
Any idea why this is happening? What should I do to force bacula to
reuse my volumes?
Should I do something with existing volumes now that I have changed them
back to files instead of folders (probably it changed voltype from 14 to 1)?
Kind regards and happy new year. :)
Ziga
*list volume pool=dc1-weekly-pool
+---------+---------------------+-----------+---------+----------------+----------+--------------+---------+------+-----------+-----------+---------+----------+---------------------+-----------+
| mediaid | volumename | volstatus | enabled | volbytes |
volfiles | volretention | recycle | slot | inchanger | mediatype |
voltype | volparts | lastwritten | expiresin |
+---------+---------------------+-----------+---------+----------------+----------+--------------+---------+------+-----------+-----------+---------+----------+---------------------+-----------+
| 284 | dc1-weekly-vol-0284 | Purged | 1 | 91,477,448,939
| 21 | 3,024,000 | 1 | 0 | 0 | File1
| 14 | 94 | 2020-11-11 11:23:40 | 0 |
| 289 | dc1-weekly-vol-0289 | Append | 1 | 49,639,547,675
| 11 | 3,024,000 | 1 | 0 | 0 | File
| 1 | 0 | 2020-10-01 11:05:10 | 0 |
| 312 | dc1-weekly-vol-0312 | Purged | 1 | 14,998,326,906
| 3 | 3,024,000 | 1 | 0 | 0 | File1
| 14 | 16 | 2020-11-21 04:50:00 | 0 |
| 319 | dc1-weekly-vol-0319 | Used | 1 | 77,232,998,541
| 17 | 3,024,000 | 1 | 0 | 0 | File1
| 14 | 79 | 2020-11-28 07:08:39 | 244,150 |
| 354 | dc1-weekly-vol-0354 | Recycle | 1 | 1
| 0 | 3,024,000 | 1 | 0 | 0 | File1
| 14 | 0 | 2020-10-24 15:26:07 | 0 |
| 385 | dc1-weekly-vol-0385 | Purged | 1 | 10,999,742,528
| 2 | 3,024,000 | 1 | 0 | 0 | File1
| 14 | 12 | 2020-10-31 05:28:03 | 0 |
| 430 | dc1_weekly | Purged | 1 | 18,458,421,588
| 4 | 3,024,000 | 1 | 0 | 0 | File1
| 14 | 20 | 2020-11-14 07:36:56 | 0 |
+---------+---------------------+-----------+---------+----------------+----------+--------------+---------+------+-----------+-----------+---------+----------+---------------------+-----------+
*list volume pool=jhost05-weekly-pool
+---------+-------------------------+-----------+---------+---------------+----------+--------------+---------+------+-----------+-----------+---------+----------+---------------------+-----------+
| mediaid | volumename | volstatus | enabled |
volbytes | volfiles | volretention | recycle | slot | inchanger |
mediatype | voltype | volparts | lastwritten | expiresin |
+---------+-------------------------+-----------+---------+---------------+----------+--------------+---------+------+-----------+-----------+---------+----------+---------------------+-----------+
| 475 | jhost05-weekly-vol-0475 | Purged | 1 |
2,293,497,625 | 0 | 3,024,000 | 1 | 0 | 0 | File1
| 14 | 4 | 2020-11-20 23:37:22 | 0 |
| 504 | jhost05-weekly-vol-0504 | Used | 1 |
2,293,497,625 | 0 | 3,024,000 | 1 | 0 | 0 | File1
| 14 | 4 | 2020-11-28 02:05:21 | 219,916 |
| 603 | jhost05-weekly-vol-0603 | Used | 1 |
2,293,497,625 | 0 | 3,024,000 | 1 | 0 | 0 | File1
| 14 | 4 | 2020-12-11 23:09:53 | 1,418,988 |
| 616 | jhost05-weekly-vol-0616 | Used | 1 |
2,293,497,625 | 0 | 3,024,000 | 1 | 0 | 0 | File1
| 14 | 4 | 2020-12-18 23:14:21 | 2,024,056 |
| 628 | jhost05-weekly-vol-0628 | Append | 1 |
2,293,497,625 | 0 | 3,024,000 | 1 | 0 | 0 | File1
| 14 | 4 | 2020-12-25 23:09:39 | 2,628,574 |
| 635 | jhost05-weekly-vol-0635 | Used | 1 |
2,293,497,620 | 0 | 3,024,000 | 1 | 0 | 0 | File1
| 1 | 0 | 2020-12-30 12:55:26 | 3,023,721 |
| 636 | jhost05-weekly-vol-0636 | Used | 1 |
2,293,497,620 | 0 | 3,024,000 | 1 | 0 | 0 | File1
| 1 | 0 | 2020-12-30 12:59:05 | 3,023,940 |
+---------+-------------------------+-----------+---------+---------------+----------+--------------+---------+------+-----------+-----------+---------+----------+---------------------+-----------+
#Example of job output:
30-Dec 12:25 bacula-dir JobId 2304: Start Backup JobId 2304,
Job=dc1-monthly-backup.2020-12-30_12.25.21_08
30-Dec 12:25 bacula-dir JobId 2304: Pruning oldest volume
"dc1-weekly-vol-0354"
30-Dec 12:25 bacula-dir JobId 2304: Using Device "FSOciCloudStandard2"
to write.
30-Dec 12:25 bacula-dir JobId 2304: Pruning oldest volume
"dc1-weekly-vol-0354"
30-Dec 12:25 bacula-dir JobId 2304: Pruning oldest volume
"dc1-weekly-vol-0354"
30-Dec 12:25 bacula-sd JobId 2304: Job
dc1-monthly-backup.2020-12-30_12.25.21_08 is waiting. Cannot find any
appendable volumes.
Please use the "label" command to create a new Volume for:
Storage: "FSOciCloudStandard2" (/mnt/ocisg/bacula/backup)
Pool: dc1-weekly-pool
Media type: File1
#Relevant part of bacula-dir conf:
Pool {
Name = dc1-weekly-pool
Pool Type = Backup
Recycle = yes # Bacula can automatically
recycle Volumes
AutoPrune = no # Prune expired volumes (catalog
job handles this)
Action On Purge = Truncate # Allow to volume truncation
Volume Use Duration = 3 days # Create new volume for each backup
Volume Retention = 35 days # one month
Maximum Volume Bytes = 500G # Limit Volume size to something
reasonable
Maximum Volumes = 6 # Limit number of Volumes in Pool
Label Format = "dc1-weekly-vol-" # Auto label
Cache Retention = 1 days # Cloud specific (delete local
cache after one day)
Maximum Volume Jobs = 1 # Write each backup to new volume
Recycle Oldest Volume = yes # In case maximum volumes is
reached - prune oldest backup
}
#Relevant part of bacula-sd conf:
Device {
Name = FSOciCloudStandard2
Device type = File
Media Type = File1
Archive Device = /mnt/ocisg/bacula/backup
LabelMedia = yes; # lets Bacula label unlabeled media
Random Access = Yes;
AutomaticMount = yes; # when device opened, read it
RemovableMedia = no;
AlwaysOpen = no;
Autochanger = yes;
}
On 28.12.2020 22:18, Heitor Faria wrote:
Hello Ziga,
If you are using the Orwcle Gateway there is no reason on Earth to use
the S3 Driver. It will only mess your system, since I believe even in
the SD startup it tries to connect to the bucket.
If you want smaller volumes, just tune the Maximum Volume Bytes and
maybe Jobs (e.g. one).
Regards,
--
MSc Heitor Faria
CEO Bacula LatAm
mobile1: + 1 909 655-8971
mobile2: + 55 61 98268-4220
América Latina
[ http://bacula.lat/]
-------- Original Message --------
From: Žiga Žvan <ziga.z...@cetrtapot.si>
Sent: Monday, December 28, 2020 04:07 PM
To: Heitor Faria <hei...@bacula.com.br>,bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] bacula-sd - file driver - cloud resource
Hello Heitor,
I am aware of that. I'm using Oracle Storage Gateway to upload data to
cloud (to S3 bucket). For bacula-sd, backup destination is ordinary
filesystem mounted over nfs. However I'm using file driver with cloud
resource (in order to have volume broken down in file parts and
therefore optimize upload amount).
Everything worked until bacula decided to reuse old volume. I'm getting
strange error (cannot download Volume). In my opinion this is some sort
of bug, because data is available on nfs destination (see example output
from cloudcache and backup folder bellow).
Could you check my configuration (bellow)? Could you confirm that this
is a bug and I did not misconfigure something?
Regards,
Ziga
On 28.12.2020 14:46, Heitor Faria wrote:
>
> Hello Ziga,
>
> The Bacula Community S3 Plugin was built for Amazon S3 protocol.
> I read that Oracle built some sort of S3 emulator for its cloud, but
> since Bacula Community is a free software, it comes without warranties
> or support.
> You could consider running the bacula-sd in debug mode to troubleshoot
> the problem. The Driver is not working properly.
> Bacula Systems already developed a specific plugin for Oracle Object
> Storages, available only in the Enterprise edition.
>
> Regards,
> --
> MSc Heitor Faria
> CEO Bacula LatAm
> mobile1: + 1 909 655-8971
> mobile2: + 55 61 98268-4220
>
> América Latina
> [ http://bacula.lat/]
>
>
>
> -------- Original Message --------
> From: Žiga Žvan <ziga.z...@cetrtapot.si>
> Sent: Monday, December 28, 2020 08:22 AM
> To: bacula-users@lists.sourceforge.net
> Subject: Re: [Bacula-users] bacula-sd - file driver - cloud resource
>
> Hi,
> What am I doing wrong? I would be grateful for a hint, why there is no
> reply to my emails.
> I believe that my problems are a bit complex for an email. Is there any
> way to discuss this in different way?
> Who should I contact in case that I would like to support development of
> oracle cloud plugin?
>
> Regards,
> Ziga
>
> On 21.12.2020 20:38, Žiga Žvan wrote:
> > Hello,
> > I'm using file driver with cloud resource. Bacula was able to backup
> > data in this way until it wrote data to new volumes. Now, after
> > retention period, I'm getting error: Fatal error: cloud_dev.c:983
> > Unable to download Volume (see output below). Data on cloud path looks
> > ok but data in local cache contains only part.1 without any data.
> >
> > Is this expected?
> > Has anybody tested this scenario?
> > Should I avoid file driver in production environment?
> >
> > Regards,
> > Ziga
> >
> >
> > [root@bacula db-01-weekly-vol-0365]# ls -la
> > /mnt/ocisg/bacula/backup/db-01-weekly-vol-0365
> > total 0
> > drwxr-----. 2 bacula disk 0 Oct 24 07:45 .
> > drwxr-xr-x. 2 bacula bacula 0 Dec 18 23:38 ..
> > -rw-r--r--. 1 bacula disk 256 Oct 24 07:43 part.1
> > -rw-r--r--. 1 bacula disk 999935992 Oct 24 07:44 part.2
> > -rw-r--r--. 1 bacula disk 999935993 Oct 24 07:44 part.3
> > -rw-r--r--. 1 bacula disk 381771773 Oct 24 07:45 part.4
> >
> > [root@bacula db-01-weekly-vol-0365]# ls -la
> > /storage/bacula/cloudcache/db-01-weekly-vol-0365
> > total 20
> > drwxr-----. 2 bacula disk 28 Dec 11 23:10 .
> > drwxr-xr-x. 344 bacula bacula 16384 Dec 18 23:26 ..
> > -rw-r--r--. 1 bacula disk 0 Dec 11 23:10 part.1
> >
> > SD config (autochanger)
> >
> > Device {
> > Name = FSOciCloudStandard2
> > Device type = Cloud
> > Cloud = OracleViaStorageGateway
> > Maximum Part Size = 1000 MB
> > Media Type = File1
> > Archive Device = /storage/bacula/cloudcache
> > LabelMedia = yes; # lets Bacula label unlabeled
> media
> > Random Access = Yes;
> > AutomaticMount = yes; # when device opened, read it
> > RemovableMedia = no;
> > AlwaysOpen = no;
> > Autochanger = yes;
> > }
> > ...
> > Device {
> > Name = FSOciCloudStandard4
> > Device type = Cloud
> > Cloud = OracleViaStorageGateway
> > Maximum Part Size = 1000 MB
> > Media Type = File1
> > Archive Device = /storage/bacula/cloudcache
> > LabelMedia = yes; # lets Bacula label unlabeled
> media
> > Random Access = Yes;
> > AutomaticMount = yes; # when device opened, read it
> > RemovableMedia = no;
> > AlwaysOpen = no;
> > Autochanger = yes;
> > }
> >
> > Cloud {
> > Name = OracleViaStorageGateway
> > Driver = "File"
> > HostName = "/mnt/ocisg/bacula/backup"
> > BucketName = "DummyBucket"
> > AccessKey = "DummyAccessKey"
> > SecretKey = "DummySecretKey"
> > Protocol = HTTPS
> > UriStyle = VirtualHost
> > Truncate Cache = AtEndOfJob
> > }
> >
> >
> > 21-Dec 19:14 bacula-dir JobId 2073: Start Backup JobId 2073,
> > Job=db-01-backup.2020-12-21_19.14.14_48
> > 21-Dec 19:14 bacula-dir JobId 2073: Using Device "FSOciCloudStandard2"
> > to write.
> > 21-Dec 19:14 bacula-sd JobId 2073: Fatal error: cloud_dev.c:983 Unable
> > to download Volume="db-01-weekly-vol-0365" label.
> > 21-Dec 19:14 bacula-sd JobId 2073: Fatal error: cloud_dev.c:983 Unable
> > to download Volume="db-01-weekly-vol-0365" label.
> > 21-Dec 19:14 db-01.prod.kr.cetrtapot.si JobId 2073: Fatal error:
> > job.c:3013 Bad response from SD to Append Data command. Wanted 3000 OK
> > data
> > , got len=25 msg="3903 Error append data: "
> > 21-Dec 19:14 bacula-sd JobId 2073: Fatal error: cloud_dev.c:983 Unable
> > to download Volume="db-01-weekly-vol-0365" label.
> > 21-Dec 19:14 bacula-sd JobId 2073: Warning: label.c:398 Open Cloud
> > device "FSOciCloudStandard2" (/storage/bacula/cloudcache) Volume
> > "db-01-weekly-vol-0365" failed: ERR=
> > 21-Dec 19:14 bacula-sd JobId 2073: Fatal error: cloud_dev.c:983 Unable
> > to download Volume="db-01-weekly-vol-0365" label.
> > 21-Dec 19:14 bacula-sd JobId 2073: Fatal error: cloud_dev.c:983 Unable
> > to download Volume="db-01-weekly-vol-0365" label.
> > 21-Dec 19:14 bacula-sd JobId 2073: Warning: label.c:398 Open Cloud
> > device "FSOciCloudStandard2" (/storage/bacula/cloudcache) Volume
> > "db-01-weekly-vol-0365" failed: ERR=
> > 21-Dec 19:14 bacula-sd JobId 2073: Marking Volume
> > "db-01-weekly-vol-0365" in Error in Catalog.
> > 21-Dec 19:14 bacula-sd JobId 2073: Fatal error: Job 2073 canceled.
> >
> >
> > On 06.12.2020 20:52, Žiga Žvan wrote:
> >> Dear all,
> >> I'm using bacula 9.6.5 in a production for a month now. I'm
> >> experiencing random backup failures from my clients. Specific hosts
> >> report errors like the outputs attached. The same host is able to
> >> perform backup at some other time. The error is more often at large
> >> backups (more errors at full backups than incremental, more errors at
> >> hosts with large data sets).
> >>
> >> I have tried to implement heartbeat interval
> >>
>
(https://www.bacula.org/9.6.x-manuals/en/main/Client_File_daemon_Configur.html#SECTION002210000000000000000)
>
> >> but there is no improvement.
> >> The error occures also on hosts in the same zone as bacula server (no
> >> router/firewall in between).
> >> Storage deamon is installed on the same server as bacula director.
> >> I'm using File cloud driver (backup to local disk via cloud
resource).
> >>
> >> Could you please suggest a solution or a way to troubleshoot this
> >> further?
> >> Thx!
> >>
> >> Regards,Ziga Zvan
> >>
> >> Backup from linux hosts (on 05-dec 3 hosts failed, 20 hosts completed
> >> without error):
> >> 05-Dec 03:26 bacula-dir JobId 1721: Fatal error: Network error with
> >> FD during Backup: ERR=Connection reset by peer
> >> 05-Dec 03:27 bacula-dir JobId 1721: Fatal error: No Job status
> >> returned from FD.
> >> 05-Dec 03:27 bacula-dir JobId 1721: Error: Bacula bacula-dir 9.6.5
> >> (11Jun20):
> >>
> >> Backup from windows hosts (on 05-dec 2 hosts failed, 5 hosts
> >> completed without error):
> >> 05-Dec 00:40 iwhost01.kranj.cetrtapot.si-fd JobId 1726: Error:
> >> lib/bsock.c:383 Write error sending 57172 bytes to Storage
> >> daemon:192.168.66.35:9103: ERR=Input/output error
> >> 05-Dec 00:40 iwhost01.kranj.cetrtapot.si-fd JobId 1726: Fatal error:
> >> filed/backup.c:848 Network send error to SD. ERR=Input/output error
> >> 05-Dec 00:40 iwhost01.kranj.cetrtapot.si-fd JobId 1726: VSS Writer
> >> (BackupComplete): "Task Scheduler Writer", State: 0x1 (VSS_WS_STABLE)
> >> 05-Dec 00:40 iwhost01.kranj.cetrtapot.si-fd JobId 1726: VSS Writer
> >> (BackupComplete): "VSS Metadata Store Writer", State: 0x1
> >> (VSS_WS_STABLE)
> >> 05-Dec 00:40 iwhost01.kranj.cetrtapot.si-fd JobId 1726: VSS Writer
> >> (BackupComplete): "Performance Counters Writer", State: 0x1
> >> (VSS_WS_STABLE)
> >> 05-Dec 00:40 iwhost01.kranj.cetrtapot.si-fd JobId 1726: VSS Writer
> >> (BackupComplete): "System Writer", State: 0x1 (VSS_WS_STABLE)
> >> 05-Dec 00:40 iwhost01.kranj.cetrtapot.si-fd JobId 1726: VSS Writer
> >> (BackupComplete): "ASR Writer", State: 0x1 (VSS_WS_STABLE)
> >> 05-Dec 00:40 iwhost01.kranj.cetrtapot.si-fd JobId 1726: VSS Writer
> >> (BackupComplete): "Shadow Copy Optimization Writer", State: 0x1
> >> (VSS_WS_STABLE)
> >> 05-Dec 01:01 bacula-dir JobId 1726: Error: bsock.c:551 Read error
> >> from Client:
> >> iwhost01.kranj.cetrtapot.si-fd:iwhost01.kranj.cetrtapot.si:9102:
> >> ERR=Connection timed out
> >> 05-Dec 01:01 bacula-dir JobId 1726: Fatal error: Network error with
> >> FD during Backup: ERR=Connection timed out
> >> 05-Dec 01:02 bacula-dir JobId 1726: Fatal error: No Job status
> >> returned from FD.
> >> 05-Dec 01:02 bacula-dir JobId 1726: Error: Bacula bacula-dir 9.6.5
> >> (11Jun20):
> >>
> >> Similar output from 21-Nov:
> >> 21-Nov 05:30 dc1.kranj.cetrtapot.si-fd JobId 1393: Error:
> >> lib/bsock.c:383 Write error sending 4 bytes to Storage
> >> daemon:192.168.66.35:9103: ERR=Input/output error
> >> 21-Nov 05:30 dc1.kranj.cetrtapot.si-fd JobId 1393: Error:
> >> lib/bsock.c:271 Socket has errors=1 on call to Storage
> >> daemon:192.168.66.35:9103
> >> 21-Nov 05:30 dc1.kranj.cetrtapot.si-fd JobId 1393: Error:
> >> lib/bsock.c:271 Socket has errors=1 on call to Storage
> >> daemon:192.168.66.35:9103
> >> 21-Nov 05:30 dc1.kranj.cetrtapot.si-fd JobId 1393: Fatal error:
> >> filed/backup.c:607 Network send error to SD. ERR=Input/output error
> >> 21-Nov 05:49 bacula-dir JobId 1393: Fatal error: Network error with
> >> FD during Backup: ERR=Connection timed out
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Bacula-users mailing list
> >> Bacula-users@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/bacula-users
> >
> >
> > _______________________________________________
> > Bacula-users mailing list
> > Bacula-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/bacula-users
>
>
> _______________________________________________
> Bacula-users mailing list
> Bacula-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
