Hi everyone,
I am running Bacula 15.0.3 (Community) on Ubuntu 24.04 using the Cloud storage
device with Driver = Amazon(aws_cloud_driver) against an S3-compatible endpoint
(GleSYS S3).
I am experiencing an intermittent issue (1-2% failure rate) where larger volume
parts (~1 GB part.2 files) fail to upload during a backup job. For example,
during a ~2.5 TB backup job for client2 (~2,500 1 GB volumes), 29 volumes
failed to upload their part.2 file.
When this happens, Bacula logs "retry=1/10 and msg= Child exited with code 1",
but immediately sets state=done on the part and moves on without attempting
retries 2 through 10.
Here is an example from the Storage Daemon log:
```
4-Nov 21:26 bacula-s3-01-vbdc-sd JobId 23: Bacula-client1-0082/part.2
state=done retry=1/10 size=1.073 GB duration=15s msg= Child exited with code
1
```
And an earlier occurrence showing the same behavior across consecutive volumes:
```
31-Oct 12:51 bacula-s3-01-vbdc-sd JobId 11: Bacula-client2-2093/part.1
state=done size=273 B duration=2s
31-Oct 12:51 bacula-s3-01-vbdc-sd JobId 11: Bacula-client2-2093/part.2
state=done retry=1/10 size=1.073 GB duration=14s msg= Child exited with code 1
```
Because Bacula records state=done instead of state=error and aborts the retry
loop after 1/10:
1. The job finishes without retrying the failed part.
2. Truncate Cache = AfterUpload does not clean up the local file, leaving
the un-uploaded part.2 (~1 GB) stranded in the local cache directory
(/backup/file01).
3. The missing part is only uploaded if I manually run `cloud upload
storage=<storage> volume=<volume>` in `bconsole`.
Relevant Configuration (bacula-sd.conf & Pool):
```
Cloud {
Name = GleSYSS3
Driver = Amazon
HostName = "objects.dc-sto1.glesys.net"
BucketName = "bacula-s3-testing"
AccessKey = “XXXX"
SecretKey = "XXXXXXXXXXXXXXXX"
Protocol = HTTPS
UriStyle = VirtualHost
Truncate Cache = AfterUpload
Upload = EachPart
Region = "dc-sto1"
}
Device {
Name = FileStorage
Device Type = Cloud
Cloud = GleSYSS3
Media Type = File
Archive Device = /backup/file01
LabelMedia = yes
Random Access = yes
AutomaticMount = yes
RemovableMedia = no
AlwaysOpen = no
Maximum Part Size = 268435456
}
```
(Pool has Maximum Volume Bytes = 1G, resulting in a ~280 B part.1 metadata file
and a ~1 GB part.2 data file).
How to Reproduce:
1. Configure a Cloud resource with Driver = Amazon, Upload = EachPart, and
Truncate Cache = AfterUpload.
2. Configure a Pool with Maximum Volume Bytes = 1G and omit Maximum Part
Size on the Device (so part.2 is ~1 GB).
3. Run large backup jobs (e.g., 1–2.5 TB spanning thousands of 1 GB volumes)
until aws_cloud_driver encounters a transient upload error (Child exited with
code 1) on part.2 (1-2% of volumes).
4. Observe that the SD logs state=done retry=1/10 ... msg= Child exited with
code 1, skips retries 2–10, and leaves the un-uploaded part.2 in the local
Archive Device cache directory.
Is it expected behavior for Driver = Amazon to mark a part as state=done and
stop after retry=1/10 when aws_cloud_driver exits with code 1, or is this a bug
in how the SD handles child exit codes from the cloud driver?
Thanks in advance!
Abdulaziz Al Ahmad
Server Technician
_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users