Hi everyone,

I am running Bacula 15.0.3 (Community) on Ubuntu 24.04 using the Cloud storage 
device with Driver = Amazon(aws_cloud_driver) against an S3-compatible endpoint 
(GleSYS S3).

I am experiencing an intermittent issue (1-2% failure rate) where larger volume 
parts (~1 GB part.2 files) fail to upload during a backup job. For example, 
during a ~2.5 TB backup job for client2 (~2,500 1 GB volumes), 29 volumes 
failed to upload their part.2 file.

When this happens, Bacula logs "retry=1/10 and msg= Child exited with code 1", 
but immediately sets state=done on the part and moves on without attempting 
retries 2 through 10.

Here is an example from the Storage Daemon log:

```
4-Nov 21:26 bacula-s3-01-vbdc-sd JobId 23: Bacula-client1-0082/part.2     
state=done    retry=1/10 size=1.073 GB duration=15s msg= Child exited with code 
1


```

And an earlier occurrence showing the same behavior across consecutive volumes:

```
31-Oct 12:51 bacula-s3-01-vbdc-sd JobId 11: Bacula-client2-2093/part.1 
state=done size=273 B duration=2s
31-Oct 12:51 bacula-s3-01-vbdc-sd JobId 11: Bacula-client2-2093/part.2 
state=done retry=1/10 size=1.073 GB duration=14s msg= Child exited with code 1
```


Because Bacula records state=done instead of state=error and aborts the retry 
loop after 1/10:

  1.  The job finishes without retrying the failed part.

  2.  Truncate Cache = AfterUpload does not clean up the local file, leaving 
the un-uploaded part.2 (~1 GB) stranded in the local cache directory 
(/backup/file01).

  3.  The missing part is only uploaded if I manually run `cloud upload 
storage=<storage> volume=<volume>` in `bconsole`.


Relevant Configuration (bacula-sd.conf & Pool):

```

Cloud {

  Name = GleSYSS3
  Driver = Amazon
  HostName = "objects.dc-sto1.glesys.net"
  BucketName = "bacula-s3-testing"
  AccessKey = “XXXX"
  SecretKey = "XXXXXXXXXXXXXXXX"
  Protocol = HTTPS
  UriStyle = VirtualHost
  Truncate Cache = AfterUpload
  Upload = EachPart
  Region = "dc-sto1"
}

Device {
  Name = FileStorage
  Device Type = Cloud
  Cloud = GleSYSS3
  Media Type = File
  Archive Device = /backup/file01
  LabelMedia = yes
  Random Access = yes
  AutomaticMount = yes
  RemovableMedia = no
  AlwaysOpen = no
  Maximum Part Size = 268435456
}

```


(Pool has Maximum Volume Bytes = 1G, resulting in a ~280 B part.1 metadata file 
and a ~1 GB part.2 data file).

How to Reproduce:

  1.  Configure a Cloud resource with Driver = Amazon, Upload = EachPart, and 
Truncate Cache = AfterUpload.

  2.  Configure a Pool with Maximum Volume Bytes = 1G and omit Maximum Part 
Size on the Device (so part.2 is ~1 GB).

  3.  Run large backup jobs (e.g., 1–2.5 TB spanning thousands of 1 GB volumes) 
until aws_cloud_driver encounters a transient upload error (Child exited with 
code 1) on part.2 (1-2% of volumes).

  4.  Observe that the SD logs state=done retry=1/10 ... msg= Child exited with 
code 1, skips retries 2–10, and leaves the un-uploaded part.2 in the local 
Archive Device cache directory.

Is it expected behavior for Driver = Amazon to mark a part as state=done and 
stop after retry=1/10 when aws_cloud_driver exits with code 1, or is this a bug 
in how the SD handles child exit codes from the cloud driver?

Thanks in advance!




Abdulaziz Al Ahmad
Server Technician


_______________________________________________
Bacula-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-users

Reply via email to