GitHub user draggeta created a discussion: NAS backup of large VMs fails
### problem
When backing up up larger VMs (the ones we've tested have 2+ TB of storage),
scheduled and manual backups eventually fail to NFS storage. The files are
written until some sort of timeout seems to happen.
The files are available on the NFS storage but every backup they stop after a
certain period of time. A side effect of this is that cloudstack doesn't apply
retention to the backups in this case so they keep filling up storage.
Smaller servers (tested until 600 GB backup correctly. There is no firewall in
the path. The hypervisors and NFS storage are in the same subnet.
It does seem to be a timeout issue as it seems the management server times out
after an hour, but the backup continues running.
```
2026-08-15 06:04:06,169 DEBUG [o.a.c.b.BackupManagerImpl]
(BackupPollTask:[ctx-f0772149]) (logid:34dc33a0) Scheduling 1 backup for VM (VM
instance
{"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"})
for backup schedule (BackupSchedule
{"id":6,"schedule":"00:06:7","scheduleType":2,"vmId":1083}) at
[2026-08-15T06:00:00+0000].
2026-08-15 06:04:06,607 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b)
Seq 10-4578471971175552676: Sending { Cmd , MgmtId: 90520740805077, via:
10(nc-lab03), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-10-1083-VM","backupPath":"i-10-1083-VM/2026.08.15.06.04.06","backupRepoType":"nfs","backupRepoAddress":"[fd33:9999:1::21]:/mnt/POOL01/Backup/cs-backup","wait":"0","bypassHostMaintenance":"false"}}]
}
2026-08-15 06:08:39,558 DEBUG [o.a.c.b.B.BackupSyncTask]
(BackgroundTaskPollManager-6:[ctx-9f798002]) (logid:1d647320) Trying to sync
backups of VM [VM instance
{"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}]
using backup provider [nas].
2026-08-15 06:13:40,517 DEBUG [o.a.c.b.B.BackupSyncTask]
(BackgroundTaskPollManager-5:[ctx-55244db9]) (logid:55fc3c21) Trying to sync
backups of VM [VM instance
{"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}]
using backup provider [nas].
2026-08-15 07:04:06,609 WARN [c.c.a.m.ClusteredAgentAttache]
(API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b)
Seq 10-4578471971175552676: Timed out on Seq 10-4578471971175552676: { Cmd ,
MgmtId: 90520740805077, via: 10(nc-lab03), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-10-1083-VM","backupPath":"i-10-1083-VM/2026.08.15.06.04.06","backupRepoType":"nfs","backupRepoAddress":"[fd33:9999:1::21]:/mnt/POOL01/Backup/cs-backup","wait":"0","bypassHostMaintenance":"false"}}]
}
2026-08-15 07:04:06,609 ERROR [o.a.c.b.NASBackupProvider]
(API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b)
Operation to initiate backup timed out for VM i-10-1083-VM
```
Looking at the files:
```shell
$ ls -la ./cs-backup/i-10-1083-VM/2026.08.15.06.04.06/
total 996803735
drwxr-xr-x 2 nobody nogroup 8 Aug 15 07:32 .
drwxr-xr-x 4 nobody nogroup 4 Aug 15 09:56 ..
-rw------- 1 nobody nogroup 950586441728 Aug 15 07:32
datadisk.d2538e1e-7166-4e57-ac85-09cdcd97822e.qcow2
-rw-r--r-- 1 nobody nogroup 8060 Aug 15 06:04 domain-config.xml
-rw-r--r-- 1 nobody nogroup 272 Aug 15 06:04 domblklist.xml
-rw-r--r-- 1 nobody nogroup 178 Aug 15 06:04 domiflist.xml
-rw-r--r-- 1 nobody nogroup 421 Aug 15 06:04 dominfo.xml
-rw------- 1 nobody nogroup 261140578304 Aug 15 06:33
root.69d79690-5753-4972-8268-5fd38b791af7.qcow2
```
### versions
Ubuntu 24.04.4
10G networking over IPv6
Truenas NFS storage
Primary storage is Ceph
Other backup solutions for other hypervisors to the same storage have no issues
with long running backups.
### The steps to reproduce the bug
1. Schedule or run backup
2. Runs for a while
3. Fails, no backups show in UI, data files seen on NFS storage
### What to do about it?
_No response_
GitHub link: https://github.com/apache/cloudstack/discussions/13903
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
