Could not open: /data1/bareos/Full-0290, ERR=No space left on device
On Tuesday, 28 November 2017 09:35:07 UTC+2, chris.fi...@ad-juster.com
wrote:
>
> Some more output from a Full job that gets stuck (after many Incremental
> jobs ran)
>
> 2017-11-27 23:30:11 bareos-sd JobId 714: Please mount append Volume
> "Full-0290" or label a new one for:
> Job: BackupCatalog.2017-11-27_23.10.00_39
> Storage: "FileStorage-Data1" (/data1/bareos)
> Pool: Full
> Media type: File
> 2017-11-27 23:30:11 bareos-sd JobId 714: Warning: mount.c:250 Open device
> "FileStorage-Data1" (/data1/bareos) Volume "Full-0290" failed:
> ERR=dev.c:664 Could not open: /data1/bareos/Full-0290, ERR=No space left on
> device
>
> ...
>
> 2017-11-27 23:30:10 bareos-sd JobId 714: Warning: label.c:376 Open device
> "FileStorage-Data1" (/data1/bareos) Volume "Full-0290" failed:
> ERR=dev.c:664 Could not open: /data1/bareos/Full-0290, ERR=No space left on
> device
>
> 2017-11-27 23:30:10 bareos-dir JobId 714: Recycled volume "Full-0290"
> 2017-11-27 23:30:10 bareos-dir JobId 714: All records pruned from Volume
> "Full-0290"; marking it "Purged"
> 2017-11-27 23:30:10 bareos-dir JobId 714: There are no more Jobs
> associated with Volume "Full-0290". Marking it purged.
> 2017-11-27 23:30:10 bareos-sd JobId 714: End of medium on Volume
> "Full-0289" Bytes=1,096,893 Blocks=17 at 27-Nov-2017 23:30.
> 2017-11-27 23:30:08 bareos-sd JobId 714: End of Volume "Full-0289" at
> 0:1096892 on device "FileStorage-Data1" (/data1/bareos). Write of 64512
> bytes got 37699.
> 2017-11-27 23:30:08 bareos-sd JobId 714: New volume "Full-0289" mounted on
> device "FileStorage-Data1" (/data1/bareos) at 27-Nov-2017 23:30.
> 2017-11-27 23:30:07 bareos-sd JobId 714: Wrote label to prelabeled Volume
> "Full-0289" on device "FileStorage-Data1" (/data1/bareos)
> 2017-11-27 23:30:07 bareos-sd JobId 714: Labeled new Volume "Full-0289" on
> device "FileStorage-Data1" (/data1/bareos).
> 2017-11-27 23:30:02 bareos-dir JobId 714: Recycled volume "Full-0289"
> 2017-11-27 23:30:02 bareos-dir JobId 714: All records pruned from Volume
> "Full-0289"; marking it "Purged"
> 2017-11-27 23:30:01 bareos-dir JobId 714: There are no more Jobs
> associated with Volume "Full-0289". Marking it purged.
> 2017-11-27 23:30:01 bareos-sd JobId 714: End of medium on Volume
> "Full-0288" Bytes=285,336,758 Blocks=4,423 at 27-Nov-2017 23:30.
> 2017-11-27 23:30:01 bareos-sd JobId 714: End of Volume "Full-0288" at
> 0:285336757 on device "FileStorage-Data1" (/data1/bareos). Write of 64512
> bytes got 43850.
> 2017-11-27 23:30:00 bareos-sd JobId 714: Wrote label to prelabeled Volume
> "Full-0288" on device "FileStorage-Data1" (/data1/bareos)
> 2017-11-27 23:30:00 bareos-sd JobId 714: Labeled new Volume "Full-0288" on
> device "FileStorage-Data1" (/data1/bareos).
> 2017-11-27 23:29:59 bareos-dir JobId 714: Using Device "FileStorage-Data1"
> to write.
> 2017-11-27 23:29:59 bareos-dir JobId 714: Start Backup JobId 714,
> Job=BackupCatalog.2017-11-27_23.10.00_39
> 2017-11-27 23:29:36 bareos-dir JobId 714: shell command: run BeforeJob
> "/usr/lib/bareos/scripts/make_catalog_backup.pl MyCatalog"
>
> On Monday, November 27, 2017 at 2:32:49 PM UTC-8,
> chris.fi...@ad-juster.com wrote:
> > Hello,
> >
> > I am currently having an issue when I run a Full job (50G vol size, 365d
> retention) the job does not switch to the next available device(s) or
> recycle an existing volume when the device(s) runs out of space, however it
> seems to work fine when I have a Incremental job (1G vol size, 30d
> retention).
> >
> > Below I have listed my setup and configuration information that I feel
> is relevant, along with output from three jobs (2 of the jobs are a small
> job that fails in Full pool but works when Incremental pool, and the 3rd is
> the output from a DB backup that spans multiple Full volumes and caused the
> last disk to fill up over night, which was hoping it would roll back to
> Data1 or at least recycle the oldest on Data4 but it did not
> >
> > Thank you for any assistance in this manner.
> > -Chris
> >
> > ##
> > ## Setup
> > ##
> > Bareos 17.2rc1/rc2
> > CentOS 7 server with 4 4TB JBOD disks for backup storage
> > Director and SD on same machine
> > Multiple FD clients
> >
> > # Director storage/File.conf
> > Storage {
> > Name = File
> > Address = backup
> > Password = <password>
> > Device = FileStorage-Data1
> > Device = FileStorage-Data2
> > Device = FileStorage-Data3
> > Device = FileStorage-Data4
> > Media Type = File
> > Maximum Concurrent Jobs = 10
> > }
> >
> > # Director pool/Full.conf
> > Pool {
> > Name = Full
> > Pool Type = Backup
> > Recycle = yes
> > AutoPrune = yes
> > Volume Retention = 365 days
> > Maximum Volume Bytes = 50G
> > Maximum Volumes = 200
> > Label Format = "Full-"
> > Recycle Oldest Volume = yes
> > Action On Purge = Truncate
> > Purge Oldest Volume = yes
> > }
> >
> > # Director pool/Incremental.conf
> > Pool {
> > Name = Incremental
> > Pool Type = Backup
> > Recycle = yes
> > AutoPrune = yes
> > Volume Retention = 30 days
> > Maximum Volume Bytes = 1G
> > Maximum Volumes = 200
> > Label Format = "Incremental-"
> > Recycle Oldest Volume = yes
> > Action On Purge = Truncate
> > Purge Oldest Volume = yes
> > }
> >
> > # Storage device/FileStorage-Data1.conf
> > Device {
> > Name = FileStorage-Data1
> > Media Type = File
> > Archive Device = /data1/bareos
> > LabelMedia = yes;
> > Random Access = yes;
> > AutomaticMount = yes;
> > RemovableMedia = no;
> > AlwaysOpen = no;
> > Description = "File device. A connecting Director must have the same
> Name and MediaType."
> > Maximum Concurrent Jobs = 10
> > }
> >
> > # Storage device/FileStorage-Data2.conf
> > Device {
> > Name = FileStorage-Data2
> > Media Type = File
> > Archive Device = /data2/bareos
> > LabelMedia = yes;
> > Random Access = yes;
> > AutomaticMount = yes;
> > RemovableMedia = no;
> > AlwaysOpen = no;
> > Description = "File device. A connecting Director must have the same
> Name and MediaType."
> > Maximum Concurrent Jobs = 10
> > }
> >
> > # Storage device/FileStorage-Data3, Data4
> > Same as above just Name and Archive Device updated accordingly
> >
> >
> > ##
> > ## Expected actions
> > ##
> > # When all drives are not full
> > When FileStorage-Data1 fills up, the volume is created on
> FileStorage-Data2..4
> >
> > # When all drives are full
> > When all drives are full, recycle the oldest volume which should be on
> FileStorage-Data1
> >
> > ## Workaround so far
> > When FileStorage-Data1 filled up, comment it out in Director
> storage/File.conf which allows it to be written to FileStorage-Data2,
> repeat till Data4.
> >
> > This allowed me to kick the can down the road until last night when
> Data4 filled up and it did not recycle and got stuck.
> >
> > I should note, that when Data1 first filled up, again only Full backups
> would not jump to Data2..4 but Incremental and Differential backups would.
> >
> >
> > ##
> > ## Output
> > ##
> > # df
> > /dev/mapper/clc-data3 3.7T 3.6T 62G 99% /data3
> > /dev/mapper/cld-data4 3.7T 3.7T 747M 100% /data4
> > /dev/mapper/clb-data2 3.7T 3.7T 40K 100% /data2
> > /dev/mapper/cla-data1 3.0T 3.0T 1001M 100% /data1
> >
> > #
> > # FAILS - Full job output (works as Incremental)
> > #
> > 2017-11-27 14:00:03 bareos-dir JobId 702: Bareos bareos-dir 17.2.4
> (21Sep17):
> > Build OS: x86_64-redhat-linux-gnu redhat Red Hat
> Enterprise Linux Server release 7.0 (Maipo)
> > JobId: 702
> > Job: Machine1-Wiki.2017-11-27_13.52.25_51
> > Backup Level: Full
> > Client: "Machine1-fd" 17.2.3 (14Aug17)
> x86_64-redhat-linux-gnu,redhat,Red Hat Enterprise Linux Server release 7.0
> (Maipo),RHEL_7,x86_64
> > FileSet: "Machine1-Wiki" 2017-10-03 15:58:33
> > Pool: "Full" (From command line)
> > Catalog: "MyCatalog" (From Client resource)
> > Storage: "File" (From Job resource)
> > Scheduled time: 27-Nov-2017 13:52:25
> > Start time: 27-Nov-2017 13:52:28
> > End time: 27-Nov-2017 14:00:03
> > Elapsed time: 7 mins 35 secs
> > Priority: 10
> > FD Files Written: 0
> > SD Files Written: 0
> > FD Bytes Written: 0 (0 B)
> > SD Bytes Written: 0 (0 B)
> > Rate: 0.0 KB/s
> > Software Compression: None
> > VSS: no
> > Encryption: no
> > Accurate: no
> > Volume name(s):
> > Volume Session Id: 2
> > Volume Session Time: 1511818757
> > Last Volume Bytes: 0 (0 B)
> > Non-fatal FD errors: 1
> > SD Errors: 0
> > FD termination status: Error
> > SD termination status: Canceled
> > Termination: Backup Canceled
> >
> > 2017-11-27 14:00:03 bareos-dir JobId 702: Fatal error: No Job status
> returned from FD.
> > 2017-11-27 14:00:03 bareos-sd JobId 702: Fatal error: Job 702 canceled.
> > 2017-11-27 14:00:03 bareos-dir JobId 702: Fatal error: Network error
> with FD during Backup: ERR=Interrupted system call
> > 2017-11-27 13:57:28 bareos-sd JobId 702: Warning: mount.c:250 Open
> device "FileStorage-Data1" (/data1/bareos) Volume "Full-0288" failed:
> ERR=dev.c:664 Could not open: /data1/bareos/Full-0288, ERR=No such file or
> directory
> >
> > 2017-11-27 13:52:28 bareos-sd JobId 702: Please mount append Volume
> "Full-0288" or label a new one for:
> > Job: Machine1-Wiki.2017-11-27_13.52.25_51
> > Storage: "FileStorage-Data1" (/data1/bareos)
> > Pool: Full
> > Media type: File
> > 2017-11-27 13:52:28 bareos-sd JobId 702: Warning: mount.c:250 Open
> device "FileStorage-Data1" (/data1/bareos) Volume "Full-0288" failed:
> ERR=dev.c:664 Could not open: /data1/bareos/Full-0288, ERR=No space left on
> device
> >
> > ...
> >
> > 2017-11-27 13:52:28 bareos-sd JobId 702: Warning: label.c:376 Open
> device "FileStorage-Data1" (/data1/bareos) Volume "Full-0288" failed:
> ERR=dev.c:664 Could not open: /data1/bareos/Full-0288, ERR=No space left on
> device
> >
> > 2017-11-27 13:52:28 bareos-dir JobId 702: Using Device
> "FileStorage-Data1" to write.
> > 2017-11-27 13:52:27 bareos-dir JobId 702: Start Backup JobId 702,
> Job=Machine1-Wiki.2017-11-27_13.52.25_51
> >
> >
> >
> > #
> > # WORKS - Incremental job output (Same job)
> > #
> > 2017-11-27 14:00:08 bareos-dir JobId 703: Bareos bareos-dir 17.2.4
> (21Sep17):
> > Build OS: x86_64-redhat-linux-gnu redhat Red Hat
> Enterprise Linux Server release 7.0 (Maipo)
> > JobId: 703
> > Job: Machine1-Wiki.2017-11-27_13.59.58_02
> > Backup Level: Incremental, since=2017-11-27 07:48:39
> > Client: "Machine1-fd" 17.2.3 (14Aug17)
> x86_64-redhat-linux-gnu,redhat,Red Hat Enterprise Linux Server release 7.0
> (Maipo),RHEL_7,x86_64
> > FileSet: "Machine1-Wiki" 2017-10-03 15:58:33
> > Pool: "Incremental" (From command line)
> > Catalog: "MyCatalog" (From Client resource)
> > Storage: "File" (From command line)
> > Scheduled time: 27-Nov-2017 13:59:58
> > Start time: 27-Nov-2017 14:00:04
> > End time: 27-Nov-2017 14:00:08
> > Elapsed time: 4 secs
> > Priority: 10
> > FD Files Written: 139
> > SD Files Written: 139
> > FD Bytes Written: 24,751,160 (24.75 MB)
> > SD Bytes Written: 24,770,990 (24.77 MB)
> > Rate: 6187.8 KB/s
> > Software Compression: 53.5 % (lzo)
> > VSS: no
> > Encryption: no
> > Accurate: no
> > Volume name(s): Incremental-0094
> > Volume Session Id: 3
> > Volume Session Time: 1511818757
> > Last Volume Bytes: 24,800,601 (24.80 MB)
> > Non-fatal FD errors: 0
> > SD Errors: 0
> > FD termination status: OK
> > SD termination status: OK
> > Termination: Backup OK
> >
> > 2017-11-27 14:00:08 bareos-dir JobId 703: sql_create.c:872 Insert of
> attributes batch table done
> > 2017-11-27 14:00:07 bareos-sd JobId 703: Elapsed time=00:00:01, Transfer
> rate=24.77 M Bytes/second
> > 2017-11-27 14:00:06 bareos-sd JobId 703: New volume "Incremental-0094"
> mounted on device "FileStorage-Data1" (/data1/bareos) at 27-Nov-2017 14:00.
> > 2017-11-27 14:00:06 bareos-sd JobId 703: Recycled volume
> "Incremental-0094" on device "FileStorage-Data1" (/data1/bareos), all
> previous data lost.
> > 2017-11-27 14:00:05 bareos-dir JobId 703: Recycled volume
> "Incremental-0094"
> > 2017-11-27 14:00:05 bareos-dir JobId 703: All records pruned from Volume
> "Incremental-0094"; marking it "Purged"
> > 2017-11-27 14:00:05 bareos-dir JobId 703: There are no more Jobs
> associated with Volume "Incremental-0094". Marking it purged.
> > 2017-11-27 14:00:05 bareos-sd JobId 703: End of medium on Volume
> "Incremental-0093" Bytes=1,073,729,940 Blocks=2 at 27-Nov-2017 14:00.
> > 2017-11-27 14:00:05 bareos-sd JobId 703: User defined maximum volume
> capacity 1,073,741,824 exceeded on device "FileStorage-Data1"
> (/data1/bareos).
> > 2017-11-27 14:00:04 bareos-sd JobId 703: Ready to append to end of
> Volume "Incremental-0093" size=1073729940
> > 2017-11-27 14:00:04 bareos-sd JobId 703: Volume "Incremental-0093"
> previously written, moving to end of data.
> > 2017-11-27 14:00:04 bareos-dir JobId 703: Using Device
> "FileStorage-Data1" to write.
> > 2017-11-27 14:00:04 bareos-dir JobId 703: Start Backup JobId 703,
> Job=Machine1-Wiki.2017-11-27_13.59.58_02
> >
> >
> > #
> > # FAILS Full job that filled Data4 device
> > #
> > 2017-11-27 07:43:33 bareos-dir JobId 687: Bareos bareos-dir 17.2.3
> (14Aug17):
> > Build OS: x86_64-redhat-linux-gnu redhat Red Hat
> Enterprise Linux Server release 7.0 (Maipo)
> > JobId: 687
> > Job: Machine2-MySQL.2017-11-26_21.05.00_43
> > Backup Level: Full
> > Client: "Machine2-fd" 17.2.3 (14Aug17)
> x86_64-redhat-linux-gnu,redhat,Red Hat Enterprise Linux Server release 7.0
> (Maipo),RHEL_7,x86_64
> > FileSet: "MySQL-Data" 2017-10-04 15:39:20
> > Pool: "Full" (From Job FullPool override)
> > Catalog: "MyCatalog" (From Client resource)
> > Storage: "File" (From Job resource)
> > Scheduled time: 26-Nov-2017 21:05:00
> > Start time: 26-Nov-2017 21:05:03
> > End time: 27-Nov-2017 07:43:32
> > Elapsed time: 10 hours 38 mins 29 secs
> > Priority: 10
> > FD Files Written: 0
> > SD Files Written: 0
> > FD Bytes Written: 0 (0 B)
> > SD Bytes Written: 846,033 (846.0 KB)
> > Rate: 0.0 KB/s
> > Software Compression: None
> > VSS: no
> > Encryption: no
> > Accurate: no
> > Volume name(s): Full-0284|Full-0285|Full-0286
> > Volume Session Id: 564
> > Volume Session Time: 1507787926
> > Last Volume Bytes: 1,354,941 (1.354 MB)
> > Non-fatal FD errors: 1
> > SD Errors: 0
> > FD termination status: Error
> > SD termination status: Canceled
> > Termination: Backup Canceled
> >
> > 2017-11-27 04:43:14 bareos-sd JobId 687: Please mount append Volume
> "Full-0287" or label a new one for:
> > Job: Machine2-MySQL.2017-11-26_21.05.00_43
> > Storage: "FileStorage-Data4" (/data4/bareos)
> > Pool: Full
> > Media type: File
> > 2017-11-27 04:38:14 bareos-sd JobId 687: Warning: mount.c:250 Open
> device "FileStorage-Data4" (/data4/bareos) Volume "Full-0287" failed:
> ERR=dev.c:663 Could not open: /data4/bareos/Full-0287, ERR=No such file or
> directory
> >
> > ...
> >
> > 2017-11-27 00:48:14 bareos-sd JobId 687: Warning: mount.c:250 Open
> device "FileStorage-Data4" (/data4/bareos) Volume "Full-0287" failed:
> ERR=dev.c:663 Could not open: /data4/bareos/Full-0287, ERR=No such file or
> directory
> >
> > 2017-11-27 00:43:14 bareos-sd JobId 687: Please mount append Volume
> "Full-0287" or label a new one for:
> > Job: Machine2-MySQL.2017-11-26_21.05.00_43
> > Storage: "FileStorage-Data4" (/data4/bareos)
> > Pool: Full
> > Media type: File
> > 2017-11-27 00:38:14 bareos-sd JobId 687: Warning: mount.c:250 Open
> device "FileStorage-Data4" (/data4/bareos) Volume "Full-0287" failed:
> ERR=dev.c:663 Could not open: /data4/bareos/Full-0287, ERR=No such file or
> directory
> >
> > ...
> >
> > 2017-11-26 21:43:13 bareos-sd JobId 687: Warning: label.c:376 Open
> device "FileStorage-Data4" (/data4/bareos) Volume "Full-0287" failed:
> ERR=dev.c:663 Could not open: /data4/bareos/Full-0287, ERR=No space left on
> device
> >
> > 2017-11-26 21:43:13 bareos-dir JobId 687: All records pruned from Volume
> "Full-0287"; marking it "Purged"
> > 2017-11-26 21:43:13 bareos-dir JobId 687: There are no more Jobs
> associated with Volume "Full-0287". Marking it purged.
> > 2017-11-26 21:43:13 bareos-dir JobId 687: 1 File on Volume "Full-0287"
> purged from catalog.
> > 2017-11-26 21:43:11 bareos-dir JobId 687: Purging oldest volume
> "Full-0287"
> > 2017-11-26 21:43:11 bareos-sd JobId 687: End of medium on Volume
> "Full-0286" Bytes=1,354,941 Blocks=21 at 26-Nov-2017 21:43.
> > 2017-11-26 21:43:11 bareos-sd JobId 687: End of Volume "Full-0286" at
> 0:1354940 on device "FileStorage-Data4" (/data4/bareos). Write of 64512
> bytes got 21315.
> > 2017-11-26 21:43:11 bareos-sd JobId 687: New volume "Full-0286" mounted
> on device "FileStorage-Data4" (/data4/bareos) at 26-Nov-2017 21:43.
> > 2017-11-26 21:43:11 bareos-sd JobId 687: Wrote label to prelabeled
> Volume "Full-0286" on device "FileStorage-Data4" (/data4/bareos)
> > 2017-11-26 21:43:10 bareos-sd JobId 687: Labeled new Volume "Full-0286"
> on device "FileStorage-Data4" (/data4/bareos).
> > 2017-11-26 21:43:07 bareos-dir JobId 687: Recycled volume "Full-0286"
> > 2017-11-26 21:43:07 bareos-dir JobId 687: All records pruned from Volume
> "Full-0286"; marking it "Purged"
> > 2017-11-26 21:43:07 bareos-dir JobId 687: There are no more Jobs
> associated with Volume "Full-0286". Marking it purged.
> > 2017-11-26 21:43:07 bareos-sd JobId 687: End of medium on Volume
> "Full-0285" Bytes=35,648,683,931 Blocks=552,590 at 26-Nov-2017 21:43.
> > 2017-11-26 21:43:06 bareos-sd JobId 687: End of Volume "Full-0285" at
> 8:1288945562 on device "FileStorage-Data4" (/data4/bareos). Write of 64512
> bytes got 61541.
> > 2017-11-26 21:26:55 bareos-sd JobId 687: New volume "Full-0285" mounted
> on device "FileStorage-Data4" (/data4/bareos) at 26-Nov-2017 21:26.
> > 2017-11-26 21:26:55 bareos-sd JobId 687: Wrote label to prelabeled
> Volume "Full-0285" on device "FileStorage-Data4" (/data4/bareos)
> > 2017-11-26 21:26:55 bareos-sd JobId 687: Labeled new Volume "Full-0285"
> on device "FileStorage-Data4" (/data4/bareos).
> > 2017-11-26 21:26:55 bareos-dir JobId 687: Recycled volume "Full-0285"
> > 2017-11-26 21:26:54 bareos-dir JobId 687: All records pruned from Volume
> "Full-0285"; marking it "Purged"
> > 2017-11-26 21:26:54 bareos-dir JobId 687: There are no more Jobs
> associated with Volume "Full-0285". Marking it purged.
> > 2017-11-26 21:26:53 bareos-sd JobId 687: End of medium on Volume
> "Full-0284" Bytes=53,687,081,703 Blocks=832,204 at 26-Nov-2017 21:26.
> > 2017-11-26 21:26:53 bareos-sd JobId 687: User defined maximum volume
> capacity 53,687,091,200 exceeded on device "FileStorage-Data4"
> (/data4/bareos).
> > 2017-11-26 21:05:03 bareos-sd JobId 687: Ready to append to end of
> Volume "Full-0284" size=4857758442
> > 2017-11-26 21:05:03 bareos-sd JobId 687: Volume "Full-0284" previously
> written, moving to end of data.
> > 2017-11-26 21:05:03 Machine2-fd JobId 687: shell command: run
> ClientBeforeJob "/usr/local/bin/mysqladmin flush-tables"
> > 2017-11-26 21:05:03 Machine2-fd JobId 687: ClientBeforeJob: Slave stopped
> > 2017-11-26 21:05:03 Machine2-fd JobId 687: shell command: run
> ClientBeforeJob "/usr/local/bin/mysqladmin stop-slave"
> > 2017-11-26 21:05:03 bareos-dir JobId 687: Using Device
> "FileStorage-Data4" to write.
> > 2017-11-26 21:05:02 bareos-dir JobId 687: Start Backup JobId 687,
> Job=Machine2-MySQL.2017-11-26_21.05.00_43
>
>
--
You received this message because you are subscribed to the Google Groups
"bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to bareos-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/bareos-users/3efd812e-c0fc-4721-8d99-00983eb821e2%40googlegroups.com.
