Craig, again I return to my issue "No space left on device".
Meanwhile I found the reason: the partition ran out of inodes. As you wrote under "How much disk space do I need?" one has to have "plenty of inodes". But what does that mean? May I ask the following: - in the "General Server Information" you give some statistical information about disk usage; would it be a good idea also to give information about inode consumption? - is it possible and would it make sense to separate the "pc" and the "pool/cpool" directories into different partitions? I just did an rsync of a BackupPC-directory and found that the files on "pc" are mostly empty or small. The file sizes in "pool/cpool" are remarkable bigger - I assume these are the "real" files. So one could create one partition for "pool/cpool" having about e.g. 64kB per inode and another partition having a block size of 1 kB and also 1 kB per inode. Maybe this would reduce disk space consumption and also allow rsyncing somewhat faster. Best regards Torsten > Thanks for the additional info, but I can't explain the behavior your are > seeing. Sending to the user-list to see if others have suggestions. > > BackupPC seems to be unable to create directories and files below > /vol/bpc4/BackupPC. The "No space left on device" errno seems inconsistent > with the output of df. > > Although it won't address any of your problems, I pushed a fix > <https://github.com/backuppc/backuppc/commit/265431fff889bca2624541b05cdfbbaeae7a0039> > for the error caused by the failure to open a directory. > > Craig > > On Sat, Feb 3, 2018 at 3:56 AM, <f...@igh.de> wrote: > > > Dear Craig, > > > > On Fri, Feb 02, 2018 at 09:19:02AM -0800, Craig Barratt via BackupPC-users > > wrote: > > > Here are a few things to check: > > > > > > - can you manually create files and directories below > > /vol/bpc4/BackupPC > > > as the backuppc user (eg, "su backuppc; cd /vol/bpc4/BackupPC; mkdir > > foo; > > > rmdir foo; touch foobar; rm foobar") > > > > yes, works as expected. > > > > > - is the file system mounted readonly? > > > > no, mount says: > > ... /vol/bpc4 type ext4 (rw,noatime,data=ordered) > > > > > - has the backuppc user lost permissions? > > > > in what sense? All BackupPC processes run as user backuppc; all files > > and directories /vol/bpc4/BackupPC and below are owned by > > backuppc:backuppc. > > > > > > df says > > Filesystem Size Used Avail Use% Mounted on > > /dev/mapper/data-bpc4 1.6T 955G 535G 65% /vol/bpc4 > > > > The file system lives on a mirror raid, which seems to be ok: > > > > md127 : active raid1 sdb1[1] sdc1[0] > > 2930266432 blocks [2/2] [UU] > > > > There are no SMART errors logged on the disks (even after SMART test). > > > > I have copied several GB to the file system and removed them again - > > no errors. > > > > I have launched a forced fsck. It succeeded without any errors: > > Durchgang 1: Inodes, Blöcke und Größen werden geprüft > > Durchgang 2: Verzeichnisstruktur wird geprüft > > Durchgang 3: Verzeichnisverknüpfungen werden geprüft > > Durchgang 4: Referenzzähler werden überprüft > > Durchgang 5: Zusammengefasste Gruppeninformation wird geprüft > > /dev/data/bpc4: 104312919/104505344 Dateien (0.0% nicht > > zusammenhängend), 256909 > > 738/417993728 Blöcke > > > > > > I went through the logs. There are more error messages from a backup > > which was presented as "running" in the summary. Rather strange sound > > the messages "IO error encountered" and "No space left on device " > > (please see below). I cannot figure out the origin of these messages. > > > > ... > > G bpc_fileReadAll: can't open > > /vol/bpc4/BackupPC/cpool/20/9a/219b13a5a32e6cd66ab3cf139bd8d35c (from > > alternatives/pg_config) > > file has vanished: "alternatives/pg_config" > > ... > > > > ... > > IO error encountered -- skipping file deletion > > G bpc_attrib_dirRead: can't open > > /vol/bpc4/BackupPC/cpool/26/24/27243e59e0d4404f14ddc7ebb0202ce6 > > G bpc_attribCache_loadPath: > > bpc_attrib_dirRead(/vol/bpc4/BackupPC/pc/zuse/169, fetc/fapache2/attrib) > > returned -1 > > ... > > > > ... > > R bpc_attrib_dirRead: can't open > > /vol/bpc4/BackupPC/cpool/1e/aa/1eab8cca5e477b6fe2530e955cca3978 > > R bpc_attribCache_loadPath: > > bpc_attrib_dirRead(/vol/bpc4/BackupPC/pc/zuse/169, > > fetc/fcups/fssl/attrib) returned -1 > > [ skipped 6 lines ] > > ... > > > > > > ... > > R > > R > > bpc_sysCall_checkFileMatch(.cpan/build/DateTime-Locale-1. > > 05-GcMGii/pm_to_blib): > > file doesn't exist > > ... > > > > ... > > R bpc_poolWrite_write: can't open/create > > /vol/bpc4/BackupPC/cpool/18801.736.0 for writingIOdone: pool > > .cpan/build/DateTime-Locale-1.05-GcMGii/blib/lib/DateTime/ > > Locale/.en_NR.pod.000000 > > ... > > > > ... > > R > > bpc_poolWrite_unmarkPendingDelete(/vol/bpc4/BackupPC/cpool/50/02/ > > 50032f7b52be451f6a6c396067253906) > > failed; errno = 2 > > R Couldn't unmark candidate matching file > > /vol/bpc4/BackupPC/cpool/50/02/50032f7b52be451f6a6c396067253906 > > (skipped; errno = 2) > > ... > > > > > > The following is very strange: > > ... > > rsync_bpc: recv_generator: mkdir > > "/.cpan/build/Dist-CheckConflicts-0.11-RNgNPa/t/lib" failed: No space > > left on device (28) > > ... > > > > dumpe2fs says: > > > > ... > > Inode count: 104505344 > > Block count: 417993728 > > Reserved block count: 20899686 > > Free blocks: 243883988 > > Free inodes: 97119479 > > ... > > > > > > > > Best regards > > > > > > Torsten Finke > > > > > > > > > > > It does look like the failure to update the filesystem exposes a bug > > which > > > I'll fix, but this is secondary to your main issue: > > > > > > Can't use an undefined value as an ARRAY reference at > > > /vol/opt/BackupPC-4.1.3/bin/BackupPC_refCountUpdate line 380 > > > > > > > > > Craig > > > > > > On Fri, Feb 2, 2018 at 5:56 AM, <f...@igh.de> wrote: > > > > > > > Dear BackupPC Users, > > > > > > > > > > > > I am running BackupPC-4.1.3 for several months without problems > > > > (really a great piece of software). > > > > > > > > About a week ago a strange type of error occured and keeps staying, > > > > which I cannot understand. The error messages are like this ("fs" is > > > > the name of one of my systems to be backed up; I did some linebreaks > > > > to increase readability): > > > > > > > > > > > > 2018-01-28 23:43:34 Can't create /vol/bpc4/BackupPC/pc/fs/149/refCnt > > > > 2018-01-29 02:09:34 Serious error: last backup > > > > /vol/bpc4/BackupPC/pc/fs/149 directory doesn't exist!!! Need to > > > > remove back to last filled backup > > > > 2018-01-29 02:09:34 Deleting backup 147 > > > > 2018-01-29 02:09:35 BackupPC_backupDelete: removing #147 > > > > 2018-01-29 02:09:35 BackupPC_backupDelete: No prior backup for merge > > > > 2018-01-29 02:09:52 BackupPC_refCountUpdate: doing fsck on fs #149 > > > > since /vol/bpc4/BackupPC/pc/fs/149/refCnt doesn't exist > > > > 2018-01-29 02:09:52 Can't use an undefined value as an ARRAY reference > > > > at /vol/opt/BackupPC-4.1.3/bin/BackupPC_refCountUpdate line 380. > > > > 2018-01-29 02:09:52 BackupPC_backupDelete: got 1 errors > > > > 2018-01-29 02:09:52 Finished BackupPC_backupDelete, status = 256 > > > > (running time: 18 sec) > > > > 2018-01-29 02:09:52 Deleting backup 148 > > > > 2018-01-29 02:09:52 BackupPC_backupDelete: removing #148 > > > > 2018-01-29 02:09:52 BackupPC_backupDelete: No prior backup for merge > > > > 2018-01-29 02:09:55 BackupPC_refCountUpdate: doing fsck on fs #149 > > > > since /vol/bpc4/BackupPC/pc/fs/149/refCnt doesn't exist > > > > 2018-01-29 02:09:55 Can't use an undefined value as an ARRAY reference > > > > at /vol/opt/BackupPC-4.1.3/bin/BackupPC_refCountUpdate line 380. > > > > 2018-01-29 02:09:55 BackupPC_backupDelete: got 1 errors > > > > 2018-01-29 02:09:55 Finished BackupPC_backupDelete, status = 256 > > > > (running time: 3 sec) > > > > 2018-01-29 02:09:55 Deleting backup 149 > > > > 2018-01-29 02:09:56 BackupPC_backupDelete: removing #149 > > > > 2018-01-29 02:09:56 BackupPC_backupDelete: No prior backup for merge > > > > 2018-01-29 02:38:20 BackupPC_refCountUpdate: host fs got 0 errors > > > > (took 1704 secs) > > > > 2018-01-29 02:38:20 Finished BackupPC_backupDelete, status = 0 > > > > (running time: 1705 sec) > > > > 2018-01-29 03:16:29 dump failed: unable to open/create > > > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z > > > > 2018-01-29 03:55:15 dump failed: unable to open/create > > > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z > > > > 2018-01-29 04:15:15 dump failed: unable to open/create > > > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z > > > > 2018-01-29 23:43:54 dump failed: unable to open/create > > > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z > > > > 2018-01-30 01:24:02 dump failed: unable to open/create > > > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z > > > > 2018-01-30 02:36:01 dump failed: unable to open/create > > > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z > > > > 2018-01-30 03:42:01 dump failed: unable to open/create > > > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z > > > > 2018-01-30 04:12:01 dump failed: unable to open/create > > > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z > > > > 2018-01-30 23:57:53 Copying backup #146 to #147 > > > > > > > > 2018-01-31 01:05:38 Can't mkpath > > > > /vol/bpc4/BackupPC/pc/fs/147/./fsysbak/f.../fLocked/f00 > > > > 2018-01-31 01:05:38 Can't copy > > > > ./fsysbak/f.../fLocked/f00/attrib_dc793dc4650baa1a8cc3b69d211573c1 > > > > to /vol/bpc4/BackupPC/pc/fs/147/./fsysbak/f.../fLocked/f00/attrib_ > > > > dc793dc4650baa1a8cc3b69d211573c1 > > > > ... > > > > > > > > The last pair of messages is then repeated for thousands of files. > > > > > > > > > > > > It seems, that there is an error concerning the sequence. > > > > > > > > I had checked for several default reasons: > > > > - No files or directories have been manipulated manually > > > > - BackupPC is almost the only task running on my backup server (there > > > > should be no side effects from other services; no user is normally > > > > working on that machine) > > > > - the file system has about 1.6 TB at a usage of 65% (this rate was > > > > never exceeded) > > > > - the system had no crash (uptime is 217 days, since setup of > > > > BackupPC). > > > > - /vol/bpc4 is a mount point of an ext4 file system (normally very > > > > reliable) > > > > - Kernel 4.1.39, OpenSuSE 42.1 - not really outdated > > > > > > > > > > > > > > > > > > > > Any advice to repair this error is highly appretiated. > > > > > > > > > > > > Best regards > > > > > > > > > > > > Torsten Finke > > > > > > > > > > > > > > -- > > ------------------------------------------------------------------------ > > Torsten Finke > > f...@igh.de > > > > Ingenieurgemeinschaft IgH > > Gesellschaft für Ingenieurleistungen mbH > > Heinz-Bäcker-Str. 34 > > D-45356 Essen > > > > ------------------------------------------------------------------------ > > > > ------------------------------------------------------------ > > ------------------ > > Check out the vibrant tech community on one of the world's most > > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > _______________________________________________ > > BackupPC-users mailing list > > BackupPC-users@lists.sourceforge.net > > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users > > Wiki: http://backuppc.wiki.sourceforge.net > > Project: http://backuppc.sourceforge.net/ > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki: http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ -- ------------------------------------------------------------------------ Dr.-Ing. Torsten Finke f...@igh.de Ingenieurgemeinschaft IgH Gesellschaft für Ingenieurleistungen mbH Heinz-Bäcker-Str. 34 D-45356 Essen ------------------------------------------------------------------------ ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
