The backup is large, but not huge. I created a new test host pointing
only to /home/raman/x and it worked just fine. I forgot to mention I
also have another BackupPC server, backing up the same client with the
same configuration, and it backs up all of these files from this
client without any issue, so the issue is definitely on the server
side, not on the client side.
I ran a test with XferLogLevel=6 and "-vvv". Here are some relevant
grep outputs -- `home/raman/x/y/2018` is a directory created in 2018,
while all of the previously created directories in `y` are backing up
without any issue --- this happens to be tax data so I have one
directory for every year, and all 2015 directories and before are
fine, but 2016 and later are not:
$ BackupPC_zcat /var/lib/BackupPC/pc/edison/XferLOG.3972.z | grep "x/y/2018"
[sender] showing directory home/raman/x/y/2018 because of pattern /home/**
[sender] make_file(home/raman/x/y/2018,*,2)
recv_file_name(home/raman/x/y/2018)
recv_file_name(home/raman/x/y/2018)
G bpc_lstat(home/raman/x/y/2018)
file has vanished: "/home/raman/x/y/2018"
recv_generator(home/raman/x/y/2018,216871)
G bpc_lstat(home/raman/x/y/2018)
G bpc_mkdir(home/raman/x/y/2018, 0700)
[sender] pushing local filters for /home/raman/x/y/2018/
[sender] showing file home/raman/x/y/2018/a.txt.pdf because of pattern /home/**
[... same thing for all other subdirectories and files ...]
recv_file_name(home/raman/x/y/2018/a.txt.pdf)
[... same thing for all other subdirectories and files ...]
recv_generator(home/raman/x/y/2018,217359)
G bpc_lstat(home/raman/x/y/2018)
G bpc_mkdir(home/raman/x/y/2018, 040775)
G bpc_lstat(home/raman/x/y/2018)
rsync_bpc: stat "/home/raman/x/y/2018" failed: No such file or directory (2)
delete_in_dir(home/raman/x/y/2018)
[generator] pushing local filters for /home/raman/x/y/2018/
G bpc_opendir(home/raman/x/y/2018)
G bpc_lstat(home/raman/x/y/2018/a.txt.pdf)
file has vanished: "/home/raman/x/y/2018/a.txt.pdf"
[... same thing for all other subdirectories and files ...]
recv_generator(home/raman/x/y/2018/a.txt.pdf,217366)
G bpc_lstat(home/raman/x/y/2018/a.txt.pdf)
G bpc_sysCall_poolFileCheck(home/raman/x/y/2018/a.txt.pdf): potential
match /var/lib/BackupPC//cpool/28/56/29571824e1fe8922304bd1924f02ce22
(len = 171990)
G bpc_lstat(home/raman/x/y/2018/a.txt.pdf)
G bpc_open(home/raman/x/y/2018/a.txt.pdf, 0x0, 00) -> 3
G bpc_lgetxattr(home/raman/x/y/2018/a.txt.pdf, user.rsync.%aacl)
G bpc_llistxattr(home/raman/x/y/2018/a.txt.pdf)
G bpc_read(3 (home/raman/x/y/2018/a.txt.pdf), buf, 171990) tmpFd = -1
G bpc_close(3 (home/raman/x/y/2018/a.txt.pdf))
[... same thing for all other subdirectories and files ...]
and some context around the "file has vanished" message:
$ BackupPC_zcat /var/lib/BackupPC/pc/edison/XferLOG.3972.z | grep -C
5 'file has vanished: "/home/raman/x/y/2018"'
G bpc_readdir -> 1996
G bpc_lstat(home/raman/x/y/1996)
[generator] make_file(home/raman/x/y/1996,*,2)
G bpc_readdir -> 2018
G bpc_lstat(home/raman/x/y/2018)
file has vanished: "/home/raman/x/y/2018"
G bpc_readdir -> 2014
G bpc_lstat(home/raman/x/y/2014)
[generator] make_file(home/raman/x/y/2014,*,2)
G bpc_readdir -> 2011
G bpc_lstat(home/raman/x/y/2011)
Lastly, I've tried a test of placing a new file into the 2015
directory, and this file does back up successfully. So it seems like
new files are captured, as long as they fall into an existing
directory, but new directories are not.
One thing I did do on this server is cancel the first full backup
after removing `--one-file-system` and deleted the backup from the web
UI, whereas the other server completed its backup without
intervention. I wonder if the server pool was corrupted somehow by
doing that? Also, the `--one-file-system` was probably added at the
same time as v4 and the resulting pool migrations on subsequent
backups, which would have *removed* a lot of stuff that was being
backed up by v3. I have run a BackupPC_fsck and there were no errors.
Regards,
Raman
On Wed, May 8, 2019 at 2:11 AM Craig Barratt via BackupPC-users
<backuppc-users@lists.sourceforge.net> wrote:
>
> Thanks for confirming; it's not a charset issue then.
>
> The empty /home due to --one-file-system from 2017 shouldn't make a
> difference by now, but it's good to keep in mind.
>
> When you look through the XferLOG file, please look out for any other
> unexpected errors. You could also increase the XferLogLevel (eg, to 5 or 6).
>
> Also, if the backup size is large, you could create a new test host (use
> ClientNameAlias to point it back to the right host) that just backs up
> /home/raman/x or even /home/raman/x/y. If the same problem happens, then it
> will be much easier to browse shorter log files. If the problem doesn't
> happen in that case, then that's a useful clue too.
>
> Craig
>
> On Tue, May 7, 2019 at 11:02 PM Raman Gupta <rocketra...@gmail.com> wrote:
>>
>> It is an ext4 filesystem. The directories are plain ASCII -- no
>> strange characters in any way. The `rsync` process on the Linux client
>> runs as root, and I have verified root has access to these files
>> without any issue. There are plenty of inodes free (`df -i` shows that
>> filesystem as only 5% used). There is no file corruption -- all the
>> data is good.
>>
>> There is a seemingly weird coincidence here though... my backups have
>> been running nightly for years. However, in late 2017 I updated
>> BackupPC on my Fedora box, and `--one-file-system` was added to the
>> rsync args without me realizing. This caused all backups from that
>> point forward to a few days ago to be missing all the new files and
>> modifications in `/home`, which is a mountpoint for a local LVM ext4
>> partition. It seems that every file and directory that falls into this
>> category are now failing with this "vanished" error. Is it possible
>> that BackupPC is confused because it is expecting to find these files
>> in prior backups?
>>
>> I will try the debugging you suggest.
>>
>> Regards,
>> Raman
>>
>> On Wed, May 8, 2019 at 1:44 AM Craig Barratt via BackupPC-users
>> <backuppc-users@lists.sourceforge.net> wrote:
>> >
>> > What sort of filesystem is this? Do those directory names contain
>> > non-ascii characters?
>> >
>> > However, the "file has vanished" error shouldn't occur on a directory, so
>> > something strange is going on.
>> >
>> > I'd recommend turning on additional debug in rsync (eg, add -vvv to
>> > $Conf{RsyncArgs}, and also look at the --debug option) and looking in the
>> > XferLOG file. When the initial file list is sent, are those directories
>> > and their contents present in the file list?
>> >
>> > Craig
>> >
>> > On Tue, May 7, 2019 at 4:26 PM Michael Stowe
>> > <michael.st...@member.mensa.org> wrote:
>> >>
>> >> On 2019-05-07 13:39, Raman Gupta wrote:
>> >>
>> >> Certain directories (and their contents) on one of my hosts are not
>> >> getting backed up at all, even with a “Full” backup.
>> >>
>> >> I use rsync as my Xfer method, with BackupPC 4.3.0 on Fedora (rpms
>> >> BackupPC-4.3.0-1.fc29.x86_64, BackupPC-XS-0.58-1.fc29.x86_64).
>> >>
>> >> Looking at the backup logs, I see messages like the following related to
>> >> the directories that are not being backed up:
>> >>
>> >> file has vanished: “/home/raman/x/y/a” file has vanished:
>> >> “/home/raman/x/y/b” file has vanished: “/home/raman/x/y/c”
>> >>
>> >> I have other directories and files successfully backed up in
>> >> “/home/raman/x/y”, but the directories “a”, “b”, and “c” (and their
>> >> content) are not being backed up.
>> >>
>> >> Note that these files have not vanished — they are not ephemeral and they
>> >> haven't been touched in days. For example:
>> >>
>> >> File: /home/raman/x/y/a
>> >> Size: 4096 Blocks: 8 IO Block: 4096 directory
>> >>
>> >> Device: fd08h/64776d Inode: 33037482 Links: 5 Access: (0775/drwxrwxr-x)
>> >> Uid: ( 1000/ raman) Gid: ( 1000/ raman) Context:
>> >> unconfined_u:object_r:user_home_t:s0 Access: 2019-05-07
>> >> 05:05:17.288857497 -0400 Modify: 2019-04-30 00:56:22.914849594 -0400
>> >> Change: 2019-04-30 00:56:22.914849594 -0400 Birth: –
>> >>
>> >> Any idea what might be happening here?
>> >>
>> >> Regards, Raman
>> >>
>> >> “File has vanished” issues can be tricky to diagnose if the file appears
>> >> to be there. What rsync is really telling you is that it built a file
>> >> list, and some of the files or directories from that list are not
>> >> accessible when it actually went to read them. Actually being deleted or
>> >> ephemeral files are two reasons, but there are others, from filename
>> >> encoding issues to inode changes to complications with remotely mounted
>> >> filesystems to corruption issues to complex file permissions.
>> >>
>> >> While I might check the file's details both before and after the rsync
>> >> run to look for changes, I recommend ensuring that these files are
>> >> reliably accessible by the rsync user, check the logs for any problems,
>> >> and working through filesystem issues. (XFS is notorious for this sort of
>> >> thing.) Also, if the volume is anything other than a local mount, that's
>> >> where I'd look first for issues; be aware that rsync's high read volume
>> >> often exposes issues not evident under less stressful usage.
>> >>
>> >> _______________________________________________
>> >> BackupPC-users mailing list
>> >> BackupPC-users@lists.sourceforge.net
>> >> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> >> Wiki: http://backuppc.wiki.sourceforge.net
>> >> Project: http://backuppc.sourceforge.net/
>> >
>> > _______________________________________________
>> > BackupPC-users mailing list
>> > BackupPC-users@lists.sourceforge.net
>> > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> > Wiki: http://backuppc.wiki.sourceforge.net
>> > Project: http://backuppc.sourceforge.net/
>>
>>
>> _______________________________________________
>> BackupPC-users mailing list
>> BackupPC-users@lists.sourceforge.net
>> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> Wiki: http://backuppc.wiki.sourceforge.net
>> Project: http://backuppc.sourceforge.net/
>
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
