Re: [BackupPC-users] server log : thousands of BackupPC_refCountUpdate: missing pool file
Hi
We noticed the log message about missing pool files appear in the log
mostly if there was an improper shutdown of the BackupPC Server while a
backup was in progress.
We developed the following script to get the affected host and backup
number for each reported missing pool file. The only remedy we found to
get rid of the error is to delete the affected backups. Please let me
know if you find any other remedy.
The way it works is that it extracts the hash of the missing pool file,
subtracts one from the first two letters of the hex value to find the
correct subfolder in the pool and then correlates that with a backup and
a host.
In the following script I think most paths are the default for a rpm
based BackupPC installation but adapt the paths to the installation on
your system. In particular
- the path to the BackupPC log file
- the path to your pool
#!/bin/bash
#
# Script for investigation of missing pool files
#
LOG_FILE_NAME="/home/backuppc/find_missing_pool_files_$(date
+%Y%m%d-%H:%m).log"
echo Search for missing pool files started. Logs will be written to
"$LOG_FILE_NAME"
MISSING_POOL_FILES_HASHES=$(grep 'missing pool' /var/log/BackupPC/LOG |
cut -d' ' -f10)
for full_hex in $MISSING_POOL_FILES_HASHES; do
hex_val=${full_hex:0:2}
hex_val_decr=$(printf "%X" $((0x$hex_val - 1)) | awk '{print
tolower($0)}')
paths="/srv/backuppc/pc/*/*/refCnt/poolCnt.1.$hex_val_decr"
echo "$hex_val - 1 = $hex_val_decr"
for path in $paths; do
echo "$path"
if test -f "$path"; then
echo one found
pool_cnt_print=$(/usr/share/BackupPC/bin/BackupPC_poolCntPrint "$path")
pool_cnt_match=$(echo "$pool_cnt_print" | grep "$full_hex")
if [ -n "$pool_cnt_match" ]; then
host=$(echo "$path" | cut -d'/' -f5)
backup_number=$(echo "$path" | cut -d'/' -f6)
echo Found "$full_hex" in Backup Nr. "$backup_number"
of host "$host" | tee -a "$LOG_FILE_NAME"
/usr/share/BackupPC/bin/BackupPC_ls -R -h "$host" -n
"$backup_number" -s / / | grep -E "$full_hex"
fi
fi
done
done
--
Tim Herren
System Engineer
On 12/11/23 05:34, Samual Flossie wrote:
Thanks for looking deeper. I agree that these missing pool file
warnings are ominous news for recovery. The link you referenced (
https://www.mail-archive.com/backuppc-users@lists.sourceforge.net/msg33090.html)
indicates to me also that in v4, the missing pool file message is not
benign.
It will take me at least a few days to run a full recovery analysis to
see if there are file recovery issues.
Do you know how to get the file/backup#/host from a pool file
reference that was reported missing in the log? That detail would
certainly help speed up my analysis and potential recovery.
I’ll update once that’s done and have some actionable data.
Sam
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/
[BackupPC-users] Missing pool files cause and remediation
Hi I'm troubleshooting an issue at work where our BackupPC4 Server showed logs about missing pool files. The Server is baremetal, has 32GB of RAM and 8 Cores. A day after a scheduled reboot of the server during a regular maintenance we start noticing the "missing pool file" messages in the logs. 2021-04-23 01:00:00 BackupPC_nightly now running BackupPC_refCountUpdate -m -s -c -P 7 -r 128-255 2021-04-23 01:00:00 BackupPC_nightly now running BackupPC_refCountUpdate -m -s -c -P 7 -r 0-127 2021-04-23 01:00:00 admin1 : __bpc_pidStart__ 18306 2021-04-23 01:00:00 admin : __bpc_pidStart__ 18307 2021-04-23 01:00:44 admin1 : BackupPC_refCountUpdate: missing pool file 81e5e7a28a0f857827a9c2de6099875601 count 1 2021-04-23 01:01:18 admin1 : BackupPC_refCountUpdate: missing pool file 83b7b97de222ecb2f2a129233d63bad801 count 1 2021-04-23 01:01:18 admin1 : BackupPC_refCountUpdate: missing pool file 82020853cb8c2718ac4ed58b2541c58c01 count 1 However the shutdown of the server during the maintenance was clean and BackupPC exited with exit-code 0 and no apparent job running. 2021-04-22 18:17:11 Got signal TERM... cleaning up (exit code = 0) 2021-04-22 18:19:34 Reading hosts file 2021-04-22 18:19:34 BackupPC 4.4.0 (Perl v5.16.3) started, pid 887 2021-04-22 18:19:34 Next wakeup is 2021-04-22 20:00:00 In total I have 56 entries of missing pool files in my logs. With a quick script we managed to find the files that are missing from the pool. The missing pool files all stem from 2 servers (out of 81) and there from 3 different backups. All missing files are quite similar and are located within /usr/lib/modules/4.18.0-240.22.1.el8_3.x86_64/kernel/drivers/ so somehow related. Does anyone have an idea what could have led to those files being missing from the pool? Is there any way to add those files back to the pool to get rid of the messages in the log. The files are still present on the servers. Thanks for any insights. Tim Herren Systems Technician Puzzle ITC GmbH ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
Re: [BackupPC-users] installation help
t; CONFIG_XFS_POSIX_ACL=y >> >> CONFIG_BTRFS_FS_POSIX_ACL=y >> >> CONFIG_FS_POSIX_ACL=y >> >> CONFIG_GENERIC_ACL=y >> >> CONFIG_TMPFS_POSIX_ACL=y >> >> CONFIG_NFS_V3_ACL=y >> >> CONFIG_NFSD_V2_ACL=y >> >> CONFIG_NFSD_V3_ACL=y >> >> CONFIG_NFS_ACL_SUPPORT=m >> >> CONFIG_CEPH_FS_POSIX_ACL=y >> >> CONFIG_CIFS_ACL=y >> >> >> >> Am I making a mistake in installing the current version? Perhaps I’d have >> better results installing the version identified in the Debian-based script? >> >> Thank you! >> >> *From:* Craig Barratt via BackupPC-users < >> backuppc-users@lists.sourceforge.net> >> *Sent:* Wednesday, August 8, 2018 00:37 >> *To:* backuppc-users@lists.sourceforge.net >> *Cc:* Craig Barratt >> *Subject:* Re: [BackupPC-users] installation help >> >> >> >> Mike, >> >> >> >> You have to build and install backuppc-xs first. Also, makeDist needs a >> --version argument. >> >> >> >> The wiki has an example script >> <https://github.com/backuppc/backuppc/wiki/Installing-BackupPC-4-from-git-on-Ubuntu-Xenial-16.04-LTS> >> for building from git. >> >> >> Craig >> >> >> >> On Tue, Aug 7, 2018 at 10:26 PM Mike Hughes wrote: >> >> I am new to git-based installations and need some help. I cloned the three >> projects and I cd into the backuppc folder to run perl configure.pl and >> it replies with: >> >> [Cent-7:root@hostname backuppc]# perl configure.pl >> >> You need to run makeDist first to create a tarball release that includes an >> >> updated configure.pl. After you unpack the tarball, run configure.pl from >> >> there. >> >> >> >> Not sure why I’d need to generate a tarball but when I run makeDist I get: >> >> [Cent-7:root@ hostname backuppc]# perl makeDist >> >> Can't locate BackupPC/XS.pm in @INC (@INC contains: ./lib >> /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl >> /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at >> lib/BackupPC/Lib.pm line 51. >> >> BEGIN failed--compilation aborted at lib/BackupPC/Lib.pm line 51. >> >> Compilation failed in require at makeDist line 59. >> >> BEGIN failed--compilation aborted at makeDist line 59. >> >> >> >> >> >> >> ---------- >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> ___ >> BackupPC-users mailing list >> BackupPC-users@lists.sourceforge.net >> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users >> Wiki:http://backuppc.wiki.sourceforge.net >> Project: http://backuppc.sourceforge.net/ >> >> >> -- >> Check out the vibrant tech community on one of the world's most >> engaging tech sites, Slashdot.org! http://sdm.link/slashdot >> ___ >> BackupPC-users mailing list >> BackupPC-users@lists.sourceforge.net >> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users >> Wiki:http://backuppc.wiki.sourceforge.net >> Project: http://backuppc.sourceforge.net/ >> >> > > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > > > > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > -- Tim Herren System Technician Puzzle ITC GmbH www.puzzle.ch Telefon +41 31 370 22 00 Mobile +41 76 211 26 06 Werfen Sie einen Blick in unseren Blog: https://www.puzzle.ch/de/blog -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Fwd: Re: BackupPC v4 for Fedora / EPEL Update
On 03.04.2017 01:25, Adam Goryachev wrote: > On 03/04/17 06:50, Bob of Donelson Trophy wrote: >> I have updated to your latest COPR and v4.1.1. Thank you for that. >> >> I have been researching signing into the web gui. Even before I >> upgraded to v4.1.1 I was having resistance to signing in with the >> backuppc user with the htpasswd I have set up. >> >> I mentioned in a previous email that the backuppc user appears in the >> /etc/BackupPC/apache.users file, as it should. But, When I sign into >> the [ipaddress]/BackupPC it gives an "Internal Server Error". If I >> modify the /etc/httpd/conf.d/BackupPC.conf file and add "Require all >> granted" and comment out the "Require valid-user" I can access the gui >> however, I cannot do much of anything. Understandable as BackupPC does >> not "know" the user I am accessing it with. When I invert this back to >> the correct configuration I am NOT presented with a user login but >> rather the "Internal Server Error". >> >> What file permissions rights should the /etc/Backup[PC/apache.users >> file have? >> >> [root@localhost ~]# ls -alh /etc/BackupPC/ >> total 192K >> drwxr-x---. 2 backuppc backuppc 112 Apr 1 06:55 . >> drwxr-xr-x. 80 root root 8.0K Apr 1 07:42 .. >> -rw-r--r--. 1 root root 47 Apr 1 18:14 apache.users >> -rw-r--r--. 1 backuppc backuppc 83K Mar 31 13:22 config.pl >> -rw-r--r--. 1 backuppc backuppc 83K Mar 31 13:22 config.pl.sample >> -rw-r--r--. 1 backuppc backuppc 2.2K Mar 31 13:22 hosts >> -rw-r--r--. 1 backuppc backuppc 2.2K Mar 31 13:22 hosts.sample >> -rw-r-. 1 backuppc backuppc0 Apr 1 17:53 LOCK >> >> Does your /etc/BackupPC/ directory have these rights? >> >> I am not sure I am looking in the correct place but, I think I am >> dealing with a permissions issue. >> >> Thoughts? >> > > Yes, please look at your apache error log file, it will tell you what > the problem is. > Basically, Apache is saying "I don't like this, and I don't know what > else to do, please get the sys admin to fix it". That's if it's a > permissions error on any of the related files (.htaccess, the password > file, or syntax errors, or options that are not permitted etc. > > It is also possible to get an internal server error if the > script/program itself can't run, usually the output of the script (error > message) will be recorded in the apache error log, so again, that is > where you will find out what the problem is. > > Regards, > Adam > Hi, I had the same problem you had. All I got when browsing to http://ipadress/BackupPC was a Error 500 - Internal Server Error. I did the installation from the rpm on a clean install of minimal CentOS 7. Your permissions in /etc/BackupPC/ look good to me, I have the same permissions. What solved it for me is that I changed the "User" and "Group" in /etc/httpd/conf/httpd.conf from "apache" to "backuppc" followed by a "systemctl restart httpd" After that all worked fine. I am not sure what would be the correct steps if you use apache on the same Host for anything else besides BackupPC. Regards, Tim -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
