[BackupPC-users] BackupPC 4.4.0: Incremental smb backup is skipping not backed-up files under certain conditions
Hi, My Problem: When doing an incremental backup with smb, only files with mtime newer than the date and time of the last backup are being backed up. To reproduce: If I copy a file with an old ctime, the file is not included in the backup even if it is not contained in the previous complete backup. Only the timestamp of the mtime seems to be relevant - even if the file is not present in any backup so far. I am not sure, if this behavior is intended or if I have misconfigured smb backup settings. The problem would be solved with the next complete backup. However, due to my circumstances, I can rarely do a full backup. And only with rsyncd, because smbclient aborts after a while, because I have to backup lots of files (> 1.000.000), which takes usually more than a day. (However, not due to configurable time limits, which I extended generously.) An incremental backup with rsyncd unfortunately takes almost as long as the full backup, so I need to do the incremental backups with smbclient. This takes only minutes (with only a few changed files), but misses new files with an old date. Any help is greatly appreciated! Regards, Jens smime.p7s Description: S/MIME cryptographic signature ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:https://github.com/backuppc/backuppc/wiki Project: https://backuppc.github.io/backuppc/
[BackupPC-users] Direct restore broken with BackupPC 4.1.x and SMB3?
Hello list!
After many hours of tests and research, I need your help.
Here is a short-as-possible report of what is happening and what I've done
already.
Actually, I have three problems, but let's try to get the first one fixed
first for a start.
Why read any further?
=
If my environment is similar to your environment, you should test a direct
restore operation on your hosts, just to be sure to have a valid
backup-system in case of emergency. (Ok, restore to zip/tar does work :-))
My environment
==
Backup-Server: BackupPC 4.1.5 running on Debian 8.10 with smbclient 4.2.14
Host: Windows Server 2012 R2
The situation
=
Backing up is no problem*, but doing a direct restore is impossible.
What happens?
=
Using the (default) configuration for smb restore:
$Conf{SmbClientRestoreCmd} = '$smbClientPath $host\\$shareName
$I_option -U $userName -E -d 1 -c tarmode\\ full -mSMB3 -Tx -';
The operation stops with:
> tar:316 tarmode is now full, system, hidden, noreset, quiet
> tar:1596 Can't mkdir test: NT_STATUS_OBJECT_NAME_COLLISION
What I've done and checked so far
=
* As BackupPC user, I can connect to the windows host in an interactive
smbclient-session.
* While in smbclient-session: I'm able to create directories and put files
on the host.
* While in smbclient-session: Trying to create a directory, that already
exists, I get the same error as above, what seems legit, since the directory
exists.
* Swapped my smb.conf with the vanilla smb.conf
* Extracting files with BackupPC_tarCreate locally does work.
* Piping BackupPC_tarCreate-command to smbclient-command manually and change
path options: if the new path does not exists, it is successfully created,
but then instantly stops with the error as seen above (since BackupPC tries
to create the path as soon as it tries to put the actual file in the new
directory).
* Checked permissions for backuppc-smb-user on windows domain/filesystem.
Although this was not likely, since smbclient-sessions do work.
* Checked another environment: BackupPC 4.1.x on Debian 9.2 and Windows
Server 2012 R2 as host on a different subnet - same error occurs. This
system is only a couple of weeks old and does not serve any other purpose.
This is, why I think that other users are affected, too.
Possible solution
=
Let BackupPC continue to work, if creation of a directory fails *just
because* it already exists. Alternatively do not try to create a directory,
if it already exists. (If smbclient is responsible for breaking the
operation, please guide me to someone to contact.)
*Since my setup has undergone some heavy migration work (moved from BackupPC
3.x to 4.x, moved my pool from v3 to v4), I have two other points to check
that might be something only existing for my setup:
- Filenames get chopped off after 97-99 characters. It's visible on the
web-frontend, if you navigate to a file in a backup.
- Some directories do net get backed up (yes, I checked permissions)
Any input is really appreciated!
Regards,
Jens
smime.p7s
Description: S/MIME cryptographic signature
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Trouble with restore
Hi, Still no progress on this one. Here is a wrap-up of what I have tried so far: - Restoring (with smb) to the same or another host: operation breaks, as soon as a file is restored in an existing directory. - Restoring (with smb) to a new directory (same or different host): directory is created, the first file is restored and then restore fails. - When I mount my shares manually as 'backuppcuser', I am able to copy/change files on Windows hosts. That should rule out any permission issues. - No Warnings or errors in SMB-Log-files on Windows hosts. - Restoring using ssh, zip or tar works as expected. I had Archive::Zip missing. Since I never used zip, it went unnoticed. What I need right now: Someone with a similar configuration/environment testing a restore (with smb) operation. My System: Debian 8.10, BackupPC 4.1.5, Hosts: Windows Server 2012 R2, Windows 7 x64 (via SMB3). Regards, Jens -Original Message- From: Jens Potthast [mailto:jens.potth...@innovation.uni-bremen.de] Sent: Mittwoch, 13. Dezember 2017 14:02 To: backuppc-users@lists.sourceforge.net Subject: [BackupPC-users] Trouble with restore Hello, using BackupPC for several years, I needed to restore a couple of files recently. The last time, I had to restore some files was years ago. At that time it was no problem. Right now, restore fails: Running: /usr/bin/smbclient server\\sharename -U backuppcuser -E -d 1 -c tarmode\ full -mSMB3 -Tx - Running: /usr/local/BackupPC/bin/BackupPC_tarCreate -h hostname -n 1886 -s sharename -t -r /Path\ to\ file -p /Path\ to\ file/filename.txt Xfer PIDs are now 2600,2601 Domain=[MYDOMAIN] OS=[] Server=[] tar:316 tarmode is now full, system, hidden, noreset, quiet tarCreate: Done: 1 files, 237695 bytes, 0 dirs, 0 specials, 0 errors tar:1596 Can't mkdir Path to filename: NT_STATUS_OBJECT_NAME_COLLISION readOutput: sysread returns 0 and got EOF (exit ok = , ) XferErr Non-zero exit status from smbclient restore failed: Non-zero exit status from smbclient What is strange: if I redirect the files to another directory, one folder or file is created before the operation fails. That should rule out all permission problems with my (Windows 2008 R2) Servers Share. I checked permissions of backuppc-user on the server, just in case. However, permissions are ok. If the directory is not there, the directory and one file is created, is the directory already there, no file is created. Therefore, my guess is that the return value is not, what BackupPC is expecting. Upping XferLogLevel does not increase log file output. I am not sure, if this is connected, but I cannot restore to zip also. Restore to tar works. Therefore, I am not completely lost right now. System: BackupPC 4.1.5, BackupPC-XS 0.57, rsync-bpc-3.0.9.9 [Debian 8.x] Any clues where to look or how to increase debug level? Regards, Jens smime.p7s Description: S/MIME cryptographic signature -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Trouble with restore
Thanks for the explanation. I'll see, if I can check permissions and logs on my Server and report back tomorrow. -Original Message- From: B [mailto:lazyvi...@gmx.com] Sent: Mittwoch, 13. Dezember 2017 15:43 To: backuppc-users@lists.sourceforge.net Subject: Re: [BackupPC-users] Trouble with restore On Wed, 13 Dec 2017 15:21:09 +0100 "Jens Potthast" <jens.potth...@innovation.uni-bremen.de> wrote: > Ok, I don't understand, what you are talking about. Sorry. >From some Samba ML posts, this error code means the server command is not able to either create the destination directory OR can't write in it. But it is Samba, not m$. > BackupPC does create a directory, if it does not already exists. If it > does, it stops with an error. It restores the first file and then it > stops. Hmm, have a look at w$ logs, if they are correctly set (? duno if it is even possible to change the log level into nsa self-service), they may reflect what the real problem is on your client as m$ is also known to sometimes group several errors under only one error code. > So, if neither directory nor files do exists, why would restore fail? I had some permissions problems in the past on a friend's Linux/w$ installation (but didn't took notes, as I do not use w$ anymore and he switched to a w$ freeware solution :/) - IIRC, there was a user substitution to the "system" user (or from, I don't remember) that clobbered the whole restoration process or something close to that. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ smime.p7s Description: S/MIME cryptographic signature -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Trouble with restore
Ok, I don't understand, what you are talking about. Sorry. BackupPC does create a directory, if it does not already exists. If it does, it stops with an error. It restores the first file and then it stops. So, if neither directory nor files do exists, why would restore fail? Anyway, thanks for helping! -Original Message- From: B [mailto:lazyvi...@gmx.com] Sent: Mittwoch, 13. Dezember 2017 14:57 To: backuppc-users@lists.sourceforge.net Subject: Re: [BackupPC-users] Trouble with restore On Wed, 13 Dec 2017 14:44:25 +0100 "Jens Potthast" <jens.potth...@innovation.uni-bremen.de> wrote: Correction, the samba ML links this code to an inability to _create_ the destination DIR (should be a m$ signification; they love Doublespeak :/) > No, that's not the problem. Files should be overwritten. *But* even > restoring to a new and empty directory fails with the same error. > > -Original Message- > From: B [mailto:lazyvi...@gmx.com] > Sent: Mittwoch, 13. Dezember 2017 14:25 > To: backuppc-users@lists.sourceforge.net > Subject: Re: [BackupPC-users] Trouble with restore > > On Wed, 13 Dec 2017 14:01:46 +0100 > "Jens Potthast" <jens.potth...@innovation.uni-bremen.de> wrote: > > > tar:1596 Can't mkdir Path to filename: > > NT_STATUS_OBJECT_NAME_COLLISION > > ^^ You obviously already have a file having the same name > in this restore DIR. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ smime.p7s Description: S/MIME cryptographic signature -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Trouble with restore
No, that's not the problem. Files should be overwritten. *But* even restoring to a new and empty directory fails with the same error. -Original Message- From: B [mailto:lazyvi...@gmx.com] Sent: Mittwoch, 13. Dezember 2017 14:25 To: backuppc-users@lists.sourceforge.net Subject: Re: [BackupPC-users] Trouble with restore On Wed, 13 Dec 2017 14:01:46 +0100 "Jens Potthast" <jens.potth...@innovation.uni-bremen.de> wrote: > tar:1596 Can't mkdir Path to filename: NT_STATUS_OBJECT_NAME_COLLISION ^^ You obviously already have a file having the same name in this restore DIR. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ smime.p7s Description: S/MIME cryptographic signature -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Trouble with restore
Hello, using BackupPC for several years, I needed to restore a couple of files recently. The last time, I had to restore some files was years ago. At that time it was no problem. Right now, restore fails: Running: /usr/bin/smbclient server\\sharename -U backuppcuser -E -d 1 -c tarmode\ full -mSMB3 -Tx - Running: /usr/local/BackupPC/bin/BackupPC_tarCreate -h hostname -n 1886 -s sharename -t -r /Path\ to\ file -p /Path\ to\ file/filename.txt Xfer PIDs are now 2600,2601 Domain=[MYDOMAIN] OS=[] Server=[] tar:316 tarmode is now full, system, hidden, noreset, quiet tarCreate: Done: 1 files, 237695 bytes, 0 dirs, 0 specials, 0 errors tar:1596 Can't mkdir Path to filename: NT_STATUS_OBJECT_NAME_COLLISION readOutput: sysread returns 0 and got EOF (exit ok = , ) XferErr Non-zero exit status from smbclient restore failed: Non-zero exit status from smbclient What is strange: if I redirect the files to another directory, one folder or file is created before the operation fails. That should rule out all permission problems with my (Windows 2008 R2) Servers Share. I checked permissions of backuppc-user on the server, just in case. However, permissions are ok. If the directory is not there, the directory and one file is created, is the directory already there, no file is created. Therefore, my guess is that the return value is not, what BackupPC is expecting. Upping XferLogLevel does not increase log file output. I am not sure, if this is connected, but I cannot restore to zip also. Restore to tar works. Therefore, I am not completely lost right now. System: BackupPC 4.1.5, BackupPC-XS 0.57, rsync-bpc-3.0.9.9 [Debian 8.x] Any clues where to look or how to increase debug level? Regards, Jens smime.p7s Description: S/MIME cryptographic signature -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Multiple issues on newly installed 4.1.0.
Hello,
After upgrading from 3.x to 4.1 and 4.1.1 and converting all backups to
4.x format, I apparently have the same problem: BackupPC_tarExtract is
using 100% CPU and the backup seems to be halted. To stop the backup, I
need to stop it exactly twice in the Web-Frontend. Don't know why it has
to stopped twice.
Manual backup with verbose logging seems to work better. Sometimes
backup stalls at 100% CPU, sometimes it works. It seems to work after
having the stalled backup stopped on the console with CRTL+C (again,
needs two attempts).
Here is my log from a failed attempt, shortened for better readability,
my comments in square brackets:
[Thousands of lines like these:]
same 644 0/0 301 Somefolder/Somefolder/Somefile.one
same 644 0/0 59177 Somefolder/Somefolder/Somefile.two
delete 644 0/0 12013
Somefolder/Somefolder/Somefile.three
delete 644 0/0 103814 Somefolder/Somefolder/Somefile.four
delete 755 0/0 0 Somefolder/Somefolder
[Then a huge amount like these:]
tarExtract: copyInodes: finished getAll()
tarExtract: copyInodes: finished getAll()
tarExtract: copyInodes: finished getAll()
tarExtract: copyInodes: finished getAll()
tarExtract: copyInodes: finished getAll()
[backup uses 100% CPU with no disk or network IO. Need to cancel with
CTRL+C (twice)]
^C^Cexiting after signal INT
__bpc_progress_state__ fail cleanup
BackupFailCleanup: nFilesTotal = 0, type = full, BackupCase = 4, inPlace
= 0, lastBkupNum = 1617
Removing prior partial backup #1617
__bpc_progress_state__ delete #1617
cmdSystemOrEval: about to system
/usr/share/backuppc/bin/BackupPC_backupDelete -h -n 1617
-l
Xfer PIDs are now 12036,11684
xferPids 12036,11684
BackupPC_backupDelete: removing #1617
__bpc_progress_state__ merge #1617 -> #1614
BackupPC_backupDelete: Merge into backup 1614
Deltas for 1614:
Uncompressed HT:
Compressed HT:
Xfer PIDs are now 12036,12037,11684
xferPids 12036,12037,11684
__bpc_progress_state__ refCnt #1618
bpc_attrib_backwardCompat: WriteOldStyleAttribFile = 0,
KeepOldAttribFiles = 0
__bpc_progress_state__ cntUpdate #1618
__bpc_progress_state__ rename #1618
bpc_poolRefFileRead: got 486418 entries (nRead = 524288)
__bpc_progress_state__ sumUpdate 0/128
bpc_poolRefFileRead: got 2 entries (nRead = 41)
bpc_poolRefFileRead: got 8 entries (nRead = 149)
bpc_poolRefFileRead: got 26 entries (nRead = 473)
[...many nearly identical lines omitted]
__bpc_progress_state__ rename total
BackupPC_refCountUpdate: host got 0 errors (took 18 secs)
Xfer PIDs are now 11684
xferPids 11684
Finished BackupPC_refCountUpdate (running time: 18 sec)
Xfer PIDs are now 11684
xferPids 11684
dump failed: aborted by user (signal=INT)
BackupPC 3.x worked flawlessly for many years. Is there anything I can
do to help fix this problem?
Regards,
Jens
Am 2017-04-10 06:01, schrieb Craig Barratt:
> I'm not sure where to start. First, I updated the documentation and pushed a
> change to update the mtime on the backup directories to match the backup
> ending time.
>
> Second, I'm ignoring a few comments since they seem like gratuitous
> complaining. Feel free to submit a pull request on git if you really care
> about those issues and I'll consider them.
>
> If your main config file is in /etc/BackupPC/config.pl [4], then your
> per-client config.pl [4] files need to be below /etc/BackupPC/pc. That's why
> putting your N$ setting in /data/BackupPC/pc/win2k8server/config.pl [5] has
> no effect.
>
> I'm not sure why the N$ backup and SCGI are consuming 100% cpu time. Are you
> sure you want to use SCGI?
>
> You could get real-time output from the N$ backup by running BackupPC_dump
> manually with the -v option. You should also increase $Conf{XferLogLevel}
> to, eg, 5:
>
> su BackupPCUser
> BackupPC_dump -v win2k8server
>
> Craig
>
> On Fri, Apr 7, 2017 at 1:28 AM, G.W. Haywood
> wrote:
>
>> Hello again,
>>
>> On Wed, 5 Apr 2017, G.W. Haywood wrote:
>>
>>> ... later I added the 'N$' share to the BackupPC
>>> configuration. That was about a week ago. Last night's backup
>>> started at 21:00. After more than 14, hours one process is still
>>> using 100% of a CPU:
>>>
>>> PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
>>> 6352 ged 20 0 109364 68124 4568 R 100.0 1.7 449:24.60
>>> /usr/bin/perl /usr/local/BackupPC/bin/BackupPC_tarExtract -h hostname -s N$
>>> -f
>>
>> After letting it run for 36 hours with no sign of it writing to the log:
>>
>> -rw-r- 1 ged ged 2580494 Apr 5 02:32 XferLOG.8.z
>>
>> I tried to stop it:
>>
>> host07:# >>> ps axufwww | grep Backup
>> ged 32382 0.0 0.3 58900 12380 ?SMar30 0:01
>> /usr/bin/perl
>> /usr/local/BackupPC/bin/BackupPC -d
>> ged 32383 0.0 0.5 74072 23428 ?SN Mar30 0:00 \_
>> /usr/bin/perl /usr/local/BackupPC/bin/BackupPC_Admin_SCGI
>> ged 32384 0.0 0.5 78456
