Re: [BackupPC-users] Ignore "file has vanished"

2016-01-12 Thread Gandalf Corvotempesta 
2016-01-12 19:04 GMT+01:00 Stephen :
> rsync warning: some files vanished before they could be transferred (code 24) 
> at main.c(1542) [generator=3.0.9.3]
> rsync_bpc exited with benign status 24 (6144)
> ^


Mine is the same. But this backup will be placed under "Failures"
table in admin interface, so, on each login I have to check
if the failed backup was due to vanished file or for any other issue.

I'm pretty sure of this, my first and only backup was finished with
error 24 and was placed under "Failure" table.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Ignore "file has vanished"

2016-01-12 Thread Stephen 
On Tue, 12 Jan 2016, Gandalf Corvotempesta wrote:

> Would be possible to instruct BPC to ignore "file has vanished" errors
> (error code 24) ?
> I'm backingup many clients that has tons of volatile file: php
> sessions, maildirs used by imap, where each file is renamed according
> to imap flags or moved in imap folders and so on.
>
> Every time i make a backup, many files are vanishing, this is
> absolutely normal but BPS mark this as an error (even rsync mark this
> as benign error)
>
> Would be possible to ignore this and mark backup as finished properly ?

This may be something unique to your config.

I have a couple of v4 servers and while most of my backups do not show such 
errors, I found one that did.. it was not a fatal error from BPC's 
perspective. The log file below is for a successful full backup where most 
files were already in the cpool.

Number of files: 245075
Number of files transferred: 22885
Total file size: 13103992399 bytes
Total transferred file size: 2394097407 bytes
Literal data: 142621907 bytes
Matched data: 1984472492 bytes
File list size: 8194691
File list generation time: 0.020 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 3190436
Total bytes received: 153583519
sent 3190436 bytes  received 153583519 bytes  70129.26 bytes/sec
total size is 13103992399  speedup is 83.59
DoneGen: 0 errors, 1402 filesExist, 107610 sizeExist, 163224 sizeExistComp, 
208769 filesTotal, 13103992399 sizeTotal, 1 filesNew, 96 sizeNew, 90 
sizeNewComp, 1121074 inode
rsync warning: some files vanished before they could be transferred (code 24) 
at main.c(1542) [generator=3.0.9.3]
rsync_bpc exited with benign status 24 (6144)
^
Xfer PIDs are now
Running BackupPC_refCountUpdate -h my.server.unc.edu on my.server.unc.edu
xferPids 18361
BackupPC_refCountUpdate: host my.server.unc.edu got 0 errors
BackupPC_refCountPrint: total errors: 0
xferPids
Finished BackupPC_refCountUpdate (running time: 25 sec)
Running BackupPC_refCountUpdate -h my.server.unc.edu -f -c on my.server.unc.edu
xferPids 18363
BackupPC_refCountUpdate: host my.server.unc.edu got 0 errors
BackupPC_refCountPrint: total errors: 0
xferPids
Finished BackupPC_refCountUpdate (running time: 3062 sec)

I'm running BPC v4.0.0alpha3 on this server.

Server and client are Ubuntu. xfer method is rsync over ssh.
Server rsync version is 3.1.0-2ubuntu0.1.
Client rsync version is 3.0.9-1ubuntu1.

Hth,
Stephen

> --
> Site24x7 APM Insight: Get Deep Visibility into Application Performance
> APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
> Monitor end-to-end web transactions and take corrective actions now
> Troubleshoot faster and improve end-user experience. Signup Now!
> http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Ignore "file has vanished"

2016-01-12 Thread Adam Goryachev 


On 12/01/2016 21:48, Gandalf Corvotempesta wrote:
> Would be possible to instruct BPC to ignore "file has vanished" errors
> (error code 24) ?
> I'm backingup many clients that has tons of volatile file: php
> sessions, maildirs used by imap, where each file is renamed according
> to imap flags or moved in imap folders and so on.
>
> Every time i make a backup, many files are vanishing, this is
> absolutely normal but BPS mark this as an error (even rsync mark this
> as benign error)
>
> Would be possible to ignore this and mark backup as finished properly ?
>
Yes, BackupPC does this automatically. I have no idea what you are doing 
wrong, or if this is even your problem though, my mail server sees this 
happen (files renamed during backups) all the time, and never causes the 
backup to fail, even though BPC logs the errors.

I'm sure that this will not help you fix whatever your problem is 
though, so I'll try to be more helpful, if you can post some more detail 
(ie, logs and configs).

Regards,
Adam

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

[BackupPC-users] Ignore "file has vanished"

2016-01-12 Thread Gandalf Corvotempesta 
Would be possible to instruct BPC to ignore "file has vanished" errors
(error code 24) ?
I'm backingup many clients that has tons of volatile file: php
sessions, maildirs used by imap, where each file is renamed according
to imap flags or moved in imap folders and so on.

Every time i make a backup, many files are vanishing, this is
absolutely normal but BPS mark this as an error (even rsync mark this
as benign error)

Would be possible to ignore this and mark backup as finished properly ?

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Ignore "file has vanished"

2016-01-12 Thread Andreas Piening 
Not a real answer to your question on instructing BPC to ignore „file has 
vanished errors“ but I have two suggestions on avoiding this error:

- You can exclude the directories with the often altered PHP files. For me this 
sounds like a machine of a WEB developer constantly editing PHP files. In this 
case the source is probably already „backed up“ by a versioning control system 
(git, mercurial, whatever…)
- You can use snapshots, whether they are LVM based or even easier ZFS based. 
You can create and remove the snapshots with SSH commands in your PRE and POST 
backup scripts for the client.

The error comes from RSYNC itself I think. I don’t think there is an option for 
rsync to ignore these „vanished files“ errors. And honestly I don’t even think 
it’s a good idea to ignore them. 

> Am 12.01.2016 um 11:48 schrieb Gandalf Corvotempesta 
> :
> 
> Would be possible to instruct BPC to ignore "file has vanished" errors
> (error code 24) ?
> I'm backingup many clients that has tons of volatile file: php
> sessions, maildirs used by imap, where each file is renamed according
> to imap flags or moved in imap folders and so on.
> 
> Every time i make a backup, many files are vanishing, this is
> absolutely normal but BPS mark this as an error (even rsync mark this
> as benign error)
> 
> Would be possible to ignore this and mark backup as finished properly ?

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Ignore "file has vanished"

2016-01-12 Thread Gandalf Corvotempesta 
2016-01-12 12:05 GMT+01:00 Andreas Piening :
> - You can exclude the directories with the often altered PHP files. For me
> this sounds like a machine of a WEB developer constantly editing PHP files.
> In this case the source is probably already „backed up“ by a versioning
> control system (git, mercurial, whatever…)

As wrote, I'm backingup also many many many mail servers.
IMAP folders with Maildir (dovecot or any other software supporting
Maildir format)
are full of files that would be renamed when users read/access the
email (at least, some
flag would be added to the file named) or moved when users moves the email
between folders.

For example, new mails would be placed in /new/ directory.
This directory *must* be backupped "as-is"
When users access the email for the first time, the email file will be
moved in /cur/
This directory *must* be backupped "as-is"
When users set an imap flag (priority, read/unread, etc etc) the mail
file is renamed.

As we have more or less 500.000 mailboxes in about 50 servers (that I
have to backup)
most of these mailboxes are used night and day, even during backups hours, thus
BPC/Rsync should be able to copy potentially mutable files with no complains

Obviously I can't exclude mails from backups and actually BPC is marking all
backups as failed as soon as one file change during the rsync procedure.

> - You can use snapshots, whether they are LVM based or even easier ZFS
> based. You can create and remove the snapshots with SSH commands in your PRE
> and POST backup scripts for the client.

I can't, i don't use LVM on all of my servers.

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Ignore "file has vanished"

2016-01-12 Thread Gandalf Corvotempesta 
2016-01-12 16:36 GMT+01:00 Les Mikesell :
> Changed/removed files are not supposed to be fatal

Probably yes, but having all hosts with items in "failed backups"  is not good.
Having too many false positive/false alarms like this will cause loss
of visibility on real issue.

Like the fable "The Boy Who Cried Wolf" ..

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Ignore "file has vanished"

2016-01-12 Thread Les Mikesell 
On Tue, Jan 12, 2016 at 5:33 AM, Gandalf Corvotempesta
 wrote:
> 2016-01-12 12:05 GMT+01:00 Andreas Piening :
>> - You can exclude the directories with the often altered PHP files. For me
>> this sounds like a machine of a WEB developer constantly editing PHP files.
>> In this case the source is probably already „backed up“ by a versioning
>> control system (git, mercurial, whatever…)
>
> As wrote, I'm backingup also many many many mail servers.
> IMAP folders with Maildir (dovecot or any other software supporting
> Maildir format)
> are full of files that would be renamed when users read/access the
> email (at least, some
> flag would be added to the file named) or moved when users moves the email
> between folders.
>
> For example, new mails would be placed in /new/ directory.
> This directory *must* be backupped "as-is"
> When users access the email for the first time, the email file will be
> moved in /cur/
> This directory *must* be backupped "as-is"
> When users set an imap flag (priority, read/unread, etc etc) the mail
> file is renamed.
>
> As we have more or less 500.000 mailboxes in about 50 servers (that I
> have to backup)
> most of these mailboxes are used night and day, even during backups hours, 
> thus
> BPC/Rsync should be able to copy potentially mutable files with no complains
>
> Obviously I can't exclude mails from backups and actually BPC is marking all
> backups as failed as soon as one file change during the rsync procedure.

Changed/removed files are not supposed to be fatal, but you may be the
first person trying to use v4 in an environment like that.  It would
not be surprising to find some bugs or inefficiencies in the new way
of handling things.

-- 
   Les Mikesell
 lesmikes...@gmail.com

--
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311=/4140
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/