Re: [BackupPC-users] Multiple backuppc server
On Tue, Jul 07, 2009 at 01:50:56PM +0100, Andy Brown wrote:
> Hi All,
> We've started to setup a large multiple server backuppc environment, and
> wanted a few thoughts/ideas/advice.
> We've got a large 2TB nas at the back of it with gig connectivity.
> Filesystem is LVM on top of OCFS2 so we have multiple front-end servers with
> read/write.
> Backuppc on each host is setup with relevant different hostnames and setup
> separate logdirectories. The actual top/backup location is shared on the main
> nas store.
>
> So
> $Conf{TopDir} = /backups/backuppc/
> $Conf{ConfDir} = '/etc/backuppc';
> $Conf{LogDir} = '/backups/backup02';
> $Conf{InstallDir} = '/usr/share/backuppc';
> $Conf{CgiDir} = '/usr/share/backuppc/cgi-bin';
> Each server has its own list of hosts in /etc/backuppc/hosts as that's
> how I'm splitting the job queues. i.e. a host only exists in one
> server hosts file (at present either backup01 or backup02).
>
> Can anyone see any pitfalls with this? The only strange thing I've
> noticed is with the trashClean process, it seems to be trying to clean
> things that the other server is creating/working on and failing with
> "Can't read /var/lib/backuppc/trash//home/blah/thing/file: No such
> file or directory". It doesn't seem a major thing so I'm ignoring it
> for now!
You will run into lots of troubles since BackupPC is not designed to
support multiple instances accessing the same storage. There are
processes like BackupPC_nightly which need to have exclusive access to
the pool (e.g. no parallel BackupPC_link running).
> Anyone see any pitfalls/problems with what I'm doing here?
What are you trying to accomplish by using multiple BackupPC instances?
For the time being, they need exclusive pool directories, that is,
exclusive TopDir.
HTH,
Tino.
--
"What we nourish flourishes." - "Was wir nähren erblüht."
www.lichtkreis-chemnitz.de
www.craniosacralzentrum.de
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
[BackupPC-users] Users managing their backups themself?
Well, finally I elucidated the mystery myself: I had to add users under the moreusers field, because all of my backups are done under the backuppc user. So now everything is going smoothly :) Thanks for the hints, Phil Les Mikesell wrote: > error403 wrote: > > > Hello, > > > > I added the htpasswd users with passwords, and they each have their hosts > > configured, but when they log in they get a page with a blank report and no > > possible options, not even starting their own backups. I even tried to > > check all possible checkbox in the cgi configuration to be sure I didn't > > miss anything. > > > > > > If you edit the hosts file, does the name in the 'user' column exactly > match the web login name (upper/lower case included)? That is what is > supposed to determine the hosts they see. > > -- > Les Mikesell > lesmikesell < at > gmail.com > > -- > Enter the BlackBerry Developer Challenge > This is your chance to win up to $100,000 in prizes! For a limited time, > vendors submitting new applications to BlackBerry App World(TM) will have > the opportunity to enter the BlackBerry Developer Challenge. See full prize > details at: http://p.sf.net/sfu/blackberry > ___ > BackupPC-users mailing list > BackupPC-users < at > lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ +-- |This was sent by krunchyf...@videotron.ca via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Users managing their backups themself?
error403 wrote: > Well, finally I elucidated the mystery myself: I had to add users under the > moreusers field, because all of my backups are done under the backuppc user. > So now everything is going smoothly :) > Normally you'd have an admin user or group that can see all of the hosts. The purpose of the 'user' field is to specify the person who gets email about problems and give web access. Moreuser's just get web access. Neither are related to the user account that runs the programs. -- Les Mikesell lesmikes...@gmail.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Multiple backuppc server
Hi, Tino Schwarze wrote on 2009-07-08 10:11:43 +0200 [Re: [BackupPC-users] Multiple backuppc server]: > On Tue, Jul 07, 2009 at 01:50:56PM +0100, Andy Brown wrote: > > We've started to setup a large multiple server backuppc environment [...] > > We've got a large 2TB nas at the back of it with gig connectivity. > > Filesystem is LVM on top of OCFS2 so we have multiple front-end servers > > with read/write. [...] The actual top/backup location is shared on the > > main nas store. > > [...] > > > > Can anyone see any pitfalls with this? > > You will run into lots of troubles [...] actually, I'm not sure you will. I'd expect subtle corruption which you won't notice until it's too late. Things like single files in backups containing the wrong contents. There might be more obvious things like garbled status.pl contents, which might make BackupPC crash or display (and use) incorrect values. I wouldn't be surprised if each BackupPC instance would remove information for hosts it doesn't know about (from status.pl, not the host directories). Random things may or may not happen. You might even be lucky and simply get away with it. Race conditions are things waiting to happen, although they may turn out not to. I don't know and it doesn't seem important to me either. Are you doing backups for the odd chance of them being correct? > BackupPC is not designed to support multiple instances accessing the same > storage. > There are > processes like BackupPC_nightly which need to have exclusive access to > the pool (e.g. no parallel BackupPC_link running). While you might even be able to rule that out by some clever scheduling (and some luck), there's no sane way to prevent more than one instance of BackupPC_link from running. > > The only strange thing I've > > noticed is with the trashClean process, it seems to be trying to clean > > things that the other server is creating/working on and failing with > > "Can't read /var/lib/backuppc/trash//home/blah/thing/file: No such > > file or directory". It doesn't seem a major thing so I'm ignoring it > > for now! This is one example of a race condition. Two trashClean processes are simultaneously trying to delete the same tree. Each file can only be deleted once, so each trashClean will fail for an arbitrary number of files (and log it, as it is unexpected). Obviously, running multiple trashClean processes on the same file system is a waste of resources, i.e. slows things down considerably compared to only one instance running. Multiple BackupPC_nightly instances would be even more wasteful by far. > What are you trying to accomplish by using multiple BackupPC instances? This is an important question. You are probably assuming that your NAS can handle more I/O than one server could generate. You are very likely wrong. Concurrent write access to a file system needs to be synchronized. Your cluster file system might do that, but it does at a cost. Your bottleneck is very likely to be disk seeks rather than raw disk I/O bandwidth. Concurrent independent access to your disk(s) will make that even more so (which is why concurrent backups under one BackupPC server are usually limited to a very small number), and I'd expect your cluster FS to make your bottleneck thinner than it already is. So, aside from not working, because BackupPC is not designed to support it, you will probably be achieving the opposite of what you want. But that's just my guess. You've got it running. What do your measurements say? Regards, Holger -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] suggestion: make wakeups start at 8am
I tend to do this on my installations, because it makes sense for my
environments.
In config.pl I put:
# set the wakeups to 'start' at 8am,
# so BackupPC_nightly will start doing maintenance
# when the main blackout period starts
$Conf{WakeupSchedule} = [8..24,1..7];
Since the default blackouts start at 7am, this setting would seem to be a
reasonable way to make the nightly maintenance job run at a time when
no backup needs to run.
The default configuration causes BackupPC to run the BackupPC_nightly job
when there are backup jobs which may need to run during the night. Many of
us have trouble getting all the backups done in the time allotted; it would
seem to make sense to use that time most efficiently.
Comments, anyone?
Have things changed in such a way that this is not relevant anymore? (I
started doing this with Backuppc 2.1.0, the first version I used).
Downsides to doing this?
--
Carl Soderstrom
Systems Administrator
Real-Time Enterprises
www.real-time.com
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
[BackupPC-users] PST files -- accelerated "retiring"?
Hi, all. Not having used BackupPC for about five years, I was wondering, first and foremost, if there had come into existence a mechanism for backing up open PST files. Secondly, since PST files are disproportionately large when it comes to incrementals, is there any way to delete them at an accelerated rate? Thanks! -Ken -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] suggestion: make wakeups start at 8am
Hi,
On Wed, Jul 8, 2009 at 10:15, Carl Wilhelm
Soderstrom wrote:
> $Conf{WakeupSchedule} = ...
> The default configuration causes BackupPC to run the BackupPC_nightly job
> when there are backup jobs which may need to run during the night...
> Comments, anyone?
Yes, I would suggest decoupling the start time for the
BackupPC_nightly job from the obscure setting of being the first hour
on which WakeupSchedule is set to.
I believe it would be much more clear if there was a separate
NightlyWakeupSchedule or such setting that would specify at which time
that process would start running. BackupPC would wake up on either
WakeupSchedule or NightlyWakeupSchedule, and then would decide if it
would run backups or BackupPC_nightly (or both) based on that hour
being on the first, second, or both variables. In that case, it would
also be much easier to set a sensible default time for
BackupPC_nightly to run.
Also, as an aside, the process should probably be renamed to
BackupPC_daily, as it makes more sense to run it during the day when
backups are not running. Isn't that right?
Filipe
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] suggestion: make wakeups start at 8am
On 07/08 11:08 , Filipe Brandenburger wrote: > Yes, I would suggest decoupling the start time for the > BackupPC_nightly job from the obscure setting of being the first hour > on which WakeupSchedule is set to. > > I believe it would be much more clear if there was a separate > NightlyWakeupSchedule or such setting that would specify at which time > that process would start running. BackupPC would wake up on either > WakeupSchedule or NightlyWakeupSchedule, and then would decide if it > would run backups or BackupPC_nightly (or both) based on that hour > being on the first, second, or both variables. In that case, it would > also be much easier to set a sensible default time for > BackupPC_nightly to run. > > Also, as an aside, the process should probably be renamed to > BackupPC_daily, as it makes more sense to run it during the day when > backups are not running. Isn't that right? I think these are superior ideas, and should be adopted. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] patch to make more robust against corrupted filesystem
I know there have been threads on here debating the relative merit of
throwing out all of the data on a backup disk that suffered major
filesystem damage, vs trying to fix it and let backuppc regenerate any
files that might have corrupted data (via setting a
RsyncCsumCacheVerifyProb of 1 and redoing all machines as full backups).
In light of just how much data I would be throwing out including several
machines that have been archived and then decomissioned, I opted to try to
restore the filesystem and let backuppc keep writing to it. And it seems
to have worked and is still working for me several months later, so I'm
confident all is well in the world.
The only trouble I had was several attrib files were corrupted, so
unpack() barfed with a die() (can't remember what it died of, and my
logs have long since been rotated away, but I suspect it was that
unpack() dies on invalid memory references rather than segfaulting,
which is awfully kind of it) and didn't let me back up anything (usually
dying about 95% of the way through a 32 hour backup, dagnamit). Below was
my solution to make it more robust. Once every machine had done a full
backup, the eval would no longer fail because there was always a valid
attrib file.
This or something like it is probably worth including in upstream, because
cosmic rays can always happen, so it's good to not let unpack() kill the
entire backup when the pool has some slightly corruped data.
--- Attrib.pm.debian-3.1.0-62009-07-09 01:15:43.0 +1000
+++ Attrib.pm.attrib_fix2009-07-09 01:14:59.0 +1000
@@ -252,11 +252,18 @@
$fd->read(\$newData, 65536);
$data .= $newData;
}
- (
-@{$a->{files}{$filename...@fldsunixw},
-@{$a->{files}{$filename...@fldsunixn},
-$data
-) = unpack("w$nFldsW N$nFldsN a*", $data);
+eval {
+ (
+@{$a->{files}{$filename...@fldsunixw},
+@{$a->{files}{$filename...@fldsunixn},
+$data
+ ) = unpack("w$nFldsW N$nFldsN a*", $data);
+ };
+ if ($@) {
+ $a->{_errStr} = "unpack: Can't read attributes for $fileName from
$file $@";
+$fd->close;
+ return;
+ }
if ( $a->{files}{$fileName}{$FldsUnixN[-1]} eq "" ) {
$a->{_errStr} = "Can't read attributes for $fileName"
. " from $file";
--
TimC
"Any sufficiently complicated C or Fortran program contains an ad hoc
informally-specified bug-ridden slow implementation of half of Common
Lisp." -- Greenspun's Tenth Rule of Programming
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] suggestion: make wakeups start at 8am
Filipe Brandenburger wrote:
>
>> $Conf{WakeupSchedule} = ...
>> The default configuration causes BackupPC to run the BackupPC_nightly job
>> when there are backup jobs which may need to run during the night...
>> Comments, anyone?
>
> Yes, I would suggest decoupling the start time for the
> BackupPC_nightly job from the obscure setting of being the first hour
> on which WakeupSchedule is set to.
>
> I believe it would be much more clear if there was a separate
> NightlyWakeupSchedule or such setting that would specify at which time
> that process would start running. BackupPC would wake up on either
> WakeupSchedule or NightlyWakeupSchedule, and then would decide if it
> would run backups or BackupPC_nightly (or both) based on that hour
> being on the first, second, or both variables. In that case, it would
> also be much easier to set a sensible default time for
> BackupPC_nightly to run.
>
> Also, as an aside, the process should probably be renamed to
> BackupPC_daily, as it makes more sense to run it during the day when
> backups are not running. Isn't that right?
I'm not sure you can make a general assumption about when backups are
run. Most places would catch machines that are on all the time at night
but there may also be a lot of desktop machines that are either started
manually through the web interface or caught outside of the blackout
time because they aren't available at night.
--
Les Mikesell
lesmikes...@gmail.com
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] suggestion: make wakeups start at 8am
On Jul 8, 2009, at 11:08 AM, Filipe Brandenburger wrote:
> Hi,
>
> On Wed, Jul 8, 2009 at 10:15, Carl Wilhelm
> Soderstrom wrote:
>> $Conf{WakeupSchedule} = ...
>> The default configuration causes BackupPC to run the
>> BackupPC_nightly job
>> when there are backup jobs which may need to run during the night...
>> Comments, anyone?
>
> Yes, I would suggest decoupling the start time for the
> BackupPC_nightly job from the obscure setting of being the first hour
> on which WakeupSchedule is set to.
>
> I believe it would be much more clear if there was a separate
> NightlyWakeupSchedule or such setting that would specify at which time
> that process would start running. BackupPC would wake up on either
> WakeupSchedule or NightlyWakeupSchedule, and then would decide if it
> would run backups or BackupPC_nightly (or both) based on that hour
> being on the first, second, or both variables. In that case, it would
> also be much easier to set a sensible default time for
> BackupPC_nightly to run.
>
> Also, as an aside, the process should probably be renamed to
> BackupPC_daily, as it makes more sense to run it during the day when
> backups are not running. Isn't that right?
>
> Filipe
I don't disagree with you, but for the last paragraph, some
environments run backups during the day when the client computers are
on.
Tony
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Multiple backuppc server
Holger Parplies wrote: >>> We've started to setup a large multiple server backuppc environment [...] >>> We've got a large 2TB nas at the back of it with gig connectivity. >>> Filesystem is LVM on top of OCFS2 so we have multiple front-end servers >>> with read/write. [...] The actual top/backup location is shared on the >>> main nas store. >>> [...] >>> >>> Can anyone see any pitfalls with this? >> You will run into lots of troubles [...] > > actually, I'm not sure you will. I'd expect subtle corruption which you won't > notice until it's too late. Things like single files in backups containing the > wrong contents. There might be more obvious things like garbled status.pl > contents, which might make BackupPC crash or display (and use) incorrect > values. I wouldn't be surprised if each BackupPC instance would remove > information for hosts it doesn't know about (from status.pl, not the host > directories). Random things may or may not happen. I thought someone had reported doing this successfully over NFS - using a high capacity commercial NAS. > You might even be lucky and simply get away with it. Race conditions are > things waiting to happen, although they may turn out not to. I don't know and > it doesn't seem important to me either. Are you doing backups for the odd > chance of them being correct? > >> BackupPC is not designed to support multiple instances accessing the same >> storage. > > >> There are >> processes like BackupPC_nightly which need to have exclusive access to >> the pool (e.g. no parallel BackupPC_link running). > > While you might even be able to rule that out by some clever scheduling (and > some luck), there's no sane way to prevent more than one instance of > BackupPC_link from running. That shouldn't matter - and in fact probably happens with multiple processes on a single server. link() should be an atomic operation so creation of a hash collision should be detected even if it is simultaneous. >>> The only strange thing I've >>> noticed is with the trashClean process, it seems to be trying to clean >>> things that the other server is creating/working on and failing with >>> "Can't read /var/lib/backuppc/trash//home/blah/thing/file: No such >>> file or directory". It doesn't seem a major thing so I'm ignoring it >>> for now! > > This is one example of a race condition. Two trashClean processes are > simultaneously trying to delete the same tree. Each file can only be deleted > once, so each trashClean will fail for an arbitrary number of files (and log > it, as it is unexpected). Obviously, running multiple trashClean processes on > the same file system is a waste of resources, i.e. slows things down > considerably compared to only one instance running. Multiple BackupPC_nightly > instances would be even more wasteful by far. The BackupPC_nightly run is the more dangerous part. There you have the possibility that it might delete a pool link at the same time another process just re-used it. In older versions, this could not run concurrently with backups for this reason. In the current version there is some sort of locking around the operations that might collide so this might or might not also work on a network filesystem. But in any case you would probably only want one nightly run and keep it outside the backup window. -- Les Mikesell lesmikes...@gmail.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] suggestion: make wakeups start at 8am
On 07/08 10:39 , Les Mikesell wrote: > I'm not sure you can make a general assumption about when backups are > run. Most places would catch machines that are on all the time at night > but there may also be a lot of desktop machines that are either started > manually through the web interface or caught outside of the blackout > time because they aren't available at night. Indeed you are correct, and I considered this. I made my initial suggestion in light of the fact that the default blackout period starts at 7am, indicating a distinct likelihood that backups are typically done at night. I do have some machines which are backed up during the day, but for them, the early morning is a poor time as well. If someone comes into work, plugs in and turns on their laptop, they don't want to be stymied by the machine being backed up (and thereby slowed down) immediately. They typically have backups scheduled for noonish, when the lunch break happens. So I think that 8am is about as good a default time as any, and better than most, for to run the cleanup processes, since it is the least likely time to run a backup. As always, YMMV, but I'm just taking a guess at the time that is likely to be the least inconvenient for the greatest number of people. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] suggestion: make wakeups start at 8am
On Wed, Jul 8, 2009 at 11:39, Les Mikesell wrote: >> Also, as an aside, the process should probably be renamed to >> BackupPC_daily, as it makes more sense to run it during the day when >> backups are not running. Isn't that right? > > I'm not sure you can make a general assumption about when backups are > run. Most places would catch machines that are on all the time at night > but there may also be a lot of desktop machines that are either started > manually through the web interface or caught outside of the blackout > time because they aren't available at night. Agreed, however some default has to be provided and the current one (1am) may not be the best possible one. As you said yourself in another thread, even though BackupPC_nightly can be run while other backups are running, it is better to do so while no or few backups are running so that they do not compete for disk I/O and there is also the locking overhead of both potentially accessing the same files (don't know exactly how that works, but just disk I/O is enough reason to do it while other backups are not running). I agree with the OP that 8am would be more sensible considering it is inside the default blackout period (when most heavy backups should be finished and less backups should be running). Someone who uses a different setup (backups during the day, at noon, etc.) will have to customize these settings anyway, so they can also choose the best time for BackupPC_nightly at that time. The suggestion on the name change is that "nightly" implies it should be run during the night, when in fact what you want is to run it once a day, at the time that is more convenient for you. Thanks, Filipe -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] suggestion: make wakeups start at 8am
On Wed, Jul 08, 2009 at 10:39:14AM -0500, Les Mikesell wrote:
> Filipe Brandenburger wrote:
> >
> >> $Conf{WakeupSchedule} = ...
> >> The default configuration causes BackupPC to run the BackupPC_nightly job
> >> when there are backup jobs which may need to run during the night...
> >> Comments, anyone?
> >
> > Yes, I would suggest decoupling the start time for the
> > BackupPC_nightly job from the obscure setting of being the first hour
> > on which WakeupSchedule is set to.
> >
> > I believe it would be much more clear if there was a separate
> > NightlyWakeupSchedule or such setting that would specify at which time
> > that process would start running. BackupPC would wake up on either
> > WakeupSchedule or NightlyWakeupSchedule, and then would decide if it
> > would run backups or BackupPC_nightly (or both) based on that hour
> > being on the first, second, or both variables. In that case, it would
> > also be much easier to set a sensible default time for
> > BackupPC_nightly to run.
> >
> > Also, as an aside, the process should probably be renamed to
> > BackupPC_daily, as it makes more sense to run it during the day when
> > backups are not running. Isn't that right?
>
> I'm not sure you can make a general assumption about when backups are
> run. Most places would catch machines that are on all the time at night
> but there may also be a lot of desktop machines that are either started
> manually through the web interface or caught outside of the blackout
> time because they aren't available at night.
Since "daily" simply implies "with a 24-hour period", while "nightly"
additionally implies that it occurs at night, i believe the more general
term should be given preference (i myself run nightly's at 3pm...).
Furthermore, i agree that separating the setting for the wake-up schedule
from that of the "nightly" schedule would be a good idea - it is just more
intuitive and flexible. And while we're at it, why keep the requirement that
it runs exactly once a day?
--
Pieter
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] suggestion: make wakeups start at 8am
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Filipe Brandenburger wrote: > On Wed, Jul 8, 2009 at 11:39, Les Mikesell wrote: >>> Also, as an aside, the process should probably be renamed to >>> BackupPC_daily, as it makes more sense to run it during the day when >>> backups are not running. Isn't that right? >> I'm not sure you can make a general assumption about when backups are >> run. Most places would catch machines that are on all the time at night >> but there may also be a lot of desktop machines that are either started >> manually through the web interface or caught outside of the blackout >> time because they aren't available at night. > > Agreed, however some default has to be provided and the current one > (1am) may not be the best possible one. > > As you said yourself in another thread, even though BackupPC_nightly > can be run while other backups are running, it is better to do so > while no or few backups are running so that they do not compete for > disk I/O and there is also the locking overhead of both potentially > accessing the same files (don't know exactly how that works, but just > disk I/O is enough reason to do it while other backups are not > running). > > I agree with the OP that 8am would be more sensible considering it is > inside the default blackout period (when most heavy backups should be > finished and less backups should be running). Someone who uses a > different setup (backups during the day, at noon, etc.) will have to > customize these settings anyway, so they can also choose the best time > for BackupPC_nightly at that time. > > The suggestion on the name change is that "nightly" implies it should > be run during the night, when in fact what you want is to run it once > a day, at the time that is more convenient for you. Maybe a more simple change is to add a comment to the blackout schedule section advising to set the nightly/daily run to during the blackout schedule. Also adding a comment to the daily/nightly section advising to schedule this for a time when you are least likely to run backups to improve performance. Also note that you can run it during a backup if desired. I also agree that setting the default time inside the default backup window is a good idea, that way if someone decides to change the default, then it is up to them to also change the other value... I also agree on renaming to daily, consider /etc/cron.daily etc, it is something that should be run every 24 hours, not needed to run at night. As to running it at intervals less than 24 hours, I'm not sure I see the advantage, better to just get it to process a smaller section of your backup pool each day, rather than run it on the whole pool every x days... Regards, Adam -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkpVKo8ACgkQGyoxogrTyiW9MwCgo16jGi/iKRakYz4iRdgXdGLG 3GMAn2aNaJHSQEt45sXgmggxpc8OrSzs =Zlrm -END PGP SIGNATURE- -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Multiple backuppc server
Hi,
Les Mikesell wrote on 2009-07-08 10:32:50 -0500 [Re: [BackupPC-users] Multiple
backuppc server]:
> [...]
Les, you are missing the important part, so I'll begin with it and repeat it a
few times throughout the mail:
> > Tino Schwarze wrote on 2009-07-08 10:11:43 +0200 [this thread]:
> > > BackupPC is not designed to support multiple instances accessing the same
> > > storage.
> Holger Parplies wrote:
> >Tino wrote:
> >>Andy Brown wrote:
> >>> We've started to setup a large multiple server backuppc environment
> >>> [...]
> >>> Can anyone see any pitfalls with this?
> >> You will run into lots of troubles [...]
> >
> > actually, I'm not sure you will. I'd expect subtle corruption which you
> > won't notice until it's too late. [...]
> > You might even be lucky and simply get away with it. Race conditions are
> > things waiting to happen, although they may turn out not to.
>
> I thought someone had reported doing this successfully over NFS - using
> a high capacity commercial NAS.
If you know there are race conditions, how much faith do you put in a report
saying it has been done "successfully" (presuming you remember correctly)?
Granted, it *might* mean that status.pl confusion will not happen. They
*might* even have figured out that the race conditions do not exist or how to
avoid them, but I wouldn't believe it without examining the reasoning behind
that, because
> > > BackupPC is not designed to support multiple instances accessing the same
> > > storage.
so it doesn't take any expensive measures to avoid race conditions resulting
from doing so anyway.
> > there's no sane way to prevent more than one instance of BackupPC_link
> > from running.
>
> That shouldn't matter - and in fact probably happens with multiple
> processes on a single server.
No, it probably doesn't. I checked that before writing what I wrote. Did you
check before contradicting me?
> link() should be an atomic operation so
> creation of a hash collision should be detected even if it is simultaneous.
Detecting it is trivial. Please provide a correct implementation of *handling*
it. It's not necessarily a "hash collision" as BackupPC uses the term, by the
way. There is currently no need to handle this, because
> > > BackupPC is not designed to support multiple instances accessing the same
> > > storage.
and it avoids it happening within a single server instance, because that is
*much* easier than handling it.
> [...]
> The BackupPC_nightly run is the more dangerous part. There you have the
> possibility that it might delete a pool link at the same time another
> process just re-used it.
You are correct in that this is something we want to avoid. If I were so
inclined, it would be trivial to contradict you with your own arguments,
though. Something like "link() should return a failure code if the source file
does not exist so this should be easily detected". Aside from the comment
that it's not "dangerous" to have no pool link for a file, it's just wasteful,
because you won't be able to reuse it for other copies.
But I won't do that. You are right. There should not be more than one instance
of BackupPC_nightly running on a pool, and BackupPC_nightly and BackupPC_link
should not run concurrently.
> In the current version there
> is some sort of locking around the operations that might collide so this
> might or might not also work on a network filesystem.
This sounds like an urban myth. Did you check how this "locking operation"
works? What version of BackupPC introduced it? I went *part of the way* through
the diffs. What I found wasn't "locking", it was design, and it will, in fact,
extend to several BackupPC servers accessing one pool. But that is only part
of the mechanism. The rest, I believe, is in fact really a form of "locking":
the provisions a BackupPC server takes to avoid two jobs from running
concurrently that shouldn't - BackupPC_nightly and BackupPC_link, including
more than one instance of either (i.e. *only one* BackupPC_nightly(*) or
BackupPC_link job may be running at one point in time). This part will
obviously *not* extend to several independent server instances accessing the
pool. In other words,
> > > BackupPC is not designed to support multiple instances accessing the same
> > > storage.
> But in any case
> you would probably only want one nightly run and keep it outside the
> backup window.
I have nothing to add to that.
So, do you insist on making the original poster believe that running several
instances of BackupPC on the same pool is a good idea, or can we maybe find
some other topic to disagree on?
Regards,
Holger
(*) With $Conf{MaxBackupPCNightlyJobs} you can split one BackupPC_nightly
job into 2, 4, 8 ... processes which will run concurrently and each process
a distinct part of the pool. In the sense of the above definition, they
comprise one logical BackupPC_nightly entity. What you can't have is
more than one BackupPC_nightly processing the *same* pa
Re: [BackupPC-users] suggestion: make wakeups start at 8am
Hi,
Adam Goryachev wrote on 2009-07-09 09:24:05 +1000 [Re: [BackupPC-users]
suggestion: make wakeups start at 8am]:
> Filipe Brandenburger wrote:
> > On Wed, Jul 8, 2009 at 11:39, Les Mikesell wrote:
> >>> Also, as an aside, the process should probably be renamed to
> >>> BackupPC_daily, as it makes more sense to run it during the day when
> >>> backups are not running. Isn't that right?
I don't think the name is important. It's an implementation detail. You can
change related config settings and the documentation without renaming the
binary. Does anybody care about the name "BackupPC_Admin"? People should read
documentation, and if they are about to change the time BackupPC_nightly is
run, they almost certainly have. I would guess that the origins of the name
are probably from the observation that most maintenance tasks are run during
the night, when they don't disturb anyone. Server backups are an example of
such a task, as are "daily" cron jobs.
Coincidentally, this maintenance task is not supposed to disturb backups ...
But you should keep in mind that not everyone has a dedicated backup server.
For example, I run BackupPC on an NFS/NIS/whatever-server to an external iSCSI
device. The server is idle during the night, so why not use it as BackupPC
server? The storage unit is independent; that is what seems to matter.
Obviously, I prefer BackupPC_nightly disturbing the backups rather than my
users :). I *have* seen it run during the day, and it wasn't being very subtle
about it.
If BackupPC_nightly is renamed, then why not to something that makes sense,
like BackupPC_dailyMaintenance or BackupPC_poolMaintenance or just
BackupPC_maintenance?
> >> I'm not sure you can make a general assumption about when backups are
> >> run. Most places would catch machines that are on all the time at night
> >> but there may also be a lot of desktop machines that are either started
> >> manually through the web interface or caught outside of the blackout
> >> time because they aren't available at night.
> >
> > Agreed, however some default has to be provided and the current one
> > (1am) may not be the best possible one.
Agreed. The default should be good enough for most people - particularly
unexperienced users -, disruptive for nobody, and follow the principle of
least surprise. Early morning seems reasonable. Inside the blackout window
makes sense but may be surprising.
> > As you said yourself in another thread, even though BackupPC_nightly
> > can be run while other backups are running, it is better to do so
> > while no or few backups are running so that they do not compete for
> > disk I/O
Which is why I'd argue for making BackupPC_nightly run time independent of
the WakeupSchedule. The WakeupSchedule defines possible times when backups
may be started (though they may continue or be started delayed due to
MaxBackups). Having an independent time for BackupPC_nightly where no backups
are considered makes sense, eg.
$Conf {WakeupSchedule} = [ 21, 22, 23, 0, 1, 2, 3 ];
$Conf {NightlySchedule} = [ 7.75 ];
> > Someone who uses a
> > different setup (backups during the day, at noon, etc.) will have to
> > customize these settings anyway, so they can also choose the best time
> > for BackupPC_nightly at that time.
Agreed.
> Maybe a more simple change is to add a comment to the blackout
> schedule section advising to set the nightly/daily run to during the
> blackout schedule. Also adding a comment to the daily/nightly section
> advising to schedule this for a time when you are least likely to run
> backups to improve performance. Also note that you can run it during a
> backup if desired.
I agree that the important parts are documentation changes. Everything can be
done with BackupPC as it is now, and probably the current default settings
are good enough for people who don't want to customize them. And, yes,
*better* default values may be even better for them :). Good and easily
accessible advice is best of all.
> I also agree that setting the default time inside the default backup
> window is a good idea,
"Backup window" or "blackout window"?
> that way if someone decides to change the
> default, then it is up to them to also change the other value...
>
> I also agree on renaming to daily, consider /etc/cron.daily etc, it is
> something that should be run every 24 hours, not needed to run at night.
"Daily" is not strictly a requirement. "Regularly" is (but BackupPC_regularly
is a bad name ;-). BackupPCNightlyPeriod effectively makes it traverse the
pool less often, and that's ok (hmm, what about e-mails, though?).
> As to running it at intervals less than 24 hours, I'm not sure I see
> the advantage, better to just get it to process a smaller section of
> your backup pool each day, rather than run it on the whole pool every
> x days...
I could imagine reasons to want a complete pool traversal on a particular day
rather than splitting it up equally. You might need to run full backups o
Re: [BackupPC-users] suggestion: make wakeups start at 8am
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Holger Parplies wrote:
> Hi,
>
> Adam Goryachev wrote on 2009-07-09 09:24:05 +1000 [Re: [BackupPC-users]
> suggestion: make wakeups start at 8am]:
>> Filipe Brandenburger wrote:
>>> On Wed, Jul 8, 2009 at 11:39, Les Mikesell wrote:
> Also, as an aside, the process should probably be renamed to
> BackupPC_daily, as it makes more sense to run it during the day when
> backups are not running. Isn't that right?
>
> I don't think the name is important. It's an implementation detail. You can
> change related config settings and the documentation without renaming the
> binary. Does anybody care about the name "BackupPC_Admin"? People should read
> documentation, and if they are about to change the time BackupPC_nightly is
> run, they almost certainly have. I would guess that the origins of the name
> are probably from the observation that most maintenance tasks are run during
> the night, when they don't disturb anyone. Server backups are an example of
> such a task, as are "daily" cron jobs.
The name only matters in that people may infer meaning where it is not
meant. Though this could be fixed through documentation, and a few
interface changes. For example, if the web interface config displayed
the setting as DailyMaintenance instead of NightlyMaintenance even
though the binary that uses the setting was Backuppc_nightly... If
someone knows enough about backuppc to find the binary name, then they
should also know enough (to find out) what it does and why.
> Coincidentally, this maintenance task is not supposed to disturb backups ...
Except it is pretty well acknowledged that it does due to the high
amount of disk IO... Of course, there are things that could be done to
off-set that, such as a small sleep every 100 "bits" it processes, but
it will *still* impact on backup performance, just less...
> But you should keep in mind that not everyone has a dedicated backup server.
> For example, I run BackupPC on an NFS/NIS/whatever-server to an external iSCSI
> device. The server is idle during the night, so why not use it as BackupPC
> server? The storage unit is independent; that is what seems to matter.
> Obviously, I prefer BackupPC_nightly disturbing the backups rather than my
> users :). I *have* seen it run during the day, and it wasn't being very subtle
> about it.
Exactly, and that is how obviously it will impact on backup performance
as well.
> If BackupPC_nightly is renamed, then why not to something that makes sense,
> like BackupPC_dailyMaintenance or BackupPC_poolMaintenance or just
> BackupPC_maintenance?
Keep in mind that is is also responsible for sending out the reminder
emails, updating the status.pl, and probably other stuff I don't know
about... It doesn't *just* maintain the pool...
> Agreed. The default should be good enough for most people - particularly
> unexperienced users -, disruptive for nobody, and follow the principle of
> least surprise. Early morning seems reasonable. Inside the blackout window
> makes sense but may be surprising.
Only in that a user may think the blackout period applies to all
backuppc activity as opposed to backup hosts activity... However, I
agree that it should be scheduled to occur (by default) during the
(default) blackout period.
>>> As you said yourself in another thread, even though BackupPC_nightly
>>> can be run while other backups are running, it is better to do so
>>> while no or few backups are running so that they do not compete for
>>> disk I/O
>
> Which is why I'd argue for making BackupPC_nightly run time independent of
> the WakeupSchedule. The WakeupSchedule defines possible times when backups
> may be started (though they may continue or be started delayed due to
> MaxBackups). Having an independent time for BackupPC_nightly where no backups
> are considered makes sense, eg.
>
> $Conf {WakeupSchedule} = [ 21, 22, 23, 0, 1, 2, 3 ];
> $Conf {NightlySchedule} = [ 7.75 ];
Not a bad idea, for example if I know all my hosts have a blackout time
of 7am - 7pm, then why do I need backuppc to wake up and schedule all
the hosts, only to find out that backups are not needed? However, I do
need it to wake up and process the pool at 10am...
Also keep in mind that unless you have a very small pool, and/or very
fast (disks) pool, then your daily maintenance will probably take much
more than 15 mins... (Mine takes around 3 hours).
>> I also agree that setting the default time inside the default backup
>> window is a good idea,
> "Backup window" or "blackout window"?
Damn, did I really say that :( I of course meant:
I also agree that setting the default time inside the default blackout
window is a good idea,
>> that way if someone decides to change the
>> default, then it is up to them to also change the other value...
>>
>> I also agree on renaming to daily, consider /etc/cron.daily etc, it is
>> something that should be run every 24 hours, not needed to run at night.
>
> "D
Re: [BackupPC-users] PST files -- accelerated "retiring"?
Ken D'Ambrosio wrote at about 09:34:42 -0400 on Wednesday, July 8, 2009: > Hi, all. Not having used BackupPC for about five years, I was wondering, > first and foremost, if there had come into existence a mechanism for > backing up open PST files. Secondly, since PST files are > disproportionately large when it comes to incrementals, is there any way > to delete them at an accelerated rate? > 1. Read the threads on using shadow mounts to backup open files like pst's. If you want, you can use the script that I posted to the group for automagically setting up and taking down shadow mounts 2. I also wrote a script, BackupPC_deleteFile for selectively deleting one or more files from one or more backups. One could presumably automate it to regularly delete past pst files using a simple cron job or equivalent. Again check out the archives. -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Bandwidth Consumption
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi all, Just wanted to supply some information on something I setup yesterday in a spare couple of hours (yeah, like any hours are spare :) I wanted to determine the amount of bandwidth was being consumed by each host I backup because sometimes I get close to exceeding the monthly quota, and also it is a good indication that something is wrong if one host is doing 5 GB a day (like trying to backup a folder of movies someone dumped onto their PC yesterday)... So, here it is: 1) Download and install http://www.intevation.de/iam It seems to be very much un-maintained/improved on since initial release many years ago, somewhat in german (I presume). 2) Setup your iptables so that you have an input and output rule pointing to a chain for each individual entry you want in your reports: new_chain client1 iptables -A INPUT -p tcp --sport 22 -i eth0 -s backuphost1 -j client1 iptables -A OUTPUT -p tcp --dport 22 -o eth0 -d backuphost1 -j client1 You can have more than one host in the same report row like this: new_chain client1 iptables -A INPUT -p tcp --sport 22 -i eth0 -s backuphost1 -j client1 iptables -A OUTPUT -p tcp --dport 22 -o eth0 -d backuphost1 -j client1 iptables -A INPUT -p tcp --sport 22 -i eth0 -s backuphost2 -j client1 iptables -A OUTPUT -p tcp --dport 22 -o eth0 -d backuphost2 -j client1 Repeating the last two lines for as many hosts as needed. You might need different rules to catch the traffic you are interested in, eg, for rsyncd change the port entry, etc... 3) Follow the rest of the instructions included in the file, and you are done. With a couple of tweaks to iam_report: #!/bin/sh IAM=/usr/local/iam/iam DUMP=/var/log/backuppc/ipacct-data WWWDIR=/var/www/backuppc-accounting YM_CURRENT=`date '+%Y-%m'` YM_LAST=`date --date='1 month ago' '+%Y-%m'` TODAY=`date '+%Y-%m-%d'` YESTERDAY=`date --date='1 day ago' '+%Y-%m-%d'` $IAM -f $YM_CURRENT-00 -t $YM_CURRENT-99 -w $WWWDIR/$YM_CURRENT.html $DUMP $IAM -f $YM_LAST-00 -t $YM_LAST-99 -w $WWWDIR/$YM_LAST.html $DUMP $IAM -f $TODAY -t $TODAY -w $WWWDIR/$TODAY.html $DUMP $IAM -f $YESTERDAY -t $YESTERDAY -w $WWWDIR/$YESTERDAY.html $DUMP ln -sf $WWWDIR/$YM_CURRENT.html $WWWDIR/00-current.html ln -sf $WWWDIR/$YM_LAST.html $WWWDIR/00-last.html ln -sf $WWWDIR/$TODAY.html $WWWDIR/00-today.html ln -sf $WWWDIR/$YESTERDAY.html $WWWDIR/00-yesterday.html I can now see stats for today, yesterday, this month, and last month, and of course, all of the daily/monthly history. Reports look like this: IP traffic (2009-07-08..2009-07-09): | traffic | cost | description | +---++---+ | 28.9 MB || filesrv (local Filesrv) | |1.0 MB || flail (local Flail) | +---++---+ | 29.9 MB || WebMan Local Hosts | +---++---+ | || | | 226.6 MB | $ 2.27 | castle (Castle) | | 76.1 MB | $ 0.76 | forex (Windows Forex) | | 27.8 MB | $ 0.28 | shared (Windows Shared) | +---++---+ | 330.5 MB | $ 3.31 | WebMan Remote Hosts | +---++---+ So you can group hosts and see the total for each group, assign a different cost / MB to each group, etc... I have no association with IAM, it just showed up on a google search, seemed to be the easiest to install, and pretty much just worked out of the box. (PS, I needed to delete line 5 from each of the .py files because it had a non standard character in it which python complained about). For someone who knows/uses python (I don't) they could fairly easily munge this into putting data into a mysql DB, or generate different reports, etc quite easily... or of course, you could just script it from nothing. The main drawback to this is that you pretty much require your hosts to have a static IP, or can define some static iptables rule which will count the traffic for you. I hope this is useful/helpful to someone else. If you have some problems setting it up with backuppc, let me know and I'll try and offer some advise... Regards, Adam - -- Adam Goryachev Website Managers www.websitemanagers.com.au -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkpVVlQACgkQGyoxogrTyiVa4ACePBREItUdC5g2g7PovO5FUwKc jiEAoM5ELp6dJSPlIm/NPeypgwGefm6F =QTFs -END PGP SIGNATURE- --
Re: [BackupPC-users] PST files -- accelerated "retiring"?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeffrey J. Kosowsky wrote: > Ken D'Ambrosio wrote at about 09:34:42 -0400 on Wednesday, July 8, 2009: > > Hi, all. Not having used BackupPC for about five years, I was wondering, > > first and foremost, if there had come into existence a mechanism for > > backing up open PST files. Secondly, since PST files are > > disproportionately large when it comes to incrementals, is there any way > > to delete them at an accelerated rate? > > > > 1. Read the threads on using shadow mounts to backup open files like >pst's. If you want, you can use the script that I posted to the >group for automagically setting up and taking down shadow mounts > > 2. I also wrote a script, BackupPC_deleteFile for selectively deleting >one or more files from one or more backups. One could presumably >automate it to regularly delete past pst files using a simple cron >job or equivalent. Again check out the archives. There is also a rsync binary that was posted which was able to automatically utilise shadow mounts to backup open files. I tried the above scripts, and got rather lost, rather than debug and work through it, I tried the rsync binary, which has worked perfectly since, and so I haven't touched it again Jeffrey, have you looked at the rsync binary which supports shadow mount backups? How does that compare to what your set of scripts do? I'd just be interested in whether your scripts offer some advantage I haven't considered... PS, recently I did a format and re-install of windows 2003 server up to the point of the first reboot (about 30-40 mins). Then I restored the entire c drive of a machine backed up using the rsync binary which supports backing up open files. Then booted the machine, which seemed to work perfectly... I'm not suggesting this is the ideal disaster recovery method, but it did work for me in this instance Regards, Adam - -- Adam Goryachev Website Managers www.websitemanagers.com.au -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkpVV5MACgkQGyoxogrTyiWAqACfXegGZZL+gHNmA/71/DXdtkBr vIwAni4FZAqCOKbSwg+Uh4q+mVaWZGtD =S1aF -END PGP SIGNATURE- -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] suggestion: make wakeups start at 8am
Les Mikesell wrote at about 10:39:14 -0500 on Wednesday, July 8, 2009:
> Filipe Brandenburger wrote:
> >
> >> $Conf{WakeupSchedule} = ...
> >> The default configuration causes BackupPC to run the BackupPC_nightly job
> >> when there are backup jobs which may need to run during the night...
> >> Comments, anyone?
> >
> > Yes, I would suggest decoupling the start time for the
> > BackupPC_nightly job from the obscure setting of being the first hour
> > on which WakeupSchedule is set to.
> >
> > I believe it would be much more clear if there was a separate
> > NightlyWakeupSchedule or such setting that would specify at which time
> > that process would start running. BackupPC would wake up on either
> > WakeupSchedule or NightlyWakeupSchedule, and then would decide if it
> > would run backups or BackupPC_nightly (or both) based on that hour
> > being on the first, second, or both variables. In that case, it would
> > also be much easier to set a sensible default time for
> > BackupPC_nightly to run.
> >
> > Also, as an aside, the process should probably be renamed to
> > BackupPC_daily, as it makes more sense to run it during the day when
> > backups are not running. Isn't that right?
>
> I'm not sure you can make a general assumption about when backups are
> run. Most places would catch machines that are on all the time at night
> but there may also be a lot of desktop machines that are either started
> manually through the web interface or caught outside of the blackout
> time because they aren't available at night.
>
> --
I'm warming to the name change to BackupPC_daily since 'daily' most
generically implies once a day whereas 'nightly' would imply during
the nighttime.
--
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] PST files -- accelerated "retiring"?
Adam Goryachev wrote at about 12:36:03 +1000 on Thursday, July 9, 2009: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Jeffrey J. Kosowsky wrote: > > Ken D'Ambrosio wrote at about 09:34:42 -0400 on Wednesday, July 8, 2009: > > > Hi, all. Not having used BackupPC for about five years, I was > > wondering, > > > first and foremost, if there had come into existence a mechanism for > > > backing up open PST files. Secondly, since PST files are > > > disproportionately large when it comes to incrementals, is there any way > > > to delete them at an accelerated rate? > > > > > > > 1. Read the threads on using shadow mounts to backup open files like > >pst's. If you want, you can use the script that I posted to the > >group for automagically setting up and taking down shadow mounts > > > > 2. I also wrote a script, BackupPC_deleteFile for selectively deleting > >one or more files from one or more backups. One could presumably > >automate it to regularly delete past pst files using a simple cron > >job or equivalent. Again check out the archives. > > There is also a rsync binary that was posted which was able to > automatically utilise shadow mounts to backup open files. I tried the > above scripts, and got rather lost, rather than debug and work through > it, I tried the rsync binary, which has worked perfectly since, and so I > haven't touched it again > > Jeffrey, have you looked at the rsync binary which supports shadow mount > backups? How does that compare to what your set of scripts do? I'd just > be interested in whether your scripts offer some advantage I haven't > considered... Not sure -- I haven't had a chance to look at it yet. I use cygwin rsync so I wasn't too excited about installing a separate rsync binary. Presumably the modified rsync is cleaner since it would be able to avoid the kludge of having to separately start up vshadow.exe (which requires additional kludges in my script since vshadow.exe is somewhat broken in XP) and to assign drive letters to vshadow. Also, assuming rsync is always running, you don't have the problem of having to use something like 'at' to re-enter as an admin-level user (this is due to a bug/feature in cygwin ssh where even ssh'ing as root doesn't give you full admin power). My script has proved to be quite robust for me across multiple usage cases so I haven't felt the need to try something new/better. > PS, recently I did a format and re-install of windows 2003 server up to > the point of the first reboot (about 30-40 mins). Then I restored the > entire c drive of a machine backed up using the rsync binary which > supports backing up open files. Then booted the machine, which seemed to > work perfectly... I'm not suggesting this is the ideal disaster recovery > method, but it did work for me in this instance > > Regards, > Adam > > - -- > Adam Goryachev > Website Managers > www.websitemanagers.com.au > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.9 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iEYEARECAAYFAkpVV5MACgkQGyoxogrTyiWAqACfXegGZZL+gHNmA/71/DXdtkBr > vIwAni4FZAqCOKbSwg+Uh4q+mVaWZGtD > =S1aF > -END PGP SIGNATURE- > > -- > Enter the BlackBerry Developer Challenge > This is your chance to win up to $100,000 in prizes! For a limited time, > vendors submitting new applications to BlackBerry App World(TM) will have > the opportunity to enter the BlackBerry Developer Challenge. See full prize > details at: http://p.sf.net/sfu/Challenge > ___ > BackupPC-users mailing list > BackupPC-users@lists.sourceforge.net > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki:http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] suggestion: make wakeups start at 8am
Hi,
Adam Goryachev wrote on 2009-07-09 12:07:11 +1000 [Re: [BackupPC-users]
suggestion: make wakeups start at 8am]:
> Holger Parplies wrote:
> > I don't think the name is important. It's an implementation detail. You can
> > change related config settings and the documentation without renaming the
> > binary. [...]
>
> The name only matters in that people may infer meaning where it is not
> meant. Though this could be fixed through documentation, and a few
> interface changes. For example, if the web interface config displayed
> the setting as DailyMaintenance instead of NightlyMaintenance even
> though the binary that uses the setting was Backuppc_nightly... If
> someone knows enough about backuppc to find the binary name, then they
> should also know enough (to find out) what it does and why.
that is what I meant. But I see that you might infer meaning where it is not
intended. You can find an instance of "BackupPC_nightly" in "ps" and wonder
why it's running in the middle of the day. Hmm.
> > Coincidentally, this maintenance task is not supposed to disturb backups ...
>
> Except it is pretty well acknowledged that it does due to the high
> amount of disk IO...
Sorry, I was unclear. I meant, this maintenance task should be run at a time
where it will not disturb backups, as opposed to other maintenance tasks that
should not disturb humans. So "nightly", from the perspective of a backup job,
is possibly during the day ;-). Or it should disturb neither backups nor
humans and be run in the morning.
There is no question that BackupPC_nightly will slow down backups (and
humans ...). Both running concurrently will almost certainly take longer
than both running sequentially (except perhaps for very slow links).
> > Agreed. The default should be good enough for most people - particularly
> > unexperienced users -, disruptive for nobody, and follow the principle of
> > least surprise. Early morning seems reasonable. Inside the blackout window
> > makes sense but may be surprising.
>
> Only in that a user may think the blackout period applies to all
> backuppc activity as opposed to backup hosts activity...
That is what I meant. "Hey, we're in the blackout period! Why is my disk going
crazy?" But whoever configured the blackout period (or just noticed there is
one) should have come across a reference to the maintenance job, and that
reference should have made clear that it is exempt from blackout.
> However, I agree that it should be scheduled to occur (by default) during the
> (default) blackout period.
I don't say I don't agree. Maybe the documentation/web interface should more
explicitly call it "backup blackout period" or something. Maybe it should even
be less definite than "blackout", since it's really only a hint that BackupPC
will ignore if necessary. It's really the opposite of a "preferred backup
window".
> > I'd argue for making BackupPC_nightly run time independent of
> > the WakeupSchedule. The WakeupSchedule defines possible times when backups
> > may be started (though they may continue or be started delayed due to
> > MaxBackups). Having an independent time for BackupPC_nightly where no
> > backups
> > are considered makes sense, eg.
> >
> > $Conf {WakeupSchedule} = [ 21, 22, 23, 0, 1, 2, 3 ];
> > $Conf {NightlySchedule} = [ 7.75 ];
>
> Not a bad idea, for example if I know all my hosts have a blackout time
> of 7am - 7pm, then why do I need backuppc to wake up and schedule all
> the hosts, only to find out that backups are not needed?
What's worse, it may decide that a backup *is* needed (because of unexpected
ping failures or something), although you never intended for one to be done at
that time.
It should be possible to define when backups are intended to be run
(WakeupSchedule), give preferences for when to run them (BlackoutPeriod) and
configure when to run the maintenance job(s). Maintenance jobs and backups are
really independent. In my opinion, using a common schedule is just saving a
configuration variable. If you want both backups and maintenance to run at
22:00, you can do that just as well with independent settings.
> Also keep in mind that unless you have a very small pool, and/or very
> fast (disks) pool, then your daily maintenance will probably take much
> more than 15 mins... (Mine takes around 3 hours).
I'm totally surprised to see times of 5 minutes on one pool and 23 on another.
I seemed to remember something like 2 hours, too. Well, I'm not complaining ;-).
> Well, we could hive off the pool maintenance into a new binary called
> backuppc_poolmaintenance and have that run separate from the
> backuppc_dailymaintenance which does the email, status.pl updates,
> etc...
That thought had occurred to me, too, at least for the e-mails. Don't the
status.pl updates (partly) reflect the results of traversing the pool?
> > I could imagine reasons to want a complete pool traversal on a particular
> > day rather than splitting it up equally. [...]
>
> True,
Re: [BackupPC-users] PST files -- accelerated "retiring"?
Jeffrey J. Kosowsky wrote: > Adam Goryachev wrote at about 12:36:03 +1000 on Thursday, July 9, 2009: > Jeffrey J. Kosowsky wrote: Hi Folks, >> Ken D'Ambrosio wrote at about 09:34:42 -0400 on Wednesday, July 8, 2009: >> > Hi, all. Not having used BackupPC for about five years, I was wondering, >> > first and foremost, if there had come into existence a mechanism for >> > backing up open PST files. Secondly, since PST files are >> > disproportionately large when it comes to incrementals, is there any way >> > to delete them at an accelerated rate? >> > > >> 1. Read the threads on using shadow mounts to backup open files like >>pst's. If you want, you can use the script that I posted to the >>group for automagically setting up and taking down shadow mounts > >> 2. I also wrote a script, BackupPC_deleteFile for selectively deleting >>one or more files from one or more backups. One could presumably >>automate it to regularly delete past pst files using a simple cron >>job or equivalent. Again check out the archives. > > There is also a rsync binary that was posted which was able to > automatically utilise shadow mounts to backup open files. I tried the > above scripts, and got rather lost, rather than debug and work through > it, I tried the rsync binary, which has worked perfectly since, and so I > haven't touched it again > I guess you're talking about the patch I created based on the work that Elias Penttilä did for rsync 2.6.x ? The code is still a bit in development, but I think this is a good moment to give an update of the state of that code. I think I now have a version that works on 32-bit Windows XP all the way up to the latest 32-bit Windows: Windows 7. Which are probably the most different versions of 32-bit Windows that I 'need' to support if I want to support as many as possible. That's also the only ones I've recently tested, although I've tested Windows 2003 in the past as well, but I don't think I've broken anything on that platform. It's a patched version of the latest rsync 3.0.6 build on Cygwin. Because Cygwin is pretty much 32-bit only and the 64-bit VSS API only seems handles 64-bit calls (or whatever it's called) it only works on Windows versions which are 32-bit. It's still in development thus some of it is still hardcoded, that means it works only in daemon mode and always uses shadow volumes, thus it's readonly (writing to a shadow volume probably makes no sense). On the other hand, it might be limited in what it supports, but that's already pretty usefull to a lot of people I hope. When creating a backup of Windows Vista or Windows 7 (possible Windows 2008 as well) Microsoft created an other hurdle called 'junction points' (sort of like symlinks for directories), which rsync had a lot of problems with. I now recoqnize they are their and skip them for now.I don't yet know if any rsync protocol changes would be needed to support them properly. I've not decided with to do with them, maybe I should talk to some people on the rsync mailinglist. It does log that it skipped them. Also usually no new ones are usually created, so you could just create a one time list (or ones a day): dir /aL /s > list.txt On those recent Windows versions (Vista, 7 and possible 2008) if I wanted to have a succesful transfer, I did have to skip: - System Volume Information (it's a hidden/system directory, which holds lots of things, including the files that store the Volume Shadow Copy information, I've not figured out why it's complaining about that) - hiberfil.sys (hibernation file, which you probably don't need on restore anyway) Those were the files Windows didn't want to transfer and still complaint about even when using Shadow Volume Copy. Everything else worked. Also you might need to start it with 'run as Administrator' and/or put the user it's running as in the 'Backup Operators' group on Vista and newer, but I've not done extensive testing with that. If you don't like wasting space it's probably a good idea to skip pagefile.sys as well, of which copying worked just fine, but possible isn't all that useful. I'd like to mention that technically Windows XP Shadow mounts work differently I have to create a seperate binary for Windows XP (rsync-xp.exe). So when deploying it, you'll need to use rsync-xp.exe (you can obviously rename it to rsync.exe) on Windows XP and rsync.exe on the newer versions. http://www.consolejunky.net/cwrsync-vss/versions/beta3/ ( I do need to clean up the patches/code in that directory a bit, without build instructions it might not be very clear how to use them, but the binaries are good ) I still need to update the readme with the current status: http://www.consolejunky.net/cwrsync-vss/ I'll do that soon, I'm kind of busy this week. You can find more discussion on the subject here: http://www.itefix.no/i2/node/11984 I hope that gives you an idea of what is possible with it now and answers any questions you may h
Re: [BackupPC-users] [OT] Linux "load" values (was: Re: Hardware considerations for building dedicated backuppc server)
On Tue, Jul 7, 2009 at 4:38 PM, Holger Parplies wrote: > No, my point was, the "load average" is an attempt to fit the state of a > system into one single number (which, as we've agreed, is only good for > getting a quick impression, nothing more). Exactly. On Linux systems (don't have enough high performance experience with other systems to say conclusively) the load average simply gives you a rough indication of the number of processes that are in the run queue. If you have one process running using 100% CPU, your load average will be 1. If you have one process running waiting 100% IO, your load average will be 1. To get a full picture you can run top (make sure if you have multiple CPUs you expand the display to show each one) or vmstat (I typically use `vmstat 1` to watch the data on 1 second intervals). vmstat has the benefit of splitting out the number of processing running on CPU and waiting on disk (see the first columns r and b) as well as showing you overall IO load and the type of CPU utilization. Here's a sample from vmstat of my desktop/server system running a local and remote backup at the same time: procs ---memory-- ---swap-- -io --system-- -cpu- r b swpd free buff cache si sobibo in cs us sy id wa st 2 2 0 35592 200332 145911600 73100 392 2884 2375 65 12 0 24 0 3 2 0 21508 200648 147397200 55036 0 2523 2641 28 6 4 63 0 2 1 0 24940 201092 147166400 81664 0 3085 2949 33 9 1 58 0 1 2 0 25512 201376 147278400 81216 1064 3094 2883 28 8 0 65 0 3 1 0 23612 201732 147576000 81256 0 3039 2939 28 8 1 64 0 Here we can see a pretty good mix of CPU and IO wait - Improving the capacity of either would lead to a reduction in backup times. -Dave -- Enter the BlackBerry Developer Challenge This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/Challenge ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
