Re: [BackupPC-users] The one-at-a-time nightly problem, debugged; DEVS PLEASE READ; long.
Robin Lee Powell rlpow...@digitalkingdom.org wrote on 11/25/2010 01:12:50 PM: The problem is that calling BackupPC_serverMesg BackupPC_nightly run when the regular nightlies are already running, or calling it twice in quick succession, doesn't result in the scheduler restarting the nightlies run from scratch (GOOD) or queuing up a second nightlies run when the first finishes (not great, but OK), it results in the scheduler eating its own face (BAD). Given your use case (free up space immediately), your GOOD and not great options are effectively identical. The extant nightly will be some random amount through its process when you submit the second nightly. It will then free up the space for the remaining amount of the pool. Once it finishes and starts a new nightly, the new nightly will start at the beginning of the pool and make its way through it, freeing up the part that the existing one missed. The only part of the process that is wasted when you run the nightly twice is the part where the second nightly job covers the part of the pool that was already processed by the first one after you submitted the second nightly job. Restarting the nightly job from scratch will not change the amount of time that it takes to free up your disk space--not one instant. It will, however, cause the nightly job to process some of your pool twice, *after* the disk space is freed up. In the end, that doesn't seem like that much of a win. So either way would work out to be practically the same. I do agree that eating its own face is *not* optimal, however! :) Timothy J. Massey Out of the Box Solutions, Inc. -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] The one-at-a-time nightly problem, debugged; DEVS PLEASE READ; long.
On Tue, Nov 30, 2010 at 03:18:46PM -0500, Timothy J Massey wrote: Robin Lee Powell rlpow...@digitalkingdom.org wrote on 11/25/2010 01:12:50 PM: The problem is that calling BackupPC_serverMesg BackupPC_nightly run when the regular nightlies are already running, or calling it twice in quick succession, doesn't result in the scheduler restarting the nightlies run from scratch (GOOD) or queuing up a second nightlies run when the first finishes (not great, but OK), it results in the scheduler eating its own face (BAD). Given your use case (free up space immediately), your GOOD and not great options are effectively identical. The extant nightly will be some random amount through its process when you submit the second nightly. It will then free up the space for the remaining amount of the pool. Once it finishes and starts a new nightly, the new nightly will start at the beginning of the pool and make its way through it, freeing up the part that the existing one missed. But it will take a lot longer. The only part of the process that is wasted when you run the nightly twice is the part where the second nightly job covers the part of the pool that was already processed by the first one after you submitted the second nightly job. Restarting the nightly job from scratch will not change the amount of time that it takes to free up your disk space--not one instant. It will, however, cause the nightly job to process some of your pool twice, *after* the disk space is freed up. Yeah, and the hosts can barely keep up with the backups required of them as it is; I need that time. But I understand your point, and had missed it before; thanks. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which this parrot is dead is ti poi spitaki cu morsi, but this sentence is false is na nei. My personal page: http://www.digitalkingdom.org/rlp/ -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] The one-at-a-time nightly problem, debugged; DEVS PLEASE READ; long.
On Sun, Nov 28, 2010 at 10:35:20PM -0800, Robin Lee Powell wrote:
On Sun, Nov 28, 2010 at 07:01:45PM -0800, Craig Barratt wrote:
Robin,
Thanks for the detailed analysis. I agree - it's pretty broken.
Preventing queuing a second nightly request when one is running
will at least avoid the problem. However, I don't recommend
killing the current running nightly.
When I ask it to run a nightly, it's because I've just manually
deleted stuff and I really need to free up disk space ASAP. This
happens regularily around here.
Just so you're aware of the use case.
In case you decide to go with a solution like mine, or someone else
wants it, here's the actually-tested version:
- -
ut00-s3 ~ # diff -uw /var/tmp/ /usr/local/bin/BackupPC
--- /var/tmp/BackupPC 2010-11-25 01:52:30.0 +
+++ /usr/local/bin/BackupPC 2010-11-29 18:18:22.0 +
@@ -522,6 +522,15 @@
$req = pop(@CmdQueue);
$host = $req-{host};
+
+ if( $BackupPCNightlyJobs 0 ! $bpc-isAdminJob($host) )
+ {
+print(LOG $bpc-timeStamp, Tried to run .$req-{cmd}. on $host
when there are nightlies running. That's bad.\n);
+
+unshift(@CmdQueue, $req);
+return;
+ }
+
if ( defined($Jobs{$host}) ) {
print(LOG $bpc-timeStamp,
Botch on admin job for $host: already in use!!\n);
@@ -1362,7 +1371,31 @@
} elsif ( $cmd =~ /^backup all$/ ) {
QueueAllPCs();
} elsif ( $cmd =~ /^BackupPC_nightly run$/ ) {
+print(LOG $bpc-timeStamp,
+ Running nightlies at user request.\n );
+ foreach my $host (keys %Jobs) {
+ if( $bpc-isAdminJob( $host ) ) {
+ my $pid = $Jobs{$host}{pid};
+ kill($bpc-sigName2num(INT), $pid);
+ delete $Jobs{$host};
+print(LOG $bpc-timeStamp,
+ Killing nightly job $host with PID $pid to make way for
manual run.\n);
+ }
+ }
+ # Clear all traces of Nightly jobs by name, just in case
+ for ( my $i = 0 ; $i $Conf{MaxBackupPCNightlyJobs} ; $i++ ) {
+ my $host = $bpc-adminJob($i);
+ if( exists $Jobs{$host} ) {
+ my $pid = $Jobs{$host}{pid};
+ kill($bpc-sigName2num(INT), $pid);
+ delete $Jobs{$host};
+ print(LOG $bpc-timeStamp,
+ Killing nightly job $host with PID $pid to make way for
manual run.\n);
+ }
+ }
$RunNightlyWhenIdle = 1;
+ $BackupPCNightlyJobs = 0;
+ $BackupPCNightlyLock = 0;
} elsif ( $cmd =~ /^backup (\S+)\s+(\S+)\s+(\S+)\s+(\S+)/ ) {
my $hostIP = $1;
$host = $2;
- -
Perhaps some redundancy/overkill there, but I've seen enough failure
modes here that I want to be careful. :)
-Robin
--
http://singinst.org/ : Our last, best hope for a fantastic future.
Lojban (http://www.lojban.org/): The language in which this parrot
is dead is ti poi spitaki cu morsi, but this sentence is false
is na nei. My personal page: http://www.digitalkingdom.org/rlp/
--
Increase Visibility of Your 3D Game App Earn a Chance To Win $500!
Tap into the largest installed PC base get more eyes on your game by
optimizing for Intel(R) Graphics Technology. Get started today with the
Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs.
http://p.sf.net/sfu/intelisp-dev2dev
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] The one-at-a-time nightly problem, debugged; DEVS PLEASE READ; long.
Robin, Thanks for the detailed analysis. I agree - it's pretty broken. Preventing queuing a second nightly request when one is running will at least avoid the problem. However, I don't recommend killing the current running nightly. I need to think some more about the best way to fix this. Craig -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] The one-at-a-time nightly problem, debugged; DEVS PLEASE READ; long.
On Sun, Nov 28, 2010 at 07:01:45PM -0800, Craig Barratt wrote: Robin, Thanks for the detailed analysis. I agree - it's pretty broken. Preventing queuing a second nightly request when one is running will at least avoid the problem. However, I don't recommend killing the current running nightly. When I ask it to run a nightly, it's because I've just manually deleted stuff and I really need to free up disk space ASAP. This happens regularily around here. Just so you're aware of the use case. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which this parrot is dead is ti poi spitaki cu morsi, but this sentence is false is na nei. My personal page: http://www.digitalkingdom.org/rlp/ -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] The one-at-a-time nightly problem, debugged; DEVS PLEASE READ; long.
Robin, Thank you for the awesome write-up. On Wed, 2010-11-24 at 18:11 -0800, Robin Lee Powell wrote: 1. Don't run a non-nightly job from the CmdQueue when there are nightly job running, *EVER*. (my copy is locally modified with debugging stuff to try to figure out these issues, so if this diff doesn't patch, that's why, sorry). -- If one would give me six lines written by the hand of the most honest man, I would find something in them to have him hanged. -- Armand Jean du Plessis, Cardinal Richelieu -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] The one-at-a-time nightly problem, debugged; DEVS PLEASE READ; long.
Robin, Thank you for the awesome write-up. On Wed, 2010-11-24 at 18:11 -0800, Robin Lee Powell wrote: 1. Don't run a non-nightly job from the CmdQueue when there are nightly job running, *EVER*. Unfortunately, that's already well-documented. NEVER, EVER call BackupPC_nightly while another is running. Let BackupPC's scheduler do its job. Regards, Tyler -- If one would give me six lines written by the hand of the most honest man, I would find something in them to have him hanged. -- Armand Jean du Plessis, Cardinal Richelieu -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] The one-at-a-time nightly problem, debugged; DEVS PLEASE READ; long.
On Thu, Nov 25, 2010 at 09:29:34AM +, Tyler J. Wagner wrote: Robin, Thank you for the awesome write-up. On Wed, 2010-11-24 at 18:11 -0800, Robin Lee Powell wrote: 1. Don't run a non-nightly job from the CmdQueue when there are nightly job running, *EVER*. Unfortunately, that's already well-documented. NEVER, EVER call BackupPC_nightly while another is running. Let BackupPC's scheduler do its job. Yeah, I didn't realize at first that that was what was causing it. Regardless, though, if you're going to have the serverMesg call, and scripts that make use of it, either the serverMesg call needs to check that nightly is already running and bail, or (better, IMO) it needs to restart nightly from the beginning. If there wasn't a reason to start it from the beginning, I wouldn't be asking. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which this parrot is dead is ti poi spitaki cu morsi, but this sentence is false is na nei. My personal page: http://www.digitalkingdom.org/rlp/ -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] The one-at-a-time nightly problem, debugged; DEVS PLEASE READ; long.
On Thu, Nov 25, 2010 at 09:29:34AM +, Tyler J. Wagner wrote: Robin, Thank you for the awesome write-up. On Wed, 2010-11-24 at 18:11 -0800, Robin Lee Powell wrote: 1. Don't run a non-nightly job from the CmdQueue when there are nightly job running, *EVER*. Unfortunately, that's already well-documented. NEVER, EVER call BackupPC_nightly while another is running. Let BackupPC's scheduler do its job. I just realized there's been a miscommunication here: 1. My exhortation you quoted wasn't directed at the *user*, it was directed at the BackupPC code. The patch enforces said exhortation. 2. I *did not* call BackupPC_nightly directly, I was fully aware of that issue. I called BackupPC_serverMesg BackupPC_nightly run, as I had been told was safe, which means I was trying to let the scheduler do its job, just as you said. The problem is that calling BackupPC_serverMesg BackupPC_nightly run when the regular nightlies are already running, or calling it twice in quick succession, doesn't result in the scheduler restarting the nightlies run from scratch (GOOD) or queuing up a second nightlies run when the first finishes (not great, but OK), it results in the scheduler eating its own face (BAD). -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which this parrot is dead is ti poi spitaki cu morsi, but this sentence is false is na nei. My personal page: http://www.digitalkingdom.org/rlp/ -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
