Re: [Bacula-users] Catalog too big / not pruning?
Wasn't sitting here the whole time, but it was 2-3 hours each run. --Jeremy -Original Message- From: Alan Brown [mailto:a...@mssl.ucl.ac.uk] Sent: Tuesday, August 11, 2009 12:13 To: Jeremy Koppel Cc: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Catalog too big / not pruning? On Thu, 6 Aug 2009, Jeremy Koppel wrote: I thought that meant it wasn't going to actually do anything, but it did reduce the DB size to 6.5GB. I had actually stopped bacula before running it this time, so perhaps that had an effect. After that, I went ahead and ran dbcheck (thanks, John), and it found a bunch of stuff: Found 1000 orphaned File records. Deleting 1000 orphaned File records. How long did it take to remove these entries? AB -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Catalog too big / not pruning?
: ) Thanks for the assist. Could be a version problem. This is on one of our older servers, and it's running Bacula version 1.36.3, which for years was the only version available on Gentoo. (looks like current version in Portage is 2.4.1). When a new server is available, I'll look into migrating it. --Jeremy -Original Message- From: Martin Simmons [mailto:mar...@lispworks.com] Sent: Friday, August 07, 2009 14:30 To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Catalog too big / not pruning? On Fri, 7 Aug 2009 07:55:08 -0700, Jeremy Koppel said: I ended up running dbcheck 3 more times. The first time got another 10,000,000, the second another 8,000,000+, and the 3rd was trivial. Running it a fourth time came up all 0s. Running another full vacuum got the DB size down to 597MB, which sounds right. Nice space saving! In theory Bacula will remove stale file records, but there was a bug in some version that could leave them. So, there has been a job in the crontab that runs the standard vacuumdb (not full), but, it does this while Bacula is running. For the past few days, while running the full vacuum, I shut it off as a precaution. Perhaps I should modify the script to shut down Bacula during the standard vacuum? Is this needed? It is probably better to avoid vacuuming while Bacula is updating the database, but there is no need to shut it down. You could run the vacuum from a Bacula admin job rather than cron if you want to control it better. I run it after the backups, so pruning has had a chance to remove records. __Martin -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Catalog too big / not pruning?
You're right; I was looking at something else entirely - I was comparing file_fp_idx and file_jobid_idx with a 3rd set of numbers I forgot to post - the numbers after running the vacuumdb, and then subsequently running the REINDEX command; they were identical. So, I guess it's no longer true that vacuumdb increases the indexes; it must reindex automatically now. I ended up running dbcheck 3 more times. The first time got another 10,000,000, the second another 8,000,000+, and the 3rd was trivial. Running it a fourth time came up all 0s. Running another full vacuum got the DB size down to 597MB, which sounds right. So, there has been a job in the crontab that runs the standard vacuumdb (not full), but, it does this while Bacula is running. For the past few days, while running the full vacuum, I shut it off as a precaution. Perhaps I should modify the script to shut down Bacula during the standard vacuum? Is this needed? --Jeremy -Original Message- From: Martin Simmons [mailto:mar...@lispworks.com] Sent: Thursday, August 06, 2009 13:11 To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Catalog too big / not pruning? On Thu, 6 Aug 2009 05:59:24 -0700, Jeremy Koppel said: We're running Postgresql 8.0.8; we can't currently update this machine (we'll have to move Bacula to a newer box when we have one available). Ran that query, and the top 4 do have very large numbers: relname | reltuples | relpages -+-+-- file| 3.28168e+07 | 592614 OK, that is 147 bytes per row, which is about what you would expect. file_fp_idx | 3.28168e+07 | 378580 file_jobid_idx | 3.28168e+07 | 368832 file_pkey | 3.28168e+07 | 364870 And running vacuumdb with the --analyze flag says: INFO: file: found 0 removable, 32828342 nonremovable row versions in 592867 pages DETAIL: 0 dead row versions cannot be removed yet. Nonremovable row versions range from 113 to 154 bytes long. ... I ran the full vacuum after that, and now it is down to 5.9GB, so I guess all those records really weren't taking up much space. Also, the indexes actually got bigger: relname | reltuples | relpages -+-+-- file| 3.28283e+07 | 592684 file_fp_idx | 3.28283e+07 |90029 file_jobid_idx | 3.28283e+07 |71896 file_pkey | 3.28283e+07 |71895 I read up on it and saw that this was expected behavior, and that running a reindex on the table should fix it. So I ran REINDEX TABLE file;, but that didn't have any effect. I'll do some looking into that today. Look again at the sizes -- they actually got 5x smaller! Initially they were very bloated compared to the table size. Also, I found the output from dbcheck curious. Of all the orphaned records it found, the file records were an even number: 10,000,000. It sort of seems like maybe dbcheck can only clean 10,000,000 records at a time. : ) So, I have just now started running it again, and so far it has found 0 bad Path records, 0 bad Filename records, etc., all 0 this time, until it got to File records, where it says again: Found 1000 File records, Deleting 1000 orphaned File records. Yes, I think there is a limit on the number of file records it will delete each time. __Martin -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Catalog too big / not pruning?
We're running Postgresql 8.0.8; we can't currently update this machine (we'll have to move Bacula to a newer box when we have one available). Ran that query, and the top 4 do have very large numbers: relname | reltuples | relpages -+-+-- file| 3.28168e+07 | 592614 file_fp_idx | 3.28168e+07 | 378580 file_jobid_idx | 3.28168e+07 | 368832 file_pkey | 3.28168e+07 | 364870 And running vacuumdb with the --analyze flag says: INFO: file: found 0 removable, 32828342 nonremovable row versions in 592867 pages DETAIL: 0 dead row versions cannot be removed yet. Nonremovable row versions range from 113 to 154 bytes long. I thought that meant it wasn't going to actually do anything, but it did reduce the DB size to 6.5GB. I had actually stopped bacula before running it this time, so perhaps that had an effect. After that, I went ahead and ran dbcheck (thanks, John), and it found a bunch of stuff: Found 1000 orphaned File records. Deleting 1000 orphaned File records. Checking for orphaned Path entries. This may take some time! Found 17688 orphaned Path records. Deleting 17688 orphaned Path records. Checking for orphaned Filename entries. This may take some time! Found 72448 orphaned Filename records. Deleting 72448 orphaned Filename records. Checking for orphaned FileSet entries. This takes some time! Found 25 orphaned FileSet records. Deleting 25 orphaned FileSet records. Checking for orphaned Client entries. Found 2 orphaned Client records. Deleting 2 orphaned Client records. Checking for orphaned Job entries. Found 1284 orphaned Job records. I ran the full vacuum after that, and now it is down to 5.9GB, so I guess all those records really weren't taking up much space. Also, the indexes actually got bigger: relname | reltuples | relpages -+-+-- file| 3.28283e+07 | 592684 file_fp_idx | 3.28283e+07 |90029 file_jobid_idx | 3.28283e+07 |71896 file_pkey | 3.28283e+07 |71895 I read up on it and saw that this was expected behavior, and that running a reindex on the table should fix it. So I ran REINDEX TABLE file;, but that didn't have any effect. I'll do some looking into that today. Also, I found the output from dbcheck curious. Of all the orphaned records it found, the file records were an even number: 10,000,000. It sort of seems like maybe dbcheck can only clean 10,000,000 records at a time. : ) So, I have just now started running it again, and so far it has found 0 bad Path records, 0 bad Filename records, etc., all 0 this time, until it got to File records, where it says again: Found 1000 File records, Deleting 1000 orphaned File records. I'll report back when it is finished. Thanks, --Jeremy -Original Message- From: Martin Simmons [mailto:mar...@lispworks.com] Sent: Wednesday, August 05, 2009 6:58 To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Catalog too big / not pruning? The job table is probably not causing the bloat, unless you have millions of rows. The space is usually consumed by the file table and its indexes. Try running vacuumdb with the --analyze and --verbose options, which prints info about the number of pages used by each table/indexes and also the number of unused rows. You can also get info from the SQL query SELECT relname, reltuples, relpages FROM pg_class ORDER BY relpages DESC; Which version of postgresql are you using? With postgresql 7, I found that regularly doing reindex on the file table indexes was needed to prevent bloat. __Martin -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Catalog too big / not pruning?
Lately, I've been going though our file server looking for disk space to reclaim, and I've come across 14GB worth of data in the Postgres DB, used only by Bacula. Reading through the Bacula manual, I see that each file record is supposed to take up 154 bytes in the DB, so I have gone through the logs to see how many records should be saved, keying on FD Files Written. Our rotation: Level=Full Pool=Weekly sun at 2:00 Level=Differential Pool=Daily FullPool=Weekly mon-fri at 2:00 File and Job Retention is set to 5 weeks in the Client directives, Volume Retention is set to 5 weeks in the Pool directives. AutoPrune is set to Yes in both places. The exception is the Monthly pool, used by 2 clients (small): Level=Full Pool=Monthly 1st sun at 1:00 Level=Full Pool=Weekly 2nd-4th sun at 1:00 Level=Differential Pool=Daily FullPool=Weekly mon-fri at 2:00 For the Monthly pool, File Retention is set to 90d, and Job Retention is set to 1y in the Client directives, and Volume Retention is set to 1y in the pool. AutoPrune is set to Yes in the Client Directives, but had been set to No in the Pool, which was a red flag. I changed it to Yes, and restarted Bacula. The numbers: In the last 5 weeks, the sum of the backed up files to the Weekly and Daily pools is 3,588,224, and of the other two, 1 client backs up 2 files to the Monthly pool a month, and the other, consistently just under 6,640 files. So, if my catalog should have a total of 3,667,928 files in it (3,588,224 + (2*12) + (6,640 * 12)) * 154 bytes per file, the DB should be 564,860,912 bytes or 538MB? Made me either wonder where my math went wrong, or why my DB is so big. So, after running a full vacuum on the Postgres DB and only reclaiming 1GB of disk space, I started poking around in the DB itself (just looking). The most interesting thing I have found so far is a table named 'job', which seems to hold all the job records. And they go back to 2006 Example: bacula=# select * from job where starttime like '2006%'; jobid | job | name | type | level | clientid | jobstatus | schedtime | starttime | endtime | jobtdate | volsessionid | volsessiontime | jobfiles | jobbytes | joberrors | jobmissingfiles | poo lid | filesetid | purgedfiles | hasbase ---+-+-+--+---+--+ ---+-+-+-+ +--++--+---+---+-+ +---+-+- 272 | Mail_Data.2006-01-26_02.00.06 | Mail Data | B| D | | f | 2006-01-26 02:00:05 | 2006-01-26 02:08:49 | 2006-01-26 02:37:33 | 1138261053 |0 | 0 |0 | 0 | 0 | 0 | | | 0 | 0 I thought I would then be clever, and in Bacula use the prune command for the old MailData job, one of the ones set to 5 weeks, but it didn't show up in the list because it was no longer a defined job resource; we don't run that particular job anymore, and haven't for months: thus there should be no jobs for 'MailData' in the catalog at all. Luckily, it was just commented out in bacula-dir.conf, so I uncommented it and restarted bacula. It then showed MailData, and said: Pruned 15 Jobs for client MailData from catalog.. Encouraged, I exited Bacula and checked the DB. They are all still there. Rinse and repeat. This time, Bacula says: No Jobs found to prune., so maybe it did get rid of a few of them after all. But the rest are sitting there derelict, just hogging space. So, why isn't my DB pruning itself? --Jeremy -- Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Restore OK -- warning file count mismatch
I'm sure I've seen this mentioned on a previous thread, but I can't seem to find it; we don't have much occasion to back up Windows machines. Maybe somebody can point me in the right direction? We're running Bacula 1.36.3-r2 (I've seen that there are newer versions, but they don't seem to be in Portage) on our server, and the Windows client in question. A test backup went fine, but on the test restore, I get: JobId: 816 Job:RestoreFiles.2007-04-23_09.30.08 Client: bcact1-fd Start time: 23-Apr-2007 09:30:10 End time: 23-Apr-2007 09:34:12 Files Expected: 3,850 Files Restored: 3,849 Bytes Restored: 1,739,460,553 Rate: 7187.9 KB/s FD Errors: 0 FD termination status: OK SD termination status: OK Termination:Restore OK -- warning file count mismatch Does this mean that it actually missed a file during restore? How do I find out which file, and make sure the backup / restore works correctly? --Jeremy - This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Resolved: FW: Autoloader: Replace tape in magazine after backups complete?
Couldnt agree more; we were taken off guard by that answer from Quantums support. We had been looking for a reasonably priced entry-level autoloader that would take LTO2 or greater, and the SuperLoader3 came in at a decent price and little rack space. Been a great performer, too. Luckily, we only lost a daily tape (differential), so it wasnt a huge deal, but if itd been from our Monthly pool, wed have been in a world of hurt. So, too late to turn back now, we were just looking for a work-around. Seems to be doing well so far, though I imagine the tapes will wear faster. --Jeremy From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of John Drescher Sent: Tuesday, October 10, 2006 12:38 To: [EMAIL PROTECTED]; [EMAIL PROTECTED] Cc: bacula-users Subject: Re: [Bacula-users] Resolved: FW: Autoloader: Replace tape in magazine after backups complete? except our Quantum SuperLoader3 has an inaccessible tape drive, and in the event of a malfunction, like we had, your tape is stuck in that drive for life (to try to remove it voids the warranty). The Bacula documentation entry for the unit's compatibility should have a LARGE warning next to it. This kind of thing is unacceptable in an individual tape drive, let alone a library. Agreed. I had a similar problem with my Exabyte magnum 224 yesterday. The drive would not unload the tape in drive 0. Peeking through the air holes in the front I saw that the tape was indeed in the drive but the robot was not attempting to unload it at all. After a power cycle of the autoloader the tape came out on the first try. To me this is an unacceptable behavior but at least there was a fix... John - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Poll - What operating systems do you run Bacula on?
No worries. We're running the entire server portion of Bacula on: Dell PowerEdge 2650 Processors: 2x Intel(R) Xeon(TM) CPU 2.40GHz, HyperThreaded RAM: 1GB OS: Gentoo Linux 2006.1 Bacula Version: 1.36.3-r2 (any word on when a newer version is coming to Portage?) Have a mix of Dell PE1800, PE1850, PE2650s, PE2850s, running Gentoo 2006.1, with the client portion of Bacula installed. 1 Windows 2003 server with the client installed. --Jeremy -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter L. Buschman Sent: Tuesday, October 10, 2006 06:52 To: bacula-users@lists.sourceforge.net Subject: [Bacula-users] Poll - What operating systems do you run Bacula on? All: If it isn't too much of an imposition, I'd like to survey the list and ask the question what operating system are you running Bacula on?. I'm interested in which OS distributions, versions and platforms are being deployed as Bacula servers. Mainly, this is to identify the highest-priority configurations for a test environment I am setting-up, but I think it would also be interesting from a broader Bacula adoption perspective to see what the distribution is. I will aggregate all of the responses and post a summary and percentage distribution of the results. If you would like to add your installation to the count but do not want to post openly to the list, please feel free to email me privately. The summarized results will be anonymous as they will only consist of rolled-up statistics. Best regards, Peter Buschman - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDE V ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] FW: Autoloader: Replace tape in magazine after backups complete?
So I've had some time to test a few things, and it looks like this is the command that Bacula is sending to see which tape is loaded: /var/lib/bacula/mtx-changer /dev/sg1 loaded 0 /dev/nst0 0 And, assuming it answers with 12 for slot 12, it ejects the tape using: /var/lib/bacula/mtx-changer /dev/sg1 unload 12 /dev/nst0 0 That seems to work okay, except that Bacula sort of stumbles around afterwards, during the next backup or restore. I've copied part of the log below, but it basically sits there for 5 minutes, until it times out I'm guessing, before it realizes that the tape has been ejected. It then times out for another 5 minutes before it tries to load the tape it needs. So, it looks like it would still be preferable to run some command inside Bacula to eject the tape so that it knows about the ejection, rather than going through the external script. This might be a question for the developers, but is there a simple way to do that? I'm still combing the manual, but haven't found anything relevant yet. Log snip: 19-Sep 14:21 ns2-dir: Start Restore Job RestoreFiles.2006-09-19_14.20.59 19-Sep 14:26 ns2-sd: RestoreFiles.2006-09-19_14.20.59 Warning: Couldn't rewind device /dev/nst0 ERR=dev.c:406 Rewind error on /dev/nst0. ERR=Input/output error. 19-Sep 14:26 ns2-sd: 3301 Issuing autochanger loaded drive 0 command. 19-Sep 14:26 ns2-sd: 3302 Autochanger loaded drive 0, result: nothing loaded. 19-Sep 14:31 ns2-sd: 3304 Issuing autochanger load slot 4, drive 0 command. 19-Sep 14:32 ns2-sd: 3305 Autochanger load slot 4, drive 0, status is OK. 19-Sep 14:32 ns2-sd: Ready to read from volume AAA003L2 on device /dev/nst0. 19-Sep 14:32 ns2-sd: Forward spacing to file:block 114:0. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeremy Koppel Sent: Monday, September 11, 2006 11:05 To: bacula-users@lists.sourceforge.net Subject: [Bacula-users] FW: Autoloader: Replace tape in magazine after backups complete? Actually, as a result of my testing last week with umount, my weekend backups didn't run; umount only logically unmounted the tape, and then Bacula sat there waiting for someone to manually (logically) remount the tape. And the manual actually says: If you have an autoloader, the mount command will not cause Bacula to operate the autoloader. Which is, I guess, what I need. Bacula is operating the autoloader for the actual backup, it just keeps the tape in the drive when the backup is finished. It looks like it's calling a program, mtx-changer, to do the actual magic, so I'm going to experiment with that once my weekend backups have completed. --Jeremy -Original Message- From: Ryan Novosielski [mailto:[EMAIL PROTECTED] Sent: Monday, September 11, 2006 10:21 To: Jeremy Koppel Subject: Re: [Bacula-users] Autoloader: Replace tape in magazine after backups complete? Those aren't commands. Commands are those that appear in the list when one types help. Try 'umount' and see what the next step in the script should be. All you're looking to do is write a sequence of console commands -- no different than running them at the console. If you don't know what the console commands are, read the manual. Jeremy Koppel wrote: I see your point. When I look through the Bacula logs, I see stuff like this: 02-Sep 01:02 ns2-sd: 3301 Issuing autochanger loaded drive 0 command. 02-Sep 01:02 ns2-sd: 3302 Autochanger loaded drive 0, result is Slot 1. 02-Sep 01:02 ns2-sd: 3303 Issuing autochanger unload slot 1, drive 0 command. But typing loaded drive 0 or autochanger loaded drive 0 or other various variants into the Bacula console just gives errors like is an illegal command. What do I need to do to actually get those commands to work? -Original Message- From: Ryan Novosielski [mailto:[EMAIL PROTECTED] Sent: Friday, September 08, 2006 15:15 To: Jeremy Koppel Cc: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Autoloader: Replace tape in magazine after backups complete? Nope. My guess is the zero has to do with autochanger slots. Important to keep in mind here is that all this is doing is running commands on the console. If umount [CR] 0 [CR] q [CR] doesn't work in the console, it's not going to work here either. I don't have an autoloader, so I don't know how prompting for them works. In any case, I'd still recommend a more specific umount storangename line -- good to be as specific as possible. =R Jeremy Koppel wrote: Mark, Arno, Thanks much for the replies. For my Gentoo installation, here's what I get: ns2 ns1 # echo -e umount\n0\nq\n | /var/lib/bacula/bconsole -c /etc/bacula/bconsole.conf Connecting to Director ns2:9101 1000 OK: ns2-dir Version: 1.36.3 (22 April 2005) Enter a period to cancel a command. umount Using default Catalog name=MyCatalog DB=bacula
[Bacula-users] FW: Autoloader: Replace tape in magazine after backups complete?
Actually, as a result of my testing last week with umount, my weekend backups didn't run; umount only logically unmounted the tape, and then Bacula sat there waiting for someone to manually (logically) remount the tape. And the manual actually says: If you have an autoloader, the mount command will not cause Bacula to operate the autoloader. Which is, I guess, what I need. Bacula is operating the autoloader for the actual backup, it just keeps the tape in the drive when the backup is finished. It looks like it's calling a program, mtx-changer, to do the actual magic, so I'm going to experiment with that once my weekend backups have completed. --Jeremy -Original Message- From: Ryan Novosielski [mailto:[EMAIL PROTECTED] Sent: Monday, September 11, 2006 10:21 To: Jeremy Koppel Subject: Re: [Bacula-users] Autoloader: Replace tape in magazine after backups complete? Those aren't commands. Commands are those that appear in the list when one types help. Try 'umount' and see what the next step in the script should be. All you're looking to do is write a sequence of console commands -- no different than running them at the console. If you don't know what the console commands are, read the manual. Jeremy Koppel wrote: I see your point. When I look through the Bacula logs, I see stuff like this: 02-Sep 01:02 ns2-sd: 3301 Issuing autochanger loaded drive 0 command. 02-Sep 01:02 ns2-sd: 3302 Autochanger loaded drive 0, result is Slot 1. 02-Sep 01:02 ns2-sd: 3303 Issuing autochanger unload slot 1, drive 0 command. But typing loaded drive 0 or autochanger loaded drive 0 or other various variants into the Bacula console just gives errors like is an illegal command. What do I need to do to actually get those commands to work? -Original Message- From: Ryan Novosielski [mailto:[EMAIL PROTECTED] Sent: Friday, September 08, 2006 15:15 To: Jeremy Koppel Cc: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Autoloader: Replace tape in magazine after backups complete? Nope. My guess is the zero has to do with autochanger slots. Important to keep in mind here is that all this is doing is running commands on the console. If umount [CR] 0 [CR] q [CR] doesn't work in the console, it's not going to work here either. I don't have an autoloader, so I don't know how prompting for them works. In any case, I'd still recommend a more specific umount storangename line -- good to be as specific as possible. =R Jeremy Koppel wrote: Mark, Arno, Thanks much for the replies. For my Gentoo installation, here's what I get: ns2 ns1 # echo -e umount\n0\nq\n | /var/lib/bacula/bconsole -c /etc/bacula/bconsole.conf Connecting to Director ns2:9101 1000 OK: ns2-dir Version: 1.36.3 (22 April 2005) Enter a period to cancel a command. umount Using default Catalog name=MyCatalog DB=bacula Automatically selected Storage: Autoloader 3002 Device /dev/nst0 unmounted. 0 0: is an illegal command. q ns2 ns1 # Am I supposed to substitute 'n' for something? --Jeremy -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Thursday, September 07, 2006 17:24 To: Jeremy Koppel Cc: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Autoloader: Replace tape in magazine after backups complete? In the message dated: Thu, 07 Sep 2006 16:55:15 EDT, The pithy ruminations from Jeremy Koppel on [Bacula-users] Autoloader: Replace tape in magazine after backups complete? w ere: = = Well, we just had an autoloader go bad on us. Quantum sent us a new Ouch. = chassis under warranty, but there was a tape in the drive when the unit = tanked out on us. There's no way to get at it manually, and no user = serviceable parts. They tell me that the only way to get it out is by = disassembling the drive, which they'll do, but the process destroys the = tape. Oh, that's bad. I've got the same thing with a long-out-of-warranty AIT2 changer. = = = = So, I'm thinking it would be best to have Bacula replace whatever tape = it's using back in the autoloader when the last job of the night runs. = Anybody know how I'd configure that? Sure. Assuming that your last backup job is to backup the Bacula catalog database, and that there's a RunAfter directive that executes a script like $BACULA/bin/delete_catalog_backup, I add to that script something like: /bin/echo -e umount\n0\nq\n | $BACULA/bin/bconsole -c $BACULA/etc/bconsole.conf
Re: [Bacula-users] Autoloader: Replace tape in magazine after backups complete?
Mark, Arno, Thanks much for the replies. For my Gentoo installation, here's what I get: ns2 ns1 # echo -e umount\n0\nq\n | /var/lib/bacula/bconsole -c /etc/bacula/bconsole.conf Connecting to Director ns2:9101 1000 OK: ns2-dir Version: 1.36.3 (22 April 2005) Enter a period to cancel a command. umount Using default Catalog name=MyCatalog DB=bacula Automatically selected Storage: Autoloader 3002 Device /dev/nst0 unmounted. 0 0: is an illegal command. q ns2 ns1 # Am I supposed to substitute n for something? --Jeremy -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Thursday, September 07, 2006 17:24 To: Jeremy Koppel Cc: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Autoloader: Replace tape in magazine after backups complete? In the message dated: Thu, 07 Sep 2006 16:55:15 EDT, The pithy ruminations from Jeremy Koppel on [Bacula-users] Autoloader: Replace tape in magazine after backups complete? w ere: = = Well, we just had an autoloader go bad on us. Quantum sent us a new Ouch. = chassis under warranty, but there was a tape in the drive when the unit = tanked out on us. There's no way to get at it manually, and no user = serviceable parts. They tell me that the only way to get it out is by = disassembling the drive, which they'll do, but the process destroys the = tape. Oh, that's bad. I've got the same thing with a long-out-of-warranty AIT2 changer. = = = = So, I'm thinking it would be best to have Bacula replace whatever tape = it's using back in the autoloader when the last job of the night runs. = Anybody know how I'd configure that? Sure. Assuming that your last backup job is to backup the Bacula catalog database, and that there's a RunAfter directive that executes a script like $BACULA/bin/delete_catalog_backup, I add to that script something like: /bin/echo -e umount\n0\nq\n | $BACULA/bin/bconsole -c $BACULA/etc/bconsole.conf This will send the umount command to the Bacula bconsole program. Substitute the correct path for your Bacula installation, and repeat the umount command for each of the tape drives within the autochanger (specifying the drive number). Mark = = = = Thanks! = = = = --Jeremy Koppel = = = = = --_=_NextPart_1655_00025162.00015147 = Content-Type: text/html; = charset=us-ascii = Content-Transfer-Encoding: quoted-printable = = html xmlns:o=3Durn:schemas-microsoft-com:office:office xmlns:w=3Durn:sc= = hemas-microsoft-com:office:word xmlns=3Dhttp://www.w3.org/TR/REC-html40 = [Gack! Horrible MS-Word HTML deleted] Mark Bergman [EMAIL PROTECTED] System Administrator Section of Biomedical Image Analysis 215-662-7310 Department of Radiology, University of Pennsylvania http://pgpkeys.pca.dfn.de:11371/pks/lookup?search=mark.bergman%40.uphs.upenn.edu The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message. - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Autoloader: Replace tape in magazine after backups complete?
Well, we just had an autoloader go bad on us. Quantum sent us a new chassis under warranty, but there was a tape in the drive when the unit tanked out on us. Theres no way to get at it manually, and no user serviceable parts. They tell me that the only way to get it out is by disassembling the drive, which theyll do, but the process destroys the tape. So, Im thinking it would be best to have Bacula replace whatever tape its using back in the autoloader when the last job of the night runs. Anybody know how Id configure that? Thanks! --Jeremy Koppel - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] ClientRunAfterJob only on successful completion?
Ive been experimenting with the ClientRunAfterJob function lately to delete files from a temporary backup location on a remote client. The database on the client backs up 2 files nightly in its own format, then my ClientRunBeforeJob script compresses these into self-extracting .exe files. The Bacula job is set to back up just those 2 files, then run the second script to wipe the directory. Seems to work great, except that last night the 2 .exe files failed to generate (my own goof), but the ClientRunAfterJob script ran anyway, even though the job failed, deleting the good backup without actually backing it up (Im in test mode, so nothing critical was lost yet). How would I tell Bacula not to run that script unless the backup is completely successful? --Jeremy Koppel ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Error on first mount
Hello, In one of our stand-alone servers, with a single tape drive, I am trying to automate backups as much as possible by having Bacula unmount and eject a tape when its time to be changed, and remount when it has been changed. Theres a script to do this with the Bacula documentation: #!/bin/sh /full-path/console -c /full-path/console.conf \ltEND_OF_DATA unmount storage=your-storage-name END_OF_DATA # the following is a shell command mt eject /full-path/console -c /full-path/console.conf END_OF_DATA mount storage=your-storage-name END_OF_DATA When I run it, however, while the unmount and eject work beautifully, the remount fails with the following error: 3901 open device failed: ERR=dev.c:289 stored: unable to open device /dev/nst0: ERR=Input/output error At that point, a subsequent mount attempt succeeds, even when run immediately after running this script manually. Has anyone else experienced this problem? Details: Gentoo Linux 2.6.13-gentoo-r3 Bacula 1.36.3-r2 (the latest in Portage) Internal SCSI DLT VS160 tape drive -- Jeremy Koppel
RE: [Bacula-users] Retention / recycling clarification
Thanks for the tip; I'll change some settings and see how it goes.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Helmut
Raubenheimer
Sent: Wednesday, March 08, 2006 17:14
To: Jeremy Koppel
Cc: bacula-users@lists.sourceforge.net
Subject: Re: [Bacula-users] Retention / recycling clarification
Hi,
as long as bacula finds appendable volumes it tries to append to them.
The volume retention period begins when the volume status changes from
append to whatever.
Add eg.
Volume Use Duration = 20 hours
to the pool resource and bacula will change status from appendable to
used 20 hours after first use. After the volume status has changed the
volume retention period begins.
Regards
Helmut
Jeremy Koppel schrieb:
Hello,
I'm getting some behavior I didn't expect while trying to
set up a rotation schedule for one of the sites I manage. They have a
single server with a single VS160 tape drive. I have it run a full
backup early Thursday and incrementals fri-wed; and have 2 tapes per
pool, which I plan to rotate. Config is below.
Trouble is, from the 'list volumes' command, it looks like
Bacula will want to append to the DailyA1 tape, instead of rewinding
the
tape and overwriting it from scratch. And I actually had that happen
at
our main site, too; it kept appending until the backup failed in the
middle, asking for a new tape. How do I get it to really recycle the
tape?
Thanks!
--Jeremy Koppel
*_Shortened output from 'list volumes':_*
+-++---++--+
--+-+--+---+---+-+
| mediaid | volumename | volstatus | volbytes | volfiles |
volretention | recycle | slot | inchanger | mediatype | lastwritten |
+-++---++--+
--+-+--+---+---+-+
| 1 | DailyA1| Append| 33,101,955,019 | 46 |
777,600 | 1 |0 | 1 | VS160 | 2006-03-01
01:00:32 |
*_Config:_*
*_ _*
# * Schedules *
Schedule {
Name = BiweeklyCycle
Run = Level=Full Pool=Weekly thu at 1:00
Run = Level=Incremental Pool=Daily fri-wed at 1:00
}
# *** Pools ***
Pool {
Name = Daily
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 9
Accept Any Volume = yes
}
Pool {
Name = Weekly
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 9 days .
Accept Any Volume = yes
}
---
This SF.Net email is sponsored by xPML, a groundbreaking scripting
language
that extends applications into web and mobile media. Attend the live
webcast
and join the prime developer group breaking into this new coding
territory!
http://sel.as-us.falkag.net/sel?cmdÿkkid0944bid$1720dat1642
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
---
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnkkid0944bid$1720dat1642
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Retention / recycling clarification
Hello,
Im
getting some behavior I didnt expect while trying to set up a rotation
schedule for one of the sites I manage. They have a single server with a
single VS160 tape drive. I have it run a full backup early Thursday and
incrementals fri-wed; and have 2 tapes per pool, which I plan to rotate.
Config is below.
Trouble
is, from the list volumes command, it looks like Bacula will want
to append to the DailyA1 tape, instead of rewinding the tape and overwriting it
from scratch. And I actually had that happen at our main site, too; it
kept appending until the backup failed in the middle, asking for a new tape.
How do I get it to really recycle the tape?
Thanks!
--Jeremy Koppel
Shortened output from list volumes:
+-++---++--+--+-+--+---+---+-+
| mediaid | volumename | volstatus |
volbytes | volfiles | volretention |
recycle | slot | inchanger | mediatype | lastwritten |
+-++---++--+--+-+--+---+---+-+
| 1 |
DailyA1 | Append | 33,101,955,019
| 46 |
777,600 | 1 |
0
|
1 | VS160 | 2006-03-01 01:00:32 |
Config:
# * Schedules
*
Schedule {
Name = BiweeklyCycle
Run = Level=Full Pool=Weekly thu at 1:00
Run = Level=Incremental Pool=Daily fri-wed at 1:00
}
# *** Pools
***
Pool {
Name = Daily
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 9
Accept Any Volume = yes
}
Pool {
Name = Weekly
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 9 days .
Accept Any Volume = yes
}
RE: [Bacula-users] Scheduling question
In the event of a restore, if I understand it correctly, let me illustrate an example. Say you have scheduled a full backup on Sunday and incremental backups Monday through Saturday. If somebody comes to you on Thursday and says that they deleted a folder on the file server yesterday (Wednesday), the restore would look for the most recent full backup (Sunday), restore it, then restore the changes in the incremental backup Monday, Tuesday, and Wednesday. Now, say I do a manual full backup on Tuesday, to take offsite. In the same restore scenario before, would Bacula then look for the most recent full backup (now Tuesday), and try to restore from it? (Since the tape would be inaccessible.) -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ryan Novosielski Sent: Monday, February 27, 2006 14:51 To: bacula-list Subject: Re: [Bacula-users] Scheduling question I'm not sure I understand the question -- jobs behave the same way whether they are run by the scheduler or if they are run from the console. Incidentally, what I meant was that I clicked on the calendar in Data Protector and scheduled each monthly backup by hand (actually, did them monthly but modified those that fell on weekends), meaning that I had monthly backups except certain months and special backups on the first weekday of those special months, etc.. I'm not actually sure how you would handle something like this in bacula, as the schedules are defined prior to launch of the daemons and I'm not aware of any capability to add to the schedule from the console. But you could manually run these backups from the console on the day in question. _ _ _ _ ___ _ _ _ |Y#| | | |\/| | \ |\ | | | Ryan Novosielski - User Support Spec. III |$| |__| | | |__/ | \| _| | [EMAIL PROTECTED] - 973/972.0922 (2-0922) \__/ Univ. of Med. and Dent.| IST/AST - NJMS Medical Science Bldg - C630 Jeremy Koppel wrote: Actually, just handling it manually is an interesting option, too, as we need manual intervention to take the tape offsite anyway. But how does Bacula handle a restore in that event? If you do differentials the rest of the week, does it diff from your automated Sunday full backup, or does it start with your full, manual, offsite backup, thus requiring you to have it onhand in the event of a restore? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ryan Novosielski Sent: Friday, February 24, 2006 09:09 To: Martin Simmons Cc: [EMAIL PROTECTED]; bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Scheduling question I learned that things like this were a problem when I started using bacula, as there is no easy way to schedule it. As it was, with Data Protector, I had been doing it by hand anyway. Doing the 1st and 3rd Tuesday is how I ultimately handled the situation. We don't have off hours staff anyway, and I had to try to avoid Monday and Friday since they are often holidays. Martin Simmons wrote: On Thu, 23 Feb 2006 20:42:09 +0100, Arno Lehmann [EMAIL PROTECTED] said: Hello, On 2/23/2006 8:11 PM, Jeremy Koppel wrote: Hello, We've been using Bacula in our company since the beginning of the year with a good deal of success, but I had a scheduling question that I hadn't anticipated earlier. As part of our backup rotation, I have 2 additional pools, Offsite1 and Offsite2 (just 1 tape each for now), that are scheduled to be written to on alternating Saturdays. How it actually breaks down is: ... * The Question: * But I see that we have a problem coming up in April; 5 Saturdays. So if I specify either pool for the 5^th Saturday, we loose our rotation. I see that what we actually need to do is to set these up for 'every other Saturday' regardless of when it falls during the month. My question is, how do you do that? I'd try using the week of year specification in the schedule. You'd simply schedule like run = pool=offsite1 w01,w03,w05,... sat at 2:00 the problems here: I don't know if this actually works :-) and you still have to worry about the odd number of weeks per year. At least only one a year, and simply by swapping the pool definitions in the job overrides of the schedules... The better solution, also absolutely untested, would be to use the python interface and implement a simple script that determines which pool to use. I even assume it's impossible at the moment, because in the manual it states that the Pool attribute of a job object in python is read-only... but that might change one fine day :-) Another way is to run both jobs every week and have a RunBeforeJob script that returns an non-zero exit status if it is the wrong week (which causes the job to abort). __Martin --- This SF.Net email is sponsored by xPML, a groundbreaking scripting
RE: [Bacula-users] Scheduling question
Thank you both for your replies. I had seen that script included with the source files, and have been playing around with it. Doesn't look like it will work for us, though. For it to work, it seems to want one 'Offsite job' to either allow or cancel. We have 15 different jobs, and what determines what pool the backup is written to are the pool overrides in the Scheduler, so an exit code would indiscriminately kill the backup job even on 'regular' backup cycles. The week of the year idea sounds like a decent work-around for the moment; at least to buy us time. Don't know any Python myself, but I can't imagine it would be too time consuming to add an appropriate option to the scheduler instead. I'll look into how to submit that as a feature request. Again, thanks! -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Martin Simmons Sent: Friday, February 24, 2006 08:11 To: [EMAIL PROTECTED] Cc: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Scheduling question On Thu, 23 Feb 2006 20:42:09 +0100, Arno Lehmann [EMAIL PROTECTED] said: Hello, On 2/23/2006 8:11 PM, Jeremy Koppel wrote: Hello, We've been using Bacula in our company since the beginning of the year with a good deal of success, but I had a scheduling question that I hadn't anticipated earlier. As part of our backup rotation, I have 2 additional pools, Offsite1 and Offsite2 (just 1 tape each for now), that are scheduled to be written to on alternating Saturdays. How it actually breaks down is: ... * The Question: * But I see that we have a problem coming up in April; 5 Saturdays. So if I specify either pool for the 5^th Saturday, we loose our rotation. I see that what we actually need to do is to set these up for 'every other Saturday' regardless of when it falls during the month. My question is, how do you do that? I'd try using the week of year specification in the schedule. You'd simply schedule like run = pool=offsite1 w01,w03,w05,... sat at 2:00 the problems here: I don't know if this actually works :-) and you still have to worry about the odd number of weeks per year. At least only one a year, and simply by swapping the pool definitions in the job overrides of the schedules... The better solution, also absolutely untested, would be to use the python interface and implement a simple script that determines which pool to use. I even assume it's impossible at the moment, because in the manual it states that the Pool attribute of a job object in python is read-only... but that might change one fine day :-) Another way is to run both jobs every week and have a RunBeforeJob script that returns an non-zero exit status if it is the wrong week (which causes the job to abort). __Martin --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmdkkid0944bid$1720dat1642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users --- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnkkid0944bid$1720dat1642 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Scheduling question
Hello,
Weve
been using Bacula in our company since the beginning of the year with a good
deal of success, but I had a scheduling question that I hadnt
anticipated earlier. As part of our backup rotation, I have 2 additional
pools, Offsite1 and Offsite2 (just 1 tape each for now), that are scheduled to
be written to on alternating Saturdays. How it actually breaks down is:
We have 15 jobs setup that point to
one of two schedules, depending on the retention we need.
The schedule then determines when
and where to put the backup; in the case of the Offsite backups, the Offsite1
pool is used on the 1st and 3rd Saturdays, and the
Offsite2 pool is used on the 2nd and 4th Saturdays.
(The actual schedule is below)
The Question:
But I see that we have a problem
coming up in April; 5 Saturdays. So if I specify either pool for the 5th
Saturday, we loose our rotation. I see that what we actually need to do
is to set these up for every other Saturday regardless of when it
falls during the month. My question is, how do you do that?
Our Schedule:
Schedule {
Name = MonthlyCycle
Run = Level=Full Pool=Offsite1 1st sat at 2:00
Run = Level=Full Pool=Offsite2 2nd sat at 2:00
Run = Level=Full Pool=Offsite1 3rd sat at 2:00
Run = Level=Full Pool=Offsite2 4th sat at 2:00
Run = Level=Full Pool=Weekly sun at 2:00
Run = Level=Differential Pool=Daily FullPool=Weekly
mon-fri at 2:00
}
Schedule {
Name = LongTerm
Run = Level=Full Pool=Offsite1 1st sat at 1:00
Run = Level=Full Pool=Offsite2 2nd sat at 1:00
Run = Level=Full Pool=Offsite1 3rd sat at 1:00
Run = Level=Full Pool=Offsite2 4th sat at 1:00
Run = Level=Full Pool=Monthly 1st sun at 1:00
Run = Level=Full Pool=Weekly 2nd-4th sun at 1:00
Run = Level=Differential Pool=Daily FullPool=Weekly
mon-fri at 1:00
}
Thanks!
--Jeremy Koppel
