Re: [Bacula-users] Multiple drives in changer
> Date: Tue, 6 Apr 2010 08:52:24 -0600 > From: Robert LeBlanc > > On Tue, Apr 6, 2010 at 6:13 AM, Matija Nalis > >> > wrote: >> > On Fri, Apr 02, 2010 at 10:36:59AM -0600, Robert LeBlanc wrote: >>> > > On Fri, Apr 2, 2010 at 2:44 AM, Matija Nalis >>> > > >>> > > > > > I think you need to set > > > Prefer Mounted Volumes = no >>> > > >>> > > I guess this is where we need clarification about what is an available >>> > > drive. I took this to mean a drive that has no tape is more available, >> > and >>> > > then a drive that does already have a tape mounted would be next in >>> > > availability. >> > >> > Hm, it looks to me that any drive which is not doing R/W operation >> > (no matter if there is a tape in drive or not) is counted as available. >> > I could be wrong on that, though. >> > >> > Anyway, the safest way to know is to test it and let the others know >> > how it goes :) >> > >From my observations of a few tests, this indeed seems to be the case. If > the drive is not being R/W to/from, it is considered available. > > >>> > > It seems that as long as no job is writing to that tape, then >>> > > the drive is available. I do want this setting to yes and not no, >> > however, I >>> > > would like to minimize tape changes, but take advantage of the multiple >>> > > drives. >> > >> > From what I see in practice, "Prefer Mounted Volumes = yes" would >> > make sure there is only one drive in each pool that does the writing. >> > >> > For example, I have pool of 4 drives and I start 10 jobs at the same >> > time, all using the same pool. I have an concurrency of >10 and >> > spooling enabled, so all the jobs run at once and start spooling to >> > disk -- but when they need to despool, one drive will grab a free >> > tape from Scratch, and all the jobs will wait for their in turn to >> > write to one tape in one drive, leaving 3 drives idle all the time. >> > Only when that tape is full, another one is loaded, and the process >> > repeats. >> > >> > I think same happens when I disable spooling, but then the 4 jobs all >> > interleave writes -- but still all of them will write on one tape in >> > one drive only. >> > >> > If you set "Prefer Mounted Volumes = no", then all 4 drives get >> > loaded with 4 fresh tapes (or just use them if right tapes are >> > already in right drives -- I guess, I have autochanger) and each >> > tape gets written to at the same time, maximizing drive (and thus, >> > the tape) usage. >> > >> > But "no" setting can (or at least could in the past) lead to >> > deadlocks sometimes (if you have autochanger), when no new jobs will >> > get serviced because drive A will wait for tape 2 that is currently >> > in drive B, and at the same time drive B will wait for tape 1 which >> > is currently in drive A. Then the manual intervention (umount/mount) >> > is needed (which is a big problem for us as we have lots of jobs/tapes). >> > >> > The (recommended) alternative is to go semi-manual way -- dedicate >> > special pool for each drive, and go with "Prefer Mounted Volumes = >> > yes" Then one can (and indeed, must) specify manually which jobs will >> > go in which pools (and hence, in which drives) and can optimize it >> > for maximum parallelism without deadlocks -- but it requires more >> > planing and is problematic if your backups are more dynamic and hard >> > to predict, and you have to redesign when you add/upgrade/remove >> > drives, and your pools might become somewhat harder to manage. >> > > This is exactly my experience, and my goal is not to use multiple drives in > the same pool at the same time, it's to use drives for different pools at > the same time one drive per pool. We are looking to bring up a lot more > storage in the future and will probably adopt the mentality of multiple > daily, weekly, monthly pools and split them up based on the number of drives > we want to run concurrently. I think that is the best way to go with Bacula > for what we want to do. > > Thanks, > > Robert LeBlanc > Life Sciences & Undergraduate Education Computer Support > Brigham Young University Your other option is to try out the "Maximum Concurrent Jobs" in the device section of your storage daemon's config. That's working well for me. One word of caution though: since the way it allocates jobs to drives is not the same as prefer mounted volumes you should carefully consider all the different concurrent parameters and how they relate to each other. Here's an example of how I first tried it (not very good)... dir.conf: Director: Maximum concurrent jobs = 20 Jobdefs: Maximum concurrent jobs = 10 sd.conf: Storage: Maximum Concurrent jobs = 20 Device1: Maximum concurrent jobs = 2 Device2: Maximum concurrent jobs = 2 The way prefer mounted volumes worked, it round robin alternated the drive assignments for each job as it started them. For This new directive it does the opposite, assigns them jobs to the first until it hits the max, then assigns the rest to the next dr
Re: [Bacula-users] Multiple drives in changer
On Tue, Apr 6, 2010 at 6:13 AM, Matija Nalis > wrote: > On Fri, Apr 02, 2010 at 10:36:59AM -0600, Robert LeBlanc wrote: > > On Fri, Apr 2, 2010 at 2:44 AM, Matija Nalis > > > > > > > I think you need to set > > > Prefer Mounted Volumes = no > > > > I guess this is where we need clarification about what is an available > > drive. I took this to mean a drive that has no tape is more available, > and > > then a drive that does already have a tape mounted would be next in > > availability. > > Hm, it looks to me that any drive which is not doing R/W operation > (no matter if there is a tape in drive or not) is counted as available. > I could be wrong on that, though. > > Anyway, the safest way to know is to test it and let the others know > how it goes :) > >From my observations of a few tests, this indeed seems to be the case. If the drive is not being R/W to/from, it is considered available. > > It seems that as long as no job is writing to that tape, then > > the drive is available. I do want this setting to yes and not no, > however, I > > would like to minimize tape changes, but take advantage of the multiple > > drives. > > From what I see in practice, "Prefer Mounted Volumes = yes" would > make sure there is only one drive in each pool that does the writing. > > For example, I have pool of 4 drives and I start 10 jobs at the same > time, all using the same pool. I have an concurrency of >10 and > spooling enabled, so all the jobs run at once and start spooling to > disk -- but when they need to despool, one drive will grab a free > tape from Scratch, and all the jobs will wait for their in turn to > write to one tape in one drive, leaving 3 drives idle all the time. > Only when that tape is full, another one is loaded, and the process > repeats. > > I think same happens when I disable spooling, but then the 4 jobs all > interleave writes -- but still all of them will write on one tape in > one drive only. > > If you set "Prefer Mounted Volumes = no", then all 4 drives get > loaded with 4 fresh tapes (or just use them if right tapes are > already in right drives -- I guess, I have autochanger) and each > tape gets written to at the same time, maximizing drive (and thus, > the tape) usage. > > But "no" setting can (or at least could in the past) lead to > deadlocks sometimes (if you have autochanger), when no new jobs will > get serviced because drive A will wait for tape 2 that is currently > in drive B, and at the same time drive B will wait for tape 1 which > is currently in drive A. Then the manual intervention (umount/mount) > is needed (which is a big problem for us as we have lots of jobs/tapes). > > The (recommended) alternative is to go semi-manual way -- dedicate > special pool for each drive, and go with "Prefer Mounted Volumes = > yes" Then one can (and indeed, must) specify manually which jobs will > go in which pools (and hence, in which drives) and can optimize it > for maximum parallelism without deadlocks -- but it requires more > planing and is problematic if your backups are more dynamic and hard > to predict, and you have to redesign when you add/upgrade/remove > drives, and your pools might become somewhat harder to manage. > This is exactly my experience, and my goal is not to use multiple drives in the same pool at the same time, it's to use drives for different pools at the same time one drive per pool. We are looking to bring up a lot more storage in the future and will probably adopt the mentality of multiple daily, weekly, monthly pools and split them up based on the number of drives we want to run concurrently. I think that is the best way to go with Bacula for what we want to do. Thanks, Robert LeBlanc Life Sciences & Undergraduate Education Computer Support Brigham Young University -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Multiple drives in changer
On Fri, Apr 02, 2010 at 10:36:59AM -0600, Robert LeBlanc wrote: > On Fri, Apr 2, 2010 at 2:44 AM, Matija Nalis > > I think you need to set > > Prefer Mounted Volumes = no > > I guess this is where we need clarification about what is an available > drive. I took this to mean a drive that has no tape is more available, and > then a drive that does already have a tape mounted would be next in > availability. Hm, it looks to me that any drive which is not doing R/W operation (no matter if there is a tape in drive or not) is counted as available. I could be wrong on that, though. Anyway, the safest way to know is to test it and let the others know how it goes :) > It seems that as long as no job is writing to that tape, then > the drive is available. I do want this setting to yes and not no, however, I > would like to minimize tape changes, but take advantage of the multiple > drives. >From what I see in practice, "Prefer Mounted Volumes = yes" would make sure there is only one drive in each pool that does the writing. For example, I have pool of 4 drives and I start 10 jobs at the same time, all using the same pool. I have an concurrency of >10 and spooling enabled, so all the jobs run at once and start spooling to disk -- but when they need to despool, one drive will grab a free tape from Scratch, and all the jobs will wait for their in turn to write to one tape in one drive, leaving 3 drives idle all the time. Only when that tape is full, another one is loaded, and the process repeats. I think same happens when I disable spooling, but then the 4 jobs all interleave writes -- but still all of them will write on one tape in one drive only. If you set "Prefer Mounted Volumes = no", then all 4 drives get loaded with 4 fresh tapes (or just use them if right tapes are already in right drives -- I guess, I have autochanger) and each tape gets written to at the same time, maximizing drive (and thus, the tape) usage. But "no" setting can (or at least could in the past) lead to deadlocks sometimes (if you have autochanger), when no new jobs will get serviced because drive A will wait for tape 2 that is currently in drive B, and at the same time drive B will wait for tape 1 which is currently in drive A. Then the manual intervention (umount/mount) is needed (which is a big problem for us as we have lots of jobs/tapes). The (recommended) alternative is to go semi-manual way -- dedicate special pool for each drive, and go with "Prefer Mounted Volumes = yes" Then one can (and indeed, must) specify manually which jobs will go in which pools (and hence, in which drives) and can optimize it for maximum parallelism without deadlocks -- but it requires more planing and is problematic if your backups are more dynamic and hard to predict, and you have to redesign when you add/upgrade/remove drives, and your pools might become somewhat harder to manage. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Multiple drives in changer
On Fri, Apr 2, 2010 at 2:44 AM, Matija Nalis > wrote: > On Thu, Apr 01, 2010 at 08:49:43AM -0600, Robert LeBlanc wrote: > > I have two LTO-3 drives in a changer and three LTO-3 pools. Earlier > version > > of Bacula would try to use an empty drive before unloading a drive when a > > tape from a different pool was requested. I used to also be able to run > in > > parallel migration jobs from two different pools at the same time. Since > > moving to 5.0.1, my second drive goes unused. Is there some change in the > > code that prevents this behavior? > > I think you need to set > Prefer Mounted Volumes = no > > to get old behaviour (maximum parallelism). > > Note however that it might lead to deadlocks (or at least, it could > in the past... don't know if that was fixed). > > Prefer Mounted Volumes = If the Prefer Mounted Volumes directive is set to yes (default yes), the Storage daemon is requested to select either an Autochanger or a drive with a valid Volume already mounted in preference to a drive that is not ready. This means that all jobs will attempt to append to the same Volume (providing the Volume is appropriate -- right Pool, ... for that job). If no drive with a suitable Volume is available, it will select the first available drive. Note, any Volume that has been requested to be mounted, will be considered valid as a mounted volume by another job. This if multiple jobs start at the same time and they all prefer mounted volumes, the first job will request the mount, and the other jobs will use the same volume. If the directive is set to no, the Storage daemon will prefer finding an unused drive, otherwise, each job started will append to the same Volume (assuming the Pool is the same for all jobs). Setting Prefer Mounted Volumes to no can be useful for those sites with multiple drive autochangers that prefer to maximize backup throughput at the expense of using additional drives and Volumes. This means that the job will prefer to use an unused drive rather than use a drive that is already in use. I guess this is where we need clarification about what is an available drive. I took this to mean a drive that has no tape is more available, and then a drive that does already have a tape mounted would be next in availability. It seems that as long as no job is writing to that tape, then the drive is available. I do want this setting to yes and not no, however, I would like to minimize tape changes, but take advantage of the multiple drives. Robert LeBlanc Life Sciences & Undergraduate Education Computer Support Brigham Young University -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Multiple drives in changer
On Thu, Apr 01, 2010 at 08:49:43AM -0600, Robert LeBlanc wrote: > I have two LTO-3 drives in a changer and three LTO-3 pools. Earlier version > of Bacula would try to use an empty drive before unloading a drive when a > tape from a different pool was requested. I used to also be able to run in > parallel migration jobs from two different pools at the same time. Since > moving to 5.0.1, my second drive goes unused. Is there some change in the > code that prevents this behavior? I think you need to set Prefer Mounted Volumes = no to get old behaviour (maximum parallelism). Note however that it might lead to deadlocks (or at least, it could in the past... don't know if that was fixed). -- Matija Nalis Odjel racunalno-informacijskih sustava i servisa Hrvatska akademska i istrazivacka mreza - CARNet Josipa Marohnica 5, 1 Zagreb tel. +385 1 6661 616, fax. +385 1 6661 766 www.CARNet.hr -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Multiple drives in changer
I have two LTO-3 drives in a changer and three LTO-3 pools. Earlier version of Bacula would try to use an empty drive before unloading a drive when a tape from a different pool was requested. I used to also be able to run in parallel migration jobs from two different pools at the same time. Since moving to 5.0.1, my second drive goes unused. Is there some change in the code that prevents this behavior? Thanks, Robert LeBlanc Life Sciences & Undergraduate Education Computer Support Brigham Young University -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Multiple drives in changer
On Thu, Apr 1, 2010 at 8:49 AM, Robert LeBlanc wrote: > I have two LTO-3 drives in a changer and three LTO-3 pools. Earlier version > of Bacula would try to use an empty drive before unloading a drive when a > tape from a different pool was requested. I used to also be able to run in > parallel migration jobs from two different pools at the same time. Since > moving to 5.0.1, my second drive goes unused. Is there some change in the > code that prevents this behavior? > > Well, I'm not sure where the inconsistencies are, but I started migrating our Daily pool from disk to tape and it unloaded the Weekly pool tape that was already loaded, then it loaded a Daily tape and ran a few migration jobs. I then run the Weekly migration job and when the daily migration job that was running finished, it unloaded the Daily tape from Drive 1 and loaded it into Drive 2 then loaded a Weekly tape into Drive 1 and is going on it's merry way running both jobs at once. I'm not sure why there is all the changing going on. Robert LeBlanc Life Sciences & Undergraduate Education Computer Support Brigham Young University -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
