Re: [Bacula-users] [Bacula-devel] Idea/suggestion for dedicated disk-based sd
Hello, I haven't seen the original messages, so I am not sure if I understand the full concept here so my remarks may not be pertinent. However, from what I see, this is basically similar to what BackuPC does. The big problem I have with it is that it does not scale well to thousands of machines. If I were thinking about changing the disk Volume format, I would start by looking at how git handles storing objects, and whether git can scale to handle a machine with 40 million file entries. One thing is sure is that, unless some new way of implementing hardlinks is implemented, you will never see Bacula using hard links in the volumes. That is a sure way to make your machine unbootable if you scale large enough Just backup enough clients with BackupPC and one day you will find that fsck no longer works. I suspect that it will require only a couple hundred million hardlinks before a Linux machine will no longer boot. Regards, Kern On Wednesday 07 April 2010 22:15:24 Phil Stracchino wrote: > On 04/07/10 12:06, Robert LeBlanc wrote (in bacula-users): > > So still thinking about this, is there any reason to not have a > > hierarchical file structure for disk based backup rather than a > > serialized stream? Here are my thought, any comments welcome to have a > > good discussion about this. > > > > SD_Base_Dir > > +- PoolA > > +- PoolB > > +- JobID1 > > +- JobID2 > > +- Clientinfo.bacula (Bacula serial file that > > holds information similar to block header) > > +- Original File Structure (File structure from > > client is maintained and repeated here, allows for browsing of files > > outside of bacula) > > +- ClientFileA > > +- ClientFileA.bacula (Bacula serial file > > that holds information similar to the unix file attribute package) > > +- ClientFileB > > +- ClientFileB.bacula > > +- ClientDirA > > +- ClientDirA.bacula > > > > Although it's great to reuse code, I think something like this would > > be very benifical to disk based backups. The would help increase dedup > > rates and some file systems like btrfs and ZFS may be able to take > > advantage of linked files (there has been some discussion on the btrfs > > list about things like this). This would also allow it to reside on > > any file system as all the ACL and information is being serialized in > > separate files which keeps unique data out of the blocks of possible > > duplicated data. I think we could even reuse a lot of the > > serialization code, so it would just differ in how it writes the > > stream of data. > > After having thought about this a bit, I believe the idea has > significant merit. Tape and disk differ significantly enough that there > is no conceptual reason not to have separate tape-specific and > disk-specific SDs. So long as the storage logically looks the same from > the point of view of other daemons, the other daemons don't need to know > that the underlying storage architecture is different. Creating a > hierarchical disk SD in this fashion that appears to the rest of Bacula > exactly the same as the existing FD does, and yet takes advantage of the > features offered by such an implementation, will not necessarily be a > trivial problem. It's a pretty major project and, if approved, wouldn't > happen right away. > > The major problem I see at the moment, architecturally speaking, is that > at the present time, this would break both migration and copy jobs > between volumes on the new disk-only SD and volumes of any kind on the > traditional SD, because Bacula does not yet support copy or migration > between different SDs. At this time, both source and destination > devices are required to be on the same SD. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Possibility of parallelising encryption?
Phil Stracchino wrote: > FWIW, memory says we tried using GnuTLS once. It turns out that for > this purpose it is (or was) horribly broken. In fact, at the time, I > seem to recall the mere presence of Red Hat's GnuTLS lib broke Bacula > altogether. Yeah. I've found it to be pretty broken in other ways - I'm STILL having to rebuild the subversion client against openssl to get working support for client certificates due to a gnutls bug, and I rebuild netatalk against openssl to get dhx password hashes that aren't supported in gnutls. It doesn't look like gnutls is any better than openssl in terms of parallel encryption support anyway. The whole affair seems to be a licensing kerfuffle rather than anything else. >> It'd be lovely to be able to use the IPP libraries in Bacula (and many >> other things) for parallel crypto and many other parallel tasks, as >> they're excellent even without special hardware. Unfortunately they're >> rather GPL-incompatible and are only "free" for non-commercial use. > > It would indeed be very nice to be able to use that kind of hardware > crypto support without having to jump through licensing hoops. You don't have to jump through any licensing hoops to use the hardware crypto. Intel have submitted patches that've been accepted into OpenSSL 1.0 . For Via's C3/C7 PadLock hardware crypto, support has been around for ages and is certainly in OpenSSL 0.9.8. Run "openssl engine" for a list of built-in engines in your version of OpenSSL. Dynamically loaded engines for rarer hardware crypto devices and/or those that need special configuration are availible in /usr/lib/ssl/engines/ . Those licensing hoops would have to be jumped to use Intel's fast parallel *software* crypto engine to do multi-core/multi-threaded AES-CTR mode encryption. Bacula should probably work with Intel's hardware crypto out of the box. If it doesn't, most likely all that'd be required would be to call: ENGINE_load_builtin_engines(); ENGINE_register_all_complete(); in init_crypto() , and: ENGINE_cleanup(); when crypto is cleaned up. See "man 3 engine". In fact, as PadLock support comes pre-loaded and Intel crypto probably will too, it may not even be neccessary to call ENGINE_load_builtin_engines() at all, only ENGINE_register_all_complete(). Asking openssl to load all engines will let it use other less common hardware crypto systems like some of the add-on hw crypto PCI cards, though, and it's cheap enough not to even be detectable in something as long-lived as bacula-fd. OpenSSL is smart enough to pick a hardware engine if one exists, and fall back to software if there's no suitable engine for the task at hand. IIRC that's all I had to do to patch PadLock support into OpenSSH when I needed it for my thin clients at work. It's a trivial change that would enable Bacula to use any builtin hardware crypto engine supported by OpenSSL. Worth making, so that by the time the new Intel hardware hits Bacula supports it? -- Craig Ringer -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Idea/suggestion for dedicated disk-based sd
Phil Stracchino wrote: >> Most practical disk backup setups will involve large RAID-5, RAID-6 or >> RAID-10 arrays. These tend to be striped across the spindles anyway, and >> the file system is rarely properly aware of how this striping occurs. > > *nod* Indeed. Nor should it care, since for general purposes, it > shouldn't matter. It's not the filesystem's problem - except of course > in advanced filesystems like ZFS where the filesystem IS the RAID > implementation. It can be of significant impact on write, as the FS can coalesce writes into stripe sized pieces, minimising read-modify-write cycle. XFS certainly and EXT4 I think, are software md RAID aware and will optimised FS creation based on number of drives and RAID level. In the case of hardware raid or stacked md, it is certainly worth working out the appropriate switch settings manually when creating the FS. Regards, Richard -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Retention Policy Pruning Files?
On 04/07/10 22:21, Craig Ringer wrote: > May, John wrote: >> Whenever I archive files, I always put them in an ‘Archive’ pool I >> created. After only about a week or two, I am no longer able to restore >> because I get the following message: >> >> For one or more of the JobIds selected, no files were found, >> so file selection is not possible. >> Most likely your retention policy pruned the files. >> >> >> Do you want to restore all the files? (yes|no): > > > You can still restore the job. You just have to restore all of it or > none of it, you don't get to pick individual files. Well, actually, you can still restore individual files. You just need to bscan the volume back into the catalog first. Looks like your retention times need some tuning, though. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Retention Policy Pruning Files?
May, John wrote: > Whenever I archive files, I always put them in an ‘Archive’ pool I > created. After only about a week or two, I am no longer able to restore > because I get the following message: > > For one or more of the JobIds selected, no files were found, > so file selection is not possible. > Most likely your retention policy pruned the files. > > > Do you want to restore all the files? (yes|no): You can still restore the job. You just have to restore all of it or none of it, you don't get to pick individual files. I frequently find this quite sufficient for my longer term backups. Do you actually need to retain detailed file lists for those? Remember, they're still in the volumes, just not in the catalog database. -- Craig Ringer -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] upgrading bacula
John Drescher wrote: >> I need to upgrade the bacula that I am running here on a fedora-linux >> server. >> Currently we run version 2.0.2 and I am attempting to move to 5.0.1 >> Questions: >> 1) Do I need to change the database? ( mysql 5.0.27 ) ( i.e. are the >> tables and indexes the same? ) > > There are several database upgrades since 2.0.2. You need to run the > bacula upgrade scripts. Also I would upgrade your mysql. 5.0.27 is as > at least as ancient as bacula-2.0.2. Yep... and has known data-loss and data-corruption bugs. Lots of them. Early MySQL 5.x was B.U.G.G.Y. -- Craig Ringer -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] client rejected Hello command (SOLVED)
On 4/7/2010 10:40 AM, Matija Nalis wrote: > On Tue, Apr 06, 2010 at 05:52:54PM -0400, Roland Roberts wrote: > >> When the client is run in the foreground in verbose mode, I get this from >> the client: >> >> archos-fd: cram-md5.c:73-0 send: auth cram-md5 >> <1018903137.1269886...@archos-fd> ssl=0 >> archos-fd: cram-md5.c:152-0 sending resp to challenge: >> a9/LnX1KKQ/Gj6+qd++t1B >> > that is strange. Looking at the code, if you get this far, one of the > 3 things should happen: > > a) timeout or socket error, in which case it would have printed > "Receive chanllenge response failed. ERR=%s", or > > b) authorization succeeds (it receives "1000 OK auth") in which case > nothing is printed and cram_md5_respond returns true, or > > c) authorization fails, and "Received bad response: %s" is printed > > As nothing is printed, one could only assume the authorization > succeeded, but things stopped somewhere afterwards. > > Can you raise debug level (say, -d200) on both the director and FD > (and maybe even on SD, just in case) and send output of that ? > Okay, problem solved. It turns out the upgrade was broken. When the director is installed, the RPM is supposed to set up a link from /usr/sbin/bacula-dir to /etc/alternatives/bacula-dir which in turns points to the real directory (in my case, the postgres version). However, the upgrade had left an old binary in place. That shouldn't have worked, but somehow I also still had links from libssl.so.6 and libcrypto.so.6 that pointed to the current versions of those shared libs. This meant bacula-dir would start, but the eventual effect was what I described before. This was not corrected even though I had tried "yum reinstall" for all of the bacula packages. It was only when started looking for things that didn't correspond to any package that I found the extra binary. I removed it and then did a yum reinstall on the bacula packages and now the director works (well, after upgrading the database and fixing some things in bacula-dir.conf that no longer work). Thank you for the above suggestions. Telling me what to expect from the auth was actually important in making me think about the ssl/crypto libraries because there had been an upgrade error with those that I had forgotten about/ignored when everything initially ran. That was just plain stupid on my part and I should have thought of it long ago. roland -- PGP Key ID: 66 BC 3B CD Roland B. Roberts, PhD RL Enterprises rol...@rlenter.com6818 Madeline Court rol...@astrofoto.org Brooklyn, NY 11220 -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Idea/suggestion for dedicated disk-based sd
On 04/07/10 18:52, Robert LeBlanc wrote: > I didn't think about the copy/migration jobs (I'm using them), and > that would be a problem. It seems for this to take off, the > copy/migration between SDs will have to be implemented. We would have > to look at the stream as a copy/migration is happening, i believe that > the record blocks are being rewritten with a new jobid and time, so it > seems that the data is already being reconstructed an rewritten. The > disk SD would have the provide the same stream from the hierarchical > file system and be able to take the reconstructed stream and build a > hierarchical file system from it. The question I have is what is the > barrier for implementing inter-sd copy/migration jobs? Technically speaking, the major step required to implement it would be to enable an SD to connect and communicate directly to another SD, something for which at present there is no provision. With that out of the way, implementing cross-SD copy-and-migrate would probably be fairly trivial. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Idea/suggestion for dedicated disk-based sd
On Wed, Apr 7, 2010 at 2:15 PM, Phil Stracchino wrote: > After having thought about this a bit, I believe the idea has > significant merit. Tape and disk differ significantly enough that there > is no conceptual reason not to have separate tape-specific and > disk-specific SDs. So long as the storage logically looks the same from > the point of view of other daemons, the other daemons don't need to know > that the underlying storage architecture is different. Creating a > hierarchical disk SD in this fashion that appears to the rest of Bacula > exactly the same as the existing FD does, and yet takes advantage of the > features offered by such an implementation, will not necessarily be a > trivial problem. It's a pretty major project and, if approved, wouldn't > happen right away. > > The major problem I see at the moment, architecturally speaking, is that > at the present time, this would break both migration and copy jobs > between volumes on the new disk-only SD and volumes of any kind on the > traditional SD, because Bacula does not yet support copy or migration > between different SDs. At this time, both source and destination > devices are required to be on the same SD. I didn't think about the copy/migration jobs (I'm using them), and that would be a problem. It seems for this to take off, the copy/migration between SDs will have to be implemented. We would have to look at the stream as a copy/migration is happening, i believe that the record blocks are being rewritten with a new jobid and time, so it seems that the data is already being reconstructed an rewritten. The disk SD would have the provide the same stream from the hierarchical file system and be able to take the reconstructed stream and build a hierarchical file system from it. The question I have is what is the barrier for implementing inter-sd copy/migration jobs? Thanks, Robert LeBlanc Life Sciences & Undergraduate Education Computer Support Brigham Young University -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Retention Policy Pruning Files?
What about archives I've already made using the 'bad' client with 'autoprune=yes' option? I tried to bscan the tape back into the database, but I still can't restore individual files. I get the error message about the files being pruned. John -Original Message- From: Martin Simmons [mailto:mar...@lispworks.com] Sent: Wednesday, April 07, 2010 1:56 PM To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Retention Policy Pruning Files? > On Wed, 7 Apr 2010 09:43:12 -0600, May, John said: > > Whenever I archive files, I always put them in an 'Archive' pool I created > After only about a week or two, I am no longer able to restore because I > get the following message: > > You have selected the following JobId: 686 > > Building directory tree for JobId(s) 686 ... > > For one or more of the JobIds selected, no files were found, > so file selection is not possible. > Most likely your retention policy pruned the files. > > Do you want to restore all the files? (yes|no): > > Can sombody take a look at my bacula-dir.conf file and see why this is > happening? > > All Archive jobs have different filesets, but go into the pool 'Archive'. > I've set a retention period of 30 years, job retention of 30 years, Recycle > = no, Auto Prune = no and Volume retention of 30 years, but the files > seem to be still getting pruned. I think the problem is that the client definition contains: > File Retention = 30 days # 30 days > AutoPrune = yes # Prune expired Jobs/Files This causes it to prune the file records of *all* jobs for that client whenever it prunes *any* job for that client. Assuming you need that file retention for other jobs on the client, you can work around the problem by adding a separate client definition for use by the archive jobs, with the same Address pararmeter as the normal one. __Martin -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Doubt about two process for restaure files
> On Mon, 05 Apr 2010 17:57:38 -0400, aspenbr said: > > Then only second process fine work, the first process don't complete process > totaly. My /var/log/message show this message at moment execution restore > command > > Apr 5 18:36:40 LinuxDefault bacula-dir: LinuxDefault-dir JobId 48: Error: > Bacula LinuxDefault-dir 5.0.0 (26Jan10): 05-Apr-2010 18:36:40 Build OS: > i686-pc-linux-gnu redhatJobId: 48 Job: > Restore-Backup-File-Clone.2010-04-05_18.36.38_43 Restore > Client: LinuxDefault Start time: 05-Apr-2010 18:36:40 > End time: 05-Apr-2010 18:36:40 Files Expected: 0 > Files Restored: 0 Bytes Restored: 0 Rate: > 0.0 KB/s FD Errors: 0 FD termination status: SD > termination status: Termination:*** Restore Error *** > Apr 5 18:36:40 LinuxDefault bacula-dir: LinuxDefault-dir JobId 48: Error: > Bacula LinuxDefault-dir 5.0.0 (26Jan10): 05-Apr-2010 18:36:40 Build OS: > i686-pc-linux-gnu redhatJobId: 48 Job: > Restore-Backup-File-Clone.2010-04-05_18.36.38_43 Restore > Client: LinuxDefault Start time: 05-Apr-2010 18:36:40 > End time: 05-Apr-2010 18:36:40 Files Expected: 0 > Files Restored: 0 Bytes Restored: 0 Rate: > 0.0 KB/s FD Errors: 1 FD termination status: SD > termination status: Termination:*** Restore Error *** > > Why the first process don't work ? What diferente between the process ? I don't know, but look for the error message in the line before "Error: Bacula LinuxDefault-dir" in the log. Also, try to get the log from the bconsole messages command (rather than syslog like above) because it will be formatted better. __Martin -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Retention Policy Pruning Files?
If that is the case, then I will need to create a few other client definitions for the same host because I do Archive/Full/Incremental backups on this same host. Obviously, the Archive I want kept around forever, the incremental only need to be kept for a month, and I need Quarterly backups kept forever. Consequently, I read another post made along the same lines: http://www.mail-archive.com/bacula-users@lists.sourceforge.net/msg35636.html -- John -Original Message- From: Martin Simmons [mailto:mar...@lispworks.com] Sent: Wednesday, April 07, 2010 1:56 PM To: bacula-users@lists.sourceforge.net Subject: Re: [Bacula-users] Retention Policy Pruning Files? > On Wed, 7 Apr 2010 09:43:12 -0600, May, John said: > > Whenever I archive files, I always put them in an 'Archive' pool I created > After only about a week or two, I am no longer able to restore because I > get the following message: > > You have selected the following JobId: 686 > > Building directory tree for JobId(s) 686 ... > > For one or more of the JobIds selected, no files were found, > so file selection is not possible. > Most likely your retention policy pruned the files. > > Do you want to restore all the files? (yes|no): > > Can sombody take a look at my bacula-dir.conf file and see why this is > happening? > > All Archive jobs have different filesets, but go into the pool 'Archive'. > I've set a retention period of 30 years, job retention of 30 years, Recycle > = no, Auto Prune = no and Volume retention of 30 years, but the files > seem to be still getting pruned. I think the problem is that the client definition contains: > File Retention = 30 days # 30 days > AutoPrune = yes # Prune expired Jobs/Files This causes it to prune the file records of *all* jobs for that client whenever it prunes *any* job for that client. Assuming you need that file retention for other jobs on the client, you can work around the problem by adding a separate client definition for use by the archive jobs, with the same Address pararmeter as the normal one. __Martin -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Idea/suggestion for dedicated disk-based sd
On 04/07/10 12:06, Robert LeBlanc wrote (in bacula-users): > So still thinking about this, is there any reason to not have a > hierarchical file structure for disk based backup rather than a > serialized stream? Here are my thought, any comments welcome to have a > good discussion about this. > > SD_Base_Dir > +- PoolA > +- PoolB > +- JobID1 > +- JobID2 > +- Clientinfo.bacula (Bacula serial file that > holds information similar to block header) > +- Original File Structure (File structure from > client is maintained and repeated here, allows for browsing of files > outside of bacula) > +- ClientFileA > +- ClientFileA.bacula (Bacula serial file > that holds information similar to the unix file attribute package) > +- ClientFileB > +- ClientFileB.bacula > +- ClientDirA > +- ClientDirA.bacula > > Although it's great to reuse code, I think something like this would > be very benifical to disk based backups. The would help increase dedup > rates and some file systems like btrfs and ZFS may be able to take > advantage of linked files (there has been some discussion on the btrfs > list about things like this). This would also allow it to reside on > any file system as all the ACL and information is being serialized in > separate files which keeps unique data out of the blocks of possible > duplicated data. I think we could even reuse a lot of the > serialization code, so it would just differ in how it writes the > stream of data. After having thought about this a bit, I believe the idea has significant merit. Tape and disk differ significantly enough that there is no conceptual reason not to have separate tape-specific and disk-specific SDs. So long as the storage logically looks the same from the point of view of other daemons, the other daemons don't need to know that the underlying storage architecture is different. Creating a hierarchical disk SD in this fashion that appears to the rest of Bacula exactly the same as the existing FD does, and yet takes advantage of the features offered by such an implementation, will not necessarily be a trivial problem. It's a pretty major project and, if approved, wouldn't happen right away. The major problem I see at the moment, architecturally speaking, is that at the present time, this would break both migration and copy jobs between volumes on the new disk-only SD and volumes of any kind on the traditional SD, because Bacula does not yet support copy or migration between different SDs. At this time, both source and destination devices are required to be on the same SD. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Retention Policy Pruning Files?
> On Wed, 7 Apr 2010 09:43:12 -0600, May, John said: > > Whenever I archive files, I always put them in an 'Archive' pool I created > After only about a week or two, I am no longer able to restore because I > get the following message: > > You have selected the following JobId: 686 > > Building directory tree for JobId(s) 686 ... > > For one or more of the JobIds selected, no files were found, > so file selection is not possible. > Most likely your retention policy pruned the files. > > Do you want to restore all the files? (yes|no): > > Can sombody take a look at my bacula-dir.conf file and see why this is > happening? > > All Archive jobs have different filesets, but go into the pool 'Archive'. > I've set a retention period of 30 years, job retention of 30 years, Recycle > = no, Auto Prune = no and Volume retention of 30 years, but the files > seem to be still getting pruned. I think the problem is that the client definition contains: > File Retention = 30 days # 30 days > AutoPrune = yes # Prune expired Jobs/Files This causes it to prune the file records of *all* jobs for that client whenever it prunes *any* job for that client. Assuming you need that file retention for other jobs on the client, you can work around the problem by adding a separate client definition for use by the archive jobs, with the same Address pararmeter as the normal one. __Martin -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Possibility of parallelising encryption?
Craig Ringer wrote: Snip interesting crypto discussion... >> I'd use the hardware encryption (which presumably has no performance >> impact), that is an option on this autochanger, except they want $2500 >> for it... > > Probably because it has a custom ASIC for the crypto algorithm in use to > allow it to go fast enough. Well, I suspect the ASIC is already onboard - the $2500 seems to get you a couple of USB keys to enable it (HP MSL series libraries). > The trouble with this is that if your tape drive/changer dies, you > generally need another one with the same hardware crypto to restore. > This is a really, really ugly situation for disaster recovery. As long as you obtain another MSL library and use one of your original USB keys, you're back in business, but I take your point. Regards, Richard -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] upgrading bacula
> On Wed, 7 Apr 2010 11:36:15 -0400, John Drescher said: > > On Wed, Apr 7, 2010 at 11:20 AM, Bill Szkotnicki wrote: > > Thanks John, > > > > I ran update_mysql_tables and the table JobHisto is missing. > > Do I need to create it? > > > > Bill > > > > ERROR 1146 (42S02) at line 8: Table 'bacula.JobHisto' doesn't exist > > ERROR 1146 (42S02) at line 9: Table 'bacula.JobHisto' doesn't exist > > ERROR 1146 (42S02) at line 10: Table 'bacula.JobHisto' doesn't exist > > Update of Bacula MySQL tables succeeded. > > > > You may have needed to run the intermediate updates. There should be > several upgrade scripts. I am not sure. I have never done this big of > a jump. Yes, to jump more than one version you need to run the appropriate update_mysql_tables_..._to_... scripts from the updatedb directory in the source code. That does work, but it sometimes needs tweaking to pass the correct command line arguments. __Martin -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Problems with bwx-console
Hi all, I'm getting troubles opening the bwx console in windows (the bconsole works perfectly). When I open the bwx-console I can see that it's interacting, for example if the director is not connected I can get an error telling me that, but the graphic never shows. PS: it used to work very well, i just had this problem today (may be it's something related to the system, caused by a virus). Please help me resolve this if you have an idea how. Many thanks, Fadwa _ Votre messagerie et bien plus où que vous soyez. Passez à Windows Live Hotmail, c'est gratuit ! https://signup.live.com/signup.aspx?id=60969-- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Poor Performance
Thank you for all the replies The bottleneck is the network or at lest something in the network and yes I have an autoloader LTO4 If I use NC with a file of 10M I need 8 seconds to transfer If I use scp I need around 2 seconds The same file without compression On Wed, Apr 7, 2010 at 10:10 AM, John Drescher wrote: >> Yes I am using a tape but should not be the tape >> 07-Apr 00:36 angel-sd JobId 1443: Despooling elapsed time = 00:02:34, >> Transfer rate = 75.16 M bytes/second >> >> >> this is with a full backup >> >> FD Files Written: 2,878,988 >> SD Files Written: 2,878,988 >> FD Bytes Written: 248,635,057,283 (248.6 GB) >> SD Bytes Written: 249,102,272,590 (249.1 GB) >> Rate: 18745.1 KB/s >> >> On my mind I was hoping something around 30KB/s >> > You mean 30MB/s. > > I get 20 to 35MB/s for full backups with LTO2 from a raid5 array to a > LTO2 autochanger that is not on the same machine as the raid. > > It looks like you have an LTO3 changer by the 75MB/s since that is a > little low for LTO4 however you named the drive LTO4Driver so I assume > its LTO4. I would expect despools of 100MB/s to 120MB/s for LTO4. > > I would first look to optimize your filesystem performance as this > appears to be where the problem is. Is your spooling drive the same as > your source data drive? That would kill performance. > > John > -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] VirtualFull mysql query blocks other jobs for a long time
ebollengier wrote: > > > Graham Keeling wrote: >> >> On Wed, Apr 07, 2010 at 08:22:09AM -0700, ebollengier wrote: >>> I tweaked my test to compare both queries, and it shows no difference >>> with >>> and without base job part... If you want to test queries on your side >>> with >>> your data, you can download my tool (accurate-test.pl) on >>> http://bacula.git.sourceforge.net/git/gitweb.cgi?p=bacula/docs;a=tree;f=docs/techlogs;hb=HEAD >>> >>> If you can tweak my script to reproduce your problem, i would be able to >>> fix >>> things. >>> >>> http://old.nabble.com/file/p28166612/diff-with-without-basejob.png >> >> I'm currently running your script to generate the test database. I think >> that is going to take a long time, so I'll leave it overnight. >> >> > > This is your first problem, on my server (just a workstation), it takes > less > than 10s to add 200,000 records... > > In fact, it's 4seconds, here... JobId=15 files=2 Insert takes 4secs for 20 records JobId=16 files=5 Insert takes 11secs for 50 records JobId=20 files=20 Insert takes 43secs for 200 records -- View this message in context: http://old.nabble.com/VirtualFull-mysql-query-blocks-other-jobs-for-a-long-time-tp28149748p28167844.html Sent from the Bacula - Users mailing list archive at Nabble.com. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] VirtualFull mysql query blocks other jobs for a long time
Graham Keeling wrote:
>
> On Wed, Apr 07, 2010 at 08:22:09AM -0700, ebollengier wrote:
>> I tweaked my test to compare both queries, and it shows no difference
>> with
>> and without base job part... If you want to test queries on your side
>> with
>> your data, you can download my tool (accurate-test.pl) on
>> http://bacula.git.sourceforge.net/git/gitweb.cgi?p=bacula/docs;a=tree;f=docs/techlogs;hb=HEAD
>>
>> If you can tweak my script to reproduce your problem, i would be able to
>> fix
>> things.
>>
>> http://old.nabble.com/file/p28166612/diff-with-without-basejob.png
>
> I'm currently running your script to generate the test database. I think
> that is going to take a long time, so I'll leave it overnight.
>
>
This is your first problem, on my server (just a workstation), it takes less
than 10s to add 200,000 records...
>
>
> Meanwhile (unless I misunderstand), I think that your graph is labelled
> strangely.
>
> When I'm generating the file records, I get this:
>
> ...
> JobId=10 files=20
> Insert takes 445secs for 200 records
> ...
> JobId=13 files=2
> Insert takes 54secs for 20 records
> ...
>
>
> So, JobIds 10 + 13:
>22 files
> 220 records
>
> Your script says:
> do_bench("10,13", 220);
> This is obviously the records number.
>
> Your graph uses this bigger number and labels it 'Files'.
>
The accurate query on jobid=10,13 will work on 2,000,000 + 200,000 files
(this is
the argument), as each filenames are different, you can expect to retrieve
2,2M files.
It's quite possible that it miss a *10 mutiplication somewhere in debug
messages (I insert files 10 by 10)
Bye
--
View this message in context:
http://old.nabble.com/VirtualFull-mysql-query-blocks-other-jobs-for-a-long-time-tp28149748p28167751.html
Sent from the Bacula - Users mailing list archive at Nabble.com.
--
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] VirtualFull mysql query blocks other jobs for a long time
On Wed, Apr 07, 2010 at 08:22:09AM -0700, ebollengier wrote:
> I tweaked my test to compare both queries, and it shows no difference with
> and without base job part... If you want to test queries on your side with
> your data, you can download my tool (accurate-test.pl) on
> http://bacula.git.sourceforge.net/git/gitweb.cgi?p=bacula/docs;a=tree;f=docs/techlogs;hb=HEAD
>
> If you can tweak my script to reproduce your problem, i would be able to fix
> things.
>
> http://old.nabble.com/file/p28166612/diff-with-without-basejob.png
I'm currently running your script to generate the test database. I think
that is going to take a long time, so I'll leave it overnight.
Meanwhile (unless I misunderstand), I think that your graph is labelled
strangely.
When I'm generating the file records, I get this:
...
JobId=10 files=20
Insert takes 445secs for 200 records
...
JobId=13 files=2
Insert takes 54secs for 20 records
...
So, JobIds 10 + 13:
22 files
220 records
Your script says:
do_bench("10,13", 220);
This is obviously the records number.
Your graph uses this bigger number and labels it 'Files'.
--
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Idea/suggestion for dedicated disk-based sd
On Tue, Apr 6, 2010 at 5:19 PM, Robert LeBlanc wrote: > On Tue, Apr 6, 2010 at 12:37 AM, Craig Ringer > wrote: > > [snip] > >> >> Is this insane? Or a viable approach to tackling some of the >> complexities of faking tape backup on disk as Bacula currently tries to do? >> > > I love Bacula and have been working hard to promote it to people I > know. The biggest problem with bacula is it's disk management. We have > a DataDomain box that is getting horrible dedup rate and after looking > at the Bacula tape stream format, I can understand why. There is so > much extra data inserted into the stream that is very helpful for tape > drives that it makes deduping the data nearly impossible. > > I would love to see the stream simplified for disk based storage. > Another thing I'd like the option for is to be able to specify a block > size and start a file on the block boundry, you could use sparse files > to skip the space without taking it up. This would allow dedup > algorithms to really be able to compress Bacula data much better. It > would be awesome if the file stored in the Bacula stream looked > exactly like on the file systm so that if you do any tier 3 storage > with dedup and run your Bacula backups to the same storage, you get > free backups. > > Dedup is gaining a lot of traction, name your favorite vendor, or as > I'm doing look at lessfs. All of these would benefit hugely from a > smart SD that knows how to handle disk storage better and make Bacula > much more attractive. With the types of backups we are doing, we > should be getting 10x easy on our DataDomain, but we are lucky to get > 4x and I think that mostly comes from compression. > > Thanks, > > Robert LeBlanc > Life Sciences & Undergraduate Education Computer Support > Brigham Young University > So still thinking about this, is there any reason to not have a hierarchical file structure for disk based backup rather than a serialized stream? Here are my thought, any comments welcome to have a good discussion about this. SD_Base_Dir +- PoolA +- PoolB +- JobID1 +- JobID2 +- Clientinfo.bacula (Bacula serial file that holds information similar to block header) +- Original File Structure (File structure from client is maintained and repeated here, allows for browsing of files outside of bacula) +- ClientFileA +- ClientFileA.bacula (Bacula serial file that holds information similar to the unix file attribute package) +- ClientFileB +- ClientFileB.bacula +- ClientDirA +- ClientDirA.bacula Although it's great to reuse code, I think something like this would be very benifical to disk based backups. The would help increase dedup rates and some file systems like btrfs and ZFS may be able to take advantage of linked files (there has been some discussion on the btrfs list about things like this). This would also allow it to reside on any file system as all the ACL and information is being serialized in separate files which keeps unique data out of the blocks of possible duplicated data. I think we could even reuse a lot of the serialization code, so it would just differ in how it writes the stream of data. Please excuse me if I'm way off here, just trying to think outside of the box a little. Robert LeBlanc Life Sciences & Undergraduate Education Computer Support Brigham Young University -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] upgrading bacula
On Wed, Apr 7, 2010 at 11:20 AM, Bill Szkotnicki wrote: > Thanks John, > > I ran update_mysql_tables and the table JobHisto is missing. > Do I need to create it? > > Bill > > ERROR 1146 (42S02) at line 8: Table 'bacula.JobHisto' doesn't exist > ERROR 1146 (42S02) at line 9: Table 'bacula.JobHisto' doesn't exist > ERROR 1146 (42S02) at line 10: Table 'bacula.JobHisto' doesn't exist > Update of Bacula MySQL tables succeeded. > You may have needed to run the intermediate updates. There should be several upgrade scripts. I am not sure. I have never done this big of a jump. John -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] upgrading bacula
Thanks John, I ran update_mysql_tables and the table JobHisto is missing. Do I need to create it? Bill ERROR 1146 (42S02) at line 8: Table 'bacula.JobHisto' doesn't exist ERROR 1146 (42S02) at line 9: Table 'bacula.JobHisto' doesn't exist ERROR 1146 (42S02) at line 10: Table 'bacula.JobHisto' doesn't exist Update of Bacula MySQL tables succeeded. John Drescher wrote: I need to upgrade the bacula that I am running here on a fedora-linux server. Currently we run version 2.0.2 and I am attempting to move to 5.0.1 Questions: 1) Do I need to change the database? ( mysql 5.0.27 ) ( i.e. are the tables and indexes the same? ) There are several database upgrades since 2.0.2. You need to run the bacula upgrade scripts. Also I would upgrade your mysql. 5.0.27 is as at least as ancient as bacula-2.0.2. My distro has mysql-5.0.90 as the latest 5.0 release but there are also 5.1 to 5.4 releases and 5.5 betas.. 2) May I just replace the binaries with the newly compiled ones? You will probably have to update your director configs as well some options have changed in the last 2.5 years. 3) Will the new bacula communicate properly with the older clients on PCs ( mostly windows 2.4.4 ) Yes If anyone can help me with this I would appreciate it. John -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] VirtualFull mysql query blocks other jobs for a long time
ebollengier wrote: > > > > Graham Keeling wrote: >> >> On Tue, Apr 06, 2010 at 09:01:13AM -0700, ebollengier wrote: >>> Hello Graham, >> >> Hello, thanks for your reply. >> >>> Graham Keeling wrote: >>> > >>> > Is there anything I can do that would speed it up? >>> > Perhaps even more importantly, what was it doing for the 33 hour >>> period >>> > between Apr 4 18:56:02 and Apr 6 03:56:12? Was it just writing the >>> > bootstrap >>> > records? >>> > >>> > On the Path table, I have these: >>> >PRIMARY KEY(PathId), >>> >INDEX (Path(255)) >>> > >>> > On the File table, I have these: >>> >PRIMARY KEY(FileId), >>> >INDEX (JobId, PathId, FilenameId), >>> >INDEX (PathId), >>> >INDEX (FilenameId) >>> > >>> > On the Job table, I have these: >>> >PRIMARY KEY(JobId), >>> >INDEX (Name(128)), >>> >INDEX (ClientId) >>> > >>> >>> Did you run tests with the original schema? It looks like you removed >>> important index such as the one >>> on JobId and you added others, and i'm not sure that MySQL can play >>> properly >>> with composed indexes (it doesn't looks like) in this treatment. >> >> In future, to avoid generating this sort of complaint, I will not fiddle >> with >> the default bacula indexes. >> However, I think that the problem is not to do with this. >> I shall explain below... >> >>> I've made tests with 400,000,000 files and 40,000,000 filenames over 600 >>> jobs, and it doesn't run fast as PostgreSQL (factor 6 between both >>> engines), >>> but it should handle the job rather quickly. (on my pc, for 2M files, >>> it's >>> between 1min and 1min30 (15seconds with PostgreSQL)) >> >> >> Here is the command that mysql takes hours over, and hence the one that I >> am >> using to test. In all of my tests, the result is the same whether I am >> using >> the default bacula indexes or my additional ones. >> >> >> SELECT Path.Path, Filename.Name, Temp.FileIndex, Temp.JobId, LStat, MD5 >> FROM >>( SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId AS PathId, >> File.FilenameId AS FilenameId, LStat, MD5 >> FROM Job, File, >> ( SELECT MAX(JobTDate) AS JobTDate, PathId, FilenameId >> FROM >> ( SELECT JobTDate, PathId, FilenameId >> FROM File >> JOIN Job USING (JobId) >> WHERE File.JobId IN (22,23,31,34,42,48,52) >> UNION ALL >> SELECT JobTDate, PathId, FilenameId >> FROM BaseFiles >> JOIN File USING (FileId) >> JOIN Job ON(BaseJobId = Job.JobId) >> WHERE BaseFiles.JobId IN (22,23,31,34,42,48,52) >> ) >> AS tmp GROUP BY PathId, FilenameId >> ) AS T1 >> WHERE (Job.JobId IN >> ( SELECT DISTINCT BaseJobId >> FROM BaseFiles >> WHERE JobId IN (22,23,31,34,42,48,52) >> ) >>) >> OR Job.JobId IN (22,23,31,34,42,48,52) >> AND T1.JobTDate = Job.JobTDate >> AND Job.JobId = File.JobId >> AND T1.PathId = File.PathId >> AND T1.FilenameId = File.FilenameId >>) AS Temp >> JOIN Filename ON (Filename.FilenameId = Temp.FilenameId) >> JOIN Path ON (Path.PathId = Temp.PathId) >> WHERE FileIndex > 0 >> ORDER BY Temp.JobId, FileIndex ASC; >> >> >> >> Since I am not using base jobs, I did the experiment of removing the >> lines >> from the query to do with base jobs. When I did that, the query changed >> from >> taking hours, to taking under a second. >> >> I then tracked it down to this specific part (i.e, remove this, and get a >> truly magnificent speed-up): >> >>(Job.JobId IN >> ( SELECT DISTINCT BaseJobId >> FROM BaseFiles >> WHERE JobId IN (22,23,31,34,42,48,52) >> ) >>) >> OR >> >> I think that mysql is repeatedly calling this section for each row that >> the SELECT returns. >> >> Now the question is what to do about it? >> > > It's very interesting, in my tests (with and without basejobs), it changes > nothing... But i will > check it again. (MySQL have a query cache, and if you run two times the > same thing, the result > is very fast) > > As we are using JobTDate to find the last record of each file, we use it > to get JobId back from the > Job table (yes, ugly, but MySQL doesn't have DISTINCT ON()), and if two > jobs have the same JobTDate, we are in trouble. So, i limit the selection > to this JobId list. > > Hope that you found the problem, it's rather easy to compute this list > once, and > use it after. > > Bye > I tweaked my test to compare both queries, and it shows no difference with and without base job part... If you want to test queries on your side with your data, you can download my tool (accurate-test.pl) on http://bacula.git.sourceforge.net/git/git
Re: [Bacula-users] client rejected Hello command
On Tue, Apr 06, 2010 at 05:52:54PM -0400, Roland Roberts wrote: > When the client is run in the foreground in verbose mode, I get this from the > client: > > archos-fd: cram-md5.c:73-0 send: auth cram-md5 > <1018903137.1269886...@archos-fd> ssl=0 > archos-fd: cram-md5.c:152-0 sending resp to challenge: > a9/LnX1KKQ/Gj6+qd++t1B that is strange. Looking at the code, if you get this far, one of the 3 things should happen: a) timeout or socket error, in which case it would have printed "Receive chanllenge response failed. ERR=%s", or b) authorization succeeds (it receives "1000 OK auth") in which case nothing is printed and cram_md5_respond returns true, or c) authorization fails, and "Received bad response: %s" is printed As nothing is printed, one could only assume the authorization succeeded, but things stopped somewhere afterwards. Can you raise debug level (say, -d200) on both the director and FD (and maybe even on SD, just in case) and send output of that ? -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Multiple drives in changer
> Date: Tue, 6 Apr 2010 08:52:24 -0600 > From: Robert LeBlanc > > On Tue, Apr 6, 2010 at 6:13 AM, Matija Nalis > >> > wrote: >> > On Fri, Apr 02, 2010 at 10:36:59AM -0600, Robert LeBlanc wrote: >>> > > On Fri, Apr 2, 2010 at 2:44 AM, Matija Nalis >>> > > >>> > > > > > I think you need to set > > > Prefer Mounted Volumes = no >>> > > >>> > > I guess this is where we need clarification about what is an available >>> > > drive. I took this to mean a drive that has no tape is more available, >> > and >>> > > then a drive that does already have a tape mounted would be next in >>> > > availability. >> > >> > Hm, it looks to me that any drive which is not doing R/W operation >> > (no matter if there is a tape in drive or not) is counted as available. >> > I could be wrong on that, though. >> > >> > Anyway, the safest way to know is to test it and let the others know >> > how it goes :) >> > >From my observations of a few tests, this indeed seems to be the case. If > the drive is not being R/W to/from, it is considered available. > > >>> > > It seems that as long as no job is writing to that tape, then >>> > > the drive is available. I do want this setting to yes and not no, >> > however, I >>> > > would like to minimize tape changes, but take advantage of the multiple >>> > > drives. >> > >> > From what I see in practice, "Prefer Mounted Volumes = yes" would >> > make sure there is only one drive in each pool that does the writing. >> > >> > For example, I have pool of 4 drives and I start 10 jobs at the same >> > time, all using the same pool. I have an concurrency of >10 and >> > spooling enabled, so all the jobs run at once and start spooling to >> > disk -- but when they need to despool, one drive will grab a free >> > tape from Scratch, and all the jobs will wait for their in turn to >> > write to one tape in one drive, leaving 3 drives idle all the time. >> > Only when that tape is full, another one is loaded, and the process >> > repeats. >> > >> > I think same happens when I disable spooling, but then the 4 jobs all >> > interleave writes -- but still all of them will write on one tape in >> > one drive only. >> > >> > If you set "Prefer Mounted Volumes = no", then all 4 drives get >> > loaded with 4 fresh tapes (or just use them if right tapes are >> > already in right drives -- I guess, I have autochanger) and each >> > tape gets written to at the same time, maximizing drive (and thus, >> > the tape) usage. >> > >> > But "no" setting can (or at least could in the past) lead to >> > deadlocks sometimes (if you have autochanger), when no new jobs will >> > get serviced because drive A will wait for tape 2 that is currently >> > in drive B, and at the same time drive B will wait for tape 1 which >> > is currently in drive A. Then the manual intervention (umount/mount) >> > is needed (which is a big problem for us as we have lots of jobs/tapes). >> > >> > The (recommended) alternative is to go semi-manual way -- dedicate >> > special pool for each drive, and go with "Prefer Mounted Volumes = >> > yes" Then one can (and indeed, must) specify manually which jobs will >> > go in which pools (and hence, in which drives) and can optimize it >> > for maximum parallelism without deadlocks -- but it requires more >> > planing and is problematic if your backups are more dynamic and hard >> > to predict, and you have to redesign when you add/upgrade/remove >> > drives, and your pools might become somewhat harder to manage. >> > > This is exactly my experience, and my goal is not to use multiple drives in > the same pool at the same time, it's to use drives for different pools at > the same time one drive per pool. We are looking to bring up a lot more > storage in the future and will probably adopt the mentality of multiple > daily, weekly, monthly pools and split them up based on the number of drives > we want to run concurrently. I think that is the best way to go with Bacula > for what we want to do. > > Thanks, > > Robert LeBlanc > Life Sciences & Undergraduate Education Computer Support > Brigham Young University Your other option is to try out the "Maximum Concurrent Jobs" in the device section of your storage daemon's config. That's working well for me. One word of caution though: since the way it allocates jobs to drives is not the same as prefer mounted volumes you should carefully consider all the different concurrent parameters and how they relate to each other. Here's an example of how I first tried it (not very good)... dir.conf: Director: Maximum concurrent jobs = 20 Jobdefs: Maximum concurrent jobs = 10 sd.conf: Storage: Maximum Concurrent jobs = 20 Device1: Maximum concurrent jobs = 2 Device2: Maximum concurrent jobs = 2 The way prefer mounted volumes worked, it round robin alternated the drive assignments for each job as it started them. For This new directive it does the opposite, assigns them jobs to the first until it hits the max, then assigns the rest to the next dr
Re: [Bacula-users] VirtualFull mysql query blocks other jobs for a long time
Since: a) I am not using Base jobs b) I am currently stuck with using MySQL c) There is not a 'proper' fix yet I am going to use the attached patch as a temporary solution to the problem. Index: src/cats/sql_cmds.c === RCS file: /cvs/netpilot/GPL/bacula-5.0.1/WORK/src/cats/sql_cmds.c,v retrieving revision 1.2 diff -u -r1.2 sql_cmds.c --- src/cats/sql_cmds.c 26 Mar 2010 10:36:03 - 1.2 +++ src/cats/sql_cmds.c 7 Apr 2010 13:56:10 - @@ -481,9 +481,7 @@ "WHERE BaseFiles.JobId IN (%s) "/* Use Max(JobTDate) to find */ ") AS tmp GROUP BY PathId, FilenameId " /* the latest file version */ ") AS T1 " -"WHERE (Job.JobId IN ( " /* Security, we force JobId to be valid */ -"SELECT DISTINCT BaseJobId FROM BaseFiles WHERE JobId IN (%s)) " -"OR Job.JobId IN (%s)) " +"WHERE Job.JobId IN (%s) " "AND T1.JobTDate = Job.JobTDate " /* Join on JobTDate to get the orginal */ "AND Job.JobId = File.JobId " /* Job/File record */ "AND T1.PathId = File.PathId " Index: src/cats/sql_get.c === RCS file: /cvs/netpilot/GPL/bacula-5.0.1/WORK/src/cats/sql_get.c,v retrieving revision 1.1 diff -u -r1.1 sql_get.c --- src/cats/sql_get.c 26 Mar 2010 10:20:41 - 1.1 +++ src/cats/sql_get.c 7 Apr 2010 13:56:10 - @@ -,7 +,7 @@ #ifdef new_db_get_file_list POOL_MEM buf2(PM_MESSAGE); Mmsg(buf2, select_recent_version_with_basejob[db_type], -jobids, jobids, jobids, jobids); +jobids, jobids, jobids); Mmsg(buf, "SELECT Path.Path, Filename.Name, Temp.FileIndex, Temp.JobId, LStat, MD5 " "FROM ( %s ) AS Temp " -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] upgrading bacula
> I need to upgrade the bacula that I am running here on a fedora-linux > server. > Currently we run version 2.0.2 and I am attempting to move to 5.0.1 > Questions: > 1) Do I need to change the database? ( mysql 5.0.27 ) ( i.e. are the > tables and indexes the same? ) There are several database upgrades since 2.0.2. You need to run the bacula upgrade scripts. Also I would upgrade your mysql. 5.0.27 is as at least as ancient as bacula-2.0.2. My distro has mysql-5.0.90 as the latest 5.0 release but there are also 5.1 to 5.4 releases and 5.5 betas.. > 2) May I just replace the binaries with the newly compiled ones? You will probably have to update your director configs as well some options have changed in the last 2.5 years. > 3) Will the new bacula communicate properly with the older clients on > PCs ( mostly windows 2.4.4 ) > Yes > > If anyone can help me with this I would appreciate it. > John -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Automatic restores BACULA
On 04/07/10 09:54, James Harper wrote: >> On 04/07/10 06:27, bitbyte wrote: >>> What do you think a restore is?Think you need to go again to the >>> elementary lessons? Periodical restores checks the status of the >>> files that are being backed up,so that we can be sure that they are >>> backed up correctly without errors? >>> >>> This just doesn't mean that we do baremetal or something like that >>> automated :):) >> >> This sounds as though you should look into the Verify feature and see > if >> it does what you need. >> > > A verify is not a substitute for a restore test. A verify tells you that > the data is on the media, it doesn't tell you that you can get it back. > > A regular full (or partial) restore test should be a part of any good > backup regime. Oh, sure. But as best I can understand the OP, it seems to me that verify is what he's looking for. However, I could easily be misunderstanding. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Poor Performance
> Yes I am using a tape but should not be the tape > 07-Apr 00:36 angel-sd JobId 1443: Despooling elapsed time = 00:02:34, > Transfer rate = 75.16 M bytes/second > > > this is with a full backup > > FD Files Written: 2,878,988 > SD Files Written: 2,878,988 > FD Bytes Written: 248,635,057,283 (248.6 GB) > SD Bytes Written: 249,102,272,590 (249.1 GB) > Rate: 18745.1 KB/s > > On my mind I was hoping something around 30KB/s > You mean 30MB/s. I get 20 to 35MB/s for full backups with LTO2 from a raid5 array to a LTO2 autochanger that is not on the same machine as the raid. It looks like you have an LTO3 changer by the 75MB/s since that is a little low for LTO4 however you named the drive LTO4Driver so I assume its LTO4. I would expect despools of 100MB/s to 120MB/s for LTO4. I would first look to optimize your filesystem performance as this appears to be where the problem is. Is your spooling drive the same as your source data drive? That would kill performance. John -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] VirtualFull mysql query blocks other jobs for a long time
On Wed, Apr 07, 2010 at 02:51:42PM +0100, Martin Simmons wrote: > Does it still run quickly if keep that Job.JobId IN clause but use the numbers > returned by > > SELECT DISTINCT BaseJobId > FROM BaseFiles > WHERE JobId IN (22,23,31,34,42,48,52) > > in place of the the nested select? In my case, there are no numbers returned. But, I assume that you are suggesting something like the command below. For me, it completes in under a second. SELECT Path.Path, Filename.Name, Temp.FileIndex, Temp.JobId, LStat, MD5 FROM ( SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId AS PathId, File.FilenameId AS FilenameId, LStat, MD5 FROM Job, File, ( SELECT MAX(JobTDate) AS JobTDate, PathId, FilenameId FROM ( SELECT JobTDate, PathId, FilenameId FROM File JOIN Job USING (JobId) WHERE File.JobId IN (22,23,31,34,42,48,52) UNION ALL SELECT JobTDate, PathId, FilenameId FROM BaseFiles JOIN File USING (FileId) JOIN Job ON(BaseJobId = Job.JobId) WHERE BaseFiles.JobId IN (22,23,31,34,42,48,52) ) AS tmp GROUP BY PathId, FilenameId ) AS T1 WHERE ( Job.JobId IN (22,23,31,34,42,48,52) OR Job.JobId IN (22,23,31,34,42,48,52) ) AND T1.JobTDate = Job.JobTDate AND Job.JobId = File.JobId AND T1.PathId = File.PathId AND T1.FilenameId = File.FilenameId ) AS Temp JOIN Filename ON (Filename.FilenameId = Temp.FilenameId) JOIN Path ON (Path.PathId = Temp.PathId) WHERE FileIndex > 0 ORDER BY Temp.JobId, FileIndex ASC -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Automatic restores BACULA
> On 04/07/10 06:27, bitbyte wrote: > > What do you think a restore is?Think you need to go again to the > > elementary lessons? Periodical restores checks the status of the > > files that are being backed up,so that we can be sure that they are > > backed up correctly without errors? > > > > This just doesn't mean that we do baremetal or something like that > > automated :):) > > This sounds as though you should look into the Verify feature and see if > it does what you need. > A verify is not a substitute for a restore test. A verify tells you that the data is on the media, it doesn't tell you that you can get it back. A regular full (or partial) restore test should be a part of any good backup regime. James -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] upgrading bacula
Hi, I need to upgrade the bacula that I am running here on a fedora-linux server. Currently we run version 2.0.2 and I am attempting to move to 5.0.1 Questions: 1) Do I need to change the database? ( mysql 5.0.27 ) ( i.e. are the tables and indexes the same? ) 2) May I just replace the binaries with the newly compiled ones? 3) Will the new bacula communicate properly with the older clients on PCs ( mostly windows 2.4.4 ) If anyone can help me with this I would appreciate it. Thanks, Bill -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula volume selection -- InChanger
> From: Tom Eastman > > Hey guys, > > I'm still having trouble controlling which volume bacula chooses to > write to when it has plenty of volumes set to 'append'. I *need* bacula > to only select a volume to append to that has InChanger=1. I kinda > thought that it would do this automatically, at least from what I've > read, but it doesn't seem to be doing so. > > I've just upgraded to 5.0.1. > > How do I force bacula to only select a tape from the list of tapes that > are InChanger=1? > > Thanks! > > Tom > Tom, In your bacula-dir.conf file, go to the Storage section. Do you have a line that says "Autochanger = yes"? If not, add it. Bob -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] VirtualFull mysql query blocks other jobs for a long time
> On Wed, 7 Apr 2010 12:40:24 +0100, Graham Keeling said: > > On Tue, Apr 06, 2010 at 09:01:13AM -0700, ebollengier wrote: > > Hello Graham, > > Hello, thanks for your reply. > > > Graham Keeling wrote: > > > > > > Hello, > > > I'm using bacula-5.0.1. > > > I have a 2.33GHz CPU with 2G of RAM. > > > I am using MySQL. > > > I had a VirtualFull scheduled for my client. > > > > > > My log says the following: > > > > > > Apr 4 18:56:02 Start Virtual Backup JobId 56, > > > Job=Linux:cvs.2010-04-04_18.56.00_03 > > > Apr 4 18:56:02 This Job is not an Accurate backup so is not equivalent > > > to a Full backup. > > > Apr 6 03:56:12 Bootstrap records written to > > > /var/lib/bacula/backup.dev.equiinet.com-.restore.1.bsr > > > Apr 6 10:23:20 Ready to read from volume "backup-0002" on device "Tower > > > 2.0" (/write/mnt/Tower 2). > > > Apr 6 10:23:20 Labeled new Volume "backup-0046" on device "Tower 2.1" > > > (/write/mnt/Tower 2). > > > ...and now, backup-0046 is being written to. > > > > > > At about Apr 6 10:00, I was logged on to the director, and attempting to > > > run > > > a 'status, director' command on bconsole. > > > bconsole printed a few bits of information, then paused. Presumably > > > because JobId 56 had locked the database. After 10:23:20, the command > > > printed > > > the rest of its information and gave me the prompt back. > > > > > > At about Apr 6 10:00, I logged into mysql, and ran 'show full > > > processlist;', > > > which gave the following. > > > > > > | Id | User | Host | db | Command | Time | State | Info > > > | 3032 | root | localhost | bacula | Query | 22464 | Locked| UPDATE > > > Job SE > > > T JobStatus='R',Level='F',StartTime='2010-04-06 > > > 03:56:14',ClientId=2,JobTDate=12 > > > 70522574,PoolId=7,FileSetId=2 WHERE JobId=56 > > > > > > | 3033 | root | localhost | bacula | Query | 22464 | executing | SELECT > > > Path.P > > > ath, Filename.Name, Temp.FileIndex, Temp.JobId, LStat, MD5 FROM ( SELECT > > > FileId, > > > Job.JobId AS JobId, FileIndex, File.PathId AS PathId, File.FilenameId AS > > > Filena > > > meId, LStat, MD5 FROM Job, File, ( SELECT MAX(JobTDate) AS JobTDate, > > > PathId, Fil > > > enameId FROM ( SELECT JobTDate, PathId, FilenameId FROM File JOIN Job > > > USING (Job > > > Id) WHERE File.JobId IN (22,23,31,34,42,48,52) UNION ALL SELECT JobTDate, > > > PathId > > > , FilenameId FROM BaseFiles JOIN File USING (FileId) JOIN Job ON > > > (BaseJobId > > > = Job.JobId) WHERE BaseFiles.JobId IN (22,23,31,34,42,48,52) ) AS tmp > > > GROUP BY P > > > athId, FilenameId ) AS T1 WHERE (Job.JobId IN ( SELECT DISTINCT BaseJobId > > > FROM B > > > aseFiles WHERE JobId IN (22,23,31,34,42,48,52)) OR Job.JobId IN > > > (22,23,31,34,42, > > > 48,52)) AND T1.JobTDate = Job.JobTDate AND Job.JobId = File.JobId AND > > > T1.PathId > > > = File.PathId AND T1.FilenameId = File.FilenameId ) AS Temp JOIN Filename > > > ON (Fi > > > lename.FilenameId = Temp.FilenameId) JOIN Path ON (Path.PathId = > > > Temp.PathId) WH > > > ERE FileIndex > 0 ORDER BY Temp.JobId, FileIndex ASC | > > > > > > 'Time 22464' is equivalent to about 6 hours, so I presume that after > > > 'Bootstrap > > > records written', bacula has been waiting for this sql command, during > > > which > > > time it could not do anything else. Backups of other clients were failing > > > because I had Max Wait Time set to something less than 6 hours. > > > > > > The following is a listing of the number of files that each previous > > > backup > > > said it had backed up, which doesn't seem extraordinarily large to my > > > mind. > > > > > > Full: 810,400 > > > Incr: 52,487 > > > Incr: 52,485 > > > Incr: 52,485 > > > Incr: 332,649 > > > Incr: 52,492 > > > Incr: 52,593 > > > Incr: 332,677 > > > Incr: 52,497 > > > Incr: 52,497 > > > Incr: 52,497 > > > > > > > > > So, my questions are: > > > > > > Does it really need to take so long? > > > Is it expected that this Job blocks everything else for 6 hours? > > > > > > > I think that this part of the code doesn't use a dedicated connection, it > > would be nice to > > improve this area. (other part of the code are using a specific connection > > to do this work) > > > > > > Graham Keeling wrote: > > > > > > Is there anything I can do that would speed it up? > > > Perhaps even more importantly, what was it doing for the 33 hour period > > > between Apr 4 18:56:02 and Apr 6 03:56:12? Was it just writing the > > > bootstrap > > > records? > > > > > > On the Path table, I have these: > > >PRIMARY KEY(PathId), > > >INDEX (Path(255)) > > > > > > On the File table, I have these: > > >PRIMARY KEY(FileId), > > >INDEX (JobId, PathId, FilenameId), > > >INDEX (PathId), > > >INDEX (FilenameId) > > > > > > On the Job table, I have these: > > >PRIMARY KEY(JobId), > > >INDEX (Name(128)), > > >INDEX (ClientId) > > > > > > > Did you run tests with the origi
Re: [Bacula-users] Poor Performance
On Wed, Apr 07, 2010 at 08:41:41AM -0400, Il Neofita wrote: > Hi > I have very poor performance when the agent send the spool to the bacula > server First the obvious question: are the server or client doing anything else while the backup is running? > I currently use a RAID5 SAS 15K, 1GB Ethernet > hdparm -t /dev/sda2 is that hdparm run on server or client ? > /dev/sda2: > Timing buffered disk reads: 692 MB in 3.00 seconds = 230.41 MB/sec that is raw buffered linear burst read speed... seeks can easily slow it down for two orders of magnitude or more -- and filesystems do require seeks. And if there are going to be lots of seeks or less depends on many factors (like what FS are you using, the amount of file fragmentation, are we talking about one big file or many many small files etc). > Any idea what I can check? 1) speed of writing on SD (server) side sync ; echo 3 > /proc/sys/vm/drop_caches time (dd if=/dev/zero of=your_spool_file bs=1k count=100; sync) # fix location of your_spool_file, and size of spool 100*1k # all examples assume that your dataset is at least 1GB large # you should probably use spool size at least 4 times the amount # of RAM to get accurate results. 2) speed of reading on FD (client) side sync ; echo 3 > /proc/sys/vm/drop_caches time tar zc / | dd of=/dev/zero bs=1k count=100 # omit 'z' if you're not doing compression, change '/' to match your fileset # note that if you're using encryption it will slow down things considerably 3) network speed between FD and SD # on server nc -l -p 12345 > /dev/null # on client time (dd if=/dev/zero bs=1k count=100 | nc -w1 server_ip 12345) # server_ip is your server IP address 4) combination of the above, should be similar to the bacula speed # on server sync ; echo 3 > /proc/sys/vm/drop_caches nc -l -p 12345 > /dev/zero # on client sync ; echo 3 > /proc/sys/vm/drop_caches time (tar zc / | dd of=/dev/zero bs=1k count=100 | nc -w1 server_ip 12345 ; sync) those should get you some idea about what part is bottleneck... -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Fatal error: backup.c:892 Network send error to SD. ERR=Connection reset by peer
Hi All, My Backup is failing on a client. The client has only one Fileset and the size is 400GB. The error is as follows Messages: 06-Apr 12:16 server-sd JobId 299: Spooling data again ... 06-Apr 12:38 server-sd JobId 299: User specified spool size reached. 06-Apr 12:38 server-sd JobId 299: Writing spooled data to Volume. Despooling 12,422,998,992 bytes ... 06-Apr 12:43 server-sd JobId 299: Despooling elapsed time = 00:04:50, Transfer rate = 42.83 M bytes/second 06-Apr 12:43 server-sd JobId 299: Spooling data again ... 06-Apr 12:54 client-fd JobId 299: Fatal error: backup.c:892 Network send error to SD. ERR=Connection reset by peer Volume Session Time: 1270457469 Last Volume Bytes: 216,986,112,000 (216.9 GB) Non-fatal FD errors:0 SD Errors: 0 FD termination status: Error SD termination status: Error Termination:*** Backup Error *** Is it possible to tell me how to enable more debug on client and storage so that i can find more clues to this issue. Many thanks, Prashant Ramhit -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Poor Performance
Yes I am using a tape but should not be the tape 07-Apr 00:36 angel-sd JobId 1443: Despooling elapsed time = 00:02:34, Transfer rate = 75.16 M bytes/second this is with a full backup FD Files Written: 2,878,988 SD Files Written: 2,878,988 FD Bytes Written: 248,635,057,283 (248.6 GB) SD Bytes Written: 249,102,272,590 (249.1 GB) Rate: 18745.1 KB/s On my mind I was hoping something around 30KB/s On Wed, Apr 7, 2010 at 8:54 AM, Simone Martina wrote: >> -- Messaggio originale --- >> Storage: "LTO4Driver" (From Job resource) >> [...] >> Rate: 6816.5 KB/s > Aren't you using a TAPE to store your backup? > IMHO, probably the speed is bottlenecked by the TAPE > trasmission rate. > > Simone > > -- > Chi ride ultimo ha fatto un backup. > -- c...@cynic.wimsey.bc.ca (Curt Sampson) > > > -- > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] PATCH: add AES-ECB support to Bacula
On 04/07/10 06:52, Craig Ringer wrote: > Bacula currently only uses the AES CBC cypher mode. This cypher can't be > effectively parallelized because block n+1 depends on block n. > > The AES ECB mode was developed to address that limitation. Support for > it is built into OpenSSL 0.9.8 . This trivial patch adds appropriate > entries in the public crypto.h enum for cyphers so that the AES ECB mode > may be requested by the fd, and translates this into OpenSSL cypher > settings in crypto.c . > > OpenSSL doesn't automatically do ECB encryption in parallel. However, > unlike the currently used CBC mode, it's possible to do. > > Right now, this patch gains you little or nothing. It also costs little > or nothing, and probably shouldn't actually be applied at present. I'm > putting it up here to keep it and the notes about crypto in the archives > in case I don't have any luck following it up with an actually parallel > implementation and others are looking into it later. > > The next step is to try to spawn worker threads to encrypt chunks in > parallel. Hopefully this will be possible with OpenSSL... Craig, you should really think about joining the -devel list. :) -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Possibility of parallelising encryption?
On 04/07/10 05:09, Craig Ringer wrote: > Richard Scobie wrote: >> Would it be possible to optimise this task by perhaps reading data in >> "chunks", which in turn can be encrypted by a core each, before being >> recombined and written out to tape? > > Bacula uses OpenSSL for crypto support. It doesn't seem to support any > other crypto libraries like NSS or GnuTLS. FWIW, memory says we tried using GnuTLS once. It turns out that for this purpose it is (or was) horribly broken. In fact, at the time, I seem to recall the mere presence of Red Hat's GnuTLS lib broke Bacula altogether. > Some hardware, like the Via C7 series of CPUs, have built-in AES crypto > hardware (PadLock) that on a single thread can do *insane* encryption > rates. On the older C3 series CPUs I've had no problems saturating a > 100MBit line with encrypted ssh data, despite the gutless 400MHz C3 CPU. > > Intel has introduced similar instructions on their Xeon 5600 series: > > http://software.intel.com/en-us/articles/boosting-openssl-aes-encryption-with-intel-ipp/ > > It'd be lovely to be able to use the IPP libraries in Bacula (and many > other things) for parallel crypto and many other parallel tasks, as > they're excellent even without special hardware. Unfortunately they're > rather GPL-incompatible and are only "free" for non-commercial use. It would indeed be very nice to be able to use that kind of hardware crypto support without having to jump through licensing hoops. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Poor Performance
> I have very poor performance when the agent send the spool to the bacula
> server
>
> I currently use a RAID5 SAS 15K, 1GB Ethernet
>
>
> hdparm -t /dev/sda2
>
>
> /dev/sda2:
> Timing buffered disk reads: 692 MB in 3.00 seconds = 230.41 MB/sec
>
>
> Any idea what I can check?
>
> 07-Apr 00:08 angel-dir JobId 1443: Start Backup JobId 1443,
> Job=fraser.2010-04-07_00.05.01_01
> 07-Apr 00:08 angel-dir JobId 1443: Using Device "LTODriver"
> 07-Apr 00:09 angel-sd JobId 1443: Volume "DMS002L4" previously written,
> moving to end of data.
> 07-Apr 00:09 angel-sd JobId 1443: Ready to append to end of Volume
> "DMS002L4" at file=237.
> 07-Apr 00:09 angel-sd JobId 1443: Spooling data ...
> 07-Apr 00:33 angel-sd JobId 1443: Job write elapsed time = 00:23:49,
> Transfer rate = 8.092 M bytes/second
> 07-Apr 00:33 angel-sd JobId 1443: Committing spooled data to Volume
> "DMS002L4". Despooling 11,574,810,050 bytes ...
> 07-Apr 00:36 angel-sd JobId 1443: Despooling elapsed time = 00:02:34,
> Transfer rate = 75.16 M bytes/second
> 07-Apr 00:36 angel-sd JobId 1443: Sending spooled attrs to the Director.
> Despooling 5,369,625 bytes ...
> 07-Apr 00:36 angel-dir JobId 1443: Bacula angel-dir 3.0.1 (30Apr09):
> 07-Apr-2010 00:36:47
> Build OS: i686-pc-linux-gnu redhat Enterprise release
> JobId: 1443
> Job: fraser.2010-04-07_00.05.01_01
> Backup Level: Incremental, since=2010-04-06 00:11:20
> Client: "fraser" 3.0.1 (30Apr09)
> i686-pc-linux-gnu,redhat,Enterprise release
> FileSet: "Fraser" 2010-01-13 08:12:58
> Pool: "IncrPool" (From Job IncPool override)
> Catalog: "MyCatalog" (From Client resource)
> Storage: "LTO4Driver" (From Job resource)
> Scheduled time: 07-Apr-2010 00:05:01
> Start time: 07-Apr-2010 00:08:31
> End time: 07-Apr-2010 00:36:47
> Elapsed time: 28 mins 16 secs
> Priority: 10
> FD Files Written: 17,598
> SD Files Written: 17,598
> FD Bytes Written: 11,560,818,747 (11.56 GB)
> SD Bytes Written: 11,563,514,692 (11.56 GB)
> Rate: 6816.5 KB/s
>
> FileSet {
> Name = "Fraser"
> Include {
> Options {
> signature = MD5
> }
> File = /etc
> File = /var/spool/mail
> File= /export/home0
> File= /export/home1
> File= /export/home2
> File= /export/home3
> File = /export/netinst
> File = /usr/local
> File = /local
> }
> Exclude {
> File = /sys
> File = /boot
> File = /dev
> File = /proc
> File = /tmp
> File = /.journal
> File = /.fsck
> }
> }
>
>
> Job {
> JobDefs = "DMSDefault"
> Name = "fraser"
> Enabled= yes
> FileSet = "Fraser"
> Level = Full
> Client=fraser
> Write Bootstrap = "/local/bacula/bin/working/Fraser.bsr"
> # Full Backup Pool = FullPool
> # Incremental Backup Pool = IncrPool
> # Differential Backup Pool = DiffPool
> Schedule = "DMS"
> SpoolData=yes
> }
>
This is not poor performance. You are doing an incremental backup. In
this case bacula will spend a large chunk of its time searching for
what files to backup and not actually backing up data.
John
--
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Poor Performance
> -- Messaggio originale --- > Storage:"LTO4Driver" (From Job resource) > [...] > Rate: 6816.5 KB/s Aren't you using a TAPE to store your backup? IMHO, probably the speed is bottlenecked by the TAPE trasmission rate. Simone -- Chi ride ultimo ha fatto un backup. -- c...@cynic.wimsey.bc.ca (Curt Sampson) -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] client rejected Hello command
> On Tue, 06 Apr 2010 17:52:54 -0400, Roland Roberts said: > > 2. Use the console to try to connect to the client (which, as I > mentioned, is on the same host as the server): > > *status client > The defined Client resources are: >1: archos-fd >2: aristarchus-fd >3: hipparchus-fd >4: tycho-fd >5: copernicus-fd > Select Client (File daemon) resource (1-5): 1 > Connecting to Client archos-fd at archos.rlent.pnet:9102 > Failed to connect to Client archos-fd. > > You have messages. > * > 29-Mar 14:10 archos-dir:*Console*.2010-03-29_14.08.08 Fatal error: File > daemon on "archos.rlent.pnet:9102" rejected Hello command > > When the client is run in the foreground in verbose mode, I get this from the > client: > > archos-fd: cram-md5.c:73-0 send: auth cram-md5 > <1018903137.1269886...@archos-fd> ssl=0 > archos-fd: cram-md5.c:152-0 sending resp to challenge: > a9/LnX1KKQ/Gj6+qd++t1B Is that all it printed? What debug level are you running at? I suggest running both fd and dir at debug level 120. __Martin -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] PATCH: add AES-ECB support to Bacula
On 7/04/2010 8:42 PM, Matija Nalis wrote: > On Wed, Apr 07, 2010 at 06:52:40PM +0800, Craig Ringer wrote: >> Bacula currently only uses the AES CBC cypher mode. This cypher can't be >> effectively parallelized because block n+1 depends on block n. >> >> The AES ECB mode was developed to address that limitation. Support for > > Actually, ECB was "developed" as it is one of the building blocks for > CBC (and other schemes). It is not secure at all by itself. Yeah. I got confused at some point while digging into all this. Thanks for following up and explicitly pointing it out, as it needs to be on record with the original post. At some point I'd got ECB and CTR mode muddled. This is why I should probably stick to something not even remotely touching on security, or at least do it when I'm reasonably awake ;-) > I really think doing pure ECB is a veeery bad way to go, as it lulls > the user in false sense of security without actually providing security. Yep. Not as bad as the recent "AES encrypted" USB flash disks that just used the user's password as a key to AES-128 encrypt/decrypt a session "key" that they merrily used to xor all the data written to / read from disk. Still inscure, though, due to known-plaintext attacks. > I know just enough crypto to know that without knowing way much > more I'm likely to make terrible mistakes (and we all still remember > Debian "fixing" OpenSSL security fiasco, don't we?). Raw ECB sounds > just like one of those mistakes. You're quite right. -- Craig Ringer -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Automatic restores BACULA
On 04/07/10 06:27, bitbyte wrote: > What do you think a restore is?Think you need to go again to the > elementary lessons? Periodical restores checks the status of the > files that are being backed up,so that we can be sure that they are > backed up correctly without errors? > > This just doesn't mean that we do baremetal or something like that > automated :):) This sounds as though you should look into the Verify feature and see if it does what you need. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] PATCH: add AES-ECB support to Bacula
On Wed, Apr 07, 2010 at 06:52:40PM +0800, Craig Ringer wrote: > Bacula currently only uses the AES CBC cypher mode. This cypher can't be > effectively parallelized because block n+1 depends on block n. > > The AES ECB mode was developed to address that limitation. Support for Actually, ECB was "developed" as it is one of the building blocks for CBC (and other schemes). It is not secure at all by itself. You may however use it to build other schemes which allow for (some/much) parallelisation and/or precomputing (like OFB, CTR etc), and which are way more secure then ECB itself (but still worse than CBC probably) > Right now, this patch gains you little or nothing. It also costs little > or nothing, and probably shouldn't actually be applied at present. I'm > putting it up here to keep it and the notes about crypto in the archives > in case I don't have any luck following it up with an actually parallel > implementation and others are looking into it later. > > The next step is to try to spawn worker threads to encrypt chunks in > parallel. Hopefully this will be possible with OpenSSL... I really think doing pure ECB is a veeery bad way to go, as it lulls the user in false sense of security without actually providing security. For examples, see nice pictures at http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Electronic_codebook_.28ECB.29 Sometimes you can even tell by the plain eye view of ECB-encrypted picture what the original picture was (See wikipedia)! Not to mention many various crypto attacks against its use in our contexts. What I would actually propose is filter support. Then a FD could pass a data stream via pipe to the standard external filter program (which reads from stdin and writes to stdout), and a user can put any third party utility there that s/he wants - from multi-core compression ones (like pbzip2) to encyption ones (like aespipe and ccyrpt and similar, or others that employ paralellisation or special hardware support), or a combination of those. Current encryption and compressions can be provided as separate applications included in default bacula distribution. What we might additionally need though, is some way to put some identifier as to which filter was used for which job, so one can retain compatibility with older backups if one decides to change the crypto/compression) I know just enough crypto to know that without knowing way much more I'm likely to make terrible mistakes (and we all still remember Debian "fixing" OpenSSL security fiasco, don't we?). Raw ECB sounds just like one of those mistakes. By using filters, we can have both maximum flexibility, and actually have the more secure encryption (by recommending the proven encryption tools to use instead of reinventing the wheel [probably badly]) -- Matija Nalis Odjel racunalno-informacijskih sustava i servisa Hrvatska akademska i istrazivacka mreza - CARNet Josipa Marohnica 5, 1 Zagreb tel. +385 1 6661 616, fax. +385 1 6661 766 www.CARNet.hr -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Simplified pools
On 7/04/2010 8:26 PM, Phil Stracchino wrote: >> Oh well, I'll use it if it still works. It's not like I can't port the >> config over to whatever is required later if it is ever dropped. Thanks >> for the tip. >> >> I'm troubled by the approach taken with Bacula of "it's possible with >> some scripting, so why should Bacula make it simple and easy to do this >> common task". Then again, perhaps I'm just trying to do things the wrong >> way and what I see as simple and common, like concurrent backups to a >> sd, isn't. > > Part of what's going on is that Bacula has explored several different > routes for scriptable automation so as to provide the maximum possible > flexibility, but so far each of them has been found to have drawbacks. > (The principal drawback in the case of Python being, in my admittedly > inexpert opinion, that Python is something of a niche language, much > less widely known and used than - for example - Perl. While I know > people who swear by Python, I also know plenty who swear *at* it, not > least for its use of whitespace to define lexical scope.) While I don't care the tiniest bit about the whitespace issue, I used to swear at Python very frequently because: - The dev team's answer to any embedding issues is "you're doing it wrong. You should rewrite your whole application as a Python module and run it under the Python interpreter, not embed the Python interpreter in your app." You can imagine how appealing THAT idea is with, say, a large Qt-based C++ GUI application... - The Global Interpreter Lock (GIL). 'nuff said. I no longer maintain Scribus's embedded scripting support and my embedded Python headaches are gone, but I'd certainly never choose it again for any embedding project when I could pick Lua or a JavaScript library instead. ( OK, that's not strictly true, it's great if your app is single-threaded and always will be, not powered by an event loop, and you want scripting users to be able to do anything without any restriction or security policy. But that's kinda rare. ) > While > different solutions have been explored at different times, generally the > original basic functionality that may be limited but works has not been > removed unless it's actively broken or causing things to break. (I'm > actually surprised that schedule-based overrides have not been removed, > given the Pool problems already discussed.) I suspect there are other use cases not covered by the pool overrides. People may need schedule-based overrides that don't neatly correspond to job levels. IMHO before they could be removed there'd need to be a way to override anything, not just pool selection, by job level. Even then it's hard to say if people would need them for other things. I guess that's what on-startup deprecation warnings are for... -- Craig Ringer -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Volumes on full disk - lots of 0 byte files
On 04/07/10 02:17, Rex Chan wrote: > Hi guys, > > once disk space becomes full, our bacula configuration starts writing > out a whole bunch of 0 byte volumes to disk. > > I can manually delete all the 0 byte files, but does the catalog > require cleanup as well? > > What changes can I make to our configuration to prevent this from > happening? Honestly, the truth is that free space monitoring isn't Bacula's job. Bacula doesn't know what else it's sharing disk space with. That said, the subject has come up before; there has been discussion of how best to handle the problem. Nothing has been implemented yet though. As best I recall, the general conclusion of the discussion was that there needed to be some kind of minimum free space directive for disk pools, such that Bacula would decline to create any new volumes if there was less than [directive value] free space on the device. I'm not sure whether this has actively gone anywhere yet. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Poor Performance
Hi
I have very poor performance when the agent send the spool to the bacula
server
I currently use a RAID5 SAS 15K, 1GB Ethernet
hdparm -t /dev/sda2
/dev/sda2:
Timing buffered disk reads: 692 MB in 3.00 seconds = 230.41 MB/sec
Any idea what I can check?
07-Apr 00:08 angel-dir JobId 1443: Start Backup JobId 1443,
Job=fraser.2010-04-07_00.05.01_01
07-Apr 00:08 angel-dir JobId 1443: Using Device "LTODriver"
07-Apr 00:09 angel-sd JobId 1443: Volume "DMS002L4" previously written,
moving to end of data.
07-Apr 00:09 angel-sd JobId 1443: Ready to append to end of Volume
"DMS002L4" at file=237.
07-Apr 00:09 angel-sd JobId 1443: Spooling data ...
07-Apr 00:33 angel-sd JobId 1443: Job write elapsed time = 00:23:49,
Transfer rate = 8.092 M bytes/second
07-Apr 00:33 angel-sd JobId 1443: Committing spooled data to Volume
"DMS002L4". Despooling 11,574,810,050 bytes ...
07-Apr 00:36 angel-sd JobId 1443: Despooling elapsed time = 00:02:34,
Transfer rate = 75.16 M bytes/second
07-Apr 00:36 angel-sd JobId 1443: Sending spooled attrs to the Director.
Despooling 5,369,625 bytes ...
07-Apr 00:36 angel-dir JobId 1443: Bacula angel-dir 3.0.1 (30Apr09):
07-Apr-2010 00:36:47
Build OS: i686-pc-linux-gnu redhat Enterprise release
JobId: 1443
Job:fraser.2010-04-07_00.05.01_01
Backup Level: Incremental, since=2010-04-06 00:11:20
Client: "fraser" 3.0.1 (30Apr09)
i686-pc-linux-gnu,redhat,Enterprise release
FileSet:"Fraser" 2010-01-13 08:12:58
Pool: "IncrPool" (From Job IncPool override)
Catalog:"MyCatalog" (From Client resource)
Storage:"LTO4Driver" (From Job resource)
Scheduled time: 07-Apr-2010 00:05:01
Start time: 07-Apr-2010 00:08:31
End time: 07-Apr-2010 00:36:47
Elapsed time: 28 mins 16 secs
Priority: 10
FD Files Written: 17,598
SD Files Written: 17,598
FD Bytes Written: 11,560,818,747 (11.56 GB)
SD Bytes Written: 11,563,514,692 (11.56 GB)
Rate: 6816.5 KB/s
FileSet {
Name = "Fraser"
Include {
Options {
signature = MD5
}
File = /etc
File = /var/spool/mail
File= /export/home0
File= /export/home1
File= /export/home2
File= /export/home3
File = /export/netinst
File = /usr/local
File = /local
}
Exclude {
File = /sys
File = /boot
File = /dev
File = /proc
File = /tmp
File = /.journal
File = /.fsck
}
}
Job {
JobDefs = "DMSDefault"
Name = "fraser"
Enabled= yes
FileSet = "Fraser"
Level = Full
Client=fraser
Write Bootstrap = "/local/bacula/bin/working/Fraser.bsr"
# Full Backup Pool = FullPool
# Incremental Backup Pool = IncrPool
# Differential Backup Pool = DiffPool
Schedule = "DMS"
SpoolData=yes
}
--
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Idea/suggestion for dedicated disk-based sd
On 04/07/10 01:13, Craig Ringer wrote: > You appear to be assuming that "disk backup" == "single disk backup" or > "set of simple disks". No, not at all. (Though in many smaller installations, it is.) I was just noting that a single-purpose disk backup tool in which elimination of interleaving and minimization of fragmentation are desired has some significantly different needs than a general-purpose filesystem. > Most practical disk backup setups will involve large RAID-5, RAID-6 or > RAID-10 arrays. These tend to be striped across the spindles anyway, and > the file system is rarely properly aware of how this striping occurs. *nod* Indeed. Nor should it care, since for general purposes, it shouldn't matter. It's not the filesystem's problem - except of course in advanced filesystems like ZFS where the filesystem IS the RAID implementation. > For what it's worth, a quick check on my volumes does reveal significant > (25% or so) fragmentation. I'm going to see if I can extend the sd with > posix_fallocate(...) support and see if I can reduce that. That would be an interesting and probably welcome proposal. > Interleaved disk volumes complicate management of retention periods and > lifetimes. They also make it harder to see what's using what space, > where. That's why I want to avoid them, not for performance reasons. Ah, that's very true. Have you looked into using migration to de-interleave jobs after backup? Bang it all onto disk as fast as possible during your backup window, sort it out at leisure later? > I'm wondering why I should worry too much about fragmentation, actually. > The array performs quite well when significantly fragmented; I haven't > noticed any significant write performance drops over time. > > It may slow restores a little, but again with a many-spindle array I'm > not sure how much practical effect it'll have. Is fragmentation > avoidance worth all this complexity? Probably not. :) -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Simplified pools
On 04/07/10 00:54, Craig Ringer wrote:
> Yeah, that works ... when you only need to control pools based on job level.
>
> My problem is that sometimes I need to control other job attibutes, such
> as spooling, by job level.
>
> The mechanism just needs to be generalised a little, so your above
> example becomes:
>
>
> JobDefs {
> Name = Backup
> Type = Backup
> Level = Full
> Full {
> Pool = Full-Tape
> }
> Differential {
> Pool = Diff-Disk
> }
> Incremental {
> Pool = Incr-Disk
> }
> Schedule = "Monthly Rotation"
> [...]
> }
>
> ... or something like that, so other things may be overridden based on
> level.
This is a good suggestion, I think. I'd go ahead and write it up and
submit it. :)
--
Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355
ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org
Renaissance Man, Unix ronin, Perl hacker, Free Stater
It's not the years, it's the mileage.
--
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Simplified pools
On 04/07/10 00:42, Craig Ringer wrote:
> Phil Stracchino wrote:
>> I can confirm that it still works in 5.x as well. I use this for disk
>> volumes:
>>
>> Label Format =
>> "FULL-$Year${Month:p/2/0/r}${Day:p/2/0/r}-${Hour:p/2/0/r}:${Minute:p/2/0/r}"
>>
>> (with alterations for Differential and Incremental pools, of course.)
>
> If it's not going to be removed and replaced with something 10x as
> complicated, perhaps it should be un-deprecated?
As far as I know, there are no plans at this time to actually remove
that functionality. I'm not really clear on why it was deprecated.
> Oh well, I'll use it if it still works. It's not like I can't port the
> config over to whatever is required later if it is ever dropped. Thanks
> for the tip.
>
> I'm troubled by the approach taken with Bacula of "it's possible with
> some scripting, so why should Bacula make it simple and easy to do this
> common task". Then again, perhaps I'm just trying to do things the wrong
> way and what I see as simple and common, like concurrent backups to a
> sd, isn't.
Part of what's going on is that Bacula has explored several different
routes for scriptable automation so as to provide the maximum possible
flexibility, but so far each of them has been found to have drawbacks.
(The principal drawback in the case of Python being, in my admittedly
inexpert opinion, that Python is something of a niche language, much
less widely known and used than - for example - Perl. While I know
people who swear by Python, I also know plenty who swear *at* it, not
least for its use of whitespace to define lexical scope.) While
different solutions have been explored at different times, generally the
original basic functionality that may be limited but works has not been
removed unless it's actively broken or causing things to break. (I'm
actually surprised that schedule-based overrides have not been removed,
given the Pool problems already discussed.)
--
Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355
ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org
Renaissance Man, Unix ronin, Perl hacker, Free Stater
It's not the years, it's the mileage.
--
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] bad response from stored to open command
The before backup script is empty at this point, so that's not taking any time. What would cause such an error? This was working until I needed to reload the firmware on my router. mtu-fd is on the internet, jon-dir is behind my router. I've got port 9103 forwarded from my router to the computer jon. I've got an entry in /etc/hosts on mtu that has the IP of the router associated with the name jon. Running tcpdump on jon I see traffic in both directions on port 9103. Any ideas? 07-Apr 01:23 jon-dir JobId 5312: Start Backup JobId 5312, Job=mtu.2010-04-07_01.05.01_00 07-Apr 01:23 jon-dir JobId 5312: Created new Volume "mtu-fd_mtu_2010-04-07_01:23:13_Incremental" in catalog. 07-Apr 01:23 jon-dir JobId 5312: Using Device "FileStorage" 07-Apr 01:23 mtu-fd JobId 5312: shell command: run ClientRunBeforeJob "/etc/bacula/before-full-backup.sh" 07-Apr 01:23 jon-dir JobId 5312: Sending Accurate information. 07-Apr 03:52 mtu-fd JobId 5312: Fatal error: Bad response from stored to open command 07-Apr 03:52 jon-dir JobId 5312: Error: Bacula jon-dir 3.0.3 (18Oct09): 07-Apr-2010 03:52:48 -- Jon Schewe | http://mtu.net/~jpschewe If you see an attachment named signature.asc, this is my digital signature. See http://www.gnupg.org for more information. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Idea/suggestion for dedicated disk-based sd
On 04/07/10 01:05, Craig Ringer wrote: > Phil Stracchino wrote: >> It is possible right now to open more than one file-based volume at a >> time. >> You simply need to define multiple storage devices under the same >> storage daemon; each device can have one volume open at a time. > > Yep. My suggestion was merely a way around the inconvenience and > configuration bloat required to do this at present. I'm saying it > shouldn't be necessary to go through all that just to achieve > concurrent disk writes. I hear you. I think it's partly a symptom of the principle that being required to handle a great number of different possible configurations tends to make it a little difficult to optimize for any specific one of them. > I'm also using a striped RAID-6 volume across eight 1TB disks, which > means access patterns are going to be somewhat random anyway. > Fragmentation just isn't a big issue. Yup, I have a similar configuration myself - eight working disks plus hot spares on RAIDZ2. I really don't concern myself about fragmentation on it; but by the same token, I really don't concern myself about interleaving either. On disk volumes, I really don't find it to be an issue (and so far I've found it comparatively little issue on LTO tape, either). >> The odds are good >> the data will actually end up in the same places on disk; it will just >> appear, from the point of view of scanning volumes, to be unfragmented, >> because the disk system hides the fragmentation from you - exactly as it >> is supposed to. As previously mentioned, this can be overcome >> essentially only by using a complete custom filesystem and disk driver >> that treats disks like tapes. > > Actually, you can do a pretty good job just by using > posix_fallocate(...) to inform the file system of how much space you > expect to need. > > The sd doesn't currently appear to use posix_fallocate(...) at all. I'll freely admit I haven't looked much at the low-level code in that area. (As a matter of fact, it's been embarrassingly long since I've looked at the low-level code at all; despite my good intentions about returning to active development, I've found other projects consuming most of my time.) > Of course. I do much the same thing myself. I'm just suggesting that > perhaps this sort of thing shouldn't have to be built by each Bacula > user, but perhaps be built in to the system to make it a bit more > practical and convenient? Now that automatic truncation of purged volumes exists, automatic deletion is a pretty small step. I can't see that there would be any objection to that change, if you were to submit a proposal on the dev list. > I don't mean to come off as a grumpy/demanding user. I'm not saying "you > should do this!", I'm seeking comments and opinion on whether these > things are worth doing. I may well tackle them myself if so, I just > don't want to waste my time on things that will get rejected. By all means contribute code. :) You're right to make sure the proposed changes will be accepted first though. -- Phil Stracchino, CDK#2 DoD#299792458 ICBM: 43.5607, -71.355 ala...@caerllewys.net ala...@metrocast.net p...@co.ordinate.org Renaissance Man, Unix ronin, Perl hacker, Free Stater It's not the years, it's the mileage. -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] VirtualFull mysql query blocks other jobs for a long time
Graham Keeling wrote: > > On Tue, Apr 06, 2010 at 09:01:13AM -0700, ebollengier wrote: >> Hello Graham, > > Hello, thanks for your reply. > >> Graham Keeling wrote: >> > >> > Hello, >> > I'm using bacula-5.0.1. >> > I have a 2.33GHz CPU with 2G of RAM. >> > I am using MySQL. >> > I had a VirtualFull scheduled for my client. >> > >> > My log says the following: >> > >> > Apr 4 18:56:02 Start Virtual Backup JobId 56, >> > Job=Linux:cvs.2010-04-04_18.56.00_03 >> > Apr 4 18:56:02 This Job is not an Accurate backup so is not >> equivalent >> > to a Full backup. >> > Apr 6 03:56:12 Bootstrap records written to >> > /var/lib/bacula/backup.dev.equiinet.com-.restore.1.bsr >> > Apr 6 10:23:20 Ready to read from volume "backup-0002" on device >> "Tower >> > 2.0" (/write/mnt/Tower 2). >> > Apr 6 10:23:20 Labeled new Volume "backup-0046" on device "Tower 2.1" >> > (/write/mnt/Tower 2). >> > ...and now, backup-0046 is being written to. >> > >> > At about Apr 6 10:00, I was logged on to the director, and attempting >> to >> > run >> > a 'status, director' command on bconsole. >> > bconsole printed a few bits of information, then paused. Presumably >> > because JobId 56 had locked the database. After 10:23:20, the command >> > printed >> > the rest of its information and gave me the prompt back. >> > >> > At about Apr 6 10:00, I logged into mysql, and ran 'show full >> > processlist;', >> > which gave the following. >> > >> > | Id | User | Host | db | Command | Time | State | Info >> > | 3032 | root | localhost | bacula | Query | 22464 | Locked| >> UPDATE >> > Job SE >> > T JobStatus='R',Level='F',StartTime='2010-04-06 >> > 03:56:14',ClientId=2,JobTDate=12 >> > 70522574,PoolId=7,FileSetId=2 WHERE JobId=56 >> > >> > | 3033 | root | localhost | bacula | Query | 22464 | executing | >> SELECT >> > Path.P >> > ath, Filename.Name, Temp.FileIndex, Temp.JobId, LStat, MD5 FROM ( >> SELECT >> > FileId, >> > Job.JobId AS JobId, FileIndex, File.PathId AS PathId, File.FilenameId >> AS >> > Filena >> > meId, LStat, MD5 FROM Job, File, ( SELECT MAX(JobTDate) AS JobTDate, >> > PathId, Fil >> > enameId FROM ( SELECT JobTDate, PathId, FilenameId FROM File JOIN Job >> > USING (Job >> > Id) WHERE File.JobId IN (22,23,31,34,42,48,52) UNION ALL SELECT >> JobTDate, >> > PathId >> > , FilenameId FROM BaseFiles JOIN File USING (FileId) JOIN Job ON >> > (BaseJobId >> > = Job.JobId) WHERE BaseFiles.JobId IN (22,23,31,34,42,48,52) ) AS tmp >> > GROUP BY P >> > athId, FilenameId ) AS T1 WHERE (Job.JobId IN ( SELECT DISTINCT >> BaseJobId >> > FROM B >> > aseFiles WHERE JobId IN (22,23,31,34,42,48,52)) OR Job.JobId IN >> > (22,23,31,34,42, >> > 48,52)) AND T1.JobTDate = Job.JobTDate AND Job.JobId = File.JobId AND >> > T1.PathId >> > = File.PathId AND T1.FilenameId = File.FilenameId ) AS Temp JOIN >> Filename >> > ON (Fi >> > lename.FilenameId = Temp.FilenameId) JOIN Path ON (Path.PathId = >> > Temp.PathId) WH >> > ERE FileIndex > 0 ORDER BY Temp.JobId, FileIndex ASC | >> > >> > 'Time 22464' is equivalent to about 6 hours, so I presume that after >> > 'Bootstrap >> > records written', bacula has been waiting for this sql command, during >> > which >> > time it could not do anything else. Backups of other clients were >> failing >> > because I had Max Wait Time set to something less than 6 hours. >> > >> > The following is a listing of the number of files that each previous >> > backup >> > said it had backed up, which doesn't seem extraordinarily large to my >> > mind. >> > >> > Full: 810,400 >> > Incr: 52,487 >> > Incr: 52,485 >> > Incr: 52,485 >> > Incr: 332,649 >> > Incr: 52,492 >> > Incr: 52,593 >> > Incr: 332,677 >> > Incr: 52,497 >> > Incr: 52,497 >> > Incr: 52,497 >> > >> > >> > So, my questions are: >> > >> > Does it really need to take so long? >> > Is it expected that this Job blocks everything else for 6 hours? >> > >> >> I think that this part of the code doesn't use a dedicated connection, it >> would be nice to >> improve this area. (other part of the code are using a specific >> connection >> to do this work) >> >> >> Graham Keeling wrote: >> > >> > Is there anything I can do that would speed it up? >> > Perhaps even more importantly, what was it doing for the 33 hour period >> > between Apr 4 18:56:02 and Apr 6 03:56:12? Was it just writing the >> > bootstrap >> > records? >> > >> > On the Path table, I have these: >> >PRIMARY KEY(PathId), >> >INDEX (Path(255)) >> > >> > On the File table, I have these: >> >PRIMARY KEY(FileId), >> >INDEX (JobId, PathId, FilenameId), >> >INDEX (PathId), >> >INDEX (FilenameId) >> > >> > On the Job table, I have these: >> >PRIMARY KEY(JobId), >> >INDEX (Name(128)), >> >INDEX (ClientId) >> > >> >> Did you run tests with the original schema? It looks like you removed >> important index such as the one >> on JobId and you added others, and i'm not sure that MySQ
Re: [Bacula-users] VirtualFull mysql query blocks other jobs for a long time
On Tue, Apr 06, 2010 at 09:01:13AM -0700, ebollengier wrote: > Hello Graham, Hello, thanks for your reply. > Graham Keeling wrote: > > > > Hello, > > I'm using bacula-5.0.1. > > I have a 2.33GHz CPU with 2G of RAM. > > I am using MySQL. > > I had a VirtualFull scheduled for my client. > > > > My log says the following: > > > > Apr 4 18:56:02 Start Virtual Backup JobId 56, > > Job=Linux:cvs.2010-04-04_18.56.00_03 > > Apr 4 18:56:02 This Job is not an Accurate backup so is not equivalent > > to a Full backup. > > Apr 6 03:56:12 Bootstrap records written to > > /var/lib/bacula/backup.dev.equiinet.com-.restore.1.bsr > > Apr 6 10:23:20 Ready to read from volume "backup-0002" on device "Tower > > 2.0" (/write/mnt/Tower 2). > > Apr 6 10:23:20 Labeled new Volume "backup-0046" on device "Tower 2.1" > > (/write/mnt/Tower 2). > > ...and now, backup-0046 is being written to. > > > > At about Apr 6 10:00, I was logged on to the director, and attempting to > > run > > a 'status, director' command on bconsole. > > bconsole printed a few bits of information, then paused. Presumably > > because JobId 56 had locked the database. After 10:23:20, the command > > printed > > the rest of its information and gave me the prompt back. > > > > At about Apr 6 10:00, I logged into mysql, and ran 'show full > > processlist;', > > which gave the following. > > > > | Id | User | Host | db | Command | Time | State | Info > > | 3032 | root | localhost | bacula | Query | 22464 | Locked| UPDATE > > Job SE > > T JobStatus='R',Level='F',StartTime='2010-04-06 > > 03:56:14',ClientId=2,JobTDate=12 > > 70522574,PoolId=7,FileSetId=2 WHERE JobId=56 > > > > | 3033 | root | localhost | bacula | Query | 22464 | executing | SELECT > > Path.P > > ath, Filename.Name, Temp.FileIndex, Temp.JobId, LStat, MD5 FROM ( SELECT > > FileId, > > Job.JobId AS JobId, FileIndex, File.PathId AS PathId, File.FilenameId AS > > Filena > > meId, LStat, MD5 FROM Job, File, ( SELECT MAX(JobTDate) AS JobTDate, > > PathId, Fil > > enameId FROM ( SELECT JobTDate, PathId, FilenameId FROM File JOIN Job > > USING (Job > > Id) WHERE File.JobId IN (22,23,31,34,42,48,52) UNION ALL SELECT JobTDate, > > PathId > > , FilenameId FROM BaseFiles JOIN File USING (FileId) JOIN Job ON > > (BaseJobId > > = Job.JobId) WHERE BaseFiles.JobId IN (22,23,31,34,42,48,52) ) AS tmp > > GROUP BY P > > athId, FilenameId ) AS T1 WHERE (Job.JobId IN ( SELECT DISTINCT BaseJobId > > FROM B > > aseFiles WHERE JobId IN (22,23,31,34,42,48,52)) OR Job.JobId IN > > (22,23,31,34,42, > > 48,52)) AND T1.JobTDate = Job.JobTDate AND Job.JobId = File.JobId AND > > T1.PathId > > = File.PathId AND T1.FilenameId = File.FilenameId ) AS Temp JOIN Filename > > ON (Fi > > lename.FilenameId = Temp.FilenameId) JOIN Path ON (Path.PathId = > > Temp.PathId) WH > > ERE FileIndex > 0 ORDER BY Temp.JobId, FileIndex ASC | > > > > 'Time 22464' is equivalent to about 6 hours, so I presume that after > > 'Bootstrap > > records written', bacula has been waiting for this sql command, during > > which > > time it could not do anything else. Backups of other clients were failing > > because I had Max Wait Time set to something less than 6 hours. > > > > The following is a listing of the number of files that each previous > > backup > > said it had backed up, which doesn't seem extraordinarily large to my > > mind. > > > > Full: 810,400 > > Incr: 52,487 > > Incr: 52,485 > > Incr: 52,485 > > Incr: 332,649 > > Incr: 52,492 > > Incr: 52,593 > > Incr: 332,677 > > Incr: 52,497 > > Incr: 52,497 > > Incr: 52,497 > > > > > > So, my questions are: > > > > Does it really need to take so long? > > Is it expected that this Job blocks everything else for 6 hours? > > > > I think that this part of the code doesn't use a dedicated connection, it > would be nice to > improve this area. (other part of the code are using a specific connection > to do this work) > > > Graham Keeling wrote: > > > > Is there anything I can do that would speed it up? > > Perhaps even more importantly, what was it doing for the 33 hour period > > between Apr 4 18:56:02 and Apr 6 03:56:12? Was it just writing the > > bootstrap > > records? > > > > On the Path table, I have these: > >PRIMARY KEY(PathId), > >INDEX (Path(255)) > > > > On the File table, I have these: > >PRIMARY KEY(FileId), > >INDEX (JobId, PathId, FilenameId), > >INDEX (PathId), > >INDEX (FilenameId) > > > > On the Job table, I have these: > >PRIMARY KEY(JobId), > >INDEX (Name(128)), > >INDEX (ClientId) > > > > Did you run tests with the original schema? It looks like you removed > important index such as the one > on JobId and you added others, and i'm not sure that MySQL can play properly > with composed indexes (it doesn't looks like) in this treatment. In future, to avoid generating this sort of complaint, I will not fiddle with the default bacula indexes. However, I t
[Bacula-users] PATCH: add AES-ECB support to Bacula
Bacula currently only uses the AES CBC cypher mode. This cypher can't be
effectively parallelized because block n+1 depends on block n.
The AES ECB mode was developed to address that limitation. Support for
it is built into OpenSSL 0.9.8 . This trivial patch adds appropriate
entries in the public crypto.h enum for cyphers so that the AES ECB mode
may be requested by the fd, and translates this into OpenSSL cypher
settings in crypto.c .
OpenSSL doesn't automatically do ECB encryption in parallel. However,
unlike the currently used CBC mode, it's possible to do.
Right now, this patch gains you little or nothing. It also costs little
or nothing, and probably shouldn't actually be applied at present. I'm
putting it up here to keep it and the notes about crypto in the archives
in case I don't have any luck following it up with an actually parallel
implementation and others are looking into it later.
The next step is to try to spawn worker threads to encrypt chunks in
parallel. Hopefully this will be possible with OpenSSL...
--
Craig Ringer
diff --git a/bacula/src/lib/crypto.c b/bacula/src/lib/crypto.c
index fb52c25..077b8d2 100644
--- a/bacula/src/lib/crypto.c
+++ b/bacula/src/lib/crypto.c
@@ -1022,6 +1022,11 @@ CRYPTO_SESSION *crypto_session_new (crypto_cipher_t cipher, alist *pubkeys)
cs->cryptoData->contentEncryptionAlgorithm = OBJ_nid2obj(NID_aes_128_cbc);
ec = EVP_aes_128_cbc();
break;
+ case CRYPTO_CIPHER_AES_128_ECB:
+ /* AES 128 bit ECB */
+ cs->cryptoData->contentEncryptionAlgorithm = OBJ_nid2obj(NID_aes_128_ecb);
+ ec = EVP_aes_128_ecb();
+ break;
#ifndef HAVE_OPENSSL_EXPORT_LIBRARY
case CRYPTO_CIPHER_AES_192_CBC:
/* AES 192 bit CBC */
@@ -1033,6 +1038,11 @@ CRYPTO_SESSION *crypto_session_new (crypto_cipher_t cipher, alist *pubkeys)
cs->cryptoData->contentEncryptionAlgorithm = OBJ_nid2obj(NID_aes_256_cbc);
ec = EVP_aes_256_cbc();
break;
+ case CRYPTO_CIPHER_AES_256_ECB:
+ /* AES 256 bit ECB */
+ cs->cryptoData->contentEncryptionAlgorithm = OBJ_nid2obj(NID_aes_256_ecb);
+ ec = EVP_aes_256_ecb();
+ break;
#endif
case CRYPTO_CIPHER_BLOWFISH_CBC:
/* Blowfish CBC */
diff --git a/bacula/src/lib/crypto.h b/bacula/src/lib/crypto.h
index cb935bd..dc3171f 100644
--- a/bacula/src/lib/crypto.h
+++ b/bacula/src/lib/crypto.h
@@ -82,7 +82,9 @@ typedef enum {
CRYPTO_CIPHER_AES_128_CBC,
CRYPTO_CIPHER_AES_192_CBC,
CRYPTO_CIPHER_AES_256_CBC,
- CRYPTO_CIPHER_BLOWFISH_CBC
+ CRYPTO_CIPHER_BLOWFISH_CBC,
+ CRYPTO_CIPHER_AES_128_ECB,
+ CRYPTO_CIPHER_AES_256_ECB
} crypto_cipher_t;
/* Crypto API Errors */
--
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Automatic restores BACULA
bitbyte wrote:
>
> bitbyte wrote:
> > Is there a way to automate restores in bacula by schedules?The jobs doesn't
> > show up in the scheduled tasks.But i am able to run the job manually.
> >
> > What to specify on the client directive here,the restore location or the
> > name of the client for which files has to be restored?
> >
> > Config :
> >
> > Job {
> > Name = "fileserver"
> > Type = Restore
> > Client=name of the machine where restore has to be done
> > FileSet="fileserver"
> > Storage = SAN
> > Pool = fileserver
> > Bootstrap = "/var/bacula/working/fileserver.bsr
> > Messages = Standard
> > Where = /dump
> > }
> >
> >
> > Schedule {
> > Name = "fileserver-restore"
> > Run = Apr 3rd wed at 15:00 pm
> >
> > }
>
>
> Got that working,forgot the schedule directive in Job section.
>
> One thing i notice here is it only restores the last full backup,is there a
> way to restore the latest with the full along with the incremental after a
> full backupt?
>
> Also if i want to do a quarterly restore like every third month 2nd
> Sunday,how is it possible in a single line?
> Should i use multiple lines only here(it works with multiple lines)?
Fixed
+--
|This was sent by e_man...@yahoo.com via Backup Central.
|Forward SPAM to ab...@backupcentral.com.
+--
--
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Automatic restores BACULA
Timo Neuvonen wrote: > "bitbyte" backupcentral.com> kirjoitti > viestissä:1270627053.m2f.331960 < at > www.backupcentral.com... > > > > > > > > > Just out of curiosity, in what kind of situation automated scheduled > restores are needed? What only comes into my mind is a computer class in > elementary school where all the computers would be restored > nightly/weekly/etc to their defaults. > > -- > ´TiN > > > > -- > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > ___ > Bacula-users mailing list > Bacula-users < at > lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users :) What do you think a restore is?Think you need to go again to the elementary lessons? Periodical restores checks the status of the files that are being backed up,so that we can be sure that they are backed up correctly without errors? This just doesn't mean that we do baremetal or something like that automated :):) +-- |This was sent by e_man...@yahoo.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Automatic restores BACULA
"bitbyte" kirjoitti viestissä:1270627053.m2f.331...@www.backupcentral.com... > > Is there a way to automate restores in bacula by schedules?The jobs > doesn't show up in the scheduled tasks.But i am able to run the job > manually. > > What to specify on the client directive here,the restore location or the > name of the client for which files has to be restored? > Just out of curiosity, in what kind of situation automated scheduled restores are needed? What only comes into my mind is a computer class in elementary school where all the computers would be restored nightly/weekly/etc to their defaults. -- ´TiN -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Simplified pools
> On Tue, 06 Apr 2010 18:05:17 -0400, Phil Stracchino said:
>
> On 04/06/10 17:28, Kevin Keane wrote:
> >> From: Phil Stracchino [mailto:ala...@metrocast.net]
> >>
> >> Actually, this is a problem which has largely been fixed in
> >> Bacula-3 and later, by changing the way overrides work.
> >
> > Really? I see this problem in bacula 3.0.3 all the time. Maybe I need
> > to change something in my configuration to take advantage of this new
> > feature. How is it supposed to work?
>
> The old Schedule-based overrides are still *supported* in 3.0.3, for
> backward compatibility, but are deprecated - precisely because of the
> problem you note - in favor of the following example syntax:
>
>
> Schedule {
> Name = "Monthly Rotation"
> Run = Full 1st mon at 03:30
> Run = Differential 2nd-5th mon at 03:30
> Run = Incremental tue-sun at 03:30
> }
>
> JobDefs {
> Name = Backup
> Type = Backup
> Level = Full
> Pool = Full-Tape
> Full Backup Pool = Full-Tape
> Differential Backup Pool = Diff-Disk
> Incremental Backup Pool = Incr-Disk
> Schedule = "Monthly Rotation"
> [...]
> }
>
>
> The change was made because, basically, it proved extremely difficult to
> avoid the pool mismatch problem using the schedule-based override code.
> Putting pool overrides in the job definition allows the job to select
> its own pool, which means that in case of level promotion, it
> automatically just Does The Right Thing.
It isn't very flexible though. E.g. the "Daily, Weekly, Monthly" example in
the manual uses two different pools for Full backups at different times, with
appropriate retention times to keep the Monthly data longest.
Schedule {
Name = "NightlySave"
Run = Level=Full Pool=Monthly 1st sat at 03:05
Run = Level=Full Pool=Weekly 2nd-5th sat at 03:05
Run = Level=Incremental Pool=Daily tue-fri at 03:05
}
http://www.bacula.org/5.0.x-manuals/en/main/main/Automatic_Volume_Recycling.html#SECTION00256
__Martin
--
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Possibility of parallelising encryption?
Richard Scobie wrote: > I have a 2.8GHz Core i7 machine backing up uncompressable data spooled > onto an 8 drive RAID5, to LTO-4 tape. > > Our requirements now dictate that data encryption must be used on the > tapes and having configured this, it seems that one core is saturated > encrypting the data and the result is that tape write speed is now about > 50% slower than when encryption is not used. > > Would it be possible to optimise this task by perhaps reading data in > "chunks", which in turn can be encrypted by a core each, before being > recombined and written out to tape? Sorry to reply-to-self, but after a bit more reading it seems that the algorithm/cypher used by Bacula, aes-256-cbc, can't really be parallel encrypted on multiple cores because encrypting one block affects the way the next block is encrypted. aes-ctr mode was developed to address that issue. It doesn't seem to be hugely widely adopted yet, but is used in IPSec, appears in revisions to the TLS standard, etc. OpenSSL supports AES CTR mode at least for 128-bit, but only single-threaded. The implementation of multi-threaded aes-ctr encryption for OpenSSH I referenced earlier was also presented to the OpenSSL folks: http://marc.info/?l=openssl-dev&m=120180007117054&w=2 but I can't find any response or follow-up to it. So I guess the short answer is "parallel encryption is way harder to do than it looks". -- Craig Ringer -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Possibility of parallelising encryption?
Richard Scobie wrote: > I have a 2.8GHz Core i7 machine backing up uncompressable data spooled > onto an 8 drive RAID5, to LTO-4 tape. > > Our requirements now dictate that data encryption must be used on the > tapes and having configured this, it seems that one core is saturated > encrypting the data and the result is that tape write speed is now about > 50% slower than when encryption is not used. The process saturating a core is the file daemon, right? > Would it be possible to optimise this task by perhaps reading data in > "chunks", which in turn can be encrypted by a core each, before being > recombined and written out to tape? Bacula uses OpenSSL for crypto support. It doesn't seem to support any other crypto libraries like NSS or GnuTLS. OpenSSL supports hardware crypto acceleration for some cyphers in a largely transparent manner. This is one option. If Bacula doesn't "just work" with hardware crypto I'd expect it to be a one-line patch to add support, going by what I've had to do to enable it in other software. Some hardware, like the Via C7 series of CPUs, have built-in AES crypto hardware (PadLock) that on a single thread can do *insane* encryption rates. On the older C3 series CPUs I've had no problems saturating a 100MBit line with encrypted ssh data, despite the gutless 400MHz C3 CPU. Intel has introduced similar instructions on their Xeon 5600 series: http://software.intel.com/en-us/articles/boosting-openssl-aes-encryption-with-intel-ipp/ I'd be lovely to be able to use the IPP libraries in Bacula (and many other things) for parallel crypto and many other parallel tasks, as they're excellent even without special hardware. Unfortunately they're rather GPL-incompatible and are only "free" for non-commercial use. ( The Intel Thread Building Blocks library *is* open source under GPLv2, though, and if Bacula wasn't already using pthreads directly would be rather nice: http://www.threadingbuildingblocks.org/ ) Anyway, if you need a software-only option, it's necessary to: 1) get OpenSSL to use multiple cores for encryption internally; 2) get Bacula to use OpenSSL to encrypt blocks using worker threads using a suitable block cypher; or 3) Use another crypto library that automatically parallelizes. None of these look easy by any stretch. (2) is probably most realistic, but as OpenSSL does some internal locking and serialization it may not be possible to encrypt on multiple threads even when using a simple block cypher where one block doesn't depend on the next or previous. I don't know much about OpenSSL and can't say more without a lot more digging. For all I know it might be necessary to ask OpenSSL for the session key, then use its low level crypto functions to encrypt blocks rather than using the higher-level stream/session interface. While not directly OpenSSL related, this might also be of interest: http://www.psc.edu/networking/projects/hpn-ssh/ http://www.psc.edu/networking/projects/hpn-ssh/papers/a14-rapier.pdf It doesn't touch on OpenSSL, but at least it's using a highly parallel AES cypher... > I'd use the hardware encryption (which presumably has no performance > impact), that is an option on this autochanger, except they want $2500 > for it... Probably because it has a custom ASIC for the crypto algorithm in use to allow it to go fast enough. The trouble with this is that if your tape drive/changer dies, you generally need another one with the same hardware crypto to restore. This is a really, really ugly situation for disaster recovery. ( Maybe things have improved since I switched away from tape, and now the hardware crypto is just an accelerator and you can still load your keys for driver-based crypto instead. I doubt it, though. ) -- Craig Ringer -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Automatic restores BACULA
bitbyte wrote:
> Is there a way to automate restores in bacula by schedules?The jobs doesn't
> show up in the scheduled tasks.But i am able to run the job manually.
>
> What to specify on the client directive here,the restore location or the name
> of the client for which files has to be restored?
>
> Config :
>
> Job {
> Name = "fileserver"
> Type = Restore
> Client=name of the machine where restore has to be done
> FileSet="fileserver"
> Storage = SAN
> Pool = fileserver
> Bootstrap = "/var/bacula/working/fileserver.bsr
> Messages = Standard
> Where = /dump
> }
>
>
> Schedule {
> Name = "fileserver-restore"
> Run = Apr 3rd wed at 15:00 pm
>
> }
Got that working,forgot the schedule directive in Job section.
One thing i notice here is it only restores the last full backup,is there a way
to restore the latest with the full along with the incremental after a full
backupt?
Also if i want to do a quarterly restore like every third month 2nd Sunday,how
is it possible in a single line?
Should i use multiple lines only here(it works with multiple lines)?
+--
|This was sent by e_man...@yahoo.com via Backup Central.
|Forward SPAM to ab...@backupcentral.com.
+--
--
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Volumes on full disk - lots of 0 byte files
On 7.4.2010 9:17, Rex Chan wrote: > Hi guys, > > once disk space becomes full, our bacula configuration starts writing > out a whole bunch of 0 byte volumes to disk. > > I can manually delete all the 0 byte files, but does the catalog > require cleanup as well? > > What changes can I make to our configuration to prevent this from > happening? > I guess you have to upgrade, when they get this bug fixed. It is a bug, no question about it. Bacula should go belly up or something. -- http://www.iki.fi/jarif/ You learn to write as if to someone else because NEXT YEAR YOU WILL BE "SOMEONE ELSE." signature.asc Description: OpenPGP digital signature -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Automatic restores BACULA
Is there a way to automate restores in bacula by schedules?The jobs doesn't
show up in the scheduled tasks.But i am able to run the job manually.
What to specify on the client directive here,the restore location or the name
of the client for which files has to be restored?
Config :
Job {
Name = "fileserver"
Type = Restore
Client=name of the machine where restore has to be done
FileSet="fileserver"
Storage = SAN
Pool = fileserver
Bootstrap = "/var/bacula/working/fileserver.bsr
Messages = Standard
Where = /dump
}
Schedule {
Name = "fileserver-restore"
Run = Apr 3rd wed at 15:00 pm
}
+--
|This was sent by e_man...@yahoo.com via Backup Central.
|Forward SPAM to ab...@backupcentral.com.
+--
--
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Upgradation to rhel 5.5 created problem in bacula setup
Recently i upgraded my server to rhel 5.5 from rhel 5.2 . After upgradation bacula stop working , starting Director stat that ' Couldn't open database' , while postgres database for catalog in accessible from psql command . What may went wrong Thanks +-- |This was sent by vishuind...@gmail.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +-- -- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
