Hello John, First, a workaround for your restore problem Bacula don't need File information to restore a backup, you can specify a bootstrap to your restore command. if you are using the WriteBootStrap option and you have it this is very simple, else you can read http://www.bacula.org/en/rel-manual/Restore_Command.html#SECTION0022130000000000000000 to generate a basic bootstrap from the job log.
restore bootstrap=/path/to/file Now, for smartalloc. If the configure option doesn't work, you can turn off SMARTALLOC in src/config.h after the configure step. This works very well. change #define SMARTALLOC 1 to #define SMARTALLOC 0 or comment them (2 lines). But for 6.000.000 files, i suppose that your backup server is too small... A full 64bit system with 4GB (ram + swap) is a minimum. Bye On Wednesday 04 June 2008 01:02:31 John Kloss wrote: > Hello, > > It was suggested that I forward this email to the bacula developers > list. > > In the time after I had sent this message originally I have been able > to compile bacula without the smartalloc routines. This required > some hacking of the code because, despite the many #ifdef's blocking > smartalloc definitions, much of the code is still dependent on the > routines. Regardless, the code did compile but does not run > (segmentation fault). > > I now suspect the issue is not in the smartalloc routines but in the > mem_pool functions. However, the configure script includes the > directive '--enable-smartalloc' which it lists as disabled by > default. The directive should read enabled by default. Considering > the difficulty in compiling without smartalloc I would suggest > removing the '--enable-smartalloc' directive altogether because as > the statement currently reads it suggests that we have a choice in > smartallocs enable/disablement, which we do-- if we're willing to > hack the code and then locate the reasons for the memory fault. > > I have tried various restore commands all to poor effect > > restore client=<client> current select all done (result-- hang) > restore client=<client> jobid=<jobid> select all done (result-- hang) > restore client=<client> jobid=<jobid> all done (result -- hang) > > My next step is to try, or rather, I am trying at this moment > > bconsole > @output /var/tmp/files.txt > *query > *12 (List Files for a selected JobId) > *<jobid> > . . . wait for 6.5 millions files to be dumped to files.txt > [EMAIL PROTECTED] > *quit > > massage data so that it represents a single list of 6.5 millions > files with no extraneous spaces or '|' separators as given by the > above command > > bconsole > restore > 7 (enter a list of files to restore) > Enter full filename: <files.txt > > I have no idea whether the above will work because it's taking > forever and a day to pull the information necessary from catalog to > perform the restore. As I mention below, I don't have forever and a > day to wait-- I have 36 hours and a coffee break. > > I'm willing to wade through code if necessary but I'd really > appreciate it if someone could at least show me where the shallow > water is. As stated, I don't have that much time, or rather, I > haven't been given much time to recover the 2.5 terabytes of data > before I am asked to leave. If the later occurs then I suppose I > will have quite a bit of time on my hands. > > Thank you. > > John Kloss. > > Begin forwarded message: > > From: John Kloss <[EMAIL PROTECTED]> > > Date: June 3, 2008 10:52:34 AM EDT > > To: baculausers <[EMAIL PROTECTED]> > > Subject: [Bacula-users] bacula-dir virtual memory limit during restore > > > > Hello, > > > > I am currently running bacula-2.2.8 compiled as a 32bit binary. > > I am using postgresql-8.3.1 as the catalog, also compiled as a 32bit > > binary. > > I am running bacula on solaris 9 running in 64bit mode which of > > course means I can run both 64bit and 32bit binaries. > > > > ulimit -v shows unlimited. I know that's a lie and that soft limit > > is 2GB. I know that I can change that to 3.5GB or so for a 32bit > > process. I have done so and then started bacula-dir. > > ulimit -d shows unlimited. I know that's also a lie and that the > > default limit is 2GB. I know that I change that to 3.5GB or so for a > > 32bit process. I have done so (along with ulimit -v) and then > > started bacula-dir. > > > > I am trying to restore 2.5 terabytes of data composed of 6.5 million > > files. > > > > My process is > > > > Run bconsole > > Choose restore > > Chose the most recent restore for a client > > Wait for the directory structure to be generated in memory (10 > > minutes tops-- postgres temp files are on a ram disk which makes life > > fast) > > Chose 'mark *' > > Watch bacula churn away for a couple of minutes and then report 6.5 > > million files marked. > > Type done. > > See that the prompt never returns. The restore never happens. > > Actually, I don't have time to wait for forever so I waited for 36 > > hours instead and saw that nothing had changed. No prompt. No > > restore. > > > > Bacula-dir consumes up to 2GB of memory and then freezes. Running > > pmap on the bacula-dir shows that, along with kernel space usage and > > library space and whatever, the heap usage is 2GB. ulimit was set to > > give bacula-dir 3.5GB. This gift of memory is apparently ignored. > > > > Thinking this was a 32bit limit I switched to a 64bit compile of > > bacula and tried the same thing. This time bacula-dir took 2.5GB of > > heap and no more even though I gifted it 10GB (I have 16GB of memory > > and it's pretty much for postgres and bacula). I still never see the > > prompt return after a 'mark *; done'. > > > > Running 64bit versions of bacula on solaris completely hoses any > > dates such as last written for media. I think this is a solaris > > sprintf thing and has nothing to do with bacula. Regardless, I don't > > want to run a 64bit version of bacula on solaris and, given the above > > limitations, it wouldn't help me anyway. > > > > Running truss -u *:: on the bacula-dir process shows it continually > > spinning in mutex locks and unlocks around memory allocation and > > frees. > > > > Previous versions of bacula (1.36) were able to restore 5.5 terabytes > > of data composed of 9 million files via the above method. Same > > machine, less memory, 32bit binaries, old version of postgres (8.0). > > The new version as I have compiled and configured cannot. > > > > How does one recover 2.5 terabytes and 6.5 million files using the > > latest version of bacula? What am I doing wrong? Is there anyway to > > change smartalloc so that it will use 3.5GBs of memory (nothing > > popped out at me when I scanned the include files)? > > > > I should note that a couple of weeks ago I had a complete system > > failure of my SAN and lost 25 terabytes of data. Bacula 1.36 > > restored all of it. I got to keep my job. Thank you bacula. > > > > John Kloss. <[EMAIL PROTECTED]> > > IT Manager, Systems Manager > > Institute of Genetic Medicine > > Johns Hopkins Medical Institution > > ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Bacula-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/bacula-devel
