Hi Kern,
now I finished testing Bacula 2.2.4 with configuration (without
"--enable-batch-insert")
./configure \
--sbindir=/bacula/bin \
--sysconfdir=/bacula/bin \
--with-pid-dir=/bacula/working \
--with-subsys-dir=/bacula/working \
--enable-smartalloc \
--with-mysql \
--with-working-dir=/bacula/working \
[EMAIL PROTECTED] \
[EMAIL PROTECTED] \
--with-smtp-host=localhost \
--with-baseport=9101 \
--with-db-name=bacula \
--with-db-user=bacula
And some job finished with: Termination:*** Backup Error ***
(Backup_Client3 Backup Unknown term code of Client3-fd Full).
24-Sep 10:50 backup-dir: Backup_Client3.2007-09-24_10.12.32 Fatal error: Can't
fill File table Query failed: INSERT INTO File (FileIndex, JobId, PathId,
FilenameId, LStat, MD5) SELECT batch.FileIndex, batch.JobId, Path.PathId,
Filename.FilenameId,batch.LStat, batch.MD5
FROM batch JOIN Path
ON (batch.Path = Path.Path) JOIN Filename ON
(batch.Name = Filename.Name) : ERR=Lock wait timeout exceeded;
try restarting transaction
But after automatic rescheduling, this Backup Job terminate properly.
Time before, I used for one year Bacula ver. 1.38.11 without any problem - I
think, it is problem or issue of Bacula 2.2.4
Do you think, that 772,757 files are "a large number of files" for Bacula?
Best Regards,
Ondrej.
---------- Original Message ----------------------------------
From: Kern Sibbald <[EMAIL PROTECTED]>
Date: Mon, 24 Sep 2007 10:45:24 +0200
>On Monday 24 September 2007 09:54, Ondrej Planka wrote:
>> Hi Kern,
>>
>> tahnks for your TIP, but this issue is still open.
>
>Please explain. I don't see why the issue should be open with what you
>describe below.
>
>> When I once more configured Bacula 2.2.4 (old version -make uninstall, make
>> distclean) without "--enable-batch-insert" - job appears to be correct -
>> now I am testing this issue.
>>
>> BUT: When backup job finished, I saw this status (command status dir):
>>
>> Running Jobs:
>> JobId Level Name Status
>> ======================================================================
>> 9963 Full Backup_client1.2007-09-24_05.42.35 has terminated
>> 9964 Full Backup_client2.2007-09-24_05.42.39 has terminated
>>
>>
>> and mysqld consume about 70% of CPU and status of client1 is
>>
>>
>> Terminated Jobs:
>> JobId Level Files Bytes Status Finished Name
>> ======================================================================
>> 9964 Full 772,757 49.17 G OK 24-Sep-07 09:24 Backup_client1
>>
>>
>> After about 20 minutes, command status dir show no Running Jobs and I got
>> emails of Bacula Job Backup - status OK.
>>
>> Is it some features of Bacula 2.2.4 ??
>
>From the information you show above (a large number of files), this looks
>reasonable. If you compare the total running time of similar jobs under
>2.2.4 and 1.38.11, 2.2.4 should be faster. With such a large number of files
>per backup, you will probably want to tune your MySQL configuration a bit --
>give it more memory, ... You can find a few pointers in the Bacula manual
>for tuning MySQL, but what you really want is in the MySQL documentation.
>
>Regards,
>
>Kern
>
>>
>> Thanks and Best Regards.
>>
>> Ondrej
>>
>> ---------- Original Message ----------------------------------
>> From: Kern Sibbald <[EMAIL PROTECTED]>
>> Date: Mon, 24 Sep 2007 08:14:39 +0200
>>
>> >Hello,
>> >
>> >I've tested this backwards and forwards with all kinds of variations,
>> >including looking at the code. Everything that I do works perfectly fine.
>> >
>> >A few points:
>> >
>> >1. Make sure you are always starting from a clean source file. If you
>> > built it on another machine then copied it, the source is not clean. The
>> > best bet is to go back and get the original tar file, alternatively a
>> > "make distclean", if you have done a ./configure once, should work.
>> >
>> >2. The default is "--enable-batch-insert" so there is no need to
>> > explicitly turn it on, it will always be if the thread safe libraries are
>> > found unless you explicitly turn it off with "--disable-batch-insert".
>> >
>> >3. The batch insert code is either on or off. I see no way it can be
>> >partially on and partially off, and if you have a clean starting
>> > environment, I cannot see how the output of ./configure can differ from
>> > what is actually done.
>> >
>> >4. The errors you guys are seeing is because batch insert is turned on but
>> >something went wrong in creating the batch table. I suspect that you have
>> > not setup the correct permissions on MySQL that allows the user you are
>> > running with (normally "bacula") to create tables. However, if that is
>> > the case, there *should* be an error message that is printed -- I don't
>> > imagine that error path has been tested though. Something to be
>> > investigated more.
>> >
>> >5. You might try doing an "ldd bacula-dir" to see what shared objects it
>> > is using.
>> >
>> >6. You can check the file <bacula-source>/src/config.h after running
>> >your ./configure to see if HAVE_BATCH_FILE_INSERT is turned on or off.
>> >
>> >7. My tests were done on MySQL 5.0.22 and MySQL 5.0.26.
>> >
>> >
>> >After having thought about it overnight, looked at the code, and
>> > successfully run the tests here, I suspect that the most likely problem
>> > is #4. If there is some "bug" here, it must be a bit unusual.
>> >
>> >Best regards,
>> >
>> >Kern
>> >
>> >On Sunday 23 September 2007 21:26, Ondrej PLANKA wrote:
>> >> Hi guys,
>> >>
>> >> I have the same problem. I upgraded Bacula 1.38.11 (SuSE Linux + Mysql
>> >> 5.0.24) to 2.2.4 and I got thousand error messages as you seen below.
>> >>
>> >> 22-Sep 07:34 noem-dir: Backup_TOV.2007-09-22_04.34.57 Fatal error:
>> >> sql_create.c:730 sql_create.c:730 insert INSERT INTO batch VALUES
>> >> (510514,10005,'/data/install/src/mysql-4.1.21/ndb/test/run-test/','','gH
>> >> BOCV EH/ C A A A Lg gAA B BG9G4N BEvk3F BE5YvP A A C','0') failed:
>> >> Table 'bacula.batch' doesn't exist
>> >>
>> >> I use this Bacula 2.2.4 configuration:
>> >>
>> >> ./configure \
>> >> --sbindir=/bacula/bin \
>> >> --sysconfdir=/bacula/bin \
>> >> --with-pid-dir=/bacula/working \
>> >> --with-subsys-dir=/bacula/working \
>> >> --enable-smartalloc \
>> >> --with-mysql \
>> >> --with-working-dir=/bacula/working \
>> >> [EMAIL PROTECTED] \
>> >> [EMAIL PROTECTED] \
>> >> --with-smtp-host=localhost \
>> >> --with-baseport=9101 \
>> >> --with-db-name=bacula \
>> >> --with-db-user=bacula \
>> >> --enable-batch-insert
>> >>
>> >> When I will remove "--enable-batch-insert " does Bacula work faster then
>> >> ver. 1.38.11 ?
>> >>
>> >> Thanks,
>> >>
>> >> Ondrej
>> >>
>> >> Kern Sibbald napsal(a):
>> >> > On Saturday 22 September 2007 23:44, Arno Lehmann wrote:
>> >> >> Hi,
>> >> >>
>> >> >> I'm seeing an interesting thing at a Bacula 2.2.4 installation.
>> >> >>
>> >> >> This is on FreeBSD 7 current, which is not an OS I know well, so I
>> >> >> might present information unrelated to each other here. See below for
>> >> >> details where I'm unsure.
>> >> >>
>> >> >> This is a Bacula installed from the ports. In
>> >> >> /usr/ports/sysutils/bacula-client/work/bacula-2.2.4/config.out I see
>> >> >> the following line:
>> >> >>
>> >> >> Batch insert enabled: no
>> >> >>
>> >> >> So I assume this Bacula was built without batch inserts enabled.
>> >> >
>> >> > That is correct. Batch inserts are not turned on.
>> >> >
>> >> >> The catalog database does not have a batch table, and I wouldn't
>> >> >> expect that to be created, too:
>> >> >>
>> >> >> grep -i batch
>> >> >> /usr/ports/sysutils/bacula-client/work/bacula-2.2.4/src/cats/make_mys
>> >> >>ql_ tab les ; echo $?
>> >> >> 1
>> >> >>
>> >> >> So, everything I see indicates that this Bacula should run without
>> >> >> batch inserts. Do you agree so far?
>> >> >>
>> >> >> But, while a job is running, I get tons of lines like these in the
>> >> >> messages:
>> >> >>
>> >> >> sql_create.c:730 insert INSERT INTO batch VALUES
>> >> >> (1028031,12,'/removed/to/protect/my/customer/from/information/leaks/.
>> >> >>..' ,'s ome file name','vK1fS DP87 IGk B Pp Pp A WOgA BAA C0F BGZxdD
>> >> >> BDIUsU BGZxdD A A C','6UkhIVIIDIW4ho1M0i9LYg') failed:
>> >> >> Table 'bacula.batch' doesn't exist
>> >> >>
>> >> >> This seems to indicate Bacula does try to work with batch inserts.
>> >> >>
>> >> >> To make this more interesting, the ./configure was actually called
>> >> >> with --enable-batch-insert. So I conclude that configure, while
>> >> >> checking for batch insert capabilities, does not turn all the
>> >> >> functions off when this would be appropriate. As far as I know, batch
>> >> >> insert needs the thread-safe mysql client libraries. These seem to be
>> >> >> present on the system:
>> >> >>
>> >> >> ls -l /usr/local/lib/mysql/*_r.*
>> >> >> -rw-r--r-- 1 root wheel 551490 Sep 19 00:46
>> >> >> /usr/local/lib/mysql/libmysqlclient_r.a
>> >> >> -rwxr-xr-x 1 root wheel 892 Sep 19 00:46
>> >> >> /usr/local/lib/mysql/libmysqlclient_r.la
>> >> >> lrwxr-xr-x 1 root wheel 22 Sep 19 00:46
>> >> >> /usr/local/lib/mysql/libmysqlclient_r.so -> libmysqlclient_r.so.16
>> >> >> -rwxr-xr-x 1 root wheel 477732 Sep 19 00:46
>> >> >> /usr/local/lib/mysql/libmysqlclient_r.so.16
>> >> >>
>> >> >> I have a hard time digging through the configure.log, but it *seems*
>> >> >> that ./configure does not look for the mysql client library in the
>> >> >> right places for this system: I see checks in
>> >> >> /usr/local/lib64/mysql/, but not in /usr/local/lib/mysql.
>> >> >
>> >> > If you have a /usr/local/lib64/... on your system, which you should
>> >> > NOT if you don't have a 64 bit CPU, then Bacula will look there first,
>> >> > then in the /usr/local/lib.
>> >> >
>> >> > Dan has added some details to the manual (unfortunately only for
>> >> > PostgreSQL), and you might want to read that as much of it applies to
>> >> > MySQL. See the Install chapter of the manual.
>> >> >
>> >> >> Can someone explain this, or give me a hint why this would be the way
>> >> >> it is? Or, even better, tell me how I can actually get batch inserts
>> >> >> enabled on this system?
>> >> >
>> >> > Normally they should be on if you have the thread safe version of the
>> >> > MySQL client loaded on your system. If you have some 64 bit
>> >> > directories, Bacula may get confused. I'd recommend that you delete
>> >> > or rename those libraries if you have a 32 bit CPU as they just waste
>> >> > space and will cause problems.
>> >> >
>> >> >> Also, I find that smaller jobs run without errors - the catalog
>> >> >> backup job does terminate without errors, and a test job I ran also
>> >> >> finished without errors.
>> >> >>
>> >> >> The catalog backup saves only one file, the test job had about 5500
>> >> >> files, and the currently still running problem job already reached
>> >> >> more than 1,000,000 files. Is there a file count threshold in the
>> >> >> batch insert code?
>> >> >>
>> >> >> As the batch table is a temporary table, I don't think it's a good
>> >> >> idea to simply create it... but how can I either ensure batch inserts
>> >> >> are not executed when not configured, or the necessary temporary
>> >> >> table is created before it's used?
>> >> >
>> >> > You shouldn't concern yourself with the "batch" table. It is a temp
>> >> > table and there is no need to create.
>> >> >
>> >> > Your problem has nothing to do with the batch insert code itself, but
>> >> > simply that it is turned off for some unknown reason.
>> >> >
>> >> > Regards,
>> >> >
>> >> > Kern
>> >> >
>> >> >> By the way - the catalog DB is MySQL 5.1.21-beta.
>> >> >>
>> >> >> Finally, one more question: As this is FreeBSD 7 current, which is
>> >> >> NOT considered stable, has anyone else a similar setup running and
>> >> >> found it to be stable?
>> >> >>
>> >> >> Thanks for your time,
>> >> >>
>> >> >> Arno
>> >> >
>> >> > ----------------------------------------------------------------------
>> >> >--- This SF.net email is sponsored by: Microsoft
>> >> > Defy all challenges. Microsoft(R) Visual Studio 2005.
>> >> > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> >> > _______________________________________________
>> >> > Bacula-devel mailing list
>> >> > [email protected]
>> >> > https://lists.sourceforge.net/lists/listinfo/bacula-devel
>> >>
>> >> ------------------------------------------------------------------------
>> >>- This SF.net email is sponsored by: Microsoft
>> >> Defy all challenges. Microsoft(R) Visual Studio 2005.
>> >> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> >> _______________________________________________
>> >> Bacula-devel mailing list
>> >> [email protected]
>> >> https://lists.sourceforge.net/lists/listinfo/bacula-devel
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Microsoft
>> Defy all challenges. Microsoft(R) Visual Studio 2005.
>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> _______________________________________________
>> Bacula-devel mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/bacula-devel
>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Bacula-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/bacula-devel
