Re: [Bacula-users] bacula-fd on windows 7 not listening on IPv6 address??
On 2011-03-15, Josh Fisher jfis...@pvct.com wrote:
On 3/15/2011 4:42 AM, Gavin McCullagh wrote:
Try using something like:
FDAddresses {
ip = {
addr = hostname
}
}
The ip keyword of FDAddresses allows selection of IPv4 and IPv6 addresses.
Unfortunately, it does not seem to work (at least around here).
Looking by the documentation it might be bug, or it might be that documentation
is wrong and ip is actually an alias for ipv4 only, and NOT ipv4+ipv6
superset..
So, specifying ip { addr = my.fqdn.name } binds only to IPv4 address (while
host my.fqdn.name return both A and records)
To make it listen to both IPv4 and IPv6 specified by FQDN hostname, I must
instead do:
FDAddresses = {
ipv4 = { addr = my.fqdn.name }
ipv6 = { addr = my.fqdn.name }
}
Then it works (binds to both IPv4 and IPv6 specific addresses corresponding
to my.fqdn.name). It would indeed by nicer if documentation worked as
advertised and I could do just:
FDAddresses = {
ip = { addr = my.fqdn.name }
}
but, it does not seem to work... I do not know if that is a bug in the code,
or in the documentation to be found at
http://www.bacula.org/5.0.x-manuals/en/main/main/Client_File_daemon_Configur.html
Also, documentation is surely wrong on another points, as it for example gives:
ipv6 = {
addr = 1.2.3.4;
port = 1205;
}
and 1.2.3.4 is not a valid IPv6 address (it is an IPv4 address)
Also of note for documentation (if it is going to be fixed), one should use
for example 192.0.2.0/24 for documentation IP addresses (and not 1.2.3.4 -
see RFC 5737), and for IPv6 documentation prefix is 2001:db8::/32 (see RFC 3849)
(I would glady provide a patch for documentation if I knew if above specified
HTML file is the source against which the patch should be provided, or is
it automatically generated from something else.)
I'd been using FDAddresses alright, but would prefer to just have bacula-fd
bind to the available addresses. Is that not a good idea? I guess the
above hostname-based directive isn't too bad.
Indeed, not specifying either FDAddress or FDAddresses should listen on
all available addresses. In the Linux client, it does. For the 3.x
Well, on my Debianu GNU/Linux, bacula-fd 5.0.2, it does not.
In absence of specified ipv6 addr in FDAddresses, it will listen only on
0.0.0.0 (all IPv4 addresses).
I must use:
FDAddresses = {
ipv6 = { addr = :: }
}
in order to make it listen to all IPv4 and IPv6 addresses (which is also
probably what OP wanted to do, and what probably should be bacula default
(which it currently isn't) if no FDAddress/FDAddresses are specified.
This should probably be explicitely documented (if it is intended and not a
deficiency of the code, of course)
--
Opinions above are GNU-copylefted.
--
Create and publish websites with WebMatrix
Use the most popular FREE web apps or write code yourself;
WebMatrix provides all the features you need to develop and
publish your website. http://p.sf.net/sfu/ms-webmatrix-sf
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] [Bacula-devel] Bacula 5.0.2 released
On Wed, Apr 28, 2010 at 07:23:07PM +0200, Kern Sibbald wrote: This is to let you know that Eric has released the tar.gz source files and the Windows binaries to the Bacula section of the Source Forge download area. This 5.0.2 version is primarily an important bug fix update to version 5.0.1. Thanks Kern and Eric and everyone involved in making this happen! BTW, http://bacula.org/en/ still says 25 January 2010: Bacula 5.0.0 has been released, maybe that should be updated too. -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] BAT?
On Tue, Apr 27, 2010 at 10:06:38AM -0700, Joseph Spenner wrote: This may be so obvious that I can't find it.. but where do I find BAT? In standard bacula distribution. when you compile your FD, SD and DIR, if Qt is autodetected (or if you pass --enable-bat to ./configure) the bat will be built also. -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] appending to a full tape?
On Tue, Apr 27, 2010 at 10:43:44AM +0200, IEM - network operating center (IOhannes m zmoelnig) wrote: i'm running bacula 2.4.4 in a debian/etch environment (i know that it is a bit outdated; but there is even one debian/sarge host that i cannot really update...) No worries, 2.4.4 is also the one in Lenny :) And it seems to me that is about the most stable version ATM (although the Branch-5.0 GIT seems to be rapidly fixing the remaining bugs). So for this once Debian long sta(b)le policy is for the good! bacula is doing a nightly backup (full every 3 months or so, diff every week, and incr every night) onto an autochanger (Quantum SuperLoader3, btw) due to some scsi resets, 2 of my daily pool tapes have been flagged as Full even though they are virtually empty (e.g. 4% fill state). Strange, I would expect them to be marked Error (but I have not run 2.4.4 for long)... in bat i see that i could change the volum status to anything, even Append, but i want to make sure that this is a good idea. You may want to look at: http://www.bacula.org/5.0.x-manuals/en/problems/problems/Tips_Suggestions.html#SECTION0036 (or http://tinyurl.com/33jhblm if your MUA breaks the link) There may be issue with number of files written (which would lead to tape being marked as error again when bacula tries to write next job to it) -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] File volumes doesnt reach Maximun volumes
On Tue, Apr 27, 2010 at 11:36:34AM +0200, Jorge Cabello wrote: I'm using 3 pools: full, differential and incremental. All of the with: Volume Retention = 1 month RecycleOldestVolume = yes Full Pool (1st of each month): Maximum volumes = 2 Differential Pool (15th of each month):Maximum volumes = 2 Incremental Pool (not 1st or 15th): Maximun Volumes = 45 My problem is that the incremental pool does not grow above 30 volumes. As I have read the number of volumes should be increasing till Maximum volumes limit is reached, right? If I'm correct, it is probably due to volume retention of 1 month - after one month, the oldest volume will be recycled. It will never go *above* 45, even if you increase retention (in that case it will stop and ask operator to insert/label new empty volumes) Note that retention does not directly influence data on tape, but just the catalog database; so you could still bscan(8) old tapes and get data back in the catalog - it is only when those volumes finally get used again that the old data will be destroyed. -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] File volumes doesnt reach Maximun volumes
On Tue, Apr 27, 2010 at 12:39:58PM +0200, Jorge Cabello wrote: I thought new files were created until Maximum volumes limit was reached. They were. But the old files were at the same time deleted according to retention period, so you you maximum volume limit was never actually reached. -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Ubuntu Lucid
On Tue, Apr 27, 2010 at 12:27:11PM +0200, Uwe Schuerkamp wrote: Hi folks, looks like lucid comes with bacula v5 which seemingly is unable to talk to our server running bacula 2.4.x: Your other clients are all 2.4.x I guess. following that, there's a client rejected hello command message on the server. Is there any way to turn on 2.x client compatibility in a 5.0.1 client? No. So your options are either to downgrade the client on Lucid (maybe there can be found a package for 2.4.x version ? If not, you may need to build it yourself) Or you can upgrade your director and SDs (it is usually possible to have a newer directory/SD talk to older FD client; but not the other way around. However 5.0 to 2.4 might be quite a stretch). -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Upgrade path from 2.4.x to 5.0.x
On Tue, Apr 27, 2010 at 02:15:35PM +0200, Foo wrote: On Mon, 26 Apr 2010 15:57:53 +0200, Matija Nalis mnalis+bac...@carnet.hr Also, can I expect problems if I added some extra indexes for speed and whatever tables/indexes were added by webacula? yes, you can expect problems :) For example, http://wiki.bacula.org/doku.php?id=faq#restore_takes_a_long_time_to_retrieve_sql_results_from_mysql_catalog http://wiki.bacula.org/doku.php?id=faq#jobs_with_accurate_filesets_take_forever_deprecated http://bugs.bacula.org/view.php?id=1472 Thanks, I'll see what I can remove. Can't access the last link though. You may need to create account in Mantis for that last one (there is a link below login form to which you get redirected) The indexes did help a lot with dbcheck though, without them it takes days to complete instead of minutes, will this speed up in 3.x/5.x or has dbcheck been updated, is it not necessary anymore, or is this something that needs to be fixed in a future release? The issue is in MySQL, as its algorithm sometimes chooses terribly wrong index, so SELECT times go up easily from 5 minutes to more than 20 hours. If they make MySQL choose good index every time, the problem will go away. Nowdays it is recommended to go with PostgreSQL instead AFAICT. As for dbcheck(8), if I recall correctly if indexes are not present, it will create indexes on start and drop them on finish (after asking). You can automate it with: yes no | dbcheck -b -f -c /etc/bacula/bacula-dir.conf if you do not want it to create indexes... (if you want it to create indexes on the fly, you modify that... probably something like yes yes instead of yes no, but I haven't checked it) I have the following indexes on File currently: Non_unique Seq_in_index || Key_name | | | Column_name | Cardinality |++ | +-+ 0| PRIMARY| 1 | FileId | 6387461 1| JobId | 1 | JobId |2750 1| JobId | 2 | PathId | 638746 1| JobId | 3 | FilenameId | 6387461 1| File_JobId_idx | 1 | JobId |2750 1| File_PathId_idx| 1 | PathId | 206047 1| File_FilenameId_idx| 1 | FilenameId | 912494 1| File_FilenameId_PathId_idx | 1 | FilenameId | 912494 1| File_FilenameId_PathId_idx | 2 | PathId | 2129153 Collation are all A, Sub_part, Packed are all NULL, Null and Comment are all empty, Index_type is all BTREE You need to retaint only primary key, and two indexes: one on column JobId and one on composite JobId,PathId,FilenameId. All other indexes should be dropped on MySQL table File. Which of these can be deleted exactly? (and how, I'm not a DBA :) drop index File_PathId_idx on File; drop index File_FilenameId_idx on File; drop index File_FilenameId_PathId_idx on File; I'm not sure about other tables; if you modified Job table that might also slow you down, but probably much less (it needs only primary key, and index on Name). -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Upgrade path from 2.4.x to 5.0.x
On Mon, Apr 26, 2010 at 03:31:21PM +0200, Foo wrote: which MySQL update scripts from which archives do I need to run to go from 2.4.4 to 5.0.1, can I skip minor revisions and just run the ones from 3.0.3 and 5.0.1 or are there other db changes between say 3.0.0 and 3.0.3a? Didn't try that, upgraded one by one Also, can I expect problems if I added some extra indexes for speed and whatever tables/indexes were added by webacula? yes, you can expect problems :) For example, http://wiki.bacula.org/doku.php?id=faq#restore_takes_a_long_time_to_retrieve_sql_results_from_mysql_catalog http://wiki.bacula.org/doku.php?id=faq#jobs_with_accurate_filesets_take_forever_deprecated http://bugs.bacula.org/view.php?id=1472 BTW, maybe I missed something, but whatever happened to 4.x? http://www.bacula.org/en/?page=news See part starting with: You might be wondering why this release jumps from 3.0.x to 5.0.0 thus skipping version 4.0.0 -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Backup sshfs mount point
On Mon, Apr 26, 2010 at 09:38:02AM -0300, Sergio Belkin wrote: It say differential because I've run that job before but ssh was unmounted so first backup was empty. So that was probably the problem. For incremental/differental to work correctly, bacula expects that the new/modified files will have their mtime timestamps updated on creation/modification, so it will know what files have changed since full backup. So differential backup will just look for files that have been modified since the timestamp of last backups (like find -mtime...) In your situtation, when you mounted sshfs, it had *new* files which had mtime *older* than last time of full backup, so bacula decided not to back up such files. So in short; you must run full backup when sshfs is mounted, only then will new differential and incremental backups work correctly (alternatively, you could touch all the files, but that is probably more trouble than it's worth). -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Full and Incremental backup to different location
On Thu, Apr 22, 2010 at 12:06:12PM -0400, Phil Stracchino wrote:
That's looking much better. One suggestion, though - Move your Pool
overrides from the Schedule to the Job or JobDefs, like this:
Schedule {
Name = Default
Run = Level=Full 1st sun at 2:05
Run = Level=Incremental mon-fri at 2:05
}
JobDefs {
[...]
Pool = File
Incremental Pool = File
Differential Pool = File
Full Pool = Tape
}
Pool overrides in the Schedule resource have been deprecated (but, for
backward compatibility, not removed) because they do not work properly
when a Job is promoted to a higher level because of a missing or failed
One somewhat unrelated question; is it possible to do the following
using the new scheme ?
- backup daily incremental in pool Pool-Daily
- backup weekly differential in pool Pool-Daily
- backup feb-dec once a month full in pool Pool-Monthly
- backup once a year (jan) full in pool Pool-Yearly
What we're trying to do is extend standard backup policy (daily=inc,
weekly=dif, monthly=full) with an full backup which would go once a
year in a separate pool (and would have a very long rentention like
15+ years, or even never expire but get larger and larger) in order a
have a very coarse historical archive (something like offline
www.archive.or )
Or is the old scheme with pool being specified in Schedule still the
only/prefered way to acomplish that ?
--
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Problems using bscan to put Files back in catalog ?
On Wed, Apr 21, 2010 at 08:31:19PM +0100, Martin Simmons wrote: Which all ended with info like (with different non-zero numbers of course) : Records added or updated in the catalog: 1 Media 1 Pool 2 Job 8992911 File So I guessed it should be ready to go. I started bacula again, run the restore -- but again I got dreaded For one or more of the JobIds selected, no files were found, so file selection is not possible error. What is the value in the Job.PurgedFiles column for those JobIds? If it is 1, then Bacula will ignore the file entries. Thanks Martin! That is indeed the case - Job 10553 (the Full backup from tape 783AHNL2) indeed has PurgedFiles=1 (all other Differental/Incremental jobs have 0). I can of course update it manually in mysql, but is that the correct way to go? And why did that happen, shouldn't bscan have updated Job.PurgedFiles column too (one would guess if you restore catalog information using bscan, you actually want it to be useable) ? -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Problems using bscan to put Files back in catalog ?
On Thu, Apr 22, 2010 at 11:04:15AM +0100, Martin Simmons wrote: On Thu, 22 Apr 2010 11:07:43 +0200, Matija Nalis said: That is indeed the case - Job 10553 (the Full backup from tape 783AHNL2) indeed has PurgedFiles=1 (all other Differental/Incremental jobs have 0). I can of course update it manually in mysql, but is that the correct way to go? Yes, in this case I think that it is best option. You could check that all files have been added to the File table. I.e. these two should be equal: select count(*) from file where jobid=10553; select jobfiles from job where jobid=10553; Yes, they are the same. So I did: UPDATE Job SET PurgedFiles=0 WHERE JobId = 10553; and the restore seems to be working now (or at least allowing me to mark the files; I hope they weill also restore normally) And why did that happen, shouldn't bscan have updated Job.PurgedFiles column too (one would guess if you restore catalog information using bscan, you actually want it to be useable) ? Yes, it looks like a bug in bscan. You could report it at http://bugs.bacula.org/. Done, reported at http://bugs.bacula.org/view.php?id=1558 Thanks again, Matija -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore hanging on Building directory tree
On Tue, Apr 20, 2010 at 01:09:45PM -0600, Jason Hill wrote: I read through that, and looked at my indexes on the File table and it doesn't appear as if I have superfluous indexes. The doc mentions: you need to drop all the indexes from File table except the primary key and indexes on (`JobId`,`PathId`,`FilenameId`) and (`JobId`) However, unless I am missing something, those are all my indexes: mysql show index from File; +---++--+--+-+---+-+--++--++-+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | +---++--+--+-+---+-+--++--++-+ | File | 0 | PRIMARY |1 | FileId | A | 9241817 | NULL | NULL | | BTREE | | | File | 1 | JobId|1 | JobId | A | 9241817 | NULL | NULL | | BTREE | | | File | 1 | JobId_2 |1 | JobId | A | 9241817 | NULL | NULL | | BTREE | | | File | 1 | JobId_2 |2 | PathId | A | 9241817 | NULL | NULL | | BTREE | | | File | 1 | JobId_2 |3 | FilenameId | A | 9241817 | NULL | NULL | | BTREE | | +---++--+--+-+---+-+--++--++-+ 5 rows in set (0.00 sec) Yes, that sounds like correct indexes for File table, but Cardinality is strange. You should probably run analyze table (if InnoDB) or optimize table (if MyISAM). Also refering to your original mail, you should note that as MySQL im multithreaded, strace(8) won't tell you much unless you run it with '-f'. The best way to check if there are queries running (and in what state) is to run show processlist mysql query. If the slowness is due to mysql queries running, you can try playing with mysql variables (some are mentioned on wiki) or you should switch to PostgreSQL which is (or so it seems) much better optimized for bacula. -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fatal error: backup.c:892 Network send error to SD. ERR=Connection reset by peer
On Sun, Apr 18, 2010 at 11:46:33AM -0500, Jon Schewe wrote: http://wiki.bacula.org/doku.php?id=faq#my_backup_starts_but_dies_after_a_while_with_connection_reset_by_peer_error [1] It actually tries that at one point in src/lib/bsock.c if TCP_KEEPIDLE support is detected, but it fails to detect it properly because netinet/tcp.h is not included. However, even after fixing that (and missing semicolon in 'int opt = heart_beat' line), it still doesn't look like it sets TCP_KEEPIDLE correctly on FD-SD connection, so maybe this codepath is not used there. Anyway I gave up debugging there and just set the system defaults. But I just though I'd mention that in case someone else wants to continue chasing the bug. Hmm, this sounds like a bug that should be fixed and once it is fixed may remove a bunch of problems with firewalls. FYI, I've put up a patch which fixes current support on bacula-devel mailing list. That support could be extended (as not all parts of bacula use that function), but it might be enough. If someone is willing to try it, let me (or better, the whole list) know how it fares and if it fixes the timeouts without the user needing to resort to changing systems defaults. -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Problems using bscan to put Files back in catalog ?
Hi, I had a long retention on jobs (3 years) and a short on files (1 month). I usually run full backups monthly, but few of those failed (due to time restrictions and other issues). I also run Differential backups weekly, so all data seems to be on tapes - it just that the File records expired. So when I tried to restore files using option 6: Select backup for a client before a specified time bacula greeted me with: ++---+---+-+-++ | JobId | Level | JobFiles | JobBytes| StartTime | VolumeName | ++---+---+-+-++ | 10,553 | F | 4,652,454 | 116,177,817,964 | 2009-11-14 11:36:46 | 783AHNL2 | | 31,236 | D | 1,680,724 | 32,677,953,918 | 2010-04-10 17:15:01 | 661AHNL2 | | 31,387 | I |37,038 | 7,804,308,239 | 2010-04-11 17:15:01 | 661AHNL2 | | 31,543 | I |71,957 | 10,034,754,063 | 2010-04-12 17:15:00 | 674AHNL2 | | 31,838 | I | 125,463 | 12,832,108,210 | 2010-04-14 17:15:00 | 674AHNL2 | | 31,992 | I | 110,707 | 11,871,570,048 | 2010-04-15 17:15:01 | 674AHNL2 | ++---+---+-+-++ You have selected the following JobIds: 10553,31236,31387,31543,31838,31992 Building directory tree for JobId(s) 10553,31236,31387,31543,31838,31992 ... ++ For one or more of the JobIds selected, no files were found, so file selection is not possible. Most likely your retention policy pruned the files. Do you want to restore all the files? (yes|no): no OK, everything is understandable so far. So, I recon I'd restore File entries by bscaning tapes (which I never used before). So I've executed (with right tapes loaded in right drives according to bscan outputs) the following commands to fully rescan all tapes involved back into catalog: bscan -s -S -v -V 783AHNL2 drv2_lto2 bscan -s -S -v -V 661AHNL2 drv3_lto2 bscan -s -S -v -V 674AHNL2 drv4_lto2 Which all ended with info like (with different non-zero numbers of course) : Records added or updated in the catalog: 1 Media 1 Pool 2 Job 8992911 File So I guessed it should be ready to go. I started bacula again, run the restore -- but again I got dreaded For one or more of the JobIds selected, no files were found, so file selection is not possible error. In mysql I executed a series of queries like (one for each job listed): select count(*) from File where JobId=10553; And they all returned non-zero results (showing millions or hundreds of files per job). So I would expect that the Files in catalog should be available, but it seems bconsole disagrees with me. Am I doing something wrong, or am I hitting a bug ? Any hints ? (I'd rescan all the volumes just to be sure that there is no catalog info missing, but it would take way too long here - weeks at best) Bacula is latest GIT Branch-5.0 (5.0.1 with some bugfixes) P.S. I know bconsole will then ask me to Regexp matching files to restore?, but that is ALSO not working for me (it seems that bacula tries to restore whole client and .bsr looks strange, but that looks like another bug, which I'm filing in bugs.bacula.org after posting this) -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula don't ignore files in section Exclude
On Fri, Apr 16, 2010 at 05:12:17PM -0300, Daniel wrote:
This problem is the continuation of my last e-mail.
I am trying to configurate the bacula to make the backup of my desktop.
I want to make copy of my /home/user folder, but I wanto to ignore the
directores .gvfs and .VirtualBox.
On the bacula-dir, the configuration is like this:
FileSet {
Name = User File Set
Include {
Options {
# Exclude = yes
}
File = /home/user/
}
Exclude {
File = /home/user/.VirtualBox/
File = /home/user/.gvfs/
}
}
The server make the backup, but don't ignore the files in section Exclude {}
Well, it should. Are you sure you restarted/reloaded your director
after changing ?
If I uncomment the line Exclude = yes, then the server don't backup
any folder... I tried reading the manual and, from what I understand, I
should put this line in the configuration, but, if I do that, the server
does not copy anything.
No, you should not put Exclude=yes, the original answer was correct
in that. It works here just like that. You could try removing the
trailing slashes though, maybe those are confusing bacula ?
There are generally 3 ways to exclude files in bacula:
1) Include {
File = /home
}
Exclude {
File = /home/blah
}
Those will backup all of /home except /home/blah.
That is the one that you are trying above, but which misteriously
doesn't work although it should.
2) Include {
Options {
wildfile = *.avi
exclude = yes
}
File = /home
}
This is somewhat confusing, but will backup all of /home but without
those files matching wildcard *.avi
3) (in newer bacula releases only)
Include {
File = /home
}
ExcludeDirContaining = .do_not_backup
This will backup all of the /home, except directories containing
file .do_not_backup (and all of it's subdirectories).
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Restore hanging on Building directory tree
On Tue, Apr 20, 2010 at 10:38:27AM -0600, Jason Hill wrote: I am at my wit's end on this one, and I am hoping someone has had this problem already. I am running the following: RHEL 5.5 (client, or workstation) Bacula 5.0.1 MySQL 5.1 My problem is that when I attempt a restore for a large backup bacula hangs on the Building directory tree Yes. See: http://wiki.bacula.org/doku.php?id=faq#restore_takes_a_long_time_to_retrieve_sql_results_from_mysql_catalog (or http://tinyurl.com/y6334ht if your MUA breaks the link) -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fatal error: backup.c:892 Network send error to SD. ERR=Connection reset by peer
On Mon, Apr 12, 2010 at 03:59:49PM -0500, Jon Schewe wrote: On 4/12/10 9:40 AM, Matija Nalis wrote: It is especially problem with bigger databases and MySQL instead of PostgreSQL, see http://bugs.bacula.org/view.php?id=1472, where it can take even several hours! (note that while it talks about restore speed, it is also related to accurate backups which employ similar SQL queries) Must be what it is then. I've been thinking about switching to postgres, but haven't because the opensuse packages for bacula are only for mysql. This may motivate me more. You should probably switch soon, before you get to like your database,,, Exporting bacula mysql tables for import in PostgreSQL can be very painful and problematic; it is much better to just drop the database and create fresh one. The backup finished, so it seems that in version 3.0.3 bacula does NOT set the socket option SO_KEEPALIVE. Hmm, yeah, I've check the code casually, and it indeed looks like the heartbeats are not setting SO_KEEPALIVE timeouts (note that it does set SO_KEEPALIVE on the socket, otherwise the advice above wouldn't work -- it just doesn't do TCP_KEEPIDLE on that[1] to specify user-defined timeouts and instead uses system defaults). The heartbeats look like are doing other things though (application-level, not socket-level), but as you saw they are not perfect for fixing network idleness problems - and so you also MUST set system defaults. I've updated the FAQ at: http://wiki.bacula.org/doku.php?id=faq#my_backup_starts_but_dies_after_a_while_with_connection_reset_by_peer_error [1] It actually tries that at one point in src/lib/bsock.c if TCP_KEEPIDLE support is detected, but it fails to detect it properly because netinet/tcp.h is not included. However, even after fixing that (and missing semicolon in 'int opt = heart_beat' line), it still doesn't look like it sets TCP_KEEPIDLE correctly on FD-SD connection, so maybe this codepath is not used there. Anyway I gave up debugging there and just set the system defaults. But I just though I'd mention that in case someone else wants to continue chasing the bug. -- Matija Nalis Odjel racunalno-informacijskih sustava i servisa Hrvatska akademska i istrazivacka mreza - CARNet Josipa Marohnica 5, 1 Zagreb tel. +385 1 6661 616, fax. +385 1 6661 766 www.CARNet.hr -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula 5.0.1 and db issues - please, share your experience
On Wed, Apr 14, 2010 at 03:29:29PM +0200, Koldo Santisteban wrote: Thanks for your answers. Stephen, do you use bconsole or bat? Perhaps the issue is only on bat. I recognize that i only use bweb and bat (on windows). For what it's worth, I used bconsole, and it was extremly slow (20+ hours) under 5.0.x, while it was relatively fast (less than 10 minutes) under 3.0.3 Regarding your comments Stephen, my bacula server is smaller than yours, but my catalog size was 400 Mb (Baculas is working since 2 months ago). I don´t tune my database, but with 3.0.3 version thats wasn´t neccesary. Wich parameters are recommending to tune? This info i think that is very useful for people with the same issue like me... i see that some people says that the better way is creating new indexes(someone says that this is the worst option), others say to custom mysql parameters...but i can´t find any official info, and, at less in my case, i don´t have enough time(and knowledge) to testing bacula with some new indexes, or customizing mysql/postgre... I miss this offcial info... I do not see how you could expect to get official info before there is official opinion formed ? There is a speed bug in bacula 5.0.x series only. It is reported, and you can track the bug status at http://bugs.bacula.org/view.php?id=1472 Once the exact problem and official resolution are known, then bug report will be closed and speed bug fixed (or official recommendations given in bug report at URL above). So you could either: 1) put up with slowness until the bug 1472 is resolved and then upgrade and/or follow official recommendations. Perhaps you can even help with it by providing additional information and/or running tests ? 2) downgrade to 3.0.3 which does not have this bug 3) spend some time reading the experiences of others and trying to adopt them to your case 4) give up -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Problems getting restore to work
On Mon, Apr 12, 2010 at 02:15:05PM -0700, Jerry Lowry wrote: Hi, I am still tweaking a new installation of bacula 5.0.1 on Centos 5.4. The backups work fine but I am trying to get the restore to work and I keep getting the following errors: Volume hardware-0014 to file:block 7:2933114700. 10-Apr 12:51 swift-fd JobId 118: Error: restore.c:1133 Write error on /backup0/bacula-restores/home/hardware/pdf/altera/quartus/81_linux/81_nios2eds_linux.tar: No space left on device The volume is a disk drive and I am trying to restore it to a dedicated restore directory on a different disk. I have checked the config files It looks to me like you've run out of disk space (Disk full) on your /backup0/bacula-restores disk -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] bacula 5.0.1 and db issues - please, share your experience
On Tue, Apr 13, 2010 at 02:42:15PM +0200, Koldo Santisteban wrote: Hello I am working with bacula 5.0.1. On first stage i setup the server with bacula 5.0.1 and Mysql, but, when i need to restore i have found that the bulid tree process take 10-12 hours (or more). I have read all about this Could you share more info on your environment ? MySQL version (and specific settings like MyISAM/InnoDB buffers etc), hardware specs, number of records in various tables (count(*) in File, Job, Path, Filename...) http://wiki.bacula.org/doku.php?id=faq#jobs_with_accurate_filesets_take_forever_deprecated http://wiki.bacula.org/doku.php?id=faq#restore_takes_a_long_time_to_retrieve_sql_results_from_mysql_catalog If you look into: http://bugs.bacula.org/view.php?id=1472 it seems that the issue is only with bacula 5 series; bacula 3.0.3 and lower are *much* faster (less then 10 minutes instead of 20+ hours). The bug is being looked into currently. Apart from this bug, there is also a discussion on this list with subject VirtualFull mysql query blocks other jobs for a?long time which tackles wider SQL issues... issues and i can see that no exists any magic solution. In order to solve it, i have migrate from mysql to postgre, but i can see the same symtopms. Again, more data ? PostgreSQL version ? is it any faster (it should be about 600% faster on same datasets according to links above) or not ? Perhaps it works better, but, in my opnion, this is not serios on a production environment. I agree. Apart from downgrading (which could be quite ok for you, but it was not possible for us for other issues, like bug 1528) you could reduce your file retention (it helps greatly) Anyway, if you can, downgrade to 3.0.3; if you can't, look into bug 1472 and share your info. If it´s possible, i will appreciate it people share their experience with bacula last version and this kind of issues. A couple of month ago i finished to deploy bacula on my environment, but now, i am considering to go back all. see the http://bugs.bacula.org/view.php?id=1472 for my info, and also for things that fix it (like undefining new_db_get_file_list in sql_get.c) -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fatal error: backup.c:892 Network send error to SD. ERR=Connection reset by peer
On Fri, Apr 09, 2010 at 07:30:19PM -0500, Jon Schewe wrote:
I have heartbeat intervals set at the following:
bacula-dir.conf:
client {
Heartbeat interval = 15 Seconds
}
storage {
Heartbeat interval = 1 minutes
}
bacula-sd.conf
storage {
Heartbeat interval = 1 minute
}
bacula-fd.conf
FileDaemon {
Heartbeat Interval = 5 seconds
}
Strange. Are you running GNU/Linux system on all the machines
(FD, SD, DIR) ? IIRC, it might not be supported on other systems,
and/or it may need additional tuning on them.
I've updated the docs at http://tinyurl.com/y8wapdu
--
Matija Nalis
Odjel racunalno-informacijskih sustava i servisa
Hrvatska akademska i istrazivacka mreza - CARNet
Josipa Marohnica 5, 1 Zagreb
tel. +385 1 6661 616, fax. +385 1 6661 766
www.CARNet.hr
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fatal error: backup.c:892 Network send error to SD. ERR=Connection reset by peer
On Mon, Apr 12, 2010 at 05:41:51AM -0500, Jon Schewe wrote: Strange. Are you running GNU/Linux system on all the machines (FD, SD, DIR) ? IIRC, it might not be supported on other systems, and/or it may need additional tuning on them. I'm running opensuse Linux for the director and storage daemon and Debian Linux for the file daemon. that is strange... can you check what are your default SO_KEEPALIVE values with: grep '' /proc/sys/net/ipv4/tcp_keepalive_* and what bacula is using for running connections - start backup first, then check if keepalive is enabled (and with what timers) with: netstat -to -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fatal error: backup.c:892 Network send error to SD. ERR=Connection reset by peer
On Mon, Apr 12, 2010 at 07:59:53AM -0500, Jon Schewe wrote: /proc/sys/net/ipv4/tcp_keepalive_time:7200 netstat -to Client: tcp0 0 client:9102 server:54043 ESTABLISHED keepalive (7196.36/0/0) That's strange. It should've been the timeouts you specified in config files, not 7200 seconds (two hours) which is system default. It looks like bacula does not use TCP_KEEPIDLE setsockopt(2) on your system. You might want to report a bug on http://bugs.bacula.org/ IMHO, it should work there. Or if not, it should probably throw a warning if you try to use it and it is not supported or fails. Apart from fixing bacula, you can override system default, for example (on both server and client) do : echo 60 /proc/sys/net/ipv4/tcp_keepalive_time (or edit /etc/sysctl.d/* or /etc/sysctl.conf to retain value across reboots). Can you try what netstat -to says after you lower that limit and rerun backups ? If netstat -to then reports smaller timers (60 or less), than it should fix your problem, so you can try turning accurate back to yes. Does that help ? -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fatal error: backup.c:892 Network send error to SD. ERR=Connection reset by peer
On Mon, Apr 12, 2010 at 09:23:51AM -0500, Jon Schewe wrote:
On 4/12/10 9:00 AM, Matija Nalis wrote:
(SO_KEEPALIVE will work even with only one side of connection having
it enabled).
So I should only need the heartbeat on that client's setup as well,
right? Getting rid of extra heart beats would be nice.
Yes, it should be enough. Note that there is no real need to get rid
of extra heartbeats, they are not really expensive (so biggest gain
is cleaner config files).
Good, let us know how it fares.
It seems to be running, but I've run into a problem with bconsole. Once
I started the job, if I run bconsole and then status dir, the console
hangs. If I strace the bconsole process it's stuck in a select call.
strace -p 18452
Process 18452 attached - interrupt to quit
select(4, [3], NULL, NULL, {9, 461287}) = 0 (Timeout)
read(3, 0x655d80, 5)= -1 EAGAIN (Resource
temporarily unavailable)
That should not be related to SO_KEEPALIVE - it should be completly
transparent to the applications if the network is working (and even
when it is not working, it should differ only in always terminating
the connection instead of sometimes terminating connection and
sometimes hanging idefinitely).
Anyway, it may be few issues with directory hanging. Most common is
you are too eager. For example, is SQL server is busy, status dir
will hang until it completes.
It is especially problem with bigger databases and MySQL instead of
PostgreSQL, see http://bugs.bacula.org/view.php?id=1472, where it can
take even several hours! (note that while it talks about restore
speed, it is also related to accurate backups which employ similar
SQL queries)
You can check for this with show processlist in MySQL (if you are
running MySQL for database, of course) if that is the case (or simply
wait).
Or you might be unlucky enough to hit a real director bug in 5.0.1,
see http://bugs.bacula.org/view.php?id=1528, but that is unlikely.
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] KeepAlive parameter for Control connection between FD and DIR?
On Thu, Apr 08, 2010 at 11:12:06AM +0200, M. Sébastien LELIEVRE wrote: Running Bacula 3.0.3, Is there a KeepAlive parameter for the Control Connection between FileDaemon and Director during a Backup/Restore operation? Yes, see http://tinyurl.com/y8wapdu that is http://wiki.bacula.org/doku.php?id=faq#my_backup_starts_but_dies_after_a_while_with_connection_reset_by_peer_error -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fatal error: backup.c:892 Network send error to SD. ERR=Connection reset by peer
On Wed, Apr 07, 2010 at 02:15:14PM +0100, Prashant Ramhit wrote: b06-Apr 12:54 client-fd JobId 299: Fatal error: backup.c:892 Network send error to SD. ERR=Connection reset by peer/b/small/pre Is it possible to tell me how to enable more debug on client and storage so that i can find more clues to this issue.br You can use -d number to increase debug level; but in your case it should be pretty clear -- something (usually router or firewall) between SD and FD (or even local firewalls on themselves) is killing TCP connection (usually because it was idle for too long). See http://tinyurl.com/y8wapdu it adding Heartbeat Interval helps you. -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Max Run Time exceede with 0s run time!
On Mon, Apr 05, 2010 at 12:46:25PM +0200, Harald Schmalzbauer wrote: Absurdly canceled job 47: Fatal error: Max run time exceeded. Job canceled. Scheduled time: 04-Apr-2010 21:01:03 Start time: 04-Apr-2010 21:39:41 End time: 04-Apr-2010 21:39:41 Elapsed time: 0 secs Hm, yeah, not much sense. Was that incremental job or ? It looks like it has 'Incremental Max Run Sched Time' instead of 'Incremental Max Run Time'... Did you try increasing it to see if that's really what is causing problems ? All other timeouts are bigger than one hour so really shouldn't be the problem. Here's my conf regarding max times: Max Start Delay = 14400 # 4h to wait after scheduled start Max Run Time = 1800 # Half an hour to run after beeing really started Incremental Max Run Time = 900 # 15 Minutes for incrementals after beeing started Max Run Sched Time = 36000 # 10 hours to wait to start job as planned Max Wait Time = 7200 # 2h to wait for resources after job really started Which version of bacula is that ? There were bugs not too far ago where 'Max Wait Time' wrongly acted like 'Max Run Time'; maybe it had similar problems with other related directives too. See http://www.bacula.org/en/dev-manual/New_Features.html -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Need help debugging SD crash
On Tue, Apr 06, 2010 at 08:40:20AM -0600, Robert LeBlanc wrote: I've tried in the past to do exactly this. Bacula will usually spit out an error that the tape could not be moved or in rarer situations say the drive is not there. I then shut down bacula-sd and try to run the mt eject command I I usually get back about ten lines that describe the error, but it does really make sense. Sometimes the drive doesn't appear as a device on the system any more. As far as the tape library, the Overland Neo 8000 most of [...] the time says soft removal error on the screen and will keep saying that if I try to have the library remove it. There is no easy way to get to the hardware eject button as the library is fully enclosed. It looks like the drive gets confused if it gets commands too fast (and/or while it is still processing previous commands)... Anyway, it looks like problem outside bacula (probably either the kernel, or a drive firmware, or both are at error). drives and our LTO-4 drive. The only thing that I can think of is that bacula is trying to take some shortcuts (issuing a command to move the tape and expecting the tape library to correctly rewind the tape, eject and then move it and maybe bacula is not quiet letting go of the drive fast enough and there gets a deadlock between the drive controlled by Bacula and the library trying to control it), or there is a kernel/driver problem. Only thing bacula does is execute mtx-changer script; it is the scripts responsibility to does everything needed for your drive / changer combination. The default script is usually good, but you may need to tailor it for your needs (if it needs manual rewind before offline, or things like that). I've set the offline=1 in mtx-changer.conf and that seems to help a little, I've still encountered some drive unmouting issues, but nothing that bacula hasn't been able to recover from on it's own or with very little manual intervention. I run mine (IBM3584) with: offline=1 offline_sleep=2 load_sleep=20 I do recall having sporadic issues with load_sleep of just a 2-3 seconds, so I've put it to 20 to allow the drive to settle fully before issuing a bunch of mt status to it in wait_for_drive(). I was pretty sure the messages were informational, I'm glad that someone can confirm that. I'll keep working on the problem to see what I can come up with. If there is a better way to tell Bacula to be stupid slow with unmount and mount requests, that may help me find where in the process things are getting hung up. Well, you can put (in 5.0.1 at least) offline_sleep and load_sleep to 30 seconds or more for example, that might help if drive is getting confused while receiving commands too fast. On older versions (3.0.x or 2.4 ?) you can edit the mtx-changer shell script itself, IIRC it had commented out sleep statements at right places already... -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Poor Performance
On Thu, Apr 08, 2010 at 10:55:55AM -0400, Il Neofita wrote: Any Idea? What I should check On Wed, Apr 7, 2010 at 1:04 PM, Il Neofita asteriskm...@gmail.com wrote: Thank you for all the replies The bottleneck is the network or at lest something in the network and yes I have an autoloader LTO4 If I use NC with a file of 10M I need 8 seconds to transfer If I use scp I need around 2 seconds The same file without compression it sounds very strange, the scp should be slower (or at best same speed) as it needs additional step of encrypting data. It could only be faster if it is (maybe by default) doing ssh compression of data on the fly, so it has less data to transfer. Also, are you sure you're correctly droping caches before each test ? Because if you run nc test first, and scp seconds, scp will have advantage of not having to access disk at all to read the file (as it will probably be in cache). Can you show exact commands and outputs you get to conduct those tests (use script(1) or something to capture it) ? you could also try using iperf(1) to measure network bandwidth between SD and FD. -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] PATCH: add AES-ECB support to Bacula
On Wed, Apr 07, 2010 at 06:52:40PM +0800, Craig Ringer wrote: Bacula currently only uses the AES CBC cypher mode. This cypher can't be effectively parallelized because block n+1 depends on block n. The AES ECB mode was developed to address that limitation. Support for Actually, ECB was developed as it is one of the building blocks for CBC (and other schemes). It is not secure at all by itself. You may however use it to build other schemes which allow for (some/much) parallelisation and/or precomputing (like OFB, CTR etc), and which are way more secure then ECB itself (but still worse than CBC probably) Right now, this patch gains you little or nothing. It also costs little or nothing, and probably shouldn't actually be applied at present. I'm putting it up here to keep it and the notes about crypto in the archives in case I don't have any luck following it up with an actually parallel implementation and others are looking into it later. The next step is to try to spawn worker threads to encrypt chunks in parallel. Hopefully this will be possible with OpenSSL... I really think doing pure ECB is a veeery bad way to go, as it lulls the user in false sense of security without actually providing security. For examples, see nice pictures at http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation#Electronic_codebook_.28ECB.29 Sometimes you can even tell by the plain eye view of ECB-encrypted picture what the original picture was (See wikipedia)! Not to mention many various crypto attacks against its use in our contexts. What I would actually propose is filter support. Then a FD could pass a data stream via pipe to the standard external filter program (which reads from stdin and writes to stdout), and a user can put any third party utility there that s/he wants - from multi-core compression ones (like pbzip2) to encyption ones (like aespipe and ccyrpt and similar, or others that employ paralellisation or special hardware support), or a combination of those. Current encryption and compressions can be provided as separate applications included in default bacula distribution. What we might additionally need though, is some way to put some identifier as to which filter was used for which job, so one can retain compatibility with older backups if one decides to change the crypto/compression) I know just enough crypto to know that without knowing way much more I'm likely to make terrible mistakes (and we all still remember Debian fixing OpenSSL security fiasco, don't we?). Raw ECB sounds just like one of those mistakes. By using filters, we can have both maximum flexibility, and actually have the more secure encryption (by recommending the proven encryption tools to use instead of reinventing the wheel [probably badly]) -- Matija Nalis Odjel racunalno-informacijskih sustava i servisa Hrvatska akademska i istrazivacka mreza - CARNet Josipa Marohnica 5, 1 Zagreb tel. +385 1 6661 616, fax. +385 1 6661 766 www.CARNet.hr -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Poor Performance
On Wed, Apr 07, 2010 at 08:41:41AM -0400, Il Neofita wrote: Hi I have very poor performance when the agent send the spool to the bacula server First the obvious question: are the server or client doing anything else while the backup is running? I currently use a RAID5 SAS 15K, 1GB Ethernet hdparm -t /dev/sda2 is that hdparm run on server or client ? /dev/sda2: Timing buffered disk reads: 692 MB in 3.00 seconds = 230.41 MB/sec that is raw buffered linear burst read speed... seeks can easily slow it down for two orders of magnitude or more -- and filesystems do require seeks. And if there are going to be lots of seeks or less depends on many factors (like what FS are you using, the amount of file fragmentation, are we talking about one big file or many many small files etc). Any idea what I can check? 1) speed of writing on SD (server) side sync ; echo 3 /proc/sys/vm/drop_caches time (dd if=/dev/zero of=your_spool_file bs=1k count=100; sync) # fix location of your_spool_file, and size of spool 100*1k # all examples assume that your dataset is at least 1GB large # you should probably use spool size at least 4 times the amount # of RAM to get accurate results. 2) speed of reading on FD (client) side sync ; echo 3 /proc/sys/vm/drop_caches time tar zc / | dd of=/dev/zero bs=1k count=100 # omit 'z' if you're not doing compression, change '/' to match your fileset # note that if you're using encryption it will slow down things considerably 3) network speed between FD and SD # on server nc -l -p 12345 /dev/null # on client time (dd if=/dev/zero bs=1k count=100 | nc -w1 server_ip 12345) # server_ip is your server IP address 4) combination of the above, should be similar to the bacula speed # on server sync ; echo 3 /proc/sys/vm/drop_caches nc -l -p 12345 /dev/zero # on client sync ; echo 3 /proc/sys/vm/drop_caches time (tar zc / | dd of=/dev/zero bs=1k count=100 | nc -w1 server_ip 12345 ; sync) those should get you some idea about what part is bottleneck... -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] client rejected Hello command
On Tue, Apr 06, 2010 at 05:52:54PM -0400, Roland Roberts wrote: When the client is run in the foreground in verbose mode, I get this from the client: archos-fd: cram-md5.c:73-0 send: auth cram-md5 1018903137.1269886...@archos-fd ssl=0 archos-fd: cram-md5.c:152-0 sending resp to challenge: a9/LnX1KKQ/Gj6+qd++t1B that is strange. Looking at the code, if you get this far, one of the 3 things should happen: a) timeout or socket error, in which case it would have printed Receive chanllenge response failed. ERR=%s, or b) authorization succeeds (it receives 1000 OK auth) in which case nothing is printed and cram_md5_respond returns true, or c) authorization fails, and Received bad response: %s is printed As nothing is printed, one could only assume the authorization succeeded, but things stopped somewhere afterwards. Can you raise debug level (say, -d200) on both the director and FD (and maybe even on SD, just in case) and send output of that ? -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Need help debugging SD crash
On Sun, Apr 04, 2010 at 01:20:49PM -0600, Robert LeBlanc wrote: I'm having problems with our SD and tapes being locked in the drive occasionally. How does it manifest exactly ? bconsole umount command returns error, or remains in some state (check with status storage) ? Which state and/or error ? Have you tried shutting down bacula-sd and ejecting tape with mt eject and/or mt offline ? Do they succeed (and the drive ejects) or do they return error (and which one) ? Double check that bacula-sd is down before you try those (they won't work if bacula-sd is still having the drive open). And if mt(1) also fails, can you eject tape manually by using tape library eject function and/or pressing hardware eject button on the drive itself (depending on the library type...) ? If mt works but bacula-sd doesn't, than you can rule out hardware and kernel -- it is bacula problem (and usually status storage will show it -- it can happen sometimes if you have more than one drive that it deadlocks by waiting for a tape that is in the other drive). At first I thought this might be a problem with our tape library. That still looks like the most probable cause to me - like a drive in the library is having problems. We've had a similar issue with one of several LTO2 drives in our library; it would (sometimes) take the tape and refuse to give it back (on mt eject and even physical button touch). Needed power cycling and long (half a minute?) button press to make it give the tape back. After it happened third time (always the same drive) we kicked it out of the library. Other drives worked OK all the time. If the hardware button always works but software commands don't, it could be fiber cables and/or GBIC/SPF (which we refused to believe at one time because drives were always detected OK and worked, albeit sometimes much slower than normal, without any errors in kernel logs, and would also lock up). You can try cleaning tape also. Then I saw these errors in the syslog. I switched out the Qlogic FC adapter thinking that maybe it was just losing all the paths to the drive. AFAIR you would get different errors if it loses path completely (but it is possible for drive to behave erratically even if it doesn't lose path) I'm still getting the errors, so I'm not sure where the hangup is. I can't tell if it's a bug in the kernel module, mt or bacula. Can someone give me some pointers to narrowing this down? This has been happening for over a year and through several kernel and bacula versions. This is Debian Squeeze Linux lsddomainsd 2.6.32-trunk-686 #1 SMP Sun Jan 10 06:32:16 UTC 2010 i686 GNU/Linux The INFO: messages themselves are just normal feature of newer 2.6.x kernels, they are informational message only (See INFO:) that tells you some system call (like open(2) or write(2) or read(2)) is taking longer than 120 seconds to complete. They didn't exist in older kernels. It is there to catch problems with I/O schedulers and problematic hardware issues -- but sometime it needs to be increased for tape drives (it is quite possible for open(2) or lseek(2) on tape to have to rewind it, and that sometimes can take more than two minutes). you can raise the current kernel limit with: echo 300 /proc/sys/kernel/hung_task_timeout_secs or (to survive reboot) by putting: kernel.hung_task_timeout_secs=300 in /etc/sysctl.conf (or a file in /etc/sysctl.d directory) But as I say, those will not help your lockup problems, just make the spurious messages go away when they are to be expected. Try the other things in the mail to narrow the problem down to bacula, kernel or hardware. -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Simplified pools
On Tue, Apr 06, 2010 at 02:36:38PM +0800, Craig Ringer wrote:
I'm sure many people using disk-based storage find that, like me, they
have three pools defined for almost every job or at least class of job,
like this:
You could use standard include (@) feature of bacula configuration
files, for example:
in /etc/bacula/pool-default-values.conf:
Storage = SystemStorage
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Use Duration = 1 day
Recycle Oldest Volume = yes
and in /etc/bacula/bacula-dir.conf:
Pool {
@/etc/bacula/pool-default-values.conf
Name = SystemFullPool
Volume Retention = 2 months
LabelFormat = SystemFull-
}
Pool {
@/etc/bacula/pool-default-values.conf
Name = SystemDiffPool
Volume Retention = 2 weeks
LabelFormat = SystemDiff-
}
Pool {
@/etc/bacula/pool-default-values.conf
Name = SystemIncrPool
Volume Retention = 1 week
LabelFormat = SystemIncr-
}
Much more readable and easier to maintain than original config:
Pool {
Name = SystemFullPool
Storage = SystemStorage
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 2 months
Volume Use Duration = 1 day
Recycle Oldest Volume = yes
LabelFormat = SystemFull-
}
Pool {
Name = SystemDiffPool
Storage = SystemStorage
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 2 weeks
Volume Use Duration = 1 day
Recycle Oldest Volume = yes
LabelFormat = SystemDiff-
}
Pool {
Name = SystemIncrPool
Storage = SystemStorage
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 1 week
Volume Use Duration = 1 day
Recycle Oldest Volume = yes
LabelFormat = SystemIncr-
}
It is even more advanced than including single files, for example you
could use something like:
@|sh -c 'for f in /etc/bacula/clients.d/*.conf ; do echo @${f} ; done'
which would include all *.conf files in /etc/bacula/clients.d/
We use that to allow different admins to edit configurations only for
their clients...
Of course you can use any external command here; for example you
could call perl script using DBI which would generate all or part of
your config files from SQL database or for example from nagios
configuration files or whatever...
--
Matija Nalis
Odjel racunalno-informacijskih sustava i servisa
Hrvatska akademska i istrazivacka mreza - CARNet
Josipa Marohnica 5, 1 Zagreb
tel. +385 1 6661 616, fax. +385 1 6661 766
www.CARNet.hr
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Multiple drives in changer
On Thu, Apr 01, 2010 at 08:49:43AM -0600, Robert LeBlanc wrote: I have two LTO-3 drives in a changer and three LTO-3 pools. Earlier version of Bacula would try to use an empty drive before unloading a drive when a tape from a different pool was requested. I used to also be able to run in parallel migration jobs from two different pools at the same time. Since moving to 5.0.1, my second drive goes unused. Is there some change in the code that prevents this behavior? I think you need to set Prefer Mounted Volumes = no to get old behaviour (maximum parallelism). Note however that it might lead to deadlocks (or at least, it could in the past... don't know if that was fixed). -- Matija Nalis Odjel racunalno-informacijskih sustava i servisa Hrvatska akademska i istrazivacka mreza - CARNet Josipa Marohnica 5, 1 Zagreb tel. +385 1 6661 616, fax. +385 1 6661 766 www.CARNet.hr -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] DVD backup
On Thu, Apr 01, 2010 at 08:23:41AM -0700, Wes Hardaker wrote: The last time I was at a store and was staring at the external USB hard-disks while holding a stack of re-writable DVDs I noticed that for about twice the stack of DVDs I could buy an external hard drive that I could always have connected and not worry about changing disks. I haven't looked back since. Of course, there are disadvantages too, especially if HDD is connected all the time. For example, the power surge that kills the main disks, might also kill the HDD used for backups. An malign virus or software error or user error (rm -rf / while mounted) might also nuke all your backups as well as your main disk. Also there is rendundancy in numbers - if you substitute one HDD for 100 of DVDs, you lose that redundancy (to gain convenience, of course). But if your (one) backup HDD fails, you loose all your backups. If one of your DVD dies however, there is quite a big chance that you'll be able to recover some older state from some of the other DVDs. Thirdly (and that is specific to WORM media), no software activity (like virus or user error) can erase backups from DVDs once they've burned -- they are read only (and will only die with age) You may alleviate some of the issues by buying stack od external HDDs and rotating them on/off-site, but that will again lose you the convenience (and gets more pricey). The main problem with DVDs however remains -- they are just too small and too slow nowdays for anything but small sites... So while there are solutions that are both convenient and relatively secure, they tent to be pretty expensive. So there is no silver bullet - the smaller sites just have to accept some disadvantages. For some, DVDs might still be the best answer. -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] ./Configure unable to make BACULA-5.0.1 from src
On Wed, Mar 31, 2010 at 05:23:43PM +, jcdole wrote: Phil Stracchino alaric at metrocast.net writes: On 03/31/10 06:24, jcdole at free.fr wrote: tray-monitor.h:36:21: error: gtk/gtk.h: No such file or directory GTK is installed. http://article.gmane.org/gmane.comp.bacula.user/59120 so the question is actually How can I tell configure to search in /usr/include/gtk2.0/gtk/gtk.h Adding the -I /usr/include/gtk2.0 to CFLAGS line should work. -- Matija Nalis Odjel racunalno-informacijskih sustava i servisa Hrvatska akademska i istrazivacka mreza - CARNet Josipa Marohnica 5, 1 Zagreb tel. +385 1 6661 616, fax. +385 1 6661 766 www.CARNet.hr -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting files/folders on restores
On Thu, Apr 01, 2010 at 11:54:51AM +0200, Marco Zimmermann wrote: How can I change this behavior? I mean restore in a way that newer files/folders will be deleted? You need to backup with Accurate = Yes, see http://www.bacula.org/en/dev-manual/New_Features.html#SECTION0031 Note that accurate will make a highier load on your machines than the default behaviour. If I imagine I would restore the whole machine like this, I have to expect that a lot of crap remains on the machine!? So the only way in order to get the exact state, I have to delete all files from the disk and to a complete restore!? Isn?t there a other way? Alternatively, if you can't or won't use Accurate=yes, you can keep doing just full backups, no incremental/differental ones. -- Matija Nalis Odjel racunalno-informacijskih sustava i servisa Hrvatska akademska i istrazivacka mreza - CARNet Josipa Marohnica 5, 1 Zagreb tel. +385 1 6661 616, fax. +385 1 6661 766 www.CARNet.hr -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting files/folders on restores
On Thu, Apr 01, 2010 at 01:48:34PM +0200, Marco Zimmermann wrote: Thanks for your reply! How can I change this behavior? I mean restore in a way that newer files/folders will be deleted? You need to backup with Accurate = Yes, see http://www.bacula.org/en/dev-manual/New_Features.html#SECTION00310 000 We already using Accurate = Yes for all our Backups. So that's obviously not the reason why the file and folder isn't deleted. Oh, I see. So the problem you have is happening only when you restore to the same client and the same place you did a backup from ? For example, if you restore to different directory (and/or to a different client), do you then only get the wanted files ? If it is *not* so, then I guess something is broken with accurate mode (and/or not *all* of your backups related to restore set have been *created* with accurate=yes) However, if restore to different (empty) directory gives you result you want, then I think that is by intention and not a bug. For example, maybe someone don't need to restore all the files, or wants to select a regexp to rename files when restoring etc, or (s)he simply wants to restore files removed by accident and not nuke all the new files created in the meantime, and many other situations like those. In those circumstances, removing all files not in selected restore fileset would be wrong (and irrepairable). And as Bacula (not any other system) cannot guess what you want, it goes by the principle of least damage (not destroying stuff implicitely because you might have wanted that). So, back to your problem, if you don't want *ANYTHING* to remain in the destination directory (the one selected in restore) *BUT* the stuff you had on backup, you need to clean destination directory yourself, for example by doing rm -rf /dest/dir before doing the restore (or using more sophisticated techniques, of course). I'm not sure if you can script that automatically in ClientRunBeforeJob (that is, if the script sees the restore variables like destination directory etc). It would be nice if one could; however if you can't you'll need to take care of it yourself (by scripting whatever does the restore, to first nuke the whole directory) -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Deleting files/folders on restores
On Thu, Apr 01, 2010 at 03:00:00PM +0200, Marco Zimmermann wrote:
Am 01.04.10 14:24 schrieb Matija Nalis unter mnalis+bac...@carnet.hr:
Oh, I see. So the problem you have is happening only when you restore
to the same client and the same place you did a backup from ?
Yes, correct, only if I restore to the same directory.
If I restore to another directory, every thing works fine.
ok
You're right, that's the point. I thought that there has to be an option to
control the behavior of deleting files/folder they don't belong to the
restored set of data.
No, as far as I know there is no such option in bacula at this time...
My Problem is, that I maybe want to restore the whole system to role back to
a certain point of my installation/configuration. So I don't know where the
files have changed/been added and where not (e.g. If I update/upgrade the
whole system, something goes wrong, an I want to roll back to the point
before the update). Do you know what I mean? I don't think that I am the
Yes, I understand. I usually prefer to do that via LVM snapshots
(*much* faster than full restore).
However, if I need to restore whole machine to previous state from
bacula tape backup, I boot from rescue USB stick (with static
bacula-fd on it), nuke all data (mkfs all partitions is fastest way),
and then do the full restore (note: on full restores, you always have
to watch out to reinitialize the boot loader at the end, or you could
end up with unbootable machine, which means more downtime)
http://www.bacula.org/en/dev-manual/Disast_Recove_Using_Bacula.html
There are alternatives -- you can restore everything to some subdir
like '/RESTORE', and then delete or move all old stuff ('/*', except
'/RESTORE' of course) and then move restored stuff to right locations
- but you should use statically linked tools for that operations or
things will break when /lib and similar directories disappears).
I do that for example if I do not have physical access to the machine
(in which case one has to be very careful :)
only one who has this demand! So I don't can script it I guess...
I need that very rarely though (I usually test upgrades on virtualized
test instances first) so the booting from USB is not the issue for me,
but I can see your problem (when it is full system we're talking about,
than you can't easily do rm -rf / and then run a restore; but you
need to boot from CD or USB or something first which is somewhat
inconvenient)
So such a feature (accurate restore, eg. nuking all files in
specified restore directory not present in restore fileset) might be
nice, but not nearly cruical (you still need to make your USB recover
stick with bacula anyway in case your machine dies and you need to do
full restore with nothing of the old machine left, and that is the
most of work).
However, if you do such restores very often and think it is important,
you could submit a feature request as described on:
http://www.bacula.org/en/?page=feature-request
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Big log....
On Thu, Apr 01, 2010 at 10:56:03AM -0300, Gilberto Nunes wrote: But now that I see... the Log.MYD is in bacula directory locate on /var/lib/mysql... Then, i thing that this Log.MYD is onwer but bacula or not??? Yeah, I think Phil is wrong, you're not talking about same thing. /var/lib/mysql/bacula/Log.MYD / MYI is bacula table, which contains logged output for jobs (any errors etc., the stuff you get with list joblog bacula command), and not the MySQL SQL log. See it's contents for yourself with something like: mysql bacula -e 'select * from Log limit 30' It should be purged automatically when the jobs themselves are purged. I do not know what would happen if you decided to purge it manually from SQL (maybe nothing, but...). So if should probably shrink by itself after some time if you reduce your Job Retention. Or maybe it is so big because some of your jobs had a ton of messages - that should also expire by itself with such jobs. Alternatively, I think you could modify your catalog line in Messages resources in config files, in order to log less (or even nothing). -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] mysql- postgres conversion. Problems...
On Sun, Mar 21, 2010 at 10:20:22PM +, Alan Brown wrote: On 21/03/10 20:43, Bruno Friedmann wrote: After moving your data from mysql to postgresql, it's important to adjust the pkey index to the correct values (last of the table) Check for each table of the db. If I remember correctly this is noted in the manual or the wiki. It's not, but I've found how to fix it. The steps given in both documents are deficient and I'll submit updates when done. Hi Alan, have you perhaps succeded in transition from MySQL to PostgreSQL for bacula ? An update to the documentation (or your explanation about problems and what you did) would be greately appreciated, as we're also being pinned down with extreme MySQL slowdowns in 5.0.x (bacula bug 1472) and must move to PostgreSQL without losing catalog data (or die trying, which I'd like to avoid if at all possible :) -- Matija Nalis Odjel racunalno-informacijskih sustava i servisa Hrvatska akademska i istrazivacka mreza - CARNet Josipa Marohnica 5, 1 Zagreb tel. +385 1 6661 616, fax. +385 1 6661 766 www.CARNet.hr -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Fwd: bacula windows file path slovak letters
On Thu, Mar 25, 2010 at 08:40:32AM +0100, Michal Juranyi wrote: Hello there, I am running Bacula 2.4.4 on Ubuntu server and 2.4.4 Windows clients. Everything works fine, except of one thing. There are localized slovak and czech WinXPs and they have localized also file paths (i.e. Application data is translated as Data aplikací). Look at the last letter, it's í, not i. Our language includes more such letters (i.e. ž,ť,č,š). The problem is that Windows can handle files with such letters in filename and so users got used to use such filenames. But Bacula as I see has a big problem with these files. I want to use Bacula to backup employees' files, like invoices etc. What exactly do you mean by has a big problem ? Do you get an error ? If so, which one (please copy it exactly) ? Or do those files silently gets dropped from catalog ? Or something else ? Please explain in more detail what do you do exactly, and what do you get (and what you were expecting, if it's not obvious). Also, what is your database backend for bacula (name and version) ? Is there any way how to teach Bacula to work with these files? I think that recent versions of bacula (last one is 5.0.1) should work out of the box with your files (but you might need to nuke the old catalog database, ie. do purge / install instead of upgrade. Or might not, dunno). Over here, bacula 5.0.1 director/SD with bacula-fd 2.4.4 (both servers and clients on debian, though), backs up and restores files containing Croatian letters in them (šđčćž ŠĐČĆŽ) without problems; so I would expect it to work with Slovak and Czech letters too. Note: your server (DIR, SD) must have same of newer versions of bacula than your FD. -- Matija Nalis Odjel racunalno-informacijskih sustava i servisa Hrvatska akademska i istrazivacka mreza - CARNet Josipa Marohnica 5, 1 Zagreb tel. +385 1 6661 616, fax. +385 1 6661 766 www.CARNet.hr -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] restores not working
On Tue, Mar 23, 2010 at 04:07:17PM +, Alan Brown wrote: Matija Nalis wrote: It is probably not hung, but just very, very slow. Yes. You probably need a LOT more ram and to tune mysql's parameters. Or maybe someone should tune the SQL queries (no, I'm not volunteering, it's not my forte) and/or the way bacula stores the catalog ? The main issue is that for us 3.0.3 was about 20-100 *times* faster for approximately same dataset than 5.0.1. I do understand that 5.0.0 added BaseFiles support, but IMHO such is a speed drop is not an acceptable tradeoff (especially if one cannot turn it off in order to get faster queries again. I'd gladly compile with --disable-basefiles if that gave me 2 orders of magnitude speedup). Queries that never took more than 3-5 *minutes* with 3.0.3 have started taking more that several *hours* with 5.0.0. (Before you ask, we've had to upgrade because there were [and still are in 5.0.1, although somewhat rarer] bugs with director stopping working -- otherwise we would've downgraded back to 3.0.x) We'd try tweaking key_buffer (and converting to InnoDB and tweakings innodb_buffer_pool_size), join_buffer_size, max_heap_table_size, tmp_table_size, sort_buffer_size, read_buffer_size, read_rnd_buffer_size but in the end we've had to reduce retention to just a few weeks in order to make the restoration happen in reasonable times (ie. getting the file selection in less than 10 minutes). Even with 48Gb ram, a few restores on our system (~255 million File records: up to 4 million files on some full backups but nost are under 100k entries) could take an hour to get past the building directory tree stage. That is really terrible, I really think the developers should look into it. We've used Legato networker before (we still do, as we're not yet successfully completed the migration; and it's looking more and more grim prospect by the day), and on approximately the same dataset (of about 500 million records spread over 100 servers) and somewhat weaker hardware, it would allow user to start selecting files to restore in matter of *seconds* (and it was using it's simple db6 files, no server/database tuning required at all) Now with bacula 5.0.1, we have to wait several *hours* before we can start selecting files to restore, and it is considered normal ?! Several minutes might be tolerated by our users (although even that is almost hundred times slower than they were used to !), but several hours most certainly isn't (and a retention drop from several months to several weeks as alternative is also isn't making them extremely happy) It's a _lot_ faster with postgresql and moderate tuning (My other gripes about the changeover notwithstanding, those are annoyances, not showstoppers) Waiting several hours to choose file for restoring might not be an issue for you; but we have users which were used to waiting just several seconds to select files to restore (and a few more minutes for restore to happen), and they are not impressed at all with bacula. As it is, it is *much faster* for us if we need to restore one file to do a complete restore of whole server and then delete 99.999% of the files, than to use the file catalog to select few files to restore. That is ridiculous situation. The lesson for us was that mysql doesn't scale to huge datasets well and we should have switched to postgres much earlier. That might be, and we'll try converting to PostgreSQL (there are issues with moving bacula data from MySQL to PostgreSQL), it seems. -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] restores not working
On Mon, Mar 22, 2010 at 01:25:38PM -0500, Jonathan R. Dundas wrote: I have a RHEL 5 x86_64 patched-current install with bacula RPM's built from the sourceforge src RPMs. I'm running MySQL community edition 5.1.43-1 mysql.com RPMs. I have updated bacula and tried this same operation with bacula 5.0.0, 5.0.1 and 5.0.2. When I try to restore a backup, bconsole appears to hang here: [...] You have selected the following JobIds: 8518,8529,8550,8571,8593,8613 Building directory tree for JobId(s) 8518,8529,8550,8571,8593,8613 ... and it stays there, never completing. The MySQL database seems hung on this query: It is probably not hung, but just very, very slow. We've had a same issue, with about 500 million records in File (and about 120GB on disk for File.ibd) on (mostly dedicated to mysql) machine (8gig RAM, 8x2.33 Xeon, different configurations with about 3-6GB for mysql buffers) -- it could take several hours for 5.0.1 until it completed and the system was ready for selecting few files to restore. :-( We switched from MyISAM to InnoDB, it didn't help. Greatly reducing the retention times did help (as it made mysql dataset much smaller). see there bacula wiki on http://tinyurl.com/yg37ujf or http://wiki.bacula.org/doku.php?id=faq#restore_takes_a_long_time_to_retrieve_sql_results_from_catalog for more info. And let the list and/or wiki know if you manage to fix it, please ! -- Opinions above are GNU-copylefted. -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Any takers, pls? Bacula Sudo Authentication Failure
On Fri, Mar 19, 2010 at 03:53:31PM +, Lucio Godoy wrote: Any takers, pls? Hi All; I have Bacula 3.0.2-1 installed on a RHEL 5.4 and it works well without any issues, but every day I get the following message from my log file watch program: sudo: Authentication Failures: (0) - bacula: 1 Time(s) Had a look with visudo, bacula is not present there, should it be? If it happens every day at specific time, I would guess it is some kind od cron job. I do not use RHEL (and do not have such a sudo issue), but you can try to search (as root) : egrep -r sudo|bacula /etc/cron* /var/spool/cron /etc/logrotate.d It it shows something, that you can see what is trying to do the sudo and pehaps can deduce why, and then decide weather you need it (in which case you would allow sudo to execute that cron job) or if you don't (in which case you would remove that cron job) -- Matija Nalis Odjel racunalno-informacijskih sustava i servisa Hrvatska akademska i istrazivacka mreza - CARNet Josipa Marohnica 5, 1 Zagreb tel. +385 1 6661 616, fax. +385 1 6661 766 www.CARNet.hr -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula Enhancements Request
On Tue, Mar 16, 2010 at 02:40:05PM -0400, Phil Stracchino wrote: On 03/16/10 14:15, Gavin McCullagh wrote: On Tue, 16 Mar 2010, Bob Cousins wrote: - Explicit capturing of boot and partitioning information for easier bare metal restores. (Bare metal restores should be easy and fast for all OSes, but that is easier said than done.) Do you mean backing up the MBR? That might be nice alright. Doing this portably is actually a rather difficult problem because there are so many different partitioning schemes and partition table formats out there. It's a mess. well, yeah, it would probably best to handle this via wrapper. (like for example mtx-changer script) as then only such a wrapper needs to be modified for specific systems. Under GNU/Linux at least there is sfdisk(8), and one can easily do a partition dump with sfdisk -d /dev/sda somefile and restore it with sfdisk /dev/sda somefile. If there was such a wrapper, each distribution could tailor it to its own needs to allow (majority of) its users to have a hassle-free out-of-the-box working environment. Clean bare-metal restore has been a feature that's been worked on since almost the earliest days of Bacula, and there still isn't a clean single solution, precisely *because* it's a complex problem. That is, it is not that hard to do it, but it is hard to do it automatically -- the cause of problem being exactly high portability of bacula to different environments. -- Matija Nalis Odjel racunalno-informacijskih sustava i servisa Hrvatska akademska i istrazivacka mreza - CARNet Josipa Marohnica 5, 1 Zagreb tel. +385 1 6661 616, fax. +385 1 6661 766 www.CARNet.hr -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Maximum Concurrent Jobs in job definition
On Fri, Mar 12, 2010 at 11:21:13AM +1100, Norberto Meijome wrote: On 11 March 2010 13:40, Norberto Meijome numard...@gmail.com wrote: Hi everyone, I have a job that runs very frequently, and successive runs take different sets of data (as determined by a pre-job script). To avoid problems, my pre-job script sets a lock file and the post-clears it. When I configure my director for concurrent jobs = 2, i sometimes get 2 of these jobs trying to run in parallel (which is understandable, many times a run has more data than the average and is still running while the next one is due). My pre-script will then cancel the execution of the 2nd script (because it finds the lock file). Great. Why the pre/post locking script ? Would it not be easier to just use Allow Duplicate Jobs = no ? See http://www.bacula.org/en/dev-manual/New_Features.html#SECTION003101000 -- Matija Nalis Odjel racunalno-informacijskih sustava i servisa Hrvatska akademska i istrazivacka mreza - CARNet Josipa Marohnica 5, 1 Zagreb tel. +385 1 6661 616, fax. +385 1 6661 766 www.CARNet.hr -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Force serialization only for a subset of hosts ?
Hi everyone, we're having a bacula backup with an order of magnitude of 150 hosts, and some of them are xen DomU virtual machines sharing same physical host. Because of the number of hosts, we need the maximum parallelization possible (so the backups can finish in allowed timeslot), which we achieve by setting Maximum Concurrent Jobs = 50 and SpoolData = yes. That part works fine, and Director/SD machine can handle the load quite well. It however breaks down when the time comes to backup DomU virtual machines. It tries to run all of them at the same time, which (while making no problem for Director/SD) puts too big a load to xen physical host (whose I/O load goes up to the skies and all DomU machines become unresponsive). We could hand-tweak the start times for each of machines, of course, but it is painful and prone to errors, and conditions change often so it would be administrative nightmare. We've tried using different priorities (with Allow Mixed Priority = no on them) for DomU hosts, and that does serialize them ok, but has a disadvantage that when any of the DomU hosts run, nothing else will run until it finishes (what reduces our concurrency way down, and so makes a problem as not all machines manage to run their daily backup in allowed timeslots) To make a clear picture, imagine the following groups of hosts: Group A = hostA1, hostA2, hostA3, ... (those are on one xen physical machine) Group B = hostB1, hostB2, hostB3, ... (those are on other xen phys. machine) Group C = all other hosts. (those are all other [non-virtual] hosts). What we want to do is to have absolute maximum concurrency possible (of up to 50 hosts backing up at once), with only restriction that at any one time, *at most* 1 host from group A will be running, and *at most* 1 host from group B will be running. There is no limit on number of group C hosts that may be running at the same time (other than global concurrency limit of 50 hosts, of course) So, for example, at any moment, having 48 backups running from group C, 1 from group A and 1 from group B is OK. So is 49 from group C, and 1 from group A. But 48 from group C, and 2 from group A is NOT ok. Does anybody have the idea how to do that (if it is possible at all with bacula 5.0.x series) ? The best I could come up with is run with global Allow Mixed Priority = yes, and setup bacula with: hostC* - priority=50, start=03:59 hostA1 - priority=20, start=04:00 hostA2 - priority=21, start=04:01 hostA3 - priority=22, start=04:02 ... hostB1 - priority=20, start=04:00 hostB2 - priority=21, start=04:01 hostB3 - priority=22, start=04:02 ... which would give us some head start of 50 parallel C hosts, and serialize A and B groups (as host with prio21 will wait for host with prio20 to finish first, and those for prio22 will wait for prio21 and prio20 etc) But as soon as groups A and B start running, the C group would slowly starve - the initial 50 C-hosts will end one by one, but new ones won't be starting until both group A and B finish (which could take quite some time in our situation -- so we would waste a good part of the backup window running just one A host and one B host at the same time). Anybody has a better idea ? Thanks! -- Opinions above are GNU-copylefted. -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
