[Bacula-users] Bacula Full backup fails : DB Error: mergejoin input data is out of order (solved and a possible reason)
Hi, the problem is solved, I just want to put it here in case anybody stumbles across the same error message(s). I recently got the following error message: Full backup fails : DB Error: mergejoin input data is out of order The postgres log had the following entry: INSERT INTO Log (JobId, Time, LogText) VALUES (52526,'2022-07-20 09:55:47','bpstation-dir JobId 52526: Fatal error: sql_create.c:841 Fill File table Query failed: INSERT INTO File (FileIndex, JobId, PathId, FilenameId, LStat, MD5, DeltaSeq) SELECT batch.FileIndex, batch.JobId, Path.PathId, Filename.FilenameId,batch.LStat, batch.MD5, batch.DeltaSeq FROM batch JOIN Path ON (batch.Path = Path.Path) JOIN Filename ON (batch.Name = Filename.Name): ERR=ERROR: mergejoin input data is out of order The client was running Bacula 9.4.1, the director and storage daemon were under 9.4.4. All previous incremental backups, the last full (6 month ago) and even subsequent incrementals on that machine and all other backups on all other computers ran smoothly. After a lot of head scratching I found out that the filename table had a lot of duplicate entries (~10,500). Where those came from, I have no idea. After dbcheck fixed those, the full backup succeeded as expected. Cheers Stefan Münkner -- -- -- this is an empty signature :-) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Bacula Full backup fails : DB Error: mergejoin input data is out of order (solved and a possible reason)
Hi, the problem is solved, I just want to put it here in case anybody stumbles across the same error message. I recently got the following error message: Full backup fails : DB Error: mergejoin input data is out of order The postgres log had the following entry: INSERT INTO Log (JobId, Time, LogText) VALUES (52526,'2022-07-20 09:55:47','bpstation-dir JobId 52526: Fatal error: sql_create.c:841 Fill File table Query failed: INSERT INTO File (FileIndex, JobId, PathId, FilenameId, LStat, MD5, DeltaSeq) SELECT batch.FileIndex, batch.JobId, Path.PathId, Filename.FilenameId,batch.LStat, batch.MD5, batch.DeltaSeq FROM batch JOIN Path ON (batch.Path = Path.Path) JOIN Filename ON (batch.Name = Filename.Name): ERR=ERROR: mergejoin input data is out of order The client was running Bacula 9.4.1, the director and storage daemon were under 9.4.4. All previous incremental backups, the last full (6 month ago) and even subsequent incrementals on that client and all other backups on all other computers ran smoothly. After a lot of head scratching I found out that the filename table had a lot of duplicate entries (~10,500). Where those came from, I have no idea. After dbcheck did fix those, the full backup succeeded as expected. Cheers Stefan Münkner -- -- this is an empty signature :-) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] 800, 000+ orphaned paths entries cannot be pruned when using bvfs
Am 23.01.19 um 11:03 schrieb Radosław Korzeniewski: > Hello, Hi, > wt., 22 sty 2019 o 14:39 Stefan Muenkner <mailto:stefan.muenk...@uni-saarland.de>> napisał(a): > > (...) > Searching for hascache in the rest of the source I found that hascache in > the job table seems to be only reset in ".bvfs_clear_cache". > > > Great! > > > >> > >> 9) Check for orphaned Path records > >> > >> Select function number: 9 > >> Pruning orphaned Path entries isn't possible when using BVFS. > >> > > > So, it seems the message above should be more descriptive and user > friendly and points to bvfs_clear_cache command. No I don't think so. As far as I remember the .bvfs_xxx commands are actually only there to help GUIs interact with the server - I think they are not meant to be widely used. It is actually the case that whenever bat (at least my version 7.4.1 here) is called and you select any job in brestore, it sets hascache=1 for the respective job (and seemingly some more). Originally I thought the occasionally crashing bat left those behind, but even if bat is closed regularly it never clears up the hascache, which I think it should. I have no idea what consequences it would have to ignore hascache for the pruning of orphaned paths, but I suspect, that behaviour is there for a reason. I only made sure, that bat was not running when I used .bvfs_clear_cache. > > best regards > -- > Radosław Korzeniewski > rados...@korzeniewski.net <mailto:rados...@korzeniewski.net> Best regards, Stefan Münkner -- this is an empty signature :-) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] 800, 000+ orphaned paths entries cannot be pruned when using bvfs
Hello Kern, thanks for the answer. I'll do. But for the time being I found a workaround. Digging in the source, I found, that dbcheck only tests whether any entry in the job table has "hascache" <>0. Even if there is just one entry, dbcheck declines to celar up the orphaned path entries. Searching for hascache in the rest of the source I found that hascache in the job table seems to be only reset in ".bvfs_clear_cache". I suspect that the occasionally crashing bat session left those lying around. I manually called ".bvfs_clear_cache yes" in bconsole without bat running and then dbcheck could clear up the orphaned path entries. Best regards, Stefan Am 20.01.19 um 13:51 schrieb Kern Sibbald: > Hello Stefan, > > Please submit a bug report on this. When running dbcheck, it is my opinion > that we should not be relying on .bvfs. If this is true (as it seems from > your > output) then I must see why and make sure it is justified. dbcheck should > always > be able to prune. > > Best regards, > Kern > > On 1/17/19 11:13 AM, Stefan Muenkner wrote: >> Hi, >> >> I have an almost 9 years running Bacula installation that accumulated more >> than 800,000 orphaned paths entries in the database (around 5% of all >> entries in the path table). dbcheck claims it cannot prune those when BVFS >> is used. >> >> >> 9) Check for orphaned Path records >> >> Select function number: 9 >> Pruning orphaned Path entries isn't possible when using BVFS. >> >> >> Is there anything I can do about it? >> >> Best regards >> Stefan >> -- this is an empty signature :-) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] 800, 000+ orphaned paths entries cannot be pruned when using bvfs
Hi, I have an almost 9 years running Bacula installation that accumulated more than 800,000 orphaned paths entries in the database (around 5% of all entries in the path table). dbcheck claims it cannot prune those when BVFS is used. 9) Check for orphaned Path records Select function number: 9 Pruning orphaned Path entries isn't possible when using BVFS. Is there anything I can do about it? Best regards Stefan -- this is an empty signature :-) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] bRestore in bat: unable to sort columns in the "File list" pane
Hi,
I've noticed lately that in bRestore which replaced the "Version Browser"
in bat in the 5.2.x release all entries in the "File list" pane are
displayed in a rather random order. Superficially it looks sometimes like
the entries are sorted alphabetically but in fact they are not.
Furthermore the entries cannot be sorted/re-sorted by clicking on the
column headers ("Type", "Size", "File name", "Date") of the columns.
Since filters do not work on directories (in that version) it is rather
cumbersome to navigate to a specific directory.
Anybody else noticed that?
I am using Bacula 5.2.6 from the opensuse Archive:Backup repo.
Best regards,
Stefan
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Bacula backup problems
On 09/06/2011 02:18 PM, René Klomp wrote: > Pre_backup:sh > > #!/bin/bash > > #scst_lv1 hangt aan de Terminal Server (192.168.32.241) > #scst_lv2 hangt aan de Exchange Server (192.168.32.231) > #scst_lv3 hangt aan de File Server (192.168.32.230) > #scst_lv4 hangt aan de Alure Server(192.168.32.232) > > sudo /usr/sbin/lvcreate -L40G -s -n scst_lv1-snap /dev/backup_vg/scst_lv1 > sudo /usr/sbin/lvcreate -L40G -s -n scst_lv2-snap /dev/backup_vg/scst_lv2 > sudo /usr/sbin/lvcreate -L40G -s -n scst_lv3-snap /dev/backup_vg/scst_lv3 > sudo /usr/sbin/lvcreate -L40G -s -n scst_lv4-snap /dev/backup_vg/scst_lv4 > > # Mount de snapshot (mooiste is als deze even groot is als het origineel, > dan kan er eigenlijk niets mis gaan > > sudo /bin/mount -t ntfs -o loop,ro,offset=$(( 63*512 )) > /dev/mapper/backup_vg-scst_lv1--snap /backupmnts/SCSTDISC1 > sudo /bin/mount -t ntfs -o loop,ro,offset=$(( 63*512 )) > /dev/mapper/backup_vg-scst_lv2--snap /backupmnts/SCSTDISC2 > sudo /bin/mount -t ntfs -o loop,ro,offset=$(( 63*512 )) > /dev/mapper/backup_vg-scst_lv3--snap /backupmnts/SCSTDISC3 > sudo /bin/mount -t ntfs -o loop,ro,offset=$(( 63*512 )) > /dev/mapper/backup_vg-scst_lv4--snap /backupmnts/SCSTDISC4 > > exit 0 > The above script silently assumes that the snapshots are not already existing _and_ are not mounted - both prerequisites cannot be relied upon since unmounting and destroying the snapshots happens somewhere else (and can fail for any reason) Call the person on holiday to fix this - it looks like this is probably the main part of the problem <---snip-snap---> > > > [root@localhost ~]# vgdisplay > /dev/loop0: read failed after 0 of 2048 at 0: Input/output error Above is the same error message you get down there vvv <--snip-snap-> > 01-Sep 08:00 backup.localdomain-dir JobId 167: BeforeJob: /dev/loop0: > read failed after 0 of 4096 at 0: Input/output error > 01-Sep 08:00 backup.localdomain-dir JobId 167: BeforeJob: Logical volume > "scst_lv4-snap" already exists in volume group "backup_vg" > 01-Sep 08:00 backup.localdomain-dir JobId 167: BeforeJob: mount: could not > find any free loop device > 01-Sep 08:00 backup.localdomain-dir JobId 167: BeforeJob: mount: could not > find any free loop device and the other error messages point in the direction of my above comment re Pre_backup.sh simply assuming that the snapshots are non existent and not mounted. The /dev/loop0 could be related to all this or might have nothing to do with it. > [...] > Judging from those "lv", "vg" and "snap" bits, it appears that Bacula > is configured to make backups using "LVM snapshots" [1]. > Supposedly that /etc/bacula/pre_backup.sh script is responsible for > creation of a snapshot for one or more LVM logical volumes (they hold > filesystems) and mounting them somewhere for the Bacula to back up. > > So without the in-depth details about how this is implemented, it's > impossible to tell anything further. > Bacula itself has nothing to do with this. > > In theory, you could provide the director's config file, the contents > of that "pre_backup.sh" script and the output of running > # vgdisplay > and > # lvdisplay > so we could try to figure out how it's supposed to work and how to > fix the situation, but I'd rather call that person out from their > holidays. > > 1. http://en.wikipedia.org/wiki/Logical_Volume_Manager_%28Linux%29 > Best regards, Stefan -- this is an empty signature :-) -- Why Cloud-Based Security and Archiving Make Sense Osterman Research conducted this study that outlines how and why cloud computing security and archiving is rapidly being adopted across the IT space for its ease of implementation, lower cost, and increased reliability. Learn more. http://www.accelacomm.com/jaw/sfnl/114/51425301/ ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] prevent windows shutdown when backup run
On 21.04.2011 10:58, John Doe wrote: > From: ruslan usifov > >> We back up client computers throw bacula-fd, but have some issue throw >> shutdown process (when user press power off button or throw "start" >> menu). We want prevent shutdown process while fd work, does exists any >> solutions for this task? > > Maybe look at: > http://www.windows-help-central.com/windows-shutdown-script.html > You'd have to write a script/program that waits until bacula-fd is done. > But I never tried it myself... > > JD > > -- > Benefiting from Server Virtualization: Beyond Initial Workload > Consolidation -- Increasing the use of server virtualization is a top > priority.Virtualization can reduce costs, simplify management, and improve > application availability and disaster protection. Learn more about boosting > the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users It might work that way, by going into a loop querying whether a backup task is currently being run and leave the loop only if no task is run (with a message to the user). But it would still leave the problem, how to detect that a backup task is being run..does bacula-fd provide anything like that.under windows... Cheers, Stefan -- Stefan Münkner FR 2.5 Biophysik Universität des Saarlandes Campus Homburg Gebäude 76 D-66421 Homburg/Saar .. phone +49 6841 16-26222fax +49 6841 16-26211 http://www.uniklinikum-saarland.de/biophysik/ -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to parse Bacula conf files
While I think it might be worthwhile having the configuration dumped in XML (== machine readable format), I think the other way around is not something that would be needed in bacula itself. There is a whole bunch of XSLT processors that would transform a XML-file into a bacula configuration file. You only need to provide the XSLT file and adapt that for future versions. Provided there are no principal problems anyway, since not everything maps cleanly into XML and back again. But in the end somebody needs to do that by actually write, testing and documenting some code. Stefan On 21.10.2010 16:24, Hugo Letemplier wrote: > a XML parse intreprator for bacula as well as in input than in output > could be a very cool project for next versions. > that would also make easier the implementation of a gui for modeling > or monitor a backup architecture. > > > 2010/10/21 Stefan Muenkner : >> On 20.10.2010 16:25, Josh Fisher wrote: >>> >>> On 10/19/2010 8:28 AM, Yuri Timofeev wrote: >>>> Hi >>>> >>>> During development the prototype of the my new system (Webacula 5.5), >>>> I tried to parse the bacula-dir.conf. >>>> However, this is probably not possible with simple regexp's. >>>> It turned out that some applications, for example, Webmin does this >>>> with a regexp >>> >>> I don't think regexp will work. The best way is to translate the C code >>> from the Bacula source into PHP. Since you are requiring PHP 5.0 or >>> above, I think, then the values could be placed in a class object, just >>> as Bacula puts them in C struct. Or, since all directives are key=value >>> pairs, you could just put them in an associative array. >>> >>> >>> -- >>> Nokia and AT&T present the 2010 Calling All Innovators-North America contest >>> Create new apps & games for the Nokia N8 for consumers in U.S. and Canada >>> $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing >>> Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store >>> http://p.sf.net/sfu/nokia-dev2dev >>> ___ >>> Bacula-users mailing list >>> Bacula-users@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/bacula-users >> >> a better solution still would be to enable bacula to dump its >> configuration as a (e.g.) XML or JSON file, probably in a rather generic >> way so that future changes to the structure/content of the config files >> would not need additional work on behalf of the bacula developers. XML or >> JSON - as examples - could be easily picked up by php, ruby, perl, >> python, java you name it whatever. >> >> -- >> this is an empty signature :-) >> >> -- >> Nokia and AT&T present the 2010 Calling All Innovators-North America contest >> Create new apps & games for the Nokia N8 for consumers in U.S. and Canada >> $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing >> Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store >> http://p.sf.net/sfu/nokia-dev2dev >> ___ >> Bacula-users mailing list >> Bacula-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bacula-users >> -- this is an empty signature :-) -- Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] How to parse Bacula conf files
On 20.10.2010 16:25, Josh Fisher wrote: > > On 10/19/2010 8:28 AM, Yuri Timofeev wrote: >> Hi >> >> During development the prototype of the my new system (Webacula 5.5), >> I tried to parse the bacula-dir.conf. >> However, this is probably not possible with simple regexp's. >> It turned out that some applications, for example, Webmin does this >> with a regexp > > I don't think regexp will work. The best way is to translate the C code > from the Bacula source into PHP. Since you are requiring PHP 5.0 or > above, I think, then the values could be placed in a class object, just > as Bacula puts them in C struct. Or, since all directives are key=value > pairs, you could just put them in an associative array. > > > -- > Nokia and AT&T present the 2010 Calling All Innovators-North America contest > Create new apps & games for the Nokia N8 for consumers in U.S. and Canada > $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing > Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store > http://p.sf.net/sfu/nokia-dev2dev > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users a better solution still would be to enable bacula to dump its configuration as a (e.g.) XML or JSON file, probably in a rather generic way so that future changes to the structure/content of the config files would not need additional work on behalf of the bacula developers. XML or JSON - as examples - could be easily picked up by php, ruby, perl, python, java you name it whatever. -- this is an empty signature :-) -- Nokia and AT&T present the 2010 Calling All Innovators-North America contest Create new apps & games for the Nokia N8 for consumers in U.S. and Canada $10 million total in prizes - $4M cash, 500 devices, nearly $6M in marketing Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store http://p.sf.net/sfu/nokia-dev2dev ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Installing with RPMs
On 04.05.2010 20:32, Joseph Spenner wrote: > --- On Tue, 5/4/10, Joseph Spenner wrote: > >> From: Joseph Spenner >> Subject: Re: [Bacula-users] Installing with RPMs >> To: bacula-users@lists.sourceforge.net >> Date: Tuesday, May 4, 2010, 11:29 AM >> --- On Tue, 5/4/10, Phil Stracchino >> >> wrote: >> I have mysql running on my Suse 11.2 64bit >>> system. How does mysql get populated with the >> bacula >>> database/tables? When I compiled from source >> earlier, >>> I found I needed to manually run some scripts: # create_bacula_database # make_bacula_tables # grant_bacula_privileges But I see no such scripts, nor did I notice any >> such >>> processes run during the RPM installation. I logged >>> into mysql and did 'show tables', but saw no bacula >> data. >>> >>> Having never installed from RPM, I can't help you >> there, >>> sorry. But the >>> RPMs SHOULD contain the scripts you reference above. >>> >> >> Ok, not sure where the mysql update scripts are. I >> might have to copy them from the source installation and run >> them manually. >> >> Also, when I try to run 'bat' or 'bconsole', it appears to >> be trying to connect to the "Director" on port 9101, but >> this process isn't running and there is no tcp/9101 >> available. >> When I start bacula using "bacula start" or using the 3 rc >> scripts created by the RPM installation: >> >> bacula-sd >> bacula-fd >> bacula-dir >> >> I can start bacula-sd, which creates a process listening on >> tcp/9103 >> I can start bacula-fd, which creates a process listening on >> tcp/9102 >> But if I try to start bacula-dir, it silently fails and >> starts no process on tcp/9101 which I suspect is what 'bat' >> and 'bconsole' want. When I try to start 'bconsole', I >> only get: >> >> Connecting to Director bacula-va-dir:9101 >> >> And my root prompt returns. >> >> 'bat' does something similar, but takes about a minute >> before failing. >> >> >From the messages file: >> May 4 17:01:27 bacula bconsole: bsock.c:135 Unable to >> connect to Director daemon on bacula-va:9101. ERR=Connection >> refused >> >> The host name bacula-va is in the hosts file, so it >> resolves. This system has multiple interfaces, and the >> bacula-va is the network where backups will take place. >> >> I suspect this problem might be due to the lack of bacula >> data in mysql. I see the following errors in >> /var/lib/bacula/log: >> >> 04-May 17:00 bacula-dir JobId 0: Fatal error: Could not >> open Catalog "MyCatalog", database "bacula". >> 04-May 17:00 bacula-dir JobId 0: Fatal error: sqlite.c:178 >> Database /var/lib/bacula/bacula.db does not exist, please >> create it. >> 04-May 17:00 bacula-dir ERROR TERMINATION >> Please correct configuration file: >> /etc/bacula/bacula-dir.conf >> >> The only files in /var/lib/bacula are: >> >> bacula-dir.bacula-dir.6892008.mail >> bacula-dir.bacula-dir.6914616.mail >> bacula-dir.bacula-dir.6914648.mail >> bacula-fd.9102.state >> bacula-sd.9103.state >> log >> >> The .mail files contain the same info as the log file (the >> error above). >> >> Before I copy over the mysql install scripts from source, >> I'd like to verify if this would be the best idea. >> > > The suspense was killing me, so I went ahead and modified the scripts from > the source installation to work here. I now have a mysql 'bacula' database. > However, the Director will still not start. > It looks to me like the bacula-mysql-5.0.2-1.su112.x86_64.rpm package might > be built wrong. The fact that it's looking for /var/lib/bacula/bacula.db > leads me to believe it's looking for sqlite > I can report the same error with the bacula-postgresql-5.0.2-1.su112.x86_64.rpm package. The director does not start and complains about an unavailable /var/lib/bacula/bacula.db > Also, I had to make several modifications to the config files in /etc/bacula > because they were hard coded to a host: OS-112vmware > > > > > > > -- > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users -- ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
