Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
On 31.12.18 20:24, Kern Sibbald wrote: > Yes, MariaDB is supposed to be MySQL compatible, but it is not really. > The first time that I installed MariaDB over MySQL (with lots of > problems -- best to remove MySQL and the DB first), I ran the Bacula > regression tests and MariaDB failed with a false detection of a lock > deadlock, which I reported. It took awhile, but the project did fix the > problem so it *might* now run Bacula with no problems. That is to be seen. I've been running Bacula on Debian for my Universities IT department since Bacula 1.x, at first with Postgres, then with MySQL and then with MariaDB once Debian switched to it. Currently there are 350 Clients with ~30,000 jobs and ~6000,000,000 Files in the database, at a size of ~200GiB. (Which reminds me to run dbcheck again, since its the new year and everything.) So far I've never seen any problems with Bacula and the database. (Yes, I do have a separate DB server with 128GiB RAM to run this on.) I don't know how Postgres would work and how much resources it would need (I have no doubt it would work perfectly now.) but I remember in the very early days, we had massive problems with Postgres, as the Vacuum process would soon take up all resources, never finish and block the backup all together. Which is why we switched to MySQL back then. Grüße, Sven. signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
Hi folks, Just for information about MySQL/MariaDB. I'm using bacula 9.0.6 under CentOS 7.5 since almost a year, with no issues whatsoever. CentOS 7.5 is shipping with MariaDB 5.5.56. AFAIK, that version of MariaDB isn't exactly rocket science, and more or less compatible with comparable MySQL versions. I prefer a working setup over bleeding edge (unstable) performance in a backup scenario. If CentOS bumps MariaDB version to 10, and it will incur problems, I will not hesitate for a moment to switch to PostgreSQL. Just my 5 cents... I wish everybody a happy and problem free new year. Peter On 31.12.2018 20:24, Kern Sibbald wrote: Hello, Yes, MariaDB is supposed to be MySQL compatible, but it is not really. The first time that I installed MariaDB over MySQL (with lots of problems -- best to remove MySQL and the DB first), I ran the Bacula regression tests and MariaDB failed with a false detection of a lock deadlock, which I reported. It took awhile, but the project did fix the problem so it *might* now run Bacula with no problems. That is to be seen. The docker idea is very good. Best regards, Kern On 12/27/18 8:32 AM, Adam Nielsen wrote: I once tried MariaDB and found that it cannot be installed on the same machine with MySQL unless you do a lot of tweaking at a very low level. Currently I have both Postgres and MySQL installed on the same machines, so supporting an additional DB is painful. MariaDB is a fork of MySQL, maintained by the original MySQL developers. Oracle continues to maintain MySQL, but MariaDB is the "spiritual" successor of MySQL. "MySQL" was named after the creator's first daughter, "My", and so since Oracle now owns this name the fork has been named after his second daughter instead, "Maria". The reason it is so hard to install MariaDB alongside MySQL is the same as why it would be difficult to install Bacula 9.4 and 9.2 at the same time - they are both different versions of the same project. I expect that if MariaDB replaces all the MySQLs, eventually if no community patches come for MariaDB I will be forced to add support for it myself ... MariaDB has already replaced MySQL in most Linux distributions, at least those aimed at the desktop. There have been very few compatibility problems. I am currently running Bacula with MariaDB on the Arch Linux distribution and haven't encountered any issues so far, but then my backups are only home-user sized. As long as Bacula sticks to the common feature set and doesn't use any of the new Oracle-MySQL-specific features then it will probably continue to work just fine with both MySQL and MariaDB for some time yet, with no special maintenance effort. If you did want to test against MySQL and MariaDB separately, this is something you could easily do with Docker containers, as you can install whichever versions of whichever packages you like inside the containers, without affecting anything on your host machine. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
Hello, Yes, MariaDB is supposed to be MySQL compatible, but it is not really. The first time that I installed MariaDB over MySQL (with lots of problems -- best to remove MySQL and the DB first), I ran the Bacula regression tests and MariaDB failed with a false detection of a lock deadlock, which I reported. It took awhile, but the project did fix the problem so it *might* now run Bacula with no problems. That is to be seen. The docker idea is very good. Best regards, Kern On 12/27/18 8:32 AM, Adam Nielsen wrote: I once tried MariaDB and found that it cannot be installed on the same machine with MySQL unless you do a lot of tweaking at a very low level. Currently I have both Postgres and MySQL installed on the same machines, so supporting an additional DB is painful. MariaDB is a fork of MySQL, maintained by the original MySQL developers. Oracle continues to maintain MySQL, but MariaDB is the "spiritual" successor of MySQL. "MySQL" was named after the creator's first daughter, "My", and so since Oracle now owns this name the fork has been named after his second daughter instead, "Maria". The reason it is so hard to install MariaDB alongside MySQL is the same as why it would be difficult to install Bacula 9.4 and 9.2 at the same time - they are both different versions of the same project. I expect that if MariaDB replaces all the MySQLs, eventually if no community patches come for MariaDB I will be forced to add support for it myself ... MariaDB has already replaced MySQL in most Linux distributions, at least those aimed at the desktop. There have been very few compatibility problems. I am currently running Bacula with MariaDB on the Arch Linux distribution and haven't encountered any issues so far, but then my backups are only home-user sized. As long as Bacula sticks to the common feature set and doesn't use any of the new Oracle-MySQL-specific features then it will probably continue to work just fine with both MySQL and MariaDB for some time yet, with no special maintenance effort. If you did want to test against MySQL and MariaDB separately, this is something you could easily do with Docker containers, as you can install whichever versions of whichever packages you like inside the containers, without affecting anything on your host machine. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
On 12/27/18 2:32 AM, Adam Nielsen wrote: >> I once tried MariaDB and found that it cannot be installed on the same >> machine with MySQL unless you do a lot of tweaking at a very low level. >> Currently I have both Postgres and MySQL installed on the same machines, >> so supporting an additional DB is painful. > > MariaDB is a fork of MySQL, maintained by the original MySQL > developers. Oracle continues to maintain MySQL, but MariaDB is the > "spiritual" successor of MySQL. Well, in one sense. I think of it more as a fork than as a successor. The real evolution is still happening in MySQL, and (so far at least) it is MySQL, not MariaDB, that major third parties such as Percona maintain compatibility with. This is becoming problematic as MariaDB and MySQL drift further and further apart. Monty Widenius did the world no favors when he decided he wanted to walk back entrusting MySQL to Sun. > MariaDB has already replaced MySQL in most Linux distributions, at > least those aimed at the desktop. There have been very few > compatibility problems. So far. MySQL 8 is a major sea change. For the first time it has a real data dictionary ... and MariaDB doesn't. This is the end of binary data compatibility between MySQL and MariaDB. > As long as Bacula sticks > to the common feature set and doesn't use any of the new > Oracle-MySQL-specific features then it will probably continue to work > just fine with both MySQL and MariaDB for some time yet, with no special > maintenance effort. And truth be told, pretty much all of the Oracle-specific features are either Oracle Enterprise add-on tools, or functionality relevant only between multiple MySQL servers. Except for MySQL 5.7's clamp-down on SQL99 compliance, which is really just cleaning up a lot of sloppy things MySQL has historically done technically-wrong, like allowing nonexistent/impossible dates as a default DATETIME value, or NOT NULL without a DEFAULT. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
Am 27.12.2018 um 08:32 schrieb Adam Nielsen: > [...] > MariaDB has already replaced MySQL in most Linux distributions, at > least those aimed at the desktop. There have been very few > compatibility problems. I am currently running Bacula with MariaDB on > the Arch Linux distribution and haven't encountered any issues so far, > but then my backups are only home-user sized. Full-ACK. (Although I'm running Bacula and MariaDB on Debian, not on Arch.) For reference: my Bacula database in MariaDB is currently ~1.5GB in size, with ~6.5 mio. entries in the "File" table. > As long as Bacula sticks > to the common feature set and doesn't use any of the new > Oracle-MySQL-specific features then it will probably continue to work > just fine with both MySQL and MariaDB for some time yet, with no special > maintenance effort. > [...] Yes, I'd assume that too. Best regards Sebastian ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
On Thu, 27 Dec 2018 02:44:52 +0100 Kern Sibbald wrote: > ... I run PostgreSQL, so if someone wants and needs MariaDB > then the project kindly requests the necessary patches and testing. LOL. I run postgres too, mysql's continuing existence has always been a mystery to me. I can sort of see the need back when the only alternative on winders was trying to get AOL CD users to build postgres in cygwin and... and even then one could find e.g. interbase for free if one looked. -- Dmitri Maziuk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
> I once tried MariaDB and found that it cannot be installed on the same > machine with MySQL unless you do a lot of tweaking at a very low level. > Currently I have both Postgres and MySQL installed on the same machines, > so supporting an additional DB is painful. MariaDB is a fork of MySQL, maintained by the original MySQL developers. Oracle continues to maintain MySQL, but MariaDB is the "spiritual" successor of MySQL. "MySQL" was named after the creator's first daughter, "My", and so since Oracle now owns this name the fork has been named after his second daughter instead, "Maria". The reason it is so hard to install MariaDB alongside MySQL is the same as why it would be difficult to install Bacula 9.4 and 9.2 at the same time - they are both different versions of the same project. > I expect that if MariaDB replaces all the MySQLs, eventually if no > community patches come for MariaDB I will be forced to add support for > it myself ... MariaDB has already replaced MySQL in most Linux distributions, at least those aimed at the desktop. There have been very few compatibility problems. I am currently running Bacula with MariaDB on the Arch Linux distribution and haven't encountered any issues so far, but then my backups are only home-user sized. As long as Bacula sticks to the common feature set and doesn't use any of the new Oracle-MySQL-specific features then it will probably continue to work just fine with both MySQL and MariaDB for some time yet, with no special maintenance effort. If you did want to test against MySQL and MariaDB separately, this is something you could easily do with Docker containers, as you can install whichever versions of whichever packages you like inside the containers, without affecting anything on your host machine. Cheers, Adam. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
Hello, Let me remind you that this is a community project and as such in many cases needs community input rather than "Kern" supporting something. I run PostgreSQL, so if someone wants and needs MariaDB then the project kindly requests the necessary patches and testing. I once tried MariaDB and found that it cannot be installed on the same machine with MySQL unless you do a lot of tweaking at a very low level. Currently I have both Postgres and MySQL installed on the same machines, so supporting an additional DB is painful. I expect that if MariaDB replaces all the MySQLs, eventually if no community patches come for MariaDB I will be forced to add support for it myself ... Best regards, Kern On 12/22/18 6:44 PM, Dmitri Maziuk via Bacula-users wrote: On Fri, 21 Dec 2018 16:34:39 -0700 Chandler wrote: Kern Sibbald wrote on 12/21/18 01:34: You should be aware that the project no long support SQLite as it is not suitable for backups of the size we are seeing today. We do leave the code in Bacula so that users can upgrade to MySQL or PostgreSQL at their ease rather than being forced. Thanks, I didn't realize that. I guess I should work on migrating to MySQL... our catalog is about 20GB and probably over 50 million file records. Skip mysql and go postgres: you're going to get mariadb that Kern doesn't support either, once you hit 50M and 1 records you'll need to figure out how to tune up your innodb, when to optimize every table, etfc. Postgres is pretty much fire-and-forget. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
Hello Chandler, On 12/22/18 12:34 AM, Chandler wrote: Kern Sibbald wrote on 12/21/18 01:34: Hello Arizona EDU admin (Chandler?), Hi Kern! Yes Chandler here... (am not using the default mail client and so I have to modify "From" address to add my name each time) I don't think you mentioned what version of Bacula you are running. We are using 5.2.13 Oh, very old. You should be aware that the project no long support SQLite as it is not suitable for backups of the size we are seeing today. We do leave the code in Bacula so that users can upgrade to MySQL or PostgreSQL at their ease rather than being forced. Thanks, I didn't realize that. I guess I should work on migrating to MySQL... our catalog is about 20GB and probably over 50 million file records. I am surprised that SQLite handles such a big catalog -- good. That said, perhaps some of the Bacula users can help you. Otherwise, I suggest that reading the SQLite documentation might help you find a solution. Good luck. Thanks.. Martin did come up with a work-around for now... Given how old your Bacula is, you are probably better to stay with your work-around almost indefinitely as long as it continues to work. Good luck, Kern ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
On Fri, 21 Dec 2018 16:34:39 -0700 Chandler wrote: > Kern Sibbald wrote on 12/21/18 01:34: > > You should be aware that the project no long support SQLite as it > > is not suitable for backups of the size we are seeing today. We do > > leave the code in Bacula so that users can upgrade to MySQL or > > PostgreSQL at their ease rather than being forced. > Thanks, I didn't realize that. I guess I should work on migrating to > MySQL... our catalog is about 20GB and probably over 50 million file > records. Skip mysql and go postgres: you're going to get mariadb that Kern doesn't support either, once you hit 50M and 1 records you'll need to figure out how to tune up your innodb, when to optimize every table, etfc. Postgres is pretty much fire-and-forget. -- Dmitri Maziuk ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] "ERR=database or disk is full" while building directory tree for restore
Kern Sibbald wrote on 12/21/18 01:34: Hello Arizona EDU admin (Chandler?), Hi Kern! Yes Chandler here... (am not using the default mail client and so I have to modify "From" address to add my name each time) I don't think you mentioned what version of Bacula you are running. We are using 5.2.13 You should be aware that the project no long support SQLite as it is not suitable for backups of the size we are seeing today. We do leave the code in Bacula so that users can upgrade to MySQL or PostgreSQL at their ease rather than being forced. Thanks, I didn't realize that. I guess I should work on migrating to MySQL... our catalog is about 20GB and probably over 50 million file records. That said, perhaps some of the Bacula users can help you. Otherwise, I suggest that reading the SQLite documentation might help you find a solution. Good luck. Thanks.. Martin did come up with a work-around for now... ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
Martin Simmons wrote on 12/21/18 04:29: The actual work is done in bacula-dir, not bconsole, so you need to restart bacula-dir with SQLITE_TMPDIR set in its environment. Thanks Martin! That worked! I added an export for SQLITE_TMPDIR to bacula-dir init script and verified the /proc//environ has the SQLITE_TMPDIR set. When building the directory tree for the restore, the free space on / doesn't run out any longer and it is apparently using the value in SQLITE_TMPDIR, although I checked that directory and it doesn't have any contents (not even hidden 'dot' files)... but the mtime did keep changing... ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
> On Thu, 20 Dec 2018 15:55:53 -0700, admin said: > > Would anyone here happen to know how i could tell the sqlite that bacula > is using, to use a different location for temporary data? I did read > that document that Martin mentioned and tried to set > SQLITE_TMPDIR="/bacula/foo" before starting bconsole but it didn't seem > to have any effect since the / disk still filled up with temporary data > when the directory tree was being built. So I guess it is using the > PRAGMA temp_store_directory or the sqlite3_temp_directory global > variable? There isn't a /etc/sqlite.conf file or anything Anyone > know how i could update that location? Or maybe i need to ask sqlite > community? The actual work is done in bacula-dir, not bconsole, so you need to restart bacula-dir with SQLITE_TMPDIR set in its environment. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
Hello Arizona EDU admin (Chandler?), I am sorry to hear you are having these problems. Although it is possible that Bacula incremental has problems, it is unlikely. There is more likely some other cause. I don't think you mentioned what version of Bacula you are running. Could you post the output from one of your successful backups. You should be aware that the project no long support SQLite as it is not suitable for backups of the size we are seeing today. We do leave the code in Bacula so that users can upgrade to MySQL or PostgreSQL at their ease rather than being forced. That said, perhaps some of the Bacula users can help you. Otherwise, I suggest that reading the SQLite documentation might help you find a solution. Good luck. Best regards, Kern On 12/20/18 11:55 PM, ad...@genome.arizona.edu wrote: Thanks all... I think this problem has also caused a problem with bacula incremental backup, perhaps the accurate information? If you look at my original post, can see the backup is quite large and a large amount of files... the incremental backup that was running (started on 12-15) was acting like a full backup... i think it was at 29 million files and over 3TB of data when i noticed something was wrong. I did free up about 1GB on the / disk and that seemed to be enough for sqlite to get by now... i am able to do a restore, and the most recent incremental backup after 12-14 was only 35GB. Would anyone here happen to know how i could tell the sqlite that bacula is using, to use a different location for temporary data? I did read that document that Martin mentioned and tried to set SQLITE_TMPDIR="/bacula/foo" before starting bconsole but it didn't seem to have any effect since the / disk still filled up with temporary data when the directory tree was being built. So I guess it is using the PRAGMA temp_store_directory or the sqlite3_temp_directory global variable? There isn't a /etc/sqlite.conf file or anything Anyone know how i could update that location? Or maybe i need to ask sqlite community? Thanks ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
Thanks all... I think this problem has also caused a problem with bacula incremental backup, perhaps the accurate information? If you look at my original post, can see the backup is quite large and a large amount of files... the incremental backup that was running (started on 12-15) was acting like a full backup... i think it was at 29 million files and over 3TB of data when i noticed something was wrong. I did free up about 1GB on the / disk and that seemed to be enough for sqlite to get by now... i am able to do a restore, and the most recent incremental backup after 12-14 was only 35GB. Would anyone here happen to know how i could tell the sqlite that bacula is using, to use a different location for temporary data? I did read that document that Martin mentioned and tried to set SQLITE_TMPDIR="/bacula/foo" before starting bconsole but it didn't seem to have any effect since the / disk still filled up with temporary data when the directory tree was being built. So I guess it is using the PRAGMA temp_store_directory or the sqlite3_temp_directory global variable? There isn't a /etc/sqlite.conf file or anything Anyone know how i could update that location? Or maybe i need to ask sqlite community? Thanks ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
On 12/18/18 10:38 PM, Gary R. Schmidt wrote: > On 2018-12-19 10:42, ad...@genome.arizona.edu wrote: >> Hi all, >> >> We are having an error of our disk running out of space while trying >> to build the directory tree for a restore job. We have an older >> server with only 16GB OS disk and that only has about 8GB free. I > [SNIP] > > That's a very small OS partition in this day and age. > >> ERR=database or disk is full > > 99.% this is that you have run out of temporary file space, most > likely space needed by your database server to complete the query. More specific: Do not use system /tmp as your DB engine's tmpdir. Allocate a separate DB tmpdir somewhere there is plenty of storage. IDEALLY it should be on a separate physical device from both the OS itself and the DB data. -- Phil Stracchino Babylon Communications ph...@caerllewys.net p...@co.ordinate.org Landline: +1.603.293.8485 Mobile: +1.603.998.6958 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
The error message suggests you are using SQLite, in which case see https://www.sqlite.org/tempfiles.html (section 5 Temporary File Storage Locations at the end). __Martin > On Tue, 18 Dec 2018 16:42:17 -0700, admin said: > > Hi all, > > We are having an error of our disk running out of space while trying to > build the directory tree for a restore job. We have an older server > with only 16GB OS disk and that only has about 8GB free. I notice the > free space will fill up while the directory tree is being built and then > the error is output below. We did have a similar error before with > spool data filling up the disk, so I did move /var/lib/bacula to our > volume storage device which is on the same server. > > I couldn't figure out what is filling up the disk, I guess there is > another directory/file that is filling up with data? Do you know which > one? We could just mount this location on our volume storage device as > well? > Thanks > > +---+---+--+---+-+--+ > | JobId | Level | JobFiles | JobBytes | StartTime | > VolumeName | > +---+---+--+---+-+--+ > | 19406 | F | 25068910 | 3459126598047 | 2018-12-03 19:00:14 | > Pac-wing2-user1-2758 | > | 19507 | I | 625628 | 16116488523 | 2018-12-07 19:00:22 | > Pac-wing2-user1-2982 | > | 19539 | I | 212478 | 6456352368| 2018-12-08 19:03:17 | > Pac-wing2-user1-2985 | > | 19564 | I | 214199 | 7256961776| 2018-12-09 19:04:07 | > Pac-wing2-user1-2988 | > | 19589 | I | 277296 | 41375685395 | 2018-12-10 19:00:22 | > Pac-wing2-user1-2991 | > | 19614 | I | 315986 | 10145548827 | 2018-12-11 19:00:39 | > Pac-wing2-user1-2994 | > | 19639 | I | 265282 | 10025498752 | 2018-12-12 19:02:14 | > Pac-wing2-user1-2997 | > | 19664 | I | 597578 | 11810773912 | 2018-12-13 19:00:01 | > Pac-wing2-user1-3001 | > | 19691 | I | 622812 | 11744357702 | 2018-12-14 19:00:16 | > Pac-wing2-user1-3005 | > +---+---+--+---+-+--+ > You have selected the following JobIds: > 19406,19507,19539,19564,19589,19614,19639,19664,19691 > > Building directory tree for JobId(s) > 19406,19507,19539,19564,19589,19614,19639,19664,19691 ... Query failed: > SELECT Path.Path, Filename.Name, T1.FileIndex, T1.JobId, LStat, DeltaSeq > FROM ( SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId > AS PathId, File.FilenameId AS FilenameId, LStat , File.DeltaSeq AS > DeltaSeq, Job.JobTDate AS JobTDate FROM Job, File, ( SELECT > MAX(JobTDate) AS JobTDate, PathId, FilenameId, DeltaSeq FROM ( SELECT > JobTDate, PathId, FilenameId, DeltaSeq FROM File JOIN Job USING (JobId) > WHERE File.JobId IN > (19406,19507,19539,19564,19589,19614,19639,19664,19691) UNION ALL SELECT > JobTDate, PathId, FilenameId, DeltaSeq FROM BaseFiles JOIN File USING > (FileId) JOIN Job ON(BaseJobId = Job.JobId) WHERE BaseFiles.JobId > IN (19406,19507,19539,19564,19589,19614,19639,19664,19691) ) AS tmp > GROUP BY PathId, FilenameId, DeltaSeq ) AS T1 WHERE (Job.JobId IN ( > SELECT DISTINCT BaseJobId FROM BaseFiles WHERE JobId IN > (19406,19507,19539,19564,19589,19614,19639,19664,19691)) OR Job.JobId IN > (19406,19507,19539,19564,19589,19614,19639,19664,19691)) AND T1.JobTDate > = Job.JobTDate AND Job.JobId = File.JobId AND T1.PathId = File.PathId > AND T1.FilenameId = File.FilenameId ) AS T1 JOIN Filename ON > (Filename.FilenameId = T1.FilenameId) JOIN Path ON (Path.PathId = > T1.PathId) WHERE FileIndex > 0 ORDER BY T1.JobTDate, FileIndex ASC: > ERR=database or disk is full > > > -- > Chandler / Systems Administrator > Arizona Genomics Institute > University Of Arizona > www.genome.arizona.edu > > > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
Hello, I concur with what Gary wrote, but would like to make it very clear -- Bacula does not need much disk to do restores except of course on the client machine. On the other hand the catalog database server (normally MySQL or PostgreSQL) will need disk in very large restores. If your SQL engine runs out of disk space, it is really a SQL server problem. The two solutions are to: give the server more disk space, give it more memory (and configure it to use that memory). If your restore is critical, I would suggest either swapping your disk out for a much bigger one, or adding a second disk, or possibly adding a network device (NFS) disk. A network disk will of course be slower. You might even be able to plugin in a giant USB disk (1-2T USB disks are common and cheap and smaller ones are very cheap) and configure the SQL server to use it. Best regards, Kern On 12/19/18 12:42 AM, ad...@genome.arizona.edu wrote: Hi all, We are having an error of our disk running out of space while trying to build the directory tree for a restore job. We have an older server with only 16GB OS disk and that only has about 8GB free. I notice the free space will fill up while the directory tree is being built and then the error is output below. We did have a similar error before with spool data filling up the disk, so I did move /var/lib/bacula to our volume storage device which is on the same server. I couldn't figure out what is filling up the disk, I guess there is another directory/file that is filling up with data? Do you know which one? We could just mount this location on our volume storage device as well? Thanks +---+---+--+---+-+--+ | JobId | Level | JobFiles | JobBytes | StartTime | VolumeName | +---+---+--+---+-+--+ | 19406 | F | 25068910 | 3459126598047 | 2018-12-03 19:00:14 | Pac-wing2-user1-2758 | | 19507 | I | 625628 | 16116488523 | 2018-12-07 19:00:22 | Pac-wing2-user1-2982 | | 19539 | I | 212478 | 6456352368 | 2018-12-08 19:03:17 | Pac-wing2-user1-2985 | | 19564 | I | 214199 | 7256961776 | 2018-12-09 19:04:07 | Pac-wing2-user1-2988 | | 19589 | I | 277296 | 41375685395 | 2018-12-10 19:00:22 | Pac-wing2-user1-2991 | | 19614 | I | 315986 | 10145548827 | 2018-12-11 19:00:39 | Pac-wing2-user1-2994 | | 19639 | I | 265282 | 10025498752 | 2018-12-12 19:02:14 | Pac-wing2-user1-2997 | | 19664 | I | 597578 | 11810773912 | 2018-12-13 19:00:01 | Pac-wing2-user1-3001 | | 19691 | I | 622812 | 11744357702 | 2018-12-14 19:00:16 | Pac-wing2-user1-3005 | +---+---+--+---+-+--+ You have selected the following JobIds: 19406,19507,19539,19564,19589,19614,19639,19664,19691 Building directory tree for JobId(s) 19406,19507,19539,19564,19589,19614,19639,19664,19691 ... Query failed: SELECT Path.Path, Filename.Name, T1.FileIndex, T1.JobId, LStat, DeltaSeq FROM ( SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId AS PathId, File.FilenameId AS FilenameId, LStat , File.DeltaSeq AS DeltaSeq, Job.JobTDate AS JobTDate FROM Job, File, ( SELECT MAX(JobTDate) AS JobTDate, PathId, FilenameId, DeltaSeq FROM ( SELECT JobTDate, PathId, FilenameId, DeltaSeq FROM File JOIN Job USING (JobId) WHERE File.JobId IN (19406,19507,19539,19564,19589,19614,19639,19664,19691) UNION ALL SELECT JobTDate, PathId, FilenameId, DeltaSeq FROM BaseFiles JOIN File USING (FileId) JOIN Job ON (BaseJobId = Job.JobId) WHERE BaseFiles.JobId IN (19406,19507,19539,19564,19589,19614,19639,19664,19691) ) AS tmp GROUP BY PathId, FilenameId, DeltaSeq ) AS T1 WHERE (Job.JobId IN ( SELECT DISTINCT BaseJobId FROM BaseFiles WHERE JobId IN (19406,19507,19539,19564,19589,19614,19639,19664,19691)) OR Job.JobId IN (19406,19507,19539,19564,19589,19614,19639,19664,19691)) AND T1.JobTDate = Job.JobTDate AND Job.JobId = File.JobId AND T1.PathId = File.PathId AND T1.FilenameId = File.FilenameId ) AS T1 JOIN Filename ON (Filename.FilenameId = T1.FilenameId) JOIN Path ON (Path.PathId = T1.PathId) WHERE FileIndex > 0 ORDER BY T1.JobTDate, FileIndex ASC: ERR=database or disk is full ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] "ERR=database or disk is full" while building directory tree for restore
On 2018-12-19 10:42, ad...@genome.arizona.edu wrote: Hi all, We are having an error of our disk running out of space while trying to build the directory tree for a restore job. We have an older server with only 16GB OS disk and that only has about 8GB free. I [SNIP] That's a very small OS partition in this day and age. ERR=database or disk is full 99.% this is that you have run out of temporary file space, most likely space needed by your database server to complete the query. If you do not have a separate /tmp (or whatever/wherever your database and OS put temporary files) then that 8Gb is just not enough. Configure your database server to put its temporary files where there is space. That said, it could be running out of space after the query has completed and Bacula is doing its thing, increasing your temporary space enough will fix it. How much is enough?? I don't know, my Bacula server has 72GB of RAM, the OS is installed on a mirrored pair of 1TB disks, and has 34GB of swap space which is shared with /tmp. Cheers, GaryB-) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] "ERR=database or disk is full" while building directory tree for restore
Hi all, We are having an error of our disk running out of space while trying to build the directory tree for a restore job. We have an older server with only 16GB OS disk and that only has about 8GB free. I notice the free space will fill up while the directory tree is being built and then the error is output below. We did have a similar error before with spool data filling up the disk, so I did move /var/lib/bacula to our volume storage device which is on the same server. I couldn't figure out what is filling up the disk, I guess there is another directory/file that is filling up with data? Do you know which one? We could just mount this location on our volume storage device as well? Thanks +---+---+--+---+-+--+ | JobId | Level | JobFiles | JobBytes | StartTime | VolumeName | +---+---+--+---+-+--+ | 19406 | F | 25068910 | 3459126598047 | 2018-12-03 19:00:14 | Pac-wing2-user1-2758 | | 19507 | I | 625628 | 16116488523 | 2018-12-07 19:00:22 | Pac-wing2-user1-2982 | | 19539 | I | 212478 | 6456352368| 2018-12-08 19:03:17 | Pac-wing2-user1-2985 | | 19564 | I | 214199 | 7256961776| 2018-12-09 19:04:07 | Pac-wing2-user1-2988 | | 19589 | I | 277296 | 41375685395 | 2018-12-10 19:00:22 | Pac-wing2-user1-2991 | | 19614 | I | 315986 | 10145548827 | 2018-12-11 19:00:39 | Pac-wing2-user1-2994 | | 19639 | I | 265282 | 10025498752 | 2018-12-12 19:02:14 | Pac-wing2-user1-2997 | | 19664 | I | 597578 | 11810773912 | 2018-12-13 19:00:01 | Pac-wing2-user1-3001 | | 19691 | I | 622812 | 11744357702 | 2018-12-14 19:00:16 | Pac-wing2-user1-3005 | +---+---+--+---+-+--+ You have selected the following JobIds: 19406,19507,19539,19564,19589,19614,19639,19664,19691 Building directory tree for JobId(s) 19406,19507,19539,19564,19589,19614,19639,19664,19691 ... Query failed: SELECT Path.Path, Filename.Name, T1.FileIndex, T1.JobId, LStat, DeltaSeq FROM ( SELECT FileId, Job.JobId AS JobId, FileIndex, File.PathId AS PathId, File.FilenameId AS FilenameId, LStat , File.DeltaSeq AS DeltaSeq, Job.JobTDate AS JobTDate FROM Job, File, ( SELECT MAX(JobTDate) AS JobTDate, PathId, FilenameId, DeltaSeq FROM ( SELECT JobTDate, PathId, FilenameId, DeltaSeq FROM File JOIN Job USING (JobId) WHERE File.JobId IN (19406,19507,19539,19564,19589,19614,19639,19664,19691) UNION ALL SELECT JobTDate, PathId, FilenameId, DeltaSeq FROM BaseFiles JOIN File USING (FileId) JOIN Job ON(BaseJobId = Job.JobId) WHERE BaseFiles.JobId IN (19406,19507,19539,19564,19589,19614,19639,19664,19691) ) AS tmp GROUP BY PathId, FilenameId, DeltaSeq ) AS T1 WHERE (Job.JobId IN ( SELECT DISTINCT BaseJobId FROM BaseFiles WHERE JobId IN (19406,19507,19539,19564,19589,19614,19639,19664,19691)) OR Job.JobId IN (19406,19507,19539,19564,19589,19614,19639,19664,19691)) AND T1.JobTDate = Job.JobTDate AND Job.JobId = File.JobId AND T1.PathId = File.PathId AND T1.FilenameId = File.FilenameId ) AS T1 JOIN Filename ON (Filename.FilenameId = T1.FilenameId) JOIN Path ON (Path.PathId = T1.PathId) WHERE FileIndex > 0 ORDER BY T1.JobTDate, FileIndex ASC: ERR=database or disk is full -- Chandler / Systems Administrator Arizona Genomics Institute University Of Arizona www.genome.arizona.edu ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
