Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
I'm not sure. I'm running v9.4.4 on the director. Most clients are running that version, although some are older (the oldest being v5.0.0 on a CentOS 6 legacy system, I think.) So if the error was introduced in v9.6.x I wouldn't be seeing it. My bacula.log goes back about a year and I only have one catreq.c error. I think it's unrelated: JobId 51404: Warning: catreq.c:619 MD5 digest not same FileIndex=-451567 as attributes FI=-451566 I'm not sure if it matters, but I'm using postgres as my catalog backend. On Tue, Feb 23, 2021 at 11:54 PM Uwe Schuerkamp wrote: > On Tue, Feb 23, 2021 at 05:52:02PM +0100, Josip Deanovic wrote: > > On Monday 2021-02-22 16:06:42 David Brodbeck wrote: > > > On Wed, Feb 17, 2021 at 6:12 AM Josip Deanovic > > > > > > wrote: > > > > It's interesting that the job had almost 2GB of attributes to > > > > despool. That's quite a large amount of attributes. > > > > > > I have a few jobs that despool that much, and one that despools over 18 > > > GB. The machine in question has 51 million files. > > > > Just to be sure... Are you talking about the attribute spool file that > > gets created in running directory and not about the spool file that gets > > created in the spool directory? > > > > I'd suspect he is talking about the attribute spool file. The size > sounds about right (1,9GB on my end for around 5 million files, 19GB > for his attribute spool with a tenfold amount of files in the backup job). > > I'm just wondering why David's job appears to be working fine without a > modification of the attribute spool size parameter in catreq.c as > outlined above... then again our job used to work for years without > any issues, things only started to go wrong on bacula 9.6.5 or > thereabouts. > > All the best, > > Uwe > > > > -- > Uwe Schürkamp | email: > Arvato Systems S4M GmbH | Sitz Köln | Amtsgericht Köln HRB 27038 > > > > > > > > > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > -- David Brodbeck (they/them) System Administrator, Department of Mathematics University of California, Santa Barbara ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
> On Wed, 24 Feb 2021 08:54:11 +0100, Uwe Schuerkamp said: > > On Tue, Feb 23, 2021 at 05:52:02PM +0100, Josip Deanovic wrote: > > On Monday 2021-02-22 16:06:42 David Brodbeck wrote: > > > On Wed, Feb 17, 2021 at 6:12 AM Josip Deanovic > > > > > > wrote: > > > > It's interesting that the job had almost 2GB of attributes to > > > > despool. That's quite a large amount of attributes. > > > > > > I have a few jobs that despool that much, and one that despools over 18 > > > GB. The machine in question has 51 million files. > > > > Just to be sure... Are you talking about the attribute spool file that > > gets created in running directory and not about the spool file that gets > > created in the spool directory? > > > > I'd suspect he is talking about the attribute spool file. The size > sounds about right (1,9GB on my end for around 5 million files, 19GB > for his attribute spool with a tenfold amount of files in the backup job). > > I'm just wondering why David's job appears to be working fine without a > modification of the attribute spool size parameter in catreq.c as > outlined above... The parameter (1000) in catreq.c is not the total size of the spool file. It is the maximum record size within the spool file, which is typically several orders of magnitude smaller than 1000. > then again our job used to work for years without > any issues, things only started to go wrong on bacula 9.6.5 or > thereabouts. That's interesting. There was a bug in 9.6.4 which changed the way that Bacula locks access to the catalog (but I'm not sure how it would cause this problem). Do you run concurrent jobs? If so, you could try updating to 9.6.7 plus this patch: https://www.bacula.org/git/cgit.cgi/bacula/commit/bacula/src/dird/dird.c?h=Branch-11.0=9ddd829414bb4b1f8330f71824f0ef5675e957a8 Don't be mislead by the change reason in this patch -- it changes much more than just the 'show catalog' command. This patch will probably work in 9.6.5 as well. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
On Tue, Feb 23, 2021 at 05:52:02PM +0100, Josip Deanovic wrote: > On Monday 2021-02-22 16:06:42 David Brodbeck wrote: > > On Wed, Feb 17, 2021 at 6:12 AM Josip Deanovic > > > > wrote: > > > It's interesting that the job had almost 2GB of attributes to > > > despool. That's quite a large amount of attributes. > > > > I have a few jobs that despool that much, and one that despools over 18 > > GB. The machine in question has 51 million files. > > Just to be sure... Are you talking about the attribute spool file that > gets created in running directory and not about the spool file that gets > created in the spool directory? > I'd suspect he is talking about the attribute spool file. The size sounds about right (1,9GB on my end for around 5 million files, 19GB for his attribute spool with a tenfold amount of files in the backup job). I'm just wondering why David's job appears to be working fine without a modification of the attribute spool size parameter in catreq.c as outlined above... then again our job used to work for years without any issues, things only started to go wrong on bacula 9.6.5 or thereabouts. All the best, Uwe -- Uwe Schürkamp | email: Arvato Systems S4M GmbH | Sitz Köln | Amtsgericht Köln HRB 27038 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
> On Thu, 18 Feb 2021 09:19:36 +0100, Uwe Schuerkamp said: > > On Wed, Feb 17, 2021 at 07:17:17PM +, Martin Simmons wrote: > > > On Wed, 17 Feb 2021 15:11:30 +0100, Josip Deanovic said: > > > > > > I am not sure if 5M files and directories could account for the > > > attribute spool file of 1.8GB in size. > > > > That is ~400 bytes per file, which is reasonable if the filenames are > > longish. > > > > __Martin > > > > Yep, and given that it's a windows fileserver some very quirky names > are probably par for the course there :-) > > I'm just wondering why I'm seeing this error at all because given > today's disk sizes I'm certain there are many such backup jobs of a > similar proportion run by bacula daily all over the world, and ours > ran fine for a long time, too before I saw this issue for the first > time. The "wanted" value is much larger than any expected value, so I think it must be caused by a bug and is probably not related to the size of the backup. In fact, the value 1969368434 looks like a fragment of a string because it decodes to the ASCII characters "r-bu" (or maybe "ub-r"). It might also be caused by filesystem corruption on the spool disk, so you could check this with fsck. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
On Monday 2021-02-22 16:06:42 David Brodbeck wrote: > On Wed, Feb 17, 2021 at 6:12 AM Josip Deanovic > > wrote: > > It's interesting that the job had almost 2GB of attributes to > > despool. That's quite a large amount of attributes. > > I have a few jobs that despool that much, and one that despools over 18 > GB. The machine in question has 51 million files. Just to be sure... Are you talking about the attribute spool file that gets created in running directory and not about the spool file that gets created in the spool directory? Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
On Wed, Feb 17, 2021 at 6:12 AM Josip Deanovic wrote: > It's interesting that the job had almost 2GB of attributes to > despool. That's quite a large amount of attributes. > I have a few jobs that despool that much, and one that despools over 18 GB. The machine in question has 51 million files. -- David Brodbeck (they/them) System Administrator, Department of Mathematics University of California, Santa Barbara ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
On Wed, Feb 17, 2021 at 07:17:17PM +, Martin Simmons wrote: > > On Wed, 17 Feb 2021 15:11:30 +0100, Josip Deanovic said: > > > > I am not sure if 5M files and directories could account for the > > attribute spool file of 1.8GB in size. > > That is ~400 bytes per file, which is reasonable if the filenames are longish. > > __Martin > Yep, and given that it's a windows fileserver some very quirky names are probably par for the course there :-) I'm just wondering why I'm seeing this error at all because given today's disk sizes I'm certain there are many such backup jobs of a similar proportion run by bacula daily all over the world, and ours ran fine for a long time, too before I saw this issue for the first time. All the best, Uwe -- Uwe Schürkamp | email: Arvato Systems S4M GmbH | Sitz Köln | Amtsgericht Köln HRB 27038 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
> On Wed, 17 Feb 2021 15:11:30 +0100, Josip Deanovic said: > > On Tuesday 2021-02-16 12:22:51 Uwe Schuerkamp wrote: > > Hi folks, > > > > thanks for all your suggestions. I compiled 9.6.7 from source and > > "patched" catreq.c to increase the maximum attribute spool size to 3x > > the original value. > > When mentioning the original value, are you referring to 1000, > the hardcoded value) or to the 1969368434 bytes which was the reported > value when the job failed? > > It's interesting that the job had almost 2GB of attributes to > despool. That's quite a large amount of attributes. > > I am not sure if 5M files and directories could account for the > attribute spool file of 1.8GB in size. That is ~400 bytes per file, which is reasonable if the filenames are longish. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
On Tuesday 2021-02-16 12:22:51 Uwe Schuerkamp wrote: > Hi folks, > > thanks for all your suggestions. I compiled 9.6.7 from source and > "patched" catreq.c to increase the maximum attribute spool size to 3x > the original value. When mentioning the original value, are you referring to 1000, the hardcoded value) or to the 1969368434 bytes which was the reported value when the job failed? It's interesting that the job had almost 2GB of attributes to despool. That's quite a large amount of attributes. I am not sure if 5M files and directories could account for the attribute spool file of 1.8GB in size. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
Hi all, On Tue, Feb 16, 2021 at 12:32:18PM +, Martin Simmons wrote: > Why 3x? The value in tne message is almost 197x the original, but is almost > certainly a junk value anyway. > sorry, I meant the value that bacula complained about not being able to allocate upped to 30... instead of 19..., I hope that clears things up. > If you have the job log then it would be useful to see it (at least the line > that contains "Sending spooled attrs to the Director..." which shows number of > bytes). > Here's the job log entry: ## 14-Feb 11:21 deniol2199-sd JobId 61332: Sending spooled attrs to the Director. Despooling 1,856,339,469 bytes ... 14-Feb 11:29 deniol2199-dir JobId 61332: Fatal error: catreq.c:762 fread attr spool error. Wanted 1969368434 bytes, maximum permitted 1000 bytes 14-Feb 11:29 deniol2199-dir JobId 61332: Error: Bacula deniol2199-dir 9.6.5 (11 Jun20): Build OS: x86_64-pc-linux-gnu redhat ## > > Also check the space in the Storage daemon's WorkingDirectory as Eric said. The storage daemon runs on the same system as the director. All the best & thanks for your help, Uwe ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
> On Tue, 16 Feb 2021 12:22:51 +0100, Uwe Schuerkamp said: > > Hi folks, > > thanks for all your suggestions. I compiled 9.6.7 from source and > "patched" catreq.c to increase the maximum attribute spool size to 3x > the original value. Why 3x? The value in tne message is almost 197x the original, but is almost certainly a junk value anyway. If you have the job log then it would be useful to see it (at least the line that contains "Sending spooled attrs to the Director..." which shows number of bytes). > Sadly I don't have the time to debug this error on a deeper level at > the moment, disk space is plenty on the director (around 400GB free) > so I doub't it's an issue with bacula's "var"-Directory filling up. Also check the space in the Storage daemon's WorkingDirectory as Eric said. __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
Hi folks, thanks for all your suggestions. I compiled 9.6.7 from source and "patched" catreq.c to increase the maximum attribute spool size to 3x the original value. Sadly I don't have the time to debug this error on a deeper level at the moment, disk space is plenty on the director (around 400GB free) so I doub't it's an issue with bacula's "var"-Directory filling up. I'll observe the behaviour during next weekend's "full" backup and report back. All the best, Uwe -- Uwe Schürkamp | email: ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
Hello Uwe, On 2/15/21 09:56, Uwe Schuerkamp wrote: > Hi folks, > > during a largish backup job (3,8TB windows fileserver, about 5m files > & directories) I'm seeing this error when the attributes are being > despooled once the job completes: > > Fatal error: catreq.c:762 fread attr spool error. Wanted 1969368434 > bytes, maximum permitted 1000 bytes It seems that an invalid message was found in the spool file, this size is the expected size of the message, that is rarely more than few hundred bytes. It can be some error message for example (low disk space?). You might want to diagnose in the Storage daemon when we write to thespool file and track when the msglen is > 100 for example. (Display the packet in the trace file for example. Hope it helps! Best Regards, Eric > I've searched the mailing list and the documentation for a possible > variable related to attribute spooling, but I can only find options > that regulate disk / job spooling. > > I'd be most grateful if someone could direct me to the correct > configuration parameter.[1] > > We're using bacula 9.6.5 compiled from source on CentOS7. > > Thanks in advance & all the best, > > Uwe > > [1] I'd also have no problem with patching catreq.c and recompiling, > but hopefully there's a supported way of adjusting this variable. > ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
On Monday 2021-02-15 18:47:35 Martin Simmons wrote: > > On Mon, 15 Feb 2021 09:56:33 +0100, Uwe Schuerkamp said: > > Hi folks, > > > > during a largish backup job (3,8TB windows fileserver, about 5m files > > & directories) I'm seeing this error when the attributes are being > > despooled once the job completes: > > > > Fatal error: catreq.c:762 fread attr spool error. Wanted 1969368434 > > bytes, maximum permitted 1000 bytes > > How large is the attribute spool file? > > This could be a bug. > > Is it repeatable? If so, does it repeat with this same number > (1969368434)? > > __Martin I don't know why some of my e-mails don't reach the e-mail list. I wrote today that the size of 1000 is hardcoded i nthe source of catreq.c. Regards! -- Josip Deanovic ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
> On Mon, 15 Feb 2021 09:56:33 +0100, Uwe Schuerkamp said: > > Hi folks, > > during a largish backup job (3,8TB windows fileserver, about 5m files > & directories) I'm seeing this error when the attributes are being > despooled once the job completes: > > Fatal error: catreq.c:762 fread attr spool error. Wanted 1969368434 > bytes, maximum permitted 1000 bytes How large is the attribute spool file? This could be a bug. Is it repeatable? If so, does it repeat with this same number (1969368434)? __Martin ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] Error on despooling attributes in catreq.c: wanted xxx bytes, maximum permitted 10000000 bytes
Hi folks, during a largish backup job (3,8TB windows fileserver, about 5m files & directories) I'm seeing this error when the attributes are being despooled once the job completes: Fatal error: catreq.c:762 fread attr spool error. Wanted 1969368434 bytes, maximum permitted 1000 bytes I've searched the mailing list and the documentation for a possible variable related to attribute spooling, but I can only find options that regulate disk / job spooling. I'd be most grateful if someone could direct me to the correct configuration parameter.[1] We're using bacula 9.6.5 compiled from source on CentOS7. Thanks in advance & all the best, Uwe [1] I'd also have no problem with patching catreq.c and recompiling, but hopefully there's a supported way of adjusting this variable. -- Uwe Schürkamp | email: Arvato Systems S4M GmbH | Sitz Köln | Amtsgericht Köln HRB 27038 ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
