Re: Error when trying to re-use Bayes database from one server to another
John Hardin wrote: > On Fri, 12 Feb 2016, Kris Deugau wrote: > >> In general though, if you're operating at a scale where one server isn't >> enough to handle your SA load, you may want to start thinking about SQL >> for Bayes, which can be shared much more easily than pushing file-based >> Bayes data around. > > Or Redis (right?) MySQL on a RAMdisk is still performing well enough here that I haven't looked at that (still running 3.3.2, in fact; 3.4.x on a new VM platform has finally hit the top of my priority list this month), but if you're changing things already, yeah, that's probably a better path to start on. -kgd
[Solved] Re: Error when trying to re-use Bayes database from one server to another
On 13/02/16 18:58, Bill Cole wrote: On 13 Feb 2016, at 3:49, Sebastian Arcus wrote: Thank you. The donor machine has db42, db44 and db44 packages installed, Based on the question below, I'll assume the second db44 above was a typo for db48, i.e. a Berkeley DB v4.8.x package. Tangentially: that's a risky mess. It's a common problem but you should try to fix it to leave just one version, which probably means rebuilding a number of pieces of software. Using db48 for everything isn't a bad choice, despite the current version being 6.something, because there are still perfectly good pieces of software that use db4x but nothing later. In any case, you have a potentially fragile system there which may have different programs using diverse Berkeley DB versions which may be broken by otherwise routine updates. If you choose to leave a working system alone rather than proactively clean it up, be sure to while the recipient machine only db42 and db44. Would it be enough to install db48 on the recipient machine, or are there also any glue/library Perl modules involved which SA uses for db access and would need to be updated as well? Any answer to that has so many conditional branches that I'm unwilling to attempt a definitive one. You definitely need to install db48 on the recipient machine if you want it to be able to read hash files created elsewhere by db48. Depending on what other software is using db42 dn db44 there, installing db48 and doing nothing else MIGHT break something. Depending on how Perl was built and/or installed on that machine and how the various db* packages are installed it MIGHT be necessary to rebuild your core Perl package and/or non-core packages which may include BerkeleyDB or (probably not) DB_File and maybe (but most likely not) SpamAssassin itself. Figuring out what exactly depends on which package on a specific system (which you've not described in any detail) is an opportunity to exercise your core system administration skills :). Thank you everybody who pitched in with suggestions. Just to confirm that in the end I decided not to mess too much with a working system and didn't upgrade to db48 on the older system. I went down the route of backing up and restoring the bayes database using sa-learn - which worked perfectly fine. There is still the question of the initial sa-learn error message which started all this. In my opinion it looks like a bug - as it says "missing file" - which is clearly not the case. Something more helpful such as "can't decode file, possible wrong format" - or anything else along those lines would be more relevant and helpful. Should I log this as a bug somewhere?
Re: Error when trying to re-use Bayes database from one server to another
On 13/02/16 18:58, Bill Cole wrote: On 13 Feb 2016, at 3:49, Sebastian Arcus wrote: Thank you. The donor machine has db42, db44 and db44 packages installed, Based on the question below, I'll assume the second db44 above was a typo for db48, i.e. a Berkeley DB v4.8.x package. Yes - sorry, you are right Tangentially: that's a risky mess. It's a common problem but you should try to fix it to leave just one version, which probably means rebuilding a number of pieces of software. Slackware current comes with all three versions - that's a default install and I checked the package list at Slackware.com. I'm afraid I'm not sure why, but I assume there is some logic to it - as the package choice in Slackware always seems to have some reasoning behind it. I also don't know why Slackware current doesn't include version 6.x (or even 4.6) - maybe something to do with the current politics of Oracle - or some other technical reason. Using db48 for everything isn't a bad choice, despite the current version being 6.something, because there are still perfectly good pieces of software that use db4x but nothing later. In any case, you have a potentially fragile system there which may have different programs using diverse Berkeley DB versions which may be broken by otherwise routine updates. If you choose to leave a working system alone rather than proactively clean it up, be sure to while the recipient machine only db42 and db44. Would it be enough to install db48 on the recipient machine, or are there also any glue/library Perl modules involved which SA uses for db access and would need to be updated as well? Any answer to that has so many conditional branches that I'm unwilling to attempt a definitive one. You definitely need to install db48 on the recipient machine if you want it to be able to read hash files created elsewhere by db48. Depending on what other software is using db42 dn db44 there, installing db48 and doing nothing else MIGHT break something. Depending on how Perl was built and/or installed on that machine and how the various db* packages are installed it MIGHT be necessary to rebuild your core Perl package and/or non-core packages which may include BerkeleyDB or (probably not) DB_File and maybe (but most likely not) SpamAssassin itself. Figuring out what exactly depends on which package on a specific system (which you've not described in any detail) is an opportunity to exercise your core system administration skills :). Thank you :-)
Re: Error when trying to re-use Bayes database from one server to another
On 13 Feb 2016, at 3:49, Sebastian Arcus wrote: Thank you. The donor machine has db42, db44 and db44 packages installed, Based on the question below, I'll assume the second db44 above was a typo for db48, i.e. a Berkeley DB v4.8.x package. Tangentially: that's a risky mess. It's a common problem but you should try to fix it to leave just one version, which probably means rebuilding a number of pieces of software. Using db48 for everything isn't a bad choice, despite the current version being 6.something, because there are still perfectly good pieces of software that use db4x but nothing later. In any case, you have a potentially fragile system there which may have different programs using diverse Berkeley DB versions which may be broken by otherwise routine updates. If you choose to leave a working system alone rather than proactively clean it up, be sure to while the recipient machine only db42 and db44. Would it be enough to install db48 on the recipient machine, or are there also any glue/library Perl modules involved which SA uses for db access and would need to be updated as well? Any answer to that has so many conditional branches that I'm unwilling to attempt a definitive one. You definitely need to install db48 on the recipient machine if you want it to be able to read hash files created elsewhere by db48. Depending on what other software is using db42 dn db44 there, installing db48 and doing nothing else MIGHT break something. Depending on how Perl was built and/or installed on that machine and how the various db* packages are installed it MIGHT be necessary to rebuild your core Perl package and/or non-core packages which may include BerkeleyDB or (probably not) DB_File and maybe (but most likely not) SpamAssassin itself. Figuring out what exactly depends on which package on a specific system (which you've not described in any detail) is an opportunity to exercise your core system administration skills :).
Re: Error when trying to re-use Bayes database from one server to another
Am 13.02.2016 um 16:56 schrieb Antony Stone: On Saturday 13 February 2016 at 16:50:56, Reindl Harald wrote: a different company with it's own infrastructure has no business to ssh-tunneling or access *my server* in any other way directly DIFFERENT NETWORKS DIFFERENT INFRASTRUCTURES DIFFERENT OWNERS DIFFERENT ADMINS NO DIRECT CONNECTION BETWEEN THE COMPANIES WE HAVE MAIL CUSTOMERS THEY HAVE MAIL CUSTOMERS DIFFERENT CUSTOMERS However, what about when those two companies have a customer-supplier relationship? The supplier is providing the service of a spam-filtering/identification database to the customer, for use on their own machines. Sounds perfectly reasonable to share this as a commercial service, to me that's exactly what happens, fetch the bayes over a webservice when the checksum has changed - nobody on both sides want to use redis for a million reasons the point is that "use redis" isn't always the answer signature.asc Description: OpenPGP digital signature
Re: Error when trying to re-use Bayes database from one server to another
On Saturday 13 February 2016 at 16:50:56, Reindl Harald wrote: > a different company with it's own infrastructure has no business to > ssh-tunneling or access *my server* in any other way directly > > DIFFERENT NETWORKS > DIFFERENT INFRASTRUCTURES > DIFFERENT OWNERS > DIFFERENT ADMINS > > NO DIRECT CONNECTION BETWEEN THE COMPANIES > WE HAVE MAIL CUSTOMERS > THEY HAVE MAIL CUSTOMERS > DIFFERENT CUSTOMERS However, what about when those two companies have a customer-supplier relationship? The supplier is providing the service of a spam-filtering/identification database to the customer, for use on their own machines. Sounds perfectly reasonable to share this as a commercial service, to me. Antony. -- I love deadlines. I love the whooshing noise they make as they go by. - Douglas Noel Adams Please reply to the list; please *don't* CC me.
Re: Error when trying to re-use Bayes database from one server to another
Am 13.02.2016 um 16:46 schrieb Marc Perkel: On 02/13/16 07:25, Reindl Harald wrote: Am 13.02.2016 um 15:59 schrieb Marc Perkel: On 02/13/16 00:42, Reindl Harald wrote: Am 13.02.2016 um 02:56 schrieb Marc Perkel: For what it's worth - just used Redis. Redis is the only thing that's worked reliably for me you can't use Redis when it comes to different servers in different networks for different clients I'm filtering 5000 domains using a single redis server and 4 SA servers looks like you refused to understand 'different networks' it's fine in your infrastructure but it won't work in the cases we have in real life where another company with independent infrastructure fetchs our bayes in context of a subscription over webservices, move the files in a temp-folder and train own samples before replace the local bayes with the result reason? 2510401 tokens with dump in and dump out is horrible slow when the number of local samples is around 1000 messages versus 82 messages we have feeded since 2014 or would you open your redis server for 3rd parties on the WAN? I'm using SSH tunneling to keep my redis private. Maybe I'm not understanding what you are trying to do? no idea how to explain it better *and* friendly as above with all details explained already a different company with it's own infrastructure has no business to ssh-tunneling or access *my server* in any other way directly DIFFERENT NETWORKS DIFFERENT INFRASTRUCTURES DIFFERENT OWNERS DIFFERENT ADMINS NO DIRECT CONNECTION BETWEEN THE COMPANIES WE HAVE MAIL CUSTOMERS THEY HAVE MAIL CUSTOMERS DIFFERENT CUSTOMERS signature.asc Description: OpenPGP digital signature
Re: Error when trying to re-use Bayes database from one server to another
On 02/13/16 07:25, Reindl Harald wrote: Am 13.02.2016 um 15:59 schrieb Marc Perkel: On 02/13/16 00:42, Reindl Harald wrote: Am 13.02.2016 um 02:56 schrieb Marc Perkel: For what it's worth - just used Redis. Redis is the only thing that's worked reliably for me you can't use Redis when it comes to different servers in different networks for different clients BDB works fine and relieable, at least without autolearning and autoexpire and having the bayes-db path read-only for the running spamd with namespaces 0 60388SPAM 0 21651HAM 02510401TOKEN insgesamt 73M -rw--- 1 sa-milt sa-milt 10M 2016-02-13 09:12 bayes_seen -rw--- 1 sa-milt sa-milt 81M 2016-02-13 09:12 bayes_toks I'm filtering 5000 domains using a single redis server and 4 SA servers looks like you refused to understand 'different networks' it's fine in your infrastructure but it won't work in the cases we have in real life where another company with independent infrastructure fetchs our bayes in context of a subscription over webservices, move the files in a temp-folder and train own samples before replace the local bayes with the result reason? 2510401 tokens with dump in and dump out is horrible slow when the number of local samples is around 1000 messages versus 82 messages we have feeded since 2014 or would you open your redis server for 3rd parties on the WAN? I'm using SSH tunneling to keep my redis private. Maybe I'm not understanding what you are trying to do? -- Marc Perkel - Sales/Support supp...@junkemailfilter.com http://www.junkemailfilter.com Junk Email Filter dot com 415-992-3400
Re: Error when trying to re-use Bayes database from one server to another
Am 13.02.2016 um 15:59 schrieb Marc Perkel: On 02/13/16 00:42, Reindl Harald wrote: Am 13.02.2016 um 02:56 schrieb Marc Perkel: For what it's worth - just used Redis. Redis is the only thing that's worked reliably for me you can't use Redis when it comes to different servers in different networks for different clients BDB works fine and relieable, at least without autolearning and autoexpire and having the bayes-db path read-only for the running spamd with namespaces 0 60388SPAM 0 21651HAM 02510401TOKEN insgesamt 73M -rw--- 1 sa-milt sa-milt 10M 2016-02-13 09:12 bayes_seen -rw--- 1 sa-milt sa-milt 81M 2016-02-13 09:12 bayes_toks I'm filtering 5000 domains using a single redis server and 4 SA servers looks like you refused to understand 'different networks' it's fine in your infrastructure but it won't work in the cases we have in real life where another company with independent infrastructure fetchs our bayes in context of a subscription over webservices, move the files in a temp-folder and train own samples before replace the local bayes with the result reason? 2510401 tokens with dump in and dump out is horrible slow when the number of local samples is around 1000 messages versus 82 messages we have feeded since 2014 or would you open your redis server for 3rd parties on the WAN? signature.asc Description: OpenPGP digital signature
Re: Error when trying to re-use Bayes database from one server to another
On 02/13/16 00:42, Reindl Harald wrote: Am 13.02.2016 um 02:56 schrieb Marc Perkel: For what it's worth - just used Redis. Redis is the only thing that's worked reliably for me you can't use Redis when it comes to different servers in different networks for different clients BDB works fine and relieable, at least without autolearning and autoexpire and having the bayes-db path read-only for the running spamd with namespaces 0 60388SPAM 0 21651HAM 02510401TOKEN insgesamt 73M -rw--- 1 sa-milt sa-milt 10M 2016-02-13 09:12 bayes_seen -rw--- 1 sa-milt sa-milt 81M 2016-02-13 09:12 bayes_toks I'm filtering 5000 domains using a single redis server and 4 SA servers. -- Marc Perkel - Sales/Support supp...@junkemailfilter.com http://www.junkemailfilter.com Junk Email Filter dot com 415-992-3400
Re: Error when trying to re-use Bayes database from one server to another
On 13/02/16 04:32, Bill Cole wrote: On 12 Feb 2016, at 17:34, Sebastian Arcus wrote: Thanks for that suggestion. I think we might be getting somewhere. On original machine: #file bayes_seen bayes_seen: Berkeley DB (Hash, version 9, native byte-order) # file bayes_toks bayes_toks: Berkeley DB (Hash, version 9, native byte-order) On the receiver machine, but with bayes files created locally: #file bayes_seen bayes_seen: Berkeley DB (Hash, version 8, native byte-order) # file bayes_toks bayes_toks: Berkeley DB (Hash, version 8, native byte-order) Could the hash version account for the errors I am seeing? Absolutely. The BDB hash storage version number only changes when a change is NOT backwards-compatible, i.e. *BY DESIGN* a library version that creates v8 files cannot read v9 files. If my recollection is correct, the v8->9 change was in BDB 4.6 and actually provided substantial performance improvements. You probably want to upgrade BDB and anything using it on the machine with the old version. Thank you. The donor machine has db42, db44 and db44 packages installed, while the recipient machine only db42 and db44. Would it be enough to install db48 on the recipient machine, or are there also any glue/library Perl modules involved which SA uses for db access and would need to be updated as well?
Re: Error when trying to re-use Bayes database from one server to another
Am 13.02.2016 um 02:56 schrieb Marc Perkel: For what it's worth - just used Redis. Redis is the only thing that's worked reliably for me you can't use Redis when it comes to different servers in different networks for different clients BDB works fine and relieable, at least without autolearning and autoexpire and having the bayes-db path read-only for the running spamd with namespaces 0 60388SPAM 0 21651HAM 02510401TOKEN insgesamt 73M -rw--- 1 sa-milt sa-milt 10M 2016-02-13 09:12 bayes_seen -rw--- 1 sa-milt sa-milt 81M 2016-02-13 09:12 bayes_toks signature.asc Description: OpenPGP digital signature
Re: Error when trying to re-use Bayes database from one server to another
Am 13.02.2016 um 02:46 schrieb Benny Pedersen: On 12. feb. 2016 20.06.52 Marc Perkel wrote: # ls -l /var/spool/spamd/bayes/ Set bayes path without bayes why do you always give wrong advises? https://wiki.apache.org/spamassassin/SiteWideBayesSetup Note that the argument to bayes_path is a combination of a directory (/var/spamassassin/bayes_db/) and a filename prefix (bayes) bayes_path /var/lib/spamass-milter/.spamassassin/bayes [root@mail-gw:~]$ ls /var/lib/spamass-milter/.spamassassin/ insgesamt 73M -rw--- 1 sa-milt sa-milt 10M 2016-02-13 09:12 bayes_seen -rw--- 1 sa-milt sa-milt 81M 2016-02-13 09:12 bayes_toks -rw--- 1 sa-milt sa-milt 18 2016-01-25 01:15 user_prefs bayes: cannot open bayes databases /var/spool/spamd/bayes/bayes_* R/W: Remove bayes from local.cf again a completly wrong advise, in a site-wide setup rsyned over different machines you *should* make sure that the path where your rsync target lives is hardwired in the config signature.asc Description: OpenPGP digital signature
Re: Error when trying to re-use Bayes database from one server to another
On 12 Feb 2016, at 17:34, Sebastian Arcus wrote: Thanks for that suggestion. I think we might be getting somewhere. On original machine: #file bayes_seen bayes_seen: Berkeley DB (Hash, version 9, native byte-order) # file bayes_toks bayes_toks: Berkeley DB (Hash, version 9, native byte-order) On the receiver machine, but with bayes files created locally: #file bayes_seen bayes_seen: Berkeley DB (Hash, version 8, native byte-order) # file bayes_toks bayes_toks: Berkeley DB (Hash, version 8, native byte-order) Could the hash version account for the errors I am seeing? Absolutely. The BDB hash storage version number only changes when a change is NOT backwards-compatible, i.e. *BY DESIGN* a library version that creates v8 files cannot read v9 files. If my recollection is correct, the v8->9 change was in BDB 4.6 and actually provided substantial performance improvements. You probably want to upgrade BDB and anything using it on the machine with the old version.
Re: Error when trying to re-use Bayes database from one server to another
For what it's worth - just used Redis. Redis is the only thing that's worked reliably for me.
Re: Error when trying to re-use Bayes database from one server to another
On 12. feb. 2016 20.06.52 Marc Perkel wrote: # ls -l /var/spool/spamd/bayes/ Set bayes path without bayes bayes: cannot open bayes databases /var/spool/spamd/bayes/bayes_* R/W: Remove bayes from local.cf Sent with AquaMail for Android http://www.aqua-mail.com
Re: Error when trying to re-use Bayes database from one server to another
On 12/02/16 21:40, Kris Deugau wrote: Sebastian Arcus wrote: On 12/02/16 20:31, Antony Stone wrote: On Friday 12 February 2016 at 17:29:23, Sebastian Arcus wrote: As per advice from this list, I have been re-using my bayes databases on several different servers running SA. On one of the servers though, the database is not accepted. Are the servers all the same distro, release and version? Are they all up to date (or at least, all equally up to date as each other)? No - there are definitely variations in kernel versions and other software versions. So far I have only made sure the SA version is the same. Any possible culprit comes to mind in terms of a certain library or software that might cause this? Standard on-disk Bayes files are (mostly) BerkeleyDB, but there are a couple of other possible back ends that could be in use depending on OS release and update level. If you have OS variations you may have different BDB versions (or even a completely different type of hash database file; look up AnyDBM_File), and you won't be able to share your Bayes by copying files around. Try running "file /path/to/bayes/files/*" on your source and problem machines (the latter with locally-created Bayes files that don't show the access problem) and see what comes up. If updating the problem machine (or updating all the rest of them, if the problem machine is "newer") is a problem for any reason, you might try sa-learn --dump to "back up" Bayes on your source machine, then sa-learn --restore to reload the data on another machine. In general though, if you're operating at a scale where one server isn't enough to handle your SA load, you may want to start thinking about SQL for Bayes, which can be shared much more easily than pushing file-based Bayes data around. Probably a good suggestion, but the only reason several servers are involved is because they belong to different clients and are on different sites - the load is not a factor. But thank you for suggesting it.
Re: Error when trying to re-use Bayes database from one server to another
On 12/02/16 21:40, Kris Deugau wrote: Sebastian Arcus wrote: On 12/02/16 20:31, Antony Stone wrote: On Friday 12 February 2016 at 17:29:23, Sebastian Arcus wrote: As per advice from this list, I have been re-using my bayes databases on several different servers running SA. On one of the servers though, the database is not accepted. Are the servers all the same distro, release and version? Are they all up to date (or at least, all equally up to date as each other)? No - there are definitely variations in kernel versions and other software versions. So far I have only made sure the SA version is the same. Any possible culprit comes to mind in terms of a certain library or software that might cause this? Standard on-disk Bayes files are (mostly) BerkeleyDB, but there are a couple of other possible back ends that could be in use depending on OS release and update level. If you have OS variations you may have different BDB versions (or even a completely different type of hash database file; look up AnyDBM_File), and you won't be able to share your Bayes by copying files around. Try running "file /path/to/bayes/files/*" on your source and problem machines (the latter with locally-created Bayes files that don't show the access problem) and see what comes up. Thanks for that suggestion. I think we might be getting somewhere. On original machine: #file bayes_seen bayes_seen: Berkeley DB (Hash, version 9, native byte-order) # file bayes_toks bayes_toks: Berkeley DB (Hash, version 9, native byte-order) On the receiver machine, but with bayes files created locally: #file bayes_seen bayes_seen: Berkeley DB (Hash, version 8, native byte-order) # file bayes_toks bayes_toks: Berkeley DB (Hash, version 8, native byte-order) Could the hash version account for the errors I am seeing? So far it seems to be the only meaningful difference.
Re: Error when trying to re-use Bayes database from one server to another
On Fri, 12 Feb 2016, Kris Deugau wrote: In general though, if you're operating at a scale where one server isn't enough to handle your SA load, you may want to start thinking about SQL for Bayes, which can be shared much more easily than pushing file-based Bayes data around. Or Redis (right?) -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- You can't reason a person out of a position if he didn't use reason to get there in the first place. -- Kristopher, at Marko's --- Today: Abraham Lincoln's and Charles Darwin's 207th Birthdays
Re: Error when trying to re-use Bayes database from one server to another
Am 12.02.2016 um 22:40 schrieb Kris Deugau: In general though, if you're operating at a scale where one server isn't enough to handle your SA load, you may want to start thinking about SQL for Bayes, which can be shared much more easily than pushing file-based Bayes data around no, that creates a single point of failure without introdcue another layer of complexity - rsync the DBD bayes files around scales and ends in independent machines signature.asc Description: OpenPGP digital signature
Re: Error when trying to re-use Bayes database from one server to another
Am 12.02.2016 um 22:36 schrieb Martin Gregorie: On Fri, 2016-02-12 at 20:58 +, Sebastian Arcus wrote: I'm on Slackware without selinux. I'm not even convinced it is a permission problem at all. If I restrict the permissions of the files on purpose, it actually reports: In that case ignore my last post, which assumed it was an SElinux problem. Could the problem be down to differing versions of the bayes database manager? If so, it may be worth letting SA set up an empty Bayes database and using the backup tool to make a backup on the source system in a version-agnostic format, e.g as a CSV file, and then restoring it to the new system with its restore tool. that won't explain "file don't exist" messages and the "bayes database manager" don't matter, it's just a BDB database perfectly working between different SA versions while the OP made clear SA is the same version on all machines a "spamassassin -D < some-message.eml" with user spamd is using would be helpful as statet in the original logs signature.asc Description: OpenPGP digital signature
Re: Error when trying to re-use Bayes database from one server to another
Sebastian Arcus wrote: > On 12/02/16 20:31, Antony Stone wrote: >> On Friday 12 February 2016 at 17:29:23, Sebastian Arcus wrote: >> >>> As per advice from this list, I have been re-using my bayes databases on >>> several different servers running SA. On one of the servers though, the >>> database is not accepted. >> Are the servers all the same distro, release and version? >> >> Are they all up to date (or at least, all equally up to date as each >> other)? > No - there are definitely variations in kernel versions and other > software versions. So far I have only made sure the SA version is the > same. Any possible culprit comes to mind in terms of a certain library > or software that might cause this? Standard on-disk Bayes files are (mostly) BerkeleyDB, but there are a couple of other possible back ends that could be in use depending on OS release and update level. If you have OS variations you may have different BDB versions (or even a completely different type of hash database file; look up AnyDBM_File), and you won't be able to share your Bayes by copying files around. Try running "file /path/to/bayes/files/*" on your source and problem machines (the latter with locally-created Bayes files that don't show the access problem) and see what comes up. If updating the problem machine (or updating all the rest of them, if the problem machine is "newer") is a problem for any reason, you might try sa-learn --dump to "back up" Bayes on your source machine, then sa-learn --restore to reload the data on another machine. In general though, if you're operating at a scale where one server isn't enough to handle your SA load, you may want to start thinking about SQL for Bayes, which can be shared much more easily than pushing file-based Bayes data around. -kgd
Re: Error when trying to re-use Bayes database from one server to another
On Fri, 2016-02-12 at 20:58 +, Sebastian Arcus wrote: > I'm on Slackware without selinux. I'm not even convinced it is a > permission problem at all. If I restrict the permissions of the files > on purpose, it actually reports: > In that case ignore my last post, which assumed it was an SElinux problem. Could the problem be down to differing versions of the bayes database manager? If so, it may be worth letting SA set up an empty Bayes database and using the backup tool to make a backup on the source system in a version-agnostic format, e.g as a CSV file, and then restoring it to the new system with its restore tool. Martin
Re: Error when trying to re-use Bayes database from one server to another
On Fri, 2016-02-12 at 15:49 -0500, Bowie Bailey wrote: > On 2/12/2016 3:45 PM, Sebastian Arcus wrote: > > On 12/02/16 20:31, Antony Stone wrote: > > > On Friday 12 February 2016 at 17:29:23, Sebastian Arcus wrote: > > > > > > > As per advice from this list, I have been re-using my bayes > > > > databases on > > > > several different servers running SA. On one of the servers > > > > though, the > > > > database is not accepted. > > > Are the servers all the same distro, release and version? > > > > > > Are they all up to date (or at least, all equally up to date as > > > each > > > other)? > > No - there are definitely variations in kernel versions and other > > software versions. So far I have only made sure the SA version is > > the > > same. Any possible culprit comes to mind in terms of a certain > > library > > or software that might cause this? > > > > The thing that always comes to mind when I see strange permission > problems is selinux. If you have it active, check to see if selinux > is blocking access to the file for some reason. > In my experience, as somebody who doesn't always leave files in their default positions, changing the absolute paths of files used by standard servers will almost always upset SELinux. For starters, set Selinux to permissive mode and reboot. Its default is Enforcing more and a reboot is accept a mode change. Running in permissive mode prevents SElinux from applying its rules to running processes but still generates violation reports. These are sent posted to the bug reporting tool. I'm running Fedora with XFCE, so names and actions will probably differ for other distros. Fedora with XFCE calls this tool Abrt and makes violations accessible from the XFCE menu bar. The violation report shows text explaining what the problem is and also suggesting what command should be run to fix it. Open a terminal window and run this command by copying and pasting it, preceded by 'sudo '. This usually fixes problems caused by moving files to a nonstandard location. Di this for all the reported violations. Check by rebooting again. If all is now clean, run in permissive mode for another few days to see that you've got the lot. If more errors are reported, rince, wash and repeat. When no more Selinux errors are appearing you can either continue running in permissive mode or switch back to enforcing mode and reboot to let that take effect. HTH Martin
Re: Error when trying to re-use Bayes database from one server to another
On 12/02/16 20:49, Bowie Bailey wrote: On 2/12/2016 3:45 PM, Sebastian Arcus wrote: On 12/02/16 20:31, Antony Stone wrote: On Friday 12 February 2016 at 17:29:23, Sebastian Arcus wrote: As per advice from this list, I have been re-using my bayes databases on several different servers running SA. On one of the servers though, the database is not accepted. Are the servers all the same distro, release and version? Are they all up to date (or at least, all equally up to date as each other)? No - there are definitely variations in kernel versions and other software versions. So far I have only made sure the SA version is the same. Any possible culprit comes to mind in terms of a certain library or software that might cause this? The thing that always comes to mind when I see strange permission problems is selinux. If you have it active, check to see if selinux is blocking access to the file for some reason. I'm on Slackware without selinux. I'm not even convinced it is a permission problem at all. If I restrict the permissions of the files on purpose, it actually reports: bayes: cannot open bayes databases /var/spool/spamd/bayes/bayes_* R/W: tie failed: Permission denied But if I reset them to correct permissions, I get: bayes: cannot open bayes databases /var/spool/spamd/bayes/bayes_* R/W: tie failed: No such file or directory That doesn't make sense. To me it looks like it doesn't like something about the content of those files, but is reporting missing files instead of what the real problem is.
Re: Error when trying to re-use Bayes database from one server to another
On 2/12/2016 3:45 PM, Sebastian Arcus wrote: On 12/02/16 20:31, Antony Stone wrote: On Friday 12 February 2016 at 17:29:23, Sebastian Arcus wrote: As per advice from this list, I have been re-using my bayes databases on several different servers running SA. On one of the servers though, the database is not accepted. Are the servers all the same distro, release and version? Are they all up to date (or at least, all equally up to date as each other)? No - there are definitely variations in kernel versions and other software versions. So far I have only made sure the SA version is the same. Any possible culprit comes to mind in terms of a certain library or software that might cause this? The thing that always comes to mind when I see strange permission problems is selinux. If you have it active, check to see if selinux is blocking access to the file for some reason. -- Bowie
Re: Error when trying to re-use Bayes database from one server to another
On 12/02/16 20:31, Antony Stone wrote: On Friday 12 February 2016 at 17:29:23, Sebastian Arcus wrote: As per advice from this list, I have been re-using my bayes databases on several different servers running SA. On one of the servers though, the database is not accepted. Are the servers all the same distro, release and version? Are they all up to date (or at least, all equally up to date as each other)? No - there are definitely variations in kernel versions and other software versions. So far I have only made sure the SA version is the same. Any possible culprit comes to mind in terms of a certain library or software that might cause this?
Re: Error when trying to re-use Bayes database from one server to another
Am 12.02.2016 um 21:31 schrieb Antony Stone: On Friday 12 February 2016 at 17:29:23, Sebastian Arcus wrote: As per advice from this list, I have been re-using my bayes databases on several different servers running SA. On one of the servers though, the database is not accepted. Are the servers all the same distro, release and version? Are they all up to date (or at least, all equally up to date as each other)? that's not the point sharing bayes here for many months from a Fedora 22/23 system to another company using Debian with perfect results and 250 tokens currently how it works? * publish a checksum of the current bayes_toks * fetch it when different * atomic mv after received the database signature.asc Description: OpenPGP digital signature
Re: Error when trying to re-use Bayes database from one server to another
On Friday 12 February 2016 at 17:29:23, Sebastian Arcus wrote: > As per advice from this list, I have been re-using my bayes databases on > several different servers running SA. On one of the servers though, the > database is not accepted. Are the servers all the same distro, release and version? Are they all up to date (or at least, all equally up to date as each other)? Antony. -- If the human brain were so simple that we could understand it, we'd be so simple that we couldn't. Please reply to the list; please *don't* CC me.
Re: Error when trying to re-use Bayes database from one server to another
On 12/02/16 19:14, Reindl Harald wrote: Am 12.02.2016 um 20:06 schrieb Marc Perkel: Any chance that the parent directory structure doesn't have enough permissions? The error message says it can't access it so there's your clue. Since the files themselves seem to have good permissions I would look at the directories. see previous mail - that was already verified looking closer "No such file or directory" is not a permission problem there was a hint "please re-run with -D" at least re-use bayes on different servers, even over different operating systems is no problem, or bayes is running on 3 own and 2 foreign machines for a long time now with great results I've checked and triple checked everything. Unless I'm missing something blindingly obvious, I don't think that error message is accurate. If I delete the bayes files and restart spamd, on running sa-learn, new ones are created in exactly the same place, with same name and same permissions - and they work fine. But the ones brought over from the other server don't work. PS - Regarding the "re-run with -D for more information" - I guess that message is slightly pointless, as it keeps on saying that even when you run it with "-D"
Re: Error when trying to re-use Bayes database from one server to another
Am 12.02.2016 um 20:06 schrieb Marc Perkel: Any chance that the parent directory structure doesn't have enough permissions? The error message says it can't access it so there's your clue. Since the files themselves seem to have good permissions I would look at the directories. see previous mail - that was already verified looking closer "No such file or directory" is not a permission problem there was a hint "please re-run with -D" at least re-use bayes on different servers, even over different operating systems is no problem, or bayes is running on 3 own and 2 foreign machines for a long time now with great results On 02/12/16 08:29, Sebastian Arcus wrote: As per advice from this list, I have been re-using my bayes databases on several different servers running SA. On one of the servers though, the database is not accepted. I re-transferred them several times over ssh, to make sure they were not corrupted. The database files are in the correct location, with correct permissions and owned by the correct user: # ls -l /var/spool/spamd/bayes/ total 5912 -rw-rw-rw- 1 spamd spamd 1310720 2016-02-09 08:42 bayes_seen -rw-rw-rw- 1 spamd spamd 4739072 2016-02-09 08:43 bayes_toks The version of SA on both the donor and receiving servers is 3.4.1. When I try to learn a new message on the receiving server (where I moved the bayes files), I get the following error: # su - spamd -c "/usr/bin/sa-learn -D --spam /New\ UnansweredSexHookup\ Request.eml" Feb 12 16:20:53.777 [12973] dbg: locker: mode is 438 Feb 12 16:20:53.778 [12973] dbg: locker: safe_lock: created /var/spool/spamd/bayes/bayes.lock.mdr-server.mdrinteriors.co.uk.12973 Feb 12 16:20:53.778 [12973] dbg: locker: safe_lock: trying to get lock on /var/spool/spamd/bayes/bayes with 0 retries Feb 12 16:20:53.778 [12973] dbg: locker: safe_lock: link to /var/spool/spamd/bayes/bayes.lock: link ok Feb 12 16:20:53.778 [12973] dbg: bayes: tie-ing to DB file R/W /var/spool/spamd/bayes/bayes_toks Feb 12 16:20:53.779 [12973] dbg: bayes: untie-ing DB file toks Feb 12 16:20:53.779 [12973] dbg: locker: safe_unlock: unlink /var/spool/spamd/bayes/bayes.lock bayes: cannot open bayes databases /var/spool/spamd/bayes/bayes_* R/W: tie failed: No such file or directory Learned tokens from 0 message(s) (1 message(s) examined) Feb 12 16:20:53.779 [12973] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x93106d0) implements 'learner_close', priority 0 ERROR: the Bayes learn function returned an error, please re-run with -D for more information at /usr/bin/sa-learn line 498 signature.asc Description: OpenPGP digital signature
Re: Error when trying to re-use Bayes database from one server to another
Any chance that the parent directory structure doesn't have enough permissions? The error message says it can't access it so there's your clue. Since the files themselves seem to have good permissions I would look at the directories. On 02/12/16 08:29, Sebastian Arcus wrote: As per advice from this list, I have been re-using my bayes databases on several different servers running SA. On one of the servers though, the database is not accepted. I re-transferred them several times over ssh, to make sure they were not corrupted. The database files are in the correct location, with correct permissions and owned by the correct user: # ls -l /var/spool/spamd/bayes/ total 5912 -rw-rw-rw- 1 spamd spamd 1310720 2016-02-09 08:42 bayes_seen -rw-rw-rw- 1 spamd spamd 4739072 2016-02-09 08:43 bayes_toks The version of SA on both the donor and receiving servers is 3.4.1. When I try to learn a new message on the receiving server (where I moved the bayes files), I get the following error: # su - spamd -c "/usr/bin/sa-learn -D --spam /New\ UnansweredSexHookup\ Request.eml" Feb 12 16:20:53.777 [12973] dbg: locker: mode is 438 Feb 12 16:20:53.778 [12973] dbg: locker: safe_lock: created /var/spool/spamd/bayes/bayes.lock.mdr-server.mdrinteriors.co.uk.12973 Feb 12 16:20:53.778 [12973] dbg: locker: safe_lock: trying to get lock on /var/spool/spamd/bayes/bayes with 0 retries Feb 12 16:20:53.778 [12973] dbg: locker: safe_lock: link to /var/spool/spamd/bayes/bayes.lock: link ok Feb 12 16:20:53.778 [12973] dbg: bayes: tie-ing to DB file R/W /var/spool/spamd/bayes/bayes_toks Feb 12 16:20:53.779 [12973] dbg: bayes: untie-ing DB file toks Feb 12 16:20:53.779 [12973] dbg: locker: safe_unlock: unlink /var/spool/spamd/bayes/bayes.lock bayes: cannot open bayes databases /var/spool/spamd/bayes/bayes_* R/W: tie failed: No such file or directory Learned tokens from 0 message(s) (1 message(s) examined) Feb 12 16:20:53.779 [12973] dbg: plugin: Mail::SpamAssassin::Plugin::Bayes=HASH(0x93106d0) implements 'learner_close', priority 0 ERROR: the Bayes learn function returned an error, please re-run with -D for more information at /usr/bin/sa-learn line 498. -- Marc Perkel - Sales/Support supp...@junkemailfilter.com http://www.junkemailfilter.com Junk Email Filter dot com 415-992-3400
Re: Error when trying to re-use Bayes database from one server to another
On 12/02/16 16:59, Reindl Harald wrote: Am 12.02.2016 um 17:29 schrieb Sebastian Arcus: As per advice from this list, I have been re-using my bayes databases on several different servers running SA. On one of the servers though, the database is not accepted. I re-transferred them several times over ssh, to make sure they were not corrupted. The database files are in the correct location, with correct permissions and owned by the correct user: # ls -l /var/spool/spamd/bayes/ total 5912 -rw-rw-rw- 1 spamd spamd 1310720 2016-02-09 08:42 bayes_seen -rw-rw-rw- 1 spamd spamd 4739072 2016-02-09 08:43 bayes_toks The version of SA on both the donor and receiving servers is 3.4.1. When I try to learn a new message on the receiving server (where I moved the bayes files), I get the following error: su - spamd stat /var stat /var/spool stat /var/spool/spamd stat /var/spool/spamd/bayes Linux is not like Windows - if ou don't have access to a parent folder you just don't have access Sorry - previous reply sent in HTML format by mistake: root@mdr-server:/# su - spamd No directory, logging in with HOME=/ spamd@mdr-server:/$ stat /var File: `/var' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 900h/2304dInode: 12 Links: 16 Access: (0755/drwxr-xr-x) Uid: (0/root) Gid: (0/ root) Access: 2016-01-18 09:28:23.0 + Modify: 2016-01-18 09:22:47.0 + Change: 2016-01-18 09:28:23.744774236 + spamd@mdr-server:/$ stat /var/spool File: `/var/spool' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 900h/2304dInode: 118 Links: 22 Access: (0755/drwxr-xr-x) Uid: (0/root) Gid: (0/ root) Access: 2015-02-03 14:28:33.0 + Modify: 2015-12-03 17:41:28.859794403 + Change: 2015-12-03 17:41:28.859794403 + spamd@mdr-server:/$ stat /var/spool/spamd File: `/var/spool/spamd' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 900h/2304dInode: 15473107Links: 3 Access: (0770/drwxrwx---) Uid: ( 1037/ spamd) Gid: ( 252/ spamd) Access: 2015-12-03 17:41:28.859794403 + Modify: 2015-12-03 17:41:32.011239989 + Change: 2015-12-03 17:46:59.187806044 + spamd@mdr-server:/$ stat /var/spool/spamd/bayes/ File: `/var/spool/spamd/bayes/' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 900h/2304dInode: 15473106Links: 3 Access: (0776/drwxrwxrw-) Uid: ( 1037/ spamd) Gid: ( 252/ spamd) Access: 2015-12-03 17:41:32.011239989 + Modify: 2016-02-12 16:20:53.778709980 + Change: 2016-02-12 16:20:53.778709980 +
Re: Error when trying to re-use Bayes database from one server to another
On 12/02/16 16:59, Reindl Harald wrote: Am 12.02.2016 um 17:29 schrieb Sebastian Arcus: As per advice from this list, I have been re-using my bayes databases on several different servers running SA. On one of the servers though, the database is not accepted. I re-transferred them several times over ssh, to make sure they were not corrupted. The database files are in the correct location, with correct permissions and owned by the correct user: # ls -l /var/spool/spamd/bayes/ total 5912 -rw-rw-rw- 1 spamd spamd 1310720 2016-02-09 08:42 bayes_seen -rw-rw-rw- 1 spamd spamd 4739072 2016-02-09 08:43 bayes_toks The version of SA on both the donor and receiving servers is 3.4.1. When I try to learn a new message on the receiving server (where I moved the bayes files), I get the following error: su - spamd stat /var stat /var/spool stat /var/spool/spamd stat /var/spool/spamd/bayes Linux is not like Windows - if ou don't have access to a parent folder you just don't have access root@mdr-server:/# su - spamd No directory, logging in with HOME=/ spamd@mdr-server:/$ stat /var File: `/var' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 900h/2304dInode: 12 Links: 16 Access: (0755/drwxr-xr-x) Uid: (0/root) Gid: (0/ root) Access: 2016-01-18 09:28:23.0 + Modify: 2016-01-18 09:22:47.0 + Change: 2016-01-18 09:28:23.744774236 + spamd@mdr-server:/$ stat /var/spool File: `/var/spool' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 900h/2304dInode: 118 Links: 22 Access: (0755/drwxr-xr-x) Uid: (0/root) Gid: (0/ root) Access: 2015-02-03 14:28:33.0 + Modify: 2015-12-03 17:41:28.859794403 + Change: 2015-12-03 17:41:28.859794403 + spamd@mdr-server:/$ stat /var/spool/spamd File: `/var/spool/spamd' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 900h/2304dInode: 15473107Links: 3 Access: (0770/drwxrwx---) Uid: ( 1037/ spamd) Gid: ( 252/ spamd) Access: 2015-12-03 17:41:28.859794403 + Modify: 2015-12-03 17:41:32.011239989 + Change: 2015-12-03 17:46:59.187806044 + spamd@mdr-server:/$ stat /var/spool/spamd/bayes/ File: `/var/spool/spamd/bayes/' Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 900h/2304dInode: 15473106Links: 3 Access: (0776/drwxrwxrw-) Uid: ( 1037/ spamd) Gid: ( 252/ spamd) Access: 2015-12-03 17:41:32.011239989 + Modify: 2016-02-12 16:20:53.778709980 + Change: 2016-02-12 16:20:53.778709980 +
Re: Error when trying to re-use Bayes database from one server to another
Am 12.02.2016 um 17:29 schrieb Sebastian Arcus: As per advice from this list, I have been re-using my bayes databases on several different servers running SA. On one of the servers though, the database is not accepted. I re-transferred them several times over ssh, to make sure they were not corrupted. The database files are in the correct location, with correct permissions and owned by the correct user: # ls -l /var/spool/spamd/bayes/ total 5912 -rw-rw-rw- 1 spamd spamd 1310720 2016-02-09 08:42 bayes_seen -rw-rw-rw- 1 spamd spamd 4739072 2016-02-09 08:43 bayes_toks The version of SA on both the donor and receiving servers is 3.4.1. When I try to learn a new message on the receiving server (where I moved the bayes files), I get the following error: su - spamd stat /var stat /var/spool stat /var/spool/spamd stat /var/spool/spamd/bayes Linux is not like Windows - if ou don't have access to a parent folder you just don't have access signature.asc Description: OpenPGP digital signature
Re: Error when trying to re-use Bayes database from one server to another
On Fri, 12 Feb 2016, Sebastian Arcus wrote: # ls -l /var/spool/spamd/bayes/ total 5912 -rw-rw-rw- 1 spamd spamd 1310720 2016-02-09 08:42 bayes_seen -rw-rw-rw- 1 spamd spamd 4739072 2016-02-09 08:43 bayes_toks When I try to learn a new message on the receiving server (where I moved the bayes files), I get the following error: bayes: cannot open bayes databases /var/spool/spamd/bayes/bayes_* R/W: tie failed: No such file or directory Directory permissions? Can the spamd user actually get to /var/spool/spamd/bayes/ ? -- John Hardin KA7OHZhttp://www.impsec.org/~jhardin/ jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79 --- ...to announce there must be no criticism of the President or to stand by the President right or wrong is not only unpatriotic and servile, but is morally treasonous to the American public. -- Theodore Roosevelt, 1918 --- Today: Abraham Lincoln's and Charles Darwin's 207th Birthdays
