Re: SA fails to search specified DATADIR for Distribution files

[JT DeLys]

FYI -- trunk/3.3.0 updates are now working.




Thanks :-)

Although, I think I'll also take earlier advice & 'stay here' (a stable &
working 3.2.x) for a bit.



--
Thanks,

   JTDeLys

Re: SA fails to search specified DATADIR for Distribution files

[Daryl C. W. O'Shea]

JT DeLys wrote:

If you were to use 3.2 it would work.


I've already 'downgraded' to svn 3.2-branch, and you're right - It works!

Reverting the changes that broke 3.3 updates is on my list of things 
to do.


Great. Thanks.


FYI -- trunk/3.3.0 updates are now working.

Daryl

Re: SA fails to search specified DATADIR for Distribution files

[Theo Van Dinter]

On Wed, Jun 27, 2007 at 04:30:12PM +0100, Nix wrote:
> But I'm glad to hear that it's meant to work and is working :) I guess
> I hit a glitch in the updates-updating process.

IIRC, there were some issues back before 3.2.0 was released, where the
3.2 branch was different than trunk but wasn't considered "stable" yet,
and people were using it, and updates may have been problematic depending
on the mix of update and code.  But now that it is the stable branch,
it should always work.

:)

-- 
Randomly Selected Tagline:
"The distributions may be developed that way, but to get into the mainline
 kernel you have to go through the pope." - David Parsons


pgp3BBjHzLXlF.pgp
Description: PGP signature

Re: SA fails to search specified DATADIR for Distribution files

[JT DeLys]

It depends on the version that the 3.2-branch announces itself as, I
think, since that determines the DNS name that's checked within the
SA channel.


Good point.

Then, just FYI, here (for the moment),

 > spamassassin --version
   SpamAssassin version 3.2.2-r547235
 running on Perl version 5.8.8

--
Thanks,

   JTDeLys

Re: SA fails to search specified DATADIR for Distribution files

[Nix]

On 27 Jun 2007, JT DeLys said:

>> The same thing is true if you install from the 3.2-branch, or at least
>> it was in the 3.2.0 days
>
> I'm currently running a build from svn 3.2-branch, and it /seems/ to
> be working correctly.  There are, at least, files being pulled into my
> LOCALSTATEDIR for both SARE & SA-Distrib rules.

It depends on the version that the 3.2-branch announces itself as, I
think, since that determines the DNS name that's checked within the
SA channel.

But I'm glad to hear that it's meant to work and is working :) I guess
I hit a glitch in the updates-updating process.

-- 
`... in the sense that dragons logically follow evolution so they would
 be able to wield metal.' --- Kenneth Eng's colourless green ideas sleep
 furiously

Re: SA fails to search specified DATADIR for Distribution files

[JT DeLys]

The same thing is true if you install from the 3.2-branch, or at least
it was in the 3.2.0 days


I'm currently running a build from svn 3.2-branch, and it /seems/ to
be working correctly.  There are, at least, files being pulled into my
LOCALSTATEDIR for both SARE & SA-Distrib rules.

--lint passes with no complaints.


--
Thanks,

   JTDeLys

Re: SA fails to search specified DATADIR for Distribution files

[Nix]

On 25 Jun 2007, Theo Van Dinter stated:
> I've asked that he open a bugzilla ticket asking for 3.3 updates, and in the
> mean time reverting to a non-development is a good idea for a non-testing
> environment.

The same thing is true if you install from the 3.2-branch, or at least
it was in the 3.2.0 days: I was running 3.2.1 according to the version,
and there were no updates available for that.

It's quite likely that people will run releases from the stable branch
and expect to be able to get updates for it: the stable branch is
supposed to be, well, stable, after all.

(I'm running 3.2.1 and keeping an eye on the branch for critical fixes
to backport now, instead, but it's hardly ideal.)

-- 
`... in the sense that dragons logically follow evolution so they would
 be able to wield metal.' --- Kenneth Eng's colourless green ideas sleep
 furiously

Re: SA fails to search specified DATADIR for Distribution files

[JT DeLys]

If you were to use 3.2 it would work.


I've already 'downgraded' to svn 3.2-branch, and you're right - It works!


Reverting the changes that broke 3.3 updates is on my list of things to do.


Great. Thanks.

--
Thanks,

   JTDeLys

Re: SA fails to search specified DATADIR for Distribution files

[Daryl C. W. O'Shea]

JT DeLys wrote:

Everything relevant ONLY seems to be in Distribution/ -- nothing in 
Updates/.


The problem is that there currently is no update for the SVN version 
(3.3.0).  If you were to use 3.2 it would work.


Reverting the changes that broke 3.3 updates is on my list of things to do.


Daryl

Re: SA fails to search specified DATADIR for Distribution files

[JT DeLys]

felicity has faster fingers.

He beat me to the puchline.

Thanks

 JTDeLys

Re: SA fails to search specified DATADIR for Distribution files

[Theo Van Dinter]

On Mon, Jun 25, 2007 at 02:47:03PM -0500, Richard Frovarp wrote:
> Your problem isn't anything to do with the datadir. It has to do with 
> the fact sa-update isn't running. Furthermore, sa-update thinks your 
> running version 3.3.0. The newest stable release is 3.2.1, so it looks 
> to me that you're running a dev version.

Yeah, we've been chatting about this in IRC.  Basically, 3.3 has no updates
available at all, but he's downloading updates from SARE, etc.  There's a
requirement that if you want the default rules, and you use sa-update, then
you have to use the updates.spamassassin.org channel, and since it doesn't
exist for 3.3, the install is in violation of that.

I've asked that he open a bugzilla ticket asking for 3.3 updates, and in the
mean time reverting to a non-development is a good idea for a non-testing
environment.

-- 
Randomly Selected Tagline:
"And, although some really nasty mind-games were played, no entities were 
 physically harmed during the making of this interactive entertainment 
 (except for the botched special-effect on the bunny rabbit that went so 
 horribly wrong and really bummed everyone out, no thanks to Mr. Boomer)."
  - From the 7th Guest


pgpf0P2670GN0.pgp
Description: PGP signature

Re: SA fails to search specified DATADIR for Distribution files

[Richard Frovarp]

JT DeLys wrote:


Checking where the updates SHOULD be,

   ls -d 
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org


returns,

  /usr/local/bin/ls: cannot access 
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org: 
No such file or directory


Well, that's a problem.

looking at output of 'sa-update --debug',

...
[18621] dbg: logger: adding facilities: all
[18621] dbg: logger: logging level is DBG
[18621] dbg: generic: SpamAssassin version 3.3.0-r543787
[18621] dbg: config: score set 0 chosen.
[18621] dbg: dns: is Net::DNS::Resolver available? yes
[18621] dbg: dns: Net::DNS version: 0.60
[18621] dbg: generic: sa-update version svn540382
[18621] dbg: generic: using update directory: 
/usr/local/etc/spamassassin/Updates/3.003000

...
[18621] dbg: channel: reading in channelfile 
/usr/local/etc/spamassassin/sa-update-channels.conf
[18621] dbg: channel: adding updates.spamassassin.org 

[18621] dbg: channel: attempting channel updates.spamassassin.org 

[18621] dbg: channel: update directory 
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org
[18621] dbg: channel: channel cf file 
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org.cf
[18621] dbg: channel: channel pre file 
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org.pre
[18621] dbg: dns: query failed: 0.3.3.updates.spamassassin.org 
 => NXDOMAIN

[18621] dbg: channel: no updates available, skipping channel
[18621] dbg: diag: updates complete, exiting with code 1

I notice the dns query FAIL.

checking the man page,

"The default channel is
   updates.spamassassin.org , 
which has updated rules since the previous

   release."


Net::DNS, which I think is responsble for those queries, /is/ working, 
as updates of other channels are working.


Ideas?


Your problem isn't anything to do with the datadir. It has to do with 
the fact sa-update isn't running. Furthermore, sa-update thinks your 
running version 3.3.0. The newest stable release is 3.2.1, so it looks 
to me that you're running a dev version.

Re: SA fails to search specified DATADIR for Distribution files

[JT DeLys]

I'm not sure what the problem is you are attempting to demonstrate...
the update from updates.spamassassin.org contains everything you need
from the default DATADIR.




The issue  parallels a discussion I've been having in IRC.

The problem is that I'm not seeing any SPF checks being done, despite the
Plugin being enabled  AND the rules/scores being defined in "50_scores.cf".

'felicity' suggested that's because the rules/scores are not being /seen/ --
and, checking with LINT, there's, indeed, no trace of those rules -- or
ANYTHING from the Distribution files being found/read.

Left to my own devices, I'm trying to figure out why not.

Again, I've specified where the Distribution files are -- why are they not
being found/read?

You state, "the update from updates.spamassassin.org contains everything you
need from the default DATADIR."

Checking my setup/install,

cd /usr/local/etc/spamassassin
grep -i spf `grep -rlni spf .` | grep -i score
./Distribution/50_scores.cf:score RCVD_IN_IADB_SPF 0 -0.001 0 -0.078
./Distribution/50_scores.cf:ifplugin Mail::SpamAssassin::Plugin::SPF
./Distribution/50_scores.cf:score USER_IN_SPF_WHITELIST -100.000
./Distribution/50_scores.cf:score USER_IN_DEF_SPF_WL -7.500
./Distribution/50_scores.cf:score ENV_AND_HDR_SPF_MATCH -7.500
./Distribution/50_scores.cf:endif # Mail::SpamAssassin::Plugin::SPF
./Distribution/50_scores.cf:#
SPF
./Distribution/50_scores.cf:# Note that the benefit for a valid SPF record
is deliberately minimal; it's
./Distribution/50_scores.cf:# likely that more spammers would quickly move
to setting valid SPF records
./Distribution/50_scores.cf:ifplugin Mail::SpamAssassin::Plugin::SPF
./Distribution/50_scores.cf:score SPF_PASS -0.001
./Distribution/50_scores.cf:score SPF_HELO_PASS -0.001
./Distribution/50_scores.cf:score SPF_FAIL 2.600 0.992 1.669 0.693
./Distribution/50_scores.cf:score SPF_HELO_FAIL 2.298 0.365 0.540 0.001
./Distribution/50_scores.cf:score SPF_HELO_NEUTRAL 2.231 2.000 0.744 0.576
./Distribution/50_scores.cf:score SPF_HELO_SOFTFAIL 2.599 1.533 1.427 0.841
./Distribution/50_scores.cf:score SPF_NEUTRAL 2.199 1.210 0.756 0.686
./Distribution/50_scores.cf:score SPF_SOFTFAIL 2.301 0.654 0.698 0.596
./Distribution/50_scores.cf:endif # Mail::SpamAssassin::Plugin::SPF

Everything relevant ONLY seems to be in Distribution/ -- nothing in
Updates/.

Checking where the updates SHOULD be,

  ls -d
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org

returns,

 /usr/local/bin/ls: cannot access
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org: No
such file or directory

Well, that's a problem.

looking at output of 'sa-update --debug',

...
[18621] dbg: logger: adding facilities: all
[18621] dbg: logger: logging level is DBG
[18621] dbg: generic: SpamAssassin version 3.3.0-r543787
[18621] dbg: config: score set 0 chosen.
[18621] dbg: dns: is Net::DNS::Resolver available? yes
[18621] dbg: dns: Net::DNS version: 0.60
[18621] dbg: generic: sa-update version svn540382
[18621] dbg: generic: using update directory:
/usr/local/etc/spamassassin/Updates/3.003000
...
[18621] dbg: channel: reading in channelfile /usr/local/etc/spamassassin/sa-
update-channels.conf
[18621] dbg: channel: adding updates.spamassassin.org
[18621] dbg: channel: attempting channel updates.spamassassin.org
[18621] dbg: channel: update directory
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org
[18621] dbg: channel: channel cf file
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org.cf
[18621] dbg: channel: channel pre file
/usr/local/etc/spamassassin/Updates/3.003000/updates_spamassassin_org.pre
[18621] dbg: dns: query failed: 0.3.3.updates.spamassassin.org => NXDOMAIN
[18621] dbg: channel: no updates available, skipping channel
[18621] dbg: diag: updates complete, exiting with code 1

I notice the dns query FAIL.

checking the man page,

"The default channel is
  updates.spamassassin.org, which has updated rules since the previous
  release."


Net::DNS, which I think is responsble for those queries, /is/ working, as
updates of other channels are working.

Ideas?

--
Thanks,

   JTDeLys

Re: SA fails to search specified DATADIR for Distribution files

[Daryl C. W. O'Shea]

JT DeLys wrote:


It seems to me that, according to,

"   Default configuration data is loaded from the first existing 
directory

   in:
"

the Default config data is NOT picked up from the first INSTANCE of a 
given .cf in the search hierarchy, just from the first DIR listed there.


Yeah, as you quoted, the docs say "first existing directory" and not 
filename.  The software, perhaps coincidentally, works the same way.


I'm not sure what the problem is you are attempting to demonstrate... 
the update from updates.spamassassin.org contains everything you need 
from the default DATADIR.



Daryl

SA fails to search specified DATADIR for Distribution files

[JT DeLys]

Exploring some problems I've been having, I have uploaded the latest
SpamAssassin SVN source code.

I'm specifying 'custom' locations for SpamAssassin local, distribution &
update files.

Despite being correctly 'told' where to pick up the files, SA seems to
ignore the DATADIR spec (where the Distribution files are), only checking in
the LOCALSTATEDIR (where the UPdates are), and traversing no further.

Can someone help find the problem?

I've pasted some of the details below.

Thanks,

  JTDeLys

svn info
Path: .
URL: http://svn.apache.org/repos/asf/spamassassin/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 550563
Node Kind: directory
Schedule: normal
Last Changed Author: jm
Last Changed Rev: 550555
Last Changed Date: 2007-06-25 10:20:48 -0700 (Mon, 25 Jun 2007)


Reading PACKAGING, the wiki, and nformation from previous list posts,


DATADIR (DEFRULESDIR):
  SpamAssassin's real logic lies in its shipped rule definitions and the
  corresponding scores. The files with these settings have to be saved
  somewhere, normally below PREFIX/share/spamassassin. The full path to
  that directory can be changed with this variable (DEFRULESDIR is a
  synonym).

CONFDIR (LOCALRULESDIR):
  SpamAssassin looks for its config files in SYSCONFDIR/mail/spamassassin.

LOCALSTATEDIR:
  "sa-update" will download rule updates into LOCALSTATEDIR/spamassassin.


I've chosen to configure & build using,

perl Makefile.PL   \
PREFIX=/usr/local/spamassassin \
DATADIR=/usr/local/etc/spamassassin/Distribution   \
CONFDIR=/usr/local/etc/spamassassin/Local  \
LOCALSTATEDIR=/usr/local/etc/spamassassin/Updates

cd spamc
perl version.h.pl

After install, sa-update (of both Distribution & SARE rules), and
sa-compile, I have

cd /usr/local/etc/spamassassin
ls Distribution/
 10_default_prefs.cf20_vbounce.cf 30_text_it.cf
 20_advance_fee.cf  23_bayes.cf   30_text_nl.cf
 20_body_tests.cf   25_accessdb.cf30_text_pl.cf
 20_compensate.cf   25_antivirus.cf   30_text_pt_br.cf
 20_dnsbl_tests.cf  25_asn.cf 50_scores.cf
 20_drugs.cf25_dcc.cf 60_awl.cf
 20_dynrdns.cf  25_dkim.cf60_shortcircuit.cf
 20_fake_helo_tests.cf  25_domainkeys.cf  60_whitelist.cf
 20_head_tests.cf   25_hashcash.cf60_whitelist_dk.cf
 20_html_tests.cf   25_pyzor.cf   60_whitelist_dkim.cf
 20_imageinfo.cf25_razor2.cf  60_whitelist_spf.cf
 20_meta_tests.cf   25_replace.cf 60_whitelist_subject.cf
 20_net_tests.cf25_spf.cf languages
 20_phrases.cf  25_textcat.cf sa-update-pubkey.txt
 20_porn.cf 25_uribl.cf   user_prefs.template
 20_ratware.cf  30_text_de.cf
 20_uri_tests.cf30_text_fr.cf

ls -rd Updates/*
 Updates/compiled  Updates/3.003000

Next, checking in the resulting "man spamassassin",

...
CONFIGURATION FILES
...

  Default configuration data is loaded from the first existing
directory
  in:

  /usr/local/etc/spamassassin/Updates/3.003000
  /usr/local/etc/spamassassin/Distribution
  /usr/local/spamassassin/share/spamassassin
  /usr/local/share/spamassassin
  /usr/share/spamassassin

  Site-specific configuration data is used to override any values which
  had already been set.  This is loaded from the first existing
directory
  in:

  /usr/local/etc/spamassassin/Local
  /usr/local/spamassassin/etc/mail/spamassassin
  /usr/local/spamassassin/etc/spamassassin
  /usr/local/etc/spamassassin
  /usr/pkg/etc/spamassassin
  /usr/etc/spamassassin
  /etc/mail/spamassassin
  /etc/spamassassin
...


The problem is that

  spamassassin --lint --nocreate-prefs --debug > /tmp/sa_debug_output.txt

reports that,

 [24122] dbg: config: using "/usr/local/etc/spamassassin/Local" for site
rules pre files
 [24122] dbg: config: read file /usr/local/etc/spamassassin/Local/init.pre
 [24122] dbg: config: using "/usr/local/etc/spamassassin/Updates/3.003000"
for sys rules pre files
 [24122] dbg: config: using "/usr/local/etc/spamassassin/Updates/3.003000"
for default rules dir

and, apparently, never picks up the original .cf's from,

  /usr/local/etc/spamassassin/Distribution

so, e.g.,

 cd /usr/local/etc/spamassassin
 find . | grep -i 50_
   ./Distribution/50_scores.cf

never gets found/read -- DESPITE having assigned

 DATADIR=/usr/local/etc/spamassassin/Distribution

It seems to me that, according to,

"   Default configuration data is loaded from the first existing
directory
  in:
"

the Default config data is NOT picked up from the first INSTANCE of a given
.cf in the search hierarchy, just from the first DIR listed there.



--
Thanks,

   JTDeLys