The version 2.4.2 of amavisd-new has been released
and is available at:
http://www.ijs.si/software/amavisd/amavisd-new-2.4.2.tar.gz
The 2.4.2 is primarily a maintenance release over 2.4.1,
but also brings one important new feature: pen pals soft-whitelisting.
amavisd-new-2.4.2 release notes
SUMMARY OF CHANGES:
- new feature: "pen pals soft-whitelisting" lowers spam score of received
replies to a message previously sent by a local user to this address;
- new features: added command line options to override certain configuration
settings from a config file, see below;
- documentation bug fixes, specially on the use of SQL data type TIMESTAMP;
- zoo decoder interface routine can now use utility unzoo(1) or zoo(1);
COMPATIBILITY WITH 2.4.1:
There are no incompatible changes since 2.4.1, but please notice below
the fixes to SQL and to LDAP documentation, which may affect you.
BUG FIXES AND CHANGES since 2.4.1:
- LDAP.schema: add missing LDAP attribute amavisSpamQuarantineCutoffLevel
to the list of allowed attributes in objectclass amavisAccount;
pointed out by Paolo Cravero;
- README_FILES/README.sql: added short for-the-impatient sections:
* BRIEF MySQL EXAMPLE of a log/report/quarantine database housekeeping
* BRIEF MySQL EQUIVALENT EXAMPLE based on time_iso if its type is TIMESTAMPS
* BRIEF PostgreSQL EXAMPLE of a log/report/quarantine database housekeeping
- README.sql PostgreSQL notes: fixed incorrect advice in README.sql which
suggested to declare a field msgs.time_iso as TIMESTAMP WITHOUT TIME ZONE
instead of the correct TIMESTAMP WITH TIME ZONE.
Previous instructions were also contradictory to suggested data type on
ALTER TABLE msgs ALTER time_iso. Using inappropriate WITHOUT TIME ZONE
when comparing time_iso to now() (which is aware of a time zone) offsets
results by a current time zone offset, which unexpectedly includes too
many or too few records in maintenance purging operation. A workaround
if WITHOUT TIME ZONE continues to be used is to: SET TIME ZONE 'UTC';
before purging.
If you were misled by previous documentation choosing WITHOUT TIME ZONE
for time_iso, and decided now to convert it to WITH TIME ZONE, the following
clause can convert time_iso to proper universal time by manually providing
appropriate time offset:
ALTER TABLE msgs ALTER COLUMN time_iso
TYPE TIMESTAMP WITH TIME ZONE
USING time_iso [-+] INTERVAL '[offset]';
Thanks to Brian Wong for a problem description and advice.
- README.sql MySQL notes: fixed incorrect advice in README.sql which suggested
to declare a field msgs.time_iso as TIMESTAMP instead of the correct
TIMESTAMP NOT NULL DEFAULT 0. The "DEFAULT 0" is mandatory to prevent
MySQL from overwriting mail reception timestamp with current local time
when other fields are updated at the end of processing of a message.
Also not to be forgotten: $timestamp_fmt_mysql *MUST* be set to 1 in
amavisd.conf with MySQL when msgs.time_iso data type is TIMESTAMP... !
- README.sql MySQL notes: if using field msgs.time_iso to select records
for purging (instead of msgs.time_num), and its data type is TIMESTAMP...
(as opposed to CHAR...), one should use function utc_timestamp() in place
of now() in the DELETE clause to make it work correctly regardless of
time zone. Alternatively, now() can continue to be used, provided that
SQL client time zone is set to UTC in the purging SQL script:
SET time_zone='+00:00'; thanks to Gary V for investigation;
- a message with only a header, without empty separator line and with no body,
lost the last line of a header on forwarding or writing to quarantine;
observed by Elias Oltmanns, reported through Debian bug tracking;
- header validity checks inappropriately reported 'header field syntax error'
as a 'header field too long' (BadHdrLong) instead of BadHdrSyntax;
- ensure that notification would not be sent if notification template is empty,
solving the following problem: when recipient notifications for bad headers
is enabled, and a message is spam with bad headers, recipient would receive
an empty notification message (because message contents category is spam
and recipient notification template for spam is empty); reported by Alex;
- changed SMTP status code 550 to 554 when rejecting mail contents,
the 550 is not envisioned in RFC 2821 as a valid reply code to a "."
after data transfer; thanks to Victor Duchovni;
- fixed case mismatch when storing e-mail address to SQL table maddr,
which wasted one unnecessary failed attempt on INSERT;
- ignore $timestamp_fmt_mysql if SQL database driver (DBD) is not 'mysql';
- perl taint workaround in lookup_sql() where SQL select clause
could become tainted; problem reported by Christer Borang;
- fixed amavisd.conf-default which stated incorrect default values of
keys 'ins_rcp' and 'ins_quar' in %sql_clause; reported by Glenn Sieb;
- limit reported boost score to three decimal places; long fractions
observed by Gary V;
- Postfix since version 20060610 uses xtext-encoded (rfc3461) strings in
XCLIENT and XFORWARD attribute values, previous versions used plain text
with neutered special characters - amavisd-new now xtext-decodes value
if it looks xtext encoded, and encodes it on sending; the change could
affect exotic host names (e.g. with a plus in host name) from broken
mailers or DNS; thanks to Ralf Hildebrandt for pointing out the recent
change in Postfix;
- improve regular expressions in the $map_full_type_to_short_type_re list
to cope better with different versions of the file(1) utility regarding
recognition of various MS executables; based on a problem report by Misha;
- use stricter suggested regular expression in amavisd.conf for matching
CLSID (Class ID extension); previous expression was loose and too easily
matched file names with braces in the name; suggested by Martin Schuster
through Debian bug tracking;
- zoo decoder interface routine (do_zoo) can now use utility unzoo(1) or the
traditional zoo(1); the unzoo(1) recognizes some additional parameters
which makes it more resilient (but still not watertight) against some
attempts to hide archive contents or to extract members to unexpected
locations, but unfortunately does not recognize all zoo compression schemes
("error, LZD not yet implemented"), and the relative modes "-j ./" or "-j X"
do not protect against all malicious cases - so it is a mixed blessing.
The way amavisd calls zoo(1) (piping members to stdout, which can be slow)
avoids some of the security problems with zoo (writing to arbitrary
directories), which were probably the main reason for ClamAV project
deciding to switch to unzoo(1);
- zoo sucks, unzoo (v4.4) sucks more: considered, but decided against changing
zoo entry in @decoders to ['unzoo','zoo'] in amavisd.conf, as was suggested
by Gábor Kövesdán. It would not necessarily be an improvement (see previous
item, misses extracting members from my test cases), so feel free to choose
between the two poor choices, I still prefer zoo(1), partly also because it
covers cases which clamd decoding misses;
- internal: saving recipient addresses to SQL table maddr is now done
earlier to make information available to pen pals code;
- explicitly test if SQL 'prepare' silently fails to return a statement
handle, just in case;
- adjusted list of pre-loaded SA modules to cater for SA 3.1.3;
NEW FEATURES:
- new feature: added command line options which override some configuration
settings from a config file (an option to override pid_file suggested by
Paul Murphy and Gábor Kövesdán):
-d log_level ... overrides $log_level
-m max_servers ... overrides $max_servers
-L lock_file ... overrides $lock_file (Net::Server serialization)
-P pid_file ... overrides $pid_file
-H home_dir ... overrides $MYHOME directory
-Q quarantine_dir ... overrides $QUARANTINEDIR directory, empty disables
-T tempbase_dir ... overrides $TEMPBASE directory
-S helpers_home_dir ... overrides $helpers_home directory (SA workplace)
-D db_home_dir ... overrides $db_home, empty arg turns off $enable_db
-R chroot_dir ... overrides $daemon_chroot_dir, empty avoids chroot
-p listen_port_or_socket ... overrides $inet_socket_port as well as
$unix_socketname, argument may be a decimal TCP port
number, or an absolute path name of a Unix socket;
may be specified multiple times: daemon can listen on
multiple inet sockets and/or multiple Unix sockets;
example: -p 10024 -p 9998 -p /var/amavis/amavisd.sock
-V ... shows version and exits
-h ... shows version and command line options, then exits
For completeness, here are remaining options, unchanged from
previous versions:
-u user ... overrides $daemon_user
-g group ... overrides $daemon_group
-c config_file ... config file path, may be specified multiple times
- new feature: "pen pals soft-whitelisting" lowers spam score of received
replies (or followup correspondence) to a message previously sent by a
local user to this address;
Pre-requisites:
* both the outgoing and the incoming mail must pass through amavisd
(although outgoing mail may have spam checks disabled if desired);
* SQL logging must be enabled (@storage_sql_dsn) and records should
be kept for at least several days (some statistics: 85% of replied mail
(or followups) is sent within a week since previous correspondence,
60% within one working day, 40% within 8 hours, 20% within 1 hour);
* @mynetworks and @local_domains_maps must reflect reality;
* the information about client IP address must be available to amavisd,
i.e. Postfix XFORWARD protocol extension must be enabled, or AM.PDP+milter;
* configuration variable $penpals_bonus_score must be set to a positive
value (such as 1.0, increase to 3 or 4 after seeing that it works),
zero disables the feature and is a default;
* $sql_clause{'sel_penpals'} must contain a SELECT clause (which by
default it does, unless overridden by an old assignment to %sql_clause
in amavisd.conf);
* sender/recipient address pair must exactly match recipient/sender pair of
previous correspondence (except for allowed case-changes in domain part),
which means that care must be taken when canonical and/or virtual mapping
is performed by MTA (such as mapping between internal and external address
forms) - if external address forms of local addresses are to be seen by
a content filter then canonical mapping (int->ext) must be done *before*
filtering and virtual mapping (ext->int) *after*; alternatively, if
internal address forms are to be seen by a content filter, then canonical
mapping should be done after filtering, and virtual mapping before;
see README.postfix, section "TO DO 'VIRTUAL ALIAS' MAPPING AND OTHER
POSTFIX CLEANUP PROCESSING BEFORE OR AFTER CONTENT FILTERING?";
How it works:
* SQL logging stores records about all mail messages processed by amavisd,
their sender, recipients, delivery status, mail contents (no changes
there, this feature was introduced with amavisd-new-2.3.0); for the
purpose of pen pals scheme only records with local-domain senders matter;
* when a message is received, a SQL lookup against a SQL logging database
is performed, looking for previous messages sent in reverse direction,
i.e. from a local user (which is now a recipient of the current mail)
to the address that is now the sender of the message being processed;
A SELECT clause in $sql_clause{'sel_penpals'} is used, which by default
only considers records of previous messages that were actually
delivered (not rejected, discarded or bounced), and were not infected.
SQL lookup returns a timestamp of the most recent such message (if any),
the difference (in seconds) between the current time and the timestamp
is an 'age' as used in the following formula;
* an exponential decay formula calculates score points to be deducted
from the SA score:
weight = 1 / 2^(age/penpals_halflife)
score_boost = -penpals_bonus_score * weight
i.e. penpals_bonus_score is multiplied by 1, 1/2, 1/4, 1/8, 1/16, ...
at age 0, 1*halflife, 2*halflife, 3*halflife, 4*halflife ...
weight is a continuous function of age (actually, in steps of one second);
* main configuration variables, members of policy banks:
$penpals_bonus_score ... a maximal (positive) score value by which
spam score is lowered when sender is known to have previously received
mail from our local user from this mail system. Zero or undef disables
pen pals lookups, and is a default.
$penpals_halflife ... exponential decay time constant in seconds,
defaults to 7 days; pen pal bonus is halved for each halflife
period since the last mail sent by a local user to the current
message's sender;
* auxiliary configuration variables, global settings:
$penpals_threshold_low ... SA score below which pen pals lookups are
not performed to save time, defaults to 1.0; undef lets the threshold
be ignored (useful for testing and statistics gathering);
$penpals_threshold_high ...
when (SA_score - $penpals_bonus_score > $penpals_threshold_high)
pen pals lookup will not be performed to save time, as it could not
influence blocking of spam even at maximal penpals bonus (age=0);
usual choice for value would be kill level or other reasonably high
value; undef lets the threshold be ignored and is a default (useful
for testing and statistics gathering);
Caveats / notes / exceptions with "pen pals soft-whitelisting":
* pen pals soft-whitelisting aids incoming mail, and internal-to-internal
mail, but has no effect on outgoing mail;
* if SQL logging was not used so far and you are considering enabling it
for a busy site, you would appreciate PostgreSQL 8.1 compared to MySQL,
as purging old records seems to be *much* faster than in MySQL 4.1,
which could lock down mail processing for an hour or more during a
weekly (or daily) purge, as opposed to minutes or seconds;
* infected messages are exempted from pen pals checks;
* mail with (unadjusted) SA score below $penpals_threshold_low (1 by default)
is exempted from pen pals check to save time and lighten the load on SQL;
similarly for high score spam which would not have a chance of being
'saved' even by a maximal pen pals bonus score;
* non-delivery notifications have null sender address, so can not match
previous correspondence and can not receive a pen pal bonus;
* sender address matching local domains but coming from outside is
not trusted and is exempted from pen pals checks; this implies that
authenticated local users posting from outside to this MTA do not
benefit from pen pals scheme when sending to local recipient - the
cleanest solution is to separate MSA from MTA (which also avoids the
SpamAssassin SPF & DUL difficulty with trusted networks for such users);
* messages from a local user to self are exempted from pen pals check;
* outgoing messages (i.e. to non-local recipients) are exempted from
pen pals checks to save some time and simplify reasoning (which reverse
mail transaction to trust?); assuming that local users rarely send
spammy mail, outgoing mail would rarely need help from pen pals checks;
* messages received from mailing list typically use ML bounce or admin
address (possibly VERPed) as the sending address, so they would not be
considered replies to postings to a mailing list from a local user;
* underlying assumption is that a local-domains sender address in mail
coming from inside can be trusted not to be faked; if this is not the case,
an internal user cooperating with a spammer can widen spam tolerance for
another internal user (but it probably does not pay off, too much trouble
for too little effect);
* if a spammer knows or can guess that a local user is frequently sending
mail to some address (e.g. a mailing list unprotected by DK or SPF), he
can gain few bonus score points by using such sending address in his spam;
* there may be multiple MTA+amavisd servers, but all must use the same
logging SQL database;
* forwarding is compatible with the pen pals scheme;
* broken forwarding like suggested for SPF, where sender address is replaced
by a forwarding mailbox address is counterproductive; for example: a local
user may also have an external mailbox at some remote provider with poor
spam protection; forwarding from the remote to a local mailbox is set up
and forwarding MTA misguidedly substitutes original sender address with a
mailbox address; spam reaching remote mailbox is forwarded to a local site
with a sender address rewritten, making it look like it is coming directly
from a user's remote mailbox, and benefiting from pen pals bonus of user's
previous correspondence with his remote mailbox;
Testing:
* set $penpals_bonus_score initially to a low value such as 1 or 0.5
to avoid surprises;
* set $penpals_threshold_low and $penpals_threshold_high to undef
to perform pen pals checks regardless of the score;
* at log level 2 (or higher) search the log for a string "penpals: "
(only shows on incoming mail sent by a non-local sender); the log also
shows mail_id of the referenced message (previous communication),
and Subject header fields of previous and current message;
Based on a feature request by Aaron P. Martinez, thanks to Gary V for
suggestions and prompting and to Michael Scheidell and Richard Bishop
for feedback.
Mark
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
AMaViS-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/amavis-user
AMaViS-FAQ:http://www.amavis.org/amavis-faq.php3
AMaViS-HowTos:http://www.amavis.org/howto/
