Re: [Mimedefang] x64 compatible?
I hope I'm not way off here, but are you running with selinux enabled or disabled? -Rich ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] SPF
Tomasz Ostrowski wrote: For folks on the road, there are plenty of workable solutions. We use OpenVPN, which works well if both ends are running Linux. Because of deficiencies in Windoze's TUN implementation, it's a bit more painful to get it working on that platform, but we managed it. It is much easier to use submission port + starttls or smtps. Both do not use smtp port which is often blocked. Agreed. That's what we do, and it works out rather well. The only problems that come up is trying to re-train users no not use their local ISP for mail delivery. -Rich ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Helo Checking
The main problem I see with yours is that it doesn't compensate for
localhost (127.0.0.1). Mine (below) checks three IP addresses:
localhost (127.0.0.1), our internal NAT'ed network (192.168.10.x), and
our external public IP address (in this example, I used 9.87.65.4).
Anyhow, here's a copy of the one I personally use. I hope it helps.
-Rich
sub filter_sender () {
my($sender, $hostip, $hostname, $helo) = @_;
# Can't be ourdomain.com unless it's one of our IP's.
if ($helo =~ /(^|.)ourdomain\.com$/i)
{
if ( ! ($hostip =~ ^192.168.10) ($hostip ne 127.0.0.1)
($hostip ne 9.87.65.4) )
{
md_syslog('warning', Host $hostip said HELO $helo);
return(0, Go away. $hostip is not a wesmo.com machine);
}
}
# The hostname better match the helo string.
if (($helo =~ /^(\d{1,3})(.)(\d{1,3})(.)(\d{1,3})(.)(\d{1,3})$/)
($hostip ne $helo))
{
md_syslog('warning', Host $hostip claims to be $helo);
return (0, Header forgery attempt, $ip claims to be $helo)
}
return (1, OK);
}
Hi all,
When I insert this snippet into my mimedefang-filter my slaves all get
busy and shut down..any Ideas?
Don Killen
sub filter_sender {
my($sender, $ip, $name, $helo) = @_;
return('CONTINUE', OK) if ($ip eq 72.242.108.6); # no further
checking if localhost
if ($helo =~ /(^|.)granis.net$/i) {
if ($ip !~ /^72.242.108./) {
return('REJECT', Connect rejected - $ip is not granis.net);
}
}
return('CONTINUE', OK);
}
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Question about sendmail/MD interaction
Nope. I've been using delay_checks for some time (probably over a year and a half) with no problems. -Rich I was asked recently if using FEATURE(`delay_checks') in sendmail.mc affected when MD was passed data by the sendmail MILTER interface. I don't think that it does, but I wanted to be sure. So, does that FEATURE change how MD interacts with sendmail? ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] use strict
Anyone care to post a strict version of their filter as an example? :-) -Rich ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Mimedefang stream-by-recipient and Mailman
We have a server that has Mailman running on it for close to 80 lists,
and, on that same server, we have our
sendmail+mimedefang+File::Scan+Anomy mail server. Mailman sends its
mail out locally to the sendmail server (not utilizing VERP as that
caused us a world of other problems in the past), and mimedefang seems
to be analyzing the messages as they go out (not what was originally
intended).
Quite suddenly, emails FROM the mailman server were not going out.
After a frazzling hour, it was discovered that a section of code in the
mimedefang filter was causing the problem, although the piece of code
was in place for well over a month. Commenting it out and restarting
mimedefang allowed the mailman mail to be sent out.
The code was:
if (stream_by_recipient())
{
return;
}
Sendmail was logging the following when a message was sent to our test list:
Apr 26 15:00:27 ourserver sendmail[11884]: j3QJ0FmI011884:
from=[EMAIL PROTECTED], size=1964, class=-30, nrcpts=5,
msgid=[EMAIL PROTECTED], proto=ESMTP,
daemon=MTA, relay=localhost.localdomain [127.0.0.1]
Apr 26 15:00:29 ourserver mimedefang.pl[11378]: j3QJ0FmJ011884: streamed
by domain or recipient and resent.
Apr 26 15:00:29 ourserver mimedefang[11362]: j3QJ0FmJ011884: Discarding
because filter instructed us to
Apr 26 15:00:29 ourserver sendmail[11884]: j3QJ0FmJ011884: Milter: data,
discard
Apr 26 15:00:29 ourserver sendmail[11884]: j3QJ0FmJ011884: discarded
Now, we've disabled stream_by_recipient, but it's global. We would
prefer to keep the stream_by_recipient() call in there (the server can
handle it) for other purposes. What would be the best way, in
filter_begin(), to utilize stream_by_recipient() for messages only
coming from external hosts (in other words, to utilize
stream_by_recipient() on all emails EXCEPT localhost)?
Thanks!
-Rich
___
Visit http://www.mimedefang.org and http://www.canit.ca
MIMEDefang mailing list
MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] OT - Using rDNS sendmail hack - your experiences
Personally, we've looked in to it. We tend to agree that AOL's position is somewhat aggressive since their techs are usually behind the time and don't support their own new technologies well. But, political opinions aside, we were leary about implementing it because, frankly, we were afraid of the possible negative impact. So, we have relied on MimeDefang to do this check for us.. However, as time has worn on (and the amount of SPAM has blossomed), we have started testing this hack on our in-house testing server. Hearing of your experiences does make me feel a bit better regarding the patch, too. Do you have any stats on how many connections this has prevented? I'd personally be interested in seeing your modified version of the hack (your hacked hack :) ) just to see and understand the differences. -Rich Hello all, this is a bit off topic but relevant. We finally decided it was probably time to implement AOL style reverse DNS checks into our MTA. Since AOL has been doing it now for something like 6 months it is a pretty fair bet that most US customers that are legit have corrected their DNS issues... or so we thought! Why reinvent the wheel... we implemented a slightly modified version of this sendmail m4 HACK here: http://www.cs.niu.edu/~rickert/cf/hack/require_rdns.m4 Which basically does this: 1. Check relay for rDNS then check the response (gethostbyaddr check) 2. If there is not PTR record FAIL 3. If you cannot find DNS record for it at all, maybe DNS is down, TEMPFAIL 4. If there is rDNS (PTR) but it appears forged (different than forward or result doesnt resolve), TEMPFAIL ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Mimedefang stream-by-recipient and Mailman
The code was:
if (stream_by_recipient())
{
return;
}
The mail should be going out, just more slowly. :-) You should see
a lot of queue files in your clientmqueue directory, which will all
be delivered at the next clientmqueue run.
The interesting thing was that nothing was getting queued up.. it was
getting discarded. :(
The fix, obviously, is:
if ($RelayAddr ne 127.0.0.1) {
return if stream_by_recipient();
}
Glad it was obvious. :) I didn't know (for sure) what variable was
available in filter_begin.
Thanks!
-Rich
___
Visit http://www.mimedefang.org and http://www.canit.ca
MIMEDefang mailing list
MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] OT: Anyone here use Dovecot imap server in production evironment?
In my initial evaluations of Dovecot and Cyrus, I did learn that Dovecot isn't truly ready for prime time just yet.. at least, not with the version I downloaded (I'm planning on trying again soon, though, as UW is painful in its delays). Ok.. I take that back.. Dovecot 0.99.13 and above are ready for prime time. Our environment makes heavy use of procmail, the storing the inbox local to the user's home directory rather than a shared store, an inconsistent storage location for mail (some users have a subfolder which is the root of all of their mail, some don't), and the use of IMAP subscriptions (.mailboxlist). Dovecot takes a little tweaking in order for it to work in a similar fashion as UW's. Most specifically, I found that the default_mail_env directive in dovecot.conf has to be exactly what you want it to be. In our case, default_mail_env = mbox:~/:INBOX=%h/mbox worked fine in addition to renaming all .mailboxlist files to .subscriptions. Dovecot takes a little extra tweaking to get it to run with xinetd, but, after some analysis, there really is little benefit to running dovecot out of xinetd if the server gets a fair amount of traffic (imap/pop traffic, that is). Running it standalone is just fine. In comparison to UW, dovecot cruises through mailboxes. :) Our environment is now sendmail + mimedefang + procmail + spamassassin on the delivery side, and dovecot on the user access side. :) -Rich ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] OT: Anyone here use Dovecot imap server in production evironment?
I tested Dovecot briefly on OpenBSD, and lost mailboxes. No idea what happened, but UW-IMAP has been much more forgiving, especially when compiled with MBX support for faster mailboxes. In my initial evaluations of Dovecot and Cyrus, I did learn that Dovecot isn't truly ready for prime time just yet.. at least, not with the version I downloaded (I'm planning on trying again soon, though, as UW is painful in its delays). Cyrus is definitely a much more serious server, and I do miss the shared mailboxes. (We worked around it with Dovecot by creating special users and giving access to the appropriate people: All the salespeople get to access shsales, for example.) One thing that is a bit of a pain with Cyrus is that it's very awkward to use procmail with Cyrus, and Sieve just isn't as flexible as procmail. Cyrus is a great mail server, no doubt, but its design makes it such that it only works in certain specific environments. In an environment that we serve here, which is a multiple domain, multiple user, nothing shared ever, mail filtering required environment, Cyrus just doesn't apply. Similar to what David states, Cyrus lacks the true ability to utilize procmail (someone on their list hinted that it was possible, but more painful than having your fingernails pulled off). What I did discover is that while there is a lot of noise about Cyrus and how wonderful it is, it was designed with a specific customer/user base in mind. UW-IMAP, while older than dirt itself, pretty much applies to any install. And, yes, UW-IMAP is simplistic in its install, while both Courier and Cyrus are scarier than hell (making compiling sendmail and building a good sendmail.cf/submit.cf look like a junior admin could do it with their eyes closed). Just my $0.02, but I find that Dovecot is quite promising as a potential drop-in replacement for UW-IMAP if Courier or Cyrus don't do it for you.. -Rich ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] timeout before data read?
(this got bounced the first time around because it contained too much
quoted material.. bah)
Wow! Thanks for the great response!! Your response would be a great
addition to the FAQ (hint, hint).
For the short term, because of this situation, I was forced to disable
MimeDefang (ack.. it killed me to do this), and the server responded well.
I agree with your assessment that the remote side was probably sending
too aggressively from whomever was sending it. I dug through the
maillog and couldn't narrow it down to a specific address, although the
recipient seemed to be the same in a number of cases.. this will take
more investigation on my part, that is for sure. :)
Our existing sendmail M4 setting is:
INPUT_MAIL_FILTER(`mimedefang',
`S=unix:/var/spool/MIMEDefang/mimedefang.sock, F=T, T=S:60s;R:60s;E:5m')dnl
With the mimdefang setting being:
MX_BUSY=600
Which, although I looked at this and said to myself That looks ok,
after pasting it here, I realized that 600 seconds is 10 minutes. Hrm...
A definite adjustment to one or both to syncronize them would be a good
thing. Although, that makes me wonder if the sendmail settings we
currently have are too aggressive...
Oh, and for those that asked, we have the max_message_size set to
1048576 (1MB). This *is* email, not ftp. :)
After getting those settings fixed, I was able to get things back up and
running again. I've had a few hiccups throughout the day as there were
a couple of situations that spiked up the load. As it is, with my
mimedefang-filter, I'm only spam scanning:
if ((-s ./INPUTMSG 100*256) (!
exists($SendmailMacros{'auth_authen'})) )
I'm going to have to dig a bit more, but, overall, the tweaking of the
timeouts did help a lot..
-Rich
Aleksandar Milivojevic wrote:
There were some discussions on the list about this recently.
Basically what happens is that you either configured
mimedfang-multiplexor or sendmail or both with too short timeouts.
Because of this either multiplexor or sendmail or both are not waiting
long enough for filtering to finish. By your description, I'd say
somebody sent you *very* large email, and sendmail is the one that got
impatient. Mail is rejected with tempfail, old process is still
spinning, remote side tries to redeliver (probaby too agressivly), new
process starts, and you end up going in circles, and your load average
hits the roof.
The first timeout is in mimedefang-multiplexor. It controls for how
long multiplexor will let the child to run before killing it. The
second timeout is in sendmail. It controlls for how long sendmail
will wait for MIMEDefang to do its job. You should set this two
timeouts to aprox. same value (maybe setting multiplexor timeout
minute or two shorter). That way, old process will be terminated by
multiplexor at about same time as sendmail gives up. It really
doesn't make any sense to have them set differently (the first that
you hit will couase tempfail).
How long should the timeout be. Depends on how large emails you are
allowing (and the speed of your machine). If you limit the size of
email in sendmail to say 1-10MB, around 15 minutes should suffice on
reasonable fast machine. If you limit it to anything larger, values
as large as 1 hour should be considered.
Small hint, do not run SpamAssassin on large emails. If mail is
larger than say 100kB, skip SpamAssassin checks. They will take
forever to complete.
Oh, BTW, multiplexor timeout is controlled in
/etc/sysconfig/mimedefang (well, at least on RedHat type systems):
MX_BUSY=1740 # (29 minutes, we give up before sendmail does)
sendmail timeouts are defined in sendmail.mc:
INPUT_MAIL_FILTER(`mimedefang',
`S=unix:/var/spool/MIMEDefang/mimedefang.sock, F=T, T=S:30m;R:30m;E:30m')
___
Visit http://www.mimedefang.org and http://www.canit.ca
MIMEDefang mailing list
[EMAIL PROTECTED]
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] STILL : MIMEDEFANG NOT TAGGING
I would like to clarify: You DO need Spamassassin to be installed. But, the spamd process it not used by MimeDefang. HOWEVER, if you want the full capabilities of Spamassassin (MimeDefang utilizes a fair amount of spamassassin's features, but not individual user preferences such as whitelists and blacklists per user), you DO need to run spamd with either a local .procmailrc file (with procmail installed, of course) or a system-wide .procmailrc file. # more /etc/procmailrc DROPPRIVS=yes :0fw * 256000 | /usr/bin/spamc -Rich James Ebright wrote: [..snip..]You do NOT need spamd or sa at all.. mimedefang takes care of that. [..snip..] On Mon, 15 Nov 2004 21:40:39 -0500, Jeff Rife wrote You don't need spamd running at all. SpamAssassin is integrated into MIMEDefang through pure Perl. ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] timeout before data read?
Quite suddenly, and consistently, today at around noon, I started seeing the following messages in my maillog, and the load average on the system went through the roof (load average shot up beyond 22!) shortly afterwards. Nov 17 15:02:51 cranium sm-mta[24072]: iAHK1XJR024072: Milter (mimedefang): timeout before data read Nov 17 15:02:51 cranium sm-mta[24072]: iAHK1XJR024072: Milter (mimedefang): to error state Nov 17 15:02:51 cranium sm-mta[24072]: iAHK1XJR024072: Milter: data, reject=451 4.3.2 Please try again later Nov 17 15:02:51 cranium sm-mta[24072]: iAHK1XJR024072: to=[EMAIL PROTECTED], delay=00:01:01, pri=32062, stat=Please try again later Now, if I bring down both mimedefang and sendmail and bring it back up again, I can get some mail to come through, but very little before it starts spitting out these errors again. And, it seems, that mimedefang keeps being forced to spawn a new process, leaving the old one around to spin its wheels, and the new process will generate the same error, and then spawn a new process because of the failure, and so on and so on, becoming a very fast and sick cycle. :( I'm pretty much at a loss as to what could be the cause since neither mimedefang nor sendmail have been rebuilt in quite a while (I am running the latest mimedefang and the latest sendmail, of course). And no changes have been put in to place in the mimedefang-filter either. :( Right now, I am limping by only by watching the maillog, seeing a failure of a slave, and restarting mimedefang as a whole... Of course, I can't keep that up forever.. -Rich ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Pounded by spam
While I know it can be easy to simply block the host, I was wondering if there was some way to avoid the problem all together by potentially identifying hosts attempting to overload the server (Denial Of Service) by throttling down the amount of allowed inbound connections (from external sources) from a single host. Yes. Sendmail =8.13.0 has several nice options. FEATURE(`ratecontrol',`nodelay',`terminate')dnl FEATURE(`conncontrol')dnl define(`confCONNECTION_RATE_WINDOW_SIZE',`60')dnl I was looking at those, in addition to the FEATURE(`greet_pause', num).. The documentation on sendmail.org's site regarding greet_pause was just a step above non-existent. I didn't check the others (ratecontrol and conncontrol).. Looking in to them now. I am the SysAdmin for an ISP here in Billings. I am unafraid of using these controls and they have really helped our situation. I limit 25 Connections/sec period. I also limit 3 connections from any one external host/min. Just out of curiosity, how, exactly, are you limiting the connections per second and connections from external hosts/domains? I occasionally get the 25 connections and deferring at that rate in my logs, but not enough to worry me and we handle ~200,000 emails a day. Adjust your connection/defer times accordingly to your normal load. Have fun and knock them dead at the gate. Thanks! -Rich ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] /etc/sysconfig/mimedefang option questions
In the /etc/sysconfig/mimedefang file, there are the following options: # If yes, turn on the multiplexor relay checking function # MX_RELAY_CHECK=yes # If yes, turn on the multiplexor sender checking function # MX_SENDER_CHECK=yes # If yes, turn on the multiplexor recipient checking function # MX_RECIPIENT_CHECK=yes What *exactly* do each of these do? I enabled them even though I am doing sender and recipient checks (rudimentary) within mimedefang-filter, but I'm curious to know if I just enabled something that was going to potentially block valid email. Additionally, by enabling the MX_RELAY_CHECK, I've gotten a few syslog broadcasts: Message from [EMAIL PROTECTED] at Fri Oct 29 12:55:19 2004 ... myhost perl: Host 211.228.227.141 claims to be my.numeric.ip.addr I did set the syslog facility to mail, and it does mention that I need to set Also set $SyslogFacility in your filter, but where in the filter should that be set? -Rich ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Pounded by spam
We just got the living daylights pounded out of us by a spam host running at 69.6.66.103. While I know it can be easy to simply block the host, I was wondering if there was some way to avoid the problem all together by potentially identifying hosts attempting to overload the server (Denial Of Service) by throttling down the amount of allowed inbound connections (from external sources) from a single host. Admittedly, this is a bit off topic.. Mimedefang.pl was the process that was getting hammered (and subsequently drove the CPU load to 16 before we shut down email all together), but I do not think that the fault lies with mimedefang (in fact, I don't think there is any 'fault' here).. it's more a configuration issue at the MTA level (in this case, sendmail). -Rich ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] filter config question
A while back I configured mimedefang to *not* scan email the is destined to the hosted mailing lists. However, I just noticed that it *is* scanning email *from* the lists out to the list members. Interestingly enough, this is causing some problems.. Is there an easy way within the filter to determine what host the message is originating from (I only want it to bypass if the originating host is the maillist server (in this case, localhost))? -Rich ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] graphdefang and mailing list activity
I've been trying to tweak mimedefang to properly log all mailing list
activity in order to get a better idea, through graphdefang, as to how
much of the overall activity is due to the mailing lists.
I've gotten the mimedefang-filter set to identify mail coming *in* to
the mailing lists and properly log it, but, as you know, the bulk of
email with regards to mailing lists is email going *out* from mailing lists.
I thought of putting something in to filter_sender, but then I backed
off thinking that I might be opening up a rather large security hole.
I've since built a small snippet to be added to the end of filter_sender:
if (is_list($sender) ($hostip == 127.0.0.1)){
md_graphdefang_log(mailing_list, , ,);
}
Where is_list is as below (with the %lists being a hash of all lists on
the system).
sub is_list
{
my($listname) = shift;
##
## The email address, as it comes in, is surrounded by
## brackets. We have to massage it a little in order to do
## proper matching. eg: '[EMAIL PROTECTED]' needs to be 'test'.
##
$listname = (split(\@, $listname))[0];
$listname = (split(\, $listname))[1];
return ($lists{lc($listname)});
}
However, if I remember correctly, the from addresses when dealing with
mailing lists are not easily matched upon... I think that is where my
logic in this scenario starts to fade..
Thoughts/ideas?
-Rich
___
Visit http://www.mimedefang.org and http://www.canit.ca
MIMEDefang mailing list
[EMAIL PROTECTED]
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Graphdefang + High CPU Load when processing
Ya know, I was considering the very same thing. My backup server does basically nothing throughout the day (it only runs backups), which makes it a great candidate. :) Just curious, but what, exactly, does the --trim option do? The documentation states that it cuts out old data from the SummaryDB, but doesn't that defeat the purpose of graphdefang when looking for longer-term trends and such? With regards to the corruption issue, the problem creeps in when Graphdefang has a LOT of information to process. In my situation, we have a medium load mail server which generates maillogs on a daily basis anywhere between 2MB and 10MB. Now, that doesn't seem to be too large, but, however, GraphDefang runs, when I was running it on a daily basis, after about a month of gathering data (maybe sooner, but it's been nearly 10 months since we were running it on a daily vs. every 30 minute basis), we noticed that the graphs suddenly flatlined. A little investigation showed that the database was corrupted (ran graphdefang.pl by hand, and no new data would get added to the database). That was the 4th occurance of the same problem, so we opted for the more frequent updates to reduce the amount of data it had to handle at any one point in time. This resolved the problem, but, now, 10 months later, we have seen that graphdefang has started spiking the CPU for a minute or more as it processes (this is a new problem for us). The machine is a solid box with 1GB of RAM.. I'll try running it on a remote server.. :-) -Rich Chris Gauch wrote: I ran into this same issue with Graphdefang, but it was fairly easy to resolve. I set up graphdefang on a remote Linux server that had a low average load. ...snip... You could also set up an rsync process to rsync the maillog onto the remote graphdefang server, rather than configuring remote syslog. On the remote syslog server running graphdefang, you should also add a CRON script that runs the graphdefang.pl --trim option. Just make sure this CRON script DOES NOT run while graphdefang is processing the SummaryDB info, or you'll corrupt the he|| out of your SummaryDB. - Chris Kevin A. McGrail wrote: I run it once a day and never have a corruption issue. If you have corruption issues, suggest looking at your DB installation. There is just very little in graphdefang that could really cause this issue. Are you having other DB corruption issues? I am very wary of using DB on servers because of the numerous issues we see but the speed benefits are great compared to something like mySQL. In other words, all over different mailing lists, I constantly read about DB corrupted this, DB corrupted that. Not to mention that DB is often implemented in a way that loads the entire DB into memory. This is great for small databases but the graphdefang database can hit half a gigabyte for a server. This causes huge spikes in the load but makes it process quicker. Perhaps your machine is running out of memory trying to load the database and just crashing? Regards, KAM ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Deadline for SPF records
Joseph Brennan wrote: One was from our user on a Verizon dialup where he was required to send through Verizon's smtp server. He reported port 587 was blocked so he could not do smtp auth to our server. This has not been confirmed. Verizon does do some funky things to try to insure that their end users use their mail server with the proper from line such. That is, however, only if you use their mail server. My experience with them at a number of different clients has been that they do not block outbound ports. So, most likely, the problem that was reported to you was somewhere between the chair and the keyboard (user error). :) One was from an IP on campus but not routed through our smtp server. Solution is to use our server or send with their own subdomain in the sender address. They chose the latter. :-) That's because they were on your IP network, but going through their own email server (aka: bad). :) The SPF advocates say all such systems must use an envelope sender with their own domain in it. The header From: can still show what human sent it. While this sounds like the right thing to do, I wonder how fast it can really be implemented and what pain will be caused in the meantime. There will be pain, that's for sure. How fast it can be done and how much pain is involved is really up to the individual organization. I had to use the p word, but official 'policy' drives everything. :) -Rich -- Richard West $14.95 Registrations mailto:[EMAIL PROTECTED] Wesmo Computer Services.com .net .org .tv .cchttp://www.wesmo.com Full Domain Web Hosting .BIZ .INFO MORE!! ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Graphdefang + High CPU Load when processing
First off, graphdefang is an awesome tool. I, too, have noticed that it is *very* easy to corrupt the graphdefang database. As was mentioned by the author and a couple of others on the list, the trick to avoid it from being corrupted is to increase the frequency that you run graphdefang... However, the cause of the database corruption is actually not the size of the database, the problem seems to come in when graphdefang has a lot to analyze in your maillog and then add to the database. If there is too much to handle in the maillog, it craps out, and takes the database with it. However, even running it every 1/2 hr on a medium load server, it causes the CPU utilization to spike up heavily.. and I mean heavily. I would be interested to see if Jonas' version overcomes the high CPU load problem.. or if anyone else has seen the same problem. -Rich On Tue, 27 Jul 2004 10:07:52 +0530, Ravi.P CMC Engr wrote: Can you please provide me the steps in order to split the DB to avoid such problem in future. Splitting the database will not avoid problems that comes from bugs or errors in the database or serializing modules. The reason split the database is because it was a bit more efficient when using DWHDB as it gave me the opportuinty to store some stuff in a flat DB_File instead. How can i proceed?? give me a detailed procedure!! You can check my version of Graphdefang at http://whatever.frukt.org if you want to see how I have replaced the seraializing with tying to DWHDB and split the database. I can hardly get more detailed than actually showing you my code. If you want to do it your own way, you could still get ideas from my code. If you don't want to use my code and you're not familiar enough with perl programming to do something similar yourself, I'd suggest that you do something simpler than what I did. Replacing the original serializing with YAML worked fine for me and is a pretty small and simple modification of the code. You can find the YAML module at CPAN. Regards /Jonas ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Globals
Thanks to all of those that responded.
Based upon all of the ideas, I came up with the following code to do the
trick.
-Rich
sub filter_begin () {
...
%lists = get_lists();
}
#***
# %PROCEDURE: get_list
# %ARGUMENTS:
# None
# %RETURNS:
# Array of lists on the system
# %DESCRIPTION:
# This function fills an array with the names of all of the mailing
# lists on the system.
#***
my %lists = ();
sub get_lists
{
return (%lists) if (%lists);
open (LISTS, /var/mailman/bin/list_lists -b|)
or die Could not execute '/var/mailman/bin/list_lists -b'.\n;
while (LISTS)
{
chop;
$lists{$_} = 1;
}
close(LISTS);
return (%lists);
}
#***
# %PROCEDURE: is_list
# %ARGUMENTS:
# listname -- the name of the mailing list
# %RETURNS:
# boolean - positive if the recipient matches a list name.
# %DESCRIPTION:
# This function matches the incoming mailing address with a list name.
# We do not scan the contents of mailing lists as spam.
#***
sub is_list
{
my($listname) = shift;
##
## The email address, as it comes in, is surrounded by
## brackets. We have to massage it a little in order to do
## proper matching. eg: '[EMAIL PROTECTED]' needs to be 'test'.
##
$listname = (split(\@, $listname))[0];
$listname = (split(\, $listname))[1];
return ($lists{lc($listname)});
}
sub filter_end ($) {
my($entity) = @_;
...
if ($RelayAddr eq 127.0.0.1 or $RelayAddr eq my.ip.address.com)
{
md_graphdefang_log('mail_out',,$RelayAddr);
}
else
{
# Do not scan any messages going to any of the mailing lists.
foreach $recipient (@Recipients)
{
if (is_list($recipient))
{
md_graphdefang_log('mail_in', , $RelayAddr);
md_graphdefang_log('mailing_list', $hits, $RelayAddr);
return;
}
}
Steffen Kaiser wrote:
On Thu, 8 Jul 2004, Rich West wrote:
Hmm, I'd populate a global variable when the slave starts or in
filter_initialize. I do so, anyway.
sub is_list
{
$listname = (split(\@, $listname))[0];
$listname = (split(\, $listname))[1];
Angle brackets are not mandatory.
foreach $list (@lists)
{
chop($list);
return 1 if ($list =~ /^$listname/i);
^ here you check only, if the
recipient begins with a name of a list.
}
return 0;
}
BTW: How about preparing the name cache a bit more in order to avoid
the foreach loop each time you lookup a name, e.g.
1) use a hash:
%mailists = ( 'list1' = 1
, 'list2' = 1, ... );
Then you can do simply return $mailists{lc($listname)}
2) or use a large string: $mailists = '@[EMAIL PROTECTED]@[EMAIL PROTECTED]@';
then do: return index($mailists, '@' . lc($listname) . '@') = 0;
(Because '@' is never part of listname, it's save.)
Bye,
___
Visit http://www.mimedefang.org and http://www.canit.ca
MIMEDefang mailing list
[EMAIL PROTECTED]
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Globals
I've mucked around with the mimedefang-filter.pl pretty heavily in the
past, but I just wanted to ask a quick question before I tried this one out.
The goal is to have it such that our hosted mailing lists do not get
passed through the spamassassin plugin since the lists are subscription
based only. Now, rather inefficiently, I have the code below, but it
litterally populates the array @lists with each and every call.
Ideally, I would like to put the array and its contents in the global
namespace to avoid the generation of the contents of the array (eg:
get_lists) each time an email is processed.
Our mailing lists do not change frequently, but new ones do get added
and older ones do get removed/archived. So, maintaining the dynamic
nature is relatively important. We don't want to restart mimedefang
just because a mailing list was added or removed, but we CAN live with
the fact that it may take some time for mimedefang to 'learn' about the
changes. In other word, I realize that if the array is populated in the
global namespace, it is there, unchanged, for the life of the slave, but
we can live with that, since slaves come and go, and as new ones come,
they will have the new list of lists. :)
-Rich
#***
# %PROCEDURE: get_list
# %ARGUMENTS:
# None
# %RETURNS:
# Array of lists on the system
# %DESCRIPTION:
# This function fills an array with the names of all of the mailing
# lists on the system.
#***
sub get_lists
{
open (LISTS, /var/mailman/bin/list_lists -b|)
or die Could not execute '/var/mailman/bin/list_lists -b'.\n;
@lists = LISTS;
close(LISTS);
return (@lists);
}
#***
# %PROCEDURE: is_list
# %ARGUMENTS:
# listname -- the name of the mailing list
# %RETURNS:
# boolean - positive if the recipient matches a list name.
# %DESCRIPTION:
# This function matches the incoming mailing address with a list name.
# We do not scan the contents of mailing lists as spam.
#***
sub is_list
{
my($listname) = shift;
my(@lists) = @_;
##
## The email address, as it comes in, is surrounded by
## brackets. We have to massage it a little in order to do
## proper matching. eg: '[EMAIL PROTECTED]' needs to be 'test'.
##
$listname = (split(\@, $listname))[0];
$listname = (split(\, $listname))[1];
foreach $list (@lists)
{
chop($list);
return 1 if ($list =~ /^$listname/i);
}
return 0;
}
sub filter_end ($) {
my($entity) = @_;
my(@lists);
my($blacklist);
...
if ($RelayAddr eq 127.0.0.1 or $RelayAddr eq
mailserver.dotted.ip.address)
{
md_graphdefang_log('mail_out',,$RelayAddr);
}
else
{
# Do not scan any messages going to any of the mailing lists.
@lists = get_lists();
foreach $recipient (@Recipients)
{
if (is_list($recipient,@lists))
{
md_graphdefang_log('mail_in', , $RelayAddr);
md_graphdefang_log('mailing_list', $hits, $RelayAddr);
return;
}
}
# Spam checks if SpamAssassin is installed
if ($Features{SpamAssassin}) {
...
___
Visit http://www.mimedefang.org and http://www.canit.ca
MIMEDefang mailing list
[EMAIL PROTECTED]
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] MIMEDefang 2.43 is released
Dumb question, but where, now, is the HELO argument accessible?
With this change, my filter_relay will no longer function... what would
be the recommended alternative?
sub filter_relay () {
my($hostip, $hostname, $helo) = @_;
# Can't be wesmo.com unless it's one of our IP's.
if ($helo =~ /(^|.)mydomain\.com$/i)
{
if ($hostip ne 127.0.0.1 and $hostip ne $myserverIP)
{
syslog('info', Host $hostip said HELO $helo);
return(0, Go away. $hostip is not a mydomain.com machine);
}
}
# The hostname better match the helo string.
if (($helo =~ /^(\d{1,3})(.)(\d{1,3})(.)(\d{1,3})(.)(\d{1,3})$/)
($hostip ne $helo))
{
syslog('info', Host $hostip claims to be $helo);
return (0, Header forgery attempt, $ip claims to be $helo)
}
return (1, OK);
}
Thanks!
-Rich
* mimedefang.pl.in: If resending a message fails during streaming,
we bounce the message and log an error at LOG_CRIT importance.
* Modified C and Perl code so that filter_relay is called
when remote client connects rather than after MAIL FROM.
This means the $helo argument is NOT available!
*** NOTE INCOMPATIBILITY ***
filter_relay no longer has access to the HELO argument, nor
does the MIMEDefang spool directory exist when filter_relay is called.
___
Visit http://www.mimedefang.org and http://www.canit.ca
MIMEDefang mailing list
[EMAIL PROTECTED]
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Spamassassin 3.0 + MD 2.42
Hrmm.. Interesting. I'll have to look in to that. :) -Rich Even if you're *not* running SpamAssassin, you can pretty easily make use of the Mail::SPF::Query module directly within your mimedefang- filter, and if you can reject a message due to SPF policy and thereby avoid a SpamAssassin call, it's more efficient! Nels Lindquist * Quidquid latine dictum sit altum viditur. Whatever is said in Latin, sounds profound. ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] Spamassassin 3.0 + MD 2.42
I've been testing SA 3.0 (CVS) plus MD 2.42 (latest release), and I am happy to report that the two are working together rather well! In addition, the SPF rules within SpamAssassin are a must! (check out the Linux Journal article(s) regarding the implementation of SPF on your servers! http://www.linuxjournal.com/article.php?sid=7327 and the follow up article at http://www.linuxjournal.com/article.php?sid=7328) -Rich -- Richard West $14.95 Registrations mailto:[EMAIL PROTECTED] Wesmo Computer Services.com .net .org .tv .cchttp://www.wesmo.com Full Domain Web Hosting .BIZ .INFO MORE!! ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Additional Spamassassin Rules
David, yes, if you place them in /usr/share/spamassassin, you risk that they will be blown away during any upgrades.. I had placed mine in /etc/mail/spamassassin, as per a bunch of posts on various web pages and list archives. I didn't finda true cookbook (how to), though.. After placing mine in /etc/mail/spamassassin with appropriate names, I looked around supidly and felt that there had to be something else to get it to work. I was unaware of the md-mx-ctrl reread functionality. So, I was restarted mimedefang all together. I didn't see anything get hit by the new rules, hence my posting. Since I did the installation of the new rules, I have been running mimedefang + spamassassin as well as using my local .procmailrc file to filter through spamc. I know, this is redundant, but I wanted to see if one scored differently than the other. After about a day I had a good solid set of spam messages (about 60) that weren't bounced but flagged (properly) as spam. I saw that, over time, some of the messages were showing the new rules in the spamassassin report. I guess some just weren't tickling those rules. :) Thanks for the quick responses! -Rich David F. Skoll [EMAIL PROTECTED] wrote: Actually, the SpamAssassin docs state that you *shouldn't* drop local rules in /usr/share/spamassassin, since they will be removed when upgrading SpamAssassin (yes, I learned the hard way . Not if you number them 70_* or higher, I believe. Regards, David. Well, my /usr/share/spamassassin/ was wiped out, so it's safer to keep local rules in /etc/mail/spamassassin/. I grab a number of SpamAssassin rule files which are named according to their function and not with numbers, for example: http://www.merchantsoverseas.com/wwwroot/gorilla/sa_rules.htm http://www.timj.co.uk/linux/bogus-virus-warnings.cf The files from these sites should definitely go into /etc/mail/spamassassin/. ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Additional Spamassassin Rules
I would be interested to know if you are running spamassassin's spamd (via individual user's .procmailrc's), or if you are using MimeDefang in combination with Spamassassin and these new rules. Thanks for posting the link! I actually stumbled across this link via the Spamassassin web site the other night.. it wasn't easy to find. Having it in the archives is a good thing.. Maybe adding it to the MimeDefang website (just as a reference) would be nice, too. I, too, have installed these rules site-wide and tweaked the values accrodingly. In my tests with a single user account, it seems that MimeDefang + Spamassassin it not picking up these new rules while the .procmailrc recipie IS picking up the new rules (and scoring accordingly). Did you have to do any 'magic' to get the new rules to work with a MimeDefang+Spamassassin configuration? -Rich -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I have been running mimedefang/spamassassin/clamd on my servers for about six months. Recently, I added a front end server with quite a bit more horsepower to offload the load from my main servers. With the extra horsepower, I have been experimenting with ways to tighten our configuration and get a higher percentage hit on incoming spam. - From reading the archives, I found reference to a link which I have found useful and I want to make sure that others in the same position as myself know of its existance. http://www.merchantsoverseas.com/wwwroot/gorilla/sa_rules.htm This site provides several spamassassin rule bundles (filename.cf) that can be downloaded and dropped directly into /etc/mail/spamassassin where they will be used by spamassassin the next time is the process is restarted. I am currently using: backhair.cf bigevil.cf nov2rules.cf oct03_headers.cf oct03_rules.cf popcornonly.cf weedsonly.cf My spam positive hit rate immediatly increased when I added the new rules and I have seen no negative impact from their use. Because some of us on the list are not as programming capable as others, I would appreciate it if others could provide tips like this on the list. I benefit greatly from the experience of others on the list and probably could not be running the system I have without their input and suggestions. Thank You all - -- EKB ___ Visit http://www.mimedefang.org and http://www.canit.ca MIMEDefang mailing list [EMAIL PROTECTED] http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
