Thanks
Darin, this is very helpful and informative.
Have you had
an issue with MS DNS where it had a hard time resolving with root
servers? I have seen this and when I add a forwarded to our upstream
provider is works fine… but I would rather depend on root hits. My
root servers are up to date with internic.
Will
-----Original
Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]]
On Behalf Of Darin
Cox
Sent: Monday,
August 01, 2005 10:00 AM
To: [email protected]
Subject: Re:
[Declude.JunkMail] Declude Woes Fixed
We
finally found our problems over the weekend. They were two-fold. I'll
document them here in the hopes someone else with this problem will
benefit.
1.
IMail and Declude are using local MS DNS. However, we had set up
Forwarders in MS DNS, and those forwarded DNS servers were not
resolving most of the DNS-based tests (thanks to our upstream providers
:( ). So, Declude had to wait for all of the DNS-based tests to time
out, which meant total processing time on a single email was about 90
seconds.
2. We
had three DNS-based tests that were timing out: RRBL, SBBL, and
SOLID. SOLID is dead and the other two report server failure, which
may be temporary. To find out if you are using any dead or failing DNS
tests, switch to LOGLEVEL DEBUG for a couple of minutes or so, and
trace a message through to see if any tests are not responding.
After
removing the DNS forwarders and the failed or dead DNS tests,
processing time was down to 8 seconds. I doubt we'll see the overload
issue again unless we don't pay attention to dead or unavailable DNS
tests, or we have a real load issue.
-----
Original Message -----
Sent:
Monday, August 01, 2005 9:04 AM
Subject: RE:
[Declude.JunkMail] Declude Woes
Just an
update…
Last week I
had disabled Declude and cleaned out my spool directory. For the next
day I had not experienced any issues. My spooled messages are staying
well under 500 and mail is flowing exactly how I want it to (without
Declude). Now, we can get away with not having Declude Junkmail for a
bit, but I really needed virus scanning, so I re-enabled Declude
Antivirus. After I verified everything was running smoothly I
downloaded F-prot and removed Mcafee before the weekend and things have
also been running very smooth. Just a note, I had run Declude
Antivirus for years without any issues and I only seem to run into
these overflow problems when I introduce Declude Junkmail.
Since the
major issues I had last week I have made three changes… I have modified
the queue manager to process 100 messages at a time. I have disabled
Declude Junkmail. I have now moved from Mcafee to F-Prot. The only
one of these changes that has ensured that mail does not go into
overflow and backup is the disabling of Declude Junkmail. I’m not so
sure it’s a DNS issue because the Imail spam filters run perfectly
fine, which I am now using in place of Declude. They do not do as good
a job identifying spam, but they are better than nothing.
Things to
think about:
I was
interested in the statement that the mail server can be setup to not
bounce messages if they are detected as SPAM. This still confuses me,
because the BOUNCEONLYIFYOUMUST statement in the $default$.junkmail
file appears to actually bounce the messages. As it is, all my filters
are set to WARN and I don’t understand why changing them to
BOUNCEONLYIFYOUMUST will help.
I do not
believe I have bulk mailers on my system, but I will not discount it.
Does anyone have a good suggestion for determining this? After looking
through my logs and running Imail log analyzer none of my own IP’s
stand out for message counts and the invalid user entries I find are
from outside sources outside of my network. These connections are
always denied and mail does not get processed onto our server. I will
think about Declude HiJack.
I still want
to use Declude Junkmail, so I will test some of the suggestions put
into place such as whitelisting my IP range and disabling outbout virus
scanning, but other than these two suggestions, I don’t see any major
changes here that will affect me.
Will
Friday:
SpamContent
13400
SpamPhrase
19107
SpamFeatures
1283
SpamUrlDomain
35018
LocalDeliver
92331
RemoteDeliver
10526
Saturday:
SpamContent
10089
SpamPhrase
20739
SpamFeatures
1090
SpamUrlDomain
27194
LocalDeliver
83297
RemoteDeliver
9008
Sunday:
SpamContent
10425
SpamPhrase
18741
SpamFeatures
793
SpamUrlDomain
25756
LocalDeliver
81033
RemoteDeliver
10142
-----Original
Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]]
On Behalf Of Matt
Sent: Friday,
July 29, 2005 2:03 PM
To: [email protected]
Subject: Re:
[Declude.JunkMail] Declude Woes
Will,
What you probably have are users bulk-mailing through your server
(evidenced by the GSE files), issues with running both IMail Antispam
and Declude, and choosing a much slower than optimal virus scanner as
your one an only engine in a CPU-challenged environment. There could
well be issues with fragmentation and not enough disks as well.
The reports that you shared are somewhat inconclusive, primarily
because running Declude plus IMail Antispam will skew these numbers.
So there is no telling what the true volume is. If you run Declude,
you should turn IMail Antispam off and just rely on Declude alone.
Your config looks fine except that you should strongly consider
spending about $50 on an F-Prot license and using that instead of
McAfee since you are CPU challenged.
The fact that you are an ISP also speaks volumes to the need of
restricting bulk-mailing among your client base. Not only can they
redline your server, they can also get you blacklisted. Adding a
gateway will do nothing for this part of the issue. You need to look
into integrating Declude Hijack, but be prepared to sort through issues
in the week that follows integrating it.
To save CPU, you could whitelist your userbase in Declude by their IP
space by listing them in your global.cfg. You could also turn off
outbound virus scanning with Declude Virus Pro (not sure if that also
applies to Declude Standard).
IMO, this isn't an issue of sizing or a bug in Declude, this is all
about optimizing your configuration where it counts most, and putting
in place reasonable protections so that bulk-mailing isn't allowed to
go through your mail server. Jumping to a gateway will just be another
partial fix and it might not provide enough juice to prevent you from
backing up, especially if the load is primarily the result of customers
bulk-mailing through your server.
If you did everything that I identified here, you would save about 1/2
to 2/3 of your current CPU utilization. That might not cut it however
since you were clearly demanding more than 100% before, but I believe
it should.
Matt
Will wrote:
Well thank
you to everyone who responded to my need for perspective and assistance.
I believe I
will be in need of a mail gateway if I am to continue to use Declude.
Also, does
anyone know how to configure an Imail/Declude setup to not bounce spam
messages?
Will
-----Original
Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]]
On Behalf Of Michael
Jaworski
Sent: Friday,
July 29, 2005 1:11 PM
To: [email protected]
Subject: RE:
[Declude.JunkMail] Declude Woes
I can second
the need for a gateway defense when under attack. We run a Windows
shop and were being crushed under multiple dictionary attacks for two
domains on a daily basis. I took the daunting task to build our first
Linux box running Postfix. The first box was a tough start though I
had a employee who had Linux experience. We are running Postfix on
OpenBSD 3.6 with MySql for dynamic update ability. (I am still working
on grabbing additions, updates and deletions from SmarterMail admins so
we can throw all our domains in Postfix and update in realtime) After a
few weeks we added a second box in the event the first box went down.
The second box was a breeze since it was basically a duplication. Both
mx records now point at the two boxes. The hardware was old 500Mhz and
1ghz cpu with 512mbs of ram each. The 1ghz is primary and takes 75% of
the load without much effort with plenty of free memory. The whole
setup allows the main server running SmarterMail/Declude
Pro/Sniffer/F-Prot to respond quickly to POP, web mail and smtp traffic
requests.
The Linus
approach only should cost you some time and old equipment as the
software is free. Our experience over the last two years showed it was
worth climbing the short Linux learning cliff. And it is true ... they
run forever
One
important note not related to using a gateway: We never bounce spam
e-mail back to the "sender". The backscatter traffic can kill you and
skew your reports.
Michael
Jaworski
Puget Sound Network, Inc.
(206) 217-0400
(800) 599-9485
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]]
On Behalf Of Will
Sent: Friday,
July 29, 2005 9:38 AM
To: [email protected]
Subject: RE:
[Declude.JunkMail] Declude Woes
Well, I’m
back at it today.
Yesterday I
disabled Declude early in the day and started working mail back into
the spool directory from the overflow directory. This was a long
process, but by the end of the day I had gone from a backlog of 150,000
files in the spool and 134,000 in the overflow directory to about 1500
(that includes logs). During this time I needed to stop and restart
the queue manger a number of time. I did this to allow me to delete
all the .gse files, which I figured would save me time discarding
them. However, by the time I got down to 1500 files and started to
watch the spool it started to increase in size again; climbing to 4000
within a matter of minutes. I stopped and restarted the queuemanager
and these files were then processed. I verified they were actually
getting processed by sending test messages to myself. At this point I
was pretty upset and confused. I looked through the sys logs and found
nothing out of the ordinary, queuemanger would simply stop. I set all
the queuemanager setting back to default and tried again without luck.
I had to stop and restart it every few minutes to get it to process a
few thousand messages. Finally, I purchased an Imail service agreement
and upgraded to 8.21. Magically, it worked. The queumanger started to
deliver messages as soon as they arrived. My thought immediately went
into conspiracy mode. It seems like this has happened before where we
had a perfectly workable solution and something completely confusing
happened and an upgrade magically fixes it!
Anyway… I
re-enabled declude and let it run overnight. Now I have a backlog
again. There are mostly D*.SMD files in the spool right now with all
their delivery Q* files in the overflow directory (*shakes fist at
overflow directory*). Time to start the process again today. I’m
disabling declude to get those messages out and one thing to note,
after I have stopped the smtp server and added smtpd.exe backing into
the delivery application, I still have about 20+ declude.exe
processes. I have stopped and started it again as well as the
queuemanager and they are still there. In fact they are creating more
declude.exe processes as I watch. I’m trying to kill them, but they
just keep coming back… having to restart so I can start processing mail.
We are an
ISP and here are some random examples of some of our Imail daily
reports to give you an idea of what kind of traffic we see:
--
=====================================================
MailPure custom filters for Declude JunkMail Pro.
http://www.mailpure.com/software/
=====================================================
