If you were to do a cache like that, you might do it using a hash of the
content. The hash wouldn't take much memory or processing time. It's an
interesting concept (we've got features like this built into server software
we have on our drawing boards.) The neat thing about this is that if the
network path changes for the message, it will still be recognized...
versions of this capability are a great way to prevent DOS attacks like
nimda - once a bad request is cached, all others like it are rejected out of
hand. For example, a hack attempt would be seen as multiple requests to a
web server with a 404 result in a specific time limit... presumably, a user
making the attempt would see the first 404 and give up... if (say 5 in 1
minute) too many of these happen in a period then the signature of the
request would be cached as a 32 bit hash value and rejected until the
"attack" stops (no such requests for a period of time.) A similar mechanism
can be used as part of a heuristic to reject spam where the signature of a
message to a bad user is hashed, and if that message is sent to too many bad
users in a period, then that message is loaded into the rejection scheme -
thus thwarting dictionary attacks... Similarly a message that fails some
other test could also be loaded and then automatically rejected even if the
case for failure of the first test is resolved by the attacker. Build in
some automated notification mechanisms and you have yourself a very robust
system... say for example trusted servers were able to share rejection
schemes... then an attack on one server would automatically be rejected by
all participating servers - protecting those that had not yet seen the
attack... but I'm letting the cat out of the bag now.
Hopefully this is a helpful glimpse.
Sorry for rambling.
_M
| -----Original Message-----
| From: [EMAIL PROTECTED]
| [mailto:[EMAIL PROTECTED]]On Behalf Of Charles Frolick
| Sent: Tuesday, September 25, 2001 11:16 AM
| To: [EMAIL PROTECTED]
| Subject: RE: [Declude.JunkMail] Declude crashing and taking up 99% CPU
|
|
| Hey Scott,
|
| Just throwing an idea out there, since most spam mail hits multiple
| addresses in the same server, often as seperate messages, how
| about a failed
| message cache that could automatically fail a message if it failed in the
| last hour or some other configurable time span? It would probably have to
| rely on a weghted combo match of several headers since they sometimes use
| different senders or servers, and I've even seen them add the recipients
| name or email in the subject and/or the body.
|
| This would help in that you are not completely reprocessing the same spam
| message hundreds possibly thousands of times on high volume servers.
|
| The only drawback I can think of might be the extra resources to
| manage the
| cache if it got huge.
|
| Another thought, it could also be useful for the AV side.
|
| Chuck Frolick
| ArgoNet, Inc.
| -----Original Message-----
| From: [EMAIL PROTECTED]
| [mailto:[EMAIL PROTECTED]]On Behalf Of R. Scott Perry
| Sent: Tuesday, September 25, 2001 9:45 AM
| To: [EMAIL PROTECTED]
| Subject: Re: [Declude.JunkMail] Declude crashing and taking up 99% CPU
|
|
|
| >I have tried my best to help with this issue, research, testing, etc.
| >During a normal business day, declude is awesome - working very nicely as
| >advertised and very reliable. Perhaps, on an email server with less
| >traffic, some declude customers wouldn't even know of any reliability
| >issues.
|
| From the information we have gathered so far, the 99% CPU issue only
| affects Declude JunkMail on v1.24 and higher, on highish volume systems
| (our mail server processes about 3,000 E-mails on an average day, and we
| haven't seen the problem even once here).
|
| >During a normal time period, we do pass a ton of emails, but during the
| >moments when the flood gates open and spam flows in, this causes many
| >declude.exe's to run. Some use 99% of the CPU, while others
| simply use up
| >memory. In my opinion, an opinion from a non-programmer, I would think
| that
| >there should only be one declude.exe running (as a service perhaps).
|
| That's actually an IMail limitation. Most programs do work that way --
| there is one process that handles all requests. However, IMail is set up
| to start a new process for each E-mail that needs to be delivered. It
| seems pretty inefficient (actually, a terrible design now that the
| Microsoft heap issue has been identified), but they chose it for a
| reason. Declude is stuck with that architecture -- IMail will start one
| declude.exe process for each E-mail, no matter what. Note that without
| Declude running, IMail will start one smtp32.exe process for each E-mail.
|
| Note that the 99% CPU issue was not reported before v1.24.
|
| >In the event that declude can't handle a "High Surge of Spam", then
| >declude should
| >pass the junk mail on to imail for normal processing.
|
| I haven't yet seen Declude not be able to handle high volumes of
| spam. It's survived spam attacks, where a spammer will try to
| send through
| 100,000's of E-mails.
|
| >With Imail limiting smtp32.exe to just 30 instances (configurable by the
| >registry), this would not cause a problem with the desktop heap.
| You could
| >even trim down the smtp32.exe count to 10. Then, when the flood gates
| open,
| >the mail will just be a little slow, but reliable.
|
| This is a separate issue, and again an IMail issue. This is the problem
| that causes the "DLL initialization failure" crashes with IMail,
| and causes
| mail delivery to be sooooooo slow when there is overflow (E-mail arriving
| when all processes are being used).
|
| We are working on a way to minimize this problem, where Declude will take
| over the limiting of the number of processes running at once, and will
| create a separate queue for E-mail that hasn't been attempted yet
| (normally, IMail just puts the mail in the spool, which should
| just contain
| E-mail that couldn't be delivered on the first attempt). Then, as soon as
| there are free processes available, Declude will start things
| back up again
| with multiple processes (rather than waiting 1/2 hour or so for the next
| queue run, which only starts up 1 process).
|
| That should help reduce the DLL initialization problem, as well as the
| IMail slow mail delivery of overflow. It's not going to be perfect, since
| IMail can start several new processes on the same timeslice, which means
| that all of those processes will start before they have a chance to stop
| themselves. So, for example, if Windows will let you run 35
| server-started
| processes before running out of their mystery heap, and IMail can start up
| to 10 processes on one timeslice, Declude would have to start the overflow
| procedure after 25 processes. If it let the 26th in, then IMail could
| create 10 more before Declude next had a chance to see how many processes
| were in memory, which would pass the 35 process limit, and crash
| the server.
|
| >I think this would make a more reliable server. Again, I am
| only TRYING to
| >help and give ideas. My intent is not to step on anyones toes.
|
| That's not a problem, ideas are what makes improvements possible.
| -Scott
|
| ---
|
| This E-mail came from the Declude.JunkMail mailing list. To
| unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
| type "unsubscribe Declude.JunkMail". You can E-mail
| [EMAIL PROTECTED] for assistance. You can visit our web
| site at http://www.declude.com .
|
|
| ---
|
| This E-mail came from the Declude.JunkMail mailing list. To
| unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
| type "unsubscribe Declude.JunkMail". You can E-mail
| [EMAIL PROTECTED] for assistance. You can visit our web
| site at http://www.declude.com .
|
---
This E-mail came from the Declude.JunkMail mailing list. To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type "unsubscribe Declude.JunkMail". You can E-mail
[EMAIL PROTECTED] for assistance. You can visit our web
site at http://www.declude.com .
