Re[2]: [sniffer] reporting spam in bulk

[Pete McNeil]

On Wednesday, January 5, 2005, 7:16:50 PM, Matt wrote:

M> Pete,

M> I've been meaning to add a link to a script from within Killer WebMail
M> that will allow me to report things to you with a single click. If I do
M> this, am I correct in assuming that I should just use something like
M> CDONTS to construct a mail and place the original source as the body?
M> If not, what would be the preferred method?

I think that should work fine for reporting spam.

M> Note that I have original D*.SMD files for everything in the range of
M> E-mails that I would consider reporting (using Declude's COPYFILE).
M> Generally speaking, this would be a customized setup, although 
M> achievable by anyone with IMail and Declude.  The hack to KWM is just
M> some JavaScript to extract the spool data file name from my message
M> headers that I insert (full headers must be turned on in Web mail), and
M> this links to an ASP script on my server that handles everything else.

This all sounds like a good idea. There are likely to be a few
IMail/WebMail folks around for a while. This sounds like it's not for
the technically timid though.

Thanks,
_M



This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re: [sniffer] reporting spam in bulk

[Matt]

Pete,
I've been meaning to add a link to a script from within Killer WebMail 
that will allow me to report things to you with a single click.  If I do 
this, am I correct in assuming that I should just use something like 
CDONTS to construct a mail and place the original source as the body?  
If not, what would be the preferred method?

Note that I have original D*.SMD files for everything in the range of 
E-mails that I would consider reporting (using Declude's COPYFILE).  
Generally speaking, this would be a customized setup, although 
achievable by anyone with IMail and Declude.  The hack to KWM is just 
some JavaScript to extract the spool data file name from my message 
headers that I insert (full headers must be turned on in Web mail), and 
this links to an ASP script on my server that handles everything else.

Matt

Pete McNeil wrote:
On Wednesday, January 5, 2005, 4:03:28 PM, Rick wrote:
RR> 100's of spams a problem, LOL!
RR> Before sniffer I was facing around 10 thousand spams a day. But then I'm
RR> coordinating 1000's of domains, so on a per domain basis, it's actually very
RR> small.
RR> I think what I'll do is route a combined spam report email to a server
RR> script which will break it down and resubmit individual messages to your
RR> spam@ address. However, this will still be sent to you as an attachment. The
RR> advantage is that the original header info will be in place, the
RR> disadvantage is that you might still be ignoring messages with attachments,
RR> right?
Not necessarily. If they are not encoded we usually get good use out
of them even if they are attachments. The trick is that they will be
one message per message - so our automated tools will help us see what
we need to see.
It would be better to see them as a redirect, followed by a simple
forward, then as a last resort an attachment. As long as they are one
at a time we should be in good shape. I'm sure Gonzo is watching and
I'll talk to him about it. Once this starts happening we'll coordinate
and give you some feedback.
RR> If you don't take spam report messages with attachments, how would you be
RR> able to get the original internet header mail info?
The trick is that unless the message comes from a clean spamtrap we
don't trust the headers anyway. Under "abuse" rules, the entire
message is always suspect, so we will only dig into the headers if we
have good reason to trust what we're looking and, and we know what we
are looking for.
Spamtrap rules are different because the delivery chain is mapped and
consistent - so we know where the goodguy headers stop and the
questionable headers begin.
Thanks!
_M
PS: I've had one other call for this mechanism - a script that will
split multiple spam attachments and forward them to us. I would be
interested to see what you develop just in case it's applicable in
other places - or perhaps adaptable as a service in some way.

This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html
 

--
=
MailPure custom filters for Declude JunkMail Pro.
http://www.mailpure.com/software/
=
This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html

RE: Re[4]: [sniffer] reporting spam in bulk

[Mike Wiegers]

I use this program to send the messages with. It's setup to use with spamcop
but you can also send to [EMAIL PROTECTED]

http://www.daesoft.com/SpamSource/



This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re[4]: [sniffer] reporting spam in bulk

[Pete McNeil]

On Wednesday, January 5, 2005, 4:03:28 PM, Rick wrote:

RR> 100's of spams a problem, LOL!

RR> Before sniffer I was facing around 10 thousand spams a day. But then I'm
RR> coordinating 1000's of domains, so on a per domain basis, it's actually very
RR> small.

RR> I think what I'll do is route a combined spam report email to a server
RR> script which will break it down and resubmit individual messages to your
RR> spam@ address. However, this will still be sent to you as an attachment. The
RR> advantage is that the original header info will be in place, the
RR> disadvantage is that you might still be ignoring messages with attachments,
RR> right?

Not necessarily. If they are not encoded we usually get good use out
of them even if they are attachments. The trick is that they will be
one message per message - so our automated tools will help us see what
we need to see.

It would be better to see them as a redirect, followed by a simple
forward, then as a last resort an attachment. As long as they are one
at a time we should be in good shape. I'm sure Gonzo is watching and
I'll talk to him about it. Once this starts happening we'll coordinate
and give you some feedback.

RR> If you don't take spam report messages with attachments, how would you be
RR> able to get the original internet header mail info?

The trick is that unless the message comes from a clean spamtrap we
don't trust the headers anyway. Under "abuse" rules, the entire
message is always suspect, so we will only dig into the headers if we
have good reason to trust what we're looking and, and we know what we
are looking for.

Spamtrap rules are different because the delivery chain is mapped and
consistent - so we know where the goodguy headers stop and the
questionable headers begin.

Thanks!
_M

PS: I've had one other call for this mechanism - a script that will
split multiple spam attachments and forward them to us. I would be
interested to see what you develop just in case it's applicable in
other places - or perhaps adaptable as a service in some way.




This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

RE: Re[2]: [sniffer] reporting spam in bulk

[Rick Robeson]

100's of spams a problem, LOL!

Before sniffer I was facing around 10 thousand spams a day. But then I'm
coordinating 1000's of domains, so on a per domain basis, it's actually very
small.

I think what I'll do is route a combined spam report email to a server
script which will break it down and resubmit individual messages to your
spam@ address. However, this will still be sent to you as an attachment. The
advantage is that the original header info will be in place, the
disadvantage is that you might still be ignoring messages with attachments,
right?

If you don't take spam report messages with attachments, how would you be
able to get the original internet header mail info?


Rick Robeson
getlocalnews.com
[EMAIL PROTECTED] 



-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] Behalf Of Pete McNeil
Sent: Wednesday, January 05, 2005 11:49 AM
To: Rick Robeson
Subject: Re[2]: [sniffer] reporting spam in bulk


On Wednesday, January 5, 2005, 1:23:59 PM, Rick wrote:

RR> It would be incredibly convenient if we could report spam emails in bulk
RR> rather than individually (i.e. select all the spam emails in outlook and
RR> then forward via one email (with all the emails as attachments to you).

RR> During this latest storm, I'm finding that I don't have time to report
100's
RR> of spam emails and just delete most of them.

RR> Any way this process can be streamlined?

Unfortunately there really is no practical way for us to break down
packages like this and deal with them efficiently.

In any case, 100's of spam sounds like a more serious problem than a
spam storm. Have you checked for other problems?

_M



This E-Mail came from the Message Sniffer mailing list. For information and
(un)subscription instructions go to
http://www.sortmonster.com/MessageSniffer/Help/Help.html


This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re[2]: [sniffer] reporting spam in bulk

[Pete McNeil]

On Wednesday, January 5, 2005, 1:23:59 PM, Rick wrote:

RR> It would be incredibly convenient if we could report spam emails in bulk
RR> rather than individually (i.e. select all the spam emails in outlook and
RR> then forward via one email (with all the emails as attachments to you).

RR> During this latest storm, I'm finding that I don't have time to report 100's
RR> of spam emails and just delete most of them.

RR> Any way this process can be streamlined?

Unfortunately there really is no practical way for us to break down
packages like this and deal with them efficiently.

In any case, 100's of spam sounds like a more serious problem than a
spam storm. Have you checked for other problems?

_M



This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

RE: [sniffer] reporting spam in bulk

[Rick Robeson]

It would be incredibly convenient if we could report spam emails in bulk
rather than individually (i.e. select all the spam emails in outlook and
then forward via one email (with all the emails as attachments to you).

During this latest storm, I'm finding that I don't have time to report 100's
of spam emails and just delete most of them.

Any way this process can be streamlined?


Rick Robeson
getlocalnews.com
[EMAIL PROTECTED] 



This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

RE: Re[2]: [sniffer] RuleBase ktk82hrr

[GlobalWeb.net Billing]

Did that - redownloaded and all is fine...

Sorry; got tied up on support calls and didn't get a round to reposting the
results sooner

Thank you!


Sincerely,

Randy Armbrecht
Global Web SolutionsR, Inc.
804-346-5300 ext. 1
877-800-GLOBAL (4562) ext. 1
http://globalweb.net

Richmond's Internet Source since 1996!
WEB HOSTING including EMAIL beginning at $29/month!
DSL Starting at $39.95.month!
Non-Profits - receive a 25% discount on most services! 

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of Pete McNeil
Sent: Wednesday, January 05, 2005 12:28 PM
To: GlobalWeb.net Billing
Subject: Re[2]: [sniffer] RuleBase ktk82hrr

On Wednesday, January 5, 2005, 10:39:19 AM, GlobalWeb.net wrote:

GnB> Ours went from 11mb to 5mb with AUTH errors now...am looking into
it

Auth errors most likely indicate a bad download. Be sure to always check
downloads with snf2check before placing them in service.

Traffic on the web server was very, very high during the period of
transition since all downloads began to take 2-3 times as long, and so the
queue built up badly -- this would increase the chance of a failed or
corrupted download.

If you download again now then you should be able to get a fresh copy
without too much trouble.

_M




This E-Mail came from the Message Sniffer mailing list. For information and
(un)subscription instructions go to
http://www.sortmonster.com/MessageSniffer/Help/Help.html


This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re: [sniffer] RuleBase size

[Pete McNeil]

On Wednesday, January 5, 2005, 10:55:35 AM, MCSE wrote:

SSMOE> Just wanted to chime in here that the original changes seem to have 
helped
SSMOE> my rulebase as it has gone down from 11,355,504 bytes to 4,871,976 bytes 
but
SSMOE> I believe I'm still at the 'standard' rulebase strength.

This is true - only ineffective rules have been removed. It was always
the case that ineffective rules would be removed, but that process was
less efficient before. As a result of the new process, all rulebases
should be somewhat smaller and all rulebases will become more
effective (relative to rule strength setting) over time.

One of the changes that have been put in place is that we now have a
differential rule boost analysis in place. This process uses messages
that hit our spamtraps and compares the scan results for the standard
rulebase and the "fullbase" which includes ever rule ever put in our
system.

If a message arrives at a spamtrap, passes through the standard
rulebase, but is captured by the fullbase, then the rules involved in
that message are "boosted" by recording an accelerated hit rate. This
pushes the rule strength of these rules above the 1.0 threshold
defined for the "standard" settings.

This mechanism tends to push marginal rules back into the mainstream
as long as they are still viable as measured by our spamtraps. In the
past these rules would have normally remained in the 0.1 to 1.0 range
along with other much less effective rules.

(Sorry for writing another book...)

_M




This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re[2]: [sniffer] RuleBase ktk82hrr

[Pete McNeil]

On Wednesday, January 5, 2005, 10:39:19 AM, GlobalWeb.net wrote:

GnB> Ours went from 11mb to 5mb with AUTH errors now...am looking into it

Auth errors most likely indicate a bad download. Be sure to always
check downloads with snf2check before placing them in service.

Traffic on the web server was very, very high during the period of
transition since all downloads began to take 2-3 times as long, and so
the queue built up badly -- this would increase the chance of a failed
or corrupted download.

If you download again now then you should be able to get a fresh copy
without too much trouble.

_M




This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

[sniffer] 2 FYIs

[John Tolmachoff (Lists)]

Bill's update script: This has been working great, with the download size
aprox 1.8MB (rule base file is about 6.25MB) and time to download about 25
seconds. Thanks for the work Bill.

Rule base changes: Thanks to Pete for the hard work, the rule base size has
now changed from about 17MB to about 6.25MB. I am on maximum rules so my
rule file is larger.

John Tolmachoff
Engineer/Consultant/Owner
eServices For You




This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

[sniffer] RuleBase size

[Shaun Sturby, MCSE Optrics Engineering]

Just wanted to chime in here that the original changes seem to have helped
my rulebase as it has gone down from 11,355,504 bytes to 4,871,976 bytes but
I believe I'm still at the 'standard' rulebase strength.

 Shaun Sturby, MCSE
 Manager - Technical Services

 Optrics Engineering - Solution Partners & Network Specialists
 Email: [EMAIL PROTECTED]   Website: www.Optrics.com
 United States:  1740 S 300 West #10 Clearfield, UT, 84015
 Phone: 1-877-430-6240  Fax: (801) 705-3150
 Canada: 6810 104 St. Edmonton, AB Canada T6H 2L6
 Phone: 1-877-463-7638  Fax: (780) 432-5630


-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] Behalf Of Landry William
Sent: Tuesday, January 04, 2005 11:22 PM
To: sniffer@SortMonster.com
Subject: RE: Re[2]: [sniffer] RuleBase ktk82hrr

Yep, just checked mine rulebase too, went from 17mb to just under 25mb.
Things still appear to be functioning okay.

Bill

_

IMail Server has scanned this e-mail for Viruses and SPAM using  
Declude Virus & Declude Junkmail available from www.Optrics.com  


This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

RE: [sniffer] RuleBase ktk82hrr

[GlobalWeb.net Billing]

Ours went from 11mb to 5mb with AUTH errors now...am looking into it


Sincerely,

Randy Armbrecht
Global Web SolutionsR, Inc.
804-346-5300 ext. 1
877-800-GLOBAL (4562) ext. 1
http://globalweb.net

Richmond's Internet Source since 1996!
WEB HOSTING including EMAIL beginning at $29/month!
DSL Starting at $39.95.month!
Non-Profits - receive a 25% discount on most services!

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of Heimir Eidskrem
Sent: Wednesday, January 05, 2005 8:43 AM
To: sniffer@SortMonster.com
Subject: Re: [sniffer] RuleBase ktk82hrr

Something is not right with my rulebase I think.

I went from 11mb to 23mb then down to 5mb.

Is that right?

H.


Pete McNeil wrote:

>On Wednesday, January 5, 2005, 1:22:29 AM, Landry wrote:
>
>
>LW> Yep, just checked mine rulebase too, went from 17mb to just under 25mb.
>LW> Things still appear to be functioning okay.
>
>The effect is as if we tuned the rulebase to allow almost all rules 
>in... like setting the rule strength threshold to 0.1 (or less).
>Everyting should work fine - maybe even catch one or two more spam.
>
>I've found a trick with the internal data that might have caused the 
>problem and I'm running an experiment. If my experiment works then I 
>may be able to re-enable some of the new tuning functions right away.
>
>I will keep the list posted, of course.
>
>_M
>
>
>
>
>This E-Mail came from the Message Sniffer mailing list. For information 
>and (un)subscription instructions go to 
>http://www.sortmonster.com/MessageSniffer/Help/Help.html
>
>
>  
>

This E-Mail came from the Message Sniffer mailing list. For information and
(un)subscription instructions go to
http://www.sortmonster.com/MessageSniffer/Help/Help.html


This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re: [sniffer] RuleBase ktk82hrr

[Heimir Eidskrem]

Great.
Thanks,
H.
Pete McNeil wrote:
On Wednesday, January 5, 2005, 8:43:13 AM, Heimir wrote:
HE> Something is not right with my rulebase I think.
HE> I went from 11mb to 23mb then down to 5mb.
HE> Is that right?
Finally, yes, this is correct.
I began to optimize the weak-rule-removal mechanisms over the last two
days. When I implemented the final piece of critical code it uncovered
a bug in the rulebase compilers. (I call that a lens effect - when
fixing one part of a system uncovers problems in another part).
Around 0100 today (deep last night) the problem was discovered.
Somehow, by deprecating the weak rules in the system the rulebase
compilers became convinced to include nearly every rule that was ever
created. So, rulebases shot up to ~24 Mbytes with 213000+ rules!
I rolled back the changes and began hunting for the cause. A couple of
hours later I had found what I thought was the problem and a way to
work around it. I made an experiment of my fix and it worked, so I was
able to roll back in the weak-rule-removal optimizations.
As a result, we are now able to have the reduced size rulebases that
we originally hoped for with all of the weak rules properly
(aggressively) removed. Thus, rulebase sizes have dropped - yours to
5Mb.
It was quite an adventure, but it seems to be working fine so far.
Hope this helps,
_M

This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html
 

This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re[2]: [sniffer] RuleBase ktk82hrr

[Pete McNeil]

On Wednesday, January 5, 2005, 8:43:13 AM, Heimir wrote:

HE> Something is not right with my rulebase I think.

HE> I went from 11mb to 23mb then down to 5mb.

HE> Is that right?

Finally, yes, this is correct.

I began to optimize the weak-rule-removal mechanisms over the last two
days. When I implemented the final piece of critical code it uncovered
a bug in the rulebase compilers. (I call that a lens effect - when
fixing one part of a system uncovers problems in another part).

Around 0100 today (deep last night) the problem was discovered.
Somehow, by deprecating the weak rules in the system the rulebase
compilers became convinced to include nearly every rule that was ever
created. So, rulebases shot up to ~24 Mbytes with 213000+ rules!

I rolled back the changes and began hunting for the cause. A couple of
hours later I had found what I thought was the problem and a way to
work around it. I made an experiment of my fix and it worked, so I was
able to roll back in the weak-rule-removal optimizations.

As a result, we are now able to have the reduced size rulebases that
we originally hoped for with all of the weak rules properly
(aggressively) removed. Thus, rulebase sizes have dropped - yours to
5Mb.

It was quite an adventure, but it seems to be working fine so far.

Hope this helps,
_M




This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re: [sniffer] RuleBase ktk82hrr

[Heimir Eidskrem]

Something is not right with my rulebase I think.
I went from 11mb to 23mb then down to 5mb.
Is that right?
H.
Pete McNeil wrote:
On Wednesday, January 5, 2005, 1:22:29 AM, Landry wrote:
LW> Yep, just checked mine rulebase too, went from 17mb to just under 25mb.
LW> Things still appear to be functioning okay.
The effect is as if we tuned the rulebase to allow almost all rules
in... like setting the rule strength threshold to 0.1 (or less).
Everyting should work fine - maybe even catch one or two more spam.
I've found a trick with the internal data that might have caused the
problem and I'm running an experiment. If my experiment works then I
may be able to re-enable some of the new tuning functions right away.
I will keep the list posted, of course.
_M

This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html
 

This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html