On Thu, May 26, 2005 at 10:30:21AM -0400, Chris Santerre wrote:
[...]
> >My intention was to have some external opinion - magazine,
> >site review, you name it - saying that when summing up
> >cost/benefit of SA comparing to other things out there, it is
> >best by far (this is my opinion).
> >
> >Regards.
>
> Understood, and very good effort by you to educate them. Mostly all the
> reviews slam the cost benefit of SA with the "Pay an employee to support
> it." line of crap.
I actually took the time to do a cost analysis myself, because I got
tired of being dragged into Dog & Pony shows from anti-spam vendors
who tell upper management they offer solutions "with 0 false
positives" (IOW, all spam is quarantined in a folder where users can
still get it - certainly not what we mean by FPs) and "we
stop spam before it hits your mailserver" (IOW, we sell a service and
you point MX records to us, rather than installing our widget on your
border).
Hope this approach can be useful to others in the same boat. If I had
let them spend $250,000 per year for a couple of years and *then*
implemented SA and MIMEDefang, I'd get an award for reducing costs. I
just avoided the costs, which doesn't excite the bean counters. :)
Here is the list of the stats I keep track of in some reporting
scripts, monthly:
* Inbound email, total
* Inbound email flagged as SPAM
* Email not flagged
* Drops due to virus content
* Inbound email discarded (if it gets more than 10 points, we
just drop the mail silently)
* Amount of times sendmail discovered an SMTP RCPT Flood
* Amount of rejected spam, comprised of:
- sendmail anti-spam rules, such as domain not existing, relay
attempt, etc.
- host in the SBL or XBL
- other MIMEDefang tests that cause rejections - HELO validity
SPF failures, etc.
- no such user
- pre-greeting traffic (THANK YOU SENDMAIL!)
* Number of calls to our Helpdesk reporting an FP, or
a problem with a partner trying to send mail due to their SPF
or other mail config problems that I see as "spammy"
* Amount of time I spend supporting this install, at our business unit
chargeback rate (if your bean counters don't use this info, divide
admins' salaries by the amount of time to get your rate...)
* Hardware cost (we depreciate over 5 years, so I use this to
calculate the "cost" of the servers per month)
We also have a customized filter using MIMEDefang that takes any MS
executable and yanks it out of the email and quarantines it for 24
hours, until we get new Clam and McAfee signatures. We found that we
get a lot of valid executables via email (engineering software
updates, etc.) so full out rejections wouldn't work. The
temp. quarantine is great (the attachment is replaced with a URL that
will be valid in 24 hours) and has completely eliminated Email-based
worm and virus outbreaks (/me knocks on wood...). We found we were
getting the worms/viruses via email through our Asian locations as
much as 12 hours before we had DAT udpates. While we were fighting a
worm that was spreading so rapidly we took email offline we got a note
from McAfee saying "hey you probably won't get infected with this, but
there is a new DAT you may want to apply soon that will catch it. Uh,
thanks McAfee....
We made a way for our Helpdesk to manually "publish" a file from the
quarantine so its URL is valid if the user confirmed that he knew the
sender, he was expecting the file, and that he had contacted the
sender and confirmed the file he received was the one actually sent.
I only describe this because we track the amount of files actually
downloaded after the quarantine as well as the amount of calls (and
percentage of executables) that need to be published immediately
(mostly due to emergency patches from vendors). This gives us some
numbers so we can say "this did not disrupt users significantly or
disrupt business".
Here are my stats for the monthly report I give to management. They
*really* like that I tell them cost per user. Since I know the Total
Cost - hardware, time, software fees (none here!) - and I know users,
I can break it down the same way as my competition (vendors). Here
was some info from my April report:
Inbound Mail: 562051
Spam [Flagged]: 31228
Ham: 530823
Dropped(>10pts): 113983
Blocked: 1200801
Total non virus
SMTP attempts: 2438886
Viruses: 3530
SMTP RCPT Floods: 772
Quarantined Exe: 1414
Downloaded Exe
after Quarantine: 101
Early Quarantine
releases: 5
FP Reports: 3
And the numbers managers like:
Percent of Exes actually downloaded: 7.14%
Percent of Exes needed immediately: 0.35%
Percent of spam BLOCKED
instead of accepted: 89.21%
Percent of mail dropped due to spam: 6.06%
Percent of mail blocked: 63.86%
Percent Viruses: 0.19%
Percent Flagged Spam: 1.66%
Percent Ham: 28.23%
FP Percent: 0.0096%
YTD Average monthly cost/user: $0.03
So I can market too: "we block 90% of the spam before [the DATA phase
of SMTP] comes onto our network". "Our FP percent is less than
1/100th of a percent". "Only 0.35% of people who are emailed an
executable need it right away and call the Helpdesk."
I have no way for users to report missed spam, so I use my anecdotal
experience and guess we get about 97-99%. (Hopefully I will give
users a "report as spam" button in their MUA in the near future, but
then someone will have to manually review it for accuracy if there is
a lot, i.e. AOL user confusion over "Delete" vs. "Delete as Spam".)
I use as reference this study
http://www.networkworld.com/reviews/2004/122004spampkg.html
from Network World, and calculate my numbers the same they do their
numbers (PPV etc.).
HTH,
Matt
--
Matthew S. Cramer <[EMAIL PROTECTED]> Office: 717-396-5032
Infrastructure Security Analyst Fax: 717-396-5590
Armstrong World Industries, Inc. Cell: 717-917-7099
