Pete,
Nothing short of Awesome, I get it, I really get it :) One last question,
does this change the behavior any within Declude? I am running the 2-2 release, in
which, I saw a huge improvement in performance. Does this new 2-3 allow it to still
be called within Declude, if so, do I simply call it the same way in the Global.cfg
file, or are there additional parameters? Thanks again for the time and typing.
Keith
-----Original Message-----
From: [EMAIL PROTECTED] on behalf of Pete McNeil
Sent: Sat 3/20/2004 2:13 AM
To: [EMAIL PROTECTED]
Cc:
Subject: Re: [sniffer] Define Persistent sniffer.
At 09:50 PM 3/19/2004, you wrote:
>Pete,
> I follow this forum pretty well, however, having been out this
> week on business it seems I have lost alot with this new feature set. If
> you don't mind, could you define Persistent Sniffer? We average well
> over a million emails a day between two servers, what impact might I see
> on our server if I run this? What is the recommended settings? Thanks
> for the aid.
(Seems I'm in the "book writing" mode this evening... sorry for the bandwidth)
____________________
Performance Metrics:
Our NT4/SP6a test bed, running IMail/Declude/Sniffer in persistent mode.
P2/450, 2x 5400rpm IDE drives, mirrored, 256M Ram (No giggles please - This
is an intentionally underpowered server - how better to stress test a
program like Sniffer?).
Sniffer in persistent mode on this box is able to process 120k msgs / month
without issue. Logs show that each message on average now takes about 100ms
total. Typical values are 20ms queue, 40ms scan though obviously some
messages take longer and occasionally longer queue times do creep in.
Prior to testing the persistent version of Sniffer, message scan times
varied wildly but averaged about 300ms per message with some messages
taking 3-5 seconds while waiting for I/O and other processes (Web Mail,
IMAP, etc...). In fact, I intentionally waited until the CPU was at 100%
(green line 100%, red line 50%+) before starting the service to see how the
creatures would handle the transition under heavy stress - The CPU dropped
so much that at first I thought I had broken something (one of those "oops"
moments).
The CPU now rests on the floor more often than not and generally runs peaks
to about 50% unless something odd is going on - such as a defrag run.
YMMV - the above data is based on a very narrow data sample and only
loosely calculated - and some of it is anecdotal. However most reports from
the field seem to support the general scale of improvement.
On the back of the envelope I can calculate something like: 1 million per
day is probably on the order of 125000 (1M/8hours) during a peak hour.
125000/3600 = about 35 per second. If message sniffer can scan about 10 per
second on an overloaded p2/450, then on a 2.4ghz machine with plenty of
memory we might expect at least a linear improvement - approximately 5x,
but we will say 4x to be safe - 40/sec covers 35/sec so we have our million
based on these assumptions.
IO not withstandng I would expect a persistent server version of Sniffer on
a well provisioned server with a 2.4ghz processor to handle 1 million per
day _IF_ that's all it had to do... since there's always more to do and
this would be a maximum load scenario, dividing this across two servers
should work nicely - though it would probably be time to start considering
a third server.
Then again, you are probably not running generic single processor servers
if you are handling 1 million messages per day ;-)
___________
Definition:
Probably the simplest definition of "Persistent Sniffer" as you put it is a
"lightweight daemon". It can't actually be launched as a daemon/service on
it's own, and it is still compatible with the self-organizing-automata
version of Sniffer, but it offers many of the performance savings of a
daemon/service - along with some added redundancy and flexibility. For
example, if the persistent server instance of Sniffer fails, then the other
instances simply return to their normal peer-server mode of operation so
there is a drop in performance, but not a loss of service.
____________
More Detail:
Versions of Message Sniffer prior to 2-2 would always load the rule-base
each time a message was to be scanned. Specifically, each instance of
Message Sniffer was isolated and did the job itself. Up to 90% of the
processing time typically required was bound in loading the rule-base file.
On our NT test bed, for example, we would regularly see queue/scan times on
the order of 1000/10, though more commonly 360/60 at the time when we
developed version 2-2.
Beginning with Version 2-2, we implemented a "cellular peer-server"
technology with Message Sniffer. This technology allows instances of
Message Sniffer running on the same server to interact and self-organize,
much like a colony of insects. In simplest terms there are two modes that
Sniffer 2-2 can be in: "server" or "client". Whenever an instance finds no
other "server" instances it will elect to be a "server" and will load the
rule-base and process messages for "client" instances for some period of
time, or until there are no other "client" instances available. This
significantly reduces the number of times the rule-base is loaded for a
given number of messages and allows Sniffer to dynamically adapt to the
loads on a given server.
Beginning with the current beta (which will become version 2-3), we can
force a sniffer instance to become a "persistent server". In simplest
terms, we trick the creature into electing a server mode, and then further
trick it into remaining alive for long periods - even if there are no
clients to serve. The result is that all other instances of Sniffer
_should_ elect to be clients - so there is only one instance that has
loaded the rule-base and that instance only reloads the rule-base
infrequently (about every 10 minutes unless there is an unusual collision
event with other Sniffer instances).
________
Effects:
In general you should see a dramatic drop in processing times for any
single message and a commensurate drop in I/O & CPU utilization by Sniffer.
One instance will be "nailed up" as a server, and all other instances will
normally elect to be clients.
____________________
Additional Comments:
Note that additional software is required to run a persistent instance of
sniffer as a service on the winx platform - but there are a number of
utilities for this and they all appear to be stable and fairly simple to use.
Note also that Sniffer still does not depend on any network resources and
therefore does not represent any additional networking load, nor any
potential network security risk.
This will change eventually as we do have plans to develop true
Daemon/Service versions of Sniffer - but in the short/medium term we are
putting our development resources into additional features and performance
enhancements.
Currently Message Sniffer uses a single code base that compiles on all
platforms without modifications and a vanishingly small amount of #ifdef
style code switches - primarily for dealing with the Sleep() function(s).
We would like to keep it this way for as long as possible - there are many
benefits.
Hope this helps,
Thanks!
_M
This E-Mail came from the Message Sniffer mailing list. For information and
(un)subscription instructions go to
http://www.sortmonster.com/MessageSniffer/Help/Help.html
<<winmail.dat>>
