Pete,
I follow this forum pretty well, however, having been out this week on business it seems I have lost alot with this new feature set. If you don't mind, could you define Persistent Sniffer? We average well over a million emails a day between two servers, what impact might I see on our server if I run this? What is the recommended settings? Thanks for the aid.
(Seems I'm in the "book writing" mode this evening... sorry for the bandwidth)
____________________ Performance Metrics:
Our NT4/SP6a test bed, running IMail/Declude/Sniffer in persistent mode. P2/450, 2x 5400rpm IDE drives, mirrored, 256M Ram (No giggles please - This is an intentionally underpowered server - how better to stress test a program like Sniffer?).
Sniffer in persistent mode on this box is able to process 120k msgs / month without issue. Logs show that each message on average now takes about 100ms total. Typical values are 20ms queue, 40ms scan though obviously some messages take longer and occasionally longer queue times do creep in.
Prior to testing the persistent version of Sniffer, message scan times varied wildly but averaged about 300ms per message with some messages taking 3-5 seconds while waiting for I/O and other processes (Web Mail, IMAP, etc...). In fact, I intentionally waited until the CPU was at 100% (green line 100%, red line 50%+) before starting the service to see how the creatures would handle the transition under heavy stress - The CPU dropped so much that at first I thought I had broken something (one of those "oops" moments).
The CPU now rests on the floor more often than not and generally runs peaks to about 50% unless something odd is going on - such as a defrag run.
YMMV - the above data is based on a very narrow data sample and only loosely calculated - and some of it is anecdotal. However most reports from the field seem to support the general scale of improvement.
On the back of the envelope I can calculate something like: 1 million per day is probably on the order of 125000 (1M/8hours) during a peak hour. 125000/3600 = about 35 per second. If message sniffer can scan about 10 per second on an overloaded p2/450, then on a 2.4ghz machine with plenty of memory we might expect at least a linear improvement - approximately 5x, but we will say 4x to be safe - 40/sec covers 35/sec so we have our million based on these assumptions.
IO not withstandng I would expect a persistent server version of Sniffer on a well provisioned server with a 2.4ghz processor to handle 1 million per day _IF_ that's all it had to do... since there's always more to do and this would be a maximum load scenario, dividing this across two servers should work nicely - though it would probably be time to start considering a third server.
Then again, you are probably not running generic single processor servers if you are handling 1 million messages per day ;-)
___________ Definition:
Probably the simplest definition of "Persistent Sniffer" as you put it is a "lightweight daemon". It can't actually be launched as a daemon/service on it's own, and it is still compatible with the self-organizing-automata version of Sniffer, but it offers many of the performance savings of a daemon/service - along with some added redundancy and flexibility. For example, if the persistent server instance of Sniffer fails, then the other instances simply return to their normal peer-server mode of operation so there is a drop in performance, but not a loss of service.
____________ More Detail:
Versions of Message Sniffer prior to 2-2 would always load the rule-base each time a message was to be scanned. Specifically, each instance of Message Sniffer was isolated and did the job itself. Up to 90% of the processing time typically required was bound in loading the rule-base file. On our NT test bed, for example, we would regularly see queue/scan times on the order of 1000/10, though more commonly 360/60 at the time when we developed version 2-2.
Beginning with Version 2-2, we implemented a "cellular peer-server" technology with Message Sniffer. This technology allows instances of Message Sniffer running on the same server to interact and self-organize, much like a colony of insects. In simplest terms there are two modes that Sniffer 2-2 can be in: "server" or "client". Whenever an instance finds no other "server" instances it will elect to be a "server" and will load the rule-base and process messages for "client" instances for some period of time, or until there are no other "client" instances available. This significantly reduces the number of times the rule-base is loaded for a given number of messages and allows Sniffer to dynamically adapt to the loads on a given server.
Beginning with the current beta (which will become version 2-3), we can force a sniffer instance to become a "persistent server". In simplest terms, we trick the creature into electing a server mode, and then further trick it into remaining alive for long periods - even if there are no clients to serve. The result is that all other instances of Sniffer _should_ elect to be clients - so there is only one instance that has loaded the rule-base and that instance only reloads the rule-base infrequently (about every 10 minutes unless there is an unusual collision event with other Sniffer instances).
________ Effects:
In general you should see a dramatic drop in processing times for any single message and a commensurate drop in I/O & CPU utilization by Sniffer. One instance will be "nailed up" as a server, and all other instances will normally elect to be clients.
____________________ Additional Comments:
Note that additional software is required to run a persistent instance of sniffer as a service on the winx platform - but there are a number of utilities for this and they all appear to be stable and fairly simple to use.
Note also that Sniffer still does not depend on any network resources and therefore does not represent any additional networking load, nor any potential network security risk.
This will change eventually as we do have plans to develop true Daemon/Service versions of Sniffer - but in the short/medium term we are putting our development resources into additional features and performance enhancements.
Currently Message Sniffer uses a single code base that compiles on all platforms without modifications and a vanishingly small amount of #ifdef style code switches - primarily for dealing with the Sleep() function(s). We would like to keep it this way for as long as possible - there are many benefits.
Hope this helps, Thanks! _M
This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
