Thanks for the additional data.
_M
At 01:12 PM 3/25/2004, you wrote:
I've been having trouble for the last 24 hrs or maybe a bit more with log uploads failing. The FTP either fails to connect, or it does connect and the upload begins and then fails after a small percentage done. Uploads are scheduled every 6 hours. Yesterday afternoon I tried renaming the log files from a couple failures and triggering the upload manually, and it also failed
An upload started a few mins ago, at 12:05 PM. It progressed almost to completion, and then ended with a reported failure from WS_FTP.
Glenn Z. WCNet
----- Original Message ----- From: "Pete McNeil" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Saturday, March 20, 2004 1:13 AM Subject: Re: [sniffer] Define Persistent sniffer.
> At 09:50 PM 3/19/2004, you wrote: > >Pete, > > I follow this forum pretty well, however, having been out this > > week on business it seems I have lost alot with this new feature set. If > > you don't mind, could you define Persistent Sniffer? We average well > > over a million emails a day between two servers, what impact might I see > > on our server if I run this? What is the recommended settings? Thanks > > for the aid. > > (Seems I'm in the "book writing" mode this evening... sorry for the bandwidth) > > ____________________ > Performance Metrics: > > Our NT4/SP6a test bed, running IMail/Declude/Sniffer in persistent mode. > P2/450, 2x 5400rpm IDE drives, mirrored, 256M Ram (No giggles please - This > is an intentionally underpowered server - how better to stress test a > program like Sniffer?). > > Sniffer in persistent mode on this box is able to process 120k msgs / month > without issue. Logs show that each message on average now takes about 100ms > total. Typical values are 20ms queue, 40ms scan though obviously some > messages take longer and occasionally longer queue times do creep in. > > Prior to testing the persistent version of Sniffer, message scan times > varied wildly but averaged about 300ms per message with some messages > taking 3-5 seconds while waiting for I/O and other processes (Web Mail, > IMAP, etc...). In fact, I intentionally waited until the CPU was at 100% > (green line 100%, red line 50%+) before starting the service to see how the > creatures would handle the transition under heavy stress - The CPU dropped > so much that at first I thought I had broken something (one of those "oops" > moments). > > The CPU now rests on the floor more often than not and generally runs peaks > to about 50% unless something odd is going on - such as a defrag run. > > YMMV - the above data is based on a very narrow data sample and only > loosely calculated - and some of it is anecdotal. However most reports from > the field seem to support the general scale of improvement. > > On the back of the envelope I can calculate something like: 1 million per > day is probably on the order of 125000 (1M/8hours) during a peak hour. > 125000/3600 = about 35 per second. If message sniffer can scan about 10 per > second on an overloaded p2/450, then on a 2.4ghz machine with plenty of > memory we might expect at least a linear improvement - approximately 5x, > but we will say 4x to be safe - 40/sec covers 35/sec so we have our million > based on these assumptions. > > IO not withstandng I would expect a persistent server version of Sniffer on > a well provisioned server with a 2.4ghz processor to handle 1 million per > day _IF_ that's all it had to do... since there's always more to do and > this would be a maximum load scenario, dividing this across two servers > should work nicely - though it would probably be time to start considering > a third server. > > Then again, you are probably not running generic single processor servers > if you are handling 1 million messages per day ;-) > > ___________ > Definition: > > Probably the simplest definition of "Persistent Sniffer" as you put it is a > "lightweight daemon". It can't actually be launched as a daemon/service on
> it's own, and it is still compatible with the self-organizing-automata > version of Sniffer, but it offers many of the performance savings of a > daemon/service - along with some added redundancy and flexibility. For > example, if the persistent server instance of Sniffer fails, then the other > instances simply return to their normal peer-server mode of operation so > there is a drop in performance, but not a loss of service. > > ____________ > More Detail: > > Versions of Message Sniffer prior to 2-2 would always load the rule-base > each time a message was to be scanned. Specifically, each instance of > Message Sniffer was isolated and did the job itself. Up to 90% of the > processing time typically required was bound in loading the rule-base file. > On our NT test bed, for example, we would regularly see queue/scan times on > the order of 1000/10, though more commonly 360/60 at the time when we > developed version 2-2. > > Beginning with Version 2-2, we implemented a "cellular peer-server" > technology with Message Sniffer. This technology allows instances of > Message Sniffer running on the same server to interact and self-organize, > much like a colony of insects. In simplest terms there are two modes that > Sniffer 2-2 can be in: "server" or "client". Whenever an instance finds no > other "server" instances it will elect to be a "server" and will load the > rule-base and process messages for "client" instances for some period of > time, or until there are no other "client" instances available. This > significantly reduces the number of times the rule-base is loaded for a > given number of messages and allows Sniffer to dynamically adapt to the > loads on a given server. > > Beginning with the current beta (which will become version 2-3), we can > force a sniffer instance to become a "persistent server". In simplest > terms, we trick the creature into electing a server mode, and then further > trick it into remaining alive for long periods - even if there are no > clients to serve. The result is that all other instances of Sniffer > _should_ elect to be clients - so there is only one instance that has > loaded the rule-base and that instance only reloads the rule-base > infrequently (about every 10 minutes unless there is an unusual collision > event with other Sniffer instances). > > ________ > Effects: > > In general you should see a dramatic drop in processing times for any > single message and a commensurate drop in I/O & CPU utilization by Sniffer. > One instance will be "nailed up" as a server, and all other instances will > normally elect to be clients. > > ____________________ > Additional Comments: > > Note that additional software is required to run a persistent instance of > sniffer as a service on the winx platform - but there are a number of > utilities for this and they all appear to be stable and fairly simple to use. > > Note also that Sniffer still does not depend on any network resources and > therefore does not represent any additional networking load, nor any > potential network security risk. > > This will change eventually as we do have plans to develop true > Daemon/Service versions of Sniffer - but in the short/medium term we are > putting our development resources into additional features and performance > enhancements. > > Currently Message Sniffer uses a single code base that compiles on all > platforms without modifications and a vanishingly small amount of #ifdef > style code switches - primarily for dealing with the Sleep() function(s). > We would like to keep it this way for as long as possible - there are many > benefits. > > Hope this helps, > Thanks! > _M > > > This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html >
This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
