Pete, your metaphors are wonderful. ... I don't know why Declude boots its priority, but in the log file at DEBUG level, there are definitely lines that indicate that it has boosted its own CPU priority at the beginning, and lowering at the end.
I wasn't too worried about raising the priority of the persistent sniffer, because I figured that I needed to do so to keep up with the Decludes, that the sleeping it did was real (not just a tight "for loop") and that it happened in "micro-pulses" of cpu processing. It's priority would be irrelevant as it sleeps, and as it waits for I/O. Kind of like a Porsche waiting at a stop light in traffic. The Porsche would wait just as fast as everybody else, but would be incrementally faster through the intersections and in closing those feet between the next gridlocked car. My inflection point on this server came at a different volume than your test cases. That's because (I think) that your test case with the small server and IMail comes with Declude doing a far light load. My Declude does a fairly heavy load of text processing, my legacy text filters that were pre-Sniffer. The result being that your little server processes more messages with queuing, while I have a bigger server that processes fewer messages without queuing, but both run near their limit. My usual peak would be 1600 messages per hour (with multiple recipients per message). If I remember correctly, the "MaxPollTime" was originally much lower. I now use the full 4 seconds, but I don't know how often that's needed. I easily see Declude processes taking longer than this, sometimes at 100% of my CPU (with Task Manager update speed set to High) I also set "Lifetime" to 0 (because I don't expect the service to need stopping), and "Persistence" to 12 hours. I'm hedging my bet with Persistence, because I normally expect a twice daily rulebase update, and my update mechanism should initiate a reload. And if it's worth mentioning, I use "SingleLine" logging to cut down on disk time. Andrew 8) -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Friday, April 01, 2005 7:12 PM To: Colbeck, Andrew Subject: Re[8]: [sniffer] Persistent Sniffer On Friday, April 1, 2005, 6:44:13 PM, Andrew wrote: <snip/> CA> I also had a bad first experience with the persistent sniffer, and CA> switch to FireDaemon from SrvAny, and set the start priority for CA> sniffer to "Above Normal"; I can't prove it, but I'm convinced that CA> without that setting, Declude starves out other processes such as CA> the persistent sniffer, and then the other sniffer clients get CA> impatient and process the files themselves, causing more cpu CA> pressure. This is an interesting thought --- when I pump it through the wetware it reminds me of a discussion I heard where Declude uses priority boosting as a kind of quazi locking mechanism to keep IMail from swiping files - or something like that... It may be different now or I may be remembering a fantasy,... however - In any case - adjusting the priority of the persistent SNF engine upward would cause a fundamental shift in power on the machine such that SNF would tend to influence the scheduling of all tasks by proxy. This would happen because during a heavy load the persistent engine would run flat out on the CPU until all of the messages in it's batch had been scanned... That particular stretch of code, for the most part, is so tight that you might be lead to believe it were a race condition if you didn't know better. Anyway --- given a sufficient priority boost - perhaps even a mild one, SNF would have a tendency to alternately usurp and release all of the CPU resources causing a kind of wave action to set up. With the exception of critical threads and IO handling, the system would tend to rock back and forth between processing messages and scanning them. (This would happen quickly enough and in large enough batches of work that most human-driven processes would never notice. It's almost like moving to a collaborative multitasking model and away from a preemptive one... actually the more I think about it, it seems like it might be a kind of blended model...) All of the other message processing (including SNF clients) would have a tendency to sleep through the scanning process - as a result, no client would ever really get the chance to become unhappy with the time it had waited... It would be like waking up after 100 years in line at the ATM machine to discover that your transaction had already been completed... only you can't account for the 100 years that went by. It's an interesting state to imagine... Also, during this time no other messages would be processed very far so the load for SNF would always be balanced. Specifically, the more heavily loaded SNF became the slower new messages would be added to it's queue - therefore a balanced equilibrium would be maintained at the full limits of the available hardware. Buffers would tend to become more full before any processing was handed off (that's actually more efficient, if less responsive)... all sorts of odd adjustments would happen to the world... The load failure sensitivity would move to new message acquisition and delivery and away from SNF scanning... and I believe, ultimately, toward the IO processes which are all going to survive because they have a higher interrupt and thread priority and because the actual workload there is much less (waiting 10s of milliseconds for the next 1500 bytes of data to show up is like waiting for the next new-year's party if you're a 2.4Ghz CPU...). I wonder if this is what really happens... (sorry for thinking out loud - just hoping to share insight if it's in there somewhere) _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
