Re: [Declude.JunkMail] OT: Windows 2000 Performance Monitor
Hi Matt, As Darrell pointed out, short-lived processes are problematic to monitor as it's difficult to get a continuous aggregate read for a particular type of process. If you're just looking for more general statistics on processor, IO, storage, RAM, etc. it works quite well to log it to a SQL Server for trending. We use the perfcheck.dll that came with either W2K Resource Kit or Support Tools and call it from a SQL job to loop through defined counters from one table every few minutes and store the sampled value in another. Currently we just clear it out when it gets too big, but have been considering aggregating to report tables for daily, weekly, and/or monthly usage trends. You can go hog wild with these things, but we've found a few simple counters are enough and give us the necessary info to project hardware needs. Darin. - Original Message - From: Matt [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, March 17, 2004 9:49 PM Subject: [Declude.JunkMail] OT: Windows 2000 Performance Monitor I've never bothered to run monitoring before, but I need to do so now so that I can make more informed decisions. Does anyone have a good config/setup that they want to share which is most effective at tracking usage primarily related to an IMail/Declude/Sniffer setup? Should I be storing this data in SQL Server? Etc. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. _ [This E-mail virus scanned by 4C Web] _ [This E-mail virus scanned by 4C Web] --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] OT: Windows 2000 Performance Monitor
Darin and Darrell, Thanks to both of you for the pointers. Certainly that saved me some time. I did manage to capture all of the process information by selecting all instances for the Process % Processor Time. Using System Monitor it was easy to set up graphs from the logs showing this info, including all numbered instances. The graphs though suck. The averages seem to help a bit more. One piece of data that I seem to be missing is F-Prot's usage though. Any idea what that shows up as? I'm looking to compare that to avgscan.exe. Also, do you guys (or anyone else) have an idea about how disk load times might be reflected as far as utilization goes? I have over 60 custom filters that get loaded for almost every message, though they only get run about 60%-70% of the time on average. I'm thinking that my excessive filter use might be an important component of my processor peaks, peaks that I need to better control because of my current mixed environment with hosting. Sniffer for instance reports very low utilization as a process, however loading the rulebase according to Pete represents about 90% of the time to process a message, but it doesn't appear to be reflected in my stats as utilization except when tracking the overall processor usage. Regardless of the pieces that are still lacking, I was definitely able to get a better grasp on some other things. Thanks, Matt Darin Cox wrote: Hi Matt, As Darrell pointed out, short-lived processes are problematic to monitor as it's difficult to get a continuous aggregate read for a particular type of process. If you're just looking for more general statistics on processor, IO, storage, RAM, etc. it works quite well to log it to a SQL Server for trending. We use the perfcheck.dll that came with either W2K Resource Kit or Support Tools and call it from a SQL job to loop through defined counters from one table every few minutes and store the sampled value in another. Currently we just clear it out when it gets too big, but have been considering aggregating to report tables for daily, weekly, and/or monthly usage trends. You can go hog wild with these things, but we've found a few simple counters are enough and give us the necessary info to project hardware needs. Darin. - Original Message - From: Matt [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, March 17, 2004 9:49 PM Subject: [Declude.JunkMail] OT: Windows 2000 Performance Monitor I've never bothered to run monitoring before, but I need to do so now so that I can make more informed decisions. Does anyone have a good config/setup that they want to share which is most effective at tracking usage primarily related to an IMail/Declude/Sniffer setup? Should I be storing this data in SQL Server? Etc. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] OT: Windows 2000 Performance Monitor
Hi Matt, Graphs are pretty but not often useful. We use daily/monthly avg and stdev for meaningful info. F-prot should show up as the exe name, fpcmd.exe Certainly threading with files remaining loaded in memory and checked for changes periodically would be a lot quicker than loading the files every time, but lacking that...shrug. I haven't seen anyone use one in years, but a ramdisk might help. That way the files do remain loaded in ram. I'm sure there's a product somewhere that still does that. I don't know of any way to separate out disk usage by one process or exe, so any performance counters there probably wouldn't do much good. If you're having disk IO problems overall, though I would suggest running SCSI RAID level 1 with multiple, striped disks so reading can be done from multiple disks at once. Darin. - Original Message - From: Matt [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, March 18, 2004 4:16 PM Subject: Re: [Declude.JunkMail] OT: Windows 2000 Performance Monitor Darin and Darrell, Thanks to both of you for the pointers. Certainly that saved me some time. I did manage to capture all of the process information by selecting all instances for the Process % Processor Time. Using System Monitor it was easy to set up graphs from the logs showing this info, including all numbered instances. The graphs though suck. The averages seem to help a bit more. One piece of data that I seem to be missing is F-Prot's usage though. Any idea what that shows up as? I'm looking to compare that to avgscan.exe. Also, do you guys (or anyone else) have an idea about how disk load times might be reflected as far as utilization goes? I have over 60 custom filters that get loaded for almost every message, though they only get run about 60%-70% of the time on average. I'm thinking that my excessive filter use might be an important component of my processor peaks, peaks that I need to better control because of my current mixed environment with hosting. Sniffer for instance reports very low utilization as a process, however loading the rulebase according to Pete represents about 90% of the time to process a message, but it doesn't appear to be reflected in my stats as utilization except when tracking the overall processor usage. Regardless of the pieces that are still lacking, I was definitely able to get a better grasp on some other things. Thanks, Matt Darin Cox wrote: Hi Matt, As Darrell pointed out, short-lived processes are problematic to monitor as it's difficult to get a continuous aggregate read for a particular type of process. If you're just looking for more general statistics on processor, IO, storage, RAM, etc. it works quite well to log it to a SQL Server for trending. We use the perfcheck.dll that came with either W2K Resource Kit or Support Tools and call it from a SQL job to loop through defined counters from one table every few minutes and store the sampled value in another. Currently we just clear it out when it gets too big, but have been considering aggregating to report tables for daily, weekly, and/or monthly usage trends. You can go hog wild with these things, but we've found a few simple counters are enough and give us the necessary info to project hardware needs. Darin. - Original Message - From: Matt [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, March 17, 2004 9:49 PM Subject: [Declude.JunkMail] OT: Windows 2000 Performance Monitor I've never bothered to run monitoring before, but I need to do so now so that I can make more informed decisions. Does anyone have a good config/setup that they want to share which is most effective at tracking usage primarily related to an IMail/Declude/Sniffer setup? Should I be storing this data in SQL Server? Etc. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. _ [This E-mail virus scanned by 4C Web] _ [This E-mail virus scanned by 4C Web] --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] OT: Windows 2000 Performance Monitor
Darin, What I'm trying to do is polish up my config in the expectation of a lot more business over the short term. Things are definitely moving around here. I'm also concerned about storms of viruses and spam attacks. I figure that now is the time to get a handle on what needs to be done in order to improve efficiency instead of when it is biting me in the a**. I figured out the fpcmd.exe thing. Curiously it didn't show up after 15 minutes when polled every second. This speaks to the incredible performance of F-Prot, in fact here are some stats from the last hour comparing fpcmd.exe to avgscan.exe: avgscan.exe - Average: 1.556 Maximum: 57.813 fpcmd.exe - Average: 0.486 Maximum: 3.125 My server started getting hammered about 30 minutes ago by a NetSky.D virus storm, once every couple of seconds all from the same computer. This has been happening for several days now in fact, but it's hit or miss when it happens. As a result, I have data showing up to 12 Declude processes at once. I assume that the other peak number of processes were also reached during that time, with avgscan.exe recording up to 7 processes, but fpcmd.exe only 2. Sniffer also only made it to 3, probably because the viruses were all blocked. I also looked a DEBUG virus logs a few days ago and found that the scan time was about 4 times longer for avgscan.exe than it was for fpcmd.exe...conclusion: AVG is not a good candidate for higher volumes, even in 32-bit mode. I think I can save myself a lot of processing by finding a more efficient second scanner, one on par with F-Prot. If my box was not doubling as a Web server, I would be willing to push it much harder. It's the peaks that bother me right now, and they're massive. This is one of the reasons why I suggested that the SKIPIFWEIGHT stuff appear in the Global.cfg, thinking that it would save the loading of these files and the minimal parsing necessary to tell Declude the limit has already been reached. Good E-mail and virus scanning takes more processing power by far than spam does because it hits every test. Regarding your suggestion about a RAMDISK, Pete is actually working on a persistent instance of Sniffer with all sorts of fancy words to describe how it works :) My machine is a 4 active drive RAID 5 array on some 10K Cheetahs. It was build for redundancy/reliability and not necessarily for speed. It does great as a Web server, but as a gateway machine, I understand fully the challenges and how that affects your choices. When I move the mail scanning onto a different box, it will be optimized for speed. Still though, I don't want to be throwing something like a inefficient virus scanner at a setup and impacting my ability to scale. It could also be that I chose inefficient switches when I configured AVG, so I'll take a look at that as well. If anyone wants to help test out virus scanners for efficiency, contact me off list and we'll come up with a standard way to test them (probably on my box if folks don't mind). Thanks again, Matt Darin Cox wrote: Hi Matt, Graphs are pretty but not often useful. We use daily/monthly avg and stdev for meaningful info. F-prot should show up as the exe name, fpcmd.exe Certainly threading with files remaining loaded in memory and checked for changes periodically would be a lot quicker than loading the files every time, but lacking that...shrug. I haven't seen anyone use one in years, but a ramdisk might help. That way the files do remain loaded in ram. I'm sure there's a product somewhere that still does that. I don't know of any way to separate out disk usage by one process or exe, so any performance counters there probably wouldn't do much good. If you're having disk IO problems overall, though I would suggest running SCSI RAID level 1 with multiple, striped disks so reading can be done from multiple disks at once. Darin. - Original Message - From: "Matt" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, March 18, 2004 4:16 PM Subject: Re: [Declude.JunkMail] OT: Windows 2000 Performance Monitor Darin and Darrell, Thanks to both of you for the pointers. Certainly that saved me some time. I did manage to capture all of the process information by selecting "all instances" for the Process % Processor Time. Using System Monitor it was easy to set up graphs from the logs showing this info, including all numbered instances. The graphs though suck. The averages seem to help a bit more. One piece of data that I seem to be missing is F-Prot's usage though. Any idea what that shows up as? I'm looking to compare that to avgscan.exe. Also, do you guys (or anyone else) have an idea about how disk load times might be reflected as far as utilization goes? I have over 60 custom filters that get loaded for almost every message, though they only get run about 60%-70% of the time on average. I'm thinking that my excessive filter use might be an important component of my processor peak
Re: [Declude.JunkMail] OT: Windows 2000 Performance Monitor
At 05:18 PM 3/18/2004, you wrote: Regarding your suggestion about a RAMDISK, Pete is actually working on a persistent instance of Sniffer with all sorts of fancy words to describe how it works :) My Matt, I'm pretty happy with the beta - so far no glitches or gotchas on my test server nor any reported from the list. If you've not yet tried it you might give it a shot. The production version is not likely to change except for cleaning out some debug/monitoring code and adding a few tuning / management features. The last thing we'll need to do is chose an MS Service stub utility to recommend, but in the mean time it's been working ok on my box as a .cmd nailed up and hidden behind a password protected screen saver setting. Just a thought. _M --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] OT: Windows 2000 Performance Monitor
Pete, I've definitely been reading up on the other list and all looks great so far. I can't leave my server logged into, so I'm going to sit back for a little bit longer until you get the service stub thing going (recommendations, et. al) It also appears that defeating Sniffer with a weight qualifier may be of almost no consequence at this point, so I'm giving up on that project. I had run into a roadblock where I couldn't get the code from Sniffer using VBScript and the directions that we found said it would work the way we did it. The funny thing is that we could get other information about the process...just not the code. Anyway, I learned a few new tricks. Thanks again, Matt Pete McNeil wrote: At 05:18 PM 3/18/2004, you wrote: Regarding your suggestion about a RAMDISK, Pete is actually working on a persistent instance of Sniffer with all sorts of fancy words to describe how it works :) My Matt, I'm pretty happy with the beta - so far no glitches or gotchas on my test server nor any reported from the list. If you've not yet tried it you might give it a shot. The production version is not likely to change except for cleaning out some debug/monitoring code and adding a few tuning / management features. The last thing we'll need to do is chose an MS Service stub utility to recommend, but in the mean time it's been working ok on my box as a .cmd nailed up and hidden behind a password protected screen saver setting. Just a thought. _M --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
[Declude.JunkMail] OT: Windows 2000 Performance Monitor
I've never bothered to run monitoring before, but I need to do so now so that I can make more informed decisions. Does anyone have a good config/setup that they want to share which is most effective at tracking usage primarily related to an IMail/Declude/Sniffer setup? Should I be storing this data in SQL Server? Etc. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
Re: [Declude.JunkMail] OT: Windows 2000 Performance Monitor
Matt, I monitor a bunch of counters (memory, cpu, process, disk, network, etc) on our servers. I roll the perf logs on a daily basis. The hard thing in tracking this stuff is that when you add process counters there is no way to track all of the individual processes for declude/imail/sniffer. What you will see is each Declude process will show like Declude#1, Declude#2, etc, etc. What I ended up doing is setting the process counters up at a busy time on my server to capture as many of the ...#1 processes. Darrell Matt writes: I've never bothered to run monitoring before, but I need to do so now so that I can make more informed decisions. Does anyone have a good config/setup that they want to share which is most effective at tracking usage primarily related to an IMail/Declude/Sniffer setup? Should I be storing this data in SQL Server? Etc. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com. Check Out DLAnalyzer a comprehensive reporting tool for Declude Junkmail Logs - http://www.dlanalyzer.com --- [This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)] --- This E-mail came from the Declude.JunkMail mailing list. To unsubscribe, just send an E-mail to [EMAIL PROTECTED], and type unsubscribe Declude.JunkMail. The archives can be found at http://www.mail-archive.com.
