Dan,

Think about the fact that the entire rulebase and the executable don't have to be read except on occasion when running as a service, and memory doesn't need to be allocated to the application each time it is run.  This should also help with the mystery heap issues that can occur on an overloaded IMail/Declude server under very heavy load.  If you ever get bursts of traffic, this can come in handy.

Matt



Dan Horne wrote:
So basically, what you are saying is that my volume is really too low to take advantage of the persistent sniffer (and such may actually decrease my performance), and I should stick with the non-service version.  Is that right?  That is about what I thought (without the details of how sniffer works, I just wanted to be sure).

Thanks, Pete.

Dan Horne

  
-----Original Message-----
From: [EMAIL PROTECTED] 
[mailto:[EMAIL PROTECTED]] On Behalf Of Pete McNeil
Sent: Tuesday, August 02, 2005 4:09 PM
To: Dan Horne
Subject: Re[2]: [sniffer] Sniffer taking a long time?

After following through all of this and looking at the .stat 
file, I think I see what's going on.

Now that it is running and producing a .stat file, the flow 
rate is very low. According to the stat data, about 6 msgs / minute.

Note the poll and loop times are in the 450 - 550 ms range.

SNF with the persistent engine is built for high throughput, 
but it's also built to play nice.

The maximum poll time gets up to 2 seconds or so (sound familiar?)

If there are no messages for a while, then everything slows 
down until the first message goes through. For that first 
message, the SNF client will probably wait about 2 seconds 
before looking for it's result because that's what the stat 
file will tell it to do.

Since the next message probably won't come around for a few 
seconds, that next message will probably wait about 2 seconds also.

If you were doing 6 messages a second then all of the times 
would be much lower and so would the individual delays.

When you turn off the persistent instance, each new message 
causes a client to look and see if there are any other peers 
acting a servers... Since the messages are far and few 
between, the client will elect to be a server (momentarily), 
will find no work but it's own, will process it's own message 
and leave. -- This is the automatic peer-server mode. It will 
always work like this unless more than one message is being 
processed at the same moment.

In peer-server mode, since there is nothing else going on and 
no persistent instance to coordinate the operations, each 
message will get processed as fast as the rulebase can be 
loaded and then the program will drop.

When the persistent instance is introduced, it sets the pace 
- and sicne there are no other messages, each client will 
wait about 2 seconds (or half a second or so with the .stat 
file contents you show) before it begins looking for it's results.

The server instance will also wait a bit before looking for 
new jobs so that the file system isn't constantly being scanned.

Of course, if a burst of messages come through then the 
pacing will speed up as much as necessary to keep up with the volume.

Hope this helps,

_M

On Tuesday, August 2, 2005, 3:38:52 PM, Dan wrote:

DH> No, I followed your instructions exactly (and not for the first 
DH> time).  I didn't add those extra values until today.  Prior to  
DH> adding the AppDirectory value, the service was taking a minute to 
DH> scan emails;  after adding it the scan time went to around 2 
DH> seconds.  I can't get it any  lower than that.  Initially 
mine was 
DH> set up exactly as you said, with only  "Application" 
containing the 
DH> path, authcode and persistent.  Today after  hearing no 
suggestions 
DH> from the list, and based on recent list messages 
mentioning the home 
DH> directory for the service, I looked at the srvany.exe 
doco  to find 
DH> out how to give it a home directory.
DH> That's when I added  AppDirectory.  I also saw and added 
DH> AppParameters at the same time and  added those as well, 
though they 
DH> seem not to be needed.
DH>  
DH> Prior to adding the AppDirectory value, I never got any 
.stat file 
DH> or any .SVR file in my sniffer dir.  After adding that value and  
DH> starting the service those files appeared.
DH>  
DH>  


DH> From: [EMAIL PROTECTED]
DH> [mailto:[EMAIL PROTECTED]] On  Behalf Of Matt
DH> Sent: Tuesday, August 02, 2005 3:24  PM
DH> To: sniffer@SortMonster.com
DH> Subject: Re: [sniffer]  Sniffer taking a long time?


  

DH> Dan,

DH> There is no AppDirectory value on my server    either.  The
DH> Parameters key has only one value under it besides Default   
DH> which is "Application", and it contains exactly what I provided
DH> below.     Could it be that you tried to hard to get everything
DH> right by tweaking these    additional keys?

DH> Something else.  Did you make sure that the    Sniffer
DH> service that you created was started?  No doubt it will work if   
DH> you follow those directions to a T, and there aren't any issues
DH> with your    server apart from this.

DH> Matt



DH> Dan Horne wrote: 
  


DH>   I removed the AppParameters value and put the authcode     
DH> and persistent back in the Application value where it was before. 
DH> It      didn't make any difference at all in the processing time,
DH> still right around      2 seconds.  I don't know how your setup is
DH> working without at least the      AppDirectory value, because mine
DH> didn't start working until I put that in,      but if it is, I
DH> can't argue.  My server load isn't anywhere near yours,      so I
DH> don't see what the problem could be with mine.  Oh well, unless   
DH> Pete responds with a suggestion, I guess I'll just keep using the 
DH> non-service version.
DH>    
DH>   Thanks anyway.

    
  

DH>   From:[EMAIL PROTECTED]
DH> [mailto:[EMAIL PROTECTED]] On Behalf Of Matt
DH> Sent: Tuesday, August 02, 2005 2:37        PM
DH> To:sniffer@SortMonster.com
DH> Subject: Re: [sniffer] Sniffer taking a long        time?


DH> Dan,

DH> I seem to recall trying to use the        AppParameters key
DH> and having difficulty with it.  I think that you        might want
DH> to try removing that key and putting everything in the       
DH> Parameters key, or at least that works for me.  If you change     
DH> 
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Sniffer\Parameters
DH> in        RegEdit to the following it might fix the issue that you
DH> are having:
  
DH> C:\IMail\Declude\Sniffer\***RULEBASE-NAME***.exe ***AUTH-CODE*** 
DH> persistent

DH> You should of course adjust the path        and service 
name as well.

DH> The directions that I provided are        working perfectly
DH> on my server so far as I can tell.  I'm running        dual 3.2
DH> Ghz 1 MB cache Xeons with 5 x 15,000 RPM drives in RAID 5.        
DH> The following three debug log entries shows between 300 ms and 550
DH> ms per        message:
  
DH> 08/02/2005 14:19:47.113 QB93D976201222A43 [2616]         
DH> SNIFFER-IP: External program started:         
DH> C:\IMail\Declude\Sniffer\executable.exe auth-code 
DH> F:\\DB93D976201222A43.SMD
DH> 08/02/2005 14:19:47.676          QB93D976201222A43 [2616]
DH> SNIFFER-IP: External program reports exit code          of 61
DH> -----------------
DH> 08/02/2005          14:19:47.488 QB9418A4800EC2A49 [6196]
DH> SNIFFER-IP: External program          started:         
DH> C:\IMail\Declude\Sniffer\executable.exe auth-code 
DH> F:\\DB9418A4800EC2A49.SMD
DH> 08/02/2005 14:19:47.770          QB9418A4800EC2A49 [6196]
DH> SNIFFER-IP: External program reports exit code          of 51
DH> -----------------
DH> 08/02/2005          14:19:49.879 QB943711501382A4D [6388]
DH> SNIFFER-IP: External program          started:         
DH> C:\IMail\Declude\Sniffer\executable.exe auth-code 
DH> F:\\DB943711501382A4D.SMD
DH> 08/02/2005 14:19:50.176          QB943711501382A4D [6388]
DH> SNIFFER-IP: External program reports exit code          of 59

DH> My stat file shows the following:
  
DH> TicToc: 1122992104
DH> Loop: 154
DH> Poll: 0
DH> Jobs:          118392
DH> Secs: 155137
DH> Msg/Min: 45.7887
DH> Current-Load:          24.4275   
DH> Average-Load: 23.8719

DH> I'm not sure why people use FireDaemon for        this.  My
DH> experience with SRVANY.exe has been absolutely flawless       
DH> since I integrated this, and it has worked on both Win2k and
DH> Windows        2003.

DH> Matt





DH> Dan Horne wrote: 
DH> OK, I have managed to get SOMETHING working, but it still 
seems too 
DH> slow and something is still not right.  I originally set up the 
DH> persistent sniffer using the instructions from this 
DH> 
post:http://www.mail-archive.com/sniffer@sortmonster.com/msg00169.ht
DH> mlThis uses SRVANY.exe.  I conjectured that possibly the service 
DH> needed a home directory, so I added an AppDirectory value to the 
DH> sniffer service's "Parameters" key in the registry.  This 
value is 
DH> set to the directory sniffer resides in.  I also (based on my 
DH> reading of the srvany.exe documentation) added another 
value to the 
DH> same key called AppParameters.  This is set to my auth 
code followed 
DH> by a space, followed by the word persistent.

DH> Now when I start the service, the time spent processing a single 
DH> message goes down to something around 2 seconds, but is still far 
DH> longer than the non-service version.  I also still had no 
.stat file 
DH> in my sniffer directory.  I did get a *.SVR file, which I 
never got before.

DH> So then I'm thinking, let's just make sure that I have the latest 
DH> version of sniffer.  I downloaded that, did the necessary 
renaming 
DH> of the files and then started the service.  NOW there is a 
DH> *.persistent.stat file.  However, the scan time is still 
at around 2 
DH> seconds.

DH> Average Scan times (based on average scan times of 5 emails each):
DH> Without sniffer service running: .033 seconds With 
sniffer service 
DH> running: 2.244 seconds

DH> The *.persistent.stat file has the following contents:

DH>       TicToc: 1122990610
DH>         Loop: 512
DH>         Poll: 445
DH>         Jobs: 34
DH>         Secs: 303
DH>      Msg/Min: 6.73267
DH> Current-Load: 8.69565   
DH> Average-Load: 10.6371

DH> Any suggestions? 

DH> Thanks,
DH> Dan Horne

DH> This E-Mail came from the Message Sniffer mailing list. For 
DH> information and (un)subscription instructions go to 
DH> http://www.sortmonster.com/MessageSniffer/Help/Help.html


DH> --
DH> =====================================================
DH> MailPure custom filters for Declude JunkMail 
DH> 
Pro.http://www.mailpure.com/software/===============================
DH> ======================




This E-Mail came from the Message Sniffer mailing list. For 
information and (un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

    

This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html


  

-- 
=====================================================
MailPure custom filters for Declude JunkMail Pro.
http://www.mailpure.com/software/
=====================================================

Reply via email to