RE: [sniffer] Downloads are slow.
Agreed, my last report showed pretty slow times. All today were slower now that I look at them. I normally see up to 1.3M with overall times around 800-900K. John C 0K .. .. .. .. .. 36.79 KB/s 50K .. .. .. .. .. 11.51 KB/s 100K .. .. .. .. .. 19.76 KB/s 150K .. .. .. .. .. 11.98 KB/s 200K .. .. .. .. .. 37.20 KB/s 250K .. .. .. .. .. 10.60 KB/s 300K .. .. .. .. .. 16.00 KB/s 350K .. .. .. .. .. 19.05 KB/s 400K .. .. .. .. .. 22.22 KB/s 450K .. .. .. .. .. 10.32 KB/s 500K .. .. .. .. .. 13.50 KB/s 550K .. .. .. .. ..2.74 KB/s 600K .. .. .. .. ..8.40 KB/s 650K .. .. .. .. ..6.00 KB/s 700K .. .. .. .. ..9.97 KB/s 750K .. .. .. .. ..6.07 KB/s 800K .. .. .. .. ..5.89 KB/s 850K .. .. .. .. ..9.20 KB/s 900K .. .. .. .. ..6.46 KB/s 950K .. .. .. .. ..4.94 KB/s 1000K .. .. .. .. ..7.67 KB/s 1050K .. .. .. .. ..9.97 KB/s 1100K .. .. .. .. .. 13.28 KB/s 1150K .. .. .. .. .. 24.61 KB/s 1200K .. .. .. .. .. 12.36 KB/s 1250K .. .. .. .. .. 31.06 KB/s 1300K .. .. .. .. ..4.87 KB/s 1350K .. .. .. .. .. 34.77 KB/s 1400K .. .. .. .. .. 14.29 KB/s 1450K .. . .. .. .. 16.24 KB/s 1500K .. .. .. .. .. 33.33 KB/s 1550K .. . .. .. .. 21.48 KB/s 1600K .. .. .. .. .. 23.19 KB/s 1650K .. .. .. .. .. 27.34 KB/s 1700K .. .. .. .. .. 14.68 KB/s 1750K .. .. .. .. .. 47.76 KB/s 1800K .. .. .. .. .. 15.17 KB/s 1850K .. .. .. .. .. 16.17 KB/s 1900K .. .. .. .. .. 18.39 KB/s 1950K .. .. .. .. .. 74.40 KB/s 2000K .. .. .. .. .. 14.10 KB/s 2050K .. .. .. .. .. 12.70 KB/s 2100K .. .. .. .. .. 29.36 KB/s 2150K .. .. .. .. .. 16.58 KB/s 2200K .. .. .. .. .. 21.62 KB/s 2250K .. .. .. .. .. 17.49 KB/s 2300K .. .. .. .. .. 11.00 KB/s 2350K .. .. .. .. .. 21.20 KB/s 2400K .. .. .. .. .. 31.69 KB/s 2450K .. .. .. .. .. 20.12 KB/s 2500K .. .. .. .. .. 57.14 KB/s 2550K .. .. .. 13.94 KB/s 15:52:29 (12.45 KB/s) - `.new.gz' saved [2646653] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Tuesday, February 07, 2006 4:46 PM To: Chuck Schick Subject: Re: [sniffer] Downloads are slow. I'm not showing this from my location and the server looks ok. I just downloaded a few rulebases, each in under 3 seconds. Please provide a traceroute -- that should show us where the issue is (if it is still there). Thanks, _M On Tuesday, February 7, 2006, 4:39:35 PM, Chuck wrote: CS> Download speeds from your server are running 17 kbps at my location. CS> Chuck Schick CS> Warp 8, Inc. CS> (303)-421-5140 CS> www.warp8.com CS> This E-Mail came from the Message Sniffer mailing list. For CS> information and (un)subscription instructions go to CS> http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-
Re: [sniffer] Downloads are slow.
I'm not showing this from my location and the server looks ok. I just downloaded a few rulebases, each in under 3 seconds. Please provide a traceroute -- that should show us where the issue is (if it is still there). Thanks, _M On Tuesday, February 7, 2006, 4:39:35 PM, Chuck wrote: CS> Download speeds from your server are running 17 kbps at my location. CS> Chuck Schick CS> Warp 8, Inc. CS> (303)-421-5140 CS> www.warp8.com CS> This E-Mail came from the Message Sniffer mailing list. For CS> information and (un)subscription instructions go to CS> http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Downloads are slow...
It's actually compared to your running .snf file, not the .new file, since the file is renamed .new as part of the download with the "-O" switch. Bill -Original Message- From: Darrell ([EMAIL PROTECTED]) [mailto:[EMAIL PROTECTED] Sent: Tuesday, December 28, 2004 11:39 AM To: sniffer@SortMonster.com Subject: Re: [sniffer] Downloads are slow... Quick question if when you have a sucessful download if abcdef.new is renamed than what is wget comparing on the next run of the script? Darrell Jim Matuska writes: > So far it seems to be working, at least it doesn't seem to be > downloading > the rulebase yet, I'll have to see if it does later when there is an > updated rulebase. My script uses a copy at the end rather than a move. > It's listed below for reference. Do you see any issues? > > wget -N http://www.sortmonster.net/Sniffer/Updates/fp0o4jye.snf -O > abcdefg.new --http-user=* --http-passwd=* > if exist abcdefg.new goto Replace > goto Done > :Replace > rename abcdefg.new abcdefg.tst > snf2check.exe abcdefg.tst abcdefg > if errorlevel 1 goto Done > echo New File Tested GOOD! > if exist abcdefg.old del abcdefg.old > rename abcdefg.snf abcdefg.old > rename abcdefg.tst abcdefg.snf > copy /V /Y abcdefg.snf C:\sniffer\abcdefg.snf > :Done > if exist abcdefg.tst del abcdefg.tst > > > Jim Matuska Jr. > Computer Tech2, CCNA > Nez Perce Tribe > Information Systems > [EMAIL PROTECTED] > > > > - Original Message - From: "Pete McNeil" > <[EMAIL PROTECTED]> > To: "Jim Matuska" > Sent: Tuesday, December 28, 2004 11:12 AM > Subject: Re[2]: [sniffer] Downloads are slow... > > >> On Tuesday, December 28, 2004, 12:49:21 PM, Jim wrote: >> >> JM> I agree that something needs to be done about the update scripts >> JM> that >> are >> JM> inadvertently downloading the full rulebase all the time. I >> JM> didn't >> even >> JM> know it but we were doing this until I went through our update >> JM> script >> again >> JM> this morning and found it didn't have the -N option in Wget, so >> JM> we >> were >> >> Watch out - you may still have not fixed it. One of the tricks with >> the -N option is that the file downloaded previously must still be in >> it's place for the comparison. If it has been moved then the -N will >> not matter. >> >> This make things a little bit more complex since you can't download a >> rulebase file on top of the one that is running. >> >> _M >> >> >> >> >> This E-Mail came from the Message Sniffer mailing list. For >> information >> and (un)subscription instructions go to >> http://www.sortmonster.com/MessageSniffer/Help/Help.html >> > > > This E-Mail came from the Message Sniffer mailing list. For > information > and (un)subscription instructions go to > http://www.sortmonster.com/MessageSniffer/Help/Help.html Check out http://www.invariantsystems.com for utilities for Declude And Imail. IMail/Declude Overflow Queue Monitoring, MRTG Integration, and Log Parsers. This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html --- This message and any included attachments are from Siemens Medical Solutions USA, Inc. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by e-mail with a copy to [EMAIL PROTECTED] Thank you This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Downloads are slow...
Title: Message Good idea, Matt. I'll put together a draft and submit it to the list for review later today. I'll also include the open-source gzip file so that everything is contained in one package (instructions, download script, and gzip file). Bill -Original Message-From: Matt [mailto:[EMAIL PROTECTED] Sent: Tuesday, December 28, 2004 11:45 AMTo: sniffer@SortMonster.comSubject: Re: [sniffer] Downloads are slow...Yep. Despite the fact that one could design a process to work properly with the -N option (also leaving the old file for comparison), since this is generally scheduled by users on the hour, it would still produce a run on the bandwidth at the top of some or even every hour. Enforcing a time bracket is not realistic.Using the program alias is the best way all around for now, and I believe that this should be promoted as the only option for IMail users at least. It appears that Pete times his notifications so that it doesn't produce backups, and I assume that notifications are sent immediately upon publishing the new customized rulebases, so it is also the fastest method to achieving an update.The code is there, but I just think that it can be better variablized to adjust for different directories and codes, so it fits appropriately in everyone's config. Packing this together with gzip and including that in the default setup would also be seemingly preferable. Throwing together a how-to that was written for the lowest common denominator would enhance the ease of use for many (pictures are nice where appropriate), and would help with reducing support.MattWoody G Fussell wrote: Why do you not use a "program alias" and only download when you receive notification that a new rule base is available? If everyone used gzip and only downloaded when notified the bandwidth could be controlled by staggering the notifications. Woody Fussell Wilbur Smith Associates -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Jim Matuska Sent: Tuesday, December 28, 2004 12:49 PM To: sniffer@SortMonster.com Subject: Re: [sniffer] Downloads are slow... I agree that something needs to be done about the update scripts that are inadvertently downloading the full rulebase all the time. I didn't even know it but we were doing this until I went through our update script again this morning and found it didn't have the -N option in Wget, so we were downloading the entire rulebase whether we needed it or not. The gzip compression is cool, and I will likely implement it soon, but I think the major problem is everyone that is using scripts that keep downloading the same file over and over again tying up the bandwidth. I would recommend 2 things to help alleviate this problem: 1. Monitor connections to rulebase downloads to see who is downloading the rulebase everytime they connect on a schedule to determine who has their scripts setup wrong, and contact them to correct it. It took me under a minute to add the -N option to wget, it should be a no brainer. 2. Correct the scripts posted on the Sniffer website to include date checking, and possibly gzip compression, I used one of those scripts for our system and assumed it would be setup correctly, but it was not. Jim Matuska Jr. Computer Tech2, CCNA Nez Perce Tribe Information Systems [EMAIL PROTECTED] - Original Message - From: "Matt" <[EMAIL PROTECTED]> To: Sent: Monday, December 27, 2004 10:03 PM Subject: Re: [sniffer] Downloads are slow... I agree entirely. If bandwidth has become an issue, it would be resolved with a focus on producing very tight and easily customizable scripts (a variables section in the top of the scripts). I believe that going the VBScript route might be the best way to go, or at least I believe that more of us can hack a more involved VBScript than a batch or CMD file. Enforcing compressed downloads and checking for timestamps prior to downloading should be done in these scripts as well. Right now the script examples assume a familiarity with scripting, and while local participants can mostly handle that stuff, the non-vocal ones are most likely to not even be aware of the issues or how to fix them, and might have scripted timed downloads because it is definitely the easiest way to go. This is probably the majority of the customer base. There is an impression for instance with Declude's user base that +80% use primarily the default config which most of us know is severely lacking in comparison to the potential that exists by tweaking the settings. With better script examples and a careful step-by-step readme promoted in a mailing to your customers, I believe that this issue could go away, or at least theoretically it should. Personally, I have mine tied to the E-mails, I download the
Re: [sniffer] Downloads are slow...
As far as I understand it wget is comparing the date stamp on the file in the local directory to the date stamp on the file at sortmonster.net, if its not don't download the file, if it is do download it. Jim Matuska Jr. Computer Tech2, CCNA Nez Perce Tribe Information Systems [EMAIL PROTECTED] - Original Message - From: "Darrell ([EMAIL PROTECTED])" <[EMAIL PROTECTED]> To: Sent: Tuesday, December 28, 2004 11:38 AM Subject: Re: [sniffer] Downloads are slow... Quick question if when you have a sucessful download if abcdef.new is renamed than what is wget comparing on the next run of the script? Darrell Jim Matuska writes: So far it seems to be working, at least it doesn't seem to be downloading the rulebase yet, I'll have to see if it does later when there is an updated rulebase. My script uses a copy at the end rather than a move. It's listed below for reference. Do you see any issues? wget -N http://www.sortmonster.net/Sniffer/Updates/fp0o4jye.snf -O abcdefg.new --http-user=* --http-passwd=* if exist abcdefg.new goto Replace goto Done :Replace rename abcdefg.new abcdefg.tst snf2check.exe abcdefg.tst abcdefg if errorlevel 1 goto Done echo New File Tested GOOD! if exist abcdefg.old del abcdefg.old rename abcdefg.snf abcdefg.old rename abcdefg.tst abcdefg.snf copy /V /Y abcdefg.snf C:\sniffer\abcdefg.snf :Done if exist abcdefg.tst del abcdefg.tst Jim Matuska Jr. Computer Tech2, CCNA Nez Perce Tribe Information Systems [EMAIL PROTECTED] - Original Message - From: "Pete McNeil" <[EMAIL PROTECTED]> To: "Jim Matuska" Sent: Tuesday, December 28, 2004 11:12 AM Subject: Re[2]: [sniffer] Downloads are slow... On Tuesday, December 28, 2004, 12:49:21 PM, Jim wrote: JM> I agree that something needs to be done about the update scripts that are JM> inadvertently downloading the full rulebase all the time. I didn't even JM> know it but we were doing this until I went through our update script again JM> this morning and found it didn't have the -N option in Wget, so we were Watch out - you may still have not fixed it. One of the tricks with the -N option is that the file downloaded previously must still be in it's place for the comparison. If it has been moved then the -N will not matter. This make things a little bit more complex since you can't download a rulebase file on top of the one that is running. _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html Check out http://www.invariantsystems.com for utilities for Declude And Imail. IMail/Declude Overflow Queue Monitoring, MRTG Integration, and Log Parsers. This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Downloads are slow...
While the "output" file is named ".new", it IS comparing the file named in the URL, in his case fp0o4jye.snf against a file with the same name in the current directory. The "output" (-o) option only comes into play IF an download is actually occurring (after the timestamp condition). With CURL things are a bit more flexible - you can specify WHICH file is used for comparison. Best Regards Andy Schmidt H&M Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Darrell ([EMAIL PROTECTED]) Sent: Tuesday, December 28, 2004 02:39 PM To: sniffer@SortMonster.com Subject: Re: [sniffer] Downloads are slow... Quick question if when you have a sucessful download if abcdef.new is renamed than what is wget comparing on the next run of the script? Darrell Jim Matuska writes: > So far it seems to be working, at least it doesn't seem to be > downloading > the rulebase yet, I'll have to see if it does later when there is an > updated rulebase. My script uses a copy at the end rather than a move. > It's listed below for reference. Do you see any issues? > > wget -N http://www.sortmonster.net/Sniffer/Updates/fp0o4jye.snf -O > abcdefg.new --http-user=* --http-passwd=* > if exist abcdefg.new goto Replace > goto Done > :Replace > rename abcdefg.new abcdefg.tst > snf2check.exe abcdefg.tst abcdefg > if errorlevel 1 goto Done > echo New File Tested GOOD! > if exist abcdefg.old del abcdefg.old > rename abcdefg.snf abcdefg.old > rename abcdefg.tst abcdefg.snf > copy /V /Y abcdefg.snf C:\sniffer\abcdefg.snf > :Done > if exist abcdefg.tst del abcdefg.tst > > > Jim Matuska Jr. > Computer Tech2, CCNA > Nez Perce Tribe > Information Systems > [EMAIL PROTECTED] > > > > - Original Message - From: "Pete McNeil" > <[EMAIL PROTECTED]> > To: "Jim Matuska" > Sent: Tuesday, December 28, 2004 11:12 AM > Subject: Re[2]: [sniffer] Downloads are slow... > > >> On Tuesday, December 28, 2004, 12:49:21 PM, Jim wrote: >> >> JM> I agree that something needs to be done about the update scripts >> JM> that >> are >> JM> inadvertently downloading the full rulebase all the time. I >> JM> didn't >> even >> JM> know it but we were doing this until I went through our update >> JM> script >> again >> JM> this morning and found it didn't have the -N option in Wget, so >> JM> we >> were >> >> Watch out - you may still have not fixed it. One of the tricks with >> the -N option is that the file downloaded previously must still be in >> it's place for the comparison. If it has been moved then the -N will >> not matter. >> >> This make things a little bit more complex since you can't download a >> rulebase file on top of the one that is running. >> >> _M >> >> >> >> >> This E-Mail came from the Message Sniffer mailing list. For >> information >> and (un)subscription instructions go to >> http://www.sortmonster.com/MessageSniffer/Help/Help.html >> > > > This E-Mail came from the Message Sniffer mailing list. For > information > and (un)subscription instructions go to > http://www.sortmonster.com/MessageSniffer/Help/Help.html Check out http://www.invariantsystems.com for utilities for Declude And Imail. IMail/Declude Overflow Queue Monitoring, MRTG Integration, and Log Parsers. This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Downloads are slow...
Yep. Despite the fact that one could design a process to work properly with the -N option (also leaving the old file for comparison), since this is generally scheduled by users on the hour, it would still produce a run on the bandwidth at the top of some or even every hour. Enforcing a time bracket is not realistic. Using the program alias is the best way all around for now, and I believe that this should be promoted as the only option for IMail users at least. It appears that Pete times his notifications so that it doesn't produce backups, and I assume that notifications are sent immediately upon publishing the new customized rulebases, so it is also the fastest method to achieving an update. The code is there, but I just think that it can be better variablized to adjust for different directories and codes, so it fits appropriately in everyone's config. Packing this together with gzip and including that in the default setup would also be seemingly preferable. Throwing together a how-to that was written for the lowest common denominator would enhance the ease of use for many (pictures are nice where appropriate), and would help with reducing support. Matt Woody G Fussell wrote: Why do you not use a "program alias" and only download when you receive notification that a new rule base is available? If everyone used gzip and only downloaded when notified the bandwidth could be controlled by staggering the notifications. Woody Fussell Wilbur Smith Associates -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Jim Matuska Sent: Tuesday, December 28, 2004 12:49 PM To: sniffer@SortMonster.com Subject: Re: [sniffer] Downloads are slow... I agree that something needs to be done about the update scripts that are inadvertently downloading the full rulebase all the time. I didn't even know it but we were doing this until I went through our update script again this morning and found it didn't have the -N option in Wget, so we were downloading the entire rulebase whether we needed it or not. The gzip compression is cool, and I will likely implement it soon, but I think the major problem is everyone that is using scripts that keep downloading the same file over and over again tying up the bandwidth. I would recommend 2 things to help alleviate this problem: 1. Monitor connections to rulebase downloads to see who is downloading the rulebase everytime they connect on a schedule to determine who has their scripts setup wrong, and contact them to correct it. It took me under a minute to add the -N option to wget, it should be a no brainer. 2. Correct the scripts posted on the Sniffer website to include date checking, and possibly gzip compression, I used one of those scripts for our system and assumed it would be setup correctly, but it was not. Jim Matuska Jr. Computer Tech2, CCNA Nez Perce Tribe Information Systems [EMAIL PROTECTED] - Original Message - From: "Matt" <[EMAIL PROTECTED]> To: Sent: Monday, December 27, 2004 10:03 PM Subject: Re: [sniffer] Downloads are slow... I agree entirely. If bandwidth has become an issue, it would be resolved with a focus on producing very tight and easily customizable scripts (a variables section in the top of the scripts). I believe that going the VBScript route might be the best way to go, or at least I believe that more of us can hack a more involved VBScript than a batch or CMD file. Enforcing compressed downloads and checking for timestamps prior to downloading should be done in these scripts as well. Right now the script examples assume a familiarity with scripting, and while local participants can mostly handle that stuff, the non-vocal ones are most likely to not even be aware of the issues or how to fix them, and might have scripted timed downloads because it is definitely the easiest way to go. This is probably the majority of the customer base. There is an impression for instance with Declude's user base that +80% use primarily the default config which most of us know is severely lacking in comparison to the potential that exists by tweaking the settings. With better script examples and a careful step-by-step readme promoted in a mailing to your customers, I believe that this issue could go away, or at least theoretically it should. Personally, I have mine tied to the E-mails, I download the zipped versions, I don't bother checking on the status, and have never noticed any issues as a result. It would be a small shame if I was missing downloads due to timeouts, but not that big of a deal if this has never caused a noticeable problem. Matt Andy Schmidt wrote: Pete, With all due respect - I think the download problem is "self-inflicted", because your web site is providing unsuitable examples to your customers! Even wi
Re: [sniffer] Downloads are slow...
Quick question if when you have a sucessful download if abcdef.new is renamed than what is wget comparing on the next run of the script? Darrell Jim Matuska writes: So far it seems to be working, at least it doesn't seem to be downloading the rulebase yet, I'll have to see if it does later when there is an updated rulebase. My script uses a copy at the end rather than a move. It's listed below for reference. Do you see any issues? wget -N http://www.sortmonster.net/Sniffer/Updates/fp0o4jye.snf -O abcdefg.new --http-user=* --http-passwd=* if exist abcdefg.new goto Replace goto Done :Replace rename abcdefg.new abcdefg.tst snf2check.exe abcdefg.tst abcdefg if errorlevel 1 goto Done echo New File Tested GOOD! if exist abcdefg.old del abcdefg.old rename abcdefg.snf abcdefg.old rename abcdefg.tst abcdefg.snf copy /V /Y abcdefg.snf C:\sniffer\abcdefg.snf :Done if exist abcdefg.tst del abcdefg.tst Jim Matuska Jr. Computer Tech2, CCNA Nez Perce Tribe Information Systems [EMAIL PROTECTED] - Original Message - From: "Pete McNeil" <[EMAIL PROTECTED]> To: "Jim Matuska" Sent: Tuesday, December 28, 2004 11:12 AM Subject: Re[2]: [sniffer] Downloads are slow... On Tuesday, December 28, 2004, 12:49:21 PM, Jim wrote: JM> I agree that something needs to be done about the update scripts that are JM> inadvertently downloading the full rulebase all the time. I didn't even JM> know it but we were doing this until I went through our update script again JM> this morning and found it didn't have the -N option in Wget, so we were Watch out - you may still have not fixed it. One of the tricks with the -N option is that the file downloaded previously must still be in it's place for the comparison. If it has been moved then the -N will not matter. This make things a little bit more complex since you can't download a rulebase file on top of the one that is running. _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html Check out http://www.invariantsystems.com for utilities for Declude And Imail. IMail/Declude Overflow Queue Monitoring, MRTG Integration, and Log Parsers. This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Downloads are slow...
Brian Gregory wrote: Does anyone have some simple instructions on how to setup to use a program alias? I'm using Bill Landry's script, have not had the time to look in to using a program alias to download upon receipt of notifications. I just setup a program alias about 2 weeks ago using this as the script http://www.sortmonster.com/MessageSniffer/Help/UserScripts/david_snifferUpdateMethod.zip that IMail calls, and this as the actual updater http://www.sortmonster.com/MessageSniffer/Help/UserScripts/Bill_Landry_snfupd.zip I did have to do some customizations, and also I had to have all the files in the sniffer directory, but it wasn't too bad. I'd be glad to help you with any specific questions, either on or off list... -Russ --- [This E-mail scanned for viruses by Declude Virus] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Downloads are slow...
Does anyone have some simple instructions on how to setup to use a program alias? I'm using Bill Landry's script, have not had the time to look in to using a program alias to download upon receipt of notifications. Thanks, Brian Gregory President Network Innovations Inc. (913) 780-0494 x104 [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Woody G Fussell Sent: Tuesday, December 28, 2004 12:25 PM To: sniffer@SortMonster.com Subject: RE: [sniffer] Downloads are slow... Why do you not use a "program alias" and only download when you receive notification that a new rule base is available? If everyone used gzip and only downloaded when notified the bandwidth could be controlled by staggering the notifications. Woody Fussell Wilbur Smith Associates -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jim Matuska Sent: Tuesday, December 28, 2004 12:49 PM To: sniffer@SortMonster.com Subject: Re: [sniffer] Downloads are slow... I agree that something needs to be done about the update scripts that are inadvertently downloading the full rulebase all the time. I didn't even know it but we were doing this until I went through our update script again this morning and found it didn't have the -N option in Wget, so we were downloading the entire rulebase whether we needed it or not. The gzip compression is cool, and I will likely implement it soon, but I think the major problem is everyone that is using scripts that keep downloading the same file over and over again tying up the bandwidth. I would recommend 2 things to help alleviate this problem: 1. Monitor connections to rulebase downloads to see who is downloading the rulebase everytime they connect on a schedule to determine who has their scripts setup wrong, and contact them to correct it. It took me under a minute to add the -N option to wget, it should be a no brainer. 2. Correct the scripts posted on the Sniffer website to include date checking, and possibly gzip compression, I used one of those scripts for our system and assumed it would be setup correctly, but it was not. Jim Matuska Jr. Computer Tech2, CCNA Nez Perce Tribe Information Systems [EMAIL PROTECTED] - Original Message - From: "Matt" <[EMAIL PROTECTED]> To: Sent: Monday, December 27, 2004 10:03 PM Subject: Re: [sniffer] Downloads are slow... >I agree entirely. If bandwidth has become an issue, it would be resolved >with a focus on producing very tight and easily customizable scripts (a >variables section in the top of the scripts). I believe that going the >VBScript route might be the best way to go, or at least I believe that more >of us can hack a more involved VBScript than a batch or CMD file. >Enforcing compressed downloads and checking for timestamps prior to >downloading should be done in these scripts as well. > > Right now the script examples assume a familiarity with scripting, and > while local participants can mostly handle that stuff, the non-vocal ones > are most likely to not even be aware of the issues or how to fix them, and > might have scripted timed downloads because it is definitely the easiest > way to go. This is probably the majority of the customer base. There is > an impression for instance with Declude's user base that +80% use > primarily the default config which most of us know is severely lacking in > comparison to the potential that exists by tweaking the settings. > > With better script examples and a careful step-by-step readme promoted in > a mailing to your customers, I believe that this issue could go away, or > at least theoretically it should. > > Personally, I have mine tied to the E-mails, I download the zipped > versions, I don't bother checking on the status, and have never noticed > any issues as a result. It would be a small shame if I was missing > downloads due to timeouts, but not that big of a deal if this has never > caused a noticeable problem. > > Matt > > > > > Andy Schmidt wrote: > >>Pete, >> >>With all due respect - I think the download problem is "self-inflicted", >>because your web site is providing unsuitable examples to your customers! >>Even with moderate bandwidth, your server would be able to handle tens of >>thousands of hits a day. Checking if an updated file exists should barely >>be noticeable - as long as it doesn't result in an unnecessary download. >>You probably suffer TWO problems: >> >>A) Most of your customers are downloading rules based on a schedule, even >>if >>no rules exists. Potential savings: 100% per download attempt. >> >>B) Your customers are not downloading "compressed" rule files. Potent
RE: [sniffer] Downloads are slow...
Why do you not use a "program alias" and only download when you receive notification that a new rule base is available? If everyone used gzip and only downloaded when notified the bandwidth could be controlled by staggering the notifications. Woody Fussell Wilbur Smith Associates -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jim Matuska Sent: Tuesday, December 28, 2004 12:49 PM To: sniffer@SortMonster.com Subject: Re: [sniffer] Downloads are slow... I agree that something needs to be done about the update scripts that are inadvertently downloading the full rulebase all the time. I didn't even know it but we were doing this until I went through our update script again this morning and found it didn't have the -N option in Wget, so we were downloading the entire rulebase whether we needed it or not. The gzip compression is cool, and I will likely implement it soon, but I think the major problem is everyone that is using scripts that keep downloading the same file over and over again tying up the bandwidth. I would recommend 2 things to help alleviate this problem: 1. Monitor connections to rulebase downloads to see who is downloading the rulebase everytime they connect on a schedule to determine who has their scripts setup wrong, and contact them to correct it. It took me under a minute to add the -N option to wget, it should be a no brainer. 2. Correct the scripts posted on the Sniffer website to include date checking, and possibly gzip compression, I used one of those scripts for our system and assumed it would be setup correctly, but it was not. Jim Matuska Jr. Computer Tech2, CCNA Nez Perce Tribe Information Systems [EMAIL PROTECTED] - Original Message - From: "Matt" <[EMAIL PROTECTED]> To: Sent: Monday, December 27, 2004 10:03 PM Subject: Re: [sniffer] Downloads are slow... >I agree entirely. If bandwidth has become an issue, it would be resolved >with a focus on producing very tight and easily customizable scripts (a >variables section in the top of the scripts). I believe that going the >VBScript route might be the best way to go, or at least I believe that more >of us can hack a more involved VBScript than a batch or CMD file. >Enforcing compressed downloads and checking for timestamps prior to >downloading should be done in these scripts as well. > > Right now the script examples assume a familiarity with scripting, and > while local participants can mostly handle that stuff, the non-vocal ones > are most likely to not even be aware of the issues or how to fix them, and > might have scripted timed downloads because it is definitely the easiest > way to go. This is probably the majority of the customer base. There is > an impression for instance with Declude's user base that +80% use > primarily the default config which most of us know is severely lacking in > comparison to the potential that exists by tweaking the settings. > > With better script examples and a careful step-by-step readme promoted in > a mailing to your customers, I believe that this issue could go away, or > at least theoretically it should. > > Personally, I have mine tied to the E-mails, I download the zipped > versions, I don't bother checking on the status, and have never noticed > any issues as a result. It would be a small shame if I was missing > downloads due to timeouts, but not that big of a deal if this has never > caused a noticeable problem. > > Matt > > > > > Andy Schmidt wrote: > >>Pete, >> >>With all due respect - I think the download problem is "self-inflicted", >>because your web site is providing unsuitable examples to your customers! >>Even with moderate bandwidth, your server would be able to handle tens of >>thousands of hits a day. Checking if an updated file exists should barely >>be noticeable - as long as it doesn't result in an unnecessary download. >>You probably suffer TWO problems: >> >>A) Most of your customers are downloading rules based on a schedule, even >>if >>no rules exists. Potential savings: 100% per download attempt. >> >>B) Your customers are not downloading "compressed" rule files. Potential >>savings: about 66%, but that's not bad either. >> >> >>One likely explanation is that at least THREE of your sample scripts do an >>unconditional and uncompressed download! Here the 3 URLs you list on your >>web site and WGET command they are using: >> >>http://www.sortmonster.com/MessageSniffer/Help/UserScripts/david_snifferUp da >>teMethod.zip >>wget http://www.sortmonster.net/Sniffer/Updates/.snf -O >>.new >>--http-user=username --http-pass
Re: [sniffer] Downloads are slow...
I agree that something needs to be done about the update scripts that are inadvertently downloading the full rulebase all the time. I didn't even know it but we were doing this until I went through our update script again this morning and found it didn't have the -N option in Wget, so we were downloading the entire rulebase whether we needed it or not. The gzip compression is cool, and I will likely implement it soon, but I think the major problem is everyone that is using scripts that keep downloading the same file over and over again tying up the bandwidth. I would recommend 2 things to help alleviate this problem: 1. Monitor connections to rulebase downloads to see who is downloading the rulebase everytime they connect on a schedule to determine who has their scripts setup wrong, and contact them to correct it. It took me under a minute to add the -N option to wget, it should be a no brainer. 2. Correct the scripts posted on the Sniffer website to include date checking, and possibly gzip compression, I used one of those scripts for our system and assumed it would be setup correctly, but it was not. Jim Matuska Jr. Computer Tech2, CCNA Nez Perce Tribe Information Systems [EMAIL PROTECTED] - Original Message - From: "Matt" <[EMAIL PROTECTED]> To: Sent: Monday, December 27, 2004 10:03 PM Subject: Re: [sniffer] Downloads are slow... I agree entirely. If bandwidth has become an issue, it would be resolved with a focus on producing very tight and easily customizable scripts (a variables section in the top of the scripts). I believe that going the VBScript route might be the best way to go, or at least I believe that more of us can hack a more involved VBScript than a batch or CMD file. Enforcing compressed downloads and checking for timestamps prior to downloading should be done in these scripts as well. Right now the script examples assume a familiarity with scripting, and while local participants can mostly handle that stuff, the non-vocal ones are most likely to not even be aware of the issues or how to fix them, and might have scripted timed downloads because it is definitely the easiest way to go. This is probably the majority of the customer base. There is an impression for instance with Declude's user base that +80% use primarily the default config which most of us know is severely lacking in comparison to the potential that exists by tweaking the settings. With better script examples and a careful step-by-step readme promoted in a mailing to your customers, I believe that this issue could go away, or at least theoretically it should. Personally, I have mine tied to the E-mails, I download the zipped versions, I don't bother checking on the status, and have never noticed any issues as a result. It would be a small shame if I was missing downloads due to timeouts, but not that big of a deal if this has never caused a noticeable problem. Matt Andy Schmidt wrote: Pete, With all due respect - I think the download problem is "self-inflicted", because your web site is providing unsuitable examples to your customers! Even with moderate bandwidth, your server would be able to handle tens of thousands of hits a day. Checking if an updated file exists should barely be noticeable - as long as it doesn't result in an unnecessary download. You probably suffer TWO problems: A) Most of your customers are downloading rules based on a schedule, even if no rules exists. Potential savings: 100% per download attempt. B) Your customers are not downloading "compressed" rule files. Potential savings: about 66%, but that's not bad either. One likely explanation is that at least THREE of your sample scripts do an unconditional and uncompressed download! Here the 3 URLs you list on your web site and WGET command they are using: http://www.sortmonster.com/MessageSniffer/Help/UserScripts/david_snifferUpda teMethod.zip wget http://www.sortmonster.net/Sniffer/Updates/.snf -O .new --http-user=username --http-passwd=password http://www.sortmonster.com/MessageSniffer/Help/UserScripts/Hank_SnifferScrip ts.zip wget http://www.sortmonster.net/Sniffer/Updates/.snf -O .new --http-user=sniffer --http-passwd=ki11sp8m http://www.sortmonster.com/MessageSniffer/Help/UserScripts/Michiel_AutoUpdat e.zip wget http://sniffer:[EMAIL PROTECTED]/Sniffer/Updates/12345678.snf -O .tst My recommendation: Replace these with examples that implement conditional, compressed downloading. Best Regards Andy Schmidt H&M Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Monday, December 27, 2004 08:10 AM To: Chuck Schick Subject: Re: [sniffer] Downloads
Re: [sniffer] Downloads are slow...
I agree entirely. If bandwidth has become an issue, it would be resolved with a focus on producing very tight and easily customizable scripts (a variables section in the top of the scripts). I believe that going the VBScript route might be the best way to go, or at least I believe that more of us can hack a more involved VBScript than a batch or CMD file. Enforcing compressed downloads and checking for timestamps prior to downloading should be done in these scripts as well. Right now the script examples assume a familiarity with scripting, and while local participants can mostly handle that stuff, the non-vocal ones are most likely to not even be aware of the issues or how to fix them, and might have scripted timed downloads because it is definitely the easiest way to go. This is probably the majority of the customer base. There is an impression for instance with Declude's user base that +80% use primarily the default config which most of us know is severely lacking in comparison to the potential that exists by tweaking the settings. With better script examples and a careful step-by-step readme promoted in a mailing to your customers, I believe that this issue could go away, or at least theoretically it should. Personally, I have mine tied to the E-mails, I download the zipped versions, I don't bother checking on the status, and have never noticed any issues as a result. It would be a small shame if I was missing downloads due to timeouts, but not that big of a deal if this has never caused a noticeable problem. Matt Andy Schmidt wrote: Pete, With all due respect - I think the download problem is "self-inflicted", because your web site is providing unsuitable examples to your customers! Even with moderate bandwidth, your server would be able to handle tens of thousands of hits a day. Checking if an updated file exists should barely be noticeable - as long as it doesn't result in an unnecessary download. You probably suffer TWO problems: A) Most of your customers are downloading rules based on a schedule, even if no rules exists. Potential savings: 100% per download attempt. B) Your customers are not downloading "compressed" rule files. Potential savings: about 66%, but that's not bad either. One likely explanation is that at least THREE of your sample scripts do an unconditional and uncompressed download! Here the 3 URLs you list on your web site and WGET command they are using: http://www.sortmonster.com/MessageSniffer/Help/UserScripts/david_snifferUpda teMethod.zip wget http://www.sortmonster.net/Sniffer/Updates/.snf -O .new --http-user=username --http-passwd=password http://www.sortmonster.com/MessageSniffer/Help/UserScripts/Hank_SnifferScrip ts.zip wget http://www.sortmonster.net/Sniffer/Updates/.snf -O .new --http-user=sniffer --http-passwd=ki11sp8m http://www.sortmonster.com/MessageSniffer/Help/UserScripts/Michiel_AutoUpdat e.zip wget http://sniffer:[EMAIL PROTECTED]/Sniffer/Updates/12345678.snf -O .tst My recommendation: Replace these with examples that implement conditional, compressed downloading. Best Regards Andy Schmidt H&M Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Monday, December 27, 2004 08:10 AM To: Chuck Schick Subject: Re: [sniffer] Downloads are slow... On Monday, December 27, 2004, 1:17:21 AM, Chuck wrote: CS> Pete: CS> It appears on weekends the sniffer downloads are really slow. I am CS> downloading at 14 minutes past the hour and I am about 1/20 th of CS> the normal speed. That is an unusual observation - I don't think weekends have anything to do with making things slower. I will look at the logs to see if I can figure out what heppened. You're not manually downloading I hope? _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Downloads are slow...
Pete, With all due respect - I think the download problem is "self-inflicted", because your web site is providing unsuitable examples to your customers! Even with moderate bandwidth, your server would be able to handle tens of thousands of hits a day. Checking if an updated file exists should barely be noticeable - as long as it doesn't result in an unnecessary download. You probably suffer TWO problems: A) Most of your customers are downloading rules based on a schedule, even if no rules exists. Potential savings: 100% per download attempt. B) Your customers are not downloading "compressed" rule files. Potential savings: about 66%, but that's not bad either. One likely explanation is that at least THREE of your sample scripts do an unconditional and uncompressed download! Here the 3 URLs you list on your web site and WGET command they are using: http://www.sortmonster.com/MessageSniffer/Help/UserScripts/david_snifferUpda teMethod.zip wget http://www.sortmonster.net/Sniffer/Updates/.snf -O .new --http-user=username --http-passwd=password http://www.sortmonster.com/MessageSniffer/Help/UserScripts/Hank_SnifferScrip ts.zip wget http://www.sortmonster.net/Sniffer/Updates/.snf -O .new --http-user=sniffer --http-passwd=ki11sp8m http://www.sortmonster.com/MessageSniffer/Help/UserScripts/Michiel_AutoUpdat e.zip wget http://sniffer:[EMAIL PROTECTED]/Sniffer/Updates/12345678.snf -O .tst My recommendation: Replace these with examples that implement conditional, compressed downloading. Best Regards Andy Schmidt H&M Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Monday, December 27, 2004 08:10 AM To: Chuck Schick Subject: Re: [sniffer] Downloads are slow... On Monday, December 27, 2004, 1:17:21 AM, Chuck wrote: CS> Pete: CS> It appears on weekends the sniffer downloads are really slow. I am CS> downloading at 14 minutes past the hour and I am about 1/20 th of CS> the normal speed. That is an unusual observation - I don't think weekends have anything to do with making things slower. I will look at the logs to see if I can figure out what heppened. You're not manually downloading I hope? _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Downloads are slow...
On Monday, December 27, 2004, 1:17:21 AM, Chuck wrote: CS> Pete: CS> It appears on weekends the sniffer downloads are really slow. I am CS> downloading at 14 minutes past the hour and I am about 1/20 th of the normal CS> speed. That is an unusual observation - I don't think weekends have anything to do with making things slower. I will look at the logs to see if I can figure out what heppened. You're not manually downloading I hope? _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
