Re: [sniffer] Downloads are slow...
On Monday, December 27, 2004, 1:17:21 AM, Chuck wrote: CS Pete: CS It appears on weekends the sniffer downloads are really slow. I am CS downloading at 14 minutes past the hour and I am about 1/20 th of the normal CS speed. That is an unusual observation - I don't think weekends have anything to do with making things slower. I will look at the logs to see if I can figure out what heppened. You're not manually downloading I hope? _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Excess spam over the weekend
Is anyone else seeing a huge flood of spam over the weekend? I have received a ton of it since Friday, a lot of it is not being picked up by sniffer either. Jim Matuska Jr.Computer Tech2, CCNANez Perce TribeInformation Systems[EMAIL PROTECTED]
Re: [sniffer] Excess spam over the weekend
On Monday, December 27, 2004, 10:24:00 AM, Jim wrote: JM Is anyone else seeing a huge flood of spam over the JM weekend? I have received a ton of it since Friday, a lot of it is JM not being picked up by sniffer either. I believe I can explain this phenomena. Over weekends and holidays folks stop submitting spam to us, so the only things we can tag are those that hit our spamtraps. I've notices quite a drop in the amount of spam we've been receiving - though the filter rate and volume at our servers is still very high (filtering 90+% of 35K/day - very little monitored spam getting through). I'm looking for some ways to increase the breadth of our spamtraps so that this cyclic drop can be mitigated... _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Excess spam over the weekend
Jim: We saw just the opposite. The amount of Spam appeared to be down over the holiday weekend. We saw less total volume and less spam in the spam traps. Chuck Schick Warp 8, Inc. (303)-421-5140 www.warp8.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jim Matuska Sent: Monday, December 27, 2004 8:24 AM To: sniffer@SortMonster.com Subject: [sniffer] Excess spam over the weekend Is anyone else seeing a huge flood of spam over the weekend? I have received a ton of it since Friday, a lot of it is not being picked up by sniffer either. Jim Matuska Jr. Computer Tech2, CCNA Nez Perce Tribe Information Systems [EMAIL PROTECTED] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Sniffer Updates
Our updates seem to be taking a very long time. I am 85% updated and the ETA shows 07:00. Is it me? Kevin This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Sniffer Updates
Kevin Stanford wrote: Our updates seem to be taking a very long time. I am 85% updated and the ETA shows 07:00. Is it me? I see stuff like this come and go... Our updates are (finally) triggered from the email notifications... Below is a snippet of the last update that shows exactly what speeds we saw, which ran at 10:45 EST this morning... Every once in a while, I will see it slow down to about 8KB/s, but rarely slower than that... Thanks, Russ (This will probably wrap and look real ugly, but the last number is the average download speed for that part of the download...) 0K .. .. .. .. .. 0% 110.38 KB/s 50K .. .. .. .. .. 1% 160.26 KB/s 100K .. .. .. .. .. 2% 71.12 KB/s 150K .. .. .. .. .. 3% 110.13 KB/s 200K .. .. .. .. .. 4% 118.76 KB/s 250K .. .. .. .. .. 5% 145.35 KB/s 300K .. .. .. .. .. 6% 168.35 KB/s 350K .. .. .. .. .. 7% 168.35 KB/s 400K .. .. .. .. .. 8% 168.35 KB/s 450K .. .. .. .. .. 9% 160.26 KB/s 500K .. .. .. .. .. 10% 159.74 KB/s 550K .. .. .. .. .. 11% 188.68 KB/s 600K .. .. .. .. .. 12% 177.30 KB/s 650K .. .. .. .. .. 13% 168.35 KB/s 700K .. .. .. .. .. 14% 177.94 KB/s 750K .. .. .. .. .. 15% 168.35 KB/s 800K .. .. .. .. .. 16% 177.94 KB/s 850K .. .. .. .. .. 17% 168.35 KB/s 900K .. .. .. .. .. 18% 168.35 KB/s 950K .. .. .. .. .. 19% 168.35 KB/s 1000K .. .. .. .. .. 20% 168.92 KB/s 1050K .. .. .. .. .. 21% 159.74 KB/s 1100K .. .. .. .. .. 22% 168.35 KB/s 1150K .. .. .. .. .. 23% 177.94 KB/s 1200K .. .. .. .. .. 24% 177.94 KB/s 1250K .. .. .. .. .. 25% 159.74 KB/s 1300K .. .. .. .. .. 26% 177.94 KB/s 1350K .. .. .. .. .. 27% 168.35 KB/s 1400K .. .. .. .. .. 28% 168.35 KB/s 1450K .. .. .. .. .. 29% 168.35 KB/s 1500K .. .. .. .. .. 30% 168.35 KB/s 1550K .. .. .. .. .. 31% 177.94 KB/s 1600K .. .. .. .. .. 32% 168.35 KB/s 1650K .. .. .. .. .. 33% 168.35 KB/s 1700K .. .. .. .. .. 34% 168.92 KB/s 1750K .. .. .. .. .. 35% 168.35 KB/s 1800K .. .. .. .. .. 36% 159.74 KB/s 1850K .. .. .. .. .. 37% 177.94 KB/s 1900K .. .. .. .. .. 38% 91.41 KB/s 1950K .. .. .. .. .. 39% 86.51 KB/s 2000K .. .. .. .. .. 40% 86.51 KB/s 2050K .. .. .. .. .. 41% 81.97 KB/s 2100K .. .. .. .. .. 42% 97.09 KB/s 2150K .. .. .. .. .. 43% 86.51 KB/s 2200K .. .. .. .. .. 44% 81.97 KB/s 2250K .. .. .. .. .. 45% 61.58 KB/s 2300K .. .. .. .. .. 46% 60.39 KB/s 2350K .. .. .. .. .. 47% 40.00 KB/s 2400K .. .. .. .. .. 48% 159.74 KB/s 2450K .. .. .. .. .. 49% 88.97 KB/s 2500K .. .. .. .. .. 50% 80.00 KB/s 2550K .. .. .. .. .. 51% 88.81 KB/s 2600K .. .. .. .. .. 52% 86.51 KB/s 2650K .. .. .. .. .. 53% 86.51 KB/s 2700K .. .. .. .. .. 54% 86.51 KB/s 2750K .. .. .. .. .. 55% 84.18 KB/s
Re: [sniffer] Sniffer Updates
I too am seeing really slow speeds, I'm running an update now and it is only downloading at about 3k/sec. Pretty bad considering we have 2 T1's and a DS3 none of which have much traffic on them this morning. Jim Matuska Jr. Computer Tech2, CCNA Nez Perce Tribe Information Systems [EMAIL PROTECTED] - Original Message - From: Russ Uhte [EMAIL PROTECTED] To: sniffer@SortMonster.com Sent: Monday, December 27, 2004 8:45 AM Subject: Re: [sniffer] Sniffer Updates Kevin Stanford wrote: Our updates seem to be taking a very long time. I am 85% updated and the ETA shows 07:00. Is it me? I see stuff like this come and go... Our updates are (finally) triggered from the email notifications... Below is a snippet of the last update that shows exactly what speeds we saw, which ran at 10:45 EST this morning... Every once in a while, I will see it slow down to about 8KB/s, but rarely slower than that... Thanks, Russ (This will probably wrap and look real ugly, but the last number is the average download speed for that part of the download...) 0K .. .. .. .. .. 0% 110.38 KB/s 50K .. .. .. .. .. 1% 160.26 KB/s 100K .. .. .. .. .. 2% 71.12 KB/s 150K .. .. .. .. .. 3% 110.13 KB/s 200K .. .. .. .. .. 4% 118.76 KB/s 250K .. .. .. .. .. 5% 145.35 KB/s 300K .. .. .. .. .. 6% 168.35 KB/s 350K .. .. .. .. .. 7% 168.35 KB/s 400K .. .. .. .. .. 8% 168.35 KB/s 450K .. .. .. .. .. 9% 160.26 KB/s 500K .. .. .. .. .. 10% 159.74 KB/s 550K .. .. .. .. .. 11% 188.68 KB/s 600K .. .. .. .. .. 12% 177.30 KB/s 650K .. .. .. .. .. 13% 168.35 KB/s 700K .. .. .. .. .. 14% 177.94 KB/s 750K .. .. .. .. .. 15% 168.35 KB/s 800K .. .. .. .. .. 16% 177.94 KB/s 850K .. .. .. .. .. 17% 168.35 KB/s 900K .. .. .. .. .. 18% 168.35 KB/s 950K .. .. .. .. .. 19% 168.35 KB/s 1000K .. .. .. .. .. 20% 168.92 KB/s 1050K .. .. .. .. .. 21% 159.74 KB/s 1100K .. .. .. .. .. 22% 168.35 KB/s 1150K .. .. .. .. .. 23% 177.94 KB/s 1200K .. .. .. .. .. 24% 177.94 KB/s 1250K .. .. .. .. .. 25% 159.74 KB/s 1300K .. .. .. .. .. 26% 177.94 KB/s 1350K .. .. .. .. .. 27% 168.35 KB/s 1400K .. .. .. .. .. 28% 168.35 KB/s 1450K .. .. .. .. .. 29% 168.35 KB/s 1500K .. .. .. .. .. 30% 168.35 KB/s 1550K .. .. .. .. .. 31% 177.94 KB/s 1600K .. .. .. .. .. 32% 168.35 KB/s 1650K .. .. .. .. .. 33% 168.35 KB/s 1700K .. .. .. .. .. 34% 168.92 KB/s 1750K .. .. .. .. .. 35% 168.35 KB/s 1800K .. .. .. .. .. 36% 159.74 KB/s 1850K .. .. .. .. .. 37% 177.94 KB/s 1900K .. .. .. .. .. 38% 91.41 KB/s 1950K .. .. .. .. .. 39% 86.51 KB/s 2000K .. .. .. .. .. 40% 86.51 KB/s 2050K .. .. .. .. .. 41% 81.97 KB/s 2100K .. .. .. .. .. 42% 97.09 KB/s 2150K .. .. .. .. .. 43% 86.51 KB/s 2200K .. .. .. .. .. 44% 81.97 KB/s 2250K .. .. .. .. .. 45% 61.58 KB/s 2300K .. .. .. .. .. 46% 60.39 KB/s 2350K .. .. .. .. .. 47% 40.00 KB/s 2400K .. .. .. .. .. 48% 159.74 KB/s 2450K .. .. .. .. .. 49% 88.97
Re: [sniffer] Sniffer Updates
It's actually getting worse now with a timed out transfer and now under 1k a sec: Resolving www.sortmonster.net... done.Connecting to www.sortmonster.net[216.88.37.61]:80... connected.HTTP request sent, awaiting response... 200 OKLength: 11,104,576 [application/x-sortmonster] 19% [== ] 2,141,361 2.99K/s ETA 48:46 09:29:12 (2.99 KB/s) - Connection closed at byte 2141361. Retrying. Connecting to www.sortmonster.net[216.88.37.61]:80... connected.HTTP request sent, awaiting response... 200 OKLength: 11,104,576 [application/x-sortmonster] 0% [ ] 87,921 993.81B/s ETA 3:04:45 Jim Matuska Jr.Computer Tech2, CCNANez Perce TribeInformation Systems[EMAIL PROTECTED] - Original Message - From: "Jim Matuska" [EMAIL PROTECTED] To: sniffer@SortMonster.com Sent: Monday, December 27, 2004 9:22 AM Subject: Re: [sniffer] Sniffer Updates I too am seeing really slow speeds, I'm running an update now and it is only downloading at about 3k/sec. Pretty bad considering we have 2 T1's and a DS3 none of which have much traffic on them this morning. Jim Matuska Jr. Computer Tech2, CCNA Nez Perce Tribe Information Systems [EMAIL PROTECTED] - Original Message - From: "Russ Uhte" [EMAIL PROTECTED] To: sniffer@SortMonster.com Sent: Monday, December 27, 2004 8:45 AM Subject: Re: [sniffer] Sniffer Updates Kevin Stanford wrote: Our updates seem to be taking a very long time. I am 85% updated and the ETA shows 07:00. Is it me? I see stuff like this come and go... Our updates are (finally) triggered from the email notifications... Below is a snippet of the last update that shows exactly what speeds we saw, which ran at 10:45 EST this morning... Every once in a while, I will see it slow down to about 8KB/s, but rarely slower than that... Thanks, Russ (This will probably wrap and look real ugly, but the last number is the average download speed for that part of the download...) 0K .. .. .. .. .. 0% 110.38 KB/s 50K .. .. .. .. .. 1% 160.26 KB/s 100K .. .. .. .. .. 2% 71.12 KB/s 150K .. .. .. .. .. 3% 110.13 KB/s 200K .. .. .. .. .. 4% 118.76 KB/s 250K .. .. .. .. .. 5% 145.35 KB/s 300K .. .. .. .. .. 6% 168.35 KB/s 350K .. .. .. .. .. 7% 168.35 KB/s 400K .. .. .. .. .. 8% 168.35 KB/s 450K .. .. .. .. .. 9% 160.26 KB/s 500K .. .. .. .. .. 10% 159.74 KB/s 550K .. .. .. .. .. 11% 188.68 KB/s 600K .. .. .. .. .. 12% 177.30 KB/s 650K .. .. .. .. .. 13% 168.35 KB/s 700K .. .. .. .. .. 14% 177.94 KB/s 750K .. .. .. .. .. 15% 168.35 KB/s 800K .. .. .. .. .. 16% 177.94 KB/s 850K .. .. .. .. .. 17% 168.35 KB/s 900K .. .. .. .. .. 18% 168.35 KB/s 950K .. .. .. .. .. 19% 168.35 KB/s 1000K .. .. .. .. .. 20% 168.92 KB/s 1050K .. .. .. .. .. 21% 159.74 KB/s 1100K .. .. .. .. .. 22% 168.35 KB/s 1150K .. .. .. .. .. 23% 177.94 KB/s 1200K .. .. .. .. .. 24% 177.94 KB/s 1250K .. .. .. .. .. 25% 159.74 KB/s 1300K .. .. .. .. .. 26% 177.94 KB/s 1350K .. .. .. .. .. 27% 168.35 KB/s 1400K .. .. .. .. .. 28% 168.35 KB/s 1450K .. .. .. .. .. 29% 168.35 KB/s 1500K .. .. .. .. .. 30% 168.35 KB/s 1550K .. .. .. .. .. 31% 177.94 KB/s 1600K .. .. .. .. .. 32% 168.35 KB/s 1650K .. .. .. .. .. 33% 168.35 KB/s 1700K .. .. .. .. .. 34% 168.92 KB/s 1750K .. .. .. .. .. 35% 168.35 KB/s 1800K .. .. .. .. .. 36% 159.74 KB/s 1850K .. .. .. .. .. 37% 177.94 KB/s 1900K .. .. .. .. ..
Re[2]: [sniffer] Sniffer Updates
On Monday, December 27, 2004, 12:46:19 PM, Landry wrote: LW Are folks taking advantage of the wget compression option before LW downloading their rulebase updates? If the slow download speeds are a LW bandwidth saturation issue on the Sniffer end, this would certainly cut down LW on the bandwidth requirements on their end and increase the download times LW for everyone. LW Also, I've got to ask, if the downloads are happening behind the scenes, LW by an automated or triggered download, why the concern about speeds, as long LW as your downloads are successful? From what I've seen in the logs, only about 5% of folks are taking advantage of gzip right now. Also, I did some incantations on the log (grep, awk, uniq etc) and came up with just under half of our customers downloading their rulebase between 1200 and 1300 today. That's between 2 and 3 times as many as should have done it ;-) -- so the backlog is explainable. This kind of thing happens for lots of reasons and there are a lot of ways to mitigate the problem. A big one on the list - certainly - is using the gzip capability. With only 5% of folks using this and average compression ratios well above 50% there is plenty of room to make a big dent in this. _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: Re[2]: [sniffer] Sniffer Updates
Title: Re: Re[2]: [sniffer] Sniffer Updates Automate harassment reminders to those of us not using it. :) I think I'll go enable gzip tonight -Original Message- From: [EMAIL PROTECTED] [EMAIL PROTECTED] To: Landry William sniffer@SortMonster.com Sent: Mon Dec 27 12:36:06 2004 Subject: Re[2]: [sniffer] Sniffer Updates On Monday, December 27, 2004, 12:46:19 PM, Landry wrote: LW Are folks taking advantage of the wget compression option before LW downloading their rulebase updates? If the slow download speeds are a LW bandwidth saturation issue on the Sniffer end, this would certainly cut down LW on the bandwidth requirements on their end and increase the download times LW for everyone. LW Also, I've got to ask, if the downloads are happening behind the scenes, LW by an automated or triggered download, why the concern about speeds, as long LW as your downloads are successful? >From what I've seen in the logs, only about 5% of folks are taking advantage of gzip right now. Also, I did some incantations on the log (grep, awk, uniq etc) and came up with just under half of our customers downloading their rulebase between 1200 and 1300 today. That's between 2 and 3 times as many as should have done it ;-) -- so the backlog is explainable. This kind of thing happens for lots of reasons and there are a lot of ways to mitigate the problem. A big one on the list - certainly - is using the gzip capability. With only 5% of folks using this and average compression ratios well above 50% there is plenty of room to make a big dent in this. _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: Re[2]: [sniffer] Sniffer Updates
Title: Re: Re[2]: [sniffer] Sniffer Updates Does anyone have any good instructions on how to modify your update scripts to use gzip? Jim Matuska Jr.Computer Tech2, CCNANez Perce TribeInformation Systems[EMAIL PROTECTED] - Original Message - From: Tom Baker | Netsmith Inc To: sniffer@SortMonster.com Sent: Monday, December 27, 2004 10:43 AM Subject: Re: Re[2]: [sniffer] Sniffer Updates Automate harassment reminders to those of us not using it. :)I think I'll go enable gzip tonight-Original Message-From: [EMAIL PROTECTED] [EMAIL PROTECTED]To: Landry William sniffer@SortMonster.comSent: Mon Dec 27 12:36:06 2004Subject: Re[2]: [sniffer] Sniffer UpdatesOn Monday, December 27, 2004, 12:46:19 PM, Landry wrote:LW Are folks taking advantage of the "wget" compression option beforeLW downloading their rulebase updates? If the slow download speeds are aLW bandwidth saturation issue on the Sniffer end, this would certainly cut downLW on the bandwidth requirements on their end and increase the download timesLW for everyone.LW Also, I've got to ask, if the downloads are happening "behind the scenes",LW by an automated or triggered download, why the concern about speeds, as longLW as your downloads are successful?From what I've seen in the logs, only about 5% of folks are takingadvantage of gzip right now.Also, I did some incantations on the log (grep, awk, uniq etc) andcame up with just under half of our customers downloading theirrulebase between 1200 and 1300 today. That's between 2 and 3 times asmany as should have done it ;-) -- so the backlog is explainable.This kind of thing happens for lots of reasons and there are a lot ofways to mitigate the problem.A big one on the list - certainly - is using the gzip capability. Withonly 5% of folks using this and average compression ratios well above50% there is plenty of room to "make a big dent" in this._MThis E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Conditional Sniffer Updates
Hi, The one thing I have not seen mentioned is the ability to do CONDITIONAL downloads - which is crucial for timed downloads when most of the time there may not even BE a more current .SNF file. Just like your browser, the HTTP Request for your latest .SNF file should ALWAYS provide the date/time stamp of your CURRENTLY active .SNF file. This way, the server will compare both dates and a download will occur ONLY, if there is LATER .SNF file on the server. (This is how your web browser controls, whether it needs to download new pages/images from sites you visited before.) Here is how CURL is used to do conditional downloads: curl http://www.sortmonster.net/Sniffer/Updates/[mylicensecode].snf -o [mylicensecode].snf.new -s -S -R -z [mylicensecode].snf -u [mywebuserid]:[mywebpassword] The -o option defines the output file. The -R option makes sure that the output file will inherit the timestamp from the Sniffer Server (if one is downloaded at all). The -z option sends the timestamp of the CURRENT SNF file to the server (in the GET request!) Since my local .SNF file has the same timestamp as the SERVER, and since every new GET request will allow the server to recognize if/that there may me no LATER .SNF file, I am only downloading when a new file is actually present! Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Monday, December 27, 2004 12:50 PM To: Russ Uhte Subject: Re[2]: [sniffer] Sniffer Updates On Monday, December 27, 2004, 11:45:59 AM, Russ wrote: RU Kevin Stanford wrote: Our updates seem to be taking a very long time. I am 85% updated and the ETA shows 07:00. Is it me? RU I see stuff like this come and go... Our updates are (finally) RU triggered from the email notifications... Below is a snippet of the RU last update that shows exactly what speeds we saw, which ran at 10:45 RU EST this morning... Every once in a while, I will see it slow down to RU about 8KB/s, but rarely slower than that... There are going to be random events like this for a while - as long as some folks still download based on a schedule rather than responding to update notifications. What happens is that sometimes a group of systems will agree to all download their rulebase files at the same time - when that happens our bandwidth gets saturated and things go slowly. (We are working on this in a number of ways.) Most of the time there is plenty of bandwidth, and if everyone always downloaded only when there was an update notification then there would always be plenty (our system paces updates to make sure this is the case as much as possible). We are in a transitional period where existing connectivity contracts prevent us from moving without incurring a significant cost (a cost we would rather not pass on to our customers). Over the next 6-9 months we will make the transition to a new rulebase format and distribution method and we will also be migrating to new hosting facilities (already running in case we encounter a serious DL problem). Since rulebase downloads should always be automated in some way, the occasional slow download should not be a problem. We will continue to monitor the situation closely - and we appreciate the reports we get. The things that you can do to help are: 1. If you haven't already, please upgrade your scripting so that your automated downloads are triggered from our update notifications. 2. If you are not going to use update notifications please be sure to use the staggered schedule we've posted here: http://www.sortmonster.com/MessageSniffer/Help/LogsHelp.html#When 3. AVOID using accelerated download software! This is the kind of software that downloads large files by opening multiple connections to the same server. Almost all of the slowdowns we experience have been associated with someone downloading a rulebase with this kind of software -- they open 100+ connections for themselves (sometimes more) and that slows things down for everyone else. We have adjusted our server's setting to mitigate this, but we can't turn it off completely without causing other performance problems ;-) Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: Re[2]: [sniffer] Sniffer Updates
Title: Re: Re[2]: [sniffer] Sniffer Updates See http://www.sortmonster.com/MessageSniffer/Help/AutomatingUpdatesHelp.htmlfor some sample scripts. Bill -Original Message-From: Jim Matuska [mailto:[EMAIL PROTECTED]Sent: Monday, December 27, 2004 10:51 AMTo: sniffer@SortMonster.comSubject: Re: Re[2]: [sniffer] Sniffer Updates Does anyone have any good instructions on how to modify your update scripts to use gzip? Jim Matuska Jr.Computer Tech2, CCNANez Perce TribeInformation Systems[EMAIL PROTECTED] - Original Message - From: Tom Baker | Netsmith Inc To: sniffer@SortMonster.com Sent: Monday, December 27, 2004 10:43 AM Subject: Re: Re[2]: [sniffer] Sniffer Updates Automate harassment reminders to those of us not using it. :)I think I'll go enable gzip tonight-Original Message-From: [EMAIL PROTECTED] [EMAIL PROTECTED]To: Landry William sniffer@SortMonster.comSent: Mon Dec 27 12:36:06 2004Subject: Re[2]: [sniffer] Sniffer UpdatesOn Monday, December 27, 2004, 12:46:19 PM, Landry wrote:LW Are folks taking advantage of the "wget" compression option beforeLW downloading their rulebase updates? If the slow download speeds are aLW bandwidth saturation issue on the Sniffer end, this would certainly cut downLW on the bandwidth requirements on their end and increase the download timesLW for everyone.LW Also, I've got to ask, if the downloads are happening "behind the scenes",LW by an automated or triggered download, why the concern about speeds, as longLW as your downloads are successful?From what I've seen in the logs, only about 5% of folks are takingadvantage of gzip right now.Also, I did some incantations on the log (grep, awk, uniq etc) andcame up with just under half of our customers downloading theirrulebase between 1200 and 1300 today. That's between 2 and 3 times asmany as should have done it ;-) -- so the backlog is explainable.This kind of thing happens for lots of reasons and there are a lot ofways to mitigate the problem.A big one on the list - certainly - is using the gzip capability. Withonly 5% of folks using this and average compression ratios well above50% there is plenty of room to "make a big dent" in this._MThis E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html ---This message and any included attachments are from Siemens Medical Solutions USA, Inc. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by e-mail with a copy to [EMAIL PROTECTED] Thank you
RE: [sniffer] Conditional Sniffer Updates
Curl is an awesome application that we also use for automating downloads. Wget also supports conditional downloads based on time/date stamp when using the -N switch. In ether case, please also use the compression support built into each application, the sniffer rulebase files can be compressed down to about 25% of their normal size before the download by using these switches. Here is an example of how to use wget to check for rulebase updates and if a new file exists, request file compression before the file is downloaded: wget -N http://www.sortmonster.net/Sniffer/Updates/LicenseID.snf -O LicenseID.new.gz --header=Accept-Encoding:gzip --http-user=sniffer --http-passwd=ki11sp8m Bill -Original Message- From: Andy Schmidt [mailto:[EMAIL PROTECTED] Sent: Monday, December 27, 2004 11:20 AM To: sniffer@SortMonster.com Subject: [sniffer] Conditional Sniffer Updates Hi, The one thing I have not seen mentioned is the ability to do CONDITIONAL downloads - which is crucial for timed downloads when most of the time there may not even BE a more current .SNF file. Just like your browser, the HTTP Request for your latest .SNF file should ALWAYS provide the date/time stamp of your CURRENTLY active .SNF file. This way, the server will compare both dates and a download will occur ONLY, if there is LATER .SNF file on the server. (This is how your web browser controls, whether it needs to download new pages/images from sites you visited before.) Here is how CURL is used to do conditional downloads: curl http://www.sortmonster.net/Sniffer/Updates/[mylicensecode].snf -o [mylicensecode].snf.new -s -S -R -z [mylicensecode].snf -u [mywebuserid]:[mywebpassword] The -o option defines the output file. The -R option makes sure that the output file will inherit the timestamp from the Sniffer Server (if one is downloaded at all). The -z option sends the timestamp of the CURRENT SNF file to the server (in the GET request!) Since my local .SNF file has the same timestamp as the SERVER, and since every new GET request will allow the server to recognize if/that there may me no LATER .SNF file, I am only downloading when a new file is actually present! Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Monday, December 27, 2004 12:50 PM To: Russ Uhte Subject: Re[2]: [sniffer] Sniffer Updates On Monday, December 27, 2004, 11:45:59 AM, Russ wrote: RU Kevin Stanford wrote: Our updates seem to be taking a very long time. I am 85% updated and the ETA shows 07:00. Is it me? RU I see stuff like this come and go... Our updates are (finally) RU triggered from the email notifications... Below is a snippet of the RU last update that shows exactly what speeds we saw, which ran at 10:45 RU EST this morning... Every once in a while, I will see it slow down to RU about 8KB/s, but rarely slower than that... There are going to be random events like this for a while - as long as some folks still download based on a schedule rather than responding to update notifications. What happens is that sometimes a group of systems will agree to all download their rulebase files at the same time - when that happens our bandwidth gets saturated and things go slowly. (We are working on this in a number of ways.) Most of the time there is plenty of bandwidth, and if everyone always downloaded only when there was an update notification then there would always be plenty (our system paces updates to make sure this is the case as much as possible). We are in a transitional period where existing connectivity contracts prevent us from moving without incurring a significant cost (a cost we would rather not pass on to our customers). Over the next 6-9 months we will make the transition to a new rulebase format and distribution method and we will also be migrating to new hosting facilities (already running in case we encounter a serious DL problem). Since rulebase downloads should always be automated in some way, the occasional slow download should not be a problem. We will continue to monitor the situation closely - and we appreciate the reports we get. The things that you can do to help are: 1. If you haven't already, please upgrade your scripting so that your automated downloads are triggered from our update notifications. 2. If you are not going to use update notifications please be sure to use the staggered schedule we've posted here: http://www.sortmonster.com/MessageSniffer/Help/LogsHelp.html#When 3. AVOID using accelerated download software! This is the kind of software that downloads large files by opening multiple connections to the same server. Almost all of the slowdowns we experience have been associated with someone downloading a rulebase with this
Re[4]: [sniffer] Sniffer Updates
On Monday, December 27, 2004, 1:51:11 PM, Jim wrote: JM Does anyone have any good instructions on how to modify your update scripts to use gzip? This is a good place to start: http://www.sortmonster.com/MessageSniffer/Help/gzip.html _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: Re[2]: [sniffer] Sniffer Updates
Title: Re: Re[2]: [sniffer] Sniffer Updates I made this one, which is probably also somewhere on the sniffer site. Change directories and keys for your use: d: cd\Batch Files\Sniffer wget http://sniffer:[EMAIL PROTECTED]/Sniffer/Updates/key.snf -O key.snf.gz --timestamping --header=Accept-Encoding:gzip gzip -d -f key.snf.gz :Check fcom32 "c:\mdaemon\sniffer\key.snf" "d:\batch files\sniffer\key.snf" if errorlevel 1 goto Test goto :Done :Test snf2check.exe key.snf password if errorlevel 1 goto Done copy /y key.snf c:\mdaemon\sniffer copy /y key.snf key.old :Done Check for wrapping by your e-mail client! I've put an empty line between every line, to make sure you see what belongs together. Next to the --timestaping feature of wget, I also use fcom32.exe to see if the file is really different than the one before. This example also uses gzip! Greets, Michiel From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jim MatuskaSent: maandag 27 december 2004 19:51To: sniffer@SortMonster.comSubject: Re: Re[2]: [sniffer] Sniffer Updates Does anyone have any good instructions on how to modify your update scripts to use gzip? Jim Matuska Jr.Computer Tech2, CCNANez Perce TribeInformation Systems[EMAIL PROTECTED] - Original Message - From: Tom Baker | Netsmith Inc To: sniffer@SortMonster.com Sent: Monday, December 27, 2004 10:43 AM Subject: Re: Re[2]: [sniffer] Sniffer Updates Automate harassment reminders to those of us not using it. :)I think I'll go enable gzip tonight-Original Message-From: [EMAIL PROTECTED] [EMAIL PROTECTED]To: Landry William sniffer@SortMonster.comSent: Mon Dec 27 12:36:06 2004Subject: Re[2]: [sniffer] Sniffer UpdatesOn Monday, December 27, 2004, 12:46:19 PM, Landry wrote:LW Are folks taking advantage of the "wget" compression option beforeLW downloading their rulebase updates? If the slow download speeds are aLW bandwidth saturation issue on the Sniffer end, this would certainly cut downLW on the bandwidth requirements on their end and increase the download timesLW for everyone.LW Also, I've got to ask, if the downloads are happening "behind the scenes",LW by an automated or triggered download, why the concern about speeds, as longLW as your downloads are successful?From what I've seen in the logs, only about 5% of folks are takingadvantage of gzip right now.Also, I did some incantations on the log (grep, awk, uniq etc) andcame up with just under half of our customers downloading theirrulebase between 1200 and 1300 today. That's between 2 and 3 times asmany as should have done it ;-) -- so the backlog is explainable.This kind of thing happens for lots of reasons and there are a lot ofways to mitigate the problem.A big one on the list - certainly - is using the gzip capability. Withonly 5% of folks using this and average compression ratios well above50% there is plenty of room to "make a big dent" in this._MThis E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Downloads are slow...
Pete, With all due respect - I think the download problem is self-inflicted, because your web site is providing unsuitable examples to your customers! Even with moderate bandwidth, your server would be able to handle tens of thousands of hits a day. Checking if an updated file exists should barely be noticeable - as long as it doesn't result in an unnecessary download. You probably suffer TWO problems: A) Most of your customers are downloading rules based on a schedule, even if no rules exists. Potential savings: 100% per download attempt. B) Your customers are not downloading compressed rule files. Potential savings: about 66%, but that's not bad either. One likely explanation is that at least THREE of your sample scripts do an unconditional and uncompressed download! Here the 3 URLs you list on your web site and WGET command they are using: http://www.sortmonster.com/MessageSniffer/Help/UserScripts/david_snifferUpda teMethod.zip wget http://www.sortmonster.net/Sniffer/Updates/.snf -O .new --http-user=username --http-passwd=password http://www.sortmonster.com/MessageSniffer/Help/UserScripts/Hank_SnifferScrip ts.zip wget http://www.sortmonster.net/Sniffer/Updates/.snf -O .new --http-user=sniffer --http-passwd=ki11sp8m http://www.sortmonster.com/MessageSniffer/Help/UserScripts/Michiel_AutoUpdat e.zip wget http://sniffer:[EMAIL PROTECTED]/Sniffer/Updates/12345678.snf -O serial.tst My recommendation: Replace these with examples that implement conditional, compressed downloading. Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Monday, December 27, 2004 08:10 AM To: Chuck Schick Subject: Re: [sniffer] Downloads are slow... On Monday, December 27, 2004, 1:17:21 AM, Chuck wrote: CS Pete: CS It appears on weekends the sniffer downloads are really slow. I am CS downloading at 14 minutes past the hour and I am about 1/20 th of CS the normal speed. That is an unusual observation - I don't think weekends have anything to do with making things slower. I will look at the logs to see if I can figure out what heppened. You're not manually downloading I hope? _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Sniffer Updates-- am I missing something
What am I missing in this thread? I use an Imail program alias that automatically runs a download script when I am notified by [EMAIL PROTECTED] when a new rule base is available; therefore only a validation needs to be preformed. I took this procedure from this list, so I know it must be common knowledge. What advantage is there to a scheduled or manual rule base update that requires conditional test? It seems to me that I have no need to check for rule base updates until I have been advised by support that one is available. Will there ever be a rule base update available before I have been notified? Woody Fussell Wilbur Smith Associates [EMAIL PROTECTED]
Re: [sniffer] Downloads are slow...
I agree entirely. If bandwidth has become an issue, it would be resolved with a focus on producing very tight and easily customizable scripts (a variables section in the top of the scripts). I believe that going the VBScript route might be the best way to go, or at least I believe that more of us can hack a more involved VBScript than a batch or CMD file. Enforcing compressed downloads and checking for timestamps prior to downloading should be done in these scripts as well. Right now the script examples assume a familiarity with scripting, and while local participants can mostly handle that stuff, the non-vocal ones are most likely to not even be aware of the issues or how to fix them, and might have scripted timed downloads because it is definitely the easiest way to go. This is probably the majority of the customer base. There is an impression for instance with Declude's user base that +80% use primarily the default config which most of us know is severely lacking in comparison to the potential that exists by tweaking the settings. With better script examples and a careful step-by-step readme promoted in a mailing to your customers, I believe that this issue could go away, or at least theoretically it should. Personally, I have mine tied to the E-mails, I download the zipped versions, I don't bother checking on the status, and have never noticed any issues as a result. It would be a small shame if I was missing downloads due to timeouts, but not that big of a deal if this has never caused a noticeable problem. Matt Andy Schmidt wrote: Pete, With all due respect - I think the download problem is self-inflicted, because your web site is providing unsuitable examples to your customers! Even with moderate bandwidth, your server would be able to handle tens of thousands of hits a day. Checking if an updated file exists should barely be noticeable - as long as it doesn't result in an unnecessary download. You probably suffer TWO problems: A) Most of your customers are downloading rules based on a schedule, even if no rules exists. Potential savings: 100% per download attempt. B) Your customers are not downloading compressed rule files. Potential savings: about 66%, but that's not bad either. One likely explanation is that at least THREE of your sample scripts do an unconditional and uncompressed download! Here the 3 URLs you list on your web site and WGET command they are using: http://www.sortmonster.com/MessageSniffer/Help/UserScripts/david_snifferUpda teMethod.zip wget http://www.sortmonster.net/Sniffer/Updates/.snf -O .new --http-user=username --http-passwd=password http://www.sortmonster.com/MessageSniffer/Help/UserScripts/Hank_SnifferScrip ts.zip wget http://www.sortmonster.net/Sniffer/Updates/.snf -O .new --http-user=sniffer --http-passwd=ki11sp8m http://www.sortmonster.com/MessageSniffer/Help/UserScripts/Michiel_AutoUpdat e.zip wget http://sniffer:[EMAIL PROTECTED]/Sniffer/Updates/12345678.snf -O serial.tst My recommendation: Replace these with examples that implement conditional, compressed downloading. Best Regards Andy Schmidt HM Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Monday, December 27, 2004 08:10 AM To: Chuck Schick Subject: Re: [sniffer] Downloads are slow... On Monday, December 27, 2004, 1:17:21 AM, Chuck wrote: CS Pete: CS It appears on weekends the sniffer downloads are really slow. I am CS downloading at 14 minutes past the hour and I am about 1/20 th of CS the normal speed. That is an unusual observation - I don't think weekends have anything to do with making things slower. I will look at the logs to see if I can figure out what heppened. You're not manually downloading I hope? _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
