RE: [sniffer] Conditional Sniffer Updates
Yes, of course, with CURL use the equivalent option to request the GZ packed file instead: curl http://www.sortmonster.net/Sniffer/Updates/[mylicensecode].snf -o [mylicensecode].snf.gz -s -S -R -z [mylicensecode].snf -H "Accept-Encoding:gzip" -u [websiteuser]:[websitepassword] Best Regards Andy Schmidt H&M Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Landry William Sent: Monday, December 27, 2004 02:38 PM To: 'sniffer@SortMonster.com' Subject: RE: [sniffer] Conditional Sniffer Updates "Curl" is an awesome application that we also use for automating downloads. "Wget" also supports conditional downloads based on time/date stamp when using the -N switch. In ether case, please also use the compression support built into each application, the sniffer rulebase files can be compressed down to about 25% of their normal size before the download by using these switches. Here is an example of how to use wget to check for rulebase updates and if a new file exists, request file compression before the file is downloaded: wget -N http://www.sortmonster.net/Sniffer/Updates/LicenseID.snf -O LicenseID.new.gz --header=Accept-Encoding:gzip --http-user=sniffer --http-passwd=ki11sp8m Bill -Original Message- From: Andy Schmidt [mailto:[EMAIL PROTECTED] Sent: Monday, December 27, 2004 11:20 AM To: sniffer@SortMonster.com Subject: [sniffer] Conditional Sniffer Updates Hi, The one thing I have not seen mentioned is the ability to do CONDITIONAL downloads - which is crucial for "timed" downloads when most of the time there may not even BE a more current .SNF file. Just like your browser, the HTTP Request for your latest .SNF file should ALWAYS provide the "date/time" stamp of your CURRENTLY active .SNF file. This way, the server will compare both dates and a download will occur ONLY, if there is LATER .SNF file on the server. (This is how your web browser controls, whether it needs to download new pages/images from sites you visited before.) Here is how CURL is used to do conditional downloads: curl http://www.sortmonster.net/Sniffer/Updates/[mylicensecode].snf -o [mylicensecode].snf.new -s -S -R -z [mylicensecode].snf -u [mywebuserid]:[mywebpassword] The "-o" option defines the output file. The "-R" option makes sure that the output file will inherit the timestamp from the Sniffer Server (if one is downloaded at all). The "-z" option sends the timestamp of the CURRENT SNF file to the server (in the GET request!) Since my local .SNF file has the same timestamp as the SERVER, and since every new GET request will allow the server to recognize if/that there may me no LATER .SNF file, I am only downloading when a new file is actually present! Best Regards Andy Schmidt H&M Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Monday, December 27, 2004 12:50 PM To: Russ Uhte Subject: Re[2]: [sniffer] Sniffer Updates On Monday, December 27, 2004, 11:45:59 AM, Russ wrote: RU> Kevin Stanford wrote: >> Our updates seem to be taking a very long time. I am 85% updated and >> the ETA shows 07:00. Is it me? RU> I see stuff like this come and go... Our updates are (finally) RU> triggered from the email notifications... Below is a snippet of the RU> last update that shows exactly what speeds we saw, which ran at RU> 10:45 EST this morning... Every once in a while, I will see it slow RU> down to about 8KB/s, but rarely slower than that... There are going to be random events like this for a while - as long as some folks still download based on a schedule rather than responding to update notifications. What happens is that sometimes a group of systems will "agree" to all download their rulebase files at the same time - when that happens our bandwidth gets saturated and things go slowly. (We are working on this in a number of ways.) Most of the time there is plenty of bandwidth, and if everyone always downloaded only when there was an update notification then there would always be plenty (our system paces updates to make sure this is the case as much as possible). We are in a transitional period where existing connectivity contracts prevent us from moving without incurring a significant cost (a cost we would rather not pass on to our customers). Over the next 6-9 months we will make the transition to a new rulebase format and distribution method and we will also be migrating to new ho
RE: [sniffer] Conditional Sniffer Updates
"Curl" is an awesome application that we also use for automating downloads. "Wget" also supports conditional downloads based on time/date stamp when using the -N switch. In ether case, please also use the compression support built into each application, the sniffer rulebase files can be compressed down to about 25% of their normal size before the download by using these switches. Here is an example of how to use wget to check for rulebase updates and if a new file exists, request file compression before the file is downloaded: wget -N http://www.sortmonster.net/Sniffer/Updates/LicenseID.snf -O LicenseID.new.gz --header=Accept-Encoding:gzip --http-user=sniffer --http-passwd=ki11sp8m Bill -Original Message- From: Andy Schmidt [mailto:[EMAIL PROTECTED] Sent: Monday, December 27, 2004 11:20 AM To: sniffer@SortMonster.com Subject: [sniffer] Conditional Sniffer Updates Hi, The one thing I have not seen mentioned is the ability to do CONDITIONAL downloads - which is crucial for "timed" downloads when most of the time there may not even BE a more current .SNF file. Just like your browser, the HTTP Request for your latest .SNF file should ALWAYS provide the "date/time" stamp of your CURRENTLY active .SNF file. This way, the server will compare both dates and a download will occur ONLY, if there is LATER .SNF file on the server. (This is how your web browser controls, whether it needs to download new pages/images from sites you visited before.) Here is how CURL is used to do conditional downloads: curl http://www.sortmonster.net/Sniffer/Updates/[mylicensecode].snf -o [mylicensecode].snf.new -s -S -R -z [mylicensecode].snf -u [mywebuserid]:[mywebpassword] The "-o" option defines the output file. The "-R" option makes sure that the output file will inherit the timestamp from the Sniffer Server (if one is downloaded at all). The "-z" option sends the timestamp of the CURRENT SNF file to the server (in the GET request!) Since my local .SNF file has the same timestamp as the SERVER, and since every new GET request will allow the server to recognize if/that there may me no LATER .SNF file, I am only downloading when a new file is actually present! Best Regards Andy Schmidt H&M Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Monday, December 27, 2004 12:50 PM To: Russ Uhte Subject: Re[2]: [sniffer] Sniffer Updates On Monday, December 27, 2004, 11:45:59 AM, Russ wrote: RU> Kevin Stanford wrote: >> Our updates seem to be taking a very long time. I am 85% updated and >> the ETA shows 07:00. Is it me? RU> I see stuff like this come and go... Our updates are (finally) RU> triggered from the email notifications... Below is a snippet of the RU> last update that shows exactly what speeds we saw, which ran at 10:45 RU> EST this morning... Every once in a while, I will see it slow down to RU> about 8KB/s, but rarely slower than that... There are going to be random events like this for a while - as long as some folks still download based on a schedule rather than responding to update notifications. What happens is that sometimes a group of systems will "agree" to all download their rulebase files at the same time - when that happens our bandwidth gets saturated and things go slowly. (We are working on this in a number of ways.) Most of the time there is plenty of bandwidth, and if everyone always downloaded only when there was an update notification then there would always be plenty (our system paces updates to make sure this is the case as much as possible). We are in a transitional period where existing connectivity contracts prevent us from moving without incurring a significant cost (a cost we would rather not pass on to our customers). Over the next 6-9 months we will make the transition to a new rulebase format and distribution method and we will also be migrating to new hosting facilities (already running in case we encounter a serious DL problem). Since rulebase downloads should always be automated in some way, the occasional slow download should not be a problem. We will continue to monitor the situation closely - and we appreciate the reports we get. The things that you can do to help are: 1. If you haven't already, please upgrade your scripting so that your automated downloads are triggered from our update notifications. 2. If you are not going to use update notifications please be sure to use the staggered schedule we've posted here: http://www.sortmonster.com/MessageSniffer/Help/LogsHelp.html#When 3. AVOID using accelerated download software! This is the kind of software that downloads large files by opening multiple connections to the same server. Almost all of the "slowdowns" we experience have been associated with someone download
