[sniffer] Re: Update Script - Choice of WGET Parameter Prevents TimeStamping
PS: And, for bonus points, to correctly support your sub-directory feature in your sample script, you would do that with the -P parameter, e.g.: wget http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf -N -P %RULEBASE_PATH% --header=Accept-Encoding:gzip --http-user=sniffer --http-passwd=ki11sp8m From: Andy Schmidt [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 08, 2008 12:41 AM To: 'Message Sniffer Community' Subject: Update Script - Choice of WGET Parameter Prevents TimeStamping Hi, I've spent some time over the last few days trying to integrate the new sniffer update scheme into my current scripts and kept hitting a wall because the update script was downloading the rulebase file with the CURRENT date/time instead of your webserver's date/time. In the past I had used CURL instead of WGET, but I'm trying to stick with the provided samples, the best I can (to make future upgrades easier). I finally figured out why the downloaded files were timestamped incorrectly (and why the conditional download that I had working with CURL was not working with WGET. The reason was your choice of WGET parameters in your sample. You currently are using: wget http://www.sortmonster.net/Sniffer/Updates/(licensecode).snf -S -O (licensecode).snf --header=Accept-Encoding:gzip --http-user=sniffer --http-passwd=ki11sp8m However, the -O parameter is not an output file parameter, but rather an OVERRIDE parameter intended for cases where MORE than one file is downloaded from the server. The '-O' parameter allows you to combine ALL these downloaded files into a SINGLE file. Since all files are combined into one large file, the file date is simply set to the current time. Clearly, this entire scenario does NOT apply to the rulebase download! Worse, this overrides the normal handling of downloads, where the output filename is controlled by the server AND the timestamp of the local file would be set to the Last-Modified header of your web server. The effect is, that the downloaded files have the wrong timestamp and thus will prevent employing a conditional download scheme in cases where the local file already exists with the correct size and timestamp. The normal command (and the one intended for YOUR application) would be: wget http://www.sortmonster.net/Sniffer/Updates/(licensecode).snf -S -N --header=Accept-Encoding:gzip --http-user=sniffer --http-passwd=ki11sp8m This will: a) Download the file, maintain the filename and (by omitting the -O) inherit the original timestamp from the web server -as it should be. b) The -N parameter will further improve the situation. If the local file already exists with the correct file size and time stamp, then an unnecessary download will be skipped! Again, I know, that you are only providing your script as a sample - but, the closer your sample tracks reality the fewer customers will see a need to adapt it and thus reducing YOUR tech support effort if the customer modifications lead to errors. Best Regards Andy Schmidt Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206
[sniffer] Re: Update Script - Choice of WGET Parameter Prevents TimeStamping
Hello Andy, Wednesday, October 8, 2008, 12:50:23 AM, you wrote: PS: And, for bonus points, to correctly support your sub-directory feature in your sample script, you would do that with the P parameter, e.g.: wget http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf-N -P %RULEBASE_PATH%--header=Accept-Encoding:gzip --http-user=sniffer --http-passwd=ki11sp8m snip/ Thanks for your help. We have to cover a lot of ground so we often get solutions from our customers and others that just want to help out. We do our best to test and edit. I will see that your suggestions / corrections are reviewed and included in our updates. In any case they will be in the mailing list archives ;-) THANKS! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Update Script - Choice of WGET Parameter Prevents TimeStamping
Hi Pete, Thanks for giving it your consideration. If you decide to revise these parameteres, then it will require an extra command in your script (because the WGET command will output the compressed file as .SNF). If you don't insist on using WGET, then CURL (also free/open software) actually has more flexible parameters that will simplify your script because it will let you compare the timestamp of the unzipped, local .SNF file against the server timestamp, e.g.: curl http://www.sortmonster.net/Sniffer/Updates/(licensecode).snf -o (licensecode).snf.gz -s -S -R -z (licensecode).snf -H Accept-Encoding:gzip -u sniffer:ki11sp8m Best Regards, Andy From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Wednesday, October 08, 2008 1:03 AM To: Message Sniffer Community Subject: [sniffer] Re: Update Script - Choice of WGET Parameter Prevents TimeStamping Hello Andy, Wednesday, October 8, 2008, 12:50:23 AM, you wrote: PS: And, for bonus points, to correctly support your sub-directory feature in your sample script, you would do that with the -P parameter, e.g.: wget http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf -N -P %RULEBASE_PATH% --header=Accept-Encoding:gzip --http-user=sniffer --http-passwd=ki11sp8m snip/ Thanks for your help. We have to cover a lot of ground so we often get solutions from our customers and others that just want to help out. We do our best to test and edit. I will see that your suggestions / corrections are reviewed and included in our updates. In any case they will be in the mailing list archives ;-) THANKS! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Update Script - Choice of WGET Parameter Prevents TimeStamping
Hello Andy, Wednesday, October 8, 2008, 1:13:50 AM, you wrote: Hi Pete, Thanks for giving it your consideration. If you decide to revise these parameteres, then it will require an extra command in your script (because the WGET command will output the compressed file as .SNF). There is actually a bit more to it than that -- The existing script generally works even though it doesn't preserve the servers's timestamp because: 1. It is usually triggered from the SNFServer when SNF detects a newer rulebase file. 2. Any rulebase fill recently downloaded is guaranteed to be newer provided the local server's clock is correct (or close to it). Also -- are you saying that with the parameters you've provided WGET would decompress the file on it's own so that we wouldn't need to do that in our script? If so, how does it know for sure where to find GZIP? If not then it would be a little dangerous to have a .snf file around that looked correct but was in fact not yet decompressed. Another consideration is that if the file name is going to collide with the existing rulebase file we would want to move that into another location so that we don't stomp the existing rulebase file until we've tested the new one. It would be preferable to use WGET since there's nothing wrong with it and we've been using it long enough that most SNF folks already have it. That doesn't mean you shouldn't provide an alternate script that works with CURL just in case someone has a preference. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Update Script - Choice of WGET Parameter Prevents TimeStamping
Hello Andy, Wednesday, October 8, 2008, 1:35:41 AM, you wrote: snip/ Also -- are you saying that with the parameters you've provided WGET would decompress the file on it's own so that we wouldn't need to do that in our script? If so, how does it know for sure where to find GZIP? snip/ Sorry--- the above was just me hitting the button before my brain caught up with my fingers. You did say that the compressed file would be output as .SNF -- _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
