The last thing before I leave for the weekend... I finally got around to updating my download/upload script so that I can upload compressed logs.
In the course of doing that, I found that my upgraded version of wget has changed its behaviour; as of the 1.10.x series, if you specify -O to specify the target filename, various options are ignored including the -N for "download only if server side is newer". Therefore, ever since I upgraded my wget, I've been downloading a compressed rulebase file on *each* run. Some of this script is antique and some of it is new. I just downloaded the "standard download script" that Bill Landry ushered into this world, and my script was certainly informed by the discussions of that on this list. (I'm not trying to replace that script, I'm just giving credit where credit is due.) My .cmd file script is attached as a .txt file; as I mentioned a while back, I use both the IMail "external script" mailbox method to launch this file when SortMonster/ARM sends me my notification, and I also run it on a schedule with the AT command so that one of them will work to get timely updates. Andrew 8)
echo off REM This file is C:\Utils\UpdateSniffer.cmd REM AC July-06-2003 REM The purpose is to truncate the MessageSniffer log and fetch a fresh copy of the REM spam signatures rulebase. REM Dependencies: REM wget.exe package (free) from hhttp://xoomer.alice.it/hherold/ (not http://www.gnu.org/software/wget/wget.html which lags!) REM gunzip.exe (free) from http://www.gzip.org/ REM fgrep.exe (free) from http://www.gnu.org/software/grep/grep.html REM timeout.exe (free) part of Microsoft Windows Server Resource Kit REM UploadScript.txt this is the text of our side of the ftp upload conversation, now built on the fly REM REM Folders used: REM d:\MessageSniffer REM d:\MessageSniffer\Download REM d:\MessageSniffer\OldLogs REM REM Oct-31-2004 AC updated to incorporate .gz compression when a download is made REM a big thanks to Bill Landry for a working example! REM Oct-30-2004 AC updated to skip the download with wget -N if it is a duplicate of a previous download REM and also to skip the update if the download was a duplicate REM Sep-07-2004 AC updated to make use of the sniffer check executable REM Jun-07-2005 AC renamed mailsniffer folder to MessageSniffer REM Jul-07-2006 AC wget defaults have changed, the (newer) option no longer works with the (rename download) option REM and I've added extra safety checking logic and gzipping, and on-the-fly build of the REM FTP upload script. REM LinceseID=This is the 8 character text that defines your rulebase and executable name. set LicenseID=12345678 REM Authorization=This is the 16 character text that is your "password" and you must keep secret. set Authorization=1234567890123456 REM FQDN=This is the fully qualified domain name at which you receive your email, e.g. bentall.com set FQDN=example.com set SnifferDrive=d: set SnifferFolder=d:\MessageSniffer if "%FQDN%" == "example.com" (echo No, really. You must set the FQDN variable to your own domain name. Now quitting. & goto quit) REM Change the current directory to the Message Sniffer drive and directory %SnifferDrive% cd %SnifferFolder% echo Clearing the old result files and logs if exist wgetresult.txt del wgetresult.txt REM Use the -N parameter to only download a newer file; note that as of wget v1.1x this is no longer REM compatible with the -O parameter to specify the destination filename. REM Note the use of the gzip encoding to save bandwidth. REM We use the -o parameter and create a new logfile each time so that we can grep the result if not exist %SnifferFolder%\Download md %SnifferFolder%\Download wget --tries=10 --wait=5 --random-wait --directory-prefix=%SnifferFolder%\Download -o wgetresult.txt --header=Accept-Encoding:gzip -N --http-user=sniffer --http-passwd=ki11sp8m http://www.sortmonster.net/Sniffer/Updates/%LicenseID%.snf REM If we didn't get the file, at least rotate the log file before quitting if errorlevel 1 goto logrot REM The errorlevel is zero whether we downloaded a fresh copy or skipped the download. REM Let's find out if we can skip the checking and backup and go straight to the log rotation and upload. REM If this grep is succesfull, the errorlevel is zero. fgrep "Server file no newer than local file" wgetresult.txt if not errorlevel 1 goto logrot ren %SnifferFolder%\Download\%LicenseID%.snf %LicenseID%.snf.gz copy %SnifferFolder%\Download\%LicenseID%.snf.gz %SnifferFolder%\Download\%LicenseID%.snf.gz.bak REM Unzip the compressed file; gunzip will automatically hack off the extension, REM creating %LicenseID%.snf and it will delete the .gz file too echo Now testing the wget'ed file gunzip -d -f -v %SnifferFolder%\Download\%LicenseID%.snf.gz REM However, sometimes wget downloads a zero length file, in which case the gunzip fails if errorlevel 1 (echo Some kind of error. Deleting the download etc and skipping the update & del %SnifferFolder%\Download\%LicenseID%.snf.* & goto logrot) REM Double-check the download with the Sniffer rulebase checker %SnifferFolder%\snf2check.exe %SnifferFolder%\Download\%LicenseID%.snf %Authorization% echo The errorlevel returned by snf2check was [%errorlevel%] if errorlevel 73 echo Malformed rulebase or mismatched auth code. if errorlevel 67 echo The rulebase does not exist. if errorlevel 65 echo Bad command line parameters. if errorlevel 1 (echo Something was wrong with the parameters or the rulebase & goto logrot) echo The new rulebase is fine. REM Backup the current rulebase if exist %SnifferFolder%\%LicenseID%.snf.old del %SnifferFolder%\%LicenseID%.snf.old ren %SnifferFolder%\%LicenseID%.snf %LicenseID%.snf.old :DoReplace REM Move the new update into the current folder move /y %SnifferFolder%\Download\%LicenseID%.snf %SnifferFolder% REM Rename the backup of the .gz file so that the next wget will be able to check if the download is newer ren %SnifferFolder%\Download\%LicenseID%.snf.gz.bak %LicenseID%.snf :DoReload REM We drop a "%LicenseID% reload" semaphore file here, although the persistent Sniffer checks every ten minutes anyway if exist %LicenseID%.reload del %LicenseID%.reload %SnifferFolder%\%LicenseID%.exe reload :LogRot echo Now at LogRot section if exist %LicenseID%.rotate del %LicenseID%.rotate %LicenseID%.exe rotate echo Pausing a few seconds to give the persistent Sniffer time to do the rotating... timeout 3 :LogZip echo Now at LogZip section REM The result of the gzip will be files named such as: %LicenseID%.log.20060707162120.gz REM The ".20" syntax is used to cleverly skip the current log, and only deal with rotated logs. if not exist %LicenseID%.log.20???????????? (echo There are no rotated logs to deal with. & goto quit) for %%x in (%LicenseID%.log.20????????????) do gzip -Nv9 -S .gz %%x REM For extra credit, we'll prepend our fully qualified domain name; here is one way to state REM a legitimate naming convention for the uploads: REM [yourfullyqualifieddomain.]yourSNFlicenseid.log[.randomstring][.gz|.zip] for %%x in (%LicenseID%.log.20*) do ren %%x bentall.com.%%x :LogUpload echo Now at LogUpload section echo Now starting the FTP upload of the sniffer logs REM We build the UploadCompressedScript.txt on the fly from here REM so that we have one fewer files to maintain and remember to enter our LicenseID. REM The UploadCompressedScript.txt looks like this: REM ............................................... echo open ftp.sortmonster.net>%SnifferFolder%\UploadScript.txt echo user snifferlog>>%SnifferFolder%\UploadScript.txt echo ki11sp8m>>%SnifferFolder%\UploadScript.txt echo literal PASV>>%SnifferFolder%\UploadScript.txt echo binary>>%SnifferFolder%\UploadScript.txt echo prompt>>%SnifferFolder%\UploadScript.txt echo mput %LicenseID%.log.20*>>%SnifferFolder%\UploadScript.txt echo close>>%SnifferFolder%\UploadScript.txt echo quit>>%SnifferFolder%\UploadScript.txt echo.>>%SnifferFolder%\UploadScript.txt ftp -n -s:%SnifferFolder%\UploadScript.txt >UploadLog.txt echo FTP upload complete. Now moving uploaded logs into the OldLogs folder: if not exist %SnifferFolder%\OldLogs md %SnifferFolder%\OldLogs echo FTP upload complete. Now moving uploaded logs into the OldLogs folder: >>UploadLog.txt move %LicenseID%.log.20????????????.gz OldLogs >>UploadLog.txt date /t >>UploadLog.txt time /t >>UploadLog.txt :quit REM the name of the email that triggered this alias script is passed as a single parameter REM so if the parameter is blank, the script was run manually, but if it contains something REM then it's a temp filename used as a semaphore to run this, so delete the file. if not "%1" == "" del %1 >nul echo Done!
############################################################# This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
