Re: [Clamav-users] Updated unofficial-sigs.sh script available
Bill Landry wrote: > Dennis Peterson wrote the following on 9/25/2007 9:45 PM -0800: >> Also - if you do all your tests up front and discover you'll need to run >> multiple instances of perl you may find you can collect multiple code >> segments into a single execution of Perl and get everything in one pass. >> Do as much as you can in a single shot once you've committed to taking >> that shot. >> > > There is probably much room for code optimization, and as time permits > I'll look into doing that. Understand these are things to think about, but surely don't drop all you're doing and jump into it. What you have is perfectly workable. There's no absolute right way to do this stuff. Your enthusiasm for the project is clearly appreciated by many including me. Mentoring is a bad habit of old, old school Unixians and I stand guilty as charged :) dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
Dennis Peterson wrote the following on 9/25/2007 9:45 PM -0800: > Also - if you do all your tests up front and discover you'll need to run > multiple instances of perl you may find you can collect multiple code > segments into a single execution of Perl and get everything in one pass. > Do as much as you can in a single shot once you've committed to taking > that shot. > There is probably much room for code optimization, and as time permits I'll look into doing that. Thanks, Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
-- Dennis Peterson said the following on 9/25/07 7:00 PM:
> Jan-Pieter Cornet wrote:
>
>> So, TIMTOTDI squared (look ma', no perl!). This does the same as
>> date +%s too:
>>
>> echo|awk '{print systime()}'
>>
>
> But not in Solaris which is where the OP's original hack was born. You
> need gawk:
>
> echo|gawk '{print systime()}'
>
If you have the Solaris Software Companion bundle, you could always
install SFWcoreu and get GNU date. By default it installs into /opt/sfw.
Amos
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
Bill Landry wrote:
> Dennis Peterson wrote the following on 9/25/2007 8:06 PM -0800:
>> Bill Landry wrote:
>>
>>
>>> Okay, let's try this again. A new update has been posted that will first
>>> try
>>> "date +%s" and if that fails, then it will automatically fall back to a perl
>>> option. I didn't update the version number, just the version info:
>>>
>> You can rip out a lot of code (well, some code) if you just use the Perl
>> date method by default and forget the date +%s stuff entirely. You
>> already have a dependency on Perl so there's no point adding another
>> tool check to see if gnu date is present as it isn't needed. It's a KISS
>> thing.
>>
>
> Yeah, but if you check, you will see that perl is always used as a
> secondary solution, so perl may never be called when the script is run.
> Besides, there may be systems that do not have perl installed (unlikely,
> but possible). And I've noticed that shelling out to perl is very slow
> compared to the primary options:
>
> time echo PING | socat - /var/amavis/clamd.sock
> PONG
>
> real0m0.003s
> user0m0.001s
> sys 0m0.002s
> -
> time perl -MIO::Socket::UNIX -we '$s = IO::Socket::UNIX->new(shift);
> $s->print("PING"); print $s->getline; $s->close' /var/amavis/clamd.sock
> 2> /dev/null
> PONG
>
> real0m0.040s
> user0m0.035s
> sys 0m0.006s
Try pgrep clamd (also not installed on all systems). It will return the
PID if it is running. pgrep won't tell you if it is unresponsive, but if
it is running and unresponsive you have bigger problems. Again, though,
this isn't significant to any reasonable system given the duty cycle.
Now if we were to explore the code simply as an exercise in code
efficiency then run time against the entire script to see what
percentage of time can be made up by making different choices as what
tools you use.
>
> If I were proficient in perl, I would have rather written the entire
> script in perl (maybe time to break open my perl book). Anyway,
> hopefully it is generic enough now, with the fall-back options, that it
> will not need much more editing.
In terms of efficiencies though you could make some simple changes - In
this section of code you call date and perl twice. Once would do it:
if [ `date +%s` -gt 0 2> /dev/null ]
then
current_time=`date +%s`
else
if [ `perl -le print+time 2> /dev/null` ]
then
current_time=`perl -le print+time`
fi
fi
Here's an alternate as a stand-alone shells script and again, there's
more than one way to do this.
#!/bin/sh
# get current time using gnu date
current_time=`date +%s 2>/dev/null`
# is it a realistic result?
if [ $current_time -lt "11" ]
# if not then fall back to Perl
then
current_time=`perl -le print+time 2> /dev/null`
if [ -z $current_time ]
then
echo "no time tool available"
exit 1 # Gratuitous exit on error
fi
fi
echo "time: $current_time"
# end
Also - if you do all your tests up front and discover you'll need to run
multiple instances of perl you may find you can collect multiple code
segments into a single execution of Perl and get everything in one pass.
Do as much as you can in a single shot once you've committed to taking
that shot.
dp
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
Dennis Peterson wrote the following on 9/25/2007 8:06 PM -0800:
> Bill Landry wrote:
>
>
>> Okay, let's try this again. A new update has been posted that will first try
>> "date +%s" and if that fails, then it will automatically fall back to a perl
>> option. I didn't update the version number, just the version info:
>>
>
> You can rip out a lot of code (well, some code) if you just use the Perl
> date method by default and forget the date +%s stuff entirely. You
> already have a dependency on Perl so there's no point adding another
> tool check to see if gnu date is present as it isn't needed. It's a KISS
> thing.
>
Yeah, but if you check, you will see that perl is always used as a
secondary solution, so perl may never be called when the script is run.
Besides, there may be systems that do not have perl installed (unlikely,
but possible). And I've noticed that shelling out to perl is very slow
compared to the primary options:
time echo PING | socat - /var/amavis/clamd.sock
PONG
real0m0.003s
user0m0.001s
sys 0m0.002s
-
time perl -MIO::Socket::UNIX -we '$s = IO::Socket::UNIX->new(shift);
$s->print("PING"); print $s->getline; $s->close' /var/amavis/clamd.sock
2> /dev/null
PONG
real0m0.040s
user0m0.035s
sys 0m0.006s
If I were proficient in perl, I would have rather written the entire
script in perl (maybe time to break open my perl book). Anyway,
hopefully it is generic enough now, with the fall-back options, that it
will not need much more editing.
> And nice script, too.
>
Thanks!
Bill
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
Kyle Lanclos wrote: > Dennis wrote: >> You can rip out a lot of code (well, some code) if you just use the Perl >> date method by default and forget the date +%s stuff entirely. > > Your mileage may vary. > > $ time perl -le print+time > > real0m0.002s > > $ time date +%s > > real0m0.001s > > (Those results were surprisingly consistent on my Linux box.) For a script that runs a few times a day it is not significant. The other optimization point to consider is development and maintenance time. Building and validating tests for conditions in arbitrary environments is difficult and prone to failure. If you can reduce the number of external tool requirements, especially platform-specific tools, it is probably a good thing. If you can easily design in platform independence with a single supertool that you know you already need then go with it. The entire script can be written in any of sh, bash, ksh, perl, ruby, php, python, or REXX. I agree it is unnecessarily cross-bred, but again, it's a very low-level utility that runs seldom and so won't benefit by a lot of optimizations and the time required to create them can be better spent with the family. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
Dennis wrote: > You can rip out a lot of code (well, some code) if you just use the Perl > date method by default and forget the date +%s stuff entirely. Your mileage may vary. $ time perl -le print+time real0m0.002s $ time date +%s real0m0.001s (Those results were surprisingly consistent on my Linux box.) My version of "keep it simple" in this situation is that if you have to invoke another language interpreter to provide a required feature, the base script itself should probably be written in a different language. If the base script will remain a shell script, you're better off invoking smaller, less-expensive programs whenever possible. This cost savings diminishes very quickly if your script has to make a lot of external program calls, where with a more fully featured language you might have handled all those external functions in-line. If it matters enough, of course, you try both, and profile the results to see which is more efficient under load. I'm probably preaching to the choir on this, and for that I apologize. It's already been a long week. --Kyle ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
Bill Landry wrote: > > Okay, let's try this again. A new update has been posted that will first try > "date +%s" and if that fails, then it will automatically fall back to a perl > option. I didn't update the version number, just the version info: You can rip out a lot of code (well, some code) if you just use the Perl date method by default and forget the date +%s stuff entirely. You already have a dependency on Perl so there's no point adding another tool check to see if gnu date is present as it isn't needed. It's a KISS thing. And nice script, too. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
Bill Landry wrote: > After a discussion on the clamav-users list yesterday of an issue a > couple of script users were experiencing with write access to the > temporary directory, I made a change to the script to overcome this > issue. There are also a couple of other script modifications to make > the integer expression handling more consistent throughout the script. > > There is no new functionality, and thus no need to upgrade unless you > were experiencing the following error when running the script: > > ERROR: Can't write to temporary directory > > I have gotten confirmation from a couple of users that were experiencing > this error that the latest version of the script resolves the issue. As > usual, the updated script can be downloaded from: > > ftp://ftp.inetmsg.com/pub/unofficial-sigs.sh > > It should be available from the SaneSecuirty "Usage" page, as well, once > Steve has gotten a chance to upload it. Okay, let's try this again. A new update has been posted that will first try "date +%s" and if that fails, then it will automatically fall back to a perl option. I didn't update the version number, just the version info: # Version 1.7c (updated 9/25/07 - Thanks to Dennis Peterson and Jan-Perter # Cornet for the perl solution for calculating seconds since epoch) # - Added timeout values to curl and rsync downloads in order to prevent # the script from hanging on a non-responsive signature host site. # - Apparently Solaris does not support "date +%s", which calculates # the number of seconds since epoch. This date function is used to # calculate when to do MBL downloads. A perl solution has been added # as a fall-back option. If "date +%s" is not supported and perl is # not found on the system, the script will report a warning message # and skip MBL updates, but the script will continue processing other # third-party signature updates. ftp://ftp.inetmsg.com/pub/unofficial-sigs.sh Thanks to Dennis Peterson and Jan-Perter Cornet for the perl (and awk/gwak) lessons, as well! I'm learning that it's never easy to make fully portable scripts, but as people continue to report issues, I'll continue looking for ways to do so. Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
Jan-Pieter Cornet wrote:
>
> So, TIMTOTDI squared (look ma', no perl!). This does the same as
> date +%s too:
>
> echo|awk '{print systime()}'
>
But not in Solaris which is where the OP's original hack was born. You
need gawk:
echo|gawk '{print systime()}'
My favorite absurd method in Solaris is:
truss date 2>&1 |awk '/time/ {print $NF}'
dp
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
On Tue, Sep 25, 2007 at 04:17:41PM -0700, Dennis Peterson wrote:
> >>> Epoch time:
> >
> > Golfed:
> >
> > perl -le print+time
>
> It wouldn't be Perl if there were only one way to do it ;)
But it's not necessarily good to include all possible ways. I mean,
this works too:
perl -ple '$_=$^T'<<<1
But that's not exactly self-documenting. Plus it's a bitch to embed in
a script because of all the quote characters, and finally, it uses a
bashism to provide a single line on stdin :) (if you don't use bash,
add "echo|" in front and remove the "<<<1").
I optimised for keystrokes, and less complicated characters that
possibly need quoting.
You could also optimize for the non-existance of perl, and use awk,
which might be even more uniformly available (eg, modern FreeBSD
comes without perl if you do a bare bones install. Then again,
FreeBSD date groks %s).
So, TIMTOTDI squared (look ma', no perl!). This does the same as
date +%s too:
echo|awk '{print systime()}'
--
Jan-Pieter Cornet <[EMAIL PROTECTED]>
!! Disclamer: The addressee of this email is not the intended recipient. !!
!! This is only a test of the echelon and data retention systems. Please !!
!! archive this message indefinitely to allow verification of the logs. !!
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
Jan-Pieter Cornet wrote: > On Tue, Sep 25, 2007 at 03:17:35PM -0700, Bill Landry wrote: >>> Epoch time: >>> perl -e 'print time() . "\n";' > > Golfed: > > perl -le print+time > > You can even leave the -l switch if used in ``, because the trailing > newline doesn't matter there. > It wouldn't be Perl if there were only one way to do it ;) dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
On Tue, Sep 25, 2007 at 03:17:35PM -0700, Bill Landry wrote: > > Epoch time: > > perl -e 'print time() . "\n";' Golfed: perl -le print+time You can even leave the -l switch if used in ``, because the trailing newline doesn't matter there. -- Jan-Pieter Cornet <[EMAIL PROTECTED]> !! Disclamer: The addressee of this email is not the intended recipient. !! !! This is only a test of the echelon and data retention systems. Please !! !! archive this message indefinitely to allow verification of the logs. !! ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
Dennis Peterson wrote: > Bill Landry wrote: >> Bill Landry wrote: >>> After a discussion on the clamav-users list yesterday of an issue a >>> couple of script users were experiencing with write access to the >>> temporary directory, I made a change to the script to overcome this >>> issue. There are also a couple of other script modifications to make >>> the integer expression handling more consistent throughout the script. >>> >>> There is no new functionality, and thus no need to upgrade unless you >>> were experiencing the following error when running the script: >>> >>> ERROR: Can't write to temporary directory >>> >>> I have gotten confirmation from a couple of users that were experiencing >>> this error that the latest version of the script resolves the issue. As >>> usual, the updated script can be downloaded from: >>> >>> ftp://ftp.inetmsg.com/pub/unofficial-sigs.sh >>> >>> It should be available from the SaneSecuirty "Usage" page, as well, once >>> Steve has gotten a chance to upload it. >> Minor updates to overcome issues reported by a few users: >> >> # Version 1.7c (updated 9/25/07) >> # - Added timeout values to curl and rsync downloads in order to prevent >> # the script from hanging on a non-responsive signature host site. >> # - Apparently Solaris does not support "date +%s", which calculates >> # the number of seconds since epoch. This date function is used to >> # calculate when to do MBL downloads. If Solaris users want to use >> # the MBL signatures, the recommendation is to install GNU date. >> > > Epoch time: > > perl -e 'print time() . "\n";' LOL, I knew I should have asked you first before posting about the update (being the master Solaris and perl user that you are ;-)! I'll add this to the script and post again when it's ready for download... Thanks Dennis! Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
Bill Landry wrote: > Bill Landry wrote: >> After a discussion on the clamav-users list yesterday of an issue a >> couple of script users were experiencing with write access to the >> temporary directory, I made a change to the script to overcome this >> issue. There are also a couple of other script modifications to make >> the integer expression handling more consistent throughout the script. >> >> There is no new functionality, and thus no need to upgrade unless you >> were experiencing the following error when running the script: >> >> ERROR: Can't write to temporary directory >> >> I have gotten confirmation from a couple of users that were experiencing >> this error that the latest version of the script resolves the issue. As >> usual, the updated script can be downloaded from: >> >> ftp://ftp.inetmsg.com/pub/unofficial-sigs.sh >> >> It should be available from the SaneSecuirty "Usage" page, as well, once >> Steve has gotten a chance to upload it. > > Minor updates to overcome issues reported by a few users: > > # Version 1.7c (updated 9/25/07) > # - Added timeout values to curl and rsync downloads in order to prevent > # the script from hanging on a non-responsive signature host site. > # - Apparently Solaris does not support "date +%s", which calculates > # the number of seconds since epoch. This date function is used to > # calculate when to do MBL downloads. If Solaris users want to use > # the MBL signatures, the recommendation is to install GNU date. > Epoch time: perl -e 'print time() . "\n";' dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Updated unofficial-sigs.sh script available
Bill Landry wrote: > After a discussion on the clamav-users list yesterday of an issue a > couple of script users were experiencing with write access to the > temporary directory, I made a change to the script to overcome this > issue. There are also a couple of other script modifications to make > the integer expression handling more consistent throughout the script. > > There is no new functionality, and thus no need to upgrade unless you > were experiencing the following error when running the script: > > ERROR: Can't write to temporary directory > > I have gotten confirmation from a couple of users that were experiencing > this error that the latest version of the script resolves the issue. As > usual, the updated script can be downloaded from: > > ftp://ftp.inetmsg.com/pub/unofficial-sigs.sh > > It should be available from the SaneSecuirty "Usage" page, as well, once > Steve has gotten a chance to upload it. Minor updates to overcome issues reported by a few users: # Version 1.7c (updated 9/25/07) # - Added timeout values to curl and rsync downloads in order to prevent # the script from hanging on a non-responsive signature host site. # - Apparently Solaris does not support "date +%s", which calculates # the number of seconds since epoch. This date function is used to # calculate when to do MBL downloads. If Solaris users want to use # the MBL signatures, the recommendation is to install GNU date. See download URL above. Steve, please post to the SaneSecurity "Usage" site when you get a change. Thanks, Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Is anyone using ClamAV on Redhat Linux
DBS Labs wrote: > Rob MacGregor, > > I am aware that your crystal ball is broken, because you could not see my > original post either, which I have included again for your benefit. This was > my second post because no one responded to my first. > You didn't mention where you installed from or how. I can tell you that the dag.wieers repository has a fantastic rpm for all the clam products. Out of the box, they simply work. I do occasionally get an error like below, but it is rare and I'm assuming caused by overloads to the db servers. For Redhat or its variants, I'd suggest using dag. The install is almost too easy and the updates come just long enough after a new release to avoid most of the bugs. This is a very active area in the dag repository. John Hinton > > I just installed ClamAV-0.91.2 on a Redhat EL 4 server for testing. There > were no errors during the install. I then edited the clamd and freshclam > configuration files because our system is behind a firewall and I have to use > a proxy server. When I run freshclam I get these error messages -ClamAV > update process started at Wed Sep 12 09:38:19 2007Connecting via > firewall.commain.cvd is up to date (version: 44, sigs: 133163, f-level: 20, > builder: sven)Connecting via firewall.comERROR: getfile: Unknown response > from remote server (IP: 199.169.119.19)ERROR: getpatch: Can't download > daily-4016.cdiff from db.us.clamav.netERROR: getfile: Unknown response from > remote server (IP: 199.169.119.19)ERROR: getpatch: Can't download > daily-4016.cdiff from db.us.clamav.netERROR: getfile: Unknown response from > remote server (IP: 199.169.119.19)ERROR: getpatch: Can't download > daily-4016.cdiff from db.us.clamav.netWARNING: Incremental update failed, > trying to download daily.cvdERROR: getfile: Unknown response from remote > server (IP: 199.169.119.19)ERROR: Can't download daily.cvd from > db.us.clamav.netTrying again in 5 secs... Our firewall does use agent > strings, which I have not configured. Should it be changed? > _ > Gear up for Halo® 3 with free downloads and an exclusive offer. It’s our way > of saying thanks for using Windows Live™. > http://gethalo3gear.com?ocid=SeptemberWLHalo3_WLHMTxt_2 > ___ > Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net > http://lurker.clamav.net/list/clamav-users.html > > > !DSPAM:46f8fd56118001280715606! > > ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] clamd problem
Oliver, Am/On Mon, 24 Sep 2007 13:21:50 +0200 schrieb/wrote Oliver Schwarz: >look into the config file of clamav. there's an option which lets it >fix stale sockets. thanks a lot, that did the trick :-) Thanks and all the best Matthias ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Is anyone using ClamAV on Redhat Linux
Rob MacGregor, I am aware that your crystal ball is broken, because you could not see my original post either, which I have included again for your benefit. This was my second post because no one responded to my first. I just installed ClamAV-0.91.2 on a Redhat EL 4 server for testing. There were no errors during the install. I then edited the clamd and freshclam configuration files because our system is behind a firewall and I have to use a proxy server. When I run freshclam I get these error messages -ClamAV update process started at Wed Sep 12 09:38:19 2007Connecting via firewall.commain.cvd is up to date (version: 44, sigs: 133163, f-level: 20, builder: sven)Connecting via firewall.comERROR: getfile: Unknown response from remote server (IP: 199.169.119.19)ERROR: getpatch: Can't download daily-4016.cdiff from db.us.clamav.netERROR: getfile: Unknown response from remote server (IP: 199.169.119.19)ERROR: getpatch: Can't download daily-4016.cdiff from db.us.clamav.netERROR: getfile: Unknown response from remote server (IP: 199.169.119.19)ERROR: getpatch: Can't download daily-4016.cdiff from db.us.clamav.netWARNING: Incremental update failed, trying to download daily.cvdERROR: getfile: Unknown response from remote server (IP: 199.169.119.19)ERROR: Can't download daily.cvd from db.us.clamav.netTrying again in 5 secs... Our firewall does use agent strings, which I have not configured. Should it be changed? _ Gear up for Halo® 3 with free downloads and an exclusive offer. It’s our way of saying thanks for using Windows Live™. http://gethalo3gear.com?ocid=SeptemberWLHalo3_WLHMTxt_2 ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
