Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-03 Thread Dennis Peterson

On 5/3/10 9:43 AM, Bill Landry wrote:



If you are seeing three copies of each unofficial database, then you have
a problem somewhere.

This topic is really OT for the ClamAV mailing list.


My net nanny buzzer went off.

dp
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-03 Thread Dennis Peterson

On 5/3/10 8:45 AM, Simon Hobson wrote:



It's actually more efficient than that !
It uses something similar to a rolling checksum to find throughout the
file. So in principal, you can add a short bit to the front of a large
file, or even chop a file up into chunks and rearrange them, and it will
still only transfer the changes.


It does only send changed parts, but it takes longer to find the differences if 
no effort is made to leave the changed bits at the end of the file.


dp
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-03 Thread Dennis Peterson

On 5/3/10 8:45 AM, Alex wrote:

Hi,

Dennis Peterson wrote:

Rsync is able to transfer only the differences between two files provided a 
version of the file being
transferred exists on the source and the destination. In addition, rsync will 
not transfer anything if it
determines there are no changes between the two files. This saves bandwidth for 
the mirrors. This


Yes, of course, this is the whole benefit of rsync.


The above is the reason for having a working area for rsync.


Yeah? So you always have effectively three copies of your data around?
I've never experienced a problem with just a local and remote copy of
data when using rsync, and it forms the basis of my backup system
(using link-dest, too).

Thanks,
Alex


Why three? You only need an unmodified version (for rsync) and a working version 
(for clamd). The point is not to simply copy a file - the point is to copy only 
the parts of the file that have changed since the most recent download. Rsync 
can do either but bandwidth is not free so I prefer to do it the most efficient 
way possible.


dp
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-03 Thread Bill Landry
On Mon, May 3, 2010 8:45 am, Alex wrote:
> Hi,
>
> Dennis Peterson wrote:
>> Rsync is able to transfer only the differences between two files
>> provided a version of the file being
>> transferred exists on the source and the destination. In addition, rsync
>> will not transfer anything if it
>> determines there are no changes between the two files. This saves
>> bandwidth for the mirrors. This
>
> Yes, of course, this is the whole benefit of rsync.
>
>> The above is the reason for having a working area for rsync.
>
> Yeah? So you always have effectively three copies of your data around?
> I've never experienced a problem with just a local and remote copy of
> data when using rsync, and it forms the basis of my backup system
> (using link-dest, too).

If you are seeing three copies of each unofficial database, then you have
a problem somewhere.

This topic is really OT for the ClamAV mailing list.  If you wish to
persist in scrutinizing the inner workings of the clamav-unofficial-sigs
script, then either review the script itself with your favorite text
editor or please move this discussion to the Sanesecurity users list,
where it would be more appropriately discussed.

Bill

___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-03 Thread Simon Hobson

Chris Meadors wrote:


Rsync treats all files as binary.  When finding changes it splits a file
into blocks, computes a checksum for each block and performs a
comparison between the sending and receiving side.  Then it only sends
the blocks which have changed.

When dealing with a text file which has been appended to, like a log,
all the initial blocks are the same.  But if the file is sorted, it's
possible only a few additional lines will disrupt most every block by
changing the start offsets through out the entire file.


It's actually more efficient than that !
It uses something similar to a rolling checksum to find throughout 
the file. So in principal, you can add a short bit to the front of a 
large file, or even chop a file up into chunks and rearrange them, 
and it will still only transfer the changes.


Andrew Tridgell's research paper is available at 
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.123.1530&rep=rep1&type=pdf

rsync is covered from section 3 onwards.


--
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-03 Thread Alex
Hi,

Dennis Peterson wrote:
> Rsync is able to transfer only the differences between two files provided a 
> version of the file being
> transferred exists on the source and the destination. In addition, rsync will 
> not transfer anything if it
> determines there are no changes between the two files. This saves bandwidth 
> for the mirrors. This

Yes, of course, this is the whole benefit of rsync.

> The above is the reason for having a working area for rsync.

Yeah? So you always have effectively three copies of your data around?
I've never experienced a problem with just a local and remote copy of
data when using rsync, and it forms the basis of my backup system
(using link-dest, too).

Thanks,
Alex









>
> After the script is run, each database that has been updated is GPG
> signature tested, then ClamAV integrity tested, and then rsynced into the
> the ClamAV production directory.  You do not want to place any database into
> the production directory until it have been tested, thus the creation and
> use of the working directories (/usr/unofficial-dbs/*) by the script.
>
> The script stops and reports/logs any failure along the way so that a
> corrupted or compromised database cannot end up in production use.
>
>> Is there support that I'm somehow missing for alerting an
>> administrator that there was a problem when it actually happens, and
>> not through just a daily report?
>
> The only error reporting the script does is via email after each script run
> and logging to a log file, depending on how the user has configured the
> script to run.
>
>> How can I verify that clamd is actually using the full set of
>> databases I've downloaded?
>
> Any database that shows up in the ClamAV production directory is used by
> ClamAV/ClamD - ls /path/to/clamav will show you what database are being
> used.
>
> Bill
> ___
> Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
> http://www.clamav.net/support/ml
>
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-03 Thread Chris Meadors
On Mon, 2010-05-03 at 05:53 -0700, Jim Preston wrote:
> Dennis Peterson wrote:
> >
> > Rsync is able to transfer only the differences between two files 
> > provided a version of the file being transferred exists on the source 
> > and the destination. In addition, rsync will not transfer anything if 
> > it determines there are no changes between the two files. 
> 
> Is this true for binary files as well? or just text files?

Rsync treats all files as binary.  When finding changes it splits a file
into blocks, computes a checksum for each block and performs a
comparison between the sending and receiving side.  Then it only sends
the blocks which have changed.

When dealing with a text file which has been appended to, like a log,
all the initial blocks are the same.  But if the file is sorted, it's
possible only a few additional lines will disrupt most every block by
changing the start offsets through out the entire file.

-- 
Chris

___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-03 Thread Jim Preston

Dennis Peterson wrote:


Rsync is able to transfer only the differences between two files 
provided a version of the file being transferred exists on the source 
and the destination. In addition, rsync will not transfer anything if 
it determines there are no changes between the two files. 

Dennis,

Is this true for binary files as well? or just text files?

Thanks, Jim
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-02 Thread Dennis Peterson

On 5/2/10 9:59 AM, Alex wrote:

Hi,


Why are some of the databases duplicated in the clamav root dir and
also in the unofficial-dbs/ss-dbs directory, such as
winnow_malware.hdb?


The rsync protocol only downloads the changes between the local and remote
files, so the local file much be available comparison.  The
unofficial/ss-dbs is also the working directory where the database's GPG
signature and integrity testing is done before rsyncing the files into the
ClamAV production directory.


By this time you must think I'm dumb or completely new at this. I'd
like to think I'm just more inquisitive and going beyond just the home
use. Time for some more coffee for me :-)

It sounds like the real reason for is duplicates is for the integrity
check, because rsync doesn't otherwise need two local copies. Also for
moving them all into place into production at once?


Rsync is able to transfer only the differences between two files provided a 
version of the file being transferred exists on the source and the destination. 
In addition, rsync will not transfer anything if it determines there are no 
changes between the two files. This saves bandwidth for the mirrors. This 
transfer economy requires the client maintain unmodified copies of the files in 
the target directory so rsync can make these comparisons. (A bit more to know: 
This also works best if the files are unsorted between versions. For a trivial 
example, a syslog file that has grown over time. Rsync will skip over the parts 
found in both files very quickly and transfer only those parts found in newer 
version. If the files have been sorted rsync has to examine and reconstruct the 
entire file.)


The above is the reason for having a working area for rsync. A reason for 
keeping a separate area for clamd is because clamd will die if a corrupt file is 
found in the clamd working area. A corrupt file could include a file that is 
being copied into the working area but is not yet complete (non-atomic 
copy/move). As clamd does check the working area from time to time the chance 
exists that this scan can occur at a time a file is being placed in the working 
area.


It is a good practice to minimize file activity in the clamd working area. As 
another matter of good practice I only use rsync to place signatures in the 
clamd working area because rsync writes to a hidden file until the transfer is 
complete at which time that file is renamed. As such it appears suddenly and so 
does not cause clamd any problems. This works for local file systems as well as 
NAS/NFS/FUSE/CIFS file systems.


dp
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-02 Thread Bill Landry

On 5/2/2010 9:59 AM, Alex wrote:

Hi,


Why are some of the databases duplicated in the clamav root dir and
also in the unofficial-dbs/ss-dbs directory, such as
winnow_malware.hdb?


The rsync protocol only downloads the changes between the local and remote
files, so the local file much be available comparison.  The
unofficial/ss-dbs is also the working directory where the database's GPG
signature and integrity testing is done before rsyncing the files into the
ClamAV production directory.


By this time you must think I'm dumb or completely new at this. I'd
like to think I'm just more inquisitive and going beyond just the home
use. Time for some more coffee for me :-)

It sounds like the real reason for is duplicates is for the integrity
check, because rsync doesn't otherwise need two local copies. Also for
moving them all into place into production at once?


Not just integrity testing, but first GPG signature tested in order to 
confirm that the files have not been compromised in any way.


After the script is run, each database that has been updated is GPG 
signature tested, then ClamAV integrity tested, and then rsynced into 
the the ClamAV production directory.  You do not want to place any 
database into the production directory until it have been tested, thus 
the creation and use of the working directories (/usr/unofficial-dbs/*) 
by the script.


The script stops and reports/logs any failure along the way so that a 
corrupted or compromised database cannot end up in production use.



Is there support that I'm somehow missing for alerting an
administrator that there was a problem when it actually happens, and
not through just a daily report?


The only error reporting the script does is via email after each script 
run and logging to a log file, depending on how the user has configured 
the script to run.



How can I verify that clamd is actually using the full set of
databases I've downloaded?


Any database that shows up in the ClamAV production directory is used by 
ClamAV/ClamD - ls /path/to/clamav will show you what database are being 
used.


Bill
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-02 Thread Alex
Hi,

>> Why are some of the databases duplicated in the clamav root dir and
>> also in the unofficial-dbs/ss-dbs directory, such as
>> winnow_malware.hdb?
>
> The rsync protocol only downloads the changes between the local and remote
> files, so the local file much be available comparison.  The
> unofficial/ss-dbs is also the working directory where the database's GPG
> signature and integrity testing is done before rsyncing the files into the
> ClamAV production directory.

By this time you must think I'm dumb or completely new at this. I'd
like to think I'm just more inquisitive and going beyond just the home
use. Time for some more coffee for me :-)

It sounds like the real reason for is duplicates is for the integrity
check, because rsync doesn't otherwise need two local copies. Also for
moving them all into place into production at once?

Is there support that I'm somehow missing for alerting an
administrator that there was a problem when it actually happens, and
not through just a daily report?

How can I verify that clamd is actually using the full set of
databases I've downloaded?

Thanks again,
Alex
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-01 Thread Bill Landry

On 5/1/2010 8:19 PM, Alex wrote:

Hi,


These are *NOT* Sanesecurity distributed scripts, so please do not add them
to the Sanesecurity section of the script's config file.  MBL already has a


Okay, got it. It wasn't clear to me that it was an "MBL" db, and the
reference in the sanesecurity database section also includes info on
the other databases.


Only the databases that the Sanesecurity site says are "distributed by 
Sanesecurity" can be configured to be downloaded in the Sanesecurity 
section of the config file and then downloaded from the Sanesecurity 
rsync servers.  The others signature databases are hosted and downloaded 
from other sites, thus the separate sections defined in the script's 
config file.



Why are some of the databases duplicated in the clamav root dir and
also in the unofficial-dbs/ss-dbs directory, such as
winnow_malware.hdb?


The rsync protocol only downloads the changes between the local and 
remote files, so the local file much be available comparison.  The 
unofficial/ss-dbs is also the working directory where the database's GPG 
signature and integrity testing is done before rsyncing the files into 
the ClamAV production directory.


Bill
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-01 Thread Alex
Hi,

> These are *NOT* Sanesecurity distributed scripts, so please do not add them
> to the Sanesecurity section of the script's config file.  MBL already has a

Okay, got it. It wasn't clear to me that it was an "MBL" db, and the
reference in the sanesecurity database section also includes info on
the other databases.

Why are some of the databases duplicated in the clamav root dir and
also in the unofficial-dbs/ss-dbs directory, such as
winnow_malware.hdb?

Thanks,
Alex
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-01 Thread Bill Landry

On 5/1/2010 1:01 PM, Bill Landry wrote:

On 5/1/2010 12:38 PM, Alex wrote:

Hi,


sent 34 bytes received 1932 bytes 786.40 bytes/sec
total size is 27032205 speedup is 13749.85
Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying
next mirror site...

Looks to me like the rsync succeeded, since it received the filelist
and actually received data. Could it instead be saying that it failed
to find an update, and not that it failed to connect?


The script only reports failed to connect if it actually failed to
connect,
not if there were no updates available.


I found the problem. Stepping through the script with execution, I
find this:

rsync --no-motd
--files-from=/var/clamav/unofficial-dbs/configs/ss-include-dbs.txt
-ctuz --contimeout=30 --timeout=30 --stats
rsync://84.19.167.43/sanesecurity /var/clamav/unofficial-dbs/ss-dbs
rsync: link_stat "/MSRBL-SPAM.ndb" (in sanesecurity) failed: No such
file or directory (2)
rsync: link_stat "/MSRBL-SPAM.ndb.sig" (in sanesecurity) failed: No
such file or directory (2)
rsync: link_stat "/MSRBL-Images.hdb" (in sanesecurity) failed: No such
file or directory (2)
rsync: link_stat "/MSRBL-Images.hdb.sig" (in sanesecurity) failed: No
such file or directory (2)
rsync: link_stat "/mbl.ndb" (in sanesecurity) failed: No such file or
directory (2)
rsync: link_stat "/mbl.ndb.sig" (in sanesecurity) failed: No such file
or directory (2)


These are *NOT* Sanesecurity distributed scripts, so please do not add


Oops, meant to say that "These are "NOT" Sanesecurity distributed 
signature databases", not "scripts".



them to the Sanesecurity section of the script's config file. MBL
already has a section in the config file, so no need to add it to the
Sanesecurity section. MSRBL has not been update since last July, so I
purposefully removed it from the script with the last update - thus, you
should not be adding these to the script anyway.


Number of files: 40
Number of files transferred: 0
Total file size: 26677157 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 1314
File list generation time: 0.232 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 34
Total bytes received: 1928

sent 34 bytes received 1928 bytes 784.80 bytes/sec
total size is 26677157 speedup is 13596.92
rsync error: some files/attrs were not transferred (see previous
errors) (code 23) at main.c(1508) [generator=3.0.7]

However, the script reports that the connection failed, which is
somewhat of a red herring. Would it be helpful to not suppress the
rsync output when it fails with something other than a simple
"connection failed" message?


The script is working as intended. User should not add signature
database that do not exist on Sanesecurity rsync servers.


These databases (that could not be retrieved) were collected from the
sanesecurity database page. Aren't they supposed to be used? I'd think
someone else would have found this problem?


No one else has reported this problem because no one else is trying to
download signature database that do not exist on Sanesecurity rsync
servers.


On a similar note, this script can replace freshclam, correct?


No it cannot replace freshclam, as the clamav-unofficial-sigs script
only update unofficial 3rd party signature databases (as the script name
itself indicates), not official ClamAV signature databases.

Bill
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-01 Thread Bill Landry

On 5/1/2010 12:38 PM, Alex wrote:

Hi,


sent 34 bytes  received 1932 bytes  786.40 bytes/sec
total size is 27032205  speedup is 13749.85
Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying
next mirror site...

Looks to me like the rsync succeeded, since it received the filelist
and actually received data. Could it instead be saying that it failed
to find an update, and not that it failed to connect?


The script only reports failed to connect if it actually failed to connect,
not if there were no updates available.


I found the problem. Stepping through the script with execution, I find this:

rsync --no-motd
--files-from=/var/clamav/unofficial-dbs/configs/ss-include-dbs.txt
-ctuz --contimeout=30 --timeout=30 --stats
rsync://84.19.167.43/sanesecurity /var/clamav/unofficial-dbs/ss-dbs
rsync: link_stat "/MSRBL-SPAM.ndb" (in sanesecurity) failed: No such
file or directory (2)
rsync: link_stat "/MSRBL-SPAM.ndb.sig" (in sanesecurity) failed: No
such file or directory (2)
rsync: link_stat "/MSRBL-Images.hdb" (in sanesecurity) failed: No such
file or directory (2)
rsync: link_stat "/MSRBL-Images.hdb.sig" (in sanesecurity) failed: No
such file or directory (2)
rsync: link_stat "/mbl.ndb" (in sanesecurity) failed: No such file or
directory (2)
rsync: link_stat "/mbl.ndb.sig" (in sanesecurity) failed: No such file
or directory (2)


These are *NOT* Sanesecurity distributed scripts, so please do not add 
them to the Sanesecurity section of the script's config file.  MBL 
already has a section in the config file, so no need to add it to the 
Sanesecurity section.  MSRBL has not been update since last July, so I 
purposefully removed it from the script with the last update - thus, you 
should not be adding these to the script anyway.



Number of files: 40
Number of files transferred: 0
Total file size: 26677157 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 1314
File list generation time: 0.232 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 34
Total bytes received: 1928

sent 34 bytes  received 1928 bytes  784.80 bytes/sec
total size is 26677157  speedup is 13596.92
rsync error: some files/attrs were not transferred (see previous
errors) (code 23) at main.c(1508) [generator=3.0.7]

However, the script reports that the connection failed, which is
somewhat of a red herring. Would it be helpful to not suppress the
rsync output when it fails with something other than a simple
"connection failed" message?


The script is working as intended.  User should not add signature 
database that do not exist on Sanesecurity rsync servers.



These databases (that could not be retrieved) were collected from the
sanesecurity database page. Aren't they supposed to be used? I'd think
someone else would have found this problem?


No one else has reported this problem because no one else is trying to 
download signature database that do not exist on Sanesecurity rsync servers.



On a similar note, this script can replace freshclam, correct?


No it cannot replace freshclam, as the clamav-unofficial-sigs script 
only update unofficial 3rd party signature databases (as the script name 
itself indicates), not official ClamAV signature databases.


Bill
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-05-01 Thread Alex
Hi,

>> sent 34 bytes  received 1932 bytes  786.40 bytes/sec
>> total size is 27032205  speedup is 13749.85
>> Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying
>> next mirror site...
>>
>> Looks to me like the rsync succeeded, since it received the filelist
>> and actually received data. Could it instead be saying that it failed
>> to find an update, and not that it failed to connect?
>
> The script only reports failed to connect if it actually failed to connect,
> not if there were no updates available.

I found the problem. Stepping through the script with execution, I find this:

rsync --no-motd
--files-from=/var/clamav/unofficial-dbs/configs/ss-include-dbs.txt
-ctuz --contimeout=30 --timeout=30 --stats
rsync://84.19.167.43/sanesecurity /var/clamav/unofficial-dbs/ss-dbs
rsync: link_stat "/MSRBL-SPAM.ndb" (in sanesecurity) failed: No such
file or directory (2)
rsync: link_stat "/MSRBL-SPAM.ndb.sig" (in sanesecurity) failed: No
such file or directory (2)
rsync: link_stat "/MSRBL-Images.hdb" (in sanesecurity) failed: No such
file or directory (2)
rsync: link_stat "/MSRBL-Images.hdb.sig" (in sanesecurity) failed: No
such file or directory (2)
rsync: link_stat "/mbl.ndb" (in sanesecurity) failed: No such file or
directory (2)
rsync: link_stat "/mbl.ndb.sig" (in sanesecurity) failed: No such file
or directory (2)

Number of files: 40
Number of files transferred: 0
Total file size: 26677157 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 1314
File list generation time: 0.232 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 34
Total bytes received: 1928

sent 34 bytes  received 1928 bytes  784.80 bytes/sec
total size is 26677157  speedup is 13596.92
rsync error: some files/attrs were not transferred (see previous
errors) (code 23) at main.c(1508) [generator=3.0.7]

However, the script reports that the connection failed, which is
somewhat of a red herring. Would it be helpful to not suppress the
rsync output when it fails with something other than a simple
"connection failed" message?

These databases (that could not be retrieved) were collected from the
sanesecurity database page. Aren't they supposed to be used? I'd think
someone else would have found this problem?

On a similar note, this script can replace freshclam, correct?

Thanks,
Alex
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-29 Thread Steve Basford
>> If you still have a copy of the headers & body, could you send me a
>> sample:
>
> Attachment sent.

Thanks for the sample Alex.

It's already being detected as:

Sanesecurity.Malware.8830.UNOFFICIAL

So, you should already be covered :)

Cheers,

Steve
Sanesecurity

___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-29 Thread Alex
>> I meant that the other day there was a URL in the body of an email
>> that passed through as ham when in fact it ended in 'ecard.exe' and,
>> should the recipient download it, would be shown to be a trojan.
>> Doesn't clamav block stuff like this, I thought?
>
> If you still have a copy of the headers & body, could you send me a sample:

Attachment sent.

Thanks,
Alex
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-28 Thread Steve Basford
> I meant that the other day there was a URL in the body of an email
> that passed through as ham when in fact it ended in 'ecard.exe' and,
> should the recipient download it, would be shown to be a trojan.
> Doesn't clamav block stuff like this, I thought?

Hi Alex,

If you still have a copy of the headers & body, could you send me a sample:

samples AT sanesecurity DOT me DOT uk

I'll run it against the dbs I've got here.

Cheers,

Steve
Sanesecurity

___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-28 Thread Alex
Hi,

>> Will amavisd now also pass to it HTML files to scan for bad URLs
>> within HTML and other email threats?
>
> I don't understand your question, but most likely it depends on your you
> have configured amavisd.

I meant that the other day there was a URL in the body of an email
that passed through as ham when in fact it ended in 'ecard.exe' and,
should the recipient download it, would be shown to be a trojan.
Doesn't clamav block stuff like this, I thought?

I'm now rejecting URLs ending in .exe right in postfix, but I thought
one of the databases was a list of bad URLs, a la blacklists...

Thanks
Alex
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-28 Thread Alex
>>> Why, are you blocking outbound rsync traffic?  If so, after 3 years of 
>>> maintaining this script and many
>>> thousands of users, this is the first time I've heard this request.
>>
>> Some of do this by default - set an outbound policy of block and allow 
>> specific traffic that's allowed. It
>> means that should a machine get compromised despite all other precautions, 
>> it can't* then be used to
>> launch an attack on others (or other servers in your own network) and/or is 
>> unable to communicate with
>> it's control centre. Just another layer of security.

Yes, exactly. That which is not expressly permitted is prohibited.

Not only once it's been compromised, but even by a trusted user that
uses rsync to download something from his own remote site to actually
do the compromising (of your system or an other system).

It's one thing on a home system, but quite another on a corporate
network where there is a policy in place.

I think it's more likely that no one has reported it previously,
rather than not implementing it.

Thanks,
Alex
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-28 Thread Bryan Blackwell
FWIW, we have the same setup where I am.  The last place I was at the network 
guys were planning to do the same thing.

--Bryan

--  Bryan Blackwell --
Unix Systems Engineer
br...@skiblack.com

On Apr 28, 2010, at 4:54 PM, Simon Hobson wrote:

>> Why, are you blocking outbound rsync traffic?  If so, after 3 years of 
>> maintaining this script and many thousands of users, this is the first time 
>> I've heard this request.
> 
> Some of do this by default - set an outbound policy of block and allow 
> specific traffic that's allowed. It means that should a machine get 
> compromised despite all other precautions, it can't* then be used to launch 
> an attack on others (or other servers in your own network) and/or is unable 
> to communicate with it's control centre. Just another layer of security.

___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-28 Thread Simon Hobson

Bill Landry wrote:

Why, are you blocking outbound rsync traffic?  If so, after 3 years 
of maintaining this script and many thousands of users, this is the 
first time I've heard this request.


Some of do this by default - set an outbound policy of block and 
allow specific traffic that's allowed. It means that should a machine 
get compromised despite all other precautions, it can't* then be used 
to launch an attack on others (or other servers in your own network) 
and/or is unable to communicate with it's control centre. Just 
another layer of security.


* Yes the attacker (assuming they got root equivalent access) can 
clear iptables - but that means they have to be proactive and risk 
making themselves more visible, not to mention they risk their remote 
install breaking networking (and also making their presence visible).


But then what would I know about administering servers :-/

--
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-28 Thread Alex
Hi,

> If you run rsync manually and then run the script after, you'll no doubt
> get a block from the server...as some mirrors only allow one rsync hit per
> hour...
>
> Just to try this out...
>
> 1. run the above rsync command manually
> 2. run the above rsync command *again*, manually

Yes, that explains it. I now understand. I hadn't realized that was the case.

Thanks again,
Alex
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-28 Thread Bill Landry

On 4/28/2010 6:01 AM, Alex wrote:

Hi,

   

The rsync mirror are defined in the script, not the config file.  However,
you can find the full list of mirrors by executing:

host rsync.sanesecurity.net
 

It might be worth mentioning this in the docs so other people can
properly configure their firewall if necessary.
   


Why, are you blocking outbound rsync traffic?  If so, after 3 years of 
maintaining this script and many thousands of users, this is the first 
time I've heard this request.

Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying
next mirror site...
   

Maybe the site was down at the time the script ran...?
 

No, I can run rsync right afterwards and it succeeds, like this:

# rsync -v rsync://ns.km33603.keymachine.de/sanesecurity/

Here's the output from the clamav-unofficial-sigs.sh script immediately after:

Sanesecurity mirror site used: ns.km33603.keymachine.de 87.118.124.191

Number of files: 40
Number of files transferred: 0
Total file size: 27032205 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 1318
File list generation time: 0.241 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 34
Total bytes received: 1932

sent 34 bytes  received 1932 bytes  786.40 bytes/sec
total size is 27032205  speedup is 13749.85
Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying
next mirror site...

Looks to me like the rsync succeeded, since it received the filelist
and actually received data. Could it instead be saying that it failed
to find an update, and not that it failed to connect?
   


The script only reports failed to connect if it actually failed to 
connect, not if there were no updates available.

How can I query clamd to find out which databases it's currently using
to scan files?
   


ClamD will use every signature database that you have located in your 
production database directory.

Will amavisd now also pass to it HTML files to scan for bad URLs
within HTML and other email threats?

   
I don't understand your question, but most likely it depends on your you 
have configured amavisd.

Is it possible to configure it to log through syslog, instead of to a file 
directly?
 

Not unless you want to edit the script.  I intentionally steered away
from using syslog so as not to pollute any of the existing system log
files, nor did I want to require script users to create a new syslog
facility in order to keep script logging separate.

I was interested in this so I can pass it to a remote log server
instead of having to monitor it locally, and also for buffering (not
that there's a lot of overhead) and for monitoring, so I don't have to
have another script that runs and watch a daemon or additional set of
log files.

Do you have any suggestions? Do you think it's necessary?
   


Personally, I don't think its necessary.  However, you could look at 
modifying the script to use"logger" if *you* feel it's necessary.

Thanks again for all your work!
   

You're welcome.

Bill
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-28 Thread Steve Basford
> No, I can run rsync right afterwards and it succeeds, like this:
>
> # rsync -v rsync://ns.km33603.keymachine.de/sanesecurity/
>
> Here's the output from the clamav-unofficial-sigs.sh script immediately
> after:

Hi Alex,

If you run rsync manually and then run the script after, you'll no doubt
get a block from the server...as some mirrors only allow one rsync hit per
hour...

Just to try this out...

1. run the above rsync command manually
2. run the above rsync command *again*, manually

Does the first one work.. and the second one fail?

If that's the case, wait 5 mins or so and run the script again, by that
time, you should hit a different mirror.

Cheers,

Steve
Sanesecurity


___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-28 Thread Alex
Hi,

> The rsync mirror are defined in the script, not the config file.  However,
> you can find the full list of mirrors by executing:
>
> host rsync.sanesecurity.net

It might be worth mentioning this in the docs so other people can
properly configure their firewall if necessary.

>> Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying
>> next mirror site...
>
> Maybe the site was down at the time the script ran...?

No, I can run rsync right afterwards and it succeeds, like this:

# rsync -v rsync://ns.km33603.keymachine.de/sanesecurity/

Here's the output from the clamav-unofficial-sigs.sh script immediately after:

Sanesecurity mirror site used: ns.km33603.keymachine.de 87.118.124.191

Number of files: 40
Number of files transferred: 0
Total file size: 27032205 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 1318
File list generation time: 0.241 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 34
Total bytes received: 1932

sent 34 bytes  received 1932 bytes  786.40 bytes/sec
total size is 27032205  speedup is 13749.85
Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying
next mirror site...

Looks to me like the rsync succeeded, since it received the filelist
and actually received data. Could it instead be saying that it failed
to find an update, and not that it failed to connect?

How can I query clamd to find out which databases it's currently using
to scan files?

Will amavisd now also pass to it HTML files to scan for bad URLs
within HTML and other email threats?

> Is it possible to configure it to log through syslog, instead of to a file 
> directly?

Not unless you want to edit the script.  I intentionally steered away
from using syslog so as not to pollute any of the existing system log
files, nor did I want to require script users to create a new syslog
facility in order to keep script logging separate.

I was interested in this so I can pass it to a remote log server
instead of having to monitor it locally, and also for buffering (not
that there's a lot of overhead) and for monitoring, so I don't have to
have another script that runs and watch a daemon or additional set of
log files.

Do you have any suggestions? Do you think it's necessary?

Thanks again for all your work!
Best regards,
Alex
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-28 Thread Bill Landry

On 4/27/2010 11:53 PM, Alex wrote:

Hi,

   

I've done some research on the best way to integrate it, but hoped
someone could point me to a current document that outlines how to do
this and help me answer some of my questions.
   

The best way to integrate them is to follow the instructions at Steve's
web site (Sane Security).
 

Great, thanks. There's an awful lot of work they've done to create
this. I've managed to get the clamav-unofficial-sigs package installed
and it appears to be working correctly so far. A few questions.

Where are the mirrors defined? I've had to add rules through the
firewall the hard way, instead of knowing what the full list are.
   


The rsync mirror are defined in the script, not the config file.  
However, you can find the full list of mirrors by executing:


host rsync.sanesecurity.net


Some of them appeared to fail, although when I run rsync manually they
succeed just fine:

Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying
next mirror site...
   


Maybe the site was down at the time the script ran...?

Running it manually, like this, results in the file list:
# rsync -av rsync://ns.km33603.keymachine.de/
sirupmusic  sirupmusic rsync
sanesecuritySaneSecurity Signatures
sanesecurity-update SaneSecurity Signatures (requires authentication)

What is sanesecurity-update? Should I somehow obtain authentication or
just use the regular sanesecurity?
   
Authentication is not required, you just need to specify the directory 
/sanesecurity

Is it possible to configure it to log through syslog, instead of to a
file directly?
   
Not unless you want to edit the script.  I intentionally steered away 
from using syslog so as not to pollute any of the existing system log 
files, nor did I want to require script users to create a new syslog 
facility in order to keep script logging separate.


Bill
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-27 Thread Alex
Hi,

>> I've done some research on the best way to integrate it, but hoped
>> someone could point me to a current document that outlines how to do
>> this and help me answer some of my questions.
>
> The best way to integrate them is to follow the instructions at Steve's
> web site (Sane Security).

Great, thanks. There's an awful lot of work they've done to create
this. I've managed to get the clamav-unofficial-sigs package installed
and it appears to be working correctly so far. A few questions.

Where are the mirrors defined? I've had to add rules through the
firewall the hard way, instead of knowing what the full list are.

Some of them appeared to fail, although when I run rsync manually they
succeed just fine:

Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying
next mirror site...

Running it manually, like this, results in the file list:
# rsync -av rsync://ns.km33603.keymachine.de/
sirupmusic  sirupmusic rsync
sanesecuritySaneSecurity Signatures
sanesecurity-update SaneSecurity Signatures (requires authentication)

What is sanesecurity-update? Should I somehow obtain authentication or
just use the regular sanesecurity?

Is it possible to configure it to log through syslog, instead of to a
file directly?

Thanks,
Alex
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


Re: [Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-27 Thread Dennis Peterson
> 
> Hi,
> 
> 
> I've done some research on the best way to integrate it, but hoped
> someone could point me to a current document that outlines how to do
> this and help me answer some of my questions.

The best way to integrate them is to follow the instructions at Steve's
web site (Sane Security). 

dp
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml


[Clamav-users] Resources for integrating with spamassassin+amavisd

2010-04-27 Thread Alex
Hi,

I'm using sa-v3.2.5 and amavisd with clamav-0.96 and it appears to be
working properly. clamdcheck periodically sends the eicar virus test
and clamd reports that it found it properly.

What is the best procedure for now integrating the sanesecurity
signatures? Are these ready for use on a production system? Why aren't
they in the default signatures included with the daily updates?

I've done some research on the best way to integrate it, but hoped
someone could point me to a current document that outlines how to do
this and help me answer some of my questions.

Thanks,
Alex
___
Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net
http://www.clamav.net/support/ml