Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
On 5/3/10 9:43 AM, Bill Landry wrote: If you are seeing three copies of each unofficial database, then you have a problem somewhere. This topic is really OT for the ClamAV mailing list. My net nanny buzzer went off. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
On 5/3/10 8:45 AM, Simon Hobson wrote: It's actually more efficient than that ! It uses something similar to a rolling checksum to find throughout the file. So in principal, you can add a short bit to the front of a large file, or even chop a file up into chunks and rearrange them, and it will still only transfer the changes. It does only send changed parts, but it takes longer to find the differences if no effort is made to leave the changed bits at the end of the file. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
On 5/3/10 8:45 AM, Alex wrote: Hi, Dennis Peterson wrote: Rsync is able to transfer only the differences between two files provided a version of the file being transferred exists on the source and the destination. In addition, rsync will not transfer anything if it determines there are no changes between the two files. This saves bandwidth for the mirrors. This Yes, of course, this is the whole benefit of rsync. The above is the reason for having a working area for rsync. Yeah? So you always have effectively three copies of your data around? I've never experienced a problem with just a local and remote copy of data when using rsync, and it forms the basis of my backup system (using link-dest, too). Thanks, Alex Why three? You only need an unmodified version (for rsync) and a working version (for clamd). The point is not to simply copy a file - the point is to copy only the parts of the file that have changed since the most recent download. Rsync can do either but bandwidth is not free so I prefer to do it the most efficient way possible. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
On Mon, May 3, 2010 8:45 am, Alex wrote: > Hi, > > Dennis Peterson wrote: >> Rsync is able to transfer only the differences between two files >> provided a version of the file being >> transferred exists on the source and the destination. In addition, rsync >> will not transfer anything if it >> determines there are no changes between the two files. This saves >> bandwidth for the mirrors. This > > Yes, of course, this is the whole benefit of rsync. > >> The above is the reason for having a working area for rsync. > > Yeah? So you always have effectively three copies of your data around? > I've never experienced a problem with just a local and remote copy of > data when using rsync, and it forms the basis of my backup system > (using link-dest, too). If you are seeing three copies of each unofficial database, then you have a problem somewhere. This topic is really OT for the ClamAV mailing list. If you wish to persist in scrutinizing the inner workings of the clamav-unofficial-sigs script, then either review the script itself with your favorite text editor or please move this discussion to the Sanesecurity users list, where it would be more appropriately discussed. Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
Chris Meadors wrote: Rsync treats all files as binary. When finding changes it splits a file into blocks, computes a checksum for each block and performs a comparison between the sending and receiving side. Then it only sends the blocks which have changed. When dealing with a text file which has been appended to, like a log, all the initial blocks are the same. But if the file is sorted, it's possible only a few additional lines will disrupt most every block by changing the start offsets through out the entire file. It's actually more efficient than that ! It uses something similar to a rolling checksum to find throughout the file. So in principal, you can add a short bit to the front of a large file, or even chop a file up into chunks and rearrange them, and it will still only transfer the changes. Andrew Tridgell's research paper is available at http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.123.1530&rep=rep1&type=pdf rsync is covered from section 3 onwards. -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
Hi, Dennis Peterson wrote: > Rsync is able to transfer only the differences between two files provided a > version of the file being > transferred exists on the source and the destination. In addition, rsync will > not transfer anything if it > determines there are no changes between the two files. This saves bandwidth > for the mirrors. This Yes, of course, this is the whole benefit of rsync. > The above is the reason for having a working area for rsync. Yeah? So you always have effectively three copies of your data around? I've never experienced a problem with just a local and remote copy of data when using rsync, and it forms the basis of my backup system (using link-dest, too). Thanks, Alex > > After the script is run, each database that has been updated is GPG > signature tested, then ClamAV integrity tested, and then rsynced into the > the ClamAV production directory. You do not want to place any database into > the production directory until it have been tested, thus the creation and > use of the working directories (/usr/unofficial-dbs/*) by the script. > > The script stops and reports/logs any failure along the way so that a > corrupted or compromised database cannot end up in production use. > >> Is there support that I'm somehow missing for alerting an >> administrator that there was a problem when it actually happens, and >> not through just a daily report? > > The only error reporting the script does is via email after each script run > and logging to a log file, depending on how the user has configured the > script to run. > >> How can I verify that clamd is actually using the full set of >> databases I've downloaded? > > Any database that shows up in the ClamAV production directory is used by > ClamAV/ClamD - ls /path/to/clamav will show you what database are being > used. > > Bill > ___ > Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net > http://www.clamav.net/support/ml > ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
On Mon, 2010-05-03 at 05:53 -0700, Jim Preston wrote: > Dennis Peterson wrote: > > > > Rsync is able to transfer only the differences between two files > > provided a version of the file being transferred exists on the source > > and the destination. In addition, rsync will not transfer anything if > > it determines there are no changes between the two files. > > Is this true for binary files as well? or just text files? Rsync treats all files as binary. When finding changes it splits a file into blocks, computes a checksum for each block and performs a comparison between the sending and receiving side. Then it only sends the blocks which have changed. When dealing with a text file which has been appended to, like a log, all the initial blocks are the same. But if the file is sorted, it's possible only a few additional lines will disrupt most every block by changing the start offsets through out the entire file. -- Chris ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
Dennis Peterson wrote: Rsync is able to transfer only the differences between two files provided a version of the file being transferred exists on the source and the destination. In addition, rsync will not transfer anything if it determines there are no changes between the two files. Dennis, Is this true for binary files as well? or just text files? Thanks, Jim ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
On 5/2/10 9:59 AM, Alex wrote: Hi, Why are some of the databases duplicated in the clamav root dir and also in the unofficial-dbs/ss-dbs directory, such as winnow_malware.hdb? The rsync protocol only downloads the changes between the local and remote files, so the local file much be available comparison. The unofficial/ss-dbs is also the working directory where the database's GPG signature and integrity testing is done before rsyncing the files into the ClamAV production directory. By this time you must think I'm dumb or completely new at this. I'd like to think I'm just more inquisitive and going beyond just the home use. Time for some more coffee for me :-) It sounds like the real reason for is duplicates is for the integrity check, because rsync doesn't otherwise need two local copies. Also for moving them all into place into production at once? Rsync is able to transfer only the differences between two files provided a version of the file being transferred exists on the source and the destination. In addition, rsync will not transfer anything if it determines there are no changes between the two files. This saves bandwidth for the mirrors. This transfer economy requires the client maintain unmodified copies of the files in the target directory so rsync can make these comparisons. (A bit more to know: This also works best if the files are unsorted between versions. For a trivial example, a syslog file that has grown over time. Rsync will skip over the parts found in both files very quickly and transfer only those parts found in newer version. If the files have been sorted rsync has to examine and reconstruct the entire file.) The above is the reason for having a working area for rsync. A reason for keeping a separate area for clamd is because clamd will die if a corrupt file is found in the clamd working area. A corrupt file could include a file that is being copied into the working area but is not yet complete (non-atomic copy/move). As clamd does check the working area from time to time the chance exists that this scan can occur at a time a file is being placed in the working area. It is a good practice to minimize file activity in the clamd working area. As another matter of good practice I only use rsync to place signatures in the clamd working area because rsync writes to a hidden file until the transfer is complete at which time that file is renamed. As such it appears suddenly and so does not cause clamd any problems. This works for local file systems as well as NAS/NFS/FUSE/CIFS file systems. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
On 5/2/2010 9:59 AM, Alex wrote: Hi, Why are some of the databases duplicated in the clamav root dir and also in the unofficial-dbs/ss-dbs directory, such as winnow_malware.hdb? The rsync protocol only downloads the changes between the local and remote files, so the local file much be available comparison. The unofficial/ss-dbs is also the working directory where the database's GPG signature and integrity testing is done before rsyncing the files into the ClamAV production directory. By this time you must think I'm dumb or completely new at this. I'd like to think I'm just more inquisitive and going beyond just the home use. Time for some more coffee for me :-) It sounds like the real reason for is duplicates is for the integrity check, because rsync doesn't otherwise need two local copies. Also for moving them all into place into production at once? Not just integrity testing, but first GPG signature tested in order to confirm that the files have not been compromised in any way. After the script is run, each database that has been updated is GPG signature tested, then ClamAV integrity tested, and then rsynced into the the ClamAV production directory. You do not want to place any database into the production directory until it have been tested, thus the creation and use of the working directories (/usr/unofficial-dbs/*) by the script. The script stops and reports/logs any failure along the way so that a corrupted or compromised database cannot end up in production use. Is there support that I'm somehow missing for alerting an administrator that there was a problem when it actually happens, and not through just a daily report? The only error reporting the script does is via email after each script run and logging to a log file, depending on how the user has configured the script to run. How can I verify that clamd is actually using the full set of databases I've downloaded? Any database that shows up in the ClamAV production directory is used by ClamAV/ClamD - ls /path/to/clamav will show you what database are being used. Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
Hi, >> Why are some of the databases duplicated in the clamav root dir and >> also in the unofficial-dbs/ss-dbs directory, such as >> winnow_malware.hdb? > > The rsync protocol only downloads the changes between the local and remote > files, so the local file much be available comparison. The > unofficial/ss-dbs is also the working directory where the database's GPG > signature and integrity testing is done before rsyncing the files into the > ClamAV production directory. By this time you must think I'm dumb or completely new at this. I'd like to think I'm just more inquisitive and going beyond just the home use. Time for some more coffee for me :-) It sounds like the real reason for is duplicates is for the integrity check, because rsync doesn't otherwise need two local copies. Also for moving them all into place into production at once? Is there support that I'm somehow missing for alerting an administrator that there was a problem when it actually happens, and not through just a daily report? How can I verify that clamd is actually using the full set of databases I've downloaded? Thanks again, Alex ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
On 5/1/2010 8:19 PM, Alex wrote: Hi, These are *NOT* Sanesecurity distributed scripts, so please do not add them to the Sanesecurity section of the script's config file. MBL already has a Okay, got it. It wasn't clear to me that it was an "MBL" db, and the reference in the sanesecurity database section also includes info on the other databases. Only the databases that the Sanesecurity site says are "distributed by Sanesecurity" can be configured to be downloaded in the Sanesecurity section of the config file and then downloaded from the Sanesecurity rsync servers. The others signature databases are hosted and downloaded from other sites, thus the separate sections defined in the script's config file. Why are some of the databases duplicated in the clamav root dir and also in the unofficial-dbs/ss-dbs directory, such as winnow_malware.hdb? The rsync protocol only downloads the changes between the local and remote files, so the local file much be available comparison. The unofficial/ss-dbs is also the working directory where the database's GPG signature and integrity testing is done before rsyncing the files into the ClamAV production directory. Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
Hi, > These are *NOT* Sanesecurity distributed scripts, so please do not add them > to the Sanesecurity section of the script's config file. MBL already has a Okay, got it. It wasn't clear to me that it was an "MBL" db, and the reference in the sanesecurity database section also includes info on the other databases. Why are some of the databases duplicated in the clamav root dir and also in the unofficial-dbs/ss-dbs directory, such as winnow_malware.hdb? Thanks, Alex ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
On 5/1/2010 1:01 PM, Bill Landry wrote: On 5/1/2010 12:38 PM, Alex wrote: Hi, sent 34 bytes received 1932 bytes 786.40 bytes/sec total size is 27032205 speedup is 13749.85 Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying next mirror site... Looks to me like the rsync succeeded, since it received the filelist and actually received data. Could it instead be saying that it failed to find an update, and not that it failed to connect? The script only reports failed to connect if it actually failed to connect, not if there were no updates available. I found the problem. Stepping through the script with execution, I find this: rsync --no-motd --files-from=/var/clamav/unofficial-dbs/configs/ss-include-dbs.txt -ctuz --contimeout=30 --timeout=30 --stats rsync://84.19.167.43/sanesecurity /var/clamav/unofficial-dbs/ss-dbs rsync: link_stat "/MSRBL-SPAM.ndb" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/MSRBL-SPAM.ndb.sig" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/MSRBL-Images.hdb" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/MSRBL-Images.hdb.sig" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/mbl.ndb" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/mbl.ndb.sig" (in sanesecurity) failed: No such file or directory (2) These are *NOT* Sanesecurity distributed scripts, so please do not add Oops, meant to say that "These are "NOT" Sanesecurity distributed signature databases", not "scripts". them to the Sanesecurity section of the script's config file. MBL already has a section in the config file, so no need to add it to the Sanesecurity section. MSRBL has not been update since last July, so I purposefully removed it from the script with the last update - thus, you should not be adding these to the script anyway. Number of files: 40 Number of files transferred: 0 Total file size: 26677157 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 1314 File list generation time: 0.232 seconds File list transfer time: 0.000 seconds Total bytes sent: 34 Total bytes received: 1928 sent 34 bytes received 1928 bytes 784.80 bytes/sec total size is 26677157 speedup is 13596.92 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1508) [generator=3.0.7] However, the script reports that the connection failed, which is somewhat of a red herring. Would it be helpful to not suppress the rsync output when it fails with something other than a simple "connection failed" message? The script is working as intended. User should not add signature database that do not exist on Sanesecurity rsync servers. These databases (that could not be retrieved) were collected from the sanesecurity database page. Aren't they supposed to be used? I'd think someone else would have found this problem? No one else has reported this problem because no one else is trying to download signature database that do not exist on Sanesecurity rsync servers. On a similar note, this script can replace freshclam, correct? No it cannot replace freshclam, as the clamav-unofficial-sigs script only update unofficial 3rd party signature databases (as the script name itself indicates), not official ClamAV signature databases. Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
On 5/1/2010 12:38 PM, Alex wrote: Hi, sent 34 bytes received 1932 bytes 786.40 bytes/sec total size is 27032205 speedup is 13749.85 Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying next mirror site... Looks to me like the rsync succeeded, since it received the filelist and actually received data. Could it instead be saying that it failed to find an update, and not that it failed to connect? The script only reports failed to connect if it actually failed to connect, not if there were no updates available. I found the problem. Stepping through the script with execution, I find this: rsync --no-motd --files-from=/var/clamav/unofficial-dbs/configs/ss-include-dbs.txt -ctuz --contimeout=30 --timeout=30 --stats rsync://84.19.167.43/sanesecurity /var/clamav/unofficial-dbs/ss-dbs rsync: link_stat "/MSRBL-SPAM.ndb" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/MSRBL-SPAM.ndb.sig" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/MSRBL-Images.hdb" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/MSRBL-Images.hdb.sig" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/mbl.ndb" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/mbl.ndb.sig" (in sanesecurity) failed: No such file or directory (2) These are *NOT* Sanesecurity distributed scripts, so please do not add them to the Sanesecurity section of the script's config file. MBL already has a section in the config file, so no need to add it to the Sanesecurity section. MSRBL has not been update since last July, so I purposefully removed it from the script with the last update - thus, you should not be adding these to the script anyway. Number of files: 40 Number of files transferred: 0 Total file size: 26677157 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 1314 File list generation time: 0.232 seconds File list transfer time: 0.000 seconds Total bytes sent: 34 Total bytes received: 1928 sent 34 bytes received 1928 bytes 784.80 bytes/sec total size is 26677157 speedup is 13596.92 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1508) [generator=3.0.7] However, the script reports that the connection failed, which is somewhat of a red herring. Would it be helpful to not suppress the rsync output when it fails with something other than a simple "connection failed" message? The script is working as intended. User should not add signature database that do not exist on Sanesecurity rsync servers. These databases (that could not be retrieved) were collected from the sanesecurity database page. Aren't they supposed to be used? I'd think someone else would have found this problem? No one else has reported this problem because no one else is trying to download signature database that do not exist on Sanesecurity rsync servers. On a similar note, this script can replace freshclam, correct? No it cannot replace freshclam, as the clamav-unofficial-sigs script only update unofficial 3rd party signature databases (as the script name itself indicates), not official ClamAV signature databases. Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
Hi, >> sent 34 bytes received 1932 bytes 786.40 bytes/sec >> total size is 27032205 speedup is 13749.85 >> Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying >> next mirror site... >> >> Looks to me like the rsync succeeded, since it received the filelist >> and actually received data. Could it instead be saying that it failed >> to find an update, and not that it failed to connect? > > The script only reports failed to connect if it actually failed to connect, > not if there were no updates available. I found the problem. Stepping through the script with execution, I find this: rsync --no-motd --files-from=/var/clamav/unofficial-dbs/configs/ss-include-dbs.txt -ctuz --contimeout=30 --timeout=30 --stats rsync://84.19.167.43/sanesecurity /var/clamav/unofficial-dbs/ss-dbs rsync: link_stat "/MSRBL-SPAM.ndb" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/MSRBL-SPAM.ndb.sig" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/MSRBL-Images.hdb" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/MSRBL-Images.hdb.sig" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/mbl.ndb" (in sanesecurity) failed: No such file or directory (2) rsync: link_stat "/mbl.ndb.sig" (in sanesecurity) failed: No such file or directory (2) Number of files: 40 Number of files transferred: 0 Total file size: 26677157 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 1314 File list generation time: 0.232 seconds File list transfer time: 0.000 seconds Total bytes sent: 34 Total bytes received: 1928 sent 34 bytes received 1928 bytes 784.80 bytes/sec total size is 26677157 speedup is 13596.92 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1508) [generator=3.0.7] However, the script reports that the connection failed, which is somewhat of a red herring. Would it be helpful to not suppress the rsync output when it fails with something other than a simple "connection failed" message? These databases (that could not be retrieved) were collected from the sanesecurity database page. Aren't they supposed to be used? I'd think someone else would have found this problem? On a similar note, this script can replace freshclam, correct? Thanks, Alex ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
>> If you still have a copy of the headers & body, could you send me a >> sample: > > Attachment sent. Thanks for the sample Alex. It's already being detected as: Sanesecurity.Malware.8830.UNOFFICIAL So, you should already be covered :) Cheers, Steve Sanesecurity ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
>> I meant that the other day there was a URL in the body of an email >> that passed through as ham when in fact it ended in 'ecard.exe' and, >> should the recipient download it, would be shown to be a trojan. >> Doesn't clamav block stuff like this, I thought? > > If you still have a copy of the headers & body, could you send me a sample: Attachment sent. Thanks, Alex ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
> I meant that the other day there was a URL in the body of an email > that passed through as ham when in fact it ended in 'ecard.exe' and, > should the recipient download it, would be shown to be a trojan. > Doesn't clamav block stuff like this, I thought? Hi Alex, If you still have a copy of the headers & body, could you send me a sample: samples AT sanesecurity DOT me DOT uk I'll run it against the dbs I've got here. Cheers, Steve Sanesecurity ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
Hi, >> Will amavisd now also pass to it HTML files to scan for bad URLs >> within HTML and other email threats? > > I don't understand your question, but most likely it depends on your you > have configured amavisd. I meant that the other day there was a URL in the body of an email that passed through as ham when in fact it ended in 'ecard.exe' and, should the recipient download it, would be shown to be a trojan. Doesn't clamav block stuff like this, I thought? I'm now rejecting URLs ending in .exe right in postfix, but I thought one of the databases was a list of bad URLs, a la blacklists... Thanks Alex ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
>>> Why, are you blocking outbound rsync traffic? If so, after 3 years of >>> maintaining this script and many >>> thousands of users, this is the first time I've heard this request. >> >> Some of do this by default - set an outbound policy of block and allow >> specific traffic that's allowed. It >> means that should a machine get compromised despite all other precautions, >> it can't* then be used to >> launch an attack on others (or other servers in your own network) and/or is >> unable to communicate with >> it's control centre. Just another layer of security. Yes, exactly. That which is not expressly permitted is prohibited. Not only once it's been compromised, but even by a trusted user that uses rsync to download something from his own remote site to actually do the compromising (of your system or an other system). It's one thing on a home system, but quite another on a corporate network where there is a policy in place. I think it's more likely that no one has reported it previously, rather than not implementing it. Thanks, Alex ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
FWIW, we have the same setup where I am. The last place I was at the network guys were planning to do the same thing. --Bryan -- Bryan Blackwell -- Unix Systems Engineer br...@skiblack.com On Apr 28, 2010, at 4:54 PM, Simon Hobson wrote: >> Why, are you blocking outbound rsync traffic? If so, after 3 years of >> maintaining this script and many thousands of users, this is the first time >> I've heard this request. > > Some of do this by default - set an outbound policy of block and allow > specific traffic that's allowed. It means that should a machine get > compromised despite all other precautions, it can't* then be used to launch > an attack on others (or other servers in your own network) and/or is unable > to communicate with it's control centre. Just another layer of security. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
Bill Landry wrote: Why, are you blocking outbound rsync traffic? If so, after 3 years of maintaining this script and many thousands of users, this is the first time I've heard this request. Some of do this by default - set an outbound policy of block and allow specific traffic that's allowed. It means that should a machine get compromised despite all other precautions, it can't* then be used to launch an attack on others (or other servers in your own network) and/or is unable to communicate with it's control centre. Just another layer of security. * Yes the attacker (assuming they got root equivalent access) can clear iptables - but that means they have to be proactive and risk making themselves more visible, not to mention they risk their remote install breaking networking (and also making their presence visible). But then what would I know about administering servers :-/ -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
Hi, > If you run rsync manually and then run the script after, you'll no doubt > get a block from the server...as some mirrors only allow one rsync hit per > hour... > > Just to try this out... > > 1. run the above rsync command manually > 2. run the above rsync command *again*, manually Yes, that explains it. I now understand. I hadn't realized that was the case. Thanks again, Alex ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
On 4/28/2010 6:01 AM, Alex wrote: Hi, The rsync mirror are defined in the script, not the config file. However, you can find the full list of mirrors by executing: host rsync.sanesecurity.net It might be worth mentioning this in the docs so other people can properly configure their firewall if necessary. Why, are you blocking outbound rsync traffic? If so, after 3 years of maintaining this script and many thousands of users, this is the first time I've heard this request. Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying next mirror site... Maybe the site was down at the time the script ran...? No, I can run rsync right afterwards and it succeeds, like this: # rsync -v rsync://ns.km33603.keymachine.de/sanesecurity/ Here's the output from the clamav-unofficial-sigs.sh script immediately after: Sanesecurity mirror site used: ns.km33603.keymachine.de 87.118.124.191 Number of files: 40 Number of files transferred: 0 Total file size: 27032205 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 1318 File list generation time: 0.241 seconds File list transfer time: 0.000 seconds Total bytes sent: 34 Total bytes received: 1932 sent 34 bytes received 1932 bytes 786.40 bytes/sec total size is 27032205 speedup is 13749.85 Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying next mirror site... Looks to me like the rsync succeeded, since it received the filelist and actually received data. Could it instead be saying that it failed to find an update, and not that it failed to connect? The script only reports failed to connect if it actually failed to connect, not if there were no updates available. How can I query clamd to find out which databases it's currently using to scan files? ClamD will use every signature database that you have located in your production database directory. Will amavisd now also pass to it HTML files to scan for bad URLs within HTML and other email threats? I don't understand your question, but most likely it depends on your you have configured amavisd. Is it possible to configure it to log through syslog, instead of to a file directly? Not unless you want to edit the script. I intentionally steered away from using syslog so as not to pollute any of the existing system log files, nor did I want to require script users to create a new syslog facility in order to keep script logging separate. I was interested in this so I can pass it to a remote log server instead of having to monitor it locally, and also for buffering (not that there's a lot of overhead) and for monitoring, so I don't have to have another script that runs and watch a daemon or additional set of log files. Do you have any suggestions? Do you think it's necessary? Personally, I don't think its necessary. However, you could look at modifying the script to use"logger" if *you* feel it's necessary. Thanks again for all your work! You're welcome. Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
> No, I can run rsync right afterwards and it succeeds, like this: > > # rsync -v rsync://ns.km33603.keymachine.de/sanesecurity/ > > Here's the output from the clamav-unofficial-sigs.sh script immediately > after: Hi Alex, If you run rsync manually and then run the script after, you'll no doubt get a block from the server...as some mirrors only allow one rsync hit per hour... Just to try this out... 1. run the above rsync command manually 2. run the above rsync command *again*, manually Does the first one work.. and the second one fail? If that's the case, wait 5 mins or so and run the script again, by that time, you should hit a different mirror. Cheers, Steve Sanesecurity ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
Hi, > The rsync mirror are defined in the script, not the config file. However, > you can find the full list of mirrors by executing: > > host rsync.sanesecurity.net It might be worth mentioning this in the docs so other people can properly configure their firewall if necessary. >> Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying >> next mirror site... > > Maybe the site was down at the time the script ran...? No, I can run rsync right afterwards and it succeeds, like this: # rsync -v rsync://ns.km33603.keymachine.de/sanesecurity/ Here's the output from the clamav-unofficial-sigs.sh script immediately after: Sanesecurity mirror site used: ns.km33603.keymachine.de 87.118.124.191 Number of files: 40 Number of files transferred: 0 Total file size: 27032205 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 1318 File list generation time: 0.241 seconds File list transfer time: 0.000 seconds Total bytes sent: 34 Total bytes received: 1932 sent 34 bytes received 1932 bytes 786.40 bytes/sec total size is 27032205 speedup is 13749.85 Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying next mirror site... Looks to me like the rsync succeeded, since it received the filelist and actually received data. Could it instead be saying that it failed to find an update, and not that it failed to connect? How can I query clamd to find out which databases it's currently using to scan files? Will amavisd now also pass to it HTML files to scan for bad URLs within HTML and other email threats? > Is it possible to configure it to log through syslog, instead of to a file > directly? Not unless you want to edit the script. I intentionally steered away from using syslog so as not to pollute any of the existing system log files, nor did I want to require script users to create a new syslog facility in order to keep script logging separate. I was interested in this so I can pass it to a remote log server instead of having to monitor it locally, and also for buffering (not that there's a lot of overhead) and for monitoring, so I don't have to have another script that runs and watch a daemon or additional set of log files. Do you have any suggestions? Do you think it's necessary? Thanks again for all your work! Best regards, Alex ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
On 4/27/2010 11:53 PM, Alex wrote: Hi, I've done some research on the best way to integrate it, but hoped someone could point me to a current document that outlines how to do this and help me answer some of my questions. The best way to integrate them is to follow the instructions at Steve's web site (Sane Security). Great, thanks. There's an awful lot of work they've done to create this. I've managed to get the clamav-unofficial-sigs package installed and it appears to be working correctly so far. A few questions. Where are the mirrors defined? I've had to add rules through the firewall the hard way, instead of knowing what the full list are. The rsync mirror are defined in the script, not the config file. However, you can find the full list of mirrors by executing: host rsync.sanesecurity.net Some of them appeared to fail, although when I run rsync manually they succeed just fine: Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying next mirror site... Maybe the site was down at the time the script ran...? Running it manually, like this, results in the file list: # rsync -av rsync://ns.km33603.keymachine.de/ sirupmusic sirupmusic rsync sanesecuritySaneSecurity Signatures sanesecurity-update SaneSecurity Signatures (requires authentication) What is sanesecurity-update? Should I somehow obtain authentication or just use the regular sanesecurity? Authentication is not required, you just need to specify the directory /sanesecurity Is it possible to configure it to log through syslog, instead of to a file directly? Not unless you want to edit the script. I intentionally steered away from using syslog so as not to pollute any of the existing system log files, nor did I want to require script users to create a new syslog facility in order to keep script logging separate. Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
Hi, >> I've done some research on the best way to integrate it, but hoped >> someone could point me to a current document that outlines how to do >> this and help me answer some of my questions. > > The best way to integrate them is to follow the instructions at Steve's > web site (Sane Security). Great, thanks. There's an awful lot of work they've done to create this. I've managed to get the clamav-unofficial-sigs package installed and it appears to be working correctly so far. A few questions. Where are the mirrors defined? I've had to add rules through the firewall the hard way, instead of knowing what the full list are. Some of them appeared to fail, although when I run rsync manually they succeed just fine: Connection to ns.km33603.keymachine.de 87.118.124.191 failed - Trying next mirror site... Running it manually, like this, results in the file list: # rsync -av rsync://ns.km33603.keymachine.de/ sirupmusic sirupmusic rsync sanesecuritySaneSecurity Signatures sanesecurity-update SaneSecurity Signatures (requires authentication) What is sanesecurity-update? Should I somehow obtain authentication or just use the regular sanesecurity? Is it possible to configure it to log through syslog, instead of to a file directly? Thanks, Alex ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
Re: [Clamav-users] Resources for integrating with spamassassin+amavisd
> > Hi, > > > I've done some research on the best way to integrate it, but hoped > someone could point me to a current document that outlines how to do > this and help me answer some of my questions. The best way to integrate them is to follow the instructions at Steve's web site (Sane Security). dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
[Clamav-users] Resources for integrating with spamassassin+amavisd
Hi, I'm using sa-v3.2.5 and amavisd with clamav-0.96 and it appears to be working properly. clamdcheck periodically sends the eicar virus test and clamd reports that it found it properly. What is the best procedure for now integrating the sanesecurity signatures? Are these ready for use on a production system? Why aren't they in the default signatures included with the daily updates? I've done some research on the best way to integrate it, but hoped someone could point me to a current document that outlines how to do this and help me answer some of my questions. Thanks, Alex ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
