Re: [Clamav-users] Why does clam die on a malformed database ?
Christopher X. Candreva wrote the following on 12/30/2006 1:39 PM -0800: On Sat, 30 Dec 2006, Bill Landry wrote: The MSRBL-Images.hdb database started showing up corrupted yesterday and This is not the only reason I ask, but the most recent. I have a script that checks that evidenly has a bug. I can either spend time fixing that, or fixing clam so it ignores the one line with an error and processes the rest of the file, and am trying to decide how best to spend time. Probably both in the end. It's a question of being brittle. Any small error in the databases stops clam dead. Hell, clam won't run if there is an empty db file ! I had wanted to leave a temporary db file around for things I wanted to block quickly, and leave it empty when there was nothing to block. Surprise -- that kills clam ! Why in the world should that happen ? I can see the arguements if the official, signed files are corrupt, but for exra added files, ignore the bad line, ignore just that FILE, but it makes no sense to me to die completely. Any mistake becomes fatal, for no good reason. You are preaching to the choir here, as you have no argument from me. I raised the same issue the last time this happened to me a few weeks ago and clamd died twice on me in one day. The script work-around to check the databases before implementing them has saved my bacon with this last string of corrupted databases from MSRBL. However, I still agree that clamd should be able to handle these kinds of issues gracefully, and in the alternative, should not simply die silently. Bill ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Re: Why does clam die on a malformed database ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 John Rudd wrote: Sander Holthaus wrote: Dennis Peterson wrote: This is a very naive or at least uninformed position to take on the monetary significance of email. The issue is that email never was designed to be used in that particular fashion. No offense, but Dennis is right. You're being naive. The issue is not how it was designed to be used. The issue is how it is actually used. The latter carries FAR more weight, outside of purely academic discussions, than the former. Even though I work at a university, I do not live in an academic world. I live in the real world. And in the real world, email is used that way, and it is a mission critical service that has financial repercussions. Arguing about how it was designed to be used is just picking nits. It has nothing to do with being naive, it has to do with understanding the constraints of the specific technology. I'm perfectly well aware of what (business) people expect from (E)SMTP, but even if I do my best to meet their expectations, I still have to rely on others and the basic fundamental design of (E)SMTP and the Internet to get the job done. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (MingW32) iD8DBQFFl5GkVf373DysOTURAqX/AKDYli887JxUpAzxp25TO20rj4pAHQCghvNw 2Z/LiVdBFtJQV0aFUKO7A2g= =N50k -END PGP SIGNATURE- ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Re: Why does clam die on a malformed database ?
On Saturday December 30, 2006 at 07:26:57 (PM) Sander Holthaus wrote: The issue is that email never was designed to be used in that particular fashion. While it may be fast and almost instant in normal circumstances, it was not designed with that in mind. The fact that businesses do expect that is something else and it is what usually gives people in IT headaches. Henry Ford never designed the original Model T with electronic ignition, air conditioning, GPS, etc. Does that mean that those items, among others, should simply be discarded? Things evolve. Even GUI's were probably not envisioned by the original PC architects, yet most people today would not choose to operate their PC's without them. -- Gerard ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Re: Why does clam die on a malformed database ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gerard Seibert wrote: On Saturday December 30, 2006 at 07:26:57 (PM) Sander Holthaus wrote: The issue is that email never was designed to be used in that particular fashion. While it may be fast and almost instant in normal circumstances, it was not designed with that in mind. The fact that businesses do expect that is something else and it is what usually gives people in IT headaches. Henry Ford never designed the original Model T with electronic ignition, air conditioning, GPS, etc. Does that mean that those items, among others, should simply be discarded? Things evolve. Even GUI's were probably not envisioned by the original PC architects, yet most people today would not choose to operate their PC's without them. That isn't a correct analogy. Last time I checked the RFC's relating to email, I still recognized a model T in there. I'm not saying that things should not evolve, I'm not saying that email should not be instant or that businesses should not use it the way they do. But the technology behind email hasn't kept up pace with the way it is being used and the way people want it to use. If people really want email to be instant, secure and reliable, they should design a new protocol with those constraints in mind. Untill then, it remains a best effort. Kind Regards, Sander Holthaus PS: If people are this interested in making (E)SMTP instant, why isn't there a blacklist of MX's using greylisting? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (MingW32) iD8DBQFFl7uKVf373DysOTURAiwbAKDUpdWPVUz3Gi7dgXcVAderPVmLoACgtOaM J907GUNefkcsW706Sb1mJ+k= =fg90 -END PGP SIGNATURE- ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Re: Why does clam die on a malformed database ?
--As of December 31, 2006 2:30:51 PM +0100, Sander Holthaus is alleged to have said: I'm not saying that things should not evolve, I'm not saying that email should not be instant or that businesses should not use it the way they do. But the technology behind email hasn't kept up pace with the way it is being used and the way people want it to use. If people really want email to be instant, secure and reliable, they should design a new protocol with those constraints in mind. Untill then, it remains a best effort. --As for the rest, it is mine. So we, the email system maintainers, should be giving our _best effort_ to make sure that the system works as quickly and reliably as possible. ;) ClamAV is a tool towards that, and (generally) does a good job of helping the admin with the task of maintaining the email service. I think we all agree there. Daniel T. Staal --- This email copyright the author. Unless otherwise noted, you are expressly allowed to retransmit, quote, or otherwise use the contents for non-commercial purposes. This copyright will expire 5 years after the author's death, or in 30 years, whichever is longer, unless such a period is in excess of local copyright law. --- ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Why does clam die on a malformed database ?
Bill Landry wrote: [ ... ] You are preaching to the choir here, as you have no argument from me. I raised the same issue the last time this happened to me a few weeks ago and clamd died twice on me in one day. The script work-around to check the databases before implementing them has saved my bacon with this last string of corrupted databases from MSRBL. However, I still agree that clamd should be able to handle these kinds of issues gracefully, and in the alternative, should not simply die silently. Agreed-- it would be nice if clamd was more robust, either by continuing to run with the other DBs (as available) and either drop the bad line or the entire bad DB file, until a new update comes along which is OK. However, improving how clamd responds to a bad DB is solving a consequence or symptom rather than the original problem. Maybe we should try to persuade the MSRBL site (and others) to use a similar checking script when pushing new versions of the DB's out, rather than checking upon receipt after people have used bandwidth to download and then have to discard a bad update...? Doesn't the main ClamAV database get tested before going out? Is there some kind of run against a big collection of known-good files which should scan cleanly to detect a DB which contains a false positive? -- -Chuck ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Why does clam die on a malformed database ?
At 11:31 AM 12/31/2006, Chuck Swiger wrote: Agreed-- it would be nice if clamd was more robust, either by continuing to run with the other DBs (as available) and either drop the bad line or the entire bad DB file, until a new update comes along which is OK. An option to drop the bad database and use rest would be good, with a big warning logged. Dropping the offending line might not work as expected. If the file is scrambled there may be bad signatures with correct syntax in the corrupt file. Doesn't the main ClamAV database get tested before going out? Is there some kind of run against a big collection of known-good files which should scan cleanly to detect a DB which contains a false positive? The official clam databases do get tested, and furthermore, freshclam protects against corrupt main/daily databases if there is a problem with distribution. This rant is mostly concerned with database updates outside of freshclam, with the example of the various add-on databases available. It has always been possible (and advisable) to test these add-on databases locally with clamscan -d /path/to/database before installing them in the live directory, and there have been scripts posted here in the past detailing how. -- Noel Jones ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Why does clam die on a malformed database ?
Chuck Swiger wrote: Bill Landry wrote: [ ... ] You are preaching to the choir here, as you have no argument from me. I raised the same issue the last time this happened to me a few weeks ago and clamd died twice on me in one day. The script work-around to check the databases before implementing them has saved my bacon with this last string of corrupted databases from MSRBL. However, I still agree that clamd should be able to handle these kinds of issues gracefully, and in the alternative, should not simply die silently. Agreed-- it would be nice if clamd was more robust, either by continuing to run with the other DBs (as available) and either drop the bad line or the entire bad DB file, until a new update comes along which is OK. However, improving how clamd responds to a bad DB is solving a consequence or symptom rather than the original problem. Maybe we should try to persuade the MSRBL site (and others) to use a similar checking script when pushing new versions of the DB's out, rather than checking upon receipt after people have used bandwidth to download and then have to discard a bad update...? Qual checking the db is not unreasonable given it can be done with automation, but it doesn't protect you from delivery failures where you get a partial db. Without a checksum you are still on the hook to qual check the tranfer. dp ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Large extremely compressed text files and clamd / amavis-new
- Mark Hennessy [EMAIL PROTECTED] wrote: Dec 29 12:56:58 host /usr/local/sbin/amavisd[89023]: (89023-01) (!)do_uncompress: Error running decompressor /usr/bin/gzip -d on p001, exit 1 at (eval 68) line 562. Dec 29 12:57:14 host /usr/local/sbin/amavisd[89023]: (89023-01) (!)Decoding of p006 (ASCII text, with very long lines) failed, leaving it unpacked: do_ascii: timed out ... While there have been reports of long scanning times on big ZIP files containing text files, it seems that the above error is coming from Amavis, not ClamD. It does not look like the message is hitting clamd at all. I use Exiscan, not Amavis, but I know that some sites have to increase the Amavis timeout to 700 seconds, otherwise Amavis gives up too quickly. But since ClamD has a ZIP engine built-in, why do you want amavis to extra the files to temporary file, when clamd could scan the file in place? And what version of ClamAV are you running? 0.88.7 has improvements to the ZIP engine. 0.99.* is supposed to be much faster on ZIP files (but still too slow for some sites). Tom ___ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://lurker.clamav.net/list/clamav-users.html
