Re: [Clamav-users] benchmarks for a LARGE site?
> > > Could someone with a LARGE site (we have about 35,000 users) post what > > hardware they use for ClamAV, and how many messages/day it handles? I'd > > like to suggest they put it on a few PCs and have their relays contact > > the milter via a network socket in a round-robin fashion. But it would > > be good to hear people's experiences with something on this large of a > > scale before I make the proposal. Dear, We are running Clamd 0.83 with a home made client (calling spamd at SMTP level). We handle on average +- 40.000 valid messages/day on a HP DL 360 G4/Bi-proc Xeon 3 GHZ/ 4GB mem/RedHat EL 3.0. No probs. It could do far more than that with such a hardware (I'm expecting 5 - 10 times more). We are also currently testing with a lighter hardware: 6 years old Intel ISP 2150/Bi-Proc Xeon 1Ghz/2 GB mem/Debian sarge. With this config we are experiencing communication probs between the client and spamd. The exact reason of these probs is not yet known. Hoping this will help. Sincerely yours, Arnaud www.contactoffice.com ___ http://lurker.clamav.net/list/clamav-users.html
[Clamav-users] 'Too many open files' on a buzy clamd contd ....
Dear all, Addendum : I forgot to mention the version : ClamAV 0.83/856/Wed Apr 27 09:00:37 2005 Sorry for this second post. Arnaud We are running a webmail service using ClamAV and get roughtly 30.000 valid mails/day. We run home-build SMTP servers calling clamd, emulating the client. The problem : After running +- 10 minutes, clamd.log reports a first message saying : 'ERROR: ScanStream: accept timeout' quickly followed other ones. After 1 or 2 minutes, we get another message : 'ERROR: accept() failed: Too many open files' and, I guess, clamd does not respond any more. Need to restart the daemon to restore the service. I tried the following tunning : 1. Increase the number of threads from 10 to 30 for reducing the queue: no changes, still errors. 2. Increase the number of MaxConnectionQueueLength to 30: no changes, still errors. Other info : Clamd runs as non-root user. Launch script is : /etc/init.d/clamav_daemon start (not modified from orginal). ClamAV is currently running and a Debian Woody with 1.5 GB mem on a 2*1Ghz Intel chassis. SpamAssassin is also running on this box. Version 3.0.2 standard (Razor, DCC, ...) Mitigating factors (;-) Running the same config on a more powerfull box does not generate the prob (2*3GH + multithreading) clamd.conf : #Automatically Generated by clamav-base postinst #To reconfigure clamd run #dpkg-reconfigure clamav-base #LocalSocket /var/run/clamav/clamd.ctl FixStaleSocket User clamav AllowSupplementaryGroups ArchiveMaxRecursion 10 ArchiveMaxFiles 1500 ArchiveMaxFileSize 30M ArchiveMaxCompressionRatio 300 ArchiveBlockEncrypted ArchiveBlockMax ReadTimeout 300 #Modified by AH 27/04/2005. Was : 10 MaxThreads 30 MaxConnectionQueueLength 15 LogFile /var/log/clamav/clamav.log LogTime LogFileMaxSize 0 PidFile /var/run/clamav/clamd.pid DatabaseDirectory /var/lib/clamav SelfCheck 3600 ScanMail ScanArchive ScanHTML ScanOLE2 ScanPE TCPSocket 3310 DetectBrokenExecutables #added by AH 27/04/2005 StreamMaxLength 20M Example of an error report : cruella:/var/log# tail -f /var/log/clamav/clamav.log Wed Apr 27 13:38:17 2005 -> Archive support enabled. Wed Apr 27 13:38:17 2005 -> Archive: RAR support disabled. Wed Apr 27 13:38:17 2005 -> Archive: Blocking encrypted archives. Wed Apr 27 13:38:17 2005 -> Archive: Blocking archives that exceed limits. Wed Apr 27 13:38:17 2005 -> Portable Executable support enabled. Wed Apr 27 13:38:17 2005 -> Detection of broken executables enabled. Wed Apr 27 13:38:17 2005 -> Mail files support enabled. Wed Apr 27 13:38:17 2005 -> OLE2 support enabled. Wed Apr 27 13:38:17 2005 -> HTML support enabled. Wed Apr 27 13:38:17 2005 -> Self checking every 3600 seconds. Wed Apr 27 13:41:21 2005 -> stream: Exploit.HTML.IFrame FOUND Wed Apr 27 13:42:42 2005 -> stream: Worm.Bagle.Gen-zippwd FOUND Wed Apr 27 13:45:09 2005 -> stream: Worm.SomeFool.P FOUND Wed Apr 27 13:45:29 2005 -> stream: Worm.SomeFool.Q FOUND Wed Apr 27 13:45:35 2005 -> stream: Worm.Mytob.A FOUND Wed Apr 27 13:46:00 2005 -> stream: Exploit.HTML.IFrame FOUND Wed Apr 27 13:47:11 2005 -> stream: Worm.SomeFool.P FOUND Wed Apr 27 13:48:06 2005 -> ERROR: ScanStream: accept timeout. Wed Apr 27 13:48:08 2005 -> ERROR: ScanStream: accept timeout. Wed Apr 27 13:48:08 2005 -> ERROR: ScanStream: accept timeout. ... Wed Apr 27 13:56:06 2005 -> ERROR: accept() failed: Too many open files Wed Apr 27 13:56:08 2005 -> ERROR: accept() failed: Too many open files Has anyone faced the same issue before ? Is there a known way to fix this problem ? Any advice ? Any help would be greatly appreciated. Thanks, Arnaud Huret ContactOffice ___ http://lurker.clamav.net/list/clamav-users.html
[Clamav-users] 'Too many open files' on a buzy clamd
Dear all, We are running a webmail service using ClamAV and get roughtly 30.000 valid mails/day. We run home-build SMTP servers calling clamd, emulating the client. The problem : After running +- 10 minutes, clamd.log reports a first message saying : 'ERROR: ScanStream: accept timeout' quickly followed other ones. After 1 or 2 minutes, we get another message : 'ERROR: accept() failed: Too many open files' and, I guess, clamd does not respond any more. Need to restart the daemon to restore the service. I tried the following tunning : 1. Increase the number of threads from 10 to 30 for reducing the queue: no changes, still errors. 2. Increase the number of MaxConnectionQueueLength to 30: no changes, still errors. Other info : Clamd runs as non-root user. Launch script is : /etc/init.d/clamav_daemon start (not modified from orginal). ClamAV is currently running and a Debian Woody with 1.5 GB mem on a 2*1Ghz Intel chassis. SpamAssassin is also running on this box. Version 3.0.2 standard (Razor, DCC, ...) Mitigating factors (;-) Running the same config on a more powerfull box does not generate the prob (2*3GH + multithreading) clamd.conf : #Automatically Generated by clamav-base postinst #To reconfigure clamd run #dpkg-reconfigure clamav-base #LocalSocket /var/run/clamav/clamd.ctl FixStaleSocket User clamav AllowSupplementaryGroups ArchiveMaxRecursion 10 ArchiveMaxFiles 1500 ArchiveMaxFileSize 30M ArchiveMaxCompressionRatio 300 ArchiveBlockEncrypted ArchiveBlockMax ReadTimeout 300 #Modified by AH 27/04/2005. Was : 10 MaxThreads 30 MaxConnectionQueueLength 15 LogFile /var/log/clamav/clamav.log LogTime LogFileMaxSize 0 PidFile /var/run/clamav/clamd.pid DatabaseDirectory /var/lib/clamav SelfCheck 3600 ScanMail ScanArchive ScanHTML ScanOLE2 ScanPE TCPSocket 3310 DetectBrokenExecutables #added by AH 27/04/2005 StreamMaxLength 20M Example of an error report : cruella:/var/log# tail -f /var/log/clamav/clamav.log Wed Apr 27 13:38:17 2005 -> Archive support enabled. Wed Apr 27 13:38:17 2005 -> Archive: RAR support disabled. Wed Apr 27 13:38:17 2005 -> Archive: Blocking encrypted archives. Wed Apr 27 13:38:17 2005 -> Archive: Blocking archives that exceed limits. Wed Apr 27 13:38:17 2005 -> Portable Executable support enabled. Wed Apr 27 13:38:17 2005 -> Detection of broken executables enabled. Wed Apr 27 13:38:17 2005 -> Mail files support enabled. Wed Apr 27 13:38:17 2005 -> OLE2 support enabled. Wed Apr 27 13:38:17 2005 -> HTML support enabled. Wed Apr 27 13:38:17 2005 -> Self checking every 3600 seconds. Wed Apr 27 13:41:21 2005 -> stream: Exploit.HTML.IFrame FOUND Wed Apr 27 13:42:42 2005 -> stream: Worm.Bagle.Gen-zippwd FOUND Wed Apr 27 13:45:09 2005 -> stream: Worm.SomeFool.P FOUND Wed Apr 27 13:45:29 2005 -> stream: Worm.SomeFool.Q FOUND Wed Apr 27 13:45:35 2005 -> stream: Worm.Mytob.A FOUND Wed Apr 27 13:46:00 2005 -> stream: Exploit.HTML.IFrame FOUND Wed Apr 27 13:47:11 2005 -> stream: Worm.SomeFool.P FOUND Wed Apr 27 13:48:06 2005 -> ERROR: ScanStream: accept timeout. Wed Apr 27 13:48:08 2005 -> ERROR: ScanStream: accept timeout. Wed Apr 27 13:48:08 2005 -> ERROR: ScanStream: accept timeout. ... Wed Apr 27 13:56:06 2005 -> ERROR: accept() failed: Too many open files Wed Apr 27 13:56:08 2005 -> ERROR: accept() failed: Too many open files Has anyone faced the same issue before ? Is there a known way to fix this problem ? Any advice ? Any help would be greatly appreciated. Thanks, Arnaud Huret ContactOffice ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Re: 0.83 potentially not catching some NetSky/SomeFool virus
> > Back to the original problem. Is Simon's answer the cause (only > > broken PE headers are detected not broken somewhere else executables)? > > Hopefully Arnaud will be able to catch one soon so we can clear up the > mystery!. > I catched two diffrent samples (NetSky.Y and Sober.gen) not catched by ClamAV but well by TrendMicro VirusWall. I submitted them through the site but I get a message saying 'already recognized'. What should I do to submit them to the team for further analysis ? Arnaud ContactOffice ___ http://lurker.clamav.net/list/clamav-users.html
Re: [Clamav-users] Re: 0.83 potentially not catching some NetSky/SomeFool virus
> So the OP has a correct configuration but his setup seems to not detect broken > executables... > > Back to the original problem. Is Simon's answer the cause (only broken PE > headers are detected not broken somewhere else executables)? > -- > René Berber As the config seems to be OK (or at least not too faulty ;-) , I'll try to catch some of these 'non-detected' examples and submit them for further analysis. Arnaud Huret ___ http://lurker.clamav.net/list/clamav-users.html
[Clamav-users] 0.83 potentially not catching some NetSky/SomeFool virus
Dear all, My apologies to the list if I waste time and bandwith. I went through the mail archive and could'nt find any (obvious) way to fix my problem. In short : we are running a 'virtual office' providing amongst different services, a webmail service. We do receive around +- 25.000 valid mails/day (more than 160.000 SMTP connexions) and installed ClamAV 0.83 on a RHEL 3.0 V4/HP DL 2*3GHZ box. We use 'home maid' java SMTP servers passing requests to clamd for evaluating each email and pass it to the final recipient in case no virus is found. FreshClam is activated and checks for new db every 2 hours. Both clamd/freshclam work very nicely, no errors, ... We started a couple of days ago and ClamAV catches more than 1800 virus a day. As we are experimenting ClamAV, we still maintain during evaluation period a second (and historic) defense line with TrendMicro VirusWall which we plan to abandon shortly. I observed that VirusWall (the second line defense) reported 8 hits on (SomeFool) Worm.Netsky.P .Y .and .W. 'DetectBrokenExecutables' is activated. (Logfiles are below). Config files is as follows (large comments stripped) : #LogFileUnlock LogFileMaxSize 0 LogTime #LogClean LogSyslog #LogFacility LOG_MAIL #LogVerbose PidFile /var/run/clamav/clamd.pid TemporaryDirectory /var/tmp DatabaseDirectory /var/clamav #LocalSocket /var/run/clamav/clamd.sock FixStaleSocket TCPSocket 3310 MaxConnectionQueueLength 30 #StreamMaxLength 20M #StreamMaxPort 32000 #MaxThreads 20 ReadTimeout 300 #IdleTimeout 60 #MaxDirectoryRecursion 20 #FollowDirectorySymlinks #FollowFileSymlinks #SelfCheck 600 #VirusEvent /usr/local/bin/send_sms 123456789 "VIRUS ALERT: %v" User clamav AllowSupplementaryGroups #ExitOnOOM #Foreground #Debug #LeaveTemporaryFiles #DisableDefaultScanOptions ## ## Executable files ## ScanPE DetectBrokenExecutables ## ## Documents ## ScanOLE2 ## ## Mail files ## ScanMail #MailFollowURLs ## ## HTML ## #ScanHTML ## ## Archives ## ScanArchive #ScanRAR ArchiveMaxFileSize 20M ArchiveMaxRecursion 10 ArchiveMaxFiles 1500 ArchiveMaxCompressionRatio 300 #ArchiveLimitMemoryUsage ArchiveBlockEncrypted ArchiveBlockMax ## ## Clamuko settings .. the rest is set to the default. After having received 32.343 mails, I got 8 hits on TrendMicro reporting these virus (apparently) not catched from ClamAV. Sort by : Date View: All Dates User: All Users Virus : All Viruses - [EMAIL] 1. Date: 04/16/2005 00:54:11 File: data.zip From: <[EMAIL PROTECTED]> To : [EMAIL PROTECTED] Action : deleted Virus : WORM_NETSKY.P [EMAIL] 2. Date: 04/16/2005 01:04:25 File: www.yahoo.fr.stlouissec.session-1292.com From: <[EMAIL PROTECTED]> To : [EMAIL PROTECTED] Action : deleted Virus : WORM_NETSKY.Y [EMAIL] 3. Date: 04/16/2005 10:42:59 File: abuselist.zip From: <[EMAIL PROTECTED]> To : [EMAIL PROTECTED] Action : deleted Virus : WORM_NETSKY.P [EMAIL] 4. Date: 04/16/2005 13:58:42 File: letter.zip From: <[EMAIL PROTECTED]> To : [EMAIL PROTECTED] Action : deleted Virus : WORM_NETSKY.P [EMAIL] 5. Date: 04/16/2005 22:13:28 File: word document.zip From: <[EMAIL PROTECTED]> To : [EMAIL PROTECTED] Action : deleted Virus : WORM_NETSKY.P [EMAIL] 6. Date: 04/17/2005 12:42:41 File: details.zip From: <[EMAIL PROTECTED]> To : [EMAIL PROTECTED] Action : deleted Virus : WORM_NETSKY.W [EMAIL] 7. Date: 04/17/2005 13:07:06 File: d4334938.zip From: To : [EMAIL PROTECTED] Action : deleted Virus : WORM_NETSKY.P [EMAIL] 8. Date: 04/17/2005 13:55:36 File: document.zip From: <[EMAIL PROTECTED]> To : [EMAIL PROTECTED] Action : deleted Virus : WORM_MyDoom.DAM Is there anything wrong in the config file ? Did I miss something ? Does anyone report/experience the same problem ? Any help would be greatly appreciated. Thanks, Arnaud Huret ContactOffice ___ http://lurker.clamav.net/list/clamav-users.html
