Re: [Clamav-users] Performance Help - 100% cpu usage
On Thu, 28 Oct 2004, Trog wrote: > > Any thoughts on how this should be accomplished? > > I may have missed it, but what are the details of your system, OS, etc. > > And, importantly, what exact version of libz (zlib) do you have > installed? Not really applicable, I'm not having stability issues. This was just discussed earlier and I was wondering if it should be persued. ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Thu, 2004-10-28 at 00:34, [EMAIL PROTECTED] wrote: > On Wed, 27 Oct 2004, Joe Maimon wrote: > > > The ClamAV authors could put a stop to this by making clamdscan and > > > clamscan the same program and then acting differently depending on > > > which name is run. This is similiar to how gzip and gunzip are > > This has been brought up before and I am surfacing it again because there > was some interest and it would add to the stability of ClamAV. Very > simply, clamdscan needs to timeout the connection to clamd after some > (sane) amount of time and run clamscan. An action could then be taken to > alert someone if clamd died (|sendmail [EMAIL PROTECTED]). When clamd > hangs on our system, mail is deferred until I realize mail has stopped and > as you can imagine, that is a bad thing. Someday I'll write a > mail-server watchdog w/ procmail and cron but I've not had time. > > Any thoughts on how this should be accomplished? I may have missed it, but what are the details of your system, OS, etc. And, importantly, what exact version of libz (zlib) do you have installed? -trog signature.asc Description: This is a digitally signed message part ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
James Lick wrote: The ClamAV authors could put a stop to this by making clamdscan and clamscan the same program and then acting differently depending on which name is run. Why? It's not a problem with clamav but a problem with broken instructions. --- Lars Hansson ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
[EMAIL PROTECTED] wrote: On Wed, 27 Oct 2004, Joe Maimon wrote: The ClamAV authors could put a stop to this by making clamdscan and clamscan the same program and then acting differently depending on which name is run. This is similiar to how gzip and gunzip are This has been brought up before and I am surfacing it again because there was some interest and it would add to the stability of ClamAV. Very simply, clamdscan needs to timeout the connection to clamd after some (sane) amount of time and run clamscan. An action could then be taken to alert someone if clamd died (|sendmail [EMAIL PROTECTED]). When clamd hangs on our system, mail is deferred until I realize mail has stopped and as you can imagine, that is a bad thing. Someday I'll write a mail-server watchdog w/ procmail and cron but I've not had time. Any thoughts on how this should be accomplished? In the clamav distribution contrib tree there is a clamwatch script (perl). It uses Unix or tcp sockets, your call. It returns 1 if clamd is running, 0 if anything bad happens. This is far better than checking only the process table (pgrep clamd or ps -ef |grep [c]lamd) as it actually tests for a known pattern, the Eicar test signature and of course exercises the entire tool. This can be run out of cron via a shell script wrapper, of course, and the return results used to run clamscan or restart clamd or let you know via email/pager that something is broken. Or all of this. Though I don't know how you might hand off the file handle without jiggering the milter or script. I use Sendmail and a third party milter (J-Chkmail) and just restart clamd if things are not right. It's not happened since 0.75.1 was released. My system is configured to tempfail the message if the milter/scanner fails and this gives me a second chance to look at the message when, hopefully, things are in better shape (hasn't happened yet) dp .. knock on wood ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
[EMAIL PROTECTED] wrote: > Oh, I completely agree, that's my job. But if clam has stability > issues, that needs to be addressed in clam. clamd->clamscan failover > code would be short and sweet and the addition to clamdscan would be > minimal compared to the cost of a complete code audit for clamd. The > mail watchdog would be specific to our server and I am not inferring > that any of you should write it. Either way, if clamd is buggy, it > should not be my duty to build a workaround but I will if clamd hasn't > stabilized. My turn to agree. Obviously, if there are stability issues, then the only place that can be addressed is within the software itself. Although, I will be honest, I have never had a single (crash|lockup|instability) with Clam. I still do believe, however, that any monitoring/fallback should be external to Clam. A shell, perl, python or whatever script is more customisable per system/platform. At the end of the day, if you have a failsafe built into software which is monitoring for it's own bugs or problems, what is to say the monitoring/fallback code may be not be susceptible to the problems it is guarding against? :) I personally have no type of monitoring running with Clam, for as I mentioned earlier, it has been rock solid for me. Matt ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Thu, 28 Oct 2004, Matt wrote: > There are ways to monitor clamd, and run clamscan if clamd is > unavailable, without expecting the software itself to do it. Clam is > fine as it is. The fault tolerance should be built around the software, > not into it. > > Not meaning to be too blunt about this, but if you have not had time to > create a watchdog for yourself, why should you expect someone else to do > the job for you? Oh, I completely agree, that's my job. But if clam has stability issues, that needs to be addressed in clam. clamd->clamscan failover code would be short and sweet and the addition to clamdscan would be minimal compared to the cost of a complete code audit for clamd. The mail watchdog would be specific to our server and I am not inferring that any of you should write it. Either way, if clamd is buggy, it should not be my duty to build a workaround but I will if clamd hasn't stabilized. -- Eric Wheeler Vice President National Security Concepts, Inc. PO Box 3567 Tualatin, OR 97062 http://www.nsci.us/ Voice: (503) 293-7656 Fax: (503) 885-0770 ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
[EMAIL PROTECTED] wrote: > On Wed, 27 Oct 2004, Joe Maimon wrote: > > > The ClamAV authors could put a stop to this by making clamdscan and > > > clamscan the same program and then acting differently depending on > > > which name is run. This is similiar to how gzip and gunzip are > > This has been brought up before and I am surfacing it again because > there was some interest and it would add to the stability of ClamAV. > Very simply, clamdscan needs to timeout the connection to clamd after > some(sane) amount of time and run clamscan. An action could then be > taken to alert someone if clamd died (|sendmail [EMAIL PROTECTED]). > When clamd hangs on our system, mail is deferred until I realize mail > has stopped and as you can imagine, that is a bad thing. Someday I'll > write a mail-server watchdog w/ procmail and cron but I've not had time. > > Any thoughts on how this should be accomplished? > There are ways to monitor clamd, and run clamscan if clamd is unavailable, without expecting the software itself to do it. Clam is fine as it is. The fault tolerance should be built around the software, not into it. Not meaning to be too blunt about this, but if you have not had time to create a watchdog for yourself, why should you expect someone else to do the job for you? Matt ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Wed, 27 Oct 2004, Joe Maimon wrote: > > The ClamAV authors could put a stop to this by making clamdscan and > > clamscan the same program and then acting differently depending on > > which name is run. This is similiar to how gzip and gunzip are This has been brought up before and I am surfacing it again because there was some interest and it would add to the stability of ClamAV. Very simply, clamdscan needs to timeout the connection to clamd after some (sane) amount of time and run clamscan. An action could then be taken to alert someone if clamd died (|sendmail [EMAIL PROTECTED]). When clamd hangs on our system, mail is deferred until I realize mail has stopped and as you can imagine, that is a bad thing. Someday I'll write a mail-server watchdog w/ procmail and cron but I've not had time. Any thoughts on how this should be accomplished? -- Eric Wheeler Vice President National Security Concepts, Inc. PO Box 3567 Tualatin, OR 97062 http://www.nsci.us/ Voice: (503) 293-7656 Fax: (503) 885-0770 ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Wed, 27 Oct 2004 20:07:20 +0800 James Lick <[EMAIL PROTECTED]> wrote: > It wouldn't be necessary to make clamscan and clamdscan the same > program > in this case. One could have clamscan check to see if it was invoked > as > clamdscan and if so refuse to run. Yes, it should be up to the end > user > to not screw up his own system, but this one issue has caused enough > grief here that such screw ups deserve a bit more direct effect. I > think it is entirely reasonable to have clamscan not work if it is > called clamdscan. I disagree. The source of the confusion must be fixed and not the victim. -- oo. Tomasz Kojm <[EMAIL PROTECTED]> (\/)\. http://www.ClamAV.net/gpg/tkojm.gpg \..._ 0DCA5A08407D5288279DB43454822DC8985A444B //\ /\ Wed Oct 27 14:21:06 CEST 2004 pgpfjFhw9DIkI.pgp Description: PGP signature ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
It wouldn't be necessary to make clamscan and clamdscan the same program in this case. One could have clamscan check to see if it was invoked as clamdscan and if so refuse to run. Yes, it should be up to the end user to not screw up his own system, but this one issue has caused enough grief here that such screw ups deserve a bit more direct effect. I think it is entirely reasonable to have clamscan not work if it is called clamdscan. -- James Lick -- éåæ -- [EMAIL PROTECTED] -- http://jameslick.com/ ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
James Lick wrote: Jason Haar wrote: I am now going to figure out a way that the installation of Qmail-Scanner will *ignore* the presense of clamdscan if its actually clamscan - that is really too gross to allow to continue. The ClamAV authors could put a stop to this by making clamdscan and clamscan the same program and then acting differently depending on which name is run. This is similiar to how gzip and gunzip are actually the same program but when called as gzip it compresses and as gunzip it uncompresses. The way I understand it most people recommend that the argv[0] mechanism be only used a) each of the programs functionality would duplicate significant portions of functionality/code b) there be a command-line switch that overrides any meaning argv[0] may have c) there should be an intelligent default Its not done nearly as often as it is _possible_ to be done, for the above reasons. Gzip and sendmail are some well known programs who do this. However, most people do not agree that sendmail is a textbook example of fine design. I believe GNU coding conventions recommends against the practice as well. Often one accomplishes the goal of (a) above by linking in some of the object files of one program to another. Or a librarywait...clamav does this already. As far as I am aware sym/hard links are currently only commonplace on unix-like systems. This would be an unneccessary hardship to the windows porters. As for the stated goal, my personal feelings is that just as users should not be trying to thwart developers, neither should developers try to thwart users. And since large portions of clamscan arguments do not apply for clamdscan, we would be provoking more confusion in that regard as well. I also suspect that there is far less similarity in the code for clamscan and clamdscan than one would expectbut I havent looked recently. As for the packager, his instructions do clearly note that it is his personal workaround preference. People who ignore that disclaimer are IMO doing so at their own risk. So are people who install complext software without reading *any* of the vendor(clamav) supplied doc. My alma mater, School Of Hard Knocks advises me that they deserve what they get. However, this list does not deserve the repeated annoyance of answering the same question. Most intelligent humans seem to feel that answering the same question repeatedly is a unique 21st century form of torture. Were I the packager, my personal workaround preference would be to configure qscan to call clamscan, instead of mucking with the clamav install. Furthermore, the documentation appears to have been updated for the .80 series -- notice the use of clamd.conf (Perhaps the workaround is meant to be overriden by newer installs of clamav?) Excuse the above rants... Joe ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Oct 26, 2004, at 4:45 AM, Eric Worthy wrote: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1290 qscand 15 0 57368 56m 696 R 50.8 5.6 172:29.51 clamdscan 25135 qscand 14 0 57368 56m 696 R 50.2 5.6 187:57.60 clamdscan 4980 qscand 15 0 57368 46m 696 R 50.2 4.6 167:42.45 clamdscan 30917 qscand 14 0 57368 56m 696 R 49.8 5.6 177:53.10 clamdscan 8861 qscand 15 0 57368 776 696 R 49.5 0.1 163:36.55 clamdscan 28183 qscand 14 0 57368 56m 696 R 49.2 5.6 182:21.71 clamdscan Is your softlimit set to about 60MB? If so I have seen a similar problem. You will need to set a limit to the size of the mails you accept. In my box I have a softlimit of 150MB and a mail size limit of 30MB. Problem surfaced with 0.75-1. Prior to that the system did not hit the softlimit and get stuck there, irrespective of mail size. Abdul Anyone have any advice on what I could be doing wrong or how to improve the performance of the scanning? Thanks, Eric *update* - 8:00pm Monday night - I rebooted and it's all back to normal for now. ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users East Coast Access Tel: 031-566-8080 Fax: 031-566-8010 Web: http://www.eastcoast.co.za ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tuesday 26 October 2004 18:47, Jim Maul shaped the electrons to say: > Scott Ryan wrote: > > > > > What are we arguing about here? I just know in my experience that you are > > seriously shooting yourself in the foot by using clamscan to scan all > > mails. Trog's suggestion of modifying qmail-scanner (if you really want > > to create the link) sounds like the sensible solution to those who use > > QMR. > > Im simply arguing the fact that someone has spent a lot of their time to > help out the community by creating the QMR setup instructions I dont think that anyone doubts that. As has been mentioned in the thread, documentation is the hardest part of any installation / build process. > and > because of some points made in that install this person is being accused > of being ignorant, stupid and breaking code. Again, I dont think that anyone thinks that the Author is 'stupid', just that the benefits of using clamdscan over clamscan is in orders of magnitude more beneficial. By suggesting to users to replace it is not wise, thats all. If you are in contact with the author maybe it is worth suggesting to him to make the change. > > Thats just flat out wrong. > > -Jim > ___ > http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users -- +--+ (0> Scott Ryan //\ Senior Unix/Linux Engineer V_/_Telkom Internet - South Africa +--+ He who controls the past, controls the future, He who controls the present, controls the past. - George Orwell, 1984 ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
Jason Haar wrote: I am now going to figure out a way that the installation of Qmail-Scanner will *ignore* the presense of clamdscan if its actually clamscan - that is really too gross to allow to continue. The ClamAV authors could put a stop to this by making clamdscan and clamscan the same program and then acting differently depending on which name is run. This is similiar to how gzip and gunzip are actually the same program but when called as gzip it compresses and as gunzip it uncompresses. -- James Lick -- éåæ -- [EMAIL PROTECTED] -- http://jameslick.com/ ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tue, Oct 26, 2004 at 03:56:20PM +0100, Trog wrote: > You don't think it's rude to break other peoples software, for which we > then have to deal with the resulting mess, as witnessed by this thread? Indeed. We now have two mailing-lists (clamav and Qmail-Scanner) where people who followed this example now have broken/underperforming systems. I appreciate the work the guy has done - writing documentation is the hardest part of development! But that sentence has done more harm than good. I am now going to figure out a way that the installation of Qmail-Scanner will *ignore* the presense of clamdscan if its actually clamscan - that is really too gross to allow to continue. I hope someone has contacted the author...? He can't fix what he doesn't know is a problem... -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tue, Oct 26, 2004 at 10:01:24AM -0400, Jim Maul said: > Keep in mind while i agree the instructions are a little messed up for > the current versions of the software it uses, the instructions are the > way they are to correct problems and certain small errors that occured > in older versions of the sofware. Basically the instructions are > outdated in my opinion. I dont believe the reason for the > clamscan/clamdscan linking is still a valid reason as well as other > "workarounds" that were put in place. The instructions should be updated. > > With that said, the person, yes, only 1 person, who created qmr is > obviously busy and this is not his full time job. I think it is great > that he has taken this amount of time out of his everyday life to > provide this great service for everyonecut him some slack will ya? > Saying "for no reason other than ignorance and gross stupidity" is quite > incorrect and even downright rude. You have NO idea why he set up the > instructions this way and you yourself are making huge assumptions. If > you have some constructive criticism here im sure it would be > appreciated but you previous comments were IMO not helpful at all. Bear in mind that this is how most packaging is done. I am a single person, handling most of the work for clamav for Debian (Thomas, hi! - does help me from time to time, but he has a full life as well). I doubt that the other packagers are working in teams to get clam into their distributions either. I grant you, each of us doing this work probably has bugs in our work - we are all human, after all. Just look at the debian BTS page for clamav - it'll give you an idea of the bugs I've introduced :) All that being said, I do not like packagers that break upstream assumptions more than necessary - replacing clamdscan with clamscan is just silly. Leave it alone, and put a big fat warning CLAMD HAS HAD STABILITY PROBLEMS - I RECOMMEND CLAMSCAN!!! all over the webpage. I have notes about stability problems in the README where appropriate. I fell that trying to obscure the normal functionality of a software package in order to make something point-n-click is doing no one a service, and making it harder to debug the real problems in it. Thanks for listening, -- -- | Stephen Gran | Conscience is a mother-in-law whose | | [EMAIL PROTECTED] | visit never ends. -- H. L. Mencken| | http://www.lobefin.net/~steve | | -- pgptIyWWlbgo7.pgp Description: PGP signature ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tue, 26 Oct 2004, Scott Ryan wrote: > > You can get some mileage by putting your MTA's temp dir on a shmfs/tmpvs or > > other type of VM filesystem if you're on a different OS to reduce the disk > > i/o cycles. By freeing I/O cycles, the cpus can do more *real* work and > > not wait precious cycles on io time. On a 2.6 kernel, vmstat will tell > > you io-wait time (wa) get a feel for where the bottle neck is. > > This can be dangerous. If your mta stores any mail here for whatever reason > and the box (again for whatever reason) reboots/dies - you lose all that > mail. Scott makes a very good point. If you don't trust your systems uptime, do not use a vmfs. This for the same reason you don't turn on write-back caching on your raid controller without a battery. I had actually meant the temp directory used by programs like amavis which break the message into file pieces, not the MTA's spool itself. All though, once upon a time we had a controller card which was dying and it kept timing out on IRQs. Moving our mail spool to a vmfs allowed the server to continue to process mail without the long io hang/timeouts until we were able to replace the card (yes, that was a long day). -- Eric Wheeler Vice President National Security Concepts, Inc. PO Box 3567 Tualatin, OR 97062 http://www.nsci.us/ Voice: (503) 293-7656 Fax: (503) 885-0770 ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tue, 26 Oct 2004, Jim Maul wrote: > Im simply arguing the fact that someone has spent a lot of their time to > help out the community by creating the QMR setup instructions[.] Jim is right here. Without the community we wouldn't be where we are. If anything, QMR has taught us something: clamdscan+clamd is faster than clamscan by itself. In fact, since Google kindly traverses archives across the web, when someone has a similar or related problem you can pretty much guess that this thread /will/ come up. That being said, we have done our job by having this discussion and the community has benefited from it. Agree or disagree, both Trog, Jim and others in this discussion have helped by simply discussing the issue. In a month, Google for QMR and clamscan. I bet you'll get the answer to the question :) -- Eric Wheeler Vice President National Security Concepts, Inc. PO Box 3567 Tualatin, OR 97062 http://www.nsci.us/ Voice: (503) 293-7656 Fax: (503) 885-0770 ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
> Scott Ryan wrote: > > >> >> What are we arguing about here? I just know in my experience that you >> are >> seriously shooting yourself in the foot by using clamscan to scan all >> mails. >> Trog's suggestion of modifying qmail-scanner (if you really want to >> create the >> link) sounds like the sensible solution to those who use QMR. >> > > Im simply arguing the fact that someone has spent a lot of their time to > help out the community by creating the QMR setup instructions and > because of some points made in that install this person is being accused > of being ignorant, stupid and breaking code. > > Thats just flat out wrong. > > -Jim > ___ > http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users > Ok, first of all, thank you all for replying to my thread. Yes, everyone can call me an idiot for not knowing 100% of what I'm doing. There are plenty of 'part time' sysadmins out there doing what I do. I work a full time job, and am helping out a small wireless company with their mail server. I know enough to get in trouble I guess. Some would say with 2200 users now, that this 'small' company needs to spend some money on a 'real SA'. I would agree with this but we're up against Comcast and Qwest and if we can get by without losing any email for the next year, we will be able to hire a real SA full time. So, now you know my situation. I do appreciate the feedback. Funny thing, is after I rebooted the server, things have been going great. The cpu's are running about 20% and we're handling over 1100 emails an hour. For some reason, (from my original post) that clamdscan was taking up all the cpu. Now, after reboot, it is fine. So, my next plan is to start reading up more on clam and how to build it from source and to take all the replies back and figure out how they will help with my performace. We're signing up 8-12 new subscribers a day so this is not an end to this. Thanks again for the replies. I didn't mean to cause any arguments. I do think qmailrocks.org and others sites are VERY valuable to people like me that don't have time to do things the right way. If the linux community really wants people to use their software, they need to come up with great pages like his (even though there are some problems with it). That being said, in my real job, if someone came to me and said they put 2000 uers on an Oracle database without thinking it through, I would tell them they are stupid. So, you can look at it both ways. Thanks, Eric ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
Scott Ryan wrote: What are we arguing about here? I just know in my experience that you are seriously shooting yourself in the foot by using clamscan to scan all mails. Trog's suggestion of modifying qmail-scanner (if you really want to create the link) sounds like the sensible solution to those who use QMR. Im simply arguing the fact that someone has spent a lot of their time to help out the community by creating the QMR setup instructions and because of some points made in that install this person is being accused of being ignorant, stupid and breaking code. Thats just flat out wrong. -Jim ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tuesday 26 October 2004 17:57, Jim Maul shaped the electrons to say: > Scott Ryan wrote: > > On Tuesday 26 October 2004 09:52, Trog shaped the electrons to say: > >>On Tue, 2004-10-26 at 03:45, Eric Worthy wrote: > >>>This is a vanilla install off qmailrocks.org site. > >> > >>This may be your problem. I seem to remember they are guilty of doing > >>very bad things to the clamav install, like linking clamdscan to > >>clamscan. > > > > Why the hell would they want to suggest that?? This would totally limit > > the ability to scale. Are there any docs suggesting what the 'benefits' > > are? When I took over here at my current job, qmailscanner was setup to > > use clamscan instead of clamdscan. We send/receive over a million mails a > > day and the cpus were sitting at 100% constantly. The first thing i did > > was to change to clamdscan and cpu usage dropped unbelievably. > > First off, the QMR install is for people who are new to this type of > setup and is NOT meant to be used in a full large volume production > environment. I dont dispute that. > If you are using the QMR setup in this type of enviroment > its your own damn fault. The suggested linking of clamdscan to clamscan > was done to eliminate usage of clamd which at the time (i believe around > ver 0.6 or so) there were some serious stbility issues > To avoid these issues the site author just suggested the linking. This is why i > suggested that the instructions are out of date. Yes the site was > updated recently, but no, this text was not changed. What are we arguing about here? I just know in my experience that you are seriously shooting yourself in the foot by using clamscan to scan all mails. Trog's suggestion of modifying qmail-scanner (if you really want to create the link) sounds like the sensible solution to those who use QMR. -- +--+ (0> Scott Ryan //\ Senior Unix/Linux Engineer V_/_Telkom Internet - South Africa +--+ He who controls the past, controls the future, He who controls the present, controls the past. - George Orwell, 1984 ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
Jim Maul wrote: > > Why the hell would they want to suggest that?? This would totally > > limit the ability to scale. Are there any docs suggesting what the > > 'benefits' are? When I took over here at my current job, qmailscanner > > was setup to use clamscan instead of clamdscan. We send/receive over a > > million mails a day and the cpus were sitting at 100% constantly. The > > first thing i did was to change to clamdscan and cpu usage dropped > > unbelievably. > > > > First off, the QMR install is for people who are new to this type of > setup and is NOT meant to be used in a full large volume production > environment. If you are using the QMR setup in this type of enviroment > its your own damn fault. The suggested linking of clamdscan to clamscan > > was done to eliminate usage of clamd which at the time (i believe around > > ver 0.6 or so) there were some serious stbility issues. To avoid these > issues the site author just suggested the linking. This is why i > suggested that the instructions are out of date. Yes the site was > updated recently, but no, this text was not changed. It will be handbags at dawn soon :) As with any tutorial, it should only be used as a general guide. Anyone who follows a tutorial to the letter really are asking for problems. The problems tutorials cause with individual installations are down to the laziness of the installer. There is ample documentation supplied by the Clam team. A percentage of people just can't be arsed to spend anytime learning, which is why these problems arise. They take the quick approach, instead of doing some initial revision. As an example, how many times do the same questions get asked on M.L's, when the answers are in the archives? ( Dog collar off :) Matt ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
Scott Ryan wrote: On Tuesday 26 October 2004 16:56, Trog shaped the electrons to say: On Tue, 2004-10-26 at 15:01, Jim Maul wrote: Keep in mind while i agree the instructions are a little messed up for the current versions of the software it uses, the instructions are the way they are to correct problems and certain small errors that occured in older versions of the sofware. Basically the instructions are outdated in my opinion. I dont believe the reason for the clamscan/clamdscan linking is still a valid reason as well as other "workarounds" that were put in place. The instructions should be updated. They were updated four days ago, and they are still grossly wrong. Yes the site was updated days ago. This was most likely to replace some tarballs with newer versions, not to update the text of the website. Just because a site update was made does not mean the instructions are up to date. Again you are making assumptions. With that said, the person, yes, only 1 person, who created qmr is obviously busy and this is not his full time job. I think it is great that he has taken this amount of time out of his everyday life to provide this great service for everyonecut him some slack will ya? I believe the same problems have been in there for over a year. Many people have used the QMR instructions and have had great success. I have been running it for almost a year now with no problems. It works great. Yes, i undid the linking of clamdscan and have switched to clamdscan, but this was a personal preference of mine and i made the change after i was comfortable with the setup and knew a little more about how it worked. Saying "for no reason other than ignorance and gross stupidity" is quite incorrect and even downright rude. You don't think it's rude to break other peoples software, for which we then have to deal with the resulting mess, as witnessed by this thread? your software is not broken, you are grossly exagerating the situation. The software is being used in a less than ideal way but for most people this does not create a resulting mess. Obviously you only hear the people that have problems on these lists. People dont post to say "hey my install has been working great nonstop!" You have NO idea why he set up the instructions this way and you yourself are making huge assumptions. If you have some constructive criticism here im sure it would be appreciated but you previous comments were IMO not helpful at all. 1. Install ClamAV as per it's documentation, and then don't break it by linking clamdscan to clamscan. Again, its hardly broken. 2. If you want to use clamscan rather than clamdscan (for no reason, other then to send your CPU load to 100%, as per this thread), configure qmail-scanner to do so, it has a configure option for this). The reason this was done was to work around bugs in older versions of clamav, not to intentionally raise CPU Load. i hope you dont seriously believe that. If you bothered to read the site a little further you would have seen the explanation as to why this linking was done. 'And the winner is by way of knockout... in the red corner - Trog!' You are being very immature about this. And finally, if anyone has issues with the QMR install, i would suggest you fire off a quick email to the maintainer and offer your constructive criticism. Hes a great guy and has offered tons of support for those in need. Obviously the clamav list is not the place to discuss this and im sorry it has even gone this far. -Jim ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
Scott Ryan wrote: On Tuesday 26 October 2004 09:52, Trog shaped the electrons to say: On Tue, 2004-10-26 at 03:45, Eric Worthy wrote: This is a vanilla install off qmailrocks.org site. This may be your problem. I seem to remember they are guilty of doing very bad things to the clamav install, like linking clamdscan to clamscan. Why the hell would they want to suggest that?? This would totally limit the ability to scale. Are there any docs suggesting what the 'benefits' are? When I took over here at my current job, qmailscanner was setup to use clamscan instead of clamdscan. We send/receive over a million mails a day and the cpus were sitting at 100% constantly. The first thing i did was to change to clamdscan and cpu usage dropped unbelievably. First off, the QMR install is for people who are new to this type of setup and is NOT meant to be used in a full large volume production environment. If you are using the QMR setup in this type of enviroment its your own damn fault. The suggested linking of clamdscan to clamscan was done to eliminate usage of clamd which at the time (i believe around ver 0.6 or so) there were some serious stbility issues. To avoid these issues the site author just suggested the linking. This is why i suggested that the instructions are out of date. Yes the site was updated recently, but no, this text was not changed. -Jim ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tuesday 26 October 2004 16:42, [EMAIL PROTECTED] shaped the electrons to say: > On Mon, 25 Oct 2004, Eric Worthy wrote: > > Anyone have any advice on what I could be doing wrong or how to improve > > the performance of the scanning? > > We always get a great performance boost in software by adding > -march=(yourcpuhere) -O2 -fomit-frame-pointer -static to the build lines. > If you can build static binaries an additional CPU register is available > for function calls (EDX iirc). If you're a quad p3 xeon, you want > -march=i686. You might also play with benching -O2 vs -03. We seem to > get mixed results depending on the nature of the software. Some perform > worse at -O3 and except for some weirdness in loop unrolling, I'm not sure > why O3 would give slower performance. Make sure that this happens for at > least clam and clamd. (caveot: some optimizations can create instability > so test it). Clam uses many libraries like libz and rebuilding those > dependent libraries may help as well (may not matter if staticly linked?) > > Another point to look at is disk o/i bottle neck. Mail has a tendency to > write-copy-write-copy-write especially when you have a scanner and an MTA. > This creation and deletion of spool files makes for a lot of journal > traffic (ext3/reiser, assuming Linux) to the hard drive. Unfortunately, > journal traffic is largely synchronous so it can rollback transactions in > the event of a failure. Filesystem noatime,notail options are your friend > here. A good solution here is to have a seperate disk preferably on a hardware raid controller for your mail queue (/var/qmail/queue if you use qmail). That coupled with reiserfs with blocksize 4096 hugely increases performance. If you are lucky to have a fibre channel SAN, you can put the queue on there for uber performance! > > You can get some mileage by putting your MTA's temp dir on a shmfs/tmpvs or > other type of VM filesystem if you're on a different OS to reduce the disk > i/o cycles. By freeing I/O cycles, the cpus can do more *real* work and > not wait precious cycles on io time. On a 2.6 kernel, vmstat will tell > you io-wait time (wa) get a feel for where the bottle neck is. This can be dangerous. If your mta stores any mail here for whatever reason and the box (again for whatever reason) reboots/dies - you lose all that mail. > > Hope this helps. We're constantly fighting io wait here due to the virus > spam and message spool/accounting database and currently our bottleneck is > definitely disk, not cpu (3.2ghz p4-ht). Same here. We use qmail and find that 128Mb Raid controller for the queue dir increase the I/O immensely. Reiserfs helps as well. -- +--+ (0> Scott Ryan //\ Senior Unix/Linux Engineer V_/_Telkom Internet - South Africa +--+ He who controls the past, controls the future, He who controls the present, controls the past. - George Orwell, 1984 ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tuesday 26 October 2004 16:56, Trog shaped the electrons to say: > On Tue, 2004-10-26 at 15:01, Jim Maul wrote: > > Keep in mind while i agree the instructions are a little messed up for > > the current versions of the software it uses, the instructions are the > > way they are to correct problems and certain small errors that occured > > in older versions of the sofware. Basically the instructions are > > outdated in my opinion. I dont believe the reason for the > > clamscan/clamdscan linking is still a valid reason as well as other > > "workarounds" that were put in place. The instructions should be > > updated. > > They were updated four days ago, and they are still grossly wrong. > > > With that said, the person, yes, only 1 person, who created qmr is > > obviously busy and this is not his full time job. I think it is great > > that he has taken this amount of time out of his everyday life to > > provide this great service for everyonecut him some slack will ya? > > I believe the same problems have been in there for over a year. > > > Saying "for no reason other than ignorance and gross stupidity" is quite > > incorrect and even downright rude. > > You don't think it's rude to break other peoples software, for which we > then have to deal with the resulting mess, as witnessed by this thread? > > > You have NO idea why he set up the > > instructions this way and you yourself are making huge assumptions. If > > you have some constructive criticism here im sure it would be > > appreciated but you previous comments were IMO not helpful at all. > > 1. Install ClamAV as per it's documentation, and then don't break it by > linking clamdscan to clamscan. > > 2. If you want to use clamscan rather than clamdscan (for no reason, > other then to send your CPU load to 100%, as per this thread), configure > qmail-scanner to do so, it has a configure option for this). 'And the winner is by way of knockout... in the red corner - Trog!' -- +--+ (0> Scott Ryan //\ Senior Unix/Linux Engineer V_/_Telkom Internet - South Africa +--+ He who controls the past, controls the future, He who controls the present, controls the past. - George Orwell, 1984 ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tuesday 26 October 2004 09:52, Trog shaped the electrons to say: > On Tue, 2004-10-26 at 03:45, Eric Worthy wrote: > > This is a vanilla install off qmailrocks.org site. > > This may be your problem. I seem to remember they are guilty of doing > very bad things to the clamav install, like linking clamdscan to > clamscan. Why the hell would they want to suggest that?? This would totally limit the ability to scale. Are there any docs suggesting what the 'benefits' are? When I took over here at my current job, qmailscanner was setup to use clamscan instead of clamdscan. We send/receive over a million mails a day and the cpus were sitting at 100% constantly. The first thing i did was to change to clamdscan and cpu usage dropped unbelievably. -- +--+ (0> Scott Ryan //\ Senior Unix/Linux Engineer V_/_Telkom Internet - South Africa +--+ He who controls the past, controls the future, He who controls the present, controls the past. - George Orwell, 1984 ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
Christopher X. Candreva wrote: On Tue, 26 Oct 2004, Niek wrote: QMR delivers the community with the open source equivelant of 'next, next, next, next, next, next, finish' installations. Is this a good thing ? Seriously. My pet peve with Windows installs is starting a 1 hour install, going to lunch, comming back, and finding out 5 minutes in it's sitting at a "click Next" window for no good reason. why is Next wait Next wait Next wait finish preferable to ./configure && make && make test && make install come back after luinch and it's done ? Its not really a next,wait,repeat install as was stated originally. I think the point he was trying to get across is that its the easy version of a potential difficult install. Regardless, some people seem to have issues with it...but to each their own. -Jim ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tue, 26 Oct 2004, Niek wrote: > QMR delivers the community with the open source equivelant of > 'next, next, next, next, next, next, finish' installations. Is this a good thing ? Seriously. My pet peve with Windows installs is starting a 1 hour install, going to lunch, comming back, and finding out 5 minutes in it's sitting at a "click Next" window for no good reason. why is Next wait Next wait Next wait finish preferable to ./configure && make && make test && make install come back after luinch and it's done ? == Chris Candreva -- [EMAIL PROTECTED] -- (914) 967-7816 WestNet Internet Services of Westchester http://www.westnet.com/ ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Mon, 25 Oct 2004, Eric Worthy wrote: > > Anyone have any advice on what I could be doing wrong or how to improve > the performance of the scanning? We always get a great performance boost in software by adding -march=(yourcpuhere) -O2 -fomit-frame-pointer -static to the build lines. If you can build static binaries an additional CPU register is available for function calls (EDX iirc). If you're a quad p3 xeon, you want -march=i686. You might also play with benching -O2 vs -03. We seem to get mixed results depending on the nature of the software. Some perform worse at -O3 and except for some weirdness in loop unrolling, I'm not sure why O3 would give slower performance. Make sure that this happens for at least clam and clamd. (caveot: some optimizations can create instability so test it). Clam uses many libraries like libz and rebuilding those dependent libraries may help as well (may not matter if staticly linked?) Another point to look at is disk o/i bottle neck. Mail has a tendency to write-copy-write-copy-write especially when you have a scanner and an MTA. This creation and deletion of spool files makes for a lot of journal traffic (ext3/reiser, assuming Linux) to the hard drive. Unfortunately, journal traffic is largely synchronous so it can rollback transactions in the event of a failure. Filesystem noatime,notail options are your friend here. You can get some mileage by putting your MTA's temp dir on a shmfs/tmpvs or other type of VM filesystem if you're on a different OS to reduce the disk i/o cycles. By freeing I/O cycles, the cpus can do more *real* work and not wait precious cycles on io time. On a 2.6 kernel, vmstat will tell you io-wait time (wa) get a feel for where the bottle neck is. Hope this helps. We're constantly fighting io wait here due to the virus spam and message spool/accounting database and currently our bottleneck is definitely disk, not cpu (3.2ghz p4-ht). -- Eric Wheeler Vice President National Security Concepts, Inc. PO Box 3567 Tualatin, OR 97062 http://www.nsci.us/ Voice: (503) 293-7656 Fax: (503) 885-0770 ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tue, 2004-10-26 at 15:01, Jim Maul wrote: > Keep in mind while i agree the instructions are a little messed up for > the current versions of the software it uses, the instructions are the > way they are to correct problems and certain small errors that occured > in older versions of the sofware. Basically the instructions are > outdated in my opinion. I dont believe the reason for the > clamscan/clamdscan linking is still a valid reason as well as other > "workarounds" that were put in place. The instructions should be updated. They were updated four days ago, and they are still grossly wrong. > > With that said, the person, yes, only 1 person, who created qmr is > obviously busy and this is not his full time job. I think it is great > that he has taken this amount of time out of his everyday life to > provide this great service for everyonecut him some slack will ya? I believe the same problems have been in there for over a year. > Saying "for no reason other than ignorance and gross stupidity" is quite > incorrect and even downright rude. You don't think it's rude to break other peoples software, for which we then have to deal with the resulting mess, as witnessed by this thread? > You have NO idea why he set up the > instructions this way and you yourself are making huge assumptions. If > you have some constructive criticism here im sure it would be > appreciated but you previous comments were IMO not helpful at all. 1. Install ClamAV as per it's documentation, and then don't break it by linking clamdscan to clamscan. 2. If you want to use clamscan rather than clamdscan (for no reason, other then to send your CPU load to 100%, as per this thread), configure qmail-scanner to do so, it has a configure option for this). -trog signature.asc Description: This is a digitally signed message part ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Monday 25 October 2004 23:17, Fajar A. Nugraha wrote: > Try 0.80. If it's not in debian's apt list, build it manually from source. You could also get it from backports.org: http://www.backports.org/package.php?search=clamav Regards Gerardo Reynoso ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On 10/26/2004 3:33 PM +0200, Trog wrote: So, I was correct, QMR completely screws up the ClamAV installation for no reason other than ignorance and gross stupidity. It also tells it's misguided users to run freshclam on-the-hour. Another bad decision. So, don't follow anything they say about installing ClamAV, and you'll be ok. -trog QMR delivers the community with the open source equivelant of 'next, next, next, next, next, next, finish' installations. Regards, Niek -- Use plain text: http://www.geoapps.com/nomime.shtml Learn to quote:http://www.netmeister.org/news/learn2quote2.html Avoid disclaimers: http://www.goldmark.org/jeff/stupid-disclaimers ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
Trog wrote: On Tue, 2004-10-26 at 14:41, Niek wrote: On 10/26/2004 3:33 PM +0200, Trog wrote: So, I was correct, QMR completely screws up the ClamAV installation for no reason other than ignorance and gross stupidity. It also tells it's misguided users to run freshclam on-the-hour. Another bad decision. So, don't follow anything they say about installing ClamAV, and you'll be ok. -trog QMR delivers the community with the open source equivelant of 'next, next, next, next, next, next, finish' installations. I don't have a problem with the theory, just they way they have fd up the practice. Keep in mind while i agree the instructions are a little messed up for the current versions of the software it uses, the instructions are the way they are to correct problems and certain small errors that occured in older versions of the sofware. Basically the instructions are outdated in my opinion. I dont believe the reason for the clamscan/clamdscan linking is still a valid reason as well as other "workarounds" that were put in place. The instructions should be updated. With that said, the person, yes, only 1 person, who created qmr is obviously busy and this is not his full time job. I think it is great that he has taken this amount of time out of his everyday life to provide this great service for everyonecut him some slack will ya? Saying "for no reason other than ignorance and gross stupidity" is quite incorrect and even downright rude. You have NO idea why he set up the instructions this way and you yourself are making huge assumptions. If you have some constructive criticism here im sure it would be appreciated but you previous comments were IMO not helpful at all. -Jim ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tue, 2004-10-26 at 14:41, Niek wrote: > On 10/26/2004 3:33 PM +0200, Trog wrote: > > So, I was correct, QMR completely screws up the ClamAV installation for > > no reason other than ignorance and gross stupidity. > > > > It also tells it's misguided users to run freshclam on-the-hour. Another > > bad decision. > > > > So, don't follow anything they say about installing ClamAV, and you'll > > be ok. > > > > -trog > > QMR delivers the community with the open source equivelant of > 'next, next, next, next, next, next, finish' installations. I don't have a problem with the theory, just they way they have fd up the practice. -trog signature.asc Description: This is a digitally signed message part ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tue, 2004-10-26 at 14:20, Jim Maul wrote: > Trog wrote: > > On Tue, 2004-10-26 at 03:45, Eric Worthy wrote: > > > >>This is a vanilla install off qmailrocks.org site. > > > > > > This may be your problem. I seem to remember they are guilty of doing > > very bad things to the clamav install, like linking clamdscan to > > clamscan. > > > the QMR install doesnt really do very bad things. It clearly explains > why it does what it does and that for high volume servers you may not > want to follow the directions entirely. Yes, it links clamdscan to > clamscan so you are ALWAYS calling clamscan once per message. On a high > volume server this isnt ideal. I would undo this linking, start clamd > and run the real clamdscan as opposed to the linked one. Your > performance should get noticeably better. So, I was correct, QMR completely screws up the ClamAV installation for no reason other than ignorance and gross stupidity. It also tells it's misguided users to run freshclam on-the-hour. Another bad decision. So, don't follow anything they say about installing ClamAV, and you'll be ok. -trog signature.asc Description: This is a digitally signed message part ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
Trog wrote: On Tue, 2004-10-26 at 03:45, Eric Worthy wrote: This is a vanilla install off qmailrocks.org site. This may be your problem. I seem to remember they are guilty of doing very bad things to the clamav install, like linking clamdscan to clamscan. -trog the QMR install doesnt really do very bad things. It clearly explains why it does what it does and that for high volume servers you may not want to follow the directions entirely. Yes, it links clamdscan to clamscan so you are ALWAYS calling clamscan once per message. On a high volume server this isnt ideal. I would undo this linking, start clamd and run the real clamdscan as opposed to the linked one. Your performance should get noticeably better. -Jim ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
I have the same problem of almost 100% CPU usage Can you kindly provide me with your clamd.conf? Best Regards, Kareem Mahgoub - Original Message - From: "Odhiambo Washington" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Tuesday, October 26, 2004 8:05 AM Subject: Re: [Clamav-users] Performance Help - 100% cpu usage > * Eric Worthy <[EMAIL PROTECTED]> [20041026 07:02]: wrote: > > I built a new 4 cpu/1 gig ram qmail/vpopmail/qmailscanner/clamv > > mail server. (Four (4) Pentium® III Xeon 700 MHz/ 1 MB Cache) > > I put it in last Thursday with it running great, then > > yesterday, about 6pm, the cpu usage went to near 100% with about 800 > > smtp transfers per hour. > > > > This morning about 8am, the cpu is at 100% and we're running about > > 1400 smtp transfers per hour. > > > > It appears that even this box can't keep up with all the scanning > > that has to take place. Here is top at this hour: > > http://t10.net/cpu.jpg > > http://t10.net/cpu2.jpg > > > > Tasks: 118 total, 7 running, 111 sleeping, 0 stopped, 0 zombie > > Cpu0 : 69.2% user, 30.8% system, 0.0% nice, 0.0% idle > > Cpu1 : 75.6% user, 24.4% system, 0.0% nice, 0.0% idle > > Cpu2 : 73.1% user, 26.9% system, 0.0% nice, 0.0% idle > > Cpu3 : 76.3% user, 23.7% system, 0.0% nice, 0.0% idle > > Mem: 1032988k total, 836408k used, 196580k free, 37472k buffers > > Swap: 128480k total, 93060k used, 35420k free, 351288k cached > > > I would suggest that you DisableDefaultScanOptions in clamd.conf > and tune values according to your system. My servers do slightly > more than 800 smtp transfers per hour and I found out that working > with the DisableDefaultScanOptions commented out brought my server > to its knees. And my server is almost like yours, except it's Pentium > III Xeon 500MHz. > > If you prefer, I can privately e-mail you my clamd.conf that works for > me. You can test it out. It's not the best tuned, especially regarding: > > MaxConnectionQueueLength > MaxThreads > > I still need to know how those two values should be determined. > What to base their values on. > > At any given moment, my SMTP service has average 300 child processes > so I used that value for MaxConnectionQueueLength. I am not sure > that is quite what it should be, but works for me is the key thing ;) > > I'll appreciate any contributions towards making decisions on those > two values. > > > > -Wash > > http://www.netmeister.org/news/learn2quote.html > > -- > +==+ > |\ _,,,---,,_ | Odhiambo Washington<[EMAIL PROTECTED]> > Zzz /,`.-'`'-. ;-;;,_ | Wananchi Online Ltd. www.wananchi.com >|,4- ) )-,_. ,\ ( `'-'| Tel: +254 20 313985-9 +254 20 313922 > '---''(_/--' `-'\_) | GSM: +254 722 743223 +254 733 744121 > +==+ > Have you ever noticed that the people who are always trying to tell > you, "There's a time for work and a time for play," never find the time > for play? > ___ > http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users > ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
* Doug Hardie <[EMAIL PROTECTED]> [20041026 10:30]: wrote: > > On Oct 25, 2004, at 23:05, Odhiambo Washington wrote: > > >I would suggest that you DisableDefaultScanOptions in clamd.conf > >and tune values according to your system. My servers do slightly > >more than 800 smtp transfers per hour and I found out that working > >with the DisableDefaultScanOptions commented out brought my server > >to its knees. And my server is almost like yours, except it's Pentium > >III Xeon 500MHz. > > > >At any given moment, my SMTP service has average 300 child processes > >so I used that value for MaxConnectionQueueLength. I am not sure > >that is quite what it should be, but works for me is the key thing ;) > > Those numbers seem unusual to me. I am handling over 2800 emails per > hour. I don't recall ever seeing more than about 50 sendmail child > processes active (except after an extended down period but even then it > doesn't seem to get much above 150). > > CPU: Pentium III/Pentium III Xeon/Celeron (701.59-MHz 686-class CPU) > > Single processor, FreeBSD 4.6, clamav 0.80. CPU utilization sits > between 80 and 95% idle. That is not any extraordinary, I believe. I don't run sendmail. I run Exim. Please see my SMTP stats as of yesterday at the following URL: http://ns2.wananchi.com/~wash/Eximstats/1/ Login as guest/guest I run a server that handles mail for more than 1750 domains and it handles both inbound as well as outbound. I run ClamAv CVS version. My uptime says: 11:57am up 9 days, 20:03, 3 users, load averages: 5.38, 5.07, 4.51 So as you can see, my load averages talk for themselves. I pass all mail less than 2MB to clamd for scanning. Anything above that size is not scanned at all. (pls don't send me viruses;)) Here is also a glimpse of the number of queue runners, and some output from 'top`:: ]1;ns2:/home/[EMAIL PROTECTED] ~] 59$ ps ax | grep exim | wc -l 279 2:/home/[EMAIL PROTECTED] ~] 60$ ps ax | grep exim | wc -l 287 2:/home/[EMAIL PROTECTED] ~] 61$ ps ax | grep exim | wc -l 309 2:/home/[EMAIL PROTECTED] ~] 62$ ps ax | grep exim | wc -l 303 2:/home/[EMAIL PROTECTED] ~] 63$ ps ax | grep exim | wc -l 300 2:/home/[EMAIL PROTECTED] ~] 64$ ps ax | grep exim | wc -l 301 2:/home/[EMAIL PROTECTED] ~] PID USERNAME PRI NICE SIZERES STATE C TIME WCPUCPU COMMAND 75616 exim2 0 16800K 6344K poll 1 18.9H 20.26% 20.26% clamd Note the WCPU and CPU values - they are normally that high, but the memory footprint is always constant as the values you see. So, yes, perhaps there is something I haven't configured properly as I mentioned earlier. I have to see those percentages for CPU usage. Here is a snippet of my dmesg: FreeBSD 4.10-STABLE #0: Fri Oct 1 09:33:18 EAT 2004 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/SRV4.x Timecounter "i8254" frequency 1193182 Hz CPU: Pentium III/Pentium III Xeon/Celeron (500.03-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x672 Stepping = 2 Features=0x387fbff real memory = 1073676288 (1048512K bytes) avail memory = 1041108992 (1016708K bytes) Programming 24 pins in IOAPIC #0 IOAPIC #0 intpin 2 -> irq 0 FreeBSD/SMP: Multiprocessor motherboard: 3 CPUs cpu0 (BSP): apic id: 2, version: 0x00040011, at 0xfee0 cpu1 (AP): apic id: 0, version: 0x00040011, at 0xfee0 cpu2 (AP): apic id: 1, version: 0x00040011, at 0xfee0 io0 (APIC): apic id: 3, version: 0x00170011, at 0xfec0 Preloaded elf kernel "kernel" at 0xc03bc000. -Wash http://www.netmeister.org/news/learn2quote.html -- +==+ |\ _,,,---,,_ | Odhiambo Washington<[EMAIL PROTECTED]> Zzz /,`.-'`'-. ;-;;,_ | Wananchi Online Ltd. www.wananchi.com |,4- ) )-,_. ,\ ( `'-'| Tel: +254 20 313985-9 +254 20 313922 '---''(_/--' `-'\_) | GSM: +254 722 743223 +254 733 744121 +==+ Please ignore previous fortune. ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Tue, 2004-10-26 at 03:45, Eric Worthy wrote: > > This is a vanilla install off qmailrocks.org site. This may be your problem. I seem to remember they are guilty of doing very bad things to the clamav install, like linking clamdscan to clamscan. -trog signature.asc Description: This is a digitally signed message part ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
On Oct 25, 2004, at 23:05, Odhiambo Washington wrote: I would suggest that you DisableDefaultScanOptions in clamd.conf and tune values according to your system. My servers do slightly more than 800 smtp transfers per hour and I found out that working with the DisableDefaultScanOptions commented out brought my server to its knees. And my server is almost like yours, except it's Pentium III Xeon 500MHz. At any given moment, my SMTP service has average 300 child processes so I used that value for MaxConnectionQueueLength. I am not sure that is quite what it should be, but works for me is the key thing ;) Those numbers seem unusual to me. I am handling over 2800 emails per hour. I don't recall ever seeing more than about 50 sendmail child processes active (except after an extended down period but even then it doesn't seem to get much above 150). CPU: Pentium III/Pentium III Xeon/Celeron (701.59-MHz 686-class CPU) Single processor, FreeBSD 4.6, clamav 0.80. CPU utilization sits between 80 and 95% idle. ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
* Eric Worthy <[EMAIL PROTECTED]> [20041026 07:02]: wrote: > I built a new 4 cpu/1 gig ram qmail/vpopmail/qmailscanner/clamv > mail server. (Four (4) Pentium® III Xeon 700 MHz/ 1 MB Cache) > I put it in last Thursday with it running great, then > yesterday, about 6pm, the cpu usage went to near 100% with about 800 > smtp transfers per hour. > > This morning about 8am, the cpu is at 100% and we're running about > 1400 smtp transfers per hour. > > It appears that even this box can't keep up with all the scanning > that has to take place. Here is top at this hour: > http://t10.net/cpu.jpg > http://t10.net/cpu2.jpg > > Tasks: 118 total, 7 running, 111 sleeping, 0 stopped, 0 zombie > Cpu0 : 69.2% user, 30.8% system, 0.0% nice, 0.0% idle > Cpu1 : 75.6% user, 24.4% system, 0.0% nice, 0.0% idle > Cpu2 : 73.1% user, 26.9% system, 0.0% nice, 0.0% idle > Cpu3 : 76.3% user, 23.7% system, 0.0% nice, 0.0% idle > Mem: 1032988k total, 836408k used, 196580k free, 37472k buffers > Swap: 128480k total, 93060k used, 35420k free, 351288k cached I would suggest that you DisableDefaultScanOptions in clamd.conf and tune values according to your system. My servers do slightly more than 800 smtp transfers per hour and I found out that working with the DisableDefaultScanOptions commented out brought my server to its knees. And my server is almost like yours, except it's Pentium III Xeon 500MHz. If you prefer, I can privately e-mail you my clamd.conf that works for me. You can test it out. It's not the best tuned, especially regarding: MaxConnectionQueueLength MaxThreads I still need to know how those two values should be determined. What to base their values on. At any given moment, my SMTP service has average 300 child processes so I used that value for MaxConnectionQueueLength. I am not sure that is quite what it should be, but works for me is the key thing ;) I'll appreciate any contributions towards making decisions on those two values. -Wash http://www.netmeister.org/news/learn2quote.html -- +==+ |\ _,,,---,,_ | Odhiambo Washington<[EMAIL PROTECTED]> Zzz /,`.-'`'-. ;-;;,_ | Wananchi Online Ltd. www.wananchi.com |,4- ) )-,_. ,\ ( `'-'| Tel: +254 20 313985-9 +254 20 313922 '---''(_/--' `-'\_) | GSM: +254 722 743223 +254 733 744121 +==+ Have you ever noticed that the people who are always trying to tell you, "There's a time for work and a time for play," never find the time for play? ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
Re: [Clamav-users] Performance Help - 100% cpu usage
Eric Worthy wrote: I built a new 4 cpu/1 gig ram qmail/vpopmail/qmailscanner/clamv mail server. (Four (4) Pentium® III Xeon 700 MHz/ 1 MB Cache) I put it in last Thursday with it running great, then yesterday, about 6pm, the cpu usage went to near 100% with about 800 smtp transfers per hour. This morning about 8am, the cpu is at 100% and we're running about 1400 smtp transfers per hour. 1400 is kind of low. I use Sparc v120 (single processor UltraSparc II 650 MHz) to handle much higher traffic than that. It appears that even this box can't keep up with all the scanning that has to take place. Here is top at this hour: This is a vanilla install off qmailrocks.org site. Debian install 3.0r1. Used apt-get to get my clam packages. clamav 0.75.1-4 Antivirus scanner for Unix clamav-base0.75.1-4 Base package for clamav, an anti-virus clamav-freshcl 0.75.1-4 Downloads clamav virus databases from the libclamav1 0.75.1-4 Virus scanner library 0.75.1 is not the latest Anyone have any advice on what I could be doing wrong or how to improve the performance of the scanning? Try 0.80. If it's not in debian's apt list, build it manually from source. If you still have performance issues with 0.80, there are some settings that you can modify later on clamd.conf. I suspect you didn't put limits on some settings (max archive size, etc.) For now, try 0.80 first. Regards, Fajar ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
[Clamav-users] Performance Help - 100% cpu usage
I built a new 4 cpu/1 gig ram qmail/vpopmail/qmailscanner/clamv mail server. (Four (4) Pentium® III Xeon 700 MHz/ 1 MB Cache) I put it in last Thursday with it running great, then yesterday, about 6pm, the cpu usage went to near 100% with about 800 smtp transfers per hour. This morning about 8am, the cpu is at 100% and we're running about 1400 smtp transfers per hour. It appears that even this box can't keep up with all the scanning that has to take place. Here is top at this hour: http://t10.net/cpu.jpg http://t10.net/cpu2.jpg Tasks: 118 total, 7 running, 111 sleeping, 0 stopped, 0 zombie Cpu0 : 69.2% user, 30.8% system, 0.0% nice, 0.0% idle Cpu1 : 75.6% user, 24.4% system, 0.0% nice, 0.0% idle Cpu2 : 73.1% user, 26.9% system, 0.0% nice, 0.0% idle Cpu3 : 76.3% user, 23.7% system, 0.0% nice, 0.0% idle Mem: 1032988k total, 836408k used, 196580k free, 37472k buffers Swap: 128480k total, 93060k used, 35420k free, 351288k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1290 qscand 15 0 57368 56m 696 R 50.8 5.6 172:29.51 clamdscan 25135 qscand 14 0 57368 56m 696 R 50.2 5.6 187:57.60 clamdscan 4980 qscand 15 0 57368 46m 696 R 50.2 4.6 167:42.45 clamdscan 30917 qscand 14 0 57368 56m 696 R 49.8 5.6 177:53.10 clamdscan 8861 qscand 15 0 57368 776 696 R 49.5 0.1 163:36.55 clamdscan 28183 qscand 14 0 57368 56m 696 R 49.2 5.6 182:21.71 clamdscan This is a vanilla install off qmailrocks.org site. Debian install 3.0r1. Used apt-get to get my clam packages. clamav 0.75.1-4 Antivirus scanner for Unix clamav-base0.75.1-4 Base package for clamav, an anti-virus clamav-freshcl 0.75.1-4 Downloads clamav virus databases from the libclamav1 0.75.1-4 Virus scanner library Anyone have any advice on what I could be doing wrong or how to improve the performance of the scanning? Thanks, Eric *update* - 8:00pm Monday night - I rebooted and it's all back to normal for now. ___ http://lists.clamav.net/cgi-bin/mailman/listinfo/clamav-users
