Bug#420391: clamav-daemon: Update of problem with 0.94
Stephen Gran said at 06/09/2008 20:26: > This one time, at band camp, Ronny Adsetts said: >> >> Quick update on this problem with the latest clamav to keep it on the >> radar: >> >> $ uname -a >> Linux allanon 2.6.24-etchnhalf.1-amd64 #1 SMP Mon Jul 21 10:36:02 UTC >> 2008 x86_64 GNU/Linux >> >> $ ps Hu -C clamd >> USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND >> clamav 30091 0.0 11.1 264900 230540 ? Ssl Sep05 0:14 >> /usr/sbin/clamd >> clamav 30091 0.0 11.1 264900 230540 ? Ssl 18:57 0:00 >> /usr/sbin/clamd >> >> So it's still there. Let me know if there's anything I can do to help. > [... description of memory fragmentation on database reloads snipped ...] > > If my understanding is correct, I'm afraid this is only going to get > worse as the databases get bigger. It can get more efficient, but the > underlying problem seems like it is likely to stay the same. That's my understanding of the problem too though I would hate to have to add more memory to the system just so that clamav can run. There are a couple of bugs in the clamav bug tracker that may be of relevance. See this one that talks about the fragmentation (marked invalid): https://wwws.clamav.net/bugzilla/show_bug.cgi?id=736 This one talks about the ref counting on the in-memory DB preventing the freed memory from being reused (or something along those lines): https://wwws.clamav.net/bugzilla/show_bug.cgi?id=1028 This latter bug has a fix that should see the light of day in 0.95 from the ticket comments. If I read the ticket right, this fix should prevent the memory usage from escalating to the extent it does now (my main live system currently shows RSS at 260688). Ronny -- Ronny Adsetts Technical Director Amazing Internet Ltd, London t: +44 20 8607 9535 f: +44 20 8607 9536 w: www.amazinginternet.com Registered office: UK House, 82 Heath Road, Twickenham TW1 4BW Registered in England. Company No. 4042957 signature.asc Description: OpenPGP digital signature
Bug#420391: clamav-daemon: Update of problem with 0.94
This one time, at band camp, Ronny Adsetts said: > Hi Stephen, > > Quick update on this problem with the latest clamav to keep it on the > radar: > > $ uname -a > Linux allanon 2.6.24-etchnhalf.1-amd64 #1 SMP Mon Jul 21 10:36:02 UTC > 2008 x86_64 GNU/Linux > > $ ps Hu -C clamd > USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND > clamav 30091 0.0 11.1 264900 230540 ? Ssl Sep05 0:14 > /usr/sbin/clamd > clamav 30091 0.0 11.1 264900 230540 ? Ssl 18:57 0:00 > /usr/sbin/clamd > > So it's still there. Let me know if there's anything I can do to help. For the bug log, what I understand about what is happening is that at startup, clamd loads a single copy of the database. At the first reload, it loads a second copy of the database, and running threads switch their database pointer to the new version as they become quiescent. This does result in some memory fragmentation, as the following ASCII art will try to show (note - this is a lie, like all descriptions of memory maps are a lie, because of the VM subsystem, but it gives an idea of what userspace sees). At startup, the memory map looks something like: || || | databases | || || | libraries, | | links, etc | After the first reload, it looks something like: || | second | | database | | copy | || || | first | | database | | copy | || | libraries, | | links, etc | And shortly after that, it looks like: || || | database | || || || || | unused| || || | libraries, | | links, etc | So when another reload happens, it is either small enough to fit in the unused space, or it's not, in which case it is appended. After that, the unused space in the middle should be big enough to load subsequent database copies, so memory usage should be roughly stable. Unfortunately, depending on the allocation, it will never quite go back to the starting memory usage, if I understand the problem correctly. If my understanding is correct, I'm afraid this is only going to get worse as the databases get bigger. It can get more efficient, but the underlying problem seems like it is likely to stay the same. -- - | ,''`.Stephen Gran | | : :' :[EMAIL PROTECTED] | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Bug#420391: clamav-daemon: Update of problem with 0.94
Package: clamav-daemon Version: 0.94.dfsg-1~volatile1 Followup-For: Bug #420391 Hi Stephen, Quick update on this problem with the latest clamav to keep it on the radar: $ uname -a Linux allanon 2.6.24-etchnhalf.1-amd64 #1 SMP Mon Jul 21 10:36:02 UTC 2008 x86_64 GNU/Linux $ ps Hu -C clamd USER PID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND clamav 30091 0.0 11.1 264900 230540 ? Ssl Sep05 0:14 /usr/sbin/clamd clamav 30091 0.0 11.1 264900 230540 ? Ssl 18:57 0:00 /usr/sbin/clamd So it's still there. Let me know if there's anything I can do to help. Ronny -- Package-specific info: --- configuration --- /etc/clamav/clamd.conf: clamd directives -- LogFile = "/var/log/clamav/clamav.log" LogFileUnlock = no LogFileMaxSize = 0 LogTime = yes LogClean = no LogVerbose = no LogSyslog = no LogFacility = "LOG_LOCAL6" PidFile = "/var/run/clamav/clamd.pid" TemporaryDirectory = "/tmp" ScanPE = yes ScanELF = yes DetectBrokenExecutables = no ScanMail = yes MailFollowURLs = no ScanPartialMessages = no PhishingSignatures = yes PhishingScanURLs = yes PhishingAlwaysBlockCloak = no PhishingAlwaysBlockSSLMismatch = no HeuristicScanPrecedence = no DetectPUA = no ExcludePUA not set IncludePUA not set StructuredDataDetection = no StructuredMinCreditCardCount = 3 StructuredMinSSNCount = 3 StructuredSSNFormatNormal = yes StructuredSSNFormatStripped = no AlgorithmicDetection = yes ScanHTML = yes ScanOLE2 = yes ScanPDF = no ScanArchive = yes MaxScanSize = 104857600 MaxFileSize = 26214400 MaxRecursion = 16 MaxFiles = 1 ArchiveLimitMemoryUsage = no ArchiveBlockEncrypted = no DatabaseDirectory = "/var/lib/clamav" TCPAddr not set TCPSocket not set LocalSocket = "/var/run/clamav/clamd.ctl" MaxConnectionQueueLength = 15 StreamMaxLength = 10485760 StreamMinPort = 1024 StreamMaxPort = 2048 MaxThreads = 12 ReadTimeout = 180 IdleTimeout = 30 MaxDirectoryRecursion = 15 ExcludePath not set FollowDirectorySymlinks = no FollowFileSymlinks = no ExitOnOOM = no Foreground = no Debug = no LeaveTemporaryFiles = no FixStaleSocket = yes User = "clamav" AllowSupplementaryGroups = yes SelfCheck = 3600 VirusEvent not set ClamukoScanOnAccess not set ClamukoScanOnOpen not set ClamukoScanOnClose not set ClamukoScanOnExec not set ClamukoIncludePath not set ClamukoExcludePath not set ClamukoMaxFileSize = 5242880 DevACOnly not set DevACDepth not set /etc/clamav/freshclam.conf: freshclam directives -- LogFileMaxSize = 0 LogTime = no LogVerbose = no LogSyslog = no LogFacility = "LOG_LOCAL6" PidFile = "/var/run/clamav/freshclam.pid" DatabaseDirectory = "/var/lib/clamav/" Foreground = no Debug = no AllowSupplementaryGroups = no DatabaseOwner = "clamav" Checks = 96 UpdateLogFile = "/var/log/clamav/freshclam.log" DNSDatabaseInfo = "current.cvd.clamav.net" DatabaseMirror = "db.local.clamav.net" DatabaseMirror = "database.clamav.net" DatabaseMirror = "db.uk.clamav.net" MaxAttempts = 5 ScriptedUpdates = yes CompressLocalDatabase = no HTTPProxyServer not set HTTPProxyPort not set HTTPProxyUsername not set HTTPProxyPassword not set HTTPUserAgent not set NotifyClamd not set OnUpdateExecute not set OnErrorExecute not set OnOutdatedExecute not set LocalIPAddress not set ConnectTimeout = 30 ReceiveTimeout = 30 Engine and signature databases -- Engine version: 0.94 Database directory: /var/lib/clamav/ main db: Format: .cld, Version: 48, Build time: Thu Sep 4 19:51:34 2008 daily db: Format: .cld, Version: 8174, Build time: Sat Sep 6 13:25:56 2008 --- data dir --- total 38440 -rw-r--r-- 1 clamav clamav 1079296 2008-09-06 13:34 daily.cld -rw-r--r-- 1 clamav clamav 38275584 2008-09-04 21:41 main.cld -rw--- 1 clamav clamav 364 2008-09-06 18:49 mirrors.dat -- System Information: Debian Release: 4.0 APT prefers stable APT policy: (500, 'stable') Architecture: amd64 (x86_64) Shell: /bin/sh linked to /bin/bash Kernel: Linux 2.6.24-etchnhalf.1-amd64 Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8) Versions of packages clamav-daemon depends on: ii clamav-base0.94.dfsg-1~volatile1 anti-virus utility for Unix - base ii clamav-freshclam [ 0.94.dfsg-1~volatile1 anti-virus utility for Unix - viru ii libc6 2.3.6.ds1-13etch7 GNU C Library: Shared libraries ii libclamav5 0.94.dfsg-1~volatile1 anti-virus utility for Unix - libr ii lsb-base 3.1-23.2etch1 Linux Standard Base 3.1 init scrip ii ucf2.0020Update Configuration File: preserv ii zlib1g 1:1.2.3-13compression library - runtime clamav-daemon recommends no packages. -- no debconf information -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
