Hi Nikolay, Sorry this slipped by me. I'd be happy to take a look at the PDF you were having scan speed issues with. I see that it's no longer available with the URL you originally provided. If you could share it again, I'll spend some time with it to try to see what's going on.
As a heads up, we have couple patch versions coming out tomorrow which I hope will show some scan time improvements and detection improvements as a result of work overhauling some scan recursion and embedded file type detection logic. I don't expect it will help in this particular case, but ... maybe! *shrugs* Regards, Micah Micah Snyder ClamAV Development Talos Cisco Systems, Inc. ________________________________ From: clamav-users <clamav-users-boun...@lists.clamav.net> on behalf of Nikolay Belaevski via clamav-users <clamav-users@lists.clamav.net> Sent: Monday, October 4, 2021 6:03 PM To: ClamAV users ML <clamav-users@lists.clamav.net> Cc: Nikolay Belaevski <nikolay.belaev...@five9.com> Subject: Re: [clamav-users] Extremely slow PDF file scanning Any feedback from ClamAV developers, please: should I open a defect for the problem or is it expected that PDF file scanning takes few minutes? Thanks, Nikolay From: Nikolay Belaevski <nikolay.belaev...@five9.com> Date: Saturday, September 25, 2021 at 11:32 To: ClamAV users ML <clamav-users@lists.clamav.net> Subject: Re: [clamav-users] Extremely slow PDF file scanning Hi Ged, Thank you for your response! You are right, the configuration below is not secure and thanks again for pointing to this! For production we’ll use different configuration that simply issues alert for the provided file. The one attached here is more a less copy of default configuration with the limits increased to extremely high values in attempt to get scanning completed and then try to tweak them down to see what exactly is causing the alert on the file. Release notes for 0.104.0 mention “Fixed bytecode match evaluation for PDF bytecode hooks in PDF file scans.”. Looks like something that’s been fixed, yes. For curiosity I have tried to disable bytecode and the scanning completes faster of course, but it must be enabled for best results AFAIK. Looking at the total size of temporary files created, I see that scan has produced 1.7Gb of temporary files extracted from PDF and that may explain why scanning takes that much time. For comparison, when I extract all images from the file using pdfimages program, I’m seeing about 21Mb. And another comparison, when I use pdftk to concatenate original file with the simple one-page text file and scan result, scan takes just two seconds and the size of temporary files produced is about 6Mb. I.e. for a slightly larger input scan time is much better! That’s the reason I’m suspecting there may be something in the original file that confuses ClamAV parser / analyzer and would be interesting for ClamAV developers to check. On the side note, I have tried few large PDF files from Google scanned library, no issues at all and they are scanned quickly. Best regards, Nikolay From: clamav-users <clamav-users-boun...@lists.clamav.net> on behalf of G.W. Haywood via clamav-users <clamav-users@lists.clamav.net> Date: Saturday, September 25, 2021 at 10:44 To: Nikolay Belaevski via clamav-users <clamav-users@lists.clamav.net> Cc: G.W. Haywood <cla...@jubileegroup.co.uk> Subject: Re: [clamav-users] Extremely slow PDF file scanning Hi there, On Fri, 24 Sep 2021, Nikolay Belaevski via clamav-users wrote: > Iʼm investigating why it takes about five minutes for ClamAV 0.104.0 > to scan PDF file. Can someone help me, please? There's a note in NEWS.md that a fault in PDF scanning was fixed in version 0.104. I don't know if it's relevant but it might be worth a look. More importantly did you read the warnings in the documentation about the settings that you've changed? Snippets from 'clamconf -n' reporting about your clamd.conf, with my comments added below each item: 8<---------------------------------------------------------------------- TCPSocket = "3310" # TCP sockets are unprotected from remote access. Careful! DisableCache = "yes" # This will cause clamd to scan the file every time it sees it, and not # to use its internal caching. Foreground = "yes" # I guess there's reason for this? Debug = "yes" # This may affect performance, but I would not expect gross effects. MaxScanTime = "600000" # Ten minutes! *Fifty* times the default of 12 seconds. According to # the man page, excessive times can cause DOS conditions. MaxScanSize = "4194304000" # 4GBytes. *Forty* times the default of 100M, and in the clamd.conf # man page there is a dire warning about setting this limit too high. # I'm also not sure how the absolute 2G limit on file size impinges. MaxFileSize = "52428800" # Twice the default. Another dire warning in the man page. MaxFiles = "50000" # Five times the default. Dire warning. PCREMaxFileSize = "104857600" # Four times the default. Specific warning about performance. 8<---------------------------------------------------------------------- I think you need to look carefully at the configuration changes which you've made, perhaps do some testing to establish whether your system can support scanning with those changes under conditions of plausible stress. Under normal circumstances my systems wouldn't scan the file you've provided - I only use ClamAV to scan mail, and if this appeared in our mail it would be rejected on principle by inspection of the message parts. When I scanned it manually, with our usual configuration, it just gave a warning about a limit that being exceeded after 1m 54s of scanning time. In my experience that's a longish but not an extreme scan time. We log the scan time for most scans. Below is an extract from our mail log, listing all the scan times this month which were longer than 10 seconds. As you can see some of the scans of rather small amounts of data took considerably longer than one might expect given the performance of other scans of much larger amounts - the two scans of more than 200kbytes of data performed this month each took a shade over 30 seconds. I haven't really investigated scan times here as they're well under anything which might cause problems. Sep 1 19:30:15 mail6 xm 10.249s, 3339 bytes Sep 1 20:04:29 mail6 xm 10.122s, 50390 bytes Sep 9 11:42:33 mail6 xm 13.563s, 3337 bytes Sep 9 21:56:56 mail6 x3 11.716s, 622 bytes Sep 9 23:29:33 mail6 xm 10.355s, 3338 bytes Sep 10 06:32:42 mail6 xm 10.153s, 3337 bytes Sep 10 06:48:42 mail6 x3 12.189s, 853 bytes Sep 10 21:14:38 mail6 xm 35.126s, 260193 bytes Sep 10 23:24:55 mail6 xm 32.949s, 218614 bytes Sep 15 16:10:04 mail6 x3 10.004s, 118 bytes Sep 15 21:39:11 mail6 xm 11.149s, 53899 bytes Sep 16 05:42:49 mail6 x3 15.139s, 175 bytes Sep 16 05:43:09 mail6 x3 10.960s, 46 bytes Sep 16 05:43:19 mail6 x3 16.012s, 195 bytes Sep 16 05:43:46 mail6 x3 10.853s, 195 bytes Sep 16 05:44:09 mail6 x3 16.696s, 1370 bytes Sep 16 05:45:22 mail6 x3 12.048s, 950 bytes Sep 16 05:48:41 mail6 x3 11.545s, 596 bytes Sep 16 05:49:19 mail6 x3 12.399s, 1115 bytes Sep 16 05:50:35 mail6 x3 10.489s, 474 bytes Sep 16 10:49:13 mail6 x3 12.740s, 1147 bytes Sep 18 05:55:10 mail6 x3 16.392s, 861 bytes Sep 18 08:54:53 mail6 x3 10.325s, 861 bytes Sep 18 18:29:43 mail6 xm 12.694s, 3342 bytes Sep 18 20:36:44 mail6 xm 10.644s, 3340 bytes Sep 18 23:32:09 mail6 x3 10.463s, 45 bytes Sep 19 04:14:03 mail6 x3 15.265s, 950 bytes Sep 19 06:29:11 mail6 x3 10.973s, 45 bytes Sep 21 19:51:45 mail6 x3 22.834s, 712 bytes These were all scanned using a TCP connection to a remote scanner on the same (1Gbit/s Ethernet) network. The processes 'xm' and 'x3' are two of the milters which our mail servers run, they pass data directly to a remote clamd over TCP as I've said - we don't use clamav-milter. HTH -- 73, Ged. _______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ________________________________ CONFIDENTIALITY NOTICE: This e-mail and any files attached may contain confidential information of Five9 and/or its affiliated entities. Access by the intended recipient only is authorized. Any liability arising from any party acting, or refraining from acting, on any information contained in this e-mail is hereby excluded. If you are not the intended recipient, please notify the sender immediately, destroy the original transmission and its attachments and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Copyright in this e-mail and any attachments belongs to Five9 and/or its affiliated entities.
_______________________________________________ clamav-users mailing list clamav-users@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-users Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
