Re: [Clamav-devel] Bug with .fp file being ignored
Hi Mark, I wanted to report that I believe we've properly fixed this issue last week. It looked to be the same issue as https://bugzilla.clamav.net/show_bug.cgi?id=12217 where I've linked to the commits where we did the work. Sorry it's taken so long, but glad we'll finally have FP signatures working properly in 0.103 đ -Micah -Original Message- From: clamav-devel On Behalf Of Mark Allan Sent: Friday, July 12, 2019 3:08 PM To: ClamAV Development Subject: [Clamav-devel] Bug with .fp file being ignored Hi, I think there's a bug with ClamAV not honouring the contents of a .fp file within the database directory. I've tested 0.101.2 as well as previous versions of ClamAV going back to 0.99.4 and the issue seems to have appeared as of 0.100.0 onwards. To re-create the issue: Find a zip file which you know reports an infection when scanned. Use sigtool --md5 to generate an FP sig of the zip file and save it in a .fp file in the databse directory. Use clamscan to scan the file and see that it still reports the file as being infected. The output from clamscan --debug shows the .fp file is being loaded, but it just doesn't seem to be being honoured for some reason. I see the same thing when I build ClamAV on macOS as well as when using the apt-get distribution on Ubuntu 18.04 Lastly, it only appears to be an issue with archive filetypes eg .zip, .dmg etc. Simple files are excluded as expected - similarly, if you generate an FP sig of a simple file and put that file within an archive, it correctly gets excluded. I'll clone the source from Git on Monday and have a dig through it myself to see if I can fix the bug, but thought I'd mention it here in case someone's already on it, or at least knows where I can start looking! Cheers Mark ___ clamav-devel mailing list clamav-devel@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-devel Please submit your patches to our Bugzilla: http://bugzilla.clamav.net Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ clamav-devel mailing list clamav-devel@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-devel Please submit your patches to our Github: https://github.com/Cisco-Talos/clamav-devel/pulls Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [Clamav-devel] Bug with .fp file being ignored
Mark, Well thatâs embarrassing. Looking at the patch now, it doesnât seem like the right fix for the issue â given the remaining bug where it alerts and then reports clean afterwards. It looks like cli_checkfp() is called within cli_append_virus(), and youâd think that would care of it â except that If you grep for âcli_append_virusâ, youâll notice that there are many cases where the return value is ignored. I suspect the real issue is that most, if not all, of those cases should check that the return value is still CL_VIRUS to honor the verdict from cli_checkfp(). Iâll put in a ticket to investigate. -Micah From: clamav-devel on behalf of Mark Allan Reply-To: ClamAV Development Date: Friday, September 20, 2019 at 4:47 PM To: ClamAV Development Subject: Re: [Clamav-devel] Bug with .fp file being ignored Hi Micah, Yes I did, and I submitted a patch back in July but there was another related issue which I wasn't able to fix. I've copied my email below with the patches. Best regards Mark --- The issue seems to have crept in with commits 3e42216cc and 28afc94c3 back in April/May 2017. Attached are patches for devel/HEAD as well as the stable 0.101.2 Tests show that the issue is fixed and doesn't appear to introduce any false negatives.however, it does produce a duplicate output line - one listed the infection found, and the second line (honouring the FP file) saying "OK". The "infected files" count is correct - see output below. Does anyone know how to fix that duplicate output? Cheers Mark virus-2009-04-13-id0007662101.zip: Osx.Worm.Leap-2 FOUND virus-2009-04-13-id0007662101.zip: OK --- SCAN SUMMARY --- Known viruses: 6168730 Engine version: 0.101.2 Scanned directories: 0 Scanned files: 1 Infected files: 0 Data scanned: 0.02 MB Data read: 0.00 MB (ratio 0.00:1) Time: 33.865 sec (0 m 33 s) > On 20 Sep 2019, at 10:29 am, Micah Snyder (micasnyd) > wrote: > > Hi Mark, > > Did you have any luck identifying the source of the bug? I admit I > bookmarked your email and failed to find time to look into it myself after > that. > > -Micah > > On 7/12/19, 6:09 PM, "clamav-devel on behalf of Mark Allan" > > wrote: > >Hi, > >I think there's a bug with ClamAV not honouring the contents of a .fp file >within the database directory. > >I've tested 0.101.2 as well as previous versions of ClamAV going back to >0.99.4 and the issue seems to have appeared as of 0.100.0 onwards. > >To re-create the issue: > >Find a zip file which you know reports an infection when scanned. >Use sigtool --md5 to generate an FP sig of the zip file and save it in a >.fp file in the databse directory. >Use clamscan to scan the file and see that it still reports the file as >being infected. > > >The output from clamscan --debug shows the .fp file is being loaded, but it >just doesn't seem to be being honoured for some reason. > >I see the same thing when I build ClamAV on macOS as well as when using the >apt-get distribution on Ubuntu 18.04 > >Lastly, it only appears to be an issue with archive filetypes eg .zip, .dmg >etc. Simple files are excluded as expected - similarly, if you generate an >FP sig of a simple file and put that file within an archive, it correctly >gets excluded. > >I'll clone the source from Git on Monday and have a dig through it myself >to see if I can fix the bug, but thought I'd mention it here in case >someone's already on it, or at least knows where I can start looking! > >Cheers >Mark >___ > >clamav-devel mailing list >clamav-devel@lists.clamav.net >https://lists.clamav.net/mailman/listinfo/clamav-devel > >Please submit your patches to our Bugzilla: http://bugzilla.clamav.net > >Help us build a comprehensive ClamAV guide: >https://github.com/vrtadmin/clamav-faq > >http://www.clamav.net/contact.html#ml > > > ___ > > clamav-devel mailing list > clamav-devel@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-devel > > Please submit your patches to our Bugzilla: http://bugzilla.clamav.net > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml ___ clamav-devel mailing list clamav-devel@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-devel Please submit your patches to our Bugzilla: http://bugzilla.clamav.net Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.cl
Re: [Clamav-devel] Bug with .fp file being ignored
Hi Micah, Yes I did, and I submitted a patch back in July but there was another related issue which I wasn't able to fix. I've copied my email below with the patches. Best regards Mark --- The issue seems to have crept in with commits 3e42216cc and 28afc94c3 back in April/May 2017. Attached are patches for devel/HEAD as well as the stable 0.101.2 Tests show that the issue is fixed and doesn't appear to introduce any false negatives.however, it does produce a duplicate output line - one listed the infection found, and the second line (honouring the FP file) saying "OK". The "infected files" count is correct - see output below. Does anyone know how to fix that duplicate output? Cheers Mark virus-2009-04-13-id0007662101.zip: Osx.Worm.Leap-2 FOUND virus-2009-04-13-id0007662101.zip: OK --- SCAN SUMMARY --- Known viruses: 6168730 Engine version: 0.101.2 Scanned directories: 0 Scanned files: 1 Infected files: 0 Data scanned: 0.02 MB Data read: 0.00 MB (ratio 0.00:1) Time: 33.865 sec (0 m 33 s) fix_devel_head.patch Description: Binary data fix_101_2.patch Description: Binary data > On 20 Sep 2019, at 10:29 am, Micah Snyder (micasnyd) > wrote: > > Hi Mark, > > Did you have any luck identifying the source of the bug? I admit I > bookmarked your email and failed to find time to look into it myself after > that. > > -Micah > > ï»żOn 7/12/19, 6:09 PM, "clamav-devel on behalf of Mark Allan" > > wrote: > >Hi, > >I think there's a bug with ClamAV not honouring the contents of a .fp file >within the database directory. > >I've tested 0.101.2 as well as previous versions of ClamAV going back to >0.99.4 and the issue seems to have appeared as of 0.100.0 onwards. > >To re-create the issue: > >Find a zip file which you know reports an infection when scanned. >Use sigtool --md5 to generate an FP sig of the zip file and save it in a >.fp file in the databse directory. >Use clamscan to scan the file and see that it still reports the file as >being infected. > > >The output from clamscan --debug shows the .fp file is being loaded, but it >just doesn't seem to be being honoured for some reason. > >I see the same thing when I build ClamAV on macOS as well as when using the >apt-get distribution on Ubuntu 18.04 > >Lastly, it only appears to be an issue with archive filetypes eg .zip, .dmg >etc. Simple files are excluded as expected - similarly, if you generate an >FP sig of a simple file and put that file within an archive, it correctly >gets excluded. > >I'll clone the source from Git on Monday and have a dig through it myself >to see if I can fix the bug, but thought I'd mention it here in case >someone's already on it, or at least knows where I can start looking! > >Cheers >Mark >___ > >clamav-devel mailing list >clamav-devel@lists.clamav.net >https://lists.clamav.net/mailman/listinfo/clamav-devel > >Please submit your patches to our Bugzilla: http://bugzilla.clamav.net > >Help us build a comprehensive ClamAV guide: >https://github.com/vrtadmin/clamav-faq > >http://www.clamav.net/contact.html#ml > > > ___ > > clamav-devel mailing list > clamav-devel@lists.clamav.net > https://lists.clamav.net/mailman/listinfo/clamav-devel > > Please submit your patches to our Bugzilla: http://bugzilla.clamav.net > > Help us build a comprehensive ClamAV guide: > https://github.com/vrtadmin/clamav-faq > > http://www.clamav.net/contact.html#ml ___ clamav-devel mailing list clamav-devel@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-devel Please submit your patches to our Bugzilla: http://bugzilla.clamav.net Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [Clamav-devel] Bug with .fp file being ignored
Hi Mark, Did you have any luck identifying the source of the bug? I admit I bookmarked your email and failed to find time to look into it myself after that. -Micah ï»żOn 7/12/19, 6:09 PM, "clamav-devel on behalf of Mark Allan" wrote: Hi, I think there's a bug with ClamAV not honouring the contents of a .fp file within the database directory. I've tested 0.101.2 as well as previous versions of ClamAV going back to 0.99.4 and the issue seems to have appeared as of 0.100.0 onwards. To re-create the issue: Find a zip file which you know reports an infection when scanned. Use sigtool --md5 to generate an FP sig of the zip file and save it in a .fp file in the databse directory. Use clamscan to scan the file and see that it still reports the file as being infected. The output from clamscan --debug shows the .fp file is being loaded, but it just doesn't seem to be being honoured for some reason. I see the same thing when I build ClamAV on macOS as well as when using the apt-get distribution on Ubuntu 18.04 Lastly, it only appears to be an issue with archive filetypes eg .zip, .dmg etc. Simple files are excluded as expected - similarly, if you generate an FP sig of a simple file and put that file within an archive, it correctly gets excluded. I'll clone the source from Git on Monday and have a dig through it myself to see if I can fix the bug, but thought I'd mention it here in case someone's already on it, or at least knows where I can start looking! Cheers Mark ___ clamav-devel mailing list clamav-devel@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-devel Please submit your patches to our Bugzilla: http://bugzilla.clamav.net Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ clamav-devel mailing list clamav-devel@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-devel Please submit your patches to our Bugzilla: http://bugzilla.clamav.net Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [Clamav-devel] Bug with .fp file being ignored
Thanks for tracking this down Mark. Sorry we didnât respond earlier. It has been a crazy couple weeks over here. Will take a look at the issue and your patches soon. -Micah Micah Snyder ClamAV Development Talos Cisco Systems, Inc. From: clamav-devel on behalf of Mark Allan Reply-To: ClamAV Development Date: Wednesday, July 17, 2019 at 11:38 AM To: ClamAV Development Subject: Re: [Clamav-devel] Bug with .fp file being ignored OK, so tracking this one down took longer than I like to admit! The issue seems to have crept in with commits 3e42216cc and 28afc94c3 back in April/May 2017. Attached are patches for devel/HEAD as well as the stable 0.101.2 Tests show that the issue is fixed and doesn't appear to introduce any false negatives.however, it does produce a duplicate output line - one listed the infection found, and the second line (honouring the FP file) saying "OK". The "infected files" count is correct - see output below. Does anyone know how to fix that duplicate output? Cheers Mark virus-2009-04-13-id0007662101.zip: Osx.Worm.Leap-2 FOUND virus-2009-04-13-id0007662101.zip: OK --- SCAN SUMMARY --- Known viruses: 6168730 Engine version: 0.101.2 Scanned directories: 0 Scanned files: 1 Infected files: 0 Data scanned: 0.02 MB Data read: 0.00 MB (ratio 0.00:1) Time: 33.865 sec (0 m 33 s) > On 12 Jul 2019, at 11:07 pm, Mark Allan wrote: > > Hi, > > I think there's a bug with ClamAV not honouring the contents of a .fp file > within the database directory. > > I've tested 0.101.2 as well as previous versions of ClamAV going back to > 0.99.4 and the issue seems to have appeared as of 0.100.0 onwards. > > To re-create the issue: > > Find a zip file which you know reports an infection when scanned. > Use sigtool --md5 to generate an FP sig of the zip file and save it in a > .fp file in the databse directory. > Use clamscan to scan the file and see that it still reports the file as being > infected. > > > The output from clamscan --debug shows the .fp file is being loaded, but it > just doesn't seem to be being honoured for some reason. > > I see the same thing when I build ClamAV on macOS as well as when using the > apt-get distribution on Ubuntu 18.04 > > Lastly, it only appears to be an issue with archive filetypes eg .zip, .dmg > etc. Simple files are excluded as expected - similarly, if you generate an FP > sig of a simple file and put that file within an archive, it correctly gets > excluded. > > I'll clone the source from Git on Monday and have a dig through it myself to > see if I can fix the bug, but thought I'd mention it here in case someone's > already on it, or at least knows where I can start looking! > > Cheers > Mark ___ clamav-devel mailing list clamav-devel@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-devel Please submit your patches to our Bugzilla: http://bugzilla.clamav.net Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml ___ clamav-devel mailing list clamav-devel@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-devel Please submit your patches to our Bugzilla: http://bugzilla.clamav.net Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
Re: [Clamav-devel] Bug with .fp file being ignored
OK, so tracking this one down took longer than I like to admit! The issue seems to have crept in with commits 3e42216cc and 28afc94c3 back in April/May 2017. Attached are patches for devel/HEAD as well as the stable 0.101.2 Tests show that the issue is fixed and doesn't appear to introduce any false negatives.however, it does produce a duplicate output line - one listed the infection found, and the second line (honouring the FP file) saying "OK". The "infected files" count is correct - see output below. Does anyone know how to fix that duplicate output? Cheers Mark virus-2009-04-13-id0007662101.zip: Osx.Worm.Leap-2 FOUND virus-2009-04-13-id0007662101.zip: OK --- SCAN SUMMARY --- Known viruses: 6168730 Engine version: 0.101.2 Scanned directories: 0 Scanned files: 1 Infected files: 0 Data scanned: 0.02 MB Data read: 0.00 MB (ratio 0.00:1) Time: 33.865 sec (0 m 33 s) fix_devel_head.patch Description: Binary data fix_101_2.patch Description: Binary data > On 12 Jul 2019, at 11:07 pm, Mark Allan wrote: > > Hi, > > I think there's a bug with ClamAV not honouring the contents of a .fp file > within the database directory. > > I've tested 0.101.2 as well as previous versions of ClamAV going back to > 0.99.4 and the issue seems to have appeared as of 0.100.0 onwards. > > To re-create the issue: > > Find a zip file which you know reports an infection when scanned. > Use sigtool --md5 to generate an FP sig of the zip file and save it in a > .fp file in the databse directory. > Use clamscan to scan the file and see that it still reports the file as being > infected. > > > The output from clamscan --debug shows the .fp file is being loaded, but it > just doesn't seem to be being honoured for some reason. > > I see the same thing when I build ClamAV on macOS as well as when using the > apt-get distribution on Ubuntu 18.04 > > Lastly, it only appears to be an issue with archive filetypes eg .zip, .dmg > etc. Simple files are excluded as expected - similarly, if you generate an FP > sig of a simple file and put that file within an archive, it correctly gets > excluded. > > I'll clone the source from Git on Monday and have a dig through it myself to > see if I can fix the bug, but thought I'd mention it here in case someone's > already on it, or at least knows where I can start looking! > > Cheers > Mark ___ clamav-devel mailing list clamav-devel@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-devel Please submit your patches to our Bugzilla: http://bugzilla.clamav.net Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
[Clamav-devel] Bug with .fp file being ignored
Hi, I think there's a bug with ClamAV not honouring the contents of a .fp file within the database directory. I've tested 0.101.2 as well as previous versions of ClamAV going back to 0.99.4 and the issue seems to have appeared as of 0.100.0 onwards. To re-create the issue: Find a zip file which you know reports an infection when scanned. Use sigtool --md5 to generate an FP sig of the zip file and save it in a .fp file in the databse directory. Use clamscan to scan the file and see that it still reports the file as being infected. The output from clamscan --debug shows the .fp file is being loaded, but it just doesn't seem to be being honoured for some reason. I see the same thing when I build ClamAV on macOS as well as when using the apt-get distribution on Ubuntu 18.04 Lastly, it only appears to be an issue with archive filetypes eg .zip, .dmg etc. Simple files are excluded as expected - similarly, if you generate an FP sig of a simple file and put that file within an archive, it correctly gets excluded. I'll clone the source from Git on Monday and have a dig through it myself to see if I can fix the bug, but thought I'd mention it here in case someone's already on it, or at least knows where I can start looking! Cheers Mark ___ clamav-devel mailing list clamav-devel@lists.clamav.net https://lists.clamav.net/mailman/listinfo/clamav-devel Please submit your patches to our Bugzilla: http://bugzilla.clamav.net Help us build a comprehensive ClamAV guide: https://github.com/vrtadmin/clamav-faq http://www.clamav.net/contact.html#ml
