The scan times are definitely better than they were - in fact, they're back
to how they were before last week's inclusion of the Phishtank signatures.
They're still almost double what they used to be though, and as far as I
can see, there are still almost 4000 Phishtank signatures in the DB:
$ sigtool --find Phishtank | wc -l
3968
Can I request that those ones also be removed please?
Best regards
Mark
On Sun, 7 Apr 2019 at 14:43, Micah Snyder (micasnyd) <micas...@cisco.com>
wrote:
> Tim,
>
>
>
> There are a couple of ways for users to drop specific categories of
> signatures at this time. Sadly, they wouldn’t have helped this last week.
> These include bytecode signatures, PUA (potentially unwanted applications)
> signatures, Email.Phishing and HTML.Phishing signatures, and the
> Safebrowsing database.
>
>
>
> If we had named the Phishtank.Phishing sigs to HTML.Phishing.Phishtank or
> Email.Phishing.Phishtank then they could have been disabled with the
> clamscan option `--phishing-sigs=no` (clamd.conf: `PhishingSignatures no`).
>
>
>
> Maybe a better option would be for us to create a new optional database
> for phishing signatures. However, the names for the databases are hardcoded
> into freshclam, so it is non-trivial to add a new database and would
> require a few changes to ClamAV’s code. We have talked about making the
> databases easier to add/remove in the future so users can have more
> categories to enable/disable. In this light, it ties in well with existing
> plans.
>
>
>
> Of note the Phishtank sigs from Friday’s daily were removed yesterday and
> scan times should be back to normal.
>
>
>
> Regards,
>
> Micah
>
>
>
> *From: *Tim Hawkins <tim.hawk...@redflaggroup.com>
> *Date: *Friday, April 5, 2019 at 6:06 PM
> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, Mark Allan <
> markjal...@gmail.com>
> *Cc: *"Micah Snyder (micasnyd)" <micas...@cisco.com>
> *Subject: *Re: [External] Re: [clamav-users] Scan very slow
>
>
>
> Hi Micah
>
>
> Does clamav partition the database so that signatures that are mainly
> associated with email scanning can be dropped out for folks only needing
> filesystems scans, none of our systems use email, and we dont make use of
> the mailer extension.
>
> Having to load all the email focused signatures could as you have observed
> impact performance.
>
> Sent from Nine <http://www.9folders.com/>
> ------------------------------
>
> *From:* "Micah Snyder (micasnyd) via clamav-users" <
> clamav-users@lists.clamav.net>
> *Sent:* Saturday, April 6, 2019 03:18
> *To:* ClamAV users ML; Mark Allan
> *Cc:* Micah Snyder (micasnyd)
> *Subject:* [External] Re: [clamav-users] Scan very slow
>
>
>
> Regarding slow scan times today (and slow scan times in general), it
> appears that the signatures we generate based on PhishTank’s feed for
> phishing URLs are resulting in very slow load and scan times.
>
>
>
> Today’s daily update saw 7448 new Phishtank signatures (much higher than
> usual) coinciding with the immediate performance drop for load time and
> scan time. One user reported that the load time today on some of his
> slower machines was slow enough to exceed the timeout for service startup (
> https://bugzilla.clamav.net/show_bug.cgi?id=12317).
>
>
>
> In limited testing on my own machine I saw the following change after
> dropping the Phishtank.Phishing signatures from daily.cvd’s daily.ldb file:
>
> - Database load time on my laptop went from 75.43203997612 seconds
> down to 14.859203100204468 seconds
> - Scan time (for an arbitrary pdf) went from 1.798 sec to 0.644 sec.
>
>
>
> After some discussion between the teams that work on ClamAV and ClamAV
> signature content and deployment, we’ve agreed to drop PhishTank signatures
> from the database until we can determine a way to craft Phishtank
> signatures without incurring such a significant performance hit.
>
>
>
> The daily update tomorrow will have the change.
>
>
>
> -Micah
>
>
>
>
> Micah Snyder
> ClamAV Development
> Talos
> Cisco Systems, Inc.
>
>
>
>
>
>
>
> *From: *clamav-users <clamav-users-boun...@lists.clamav.net> on behalf of
> "Micah Snyder (micasnyd) via clamav-users" <clamav-users@lists.clamav.net>
> *Reply-To: *ClamAV users ML <clamav-users@lists.clamav.net>
> *Date: *Friday, April 5, 2019 at 1:08 PM
> *To: *Mark Allan <markjal...@gmail.com>, ClamAV users ML <
> clamav-users@lists.clamav.net>
> *Cc: *"Micah Snyder (micasnyd)" <micas...@cisco.com>
> *Subject: *Re: [clamav-users] Scan very slow
>
>
>
> Hi Mark,
>
>
>
> Sorry about the delay in responding. I hadn’t looked at my clamav-users
> filter this morning. Just investigating now. Will respond when I know
> more.
>
>
>
> -Micah
>
>
>
> *From: *Mark Allan <markjal...@gmail.com>
> *Date: *Friday, April 5, 2019 at 9:12 AM
> *To: *ClamAV users ML <clamav-users@lists.clamav.net>, "Micah Snyder
> (micasnyd)" <micas...@cisco.com>
> *Subject: *Re: [clamav-users] Scan very slow
>
>
>
> Also CC'ing Micah directly as the mailing list would appear to be offline
> (at least lists.clamav.net isn't responding to http requests anyway)
>
>
>
> It looks like scan times have gone through the roof. As Oya said, they're
> still considerably higher than they were a couple of months ago, but
> today's scan time is insane.
>
>
>
> Yesterday's scan using
>
> 0.101.2:58:25409:1554370140:1:63:48554:328
>
> took 7m 3s
>
>
>
> On the same hardware, scanning the same read-only disk image, with today's
> scan using
>
> 0.101.2:58:25410:1554452941:1:63:48557:328
>
> the scan time has jumped to 26m 15s
>
>
>
> This is the longest it has ever taken to scan this volume (cf my previous
> email of 25th March)
>
>
>
> Is there anything that can be excluded?
>
>
>
> Best regards
>
> Mark
>
>
>
> On Mon, 1 Apr 2019 at 17:11, Micah Snyder (micasnyd) via clamav-users <
> clamav-users@lists.clamav.net> wrote:
>
> Thanks Oya for the update. We will continue to investigate the signature
> performance issue.
>
> Regards,
> Micah
>
> On 3/28/19, 9:50 AM, "clamav-users on behalf of Tsutomu Oyamada" <
> clamav-users-boun...@lists.clamav.net on behalf of oyam...@promark-inc.com>
> wrote:
>
> Hi Micah
>
> It seems that the scanning slow down issue of this time has been
> solved
> at some level with CVD Update of the other day.
> However, there is still big discrepancy in between the current
> condition and
> the last condition in one month ago.
>
> Date Files Scan time
> 2019/02/15 2550338 08:53:57
> 2019/03/15 2612792 19:22:54
> 2019/03/26 2634489 18:13:56
> 2019/03/27 2637201 18:10:05
>
> We know the improvement of this time is due to the details of CVD,
> because
> we did not make any change on the user's system.
> We are going to try some tuning for scanning.
>
> We like to know if you still have some room to make further improvement
> for this slow down issue.
> Thank you for your help, in advance.
>
> Best regards,
> Oya
>
> On Mon, 25 Mar 2019 15:45:02 +0000
> "Micah Snyder \(micasnyd\) via clamav-users" <
> clamav-users@lists.clamav.net> wrote:
>
> > Hi Mark, all:
> >
> > I’m disappointed to hear that it is still slow for you.
> >
> > We found that the target-type of signatures used for
> PhishTank.Phishing signatures were causing a significant slowdown. We
> have dropped them as of this past Saturday (
> https://lists.gt.net/clamav/virusdb/75279 ) and in the last two updates
> have been re-adding them with more specific scan target types. We’re now
> investigating some other optimizations we can make for the next major
> ClamAV release to improve scan times but at present we don’t have any other
> leads for signatures that may be slowing down scans.
> >
> > Regards,
> > Micah
> >
> >
> > From: clamav-users <clamav-users-boun...@lists.clamav.net> on
> behalf of Mark Allan via clamav-users <clamav-users@lists.clamav.net>
> > Reply-To: ClamAV users ML <clamav-users@lists.clamav.net>
> > Date: Monday, March 25, 2019 at 9:37 AM
> > To: ClamAV users ML <clamav-users@lists.clamav.net>
> > Cc: Mark Allan <markjal...@gmail.com>
> > Subject: Re: [clamav-users] Scan very slow
> >
> > Cheers Steve,
> >
> > In the interest of completeness, here's the scan from today (TXT
> from DNS: 0.101.1:58:25399:1553509741:1:63:48528:328) showing a marked
> improvement in scan time, although at 6m 7s it's still almost twice what it
> used to be.
> >
> > Mark
> >
> > On Mon, 25 Mar 2019 at 12:56, Steve Basford <
> steveb_cla...@sanesecurity.com<mailto:steveb_cla...@sanesecurity.com>>
> wrote:
> > On 2019-03-25 10:52, Mark Allan via clamav-users wrote:
> > > Hi all,
> > >
> > te.
> > >
> > > Hopefully this helps someone to narrow things down a bit.
> > >
> > > Mark
> > >
> >
> > 18/3/19 10m 49s TXT from DNS:
> > 0.101.1:58:25392:1552904941:1:63:48507:328 ***
> >
> > Here's the changes for the above update:
> >
> > https://lists.gt.net/clamav/virusdb/75154
> >
> > You can also check sigs quickly per update:
> >
> > https://lists.gt.net/clamav/virusdb/
> >
> >
> >
> > --
> > Cheers,
> >
> > Steve
> > Twitter: @sanesecurity
> >
> > _______________________________________________
> >
> > clamav-users mailing list
> > clamav-users@lists.clamav.net<mailto:clamav-users@lists.clamav.net>
> > https://lists.clamav.net/mailman/listinfo/clamav-users
> >
> >
> > Help us build a comprehensive ClamAV guide:
> > https://github.com/vrtadmin/clamav-faq
> >
> > http://www.clamav.net/contact.html#ml
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
>
> _______________________________________________
>
> clamav-users mailing list
> clamav-users@lists.clamav.net
> https://lists.clamav.net/mailman/listinfo/clamav-users
>
>
> Help us build a comprehensive ClamAV guide:
> https://github.com/vrtadmin/clamav-faq
>
> http://www.clamav.net/contact.html#ml
>
>
>
> *DISCLAIMER*
>
> The information contained in this email and any attachments are
> confidential. It is intended solely for the individual or entity to whom
> they are addressed. Access to this email by anyone else is unauthorized.
>
> If you are not the intended recipient, any disclosure, copying,
> distribution or any action taken or omitted to be taken in reliance on it,
> is prohibited and may be unlawful. If you have received this communication
> in error, please notify us immediately by responding to this email and then
> delete it from your system.
>
> The Red Flag Group is neither liable for the proper and complete
> transmission of the information contained in this communication nor for any
> delay in its receipt.
>
> Any advice, recommendations or opinion contained within this email or its
> attachments are not to be construed as legal advice.
>
_______________________________________________
clamav-users mailing list
clamav-users@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-users
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
