This is actually really good data. Thanks for taking the time out to evaluate these files.
First, have you modified bofhland_cracked_URL.ndb at all? I'm getting 20+ seconds to load that. On the flip side, I'm getting sub-second loading times for winnow_phish_complete.ndb, winnow_phish_complete_url.ndb and phish.ndb. I'm running this on a beefy macbook pro with 16Gb of RAM, so I'm not sure if that helps or not in this particular case. Scamnailer is a little longer at 1.5 seconds. But, if I were guessing, the pattern for "http://"; for winnow_phish. for phish.ndb, it looks like a lot of sigs in the form PK{WILDCARD_ANY_STRING(LENGTH==28)}.... Which would demonstrate the same behavior. We'll have to do more checkingon scamnailer. There is a ton of alternating patterns, and really no repeating static contents that I can see in a cursory glance. We'll check it out and get more information. Again, thanks for the data, we'll keep it in mind as we work on coming versions. Matt On Thu, Aug 15, 2013 at 7:45 AM, Steve Basford < steveb_cla...@sanesecurity.com> wrote: > > > > >> I've done some analysis of ClamAV with just this signature set, and the > >> loading is simply slowing down as it runs through the list. > > * Third Party dbs * > > Hi, > > While looking into the database loading time issue, thought it might be > an idea to quickly scan the same "small" file with each database, just to > see what scanning time each database took and the amount of memory the > *single* database used. > > When using multiple db's it's not the whole story... but just in case it's > useful.... > > bofhland_cracked_URL.ndb: Time: 6.593 sec > bofhland_cracked_URL.ndb: Memory: 29.777 MB > > bofhland_malware_attach.hdb: Time: 0.047 sec > bofhland_malware_attach.hdb: Memory: 4.331 MB > > bofhland_malware_URL.ndb: Time: 0.125 sec > bofhland_malware_URL.ndb: Memory: 7.816 MB > > bofhland_phishing_URL.ndb: Time: 0.047 sec > bofhland_phishing_URL.ndb: Memory: 4.741 MB > > crdfam.clamav.hdb: Time: 0.062 sec > crdfam.clamav.hdb: Memory: 5.046 MB > > foxhole_all.ccdb: Time: 0.046 sec > foxhole_all.cdb: Memory: 4.308 MB > > foxhole_filename.ccdb: Time: 0.047 sec > foxhole_filename.cdb: Memory: 4.308 MB > > foxhole_generic.ccdb: Time: 0.047 sec > foxhole_generic.cdb: Memory: 4.312 MB > > junk.ndb: Time: 0.860 sec > junk.ndb: Memory: 18.866 MB > > jurlbl.ndb: Time: 0.078 sec > jurlbl.ndb: Memory: 5.281 MB > > jurlbla.ndb: Time: 0.125 sec > jurlbla.ndb: Memory: 6.386 MB > > lott.ndb: Time: 0.078 sec > lott.ndb: Memory: 5.206 MB > > phish.ndb: Time: 2.390 sec > phish.ndb: Memory: 14.546 MB > > phishtank.ndb: Time: 0.157 sec > phishtank.ndb: Memory: 5.699 MB > > porcupine.ndb: Time: 0.078 sec > porcupine.ndb: Memory: 5.898 MB > > rogue.hdb: Time: 0.047 sec > rogue.hdb: Memory: 4.652 MB > > scam.ndb: Time: 0.407 sec > scam.ndb: Memory: 11.585 MB > > scamnailer.ndb: Time: 4.609 sec > scamnailer.ndb: Memory: 22.085 MB > > spam.lcdb: Time: 0.047 sec > spam.ldb: Memory: 4.515 MB > > spamattach.hdb: Time: 0.047 sec > spamattach.hdb: Memory: 4.308 MB > > spamimg.hdb: Time: 0.047 sec > spamimg.hdb: Memory: 4.398 MB > > spear.ndb: Time: 0.610 sec > spear.ndb: Memory: 12.140 MB > > spearl.ndb: Time: 0.063 sec > spearl.ndb: Memory: 5.089 MB > > winnow.attachments.hdb: Time: 0.047 sec > winnow.attachments.hdb: Memory: 4.370 MB > > winnow.complex.patterns.lcdb: Time: 0.047 sec > winnow.complex.patterns.ldb: Memory: 4.320 MB > > winnow_bad_cw.hdb: Time: 0.046 sec > winnow_bad_cw.hdb: Memory: 4.308 MB > > winnow_extended_malware.hdb: Time: 0.109 sec > winnow_extended_malware.hdb: Memory: 7.413 MB > > winnow_extended_malware_links.ndb: Time: 0.046 sec > winnow_extended_malware_links.ndb: Memory: 4.308 MB > > winnow_malware.hdb: Time: 0.110 sec > winnow_malware.hdb: Memory: 7.777 MB > > winnow_malware_links.ndb: Time: 0.125 sec > winnow_malware_links.ndb: Memory: 7.128 MB > > winnow_phish_complete.ndb: Time: 4.907 sec > winnow_phish_complete.ndb: Memory: 7.577 MB > > winnow_phish_complete_url.ndb: Time: 4.922 sec > winnow_phish_complete_url.ndb: Memory: 7.577 MB > > winnow_spam_complete.ndb: Time: 0.125 sec > winnow_spam_complete.ndb: Memory: 7.097 MB > > > Cheers, > > Steve > Sanesecurity > > _______________________________________________ > Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net > http://www.clamav.net/support/ml > _______________________________________________ Help us build a comprehensive ClamAV guide: visit http://wiki.clamav.net http://www.clamav.net/support/ml
