Re: [Clamav-users] Idea for more timely virusdb updates
On Tue, 24 Aug 2004 10:26:50 +0700 Fajar A. Nugraha [EMAIL PROTECTED] wrote: I'm concerned about the fact that you only use one NS for cvd.clamav.net though. There will be at least 6 when the testing period is over. Is there a fallback scenario (automagically revert to HTTP GETs) if that NS fails? Yes, it's already implemented. -- oo. Tomasz Kojm [EMAIL PROTECTED] (\/)\. http://www.ClamAV.net/gpg/tkojm.gpg \..._ 0DCA5A08407D5288279DB43454822DC8985A444B //\ /\ Thu Aug 26 03:16:16 CEST 2004 pgpF9IjwBsdRs.pgp Description: PGP signature
Re: [Clamav-users] Idea for more timely virusdb updates
Hi, new freshclam is ready for testing: Sun Aug 22 02:07:13 CEST 2004 (tk) -- * freshclam: Support version verification through DNS (DNSDatabaseInfo). Based on idea by Christopher X. Candreva chris*westnet.com, see http://www.gossamer-threads.com/lists/clamav/users/11102 Because most users are happy with hourly checks we didn't decide to use a ttl-related instant checking (i.e. sleep(ttl); check; slep(ttl)...) in daemon mode but removed the limit of maximal checks instead (only in DNSDatabaseInfo mode). We're not willing to use DNS for database updates. On September 1, the databases will be restructured and the size of daily.cvd limited. Thanks, -- oo. Tomasz Kojm [EMAIL PROTECTED] (\/)\. http://www.ClamAV.net/gpg/tkojm.gpg \..._ 0DCA5A08407D5288279DB43454822DC8985A444B //\ /\ Sun Aug 22 02:28:48 CEST 2004 pgpeRq8VsJIKw.pgp Description: PGP signature
Re: [Clamav-users] Idea for more timely virusdb updates
libresolv don't exist on FreeBSD. Regards gmake[2]: Entering directory `/usr/ports/security/clamav-devel/work/clamav-devel-20040822/clamdscan' /bin/sh ../libtool --mode=link cc -O3 -pipe -funroll-loops -ffast-math -march=pentiumpro -I/usr/local/include -L/usr/local/lib -lcipher -o clamdscan output.o cfgparser.o memory.o clamdscan.o client.o ../clamscan/options.o ../clamscan/getopt.o ../libclamav/libclamav.la -lresolv cc -O3 -pipe -funroll-loops -ffast-math -march=pentiumpro -I/usr/local/include -o .libs/clamdscan output.o cfgparser.o memory.o clamdscan.o client.o ../clamscan/options.o ../clamscan/getopt.o -L/usr/local/lib ../libclamav/.libs/libclamav.so -lcipher -lz -lgmp -lc_r -lresolv -Wl,--rpath -Wl,/usr/local/lib /usr/libexec/elf/ld: cannot find -lresolv Tomasz Kojm wrote: Hi, new freshclam is ready for testing: Sun Aug 22 02:07:13 CEST 2004 (tk) -- * freshclam: Support version verification through DNS (DNSDatabaseInfo). Based on idea by Christopher X. Candreva chris*westnet.com, see http://www.gossamer-threads.com/lists/clamav/users/11102 Because most users are happy with hourly checks we didn't decide to use a ttl-related instant checking (i.e. sleep(ttl); check; slep(ttl)...) in daemon mode but removed the limit of maximal checks instead (only in DNSDatabaseInfo mode). We're not willing to use DNS for database updates. On September 1, the databases will be restructured and the size of daily.cvd limited. Thanks, -- Marcus Grando Grupos Internet S/A marcus(at)corp.grupos.com.br --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Idea for more timely virusdb updates
I still don't see why rsync can't be used here. It can easily do incremental updates. True. However, (1) many firewall admins allow outgoing HTTP and DNS ports; I cannot say the same for rsync port. (2) The uncompressed signature (viruses.db*) files is a good candidate for rsync (or even a simple diff command). I don't know how well rsync or diff performs on the compressed-signed *.cvd. Hmmm... interesting points... but what about this option? Rsync and diff are generic patching mechanisms meant to accomodate data without a known format - we don't have that problem here. My understanding is that for the most part database updates are additions, though sometimes there may be deletions or updates to preexisting keys Lets say on the SERVER side, those updates were kept in something of the form: version|status|signature|md5 Where version is the version number containing the change... status is + (new sig), - (remove sig), or = (update sig) (the sematics are important, the values of the enum are not of course) and signature contains whatever the current fields of the database are... md5 would be the checksum of a database if all patches applied to this point are sucessful Then, any freshclam could connect, something like: http://somemirror.db?version=xxx The server would then return all updates xxx, which would allow the freshclam to patch it's local database, and verify the last md5 is a match for the md5 of the updated local db. If the update fails to produce a matching checksum, freshclam could then pull a fresh copy in it's entirety. This would mean the mirrors would have to support basic scripting (PHP?) but we could trade a significant portion of the bandwidth for a few cpu cycles... m/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Saturday 14 August 2004 07:11 am, Jason Haar wrote: On Sat, Aug 14, 2004 at 01:07:06PM +0700, Fajar Nugraha wrote: How about incremental updates? Can it possibly make way to next freshclam versions too? Too true. Some commercial AVs separate patterns into a library (large) plus incremental files (small). Typically the next library release contains the previous incremental patterns, and the actual incremental files are deleted - stopping crustiness from developing... I still don't see why rsync can't be used here. It can easily do incremental updates. -Jeremy -- Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc. [EMAIL PROTECTED] ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Sun, 15 Aug 2004 08:25:36 -0500 Jeremy Kitchen [EMAIL PROTECTED] wrote: On Saturday 14 August 2004 07:11 am, Jason Haar wrote: On Sat, Aug 14, 2004 at 01:07:06PM +0700, Fajar Nugraha wrote: How about incremental updates? I still don't see why rsync can't be used here. It can easily do incremental updates. True. However, (1) many firewall admins allow outgoing HTTP and DNS ports; I cannot say the same for rsync port. (2) The uncompressed signature (viruses.db*) files is a good candidate for rsync (or even a simple diff command). I don't know how well rsync or diff performs on the compressed-signed *.cvd. Regards, Fajar --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
At 18:40 15.08.2004, Fajar Nugraha wrote: On Sun, 15 Aug 2004 08:25:36 -0500 Jeremy Kitchen [EMAIL PROTECTED] wrote: On Saturday 14 August 2004 07:11 am, Jason Haar wrote: On Sat, Aug 14, 2004 at 01:07:06PM +0700, Fajar Nugraha wrote: How about incremental updates? I still don't see why rsync can't be used here. It can easily do incremental updates. True. However, (1) many firewall admins allow outgoing HTTP and DNS ports; I cannot say the same for rsync port. (2) The uncompressed signature (viruses.db*) files is a good candidate for rsync (or even a simple diff command). I don't know how well rsync or diff performs on the compressed-signed *.cvd. badly., unless you make those incremental too, e.g. daily.454.cvd and provide means to join those. cheers Erich THINK Püntenstrasse 39 8143 Stallikon mailto:[EMAIL PROTECTED] PGP Fingerprint: BC9A 25BC 3954 3BC8 C024 8D8A B7D4 FF9D 05B8 0A16 --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Am Friday 13 August 2004 23:23 schrieb Mitch (WebCob): Hi, DNS for serial numbers plus HTTP for actual data transfer still sounds New version of freshclam will work in this way. the mirrors at all. Once this is tested, an update to recommended polling times would be appreciated (for anyone not running freshclam as a daemon) Actually this is not required. A very nice feature of DNS is that it not only transports the information (serial number) but also the metainformation when to refetch the data. This means that the clamav nameserver control the TTL (time to live) of the data. A nice side effect of this is that it allows for a flexible _server_ based control of the DNS retry intervals of the clients. E.g. the virusdb people get a new signature/virus. While investigating the data they might consider to already lower the TTL so that the clients check the DNS more frequently. As soon as the new signature/virusdb is available the clients notice this rather fast. In addition by managing the TTL the server can control how hard it is hit when a new signature is available. Shorter TTL means less smoothing but better latency. IMHO using the DNS intelligently helps not only for the QoS but also the internet as a whole because it avoids unnecessary and unproductive load on all systems involved. Side note: Lets hope that no 1$107 gets the idea to patent such a mechanism as it just happend recently to us KDE people. (A KDE person had a UI idea and presented it on a KDE ml. KDE implemented this idea in the following months but MS took the idea, added the obvious and patented it :-(. MS even cites the KDE ml as the source of the initial idea.) Regards, -- martin Dipl.-Phys. Martin Konold e r f r a k o n Erlewein, Frank, Konold Partner - Beratende Ingenieure und Physiker Nobelstrasse 15, 70569 Stuttgart, Germany fon: 0711 67400963, fax: 0711 67400959 email: [EMAIL PROTECTED] --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Fri, 13 Aug 2004 22:34:43 +0200 Tomasz Kojm [EMAIL PROTECTED] wrote: On Sat, 14 Aug 2004 08:02:51 +1200 Jason Haar [EMAIL PROTECTED] wrote: DNS for serial numbers plus HTTP for actual data transfer still sounds New version of freshclam will work in this way. Big thanks to all for the interesting thread ! How about incremental updates? Can it possibly make way to next freshclam versions too? Checking serial number (or version) via DNS will certainly reduce some traffic on the mirrors, but I'm not sure how much of the 91.53 GB (last month traffic here for *.cvd) will be reduced. Version checking (assuming most clients use the new freshclam) in my raw calculation uses 173840 (cvd hit) * 512 bytes = 85 MB. Not much. If we can use incremental update, for each database update we'll be distributing small ammount of data (say ... 1 - 5 kB) instead of the 162kB daily.cvd (or more) that we have now. If we distribute the incremental update via DNS (I still have a thing for DNS), ISPs will cache this too so mirror traffic will be much much much reduced. I have a Linode with 25 GB monthly bandwidth on a fast datacenter, and I'll be happy to contribute half of it for clamav db mirror. But since mirror bandwidth requirement is SOOO high, I can't do that can I? I'm sure that with lower bandwidth requirement many people will be happy to pinch in. Regards, Fajar --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Sat, Aug 14, 2004 at 01:07:06PM +0700, Fajar Nugraha wrote: How about incremental updates? Can it possibly make way to next freshclam versions too? Too true. Some commercial AVs separate patterns into a library (large) plus incremental files (small). Typically the next library release contains the previous incremental patterns, and the actual incremental files are deleted - stopping crustiness from developing... daily.cvd is 160K at the moment, and will be totally re-downloaded the next time it's updated. However, this fiddling around trying to remember which old one-off pattern files can now be deleted/ignored does add complexity - something daily.cvd doesn't have to worry about... I suspect the digital signing of those files might be screwing up alternative options on this? After all, you don't want to be checking sigs on 12 files before actually being able to start doing AV! Perhaps a different way could be brought into play... Can the digital sig be separated from the daily file, and have the current CVD file changed into a better format that rsync could do a good job of incrementing/appended it? That way an update would in effect be appending some lines to the daily.RAW file, plus downloading the new (tiny) digital signature of that file? Just a thought - Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Wed, Aug 11, 2004 at 08:34:48PM +0200, Martin Konold wrote:
The problem with bittorent is that bittorent addresses a different problem
domain.
clamav pattern update:
- frequently changing small number of small files distributed from a single
point to many
bittorrent:
- slowly changing high number of potentially very big files distributed from
many sources to many destinations.
This isn't correct. You somehow confusingly assume all current
bittorrent downloads are related? They are not. Each individual .torrent
starts out as a one-to-many distribution.
The nice thing about bittorrent is that practically immediately after
a third client connects, it becomes a many-to-many transfer, utilising
the available upload capacity of all clients.
So each individual torrent you find on those popular websites that list
all torrents, started as a single-point-to-many distribution. And the
number of torrents available there isn't slowly changing, in fact, it's
often changing way faster than new virus definitions are released :)
The main difference is that most currently offered torrents comprise
many megabytes, while a virus definition file would only be a few
kilobytes. But that doesn't invalidate the protocol, certainly not with
a high number of downloaders.
If anyone has questions on how the bittorrent protocol works, there
is quite a bit of info on the official website:
http://www.bitconjurer.org/BitTorrent/
and there's a wiki FAQ: http://wiki.theory.org/index.php/BitTorrentFAQ
--
#!perl -wpl # mmfppfmpmmpp mmpffm [EMAIL PROTECTED]
$p=3-2*/[^\W\dmpf_]/i;s.[a-z]{$p}.vec($f=join('',$p-1?chr(sub{$_[0]*9+$_[1]*3+
$_[2]}-(map{/p|f/i+/f/i}split//,$)+97):qw(m p f)[map{((ord$)%32-1)/$_%3}(9,
3,1)]),5,1)='`'lt$;$f.eig;# Jan-Pieter Cornet
---
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
___
Clamav-users mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Wed, Aug 11, 2004 at 03:07:35PM +0200, Lionel Bouton wrote: The ideal setup would be to push updates instead of clients polling them. It would requires a separate architecture though (HTTP mirrors can't push things). Since some time I am thinking of a bittorrent approach too. Bittorrent i... All this should fail for the *majority* of ClamAV sites!! Push updates implies people have put clam servers out on the Internet so that they are reachable - I don't think so! That's what firewalls were invented for. Similarly, BitTorrent *requires* raw Internet access in order to operate - again - not a normal situation for an AV server. DNS for serial numbers plus HTTP for actual data transfer still sounds best to me... All outgoing connections only, all well established (nothing exotic) -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Sat, 14 Aug 2004 08:02:51 +1200 Jason Haar [EMAIL PROTECTED] wrote: DNS for serial numbers plus HTTP for actual data transfer still sounds New version of freshclam will work in this way. Big thanks to all for the interesting thread ! -- oo. Tomasz Kojm [EMAIL PROTECTED] (\/)\. http://www.ClamAV.net/gpg/tkojm.gpg \..._ 0DCA5A08407D5288279DB43454822DC8985A444B //\ /\ Fri Aug 13 22:26:46 CEST 2004 pgp2afXjVont1.pgp Description: PGP signature
RE: [Clamav-users] Idea for more timely virusdb updates
DNS for serial numbers plus HTTP for actual data transfer still sounds New version of freshclam will work in this way. Big thanks to all for the interesting thread ! Sounds cool Tomasz! Be interested to hear if this helps reduce the load on the mirrors at all. Once this is tested, an update to recommended polling times would be appreciated (for anyone not running freshclam as a daemon) Thanks! m/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Idea for more timely virusdb updates
Similarly, BitTorrent *requires* raw Internet access in order to operate - again - not a normal situation for an AV server. Don't know what exactly you meant by raw as opposed to sauteed, broiled, baked or toasted, but BitTorrent does NOT require unfirewalled access. It does require a small port range to be forwarded to it, BUT that port range is not required to be the same on any two hosts. When the host contacts the tracker, it tells the tracker which ports it is listening on so the tracker can distribute load to it. m/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Fri, 13 Aug 2004, Tomasz Kojm wrote: New version of freshclam will work in this way. Big thanks to all for the interesting thread ! That's C-a-n-d-r-e-v-a . For the CHANGES file. :-) -Chris == Chris Candreva -- [EMAIL PROTECTED] -- (914) 967-7816 WestNet Internet Services of Westchester http://www.westnet.com/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Fri, Aug 13, 2004 at 02:22:55PM -0700, Mitch (WebCob) wrote: Don't know what exactly you meant by raw as opposed to sauteed, broiled, baked or toasted, but BitTorrent does NOT require unfirewalled access. It does require a small port range to be forwarded to it, BUT that port range is not required to be the same on any two hosts. Well that means raw :-) It means the *incoming* ports involved need to be open to the Internet. Creating outgoing SMTP/HTTP/FTP from within a firewalled network doesn't mean you have to open up your firewall for any incoming ports - BitTorrent does require that. That falls into the must have really good business case - can we put you in a standalone DMZ? case for most largish companies. -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Aug 11, 2004, at 1:22 PM, Martin Konold wrote: Am Wednesday 11 August 2004 16:19 schrieb Bart Silverstrim: Hi Bart, DNS was developed exactly for this kind of purpose. Storing non-DNS related information for retrieval? As I understand the proposition (and the original lecture that this idea was based on), it's was for hiding information in a very small records area of DNS for propogating information...I don't think the designers for DNS had spreading AV signatures (or files or other things that have been proposed, non-Clam related) in mind at the time. Please be aware of the fact that I don't think that DNS is the correct tool to distribute files but for distributing something like a serial number it fits perfectly. The actual download of the data has to be done outside of the DNS system. Okay, this would be where my misunderstanding was entering the picture :-) I was thinking DNS would be the distribution engine, not the notification engine. Sorry for the confusion! Abusing the DNS to directly transfer files etc is not appropriate as the DNS infrastructure is not ready for such kind of abuse. Give it time...someone's going to do it. I'm surprised someone hasn't found a hole in DNS that would allow it to act as a way to distribute viruses via DNS records yet... -Bart --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Aug 11, 2004, at 10:40 AM, Damian Menscher wrote: On Wed, 11 Aug 2004, Lionel Bouton wrote: Since some time I am thinking of a bittorrent approach too. Bittorrent is quite efficient at distributing files and there are implementations allowing multiple trackers to distribute the remaining server-side load. Please take this as a question rather than a criticism of the approach: My experience with bittorrent has been with downloading huge things, like Fedora. snip I've never used bittorrent so I'm afraid I can't comment there. With regard to all the other ideas: Please remember to keep this *simple*. Here's where I, IMHO, think we stand: Opening a new port on a mailserver so updates can be pushed to it is a BAD idea. As a sysadmin, I would not allow such a thing on my production machines. It creates a huge security risk, since now you have one more opening to a remote root vulnerability. Just a clarification of what I accidentally proposed earlier: it wouldn't be so much a mail server doing this, just a daemon and application modeled after mail. I think it's pretty clear thanks to SPAMmers all around the Internet that email protocols are broken, but the basic simplicity of the model behind SMTP could be applied to send out a subscription of encrypted and signed updates to people who sign up for it...it would be more like a whitelisted email system in *concept* only. I was proposing that only because the basics are already out there in framework in the form of email...but we don't want something that accepts random or additional info. Just updates for our Clam programs. Yes, it could be another root vulnerability, and it would be a bigger target because AV kiddies are usually the kind that are more likely to attempt DOS attacks against servers if there's a central target to hit. I'm really starting to like the idea of a mailing list that can have dedicated (and random for each site) subscription addresses and pipe the list straight into sigtool --add. It means we'd have to find someone to host the list, but that's probably no more difficult than finding someone to host a mirror. Presumably there could even be multiple mirrors sending the list, to improve speed (taking an idea from spammers who use open relays to do the hard part). One thing to add to the mailing list approach: there needs to be some sort of heartbeat or dead man's switch -- a way to know that the mailing list is functional, but there are no needed updates, rather than that the mailing list has broken. I suppose this might be a use for that latest-db-version.clamav.net idea. Here's a second idea to combine with the first...use a freenet model. At least I think that's the name... It's P2P and anonymous; and (my memory is foggy...can someone confirm the details?) it is kind of like a mesh network within a network. It was originally meant as a totally free and distributed way for P2P transfers of information. Everyone shares information and it gets distributed on computers, and you as a client/server have no control or idea what is in your allocated sharing space. Could be illegal material, could be shakespeare, you don't know (Freenet, that is). If we had a meshed system of a live network within a network of updates with this model, it may be an interesting infrastructure not only for rapid updates, but impossible (improbable?) denial of service attacks, and the possibility of even tagging exe's for analysis later...they could just be swept up in the grid and analyzed when they reach the appropriate team members. The sigs could be updates and swept into the grid where they'd be distributed to sysadmins. Again, probably impractical, but just enjoying the brainstorming that's going on on the list recently. :-) It would be more complicated that previous ideas, yes, but it may lay groundwork for future features or ideas, like maybe a way to monitor virus activity and send out statistics to users who wish to setup that ability for monitoring outbreaks in certain regions of the Internet. -Bart --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Idea for more timely virusdb updates
I've already mentioned this jokingly, but I was half serious: I think setting up a bittorrent would solve a lot of the bandwidth problems. Been playing with that a bit recently - the more I think about it, the more I like it... saw a website that has built a custom tracker to manage leeches, and prevent people (regardless of client) from sponging without contributing... The old way could remain, for offline / intermittantly or heavily firewalled users... The addition of DNS version management could reduce overhead bandwidth that occurs during useless polls... The new way could provide higher frequency updates for those willing to share and contribute some bytes. m/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Am Mittwoch, 11. August 2004 04:56 schrieb Robert Blayzor: Hi, In a perfect world, wouldn't this be the ultimate application for say, multicast? No, multicast is not the most efficient solution and does no caching. Honestly imho the DNS approach is the most feasable and the most efficient solution as it provides incoherent caching uses udp and reuses a widly available infrastructure. Regards, -- martin Dipl.-Phys. Martin Konold e r f r a k o n Erlewein, Frank, Konold Partner - Beratende Ingenieure und Physiker Nobelstrasse 15, 70569 Stuttgart, Germany fon: 0711 67400963, fax: 0711 67400959 email: [EMAIL PROTECTED] --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Idea for more timely virusdb updates
Daniel J McDonald wrote: That's one of the things that seems to be driving the size of daily.cvd up - updating main.cvd entails a massive distribution of files to the world. Current main.cvd = 1103636 bytes, last updated on July 8 Current daily.cvd = 156470 bytes A bit of mental arithmetic suggests thatdaily.cvd grows by about 5KB per day. A few sums in my head suggest that total download savings in a month if main.cvd was updated fortnightly would be around 200KB (circa 3100KB total download instead of 3300KB), a virtually insignificant difference. Perhaps a tiered approach to the update files, with main.cvd, monthly.cvd, weekly.cvd, daily.cvd, and hot.cvd The advantage there is that the really big update could be distributed very seldom - perhaps only with new code (the code generally has to be upgraded every few months to deal with a new threat anyway). Big updates often remove false positives, improve detections of existing viruses, so might still need monthly (or more frequent) updating. If you had overlapping signatures between the files, you could add a fuzzy-factor into freshclam that it might not bring down the latest weekly/monthly if the other files overlap completely. That would distribute the load on the freshclam servers for the larger updates, and there would just be the very small daily.cvd (and perhaps hot.cvd) downloads. If we could use incremental (or, more correctly, differential) updates which effectively create a new main.cvd then we could have a large reduction in the load on the download servers. However, we then have the problem of ensuring that main.cvd remains consistent. I like the idea of using DNS to signal the change - maybe just for hot.cvd. so, whenever a major virus breakout occurs, the new sig would be added to hot.cvd and the DNS TXT record changed. 10,000 users pulling down a 2-3K file is not terribly hard for a server with decent bandwidth I've known DNS servers to completely ignore TTL figures and cache stuff which should have expired, so this might not be reliable. Cheers, Phil Phil Randal Network Engineer Herefordshire Council Hereford, UK --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Aug 10, 2004, at 2:30 PM, Jeremy Kitchen wrote: On Tuesday 10 August 2004 12:23 pm, Damian Menscher wrote: The gpg-signature prevents spoofing. And the sequence numbers keep everyone current. The major problems I see are getting clamd to recognize a message targeted for it, and the obvious problems of DoS attacks (someone sending spoofed messages that would suck CPU time decoding the gpg signature). yes, that's an unfortunate problem with this idea, however, if you used, as I stated, a special address that uses program delivery, you'd have to hack the listserver to get everyone's 'subscription' address to be able to do this. Instead of having this piggyback on email, I was thinking more along the line of a separate protocol just *modeled* after email. Separate port, separate server daemon for it...maybe it would lessen the chances of your updates getting filtered by spam filters and/or targeted for probes and overflow attacks in the process. That way it isn't hacking the MTAs out there to do work that isn't meant or related to them...never liked the idea of bending programs backwards to tack on added functionality. Seems to be another vector for bug creep :-) [I haven't given up on DNS updates yet, but it's hard to come up with a clean way to distribute 256 bytes of data that way, which means even single rules don't always fit.] I wouldn't distribute the rule in DNS, however, a timestamp of sorts in dns isn't a bad idea. While DNS is an interesting idea, I'm worried more about what kind of bugs and glitches this is going to uncover in the process (or what kind of attacks would be crafted should this idea catches on.) Let's say the idea does become popular, and clam and other programs out there start taking advantage of it...I don't know about all of you, but I didn't set up a DNS server on a system meant for constant hits from other sources querying it; it's just a little system that can handle the load of a small network and that's pretty much it :-) And what about systems that restrict querying to certain IPs? If a service starts getting abused, that tends to be when (clueful) admins start taking steps to lock things down; many places with NTP servers, for example, will host a public site until too many people start hitting and if it starts to become a burden, the time server suddenly disappears. :-( -Bart --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Am Mittwoch, 11. August 2004 13:53 schrieb Bart Silverstrim: Hi Bart, the idea does become popular, and clam and other programs out there start taking advantage of it DNS was developed exactly for this kind of purpose. ...I don't know about all of you, but I didn't set up a DNS server on a system meant for constant hits from other sources querying it Sorry, I dont want to sound impolite but please try to become familiar how DNS works in todays internet. ; it's just a little system that can handle the load of a small network and that's pretty much it :-) Your little DNS server will only get queries from you very own clamav installation but not from _anyone_ else. And what about systems that restrict querying to certain IPs? ?? This makes no sense. If you system is able to do http with using fqdn then it is also able to use the DNS. If a service starts getting abused, that tends to be when (clueful) admins start taking steps to lock things down; many places with NTP servers, for example, will host a public site until too many people start hitting and if it starts to become a burden, the time server suddenly disappears. :-( Typically the DNS load will be put on the ISPs. The ISPs prever _very_ much people using the DNS which is an incoherent or loosly coherent caching and distributed database to permanent polling, multicast or pushing. Funny enough: The protocol ideas you are proposeing are putting _more_ load on the DNS(*) than the direct DNS idea. (*) it is save to assume that your protocol ideas don't use static ip numbers but use DNS to do gethostbyname() resolving. Regards, -- martin Dipl.-Phys. Martin Konold e r f r a k o n Erlewein, Frank, Konold Partner - Beratende Ingenieure und Physiker Nobelstrasse 15, 70569 Stuttgart, Germany fon: 0711 67400963, fax: 0711 67400959 email: [EMAIL PROTECTED] --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Martin Konold wrote: No, multicast is not the most efficient solution and does no caching. Honestly imho the DNS approach is the most feasable and the most efficient solution as it provides incoherent caching uses udp and reuses a widly available infrastructure. Why would you need caching? Especially when using PIM sparse. -- Robert Blayzor, BOFH INOC, LLC [EMAIL PROTECTED] PGP: http://www.inoc.net/~dev/ Key fingerprint = 1E02 DABE F989 BC03 3DF5 0E93 8D02 9D0B CB1A A7B0 There are two major products that come out of Berkeley: LSD and UNIX. We don't believe this to be a coincidence. - Jeremy S. Anderson --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Idea for more timely virusdb updates
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Dennis Peterson Jeremy Kitchen wrote: On Tuesday 10 August 2004 02:41 pm, Damian Menscher wrote: [snip: using a program delivery to process update mailing list mails] With sendmail, you could add to /etc/aliases something like: clamav-updates | sigtool --add that's the ticket. And a cool little DOS tool. Nothing like a well-known email address for a little fun having. I imagine the blackhats will slam that rather quickly. The list can have a simple algorithm to filter any 'easy' or mnemotenic email address thats trying to subscribe. That way we can reduce the problem. In concern of cvd corruption because a malformed patches, I think that the updater could backup some old states of the sig file in order to have the posibility to downgrade without downloading sigs again (manually or something). In any case, the patch can always have a 'before MD5' and an 'after MD5', just to be sure. Another idea to save bandwidth (in a pull fashion) could be to download instead of: daily.cvd something like: daily.cvd?myver=XXXMD5=XX Were _myver_ and _MD5_ are my sigfile's version and it's MD5 respectively. The server could make a patch to the last version in real time and send it, instead of sending the 'big' file. The realtime patch could be done cat'ing all patches needed together or already having created all the respective daily_fromN_toCURRENT.cvd patches. Could also be implemented directly into main.cvd. Brainstorming is great... as long as I don't have to code resulting ideas ;) Thanks to all for this great bits. -Samuel --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Christopher X. Candreva wrote the following on 08/10/2004 07:40 PM : If people can't check for database updates more often than once an hour, then there is a pressing need. The mirror page talkes about the need for mirrors, about exponential growth, and how at least a 10mbit pipe is needed to host a mirror. It puts March 2004 traffic at about 120gig/month The bandwidth used on our mirrors (clamav.inet6.fr) as lowered since then mainly thanks to the ability of new clients to abort the transmission of a file when it is the same version as the one they already have. The ideal setup would be to push updates instead of clients polling them. It would requires a separate architecture though (HTTP mirrors can't push things). Since some time I am thinking of a bittorrent approach too. Bittorrent is quite efficient at distributing files and there are implementations allowing multiple trackers to distribute the remaining server-side load. If I'm not mistaken, there's only one thing to add : querying for the latest available torrent which must be signed by the db authors : DNS would be great for this indeed (hash sigs prevent cache poisoning and thus distribution of invalid db, but not DoS). Lionel. --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Aug 11, 2004, at 10:32 AM, Martin Konold wrote: Am Mittwoch, 11. August 2004 13:53 schrieb Bart Silverstrim: Hi Bart, the idea does become popular, and clam and other programs out there start taking advantage of it DNS was developed exactly for this kind of purpose. Storing non-DNS related information for retrieval? As I understand the proposition (and the original lecture that this idea was based on), it's was for hiding information in a very small records area of DNS for propogating information...I don't think the designers for DNS had spreading AV signatures (or files or other things that have been proposed, non-Clam related) in mind at the time. Also I was worried about the fact that DNS servers usually got traffic for updates from peers and client/server lookups, not spreading files...that would boost their hits and bandwidth. ...I don't know about all of you, but I didn't set up a DNS server on a system meant for constant hits from other sources querying it Sorry, I dont want to sound impolite but please try to become familiar how DNS works in todays internet. Prefacing it with not wanting to sound impolite still makes it sound impolite but that's okay :-) I'm not a DNS expert by any means. It's been five years or so since I set up a bare linux system as an authoritative DNS server, other setups I've made were all in-house caching servers. I'll make no claim to knowing how things work exactly or the pitfalls of trying this out, or the effect it would have on the servers. But I *have* had enough experience to say that if I have these questions, chances are someone else on the lists has them too, vocalizing them or not...and they'll hopefully get an education from the answers I get as the result of my chiming in :-) ; it's just a little system that can handle the load of a small network and that's pretty much it :-) Your little DNS server will only get queries from you very own clamav installation but not from _anyone_ else. The proposal is just to have a few DNS servers, the authoritative ones, seeded with the info then? Others would just cache it? Duh. Makes sense. And what about systems that restrict querying to certain IPs? ?? This makes no sense. If you system is able to do http with using fqdn then it is also able to use the DNS. I think at the time I was thinking about DNS servers updating from peers, distributing the load of the records that are spread piecemeal. Funny enough: The protocol ideas you are proposeing are putting _more_ load on the DNS(*) than the direct DNS idea. (*) it is save to assume that your protocol ideas don't use static ip numbers but use DNS to do gethostbyname() resolving. You would be correct; it would be the load made by any listserv though. any spreading of an email like server's load would increase lookups (unless assisted by local caches heavily). It's not necessarily the load on the server I think I was worrying about, per se...it's trying to shoehorn the protocol to do more than it was supposed to. It's probably a misunderstanding of the idea for spreading the information through DNS, but I would think that the DNS idea would hit a wall or block that would be imposed by the restrictions of DNS itself and the way it works, so a new mechanism would have to supplement it or some other clever hack. I would think that it would be better to start a propagation idea from scratch rather than a neat idea (I'm not trying to disregard it...heck, I'm probably missing something obvious and am worrying about a non-issue) for extending a protocol meant for task A being extended to also be able to do task B. -Bart --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Am Mittwoch, 11. August 2004 14:47 schrieb Robert Blayzor: Hi, No, multicast is not the most efficient solution and does no caching. Honestly imho the DNS approach is the most feasable and the most efficient solution as it provides incoherent caching uses udp and reuses a widly available infrastructure. Why would you need caching? PIM sparse helps in some situations but it put extra load on the routers. I like the fact the PIM sparse is a request-only service but still imho a pull is better than a push approach. For enterprise use and its security policies using http for the download is most appropriate especially when using caching upstream http proxies. Especially when using PIM sparse. Regards, -- martin Dipl.-Phys. Martin Konold e r f r a k o n Erlewein, Frank, Konold Partner - Beratende Ingenieure und Physiker Nobelstrasse 15, 70569 Stuttgart, Germany fon: 0711 67400963, fax: 0711 67400959 email: [EMAIL PROTECTED] --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Wed, 11 Aug 2004, Lionel Bouton wrote: Since some time I am thinking of a bittorrent approach too. Bittorrent is quite efficient at distributing files and there are implementations allowing multiple trackers to distribute the remaining server-side load. Please take this as a question rather than a criticism of the approach: My experience with bittorrent has been with downloading huge things, like Fedora. It tends to start up really slowly (since it has to find peers) and then speed up. But the speedup doesn't occur until several megs have been downloaded. If we're only sending a 1-meg main.cvd, then wouldn't bittorrent lose its advantage to all the overhead of finding peers? With regard to all the other ideas: Please remember to keep this *simple*. Here's where I, IMHO, think we stand: Opening a new port on a mailserver so updates can be pushed to it is a BAD idea. As a sysadmin, I would not allow such a thing on my production machines. It creates a huge security risk, since now you have one more opening to a remote root vulnerability. The idea of DNS sounds really good, but it doesn't appear we can fit all the data there. And putting just a version number there appears to make things worse, since it will just make everyone hit the mirrors at the same time. If we can somehow distribute signatures that way it would be nice, but it just doesn't seem practical. I'm really starting to like the idea of a mailing list that can have dedicated (and random for each site) subscription addresses and pipe the list straight into sigtool --add. It means we'd have to find someone to host the list, but that's probably no more difficult than finding someone to host a mirror. Presumably there could even be multiple mirrors sending the list, to improve speed (taking an idea from spammers who use open relays to do the hard part). One thing to add to the mailing list approach: there needs to be some sort of heartbeat or dead man's switch -- a way to know that the mailing list is functional, but there are no needed updates, rather than that the mailing list has broken. I suppose this might be a use for that latest-db-version.clamav.net idea. Damian Menscher -- -=#| Physics Grad Student SysAdmin @ U Illinois Urbana-Champaign |#=- -=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=- -=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=- -=#| [EMAIL PROTECTED] www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=- -=#| The above opinions are not necessarily those of my employers. |#=- --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Am Wednesday 11 August 2004 16:19 schrieb Bart Silverstrim: Hi Bart, DNS was developed exactly for this kind of purpose. Storing non-DNS related information for retrieval? As I understand the proposition (and the original lecture that this idea was based on), it's was for hiding information in a very small records area of DNS for propogating information...I don't think the designers for DNS had spreading AV signatures (or files or other things that have been proposed, non-Clam related) in mind at the time. Please be aware of the fact that I don't think that DNS is the correct tool to distribute files but for distributing something like a serial number it fits perfectly. The actual download of the data has to be done outside of the DNS system. Also I was worried about the fact that DNS servers usually got traffic for updates from peers and client/server lookups, not spreading files...that would boost their hits and bandwidth. I am afraid this is a missunderstanding as I am not advocating to distribute files via the dns but only a serial number. vocalizing them or not...and they'll hopefully get an education from the answers I get as the result of my chiming in :-) I think this is totally fine :-) was supposed to. It's probably a misunderstanding of the idea for spreading the information through DNS, but I would think that the DNS idea would hit a wall or block that would be imposed by the restrictions of DNS itself and the way it works, so a new mechanism would have to supplement it or some other clever hack. I would think that it would be better to start a propagation idea from scratch rather than a neat idea (I'm not trying to disregard it...heck, I'm probably missing something obvious and am worrying about a non-issue) for extending a protocol meant for task A being extended to also be able to do task B. In short: Using DNS for spreading _small_ pieces of information which typically have some reasonable TTL (time to live) and which can deal with the loose coherency of DNS is totally fine. Abusing the DNS to directly transfer files etc is not appropriate as the DNS infrastructure is not ready for such kind of abuse. Regards, -- martin Dipl.-Phys. Martin Konold e r f r a k o n Erlewein, Frank, Konold Partner - Beratende Ingenieure und Physiker Nobelstrasse 15, 70569 Stuttgart, Germany fon: 0711 67400963, fax: 0711 67400959 email: [EMAIL PROTECTED] --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Damian Menscher wrote the following on 08/11/2004 04:40 PM : On Wed, 11 Aug 2004, Lionel Bouton wrote: Since some time I am thinking of a bittorrent approach too. Bittorrent is quite efficient at distributing files and there are implementations allowing multiple trackers to distribute the remaining server-side load. Please take this as a question rather than a criticism of the approach: My experience with bittorrent has been with downloading huge things, like Fedora. It tends to start up really slowly (since it has to find peers) and then speed up. But the speedup doesn't occur until several megs have been downloaded. If we're only sending a 1-meg main.cvd, then wouldn't bittorrent lose its advantage to all the overhead of finding peers? It depends on the torrents you download. Some starts really quickly. From my experience it really depends on how much clients are really available to upload files and the ratio of good/bad clients known by the tracker : when numerous clients stop distributing content after finishing a download, some (and sometimes most of the) clients the tracker send you are now unavailable and you then start having timeouts and asking other clients to the tracker. In a more controlled environnement where the client behaviour will be hard-coded in a hypothetical freshclam-torrent to serve the last two db files and properly warn the trackers when they stop distributing contents, you most probably won't experience such problems. Remember that bittorrent was designed specifically to handle the very same problem we are discussing, the quality of some popular implementations may damage the efficiency of the protocol, but in the clamav context we don't have to deal with this problem. With regard to all the other ideas: Please remember to keep this *simple*. Here's where I, IMHO, think we stand: Opening a new port on a mailserver so updates can be pushed to it is a BAD idea. As a sysadmin, I would not allow such a thing on my production machines. It creates a huge security risk, since now you have one more opening to a remote root vulnerability. The freshclam process is less risky because tricking freshclam into connecting to an arbitrary system requires injecting DNS entries somewhere, controlling the routing process or even injecting content in TCP conversations which all are more complex techniques than exploiting a buffer overflow on a server for example. But the problem is not really in opening ports or not but in controlling the quality of the code you run. TCP and DNS are not secure so freshclam must add its security layer above, theoritically it doesn't really matter if it is running as a TCP client or server. The idea of DNS sounds really good, but it doesn't appear we can fit all the data there. And putting just a version number there appears to make things worse, since it will just make everyone hit the mirrors at the same time. Using DNS doesn't mean the freshclam processes will all be waking up at the same time to query the DNS entry. They may use shorter delays than the current 1 or 2 hours, but even with one second delay, the queries leaving the local DNS servers in the recursive process would be distributed on a TTL-wide period. The TTL being controlled by the authoritative servers. If there's a problem of bandwidth/load on the mirrors, then move it away from them by designing a new (parallel ?) distribution mechanism or throw more mirrors on it. But as I said I'm not yet convinced there's a real problem. Shorting the delay from db update to db install on each mail server is good, but don't forget that 1 hour is already low compared to the time between the first virus appearance and its signature addition in the database. We will never eliminate that delay and must do with it. If we can somehow distribute signatures that way it would be nice, but it just doesn't seem practical. I'm really starting to like the idea of a mailing list that can have dedicated (and random for each site) subscription addresses and pipe the list straight into sigtool --add. It means we'd have to find someone to host the list, but that's probably no more difficult than finding someone to host a mirror. Are you kidding ? Our mirrors have seen more than 12 clamav clients last month and we only see a *part* of the *european* clients. Mirrors handled with round-robin DNS are fairly easy to setup, but sending 12 mails for each db update can very well end up burying one mail server under the load. Presumably there could even be multiple mirrors sending the list, to improve speed (taking an idea from spammers who use open relays to do the hard part). Using spammers' techniques could help but at quite a cost in complexity and in the end I don't think it would be practical for the end-users. -- Lionel Bouton - inet6 - o Siege social: 51, rue de Verdun
Re: [Clamav-users] Idea for more timely virusdb updates
Am Wednesday 11 August 2004 16:40 schrieb Damian Menscher: Hi, like Fedora. It tends to start up really slowly (since it has to find peers) and then speed up. But the speedup doesn't occur until several megs have been downloaded. If we're only sending a 1-meg main.cvd, then wouldn't bittorrent lose its advantage to all the overhead of finding peers? IMHO this is a very valid concern. With regard to all the other ideas: Please remember to keep this *simple*. Here's where I, IMHO, think we stand: Opening a new port on a mailserver so updates can be pushed to it is a BAD idea. As a sysadmin, I would not allow such a thing on my production machines. It creates a huge security risk, since now you have one more opening to a remote root vulnerability. Opening another port is simply no option for any serious enterprise use. There is simply no way to open another port in the firewall. In addition I am confident that IANA will not allow to reserve a fixed port number for this service. After all port numbers are a limited resource with todays IPv4 networks. The idea of DNS sounds really good, but it doesn't appear we can fit all the data there. Yes. And putting just a version number there appears to make things worse, since it will just make everyone hit the mirrors at the same time. This is not really such a big problem as the DNS is still no push but a pull service and the incoherency of the DNS leads to a smoothing effect. If we can somehow distribute signatures that way it would be nice, but it just doesn't seem practical. I agree with you. I'm really starting to like the idea of a mailing list that can have This is a very bad idea. As someone who is used to run _very_ big mailing lists I can tell you that the resources to run a _big_ mailing list are 3 or even 4 magnitudes bigger than a simple webserver offering the very same single file to everyone interested via HTTP GET. Offering this file e.g. 1MB via http get allows very easy to saturate any backbone with useful data _without_ the need to handle DNS lookups, generating an email, try delivery (multiple packages back and forth) and then finally having about 3-5 percent of the connection be failures -- retries, Using the very same hw resources (cpu, memory and bandwidth) with http get allows for much more (think about a factor of 100 or 1000) information be spread within a timeinterval. Things to think about: - Effort required to create a mail body - overhead of 7bit email encoding - effort required to do the email envelope - Effort for queuing many emails (much copying on the server) - No caching on the intermediate servers (only proxying) - Handling bounces etc. - Doing many DNS lookups Regards, -- martin Dipl.-Phys. Martin Konold e r f r a k o n Erlewein, Frank, Konold Partner - Beratende Ingenieure und Physiker Nobelstrasse 15, 70569 Stuttgart, Germany fon: 0711 67400963, fax: 0711 67400959 email: [EMAIL PROTECTED] --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Lionel Bouton wanted us to know: Since some time I am thinking of a bittorrent approach too. Bittorrent is quite efficient at distributing files and there are implementations allowing multiple trackers to distribute the remaining server-side load. There's a libbt library written in C that could accomplish this without having to spawn a scripting language interpreter (eg python). If I'm not mistaken, there's only one thing to add : querying for the latest available torrent which must be signed by the db authors : DNS would be great for this indeed (hash sigs prevent cache poisoning and thus distribution of invalid db, but not DoS). I like this idea, but want to study it a little more as I'm afraid of what will happen when a virus comes out that DOS's clamav nameservers (thus preventing or making much more difficult the proliferation of signatures to detect it). I think in such a case though, a fallback of poll for it anyway would be sufficient but only if DNS A records are still responsive. -- Regards... Todd They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. --Benjamin Franklin Linux kernel 2.6.3-15mdkenterprise 2 users, load average: 0.14, 0.06, 0.02 --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Idea for more timely virusdb updates
Opening another port is simply no option for any serious enterprise use. There is simply no way to open another port in the firewall. In addition I am confident that IANA will not allow to reserve a fixed port number for this service. After all port numbers are a limited resource with todays IPv4 networks. bittorrent doesn't rely on a fixed port - it doesn't need one. If I understand it right, seeders (people with full copies) and peers (people with partial downloads) register their ports with the tracker, and if shutdown properly, de-register themselves. The problem with slow starts DEFINITELY has something to do with seeders and peers not deregistering themselves (I see it in logs) and have seen FAQ's on web sites hosting torrent files to this effect. With a closed loop distribution system with a custom client that guaranteed a 10X ratio and then cleanly deregistered itself, we would have a very powerful distribution system. Might even make a project on it's own. m/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Am Wednesday 11 August 2004 18:56 schrieb Mitch (WebCob): Hi Mitch, service. After all port numbers are a limited resource with todays IPv4 networks. bittorrent doesn't rely on a fixed port - it doesn't need one. My above comment was an answer to the idea of letting an MTA run on an extra fixed port. It was in no way related to bittorrent. The problem with bittorent is that bittorent addresses a different problem domain. clamav pattern update: - frequently changing small number of small files distributed from a single point to many bittorrent: - slowly changing high number of potentially very big files distributed from many sources to many destinations. Regards, -- martin Dipl.-Phys. Martin Konold e r f r a k o n Erlewein, Frank, Konold Partner - Beratende Ingenieure und Physiker Nobelstrasse 15, 70569 Stuttgart, Germany fon: 0711 67400963, fax: 0711 67400959 email: [EMAIL PROTECTED] --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
OK folks, chiming in again with my more or less educated opinion. DNS was built exactly to distribute fast and reliably small chunks of information (most or the time address related information to start with). Is an MX record now exactly an address related information, one would guess yes. Was DNS based blacklisting thought about when the BIND system was developed? I doubt it, still I believe most of you admins around use it in some form. The DNS system will survive much heavier load then we could possibly ever burden it with. I believe a pull method is preferred over a push method, because it is easier to manage regionally. A system administrator could even decide to use a specific group of servers. I doubt that any sensible sysadmin will allow a push to be implemented anyway seee security risk. To build smaller update chunks I beg you to reconsider the patch method, even though the files are binary. They could easily be converted to a textual multi line representation, then diffed with enough redundant information to assert the patch is inserted at the right place, possibly compressed for transport and applied to a text converted file, which of course would need to be converted back, checksummed, compared, you name it. All the necessary tools to do this are standard on *nix systems and there is always cygwin for the Redmond addicts. cheers Erich THINK Püntenstrasse 39 8143 Stallikon mailto:[EMAIL PROTECTED] PGP Fingerprint: BC9A 25BC 3954 3BC8 C024 8D8A B7D4 FF9D 05B8 0A16 --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Mitch (WebCob) wrote: What about a deeper mirroring system? Perhaps one that supports notification? One of the things I like about BIND (not enough to use it, but still an admired concept ;-) is the way zones can be distributed... notification speeds things up if it works, polling creates a failsafe in which a missing notify doesn't cause the world to end... right, but as discussed below, generally bind servers don't have 100k people waiting for notifications and updates. Hourly polls is a good thing - but if the system worked both ways, the mirror could signal the end clients that it's time to download... those notifies could be send only to clients that had registered to receive it (an option in freshclam) and would not push the data, but trigger a freshclam pull. with that option, the 'clients' would either have to remain connected the entire time, which is completely not feasable, or somehow the database mirrors would have to either 'remember' who to notify, or have some sort of registry of people to notify (I can see how one might do this with a paid mirror service), and then send out notifications (even a single UDP packet to 100k servers could be quite bandwidth intensive. The architecture could work, yes, but it doesn't scale well, and I don't think the clamav team has the resources to do this sort of ass-kissing for free. They're already providing a wonderful service to the internet community, we cannot bite the hand that feeds us. Another problem with this notification is there are still the spikes when the notifications come out that EVERYONE AND THEIR BROTHER contacts the database mirrors for updates. Your solution doesn't solve any problems imposed by Christopher's idea, and actually introduces more. In my opinion, the existing system is fine, and if you want better, you should talk to the clamav folks about setting up some sort of 'priority' mirror, in which you would pay a fee for having more enhanced services, like notification, dns update polling, etc. And of course, proceeds (or at least a major part of) would go to the clamav team for being the most kick ass anti-virus product out there. I'm not sure how the official procedure would be to roll something like this out, but now that I think of it, I may just go about working on something like this. Gotta pay for my colocation somehow :) Tomasz, et al.: Please expect to see an email from me by the end of the work day tomorrow (or rather, today, but I haven't slept yet) It could provide faster update response and smooth out the spikes in download traffic, and could be used to maintain a larger set of mirrors... without increasing polling frquency... a new freshclam server could allow all larger users to easily run their own mirrors for internal distribution... I would think that most 'larger users' (5+ node mail server cluster) would already have an internal mirror. It's not difficult to do, and has been discussed on this list, and in the clamav documentation many times. Just a few ideas... hey, brainstorming is good, it's just the ideas aren't always ;) -Jeremy -- Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc. [EMAIL PROTECTED] ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Tue, 10 Aug 2004, Fajar A. Nugraha wrote: The only snag, is that TXT record is limited to a number of bytes ( I tried putting 4096 bytes on it, it didn't work). Now, the question is, can the daily (or hourly) updates fit in a single TXT record? I don't know that putting ALL of the records in DNS is necessary. The only reason I was putting the version number there was to allow quick, more frequent checks to see if you had the current version. It's possible to run DNS with very short TTL times, even 0. In terms of load on the servers: a smaller file would certainly help. I can see the simplicity of just haveing two files to grab being attractive. However, daily.cfg is now about 150k . I don't consider this big in the scheme of things, but if we are talking about hundreds or thousands of people trying to get the file, then the difference could be significant. Suppose there was a numbered file for each version that would 'upgrade' you from the previous version. IE, if I'm at 444, and current is 445, I grab 445.cvd. If I'm at 440, I grab 441.cvd, 442.cvd, through 445.cvd. Downside -- obviously harder to maintain. Upside -- someone who is staying constantly up to date is grabbing only a few bytes off the server at a time. I think this is simpler than putting all the data into DNS. I've also thought about rsync -- if putting the cvd files on an rsync server would lighten the load at all. == Chris Candreva -- [EMAIL PROTECTED] -- (914) 967-7816 WestNet Internet Services of Westchester http://www.westnet.com/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
At 16:03 10.08.2004, you wrote: ... I've also thought about rsync -- if putting the cvd files on an rsync server would lighten the load at all. Oh it would, rsync is quite effective. Another possibility might be to patch the .cvd file(s) 0.02 Erich THINK Püntenstrasse 39 8143 Stallikon mailto:[EMAIL PROTECTED] PGP Fingerprint: BC9A 25BC 3954 3BC8 C024 8D8A B7D4 FF9D 05B8 0A16 --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Erich Titl wanted us to know: I've also thought about rsync -- if putting the cvd files on an rsync server would lighten the load at all. Oh it would, rsync is quite effective. Another possibility might be to patch the .cvd file(s) Agree with rsync, depends how much changes in the file per download, but I think it can be marvelously efficient. Disagree with patching, binary files don't lend themselves to diff/patch very well. 0.02 Now we have 0.04 :-) -- Regards... Todd We should not be building surveillance technology into standards. Law enforcement was not supposed to be easy. Where it is easy, it's called a police state. -- Jeff Schiller on NANOG Linux kernel 2.6.3-15mdkenterprise 2 users, load average: 0.09, 0.03, 0.01 --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Erich Titl wrote the following on 08/10/2004 05:12 PM : At 16:03 10.08.2004, you wrote: ... I've also thought about rsync -- if putting the cvd files on an rsync server would lighten the load at all. Oh it would, rsync is quite effective. Not much with compressed files like *.cvd. Another possibility might be to patch the .cvd file(s) That was one proposition I made last year. But in practice it seems there isn't really a pressing need now. Lionel. -- Lionel Bouton - inet6 - o Siege social: 51, rue de Verdun - 92158 Suresnes / _ __ _ Acces Bureaux: 33 rue Benoit Malon - 92150 Suresnes / /\ /_ / /_ France \/ \/_ / /_/ Tel. +33 (0) 1 41 44 85 36 Inetsys S.A.Fax +33 (0) 1 46 97 20 10 --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Idea for more timely virusdb updates
right, but as discussed below, generally bind servers don't have 100k people waiting for notifications and updates. Nope, true... but like I suggested, the notification tree doesn't have to be flat... One server notifying 10 servers is time consuming and sure - costs a lot of bandwidth... Lets assume that each notify takes 5 seconds... we have to have SOMETHING to measure... 1 server notifying 10 servers takes 50 seconds. That's a little over a day to push the notification - bad idea ;-) 1 server notifying 100 servers, which each in turn notify 100 servers and so on... 1 to 100: 100 seconds each of them notifying 100: 100 seconds (total notified 10100) each of them notifying 100: 100 seconds (total notified 1010100!) in 5 minutes! That's 10 times your value of 10 servers. Each server would only have to know about 100 others. Not a huge database - wouldn't even have to be written to file. Each server could be responsible for polling it's master once per hour. Hourly polls is a good thing - but if the system worked both ways, the mirror could signal the end clients that it's time to download... those notifies could be send only to clients that had registered to receive it (an option in freshclam) and would not push the data, but trigger a freshclam pull. with that option, the 'clients' would either have to remain connected the entire time, which is completely not feasable, or somehow the database mirrors would have to either 'remember' who to notify, or have some sort of registry of people to notify (I can see how one might do this with a paid mirror service), and then send out notifications (even a single UDP packet to 100k servers could be quite bandwidth intensive. The architecture could work, yes, but it doesn't scale well, and I don't think the clamav team has the resources to do this sort of ass-kissing for free. They're already providing a wonderful service to the internet community, we cannot bite the hand that feeds us. I wasn't proposing that it had to be done for free (not that it can't be with the factor tree I explained above). It might even reduce the cost of database distribution. If each server is only pushing 100 updates @ 200KB per update (2MB total) we can get 500 pushes per month for only a couple dollars. Another problem with this notification is there are still the spikes when the notifications come out that EVERYONE AND THEIR BROTHER contacts the database mirrors for updates. Your solution doesn't solve any problems imposed by Christopher's idea, and actually introduces more. 100 servers for 200KB (20MB is hardly a spike.) and as for clients remaining connected, that is what a server is - connected. This isn't for end users, or local workstations. It's an OPTION for people who process a lot of data, are at high risk, and need immediate response. Then their own internal freshclam clients can poll their local authoritative server as often as they want, or use the same procedure to distribute to them (if they are full time connected that is). In my opinion, the existing system is fine, and if you want better, you should talk to the clamav folks about setting up some sort of 'priority' Yeah, we could, but I don't think it needs that. And setting up an internal mirror doesn't address the response time of the updates, unless I start hammering the main freshclam every few minutes... and I just don't think that would be friendly. With the sort of hierarchical distribution I'm talking about, you could even use an ranking system to automatically organize the distribtion (while I'm on a roll ;-)... What I mean is that everyone would contact one of the root mirrors initially. In the request to be notified, it would indicate the number of clients it serves. If less than a certain number, then it could be referred to a child of the root server. If that child becomes unavailable it could contact the root again (at the next hourly polling time). How many servers are there on the Internet? We could probably handle the whole lot of them with no more than 4 or 5 levels. Push an update to the world in under 10 minutes. Think how many virus laden emails this could stop. (visions of f5...) in fact, the root server could hand out the IP's of all child servers not fully loaded. The client could register with the nearest (by route time) one - just ranting... m/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Aug 10, 2004, at 5:57 AM, Jeremy Kitchen wrote: Mitch (WebCob) wrote: Just a few ideas... hey, brainstorming is good, it's just the ideas aren't always ;) Another stupid idea...how about a mechanism where clam can have updates pushed to it, so servers controlled by the clam team can distribute mini updates to them. The admins would have to subscribe to it, like a listserv, only instead of through email, it's done through this theoretical mechanism. There wouldn't be traffic spikes (as big) for times where there *aren't* updated db's available, only when there are updates, and the updates are sent out as the clam servers are able to handle the load. Maybe like a modified GPG-signed listserv system only on it's own clam update daemon port...take a little more configuration since the people installing clam would have to subscribe and install a GPG key or something like that in the process, but that shouldn't be something back-breaking to figure out. Maintenance would have to be done for the subscription mechanism, etc., like a listserv would, but it may be something that could be done. May even be extendable so that a master server for a network could receive the updates from the clam site (pushed from clamserv) then in turn be told to push them out to machines on the internal network. (I know this could already be set up, but it may be easier through this type of model to set up and maintain...) I'm probably overlooking something obvious, but again...just an idea, right? :-) -Bart --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Tue, 10 Aug 2004, Bart Silverstrim wrote: Maybe like a modified GPG-signed listserv system only on it's own clam update daemon port...take a little more configuration since the people installing clam would have to subscribe and install a GPG key or something like that in the process, but that shouldn't be something back-breaking to figure out. Ok, this is turning into a scary beast. But we already have several mailing lists (clamav-users, for example) which can obviously handle a bit of a load. Might be interesting to concoct a specially-formatted message that the milter (or clamd itself) could recognize as a database update, and automatically append to its list of signatures. I'd imagine a format something like: ---gpg-cleartext-signed-message--- BEGIN clamd update 24.449 Worm.bagle.zz CCCEEFEFKL.. Worm.SkyNet.zz 14445577 ... END ---gpg-signature--- JDSLJGIREJIOJDGLSJLGHSLKJGLKSDJLKGJSLKJGIEJ*Y*G($Y*HHIO4k245j2jk kdjaflkjkh325hjk35h2jkhkjhjkfdhjh42jkh345jk2h35jk2hkjhjkfhjskh32 fhjkhafdjhajk53h2jk5h3j2kh35jkhfay983489527938572035230398udfsfs ---end-signature--- When scanning stuff like this, clamd could automagically decode the gpg signature and test that it is valid. If so, it looks at the sequence number (24.449 in this case). If that's the next one in the series, it appends the rules to its database. If not, it assumes it lost a message somewhere and contacts a mirror via HTTP to get main 24 and daily 449. Doing something like this would push a lot of the distribution load onto sourceforge (which seems to get messages out to this list in about 1/2 hour). The gpg-signature prevents spoofing. And the sequence numbers keep everyone current. The major problems I see are getting clamd to recognize a message targeted for it, and the obvious problems of DoS attacks (someone sending spoofed messages that would suck CPU time decoding the gpg signature). Anyway, just another wild-n-crazy idea to throw out there. I'm guessing we're better off with the current method for now, but this might be an interesting possibility for the future. [I haven't given up on DNS updates yet, but it's hard to come up with a clean way to distribute 256 bytes of data that way, which means even single rules don't always fit.] Damian Menscher -- -=#| Physics Grad Student SysAdmin @ U Illinois Urbana-Champaign |#=- -=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=- -=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=- -=#| [EMAIL PROTECTED] www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=- -=#| The above opinions are not necessarily those of my employers. |#=- --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Tue, 10 Aug 2004, Lionel Bouton wrote: Another possibility might be to patch the .cvd file(s) That was one proposition I made last year. But in practice it seems there isn't really a pressing need now. If people can't check for database updates more often than once an hour, then there is a pressing need. The mirror page talkes about the need for mirrors, about exponential growth, and how at least a 10mbit pipe is needed to host a mirror. It puts March 2004 traffic at about 120gig/month Some quick calculations: daily.cvd is about 150k compressed, 334k uncompressed -- let's say 50%. Greping the virses added for updated 447 gave me about 3k uncompressed -- so let's say 2k compressed, on the outside. For 2k of update, everyone downloaded 150k. That shows (at least for that update) only 1.3% of what was downloaded was needed. If only 1.3% of every update is actually needed, and people only downloaded what they needed, the traffic on the mirrors would drop from 120gig/month to 1.6 gig/month. If I am completely off by a factor of 10 -- say only 10% of every update is actually needed, traffic on the mirrors drops from 120gig to 12gig. There are a lot of assumptions here, but I would think ever reducing the load on the virus servers by half would be significant. == Chris Candreva -- [EMAIL PROTECTED] -- (914) 967-7816 WestNet Internet Services of Westchester http://www.westnet.com/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Tuesday 10 August 2004 12:23 pm, Damian Menscher wrote: Ok, this is turning into a scary beast. But we already have several mailing lists (clamav-users, for example) which can obviously handle a bit of a load. Might be interesting to concoct a specially-formatted message that the milter (or clamd itself) could recognize as a database update, and automatically append to its list of signatures. this is actually a pretty decent idea. I think it would be best to, rather than have clamd try to detect it, have a special address on the machine that processes the message via a program. Most MTAs I'm aware of (at least on the unix side) can do this, I know qmail can for sure. I'd imagine a format something like: [snip email message for the update] Doing something like this would push a lot of the distribution load onto sourceforge (which seems to get messages out to this list in about 1/2 hour). for something like this I wouldn't use sourceforge's mail servers :P They're already bogged down as it is, us adding load to them like this would be bad, and the notifications would eventually get slower, and slower, and slower... having a dedicated list server for this purpose would be the best. The gpg-signature prevents spoofing. And the sequence numbers keep everyone current. The major problems I see are getting clamd to recognize a message targeted for it, and the obvious problems of DoS attacks (someone sending spoofed messages that would suck CPU time decoding the gpg signature). yes, that's an unfortunate problem with this idea, however, if you used, as I stated, a special address that uses program delivery, you'd have to hack the listserver to get everyone's 'subscription' address to be able to do this. Anyway, just another wild-n-crazy idea to throw out there. I'm guessing we're better off with the current method for now, but this might be an interesting possibility for the future. it definitely is interesting. [I haven't given up on DNS updates yet, but it's hard to come up with a clean way to distribute 256 bytes of data that way, which means even single rules don't always fit.] I wouldn't distribute the rule in DNS, however, a timestamp of sorts in dns isn't a bad idea. -Jeremy -- Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc. [EMAIL PROTECTED] ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Tue, 10 Aug 2004, Damian Menscher wrote: Anyone know if it's really feasible for us to obtain a mailserver that can send out 2k emails to all (100,000?) users in a short (5-10 mins) time? I haven't been following the whole discussion, but I thought this was mostly to provide support to power users. I think the average small-time admin would be happy with the hourly updates. Jeffrey Moskot System Administrator [EMAIL PROTECTED] --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On 2004-08-10 14:41:28 -0500, Damian Menscher wrote: On Tue, 10 Aug 2004, Jeremy Kitchen wrote: On Tuesday 10 August 2004 12:23 pm, Damian Menscher wrote: Ok, this is turning into a scary beast. But we already have several mailing lists (clamav-users, for example) which can obviously handle a bit of a load. Might be interesting to concoct a specially-formatted message that the milter (or clamd itself) could recognize as a database update, and automatically append to its list of signatures. [...] Before people get too excited about this idea, though, there are some issues that need to be fixed. Anyone know if it's really feasible for us to obtain a mailserver that can send out 2k emails to all (100,000?) users in a short (5-10 mins) time? How about using NNTP instead of SMTP? Then the clamav server doesn't have to push out those messages to everybody but only to its neighbours which will distribute it further. hp -- _ | Peter J. Holzer| Je höher der Norden, desto weniger wird |_|_) | Sysadmin WSR | überhaupt gesprochen, also auch kein Dialekt. | | | [EMAIL PROTECTED] | Hallig Gröde ist fast gänzlich dialektfrei. __/ | http://www.hjp.at/ | -- Hannes Petersen in desd pgpcRxcR1GytI.pgp Description: PGP signature
Re: [Clamav-users] Idea for more timely virusdb updates
Christopher X. Candreva wrote:
This thread on Trojan.JS.RunMe had me thinking: Hourly virus updates is
better than any of the commercial virus scanners, but obviously still has
issues, especially since a bunch of us obviously submitted updates that had
already been entered. I gather from these posts that the virusdb's actually
have some form of version number.
This could actualy be easily accomplished also by attaching a soa record
to a zone ... for example
dbversion.clamav.net
Incrementing the serial for that should be trivial enough.writing a
mechanism to rapidly query against it and then to invoke a freshclam is
left as an exercise to the reader.
Presumably then the lists of Nameservers for that particular zone would
be expanded to about 10 or more. Notification from whatever master zone
server could be trivialy accomplished on that.
We should probably consider that the load balancing of all those end
users/isp's DNS resolvers may not be all it can be, particularly the
selection of which nameserver to talk to out of many for a particular zone.
Anyways I did a dig Arent CNAMEs that Point to CNAMEs contrary to RFC?
Might that be behind the infrequent dns resolution complaints?
Also... Is there any single name that covers ALL mirrors?
Also Any insight as to how the { presumably dynamic } selection to
alias the db-local to db.america is done?
c:\Documents and Settings\joe.JOE.000dig database.clamav.net
; DiG 9.2.3rc3 database.clamav.net
;; global options: printcmd
;; Got answer:
;; -HEADER- opcode: QUERY, status: NOERROR, id: 41
;; flags: qr rd ra; QUERY: 1, ANSWER: 15, AUTHORITY: 5, ADDITIONAL: 2
;; QUESTION SECTION:
;database.clamav.net. IN A
;; ANSWER SECTION:
database.clamav.net.5 IN CNAME db.local.clamav.net.
db.local.clamav.net.7200IN CNAME db.america.clamav.net.
db.america.clamav.net. 5 IN A 128.121.60.235
db.america.clamav.net. 5 IN A 196.40.71.226
db.america.clamav.net. 5 IN A 199.239.233.95
db.america.clamav.net. 5 IN A 200.68.106.39
db.america.clamav.net. 5 IN A 24.244.193.21
db.america.clamav.net. 5 IN A 38.136.139.7
db.america.clamav.net. 5 IN A 64.18.103.6
db.america.clamav.net. 5 IN A 64.69.64.158
db.america.clamav.net. 5 IN A 65.75.154.69
db.america.clamav.net. 5 IN A 65.77.42.207
db.america.clamav.net. 5 IN A 66.139.75.171
db.america.clamav.net. 5 IN A 67.18.205.218
db.america.clamav.net. 5 IN A 69.93.108.98
;; AUTHORITY SECTION:
clamav.net. 7200IN NS ns5.clamav.net.
clamav.net. 7200IN NS ns1.oltrelinux.com.
clamav.net. 7200IN NS ns2.clamav.net.
clamav.net. 7200IN NS ns3.clamav.net.
clamav.net. 7200IN NS ns4.clamav.net.
;; ADDITIONAL SECTION:
ns1.oltrelinux.com. 38516 IN A 194.242.226.43
ns5.clamav.net. 153717 IN A 80.69.66.9
;; Query time: 671 msec
;; SERVER: 64.95.32.37#53(64.95.32.37)
;; WHEN: Tue Aug 10 16:40:04 2004
;; MSG SIZE rcvd: 429
---
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
___
Clamav-users mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Tue, 2004-08-10 at 12:40, Christopher X. Candreva wrote: If people can't check for database updates more often than once an hour, then there is a pressing need. [...] If only 1.3% of every update is actually needed, and people only downloaded what they needed, the traffic on the mirrors would drop from 120gig/month to 1.6 gig/month. If I am completely off by a factor of 10 -- say only 10% of every update is actually needed, traffic on the mirrors drops from 120gig to 12gig. That's one of the things that seems to be driving the size of daily.cvd up - updating main.cvd entails a massive distribution of files to the world. Perhaps a tiered approach to the update files, with main.cvd, monthly.cvd, weekly.cvd, daily.cvd, and hot.cvd The advantage there is that the really big update could be distributed very seldom - perhaps only with new code (the code generally has to be upgraded every few months to deal with a new threat anyway). If you had overlapping signatures between the files, you could add a fuzzy-factor into freshclam that it might not bring down the latest weekly/monthly if the other files overlap completely. That would distribute the load on the freshclam servers for the larger updates, and there would just be the very small daily.cvd (and perhaps hot.cvd) downloads. I like the idea of using DNS to signal the change - maybe just for hot.cvd. so, whenever a major virus breakout occurs, the new sig would be added to hot.cvd and the DNS TXT record changed. 10,000 users pulling down a 2-3K file is not terribly hard for a server with decent bandwidth -- Daniel J McDonald, CCIE 2495, CNX Austin Energy --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Tue, Aug 10, 2004 at 10:39:19PM +0200, Peter J. Holzer wrote:
On 2004-08-10 14:41:28 -0500, Damian Menscher wrote:
[... about sending clamav updates quickly to all subscribers]
Anyone know if it's really feasible for us to obtain a mailserver that
can send out 2k emails to all (100,000?) users in a short (5-10 mins)
time?
How about using NNTP instead of SMTP? Then the clamav server doesn't
Why use such an old protocol that isn't suited to binary transfers.
I've already mentioned this jokingly, but I was half serious: I think
setting up a bittorrent would solve a lot of the bandwidth problems.
You would need some place to get the daily.cvd.torrent file, which seems
to be about 170 bytes when I tried creating one yesterday (Small enough
to fit base64-encoded in a DNS TXT record, if you insist, but I doubt
that that is prudent to rely upon). Then you'd need a decent tracker,
or a bunch of trackers, and at least one seeder per tracker. I guess
that the current db.*.clamav.net hosts can easily host both a tracker
and a seeder.
If you then distribute a downloading clients that keeps seeding for just
1 hour (or until a preset share ratio was reached, say, 10x), you would
very quickly take a HUGE load off the download servers... and everyone
using clamav would automatically help the project by donating bandwidth
for the updates.
P2P - it's not just for downloading pirated Metallica mp3s.
HTH,
--
#!perl -wpl # mmfppfmpmmpp mmpffm [EMAIL PROTECTED]
$p=3-2*/[^\W\dmpf_]/i;s.[a-z]{$p}.vec($f=join('',$p-1?chr(sub{$_[0]*9+$_[1]*3+
$_[2]}-(map{/p|f/i+/f/i}split//,$)+97):qw(m p f)[map{((ord$)%32-1)/$_%3}(9,
3,1)]),5,1)='`'lt$;$f.eig;# Jan-Pieter Cornet
---
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
___
Clamav-users mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Tuesday 10 August 2004 02:41 pm, Damian Menscher wrote: [snip: using a program delivery to process update mailing list mails] With sendmail, you could add to /etc/aliases something like: clamav-updates| sigtool --add that's the ticket. Anyone know if it's really feasible for us to obtain a mailserver that can send out 2k emails to all (100,000?) users in a short (5-10 mins) time? Assuming those numbers are reasonable, that means 200 meg of data. Combined with SMTP overhead, it seems like it would be troublesome. Additionally, there are potential bandwidth issues if you consider we'd need to do that several times/day. well, I would think this would be an 'optional' thing you could do, or maybe part of a 'premium' service provided for a fee. As Jef mentioned, most small time folks are perfectly happy with hourly updates in a pull configuration. Updating the main database is one concern. Sending out a 2-meg email to everyone seems like it might be too much load, but sending out the 1K email telling everyone to get it means the mirrors will get swamped. I can't think of a way around this, but hopefully someone else can? well, I would hope that while also grabbing these daily.cvd updates via email, that the admin is also running freshclam (perhaps less frequently now that he/she only needs to check main.cvd once a day) to grab the main.cvd and doesn't need notification for it. Forgive my ignorance if I'm not interpreting the role of the main/daily.cvd files correctly: main.cvd: updated daily with all of the updates done to daily.cvd throughout the day daily.cvd: 0sec updates to the database, get rolled into main.cvd nightly Also, this doesn't give much provision for removing bad signatures (that cause false positives) since it really just appends rules. We'd need to figure out a way to delete signatures also. I could imagine doing this by including a null signature, or using some other flag. true. perhaps the first line of the email could be a command, and a simple sh/perl/c program could parse it and then call the proper commands to add or remove the signature that follows. Finally, there's the whole issue of multiplying your points of failure. If your current database is screwed, appending more to it will leave it screwed. And if you add stuff to it a few times a day, chances are it will get screwed up at some point. At least this issue has a simple fix: include an MD5 sum with the update which must match your MD5 sum after applying the update. If they don't match, you know something went wrong, either with this update or a previous one. (This has the danger that if the developers send an email with an incorrect MD5 hash, everyone will thrash the mirrors.) eek. Note to the developers: please don't feel like you have to code up any of our random ideas. I'm just having fun brainstorming about how to optimize this process. I expect in another few days of discussion we'll have converged on a fairly sane idea. or scrap the whole idea all together :) -Jeremy -- Jeremy Kitchen ++ Systems Administrator ++ Inter7 Internet Technologies, Inc. [EMAIL PROTECTED] ++ www.inter7.com ++ 866.528.3530 ++ 847.492.0470 int'l kitchen @ #qmail #gentoo on EFnet ++ scriptkitchen.com/qmail --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Jeremy Kitchen wrote: or scrap the whole idea all together :) Maybe the best thing written on the subject today! ;-) j/k But really, what's the problem? Shouldn't big time folks complain to the commercial companies to whom they pay for service and still they got updates later than Clam? Instead hundreds of mails are written here with one solution more far out than the other. Please, I *think* you might have caught the attention of the developers by now so please let them think about this for a moment. They still beat everyone else so I just want to say thank you. Everything works great! In combination with MailScanner which checks inside zip files and blocks executables I stopped all the viruses even before Clam was updated. From what I have seen from reading this list for some time many of you seem to rely to heavily on too few layers of protections. Maybe that's why you must have the updates immediately with no regard to server load or maybe I missed the solution that took care of that one too in the flood of mail. Premium servers for a fee is the best solution I have seen so far. No offence meant to anyone in particular. -- /Peter Bonivart --Unix lovers do it in the Sun Sun Fire V210, Solaris 9, Sendmail 8.12.10, MailScanner 4.32.5, SpamAssassin 2.63 + DCC 1.2.50, ClamAV 0.75.1 + GMP 4.1.2, Vispan 1.4 --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
Jeremy Kitchen wrote: On Tuesday 10 August 2004 02:41 pm, Damian Menscher wrote: [snip: using a program delivery to process update mailing list mails] With sendmail, you could add to /etc/aliases something like: clamav-updates | sigtool --add that's the ticket. And a cool little DOS tool. Nothing like a well-known email address for a little fun having. I imagine the blackhats will slam that rather quickly. dp --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
OK, here's my pitch I like the DNS idea as a way to push out just the version number of the update. This pattern serial number would be the current version of the CVD file. A record like this in tinydns: 'dbversion.clamav.net:447:600 would create a DNS TXT record for dbversion.clamav.net with a value of 447 with a TTL of 600 sec (10 minutes). I see no point in any more information being recorded. If freshclam were to initially do that DNS lookup, it could afford to look every 10 minutes instead of hourly, and would dramatically cut down on the amount of HTTP (or any other TCP) transactions required. I think all the comments about using SMTP or NNTP suffer the same problem as HTTP - they are no where near as fast or as natively multicast as DNS is - oh yeah - and it's UDP too. DNS natively shares the load, whereas all other load sharing solutions would have to be created. So I'd envisage freshclam doing the DNS lookup, and if the pattern number TXT record returned is *different* (not smaller! DNS cache poisoning can affect this solution, so just choose DIFFERENT) than the current pattern number, then it should check for an update. This has the advantage that it could just be a new bit of code added in front of the existing freshclam code. The TTL 0 allows you to even cut down the load on the primary DNS servers. The ClamAV team should make a policy saying people aren't allowed to check for updates more often than every TTL seconds and this within freshclam would enforce it. Just my 2c worth -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
In a perfect world, wouldn't this be the ultimate application for say, multicast? Just keep casting the database over and over, when it changes, you instantly have it! ;-) -- Robert Blayzor, BOFH INOC, LLC [EMAIL PROTECTED] PGP: http://www.inoc.net/~dev/ Key fingerprint = 1E02 DABE F989 BC03 3DF5 0E93 8D02 9D0B CB1A A7B0 Press Ctrl-Alt-Del now for IQ test. --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Idea for more timely virusdb updates
The mirror page talkes about the need for mirrors, about exponential growth, and how at least a 10mbit pipe is needed to host a mirror. It puts March 2004 traffic at about 120gig/month I think I read it differently... I thought it was 120GB / month per mirror (at that point in time there were 11 mirrors!) QUOTE (http://www.clamav.net/doc/mirrors/clamav-mirror-howto.txt) Without mirrors, the traffic on our main site was 100GB/month (May 2003). On Feb 2004 the traffic on each mirror (11 in total) reached 120GB/month. END QUOTE Not sure if I read it wrong, but that would put total consumption about 1320 GB - makes it more urgent doesn't it? Unfortunately the round robin - no limits nature makes the entry price for people who want to help too high for some. I wonder in the short term if there is a way to create a lower % hit mirror which could say take 10% of the normal average... at 12GB / month there might be more takers m/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Mon, 9 Aug 2004, Christopher X. Candreva wrote: This thread on Trojan.JS.RunMe had me thinking: Hourly virus updates is better than any of the commercial virus scanners, but obviously still has issues, especially since a bunch of us obviously submitted updates that had already been entered. I gather from these posts that the virusdb's actually have some form of version number. Suppose there was a DNS entry, say virusdb.clamav.net (or version.virusdb.clamav.net, etc), that returned simply a text record with the current DB version in it. Then, it would be possible to check the version with a relatively cheap single UDP packet, rather than a full http check, and people could check for DB updates more often than once an hour without taxing the distribution system. That's a very interesting idea, but I can imagine a few problems: - we'd have to have a very short time-to-live or it would get stale - the dns might know about the update before the mirrors all get it - if everyone finds out about an update within 5 minutes of each other, the mirrors might not handle the load After seeing a Defcon talk on putting arbitrary data in DNS, though, I wonder if we could put the daily updates (gpg signed) into DNS? That would take a lot of load off the mirrors (occasional checks for main.cvd updates are all that is required). And caching DNS servers would distribute the load a bit. Anyway, just another crazy idea for the developers to consider. Damian Menscher -- -=#| Physics Grad Student SysAdmin @ U Illinois Urbana-Champaign |#=- -=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=- -=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=- -=#| [EMAIL PROTECTED] www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=- -=#| The above opinions are not necessarily those of my employers. |#=- --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Mon, 2004-08-09 at 16:55 -0400, Christopher X. Candreva wrote: This thread on Trojan.JS.RunMe had me thinking: Hourly virus updates is better than any of the commercial virus scanners, but obviously still has issues, especially since a bunch of us obviously submitted updates that had already been entered. I gather from these posts that the virusdb's actually have some form of version number. Suppose there was a DNS entry, say virusdb.clamav.net (or version.virusdb.clamav.net, etc), that returned simply a text record with the current DB version in it. Then, it would be possible to check the version with a relatively cheap single UDP packet, rather than a full http check, and people could check for DB updates more often than once an hour without taxing the distribution system. If nothing else, if this TXT record existing we could hack together some shell script to check it and run freshclam as needed. Then all users would sworm to download the new sig, as soon as that serial number incrimented, flooding the download server with update requests. --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Mon, Aug 09, 2004 at 05:33:05PM -0400, Chris Meadors wrote:
Suppose there was a DNS entry, say virusdb.clamav.net (or
version.virusdb.clamav.net, etc), that returned simply a text record with
the current DB version in it. Then, it would be possible to check the
version with a relatively cheap single UDP packet, rather than a full http
check, and people could check for DB updates more often than once an hour
without taxing the distribution system.
Then all users would sworm to download the new sig, as soon as that
serial number incrimented, flooding the download server with update
requests.
Only tracker.clamav.net (can be loadbalanced) should be able to handle
a fair number of connections, but daily.cvd.torrent is small enough
you could put it in a DNS TXT record :) (OK, DNS is far from secure,
so reliability will be at stake in that case... you might need to
cryptographically sign the file).
(1/2 :-)
--
#!perl -wpl # mmfppfmpmmpp mmpffm [EMAIL PROTECTED]
$p=3-2*/[^\W\dmpf_]/i;s.[a-z]{$p}.vec($f=join('',$p-1?chr(sub{$_[0]*9+$_[1]*3+
$_[2]}-(map{/p|f/i+/f/i}split//,$)+97):qw(m p f)[map{((ord$)%32-1)/$_%3}(9,
3,1)]),5,1)='`'lt$;$f.eig;# Jan-Pieter Cornet
---
SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media
100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33
Save 50% off Retail on Ink Toner - Free Shipping and Free Gift.
http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285
___
Clamav-users mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Idea for more timely virusdb updates
What about a deeper mirroring system? Perhaps one that supports notification? One of the things I like about BIND (not enough to use it, but still an admired concept ;-) is the way zones can be distributed... notification speeds things up if it works, polling creates a failsafe in which a missing notify doesn't cause the world to end... Hourly polls is a good thing - but if the system worked both ways, the mirror could signal the end clients that it's time to download... those notifies could be send only to clients that had registered to receive it (an option in freshclam) and would not push the data, but trigger a freshclam pull. It could provide faster update response and smooth out the spikes in download traffic, and could be used to maintain a larger set of mirrors... without increasing polling frquency... a new freshclam server could allow all larger users to easily run their own mirrors for internal distribution... Just a few ideas... m/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Idea for more timely virusdb updates
On Monday, August 09, 2004 11:18 PM [EST], Fajar A. Nugraha wrote: You know, this isn't so crazy after all. I put arbitrary data on my DNS server so that exim can get config data using dnsdb lookup. Its cheaper than mysql lookup (Plus, you eliminate single point of failure), and you can still update config from a central location instead of updating each server config. The only snag, is that TXT record is limited to a number of bytes ( I tried putting 4096 bytes on it, it didn't work). Now, the question is, can the daily (or hourly) updates fit in a single TXT record? If it must span multiple records than it will be somewhat complicated Regards, Fajar I'd not recommend putting all the data in TXT records.TXT records can be a max of 255 characters (anything more and you'll have problems with other resolvers and such). But yeah, the version number in the TXT records would be good, set the TTL to about 30-60 mins, and have the freshclam client query and check the version. I could assist with implementing the necessary server side scripts to make the DNS management part really easy (I do something similar to this all the time, rbldnsd makes stuff like this stupidly simple and quick). Hell, I could even offer the DNSbl servers that the AHBL has to host the zones if needbe. -- Brian Bruns The Summit Open Source Development Group Open Solutions For A Closed World / Anti-Spam Resources http://www.sosdg.org The Abusive Hosts Blocking List http://www.ahbl.org --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
