Re: [Clamav-users] Second-tier Mirrors...
I am not sure if this will work or not since I haven't tried it, but what about taking advantage of this: http://it.slashdot.org/article.pl?sid=04/08/28/2330252&tid=95&tid=218 If you could use this URL to get the database it should work. http://database.clamav.net.nyud.net:8090/ Also, I could donate some bandwidth as a mirror. Contact me off list if you are interested. -- Jacob S. Barrett [EMAIL PROTECTED] www.amduat.net "I don't suffer from insanity, I enjoy every minute of it." --- This SF.Net email is sponsored by BEA Weblogic Workshop FREE Java Enterprise J2EE developer tools! Get your free copy of BEA WebLogic Workshop 8.1 today. http://ads.osdn.com/?ad_id=5047&alloc_id=10808&op=click ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Second-tier Mirrors...
On Wed, 2004-08-25 at 16:28, Mitch (WebCob) wrote: <..snip..> > Perhaps (not sure of the DNS system in place) could be arranged so that 10% > of the requests a full primary mirror receives could be directed to a > secondary level mirror. With a committment of only roughly 10GB per month, > we'd get more volunteers (I'd volunteer 2). We could/would mirror at 10GB/month. -- Matthew Keller signat-url: http://mattwork.potsdam.edu/signat-url/ "No one ever says, 'I can't read that ASCII E-mail you sent me.'" --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Second-tier Mirrors...
Mitch (WebCob) wrote the following on 08/26/2004 10:47 AM : -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Graham Toal Aren't we missing something obvious here? Shouldn't we be using some sort of distributed technology like BitTorrent? That's been asked and answered... Bittorrent is meant to optimize download of large files when there are many peers. Not really, the protocol is tunable to scales down too (just use small chunks for the torrents). Sure there will be more overhead BW-wise because the aggregated BW is always more than the one you get using a central point of distribution and the overhead gets bigger when the files' sizes go down. But this isn't the point, the point is : you get rid of the central distribution point(s) bottleneck. The aggregated BW is distributed on as many links as freshclam instances... You may want to have more than one tracker though with DNS views to redirect people to a local tracker, I don't think a tracker scales well beyond 10 clients today (you could use the multi-tracker extensions too). We could effect the many peers, but the size of the files involved are often finished downloading before a torrent file is downloaded parsed, and attempted (there are always unreachable / not responding hosts / slow hosts / bad routes etc.) a summarization of my understanding anyways. m/ In fact the main obstacle is the firewall setup needed for such a scheme for each client. One possibility would be to provide 2 distributions paths : the current one, DNS-enhanced for administrators worrying about a new port open, a parallel one for people looking for near zero-delay, using the bittorrent protocol with adequate sigs for torrent as I described earlier on this list. Anybody with some time to spare ? -- Lionel Bouton - inet6 - o Siege social: 51, rue de Verdun - 92158 Suresnes / _ __ _ Acces Bureaux: 33 rue Benoit Malon - 92150 Suresnes / /\ /_ / /_ France \/ \/_ / /_/ Tel. +33 (0) 1 41 44 85 36 Inetsys S.A.Fax +33 (0) 1 46 97 20 10 --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Second-tier Mirrors...
> -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] Behalf Of Graham > Toal > > Aren't we missing something obvious here? Shouldn't we be using some > sort of distributed technology like BitTorrent? > That's been asked and answered... Bittorrent is meant to optimize download of large files when there are many peers. We could effect the many peers, but the size of the files involved are often finished downloading before a torrent file is downloaded parsed, and attempted (there are always unreachable / not responding hosts / slow hosts / bad routes etc.) a summarization of my understanding anyways. m/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Second-tier Mirrors...
> In fact that is why I chose to use the term franchisee earlier - so far > I've described only the first layer in all of this. A full working model > would be a tree structure not unlike the Amway model. Only without the > huckstering. > Associate it with a business model that creates revenue for the nodes and > the mother ship and you are on your way. I've actually done this once (taken a distributed project commercial) quite successfully, so I have a pretty good idea of what is involved; and I also have a very clear understanding from the customer side of why you would want a paid-for commercial AV service and why you would pay more to someone like NAI when it would appear on the surface that you get better, faster, and cheaper service free from this project. Having said that, I do not believe that what you are suggesting is good for this project. The air of commercial respectability that would enable clamav to penetrate the market currently held by commercial AV companies would *not* be created by implementing this distributed franchise idea; in fact to the average businessman it would look more chaotic than it does already. And from the clamav project side, to implement what is really needed to make it commercial-grade quality with guarantees of service, is a major undertaking that the principals may not want to get involved with, especially if they are dedicated to providing an alternative to commercial services. My recommendation is to keep it free, and rely on distributed systems and finding sufficient volunteers to handle the bandwidth issues, and perhaps modifying the technology so that in the style of BitTorrent, you get good performance out of it only if you are willing to contribute bandwidth yourself. This can all be done by code, with the central clearing house no longer the main source of data for everyone but only the seed for the network and the orchestrator and arbiter of fair performance - sending control packets around in real time is far less overhead than sending databases... Graham --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Second-tier Mirrors...
On Wed, 25 Aug 2004, Dennis Peterson wrote: > Tim Howell said: > > > > I would love to setup a mirror, but 10Mbps and 100GB/month is more than > > I've got available. > > Those figures don't account for clever scripting that some folks are > willing to implement that defeat the round-robin distribution now in > place. If you have a faster than average site you will be targetted by > aggressive systems as the trend continues. Jerks that target you are trivial to filter (or serve old DBs to, if you're feeling cruel). It's the jerks that follow the round-robin that are hard to track. This idea of splitting a "first-tier" cname among 10+ "second-tier" mirrors sounds like a really good idea, and one that would allow me to offer a mirror. My biggest concern is that we need to ensure the second-tier mirrors are just as stable (uptime requirements) as the primary mirrors, to avoid freshclam errors. Does anyone have stats on how much the DNS-trick will save on overall mirror bandwidth? Damian Menscher -- -=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=- -=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=- -=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=- -=#| <[EMAIL PROTECTED]> www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=- -=#| The above opinions are not necessarily those of my employers. |#=- --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Second-tier Mirrors...
Tim Howell said: > > > I would love to setup a mirror, but 10Mbps and 100GB/month is more than > I've got available. > Those figures don't account for clever scripting that some folks are willing to implement that defeat the round-robin distribution now in place. If you have a faster than average site you will be targetted by aggressive systems as the trend continues. dp --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Second-tier Mirrors...
Graham Toal said: >> Currently each mirror contributes around 100GB of traffic monthly >> >> Perhaps (not sure of the DNS system in place) could be arranged so that >> 10% >> of the requests a full primary mirror receives could be directed to a >> secondary level mirror. With a committment of only roughly 10GB per >> month, >> we'd get more volunteers (I'd volunteer 2). >> >> Also, while I'm at it, sponsors of open source products are often >> credited - >> the mirrors should have a web page crediting the responsible hosts with >> banners / links to them if they would like it... (or has this appeared >> at >> some point since I started participating)... Of course tier 1 mirrors >> would >> get top billing. ;-) > > Aren't we missing something obvious here? Shouldn't we be using some > sort of distributed technology like BitTorrent? > > I'm sure I've seen distribution schemes that organise a tree of downloads > with a suitable branching ratio so that no one server is overloaded, and > I've also seen tricks for pushing out one copy which is daisy-chained > through many servers in real time so that everyone in the chain has > one input feed and one output feed, but everyone receives the data > at the same time, modulo a few seconds latency from end to end... In fact that is why I chose to use the term franchisee earlier - so far I've described only the first layer in all of this. A full working model would be a tree structure not unlike the Amway model. Only without the huckstering. > > We need to be looking at more creative schemes in line with modern > distributed and parallel networks. If it's good enough for the warez > kiddies it should be good enough for us. > > I bet you with a few hours thought we could come up with a hybrid > scheme which pushed out updates over a hierarchy to a sufficient > number of 24x7 fixed-ip machines, from which dynamic unpushable > clients could poll and fetch updates from without a damaging overhead > on any one server. Associate it with a business model that creates revenue for the nodes and the mother ship and you are on your way. dp --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Second-tier Mirrors...
> Currently each mirror contributes around 100GB of traffic monthly > > Perhaps (not sure of the DNS system in place) could be arranged so that 10% > of the requests a full primary mirror receives could be directed to a > secondary level mirror. With a committment of only roughly 10GB per month, > we'd get more volunteers (I'd volunteer 2). > > Also, while I'm at it, sponsors of open source products are often credited - > the mirrors should have a web page crediting the responsible hosts with > banners / links to them if they would like it... (or has this appeared at > some point since I started participating)... Of course tier 1 mirrors would > get top billing. ;-) Aren't we missing something obvious here? Shouldn't we be using some sort of distributed technology like BitTorrent? I'm sure I've seen distribution schemes that organise a tree of downloads with a suitable branching ratio so that no one server is overloaded, and I've also seen tricks for pushing out one copy which is daisy-chained through many servers in real time so that everyone in the chain has one input feed and one output feed, but everyone receives the data at the same time, modulo a few seconds latency from end to end... We need to be looking at more creative schemes in line with modern distributed and parallel networks. If it's good enough for the warez kiddies it should be good enough for us. I bet you with a few hours thought we could come up with a hybrid scheme which pushed out updates over a hierarchy to a sufficient number of 24x7 fixed-ip machines, from which dynamic unpushable clients could poll and fetch updates from without a damaging overhead on any one server. We can't afford to use the old gentle methods. The viruses don't. We need to push updates out almost instantly. Blaster went round the world in 15 minutes. We have to be able to fully distribute our updates in 5 - efficiently, but not in a way that causes net congestion on any one link. G --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Second-tier Mirrors...
> I would love to setup a mirror, but 10Mbps and 100GB/month is more than > I've got available. > > --TWH By my count that makes 5 of us I recall seeing volunteer and it isn't even an option yet. As we are already trampling the rules with cnames to cnames... what about this... the second tier cnames could exist as multiple rr ips in a single mirror cname... thereby sharing the required bandwidth. Just an idea. I don't think the problem will be adding a few more sites to push updates too... it would probably be much worse to manage the complexity of 2 tier deployment than to just update the secondary mirrors after the primary mirrors from the same source... 10 updates vs 100 updates won't kill the main source... when it becomes 1000's maybe we worry ;-) m/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Second-tier Mirrors...
On Mon, 23 Aug 2004, Damian Menscher wrote: >> It would help if you could define what you mean by a second-tier mirror. >> If you allow just anyone to connect, then what makes you second-tier >> instead of primary-tier? And if you restrict your connections to come >> from within your domain, then why do you need to become an official >> mirror at all? >Since there was no response, I'll offer an idea: >What about one set of mirrors that host the main.cvd and another set of mirrors >that host the daily.cvd? Assuming people use the DNS to check what updates they >need, they could then connect to the appropriate "class" of mirrors to get the >actual updates. >It seems to me that this could be a simple way to split the load and allow >potential mirrors to choose how much they want to host (main only, daily only, or >both). Sorry about not responding. Mitch clarifies the idea a little bit in another post. Basically, the DNS records would be changed so that second-tier mirrors get a smaller percentage of hits than first-tier mirrors. Maybe you even have a "master" second-tier mirror that is the only one which actually receives pushes from the ClamAV team; as soon as it receives an update it begins the process of pushing it out to the rest of the second tier mirrors. It would also be responsible for keeping track of whether or not the second-tier mirrors are functioning, and any other housekeeping that could be off-loaded from the main ClamAV servers to prevent too many mirrors becoming a hassle for the ClamAV team. I would love to setup a mirror, but 10Mbps and 100GB/month is more than I've got available. --TWH --MIS Supervisor, FEFC Fullerton (http://www.evfreefullerton.com) --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
RE: [Clamav-users] Second-tier Mirrors...
> > > Someone recently suggested the idea of allowing sites with > less than the > > > mirror site requirements becoming second-tier mirrors. This thread is > > > an attempt to see what kind of interest there is in such an > idea and for > > > the developers to respond whether or not the idea has merit. > > > > It would help if you could define what you mean by a second-tier mirror. > > If you allow just anyone to connect, then what makes you second-tier > > instead of primary-tier? And if you restrict your connections to come > > from within your domain, then why do you need to become an official > > mirror at all? > > Since there was no response, I'll offer an idea: > > What about one set of mirrors that host the main.cvd and another set of > mirrors that host the daily.cvd? Assuming people use the DNS to check > what updates they need, they could then connect to the appropriate > "class" of mirrors to get the actual updates. > > It seems to me that this could be a simple way to split the load and > allow potential mirrors to choose how much they want to host (main only, > daily only, or both). > I suggested this (2 tier mirrors) at one point - not sure if it was me you are referring to or not... I was thinking something like this: Currently each mirror contributes around 100GB of traffic monthly Perhaps (not sure of the DNS system in place) could be arranged so that 10% of the requests a full primary mirror receives could be directed to a secondary level mirror. With a committment of only roughly 10GB per month, we'd get more volunteers (I'd volunteer 2). Also, while I'm at it, sponsors of open source products are often credited - the mirrors should have a web page crediting the responsible hosts with banners / links to them if they would like it... (or has this appeared at some point since I started participating)... Of course tier 1 mirrors would get top billing. ;-) m/ --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Second-tier Mirrors...
On Mon, 23 Aug 2004, Damian Menscher wrote: > On Mon, 23 Aug 2004, Tim Howell wrote: > > > Someone recently suggested the idea of allowing sites with less than the > > mirror site requirements becoming second-tier mirrors. This thread is > > an attempt to see what kind of interest there is in such an idea and for > > the developers to respond whether or not the idea has merit. > > It would help if you could define what you mean by a second-tier mirror. > If you allow just anyone to connect, then what makes you second-tier > instead of primary-tier? And if you restrict your connections to come > from within your domain, then why do you need to become an official > mirror at all? Since there was no response, I'll offer an idea: What about one set of mirrors that host the main.cvd and another set of mirrors that host the daily.cvd? Assuming people use the DNS to check what updates they need, they could then connect to the appropriate "class" of mirrors to get the actual updates. It seems to me that this could be a simple way to split the load and allow potential mirrors to choose how much they want to host (main only, daily only, or both). Damian Menscher -- -=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=- -=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=- -=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=- -=#| <[EMAIL PROTECTED]> www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=- -=#| The above opinions are not necessarily those of my employers. |#=- --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
Re: [Clamav-users] Second-tier Mirrors...
On Mon, 23 Aug 2004, Tim Howell wrote: > Someone recently suggested the idea of allowing sites with less than the > mirror site requirements becoming second-tier mirrors. This thread is > an attempt to see what kind of interest there is in such an idea and for > the developers to respond whether or not the idea has merit. It would help if you could define what you mean by a second-tier mirror. If you allow just anyone to connect, then what makes you second-tier instead of primary-tier? And if you restrict your connections to come from within your domain, then why do you need to become an official mirror at all? Damian Menscher -- -=#| Physics Grad Student & SysAdmin @ U Illinois Urbana-Champaign |#=- -=#| 488 LLP, 1110 W. Green St, Urbana, IL 61801 Ofc:(217)333-0038 |#=- -=#| 4602 Beckman, VMIL/MS, Imaging Technology Group:(217)244-3074 |#=- -=#| <[EMAIL PROTECTED]> www.uiuc.edu/~menscher/ Fax:(217)333-9819 |#=- -=#| The above opinions are not necessarily those of my employers. |#=- --- SF.Net email is sponsored by Shop4tech.com-Lowest price on Blank Media 100pk Sonic DVD-R 4x for only $29 -100pk Sonic DVD+R for only $33 Save 50% off Retail on Ink & Toner - Free Shipping and Free Gift. http://www.shop4tech.com/z/Inkjet_Cartridges/9_108_r285 ___ Clamav-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/clamav-users
