On Wed, Nov 20, 2002 at 05:48:12PM -0800, Tyler Riddle wrote: > This was posted to tech at freenetproject.org but I did > not see any word of here or any activity regarding it > on the tech list so I'm posting it here. This is > something I have been advocating for a while (not > entirely in this manor, but the functionality is the > same) and I also posted to tech@ some time ago and got > ignored as well. This would be a great feature but I > see a huge potential for abuse as well. I think it > would be worth it to discuss how to pull this > particular feature off if it is at all possible with > out bringing the network to it's knees. > > Tyler > > Begin original message: > > The problem: > Freenet is slow. Loading a freesite from an activelink > takes a long > time, and when you finally get the HTML file, it takes > even longer > before you see the graphics (assuming you get to see > all of them). If > you then navigate to a secondary page of that > freesite, odds are that > it won't even load. Whereas ZIPping it up (jar = zip), means that you fetch the whole freesite, which will take even longer - but once you've got it subpages will load quickly. > The problem isn't one of bandwidth, but one of > latency. For each file > that is needed, one or more key(s) must be retrieved, > and for each key > that is needed, a little "roamer" is sent out on the > net to visit each > and every node (within a radius) one by one, hoping to > run into the > sought after key and come home with it. This takes a > long time. I > therefore submit that, because of this, it takes much > longer to > retrieve ten 10KB files than it takes to retrieve one > 100KB file, it's It's a bit slower, but you can run the 10 requests in parallel. But here's the nice trick: if you FEC encode the 100kB file into 15 10kB chunks, only 10 of which are needed, and use a reasonable number of parrallel requests, you can fetch 10 of the 15 in a reasonable time, much less than the time taken to fetch all 10 non-redundantly. And in fact, the variability of the time to fetch 10 of 15 is probably better than the variability to fetch the single file, and given the latencies involved it probably doesn't take any longer either. So zip the site up, then FEC encode it, if it's not REALLY small (64k or whatever). It should load not much slower than the 100kB file, and its loading time should be much more predictable (freenet latency is not just high, it's highly variable). > the nature of the beast. Furthermore, since it takes > ten little roamers > to retrieve the ten keys and only one to retrieve the > one key, it > causes ten times the traffic on the net, compounding > the problem. Yeah. It does produce more traffic, which may not be a good thing. > A solution: > JAR files. Take the few HTML files and the few graphic > files that make > a freesite, pack them in a .jar file and insert it at > a "/site//" type > URI. URI's like "/site//index.html" and > "/site//images/activelink.gif" > would retrieve the .jar file and extract the target > file. Right. So once you've loaded the activelink you've loaded the entire site. It will take a bit longer, but as long as the site is small it won't take an unreasonable time. The other possible disadvantage is that files in the JAR file don't collide with other files that have the same value - freenet normally saves space by files on one freesite which are the same as files on another having the same CHK. > The advantages: > 1. you only need to retrieve one file to render a > complete freesite > (including secondary pages). > 2. by doing so, you replicate the whole site, not just > the page(s) you > visited. This in itself is a good thing. > 3. activelinks replicate the whole site as well. This is debatable, but we should provide the tradeoff to site authors at least for small sites. > 4. the .jar files are compressed, improving DS usage > and transmission > times. Yeah, for the HTML and plaintext. Not much else. We need to implement something like Content-Transfer-Encoding: gzip, separately from JAR sites, just for big text/html/xml/svg/smil/whatever files. But it's not a priority. > 5. insertions are faster and more efficient. Maybe so. > 6. chatter on the net is reduced dramatically. Reduced traffic, yeah, that's a good thing. > 7. using real .jar files leverages existing technology > in the JDK. Um, the JAR file format is identical to the PKZIP format, which is MANY years older than Java :) > 8. users could even use the jar tool to create them > (until insertion > tools get smarter). > A few things to keep in mind: > 1. the .jar files should be kept under 1MB to keep > them from splitting > (and for practical reasons). Yeah. But we should support FEC splitfiles for JAR sites, for reason (2). Non-redundant splitfiles for jar sites is stupid, but redundant splitfile jar sites, for medium sized sites, is reasonable. > 2. freesites should not include files for download in > the .jar files, > instead they should be inserted separately and linked > to by CHK. Yeah. Up to the freesite tool authors... probably could be done by file type and file size. > 3. .jar files not accessed through a "//" URI should > simply be > retrieved. Yeah, the method should be set out by the metadata in the manifest. Either it redirects to CHKs, or it redirects to files in the zip... Or we make a new keytype, ZIP@<chk fields>/<filename>, and use that in the manifest :). The final tweak is to make sure it works for the SSK being both the ZIP and the manifest, for really small sites. > 4. users can still use other methods if they prefer > (maps/manifests). > This simple feature has the potential to greatly > improve Freenet's > worst perceived shortcoming. Well... I'm not convinced of that. It would however help to have the option for a freesite to be atomic (in the sense of if you have part of it you have the whole site). > Yves Lempereur > > ===== > AIM:rllybites Y! Messenger:triddle_1999 >
-- Matthew Toseland toad at amphibian.dyndns.org amphibian at users.sourceforge.net Freenet/Coldstore open source hacker. Employed full time by Freenet Project Inc. from 11/9/02 to 11/1/03 http://freenetproject.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20021121/6012550a/attachment.pgp>
