On Thu, Aug 03, 2000 at 10:14:51PM -0400, Travis Bemann wrote: > On Thu, Aug 03, 2000 at 06:11:08PM -0500, Scott G. Miller wrote: > > Basically, all the design criteria for a datstore index are the same as > > for a general database indice. > > > > > 1. How do you efficiently search indices (remember that datastore > > > indices are like the flat filesystems on old bitty boxen)? > > B+ tree, or a hash structure. > > Actually, I was already thinking of using a binary tree. Then you need to do some research on data structures. A binary tree is *not* a scalable datastructure for high volume disk access. Read up on a B+ tree and come back to me.
> > Any modern 2nd-level storage algo handles this. > > Yeah, and that is what I am planning to do with nfreenetd. Have you > implemented such an algorithm in the Java Freenet node? I believe there is code waiting to be rolled in, going into 0.4 due to time constraints. > > You don't remember the 2 GB file size limit imposed by Linux? Of > course, breaking up files and using deletion resistant coding would > get around this, but you need to make software as foolproof as > possible. Yeah, breaking up files would solve this, but some idiot > with an SDSL line *will* eventually try to put a full uncompressed > DeCSSed version of The Matrix onto Freenet. You need to consider this > case, even if you don't think it will ever happen. You're trying to put all the files into a single flat db file? Thats just bad coding. If you really want the kind of functionality thats that scalable, why dont you just interface a relational database? Why duplicate the effort? > > > > I have to challenge this assumption. Datastores aren't meant to be this > > multi-gigabyte repository. Running a site like flys in the face of the > > whole idea behind Freenet and distributed datastores. You don't *need* to > > run these enormous centralized/webserver like nodes. > > This depends a lot upon Freenet's topology, which depends on > mechanisms which nodes find out about each other through. If it takes > a UUCPNET type topology, you *will* have these sorts of nodes. I'm arguing that if these kinds of nodes must exist, then Freenet isn't doing its job. > > In fact, I think a lot of your criticisms really don't make much > > sense. You're not designing a webserver. It doesnt *have* to scale to > > enterprise levels. > > Okay, so what if I have a pile of money I want to waste and I use it > to buy a nice quad Pentium III box with a nice big 40 GB hard drive > and a really nice video card (for playing Quake III Arena)? I have > all this hard drive space to spare, so I devote 20 gigs to a Freenet > node datastore. I also have a real nice SDSL line, and I don't > download piles of pr0n and music all day, so I dedicate a large amount > of bandwidth to the Freenet node. People who know that I have a fast > computer with a ton of bandwidth and a massive datastore will start > hooking up their local nodes to my node. Next thing you find, my > computer would get bogged down if it wasn't for my nice scalable > Freenet node implementation. Yeah, this would not happen in the ideal > case, but the ideal case rarely occurs in most situations. In a > situation like this one, you do want a very scalable Freenet node. > Yeah, this might not be likely to happen in your mind, but it *will* > happen. Fine. Go for it. Never said you couldn't do it, just that it really isn't necessary to have these monster freenet processes for freenet to succeed. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 232 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20000803/0da79a66/attachment.pgp>
