On Thu, Aug 03, 2000 at 10:47:44PM -0500, Scott G. Miller wrote: > On Thu, Aug 03, 2000 at 10:14:51PM -0400, Travis Bemann wrote: > > On Thu, Aug 03, 2000 at 06:11:08PM -0500, Scott G. Miller wrote: > > > Basically, all the design criteria for a datstore index are the same as > > > for a general database indice. > > > > > > > 1. How do you efficiently search indices (remember that datastore > > > > indices are like the flat filesystems on old bitty boxen)? > > > B+ tree, or a hash structure. > > > > Actually, I was already thinking of using a binary tree. > Then you need to do some research on data structures. A binary tree is > *not* a scalable datastructure for high volume disk access. Read up on a > B+ tree and come back to me.
No, the reason why I want to use a binary tree is to do high speed file index lookups. The reason why is that you don't have to compare the key you are looking for with each other key for other files until you come upon the file the key you have points to. And also, this would greatly reduce the time spent in the worst case (which would be searching for a nonexistant item with a plain sequential search). > > You don't remember the 2 GB file size limit imposed by Linux? Of > > course, breaking up files and using deletion resistant coding would > > get around this, but you need to make software as foolproof as > > possible. Yeah, breaking up files would solve this, but some idiot > > with an SDSL line *will* eventually try to put a full uncompressed > > DeCSSed version of The Matrix onto Freenet. You need to consider this > > case, even if you don't think it will ever happen. > You're trying to put all the files into a single flat db file? Thats just > bad coding. If you really want the kind of functionality thats that > scalable, why dont you just interface a relational database? Why > duplicate the effort? No, I'm not silly enough to put all the stuff in a single file. As for the mention of breaking up the *index*, that was just so that the node would handle an extreme case (which would probably never happen, but remember that there have been plenty of stuff that have been said that they could never happen and did happen). > > > > > > I have to challenge this assumption. Datastores aren't meant to be this > > > multi-gigabyte repository. Running a site like flys in the face of the > > > whole idea behind Freenet and distributed datastores. You don't *need* to > > > run these enormous centralized/webserver like nodes. > > > > This depends a lot upon Freenet's topology, which depends on > > mechanisms which nodes find out about each other through. If it takes > > a UUCPNET type topology, you *will* have these sorts of nodes. > > I'm arguing that if these kinds of nodes must exist, then Freenet isn't > doing its job. Well, to determine this kind of stuff, we will have to do testing and modeling (with Serapis, of course). > > > In fact, I think a lot of your criticisms really don't make much > > > sense. You're not designing a webserver. It doesnt *have* to scale to > > > enterprise levels. > > > > Okay, so what if I have a pile of money I want to waste and I use it > > to buy a nice quad Pentium III box with a nice big 40 GB hard drive > > and a really nice video card (for playing Quake III Arena)? I have > > all this hard drive space to spare, so I devote 20 gigs to a Freenet > > node datastore. I also have a real nice SDSL line, and I don't > > download piles of pr0n and music all day, so I dedicate a large amount > > of bandwidth to the Freenet node. People who know that I have a fast > > computer with a ton of bandwidth and a massive datastore will start > > hooking up their local nodes to my node. Next thing you find, my > > computer would get bogged down if it wasn't for my nice scalable > > Freenet node implementation. Yeah, this would not happen in the ideal > > case, but the ideal case rarely occurs in most situations. In a > > situation like this one, you do want a very scalable Freenet node. > > Yeah, this might not be likely to happen in your mind, but it *will* > > happen. > Fine. Go for it. Never said you couldn't do it, just that it really > isn't necessary to have these monster freenet processes for freenet to > succeed. Extreme cases will happen, and you should account for them even if they are quite unlikely. -- Travis Bemann Sendmail is still screwed up on my box. My email address is really bemann at execpc.com. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 3360 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20000803/cdae0b17/attachment.pgp>
