On Fri, Dec 16, 2005 at 10:12:00AM +0100, tubbie at games-on.nl wrote: > Original message was posted November the 12th. Wasn't subscribed to the > mailing list then so this is from the archives.. > > > "It would be nice to keep temp files, and client cache, in the datastore. > This would give freenet a fixed space usage, and eliminate > out-of-disk-space and too-many-open-files errors." > > It would just move the problem, the datastore can still get full. Imho it > is very easy to explain to the user that when you download things it will > cost him additional diskspace.
No, because Java has no way of telling how much disk space is used at present. Whereas we have full control of the store. We can certainly limit the temp space used from the store, and the client-cache space used from the store; we will have to do this even if they are on disk as temp files. And putting them in the store does gain some security (and some space), as at present we use individual files which are padded to the next power of two size; some information i.e. the size of the temp files we were using can be extracted relatively easily. And any files which hadn't been closed yet, we can get the full length. Of course in 0.5, we don't encrypt and we don't pad. :) > > "It would also be nice if aforesaid attacker > could not trivially obtain a list of keys in the store, without having a > list of known keys already and doing some computational work." > > Agreed, this isn't too difficult. Right. > > "When we want to make a temp file, we simply create a series of fake > keys. We keep these only in RAM. The on-disk structure is not > affected. We will have to keep these blocks from being reallocated, > but that can be done in RAM; we are using the file for something > anyway, so the small memory overhead is irrelevant." > > What is the content of these temp files? I'll presume these are > in-progress downloads and (maybe) in progress uploads. If that is the case > how would we handle restarts of the node. If the node would restart we > would lose all this data. To handle this we would need to store this > information on hard drive, and then the whole purpose would be lost. Or am > i missing something? Yeah, I am here talking about immediate mode downloads, FEC encodes and so on; stuff that is transient anyway. Persistent downloads through the download manager will need to be stored to disk. There are various options for this, such as writing the downloaded data directly to the target file (renamed so users don't use it until it's done), which is what Frost does. But we will always need temp buckets for e.g. fproxy. > > "Should we save the free blocks list to disk? Free blocks are > generally blocks which have been used for temp files and so on, > rather than real keys. We don't want to give this information away > to an attacker." > > Again, how would we handle restarts. We need to know which blocks are > empty or we wouldn't be able to know where to write. To find out which > blocks are empty we have to walk through all the keys in the database and > 'mark'(in memory) which ones are empty. This can make the initialisation > off the database take very long, and it wouldn't scale very well. For a > would-be attacker, these few minutes are hardly a problem but for a > regular user these minutes would get pretty annoying. No, we don't. We simply use the LRU! We preallocate to the max, with random (but verifying) data. We randomize the LRU. Real data gets added to the most-recently-used end, and we delete stuff to use the space from the least-recently-used end. That's the point I was making. Yes, using it for temp files does change this slightly (if it's currently in use we can't allocate it), but that's orthogonal. > > > I like the idea of 'preallocation' and 'multiple store files'(as described > in follow up messages). But this does require an on-disk list of empty > blocks due to above reason. No, it doesn't. That flows directly from the above idea of obscuring the keys. All we have to do is keep a full LRU at all times, even if that means putting garbage keys into it. The attacker won't be able to tell that they are garbage keys. -- Matthew J Toseland - toad at amphibian.dyndns.org Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: <https://emu.freenetproject.org/pipermail/tech/attachments/20051216/a3d68b55/attachment.pgp>
