On Thu, Mar 06, 2003 at 12:20:50AM -0500, Scott Young wrote: > On Wed, 2003-03-05 at 23:43, Frank v Waveren wrote: > > On Wed, Mar 05, 2003 at 08:39:37PM -0800, Ian Clarke wrote: > > > Yes they should, good observation. We do, however, need to be careful > > > about memory usage (and, to a lesser extent, disk usage) when sewing the > > > segments back together (it would be tempting to load them all into RAM > > > at once and stick-em together, but that would clearly be a bad thing). > Actually, my processor (Athlon XP 1800) seems to be the major bottleneck > when decoding FEC blocks, so loading into RAM wouldn't be that much of a > performance boost. Plus, I don't think many JVMs would like to store > and decode 128 1 MB segments in memory. > > > Why? There's no magic involved in the joining of segments, so just > > store them all to tempfiles and then read through them sequentially? > > If the segments are all stored in temp files, then sewn together, then > twice as much free space as the file itself (unless the temp files are > deleted as they are "sewn" into the file, which seems to not be > happening). This is actually three times, if you include the space the No. For uploads, we end up using twice, because of a bug/misfeature in the MIME handling code. For downloads, we stream the buckets straight out, the worst case for a segment is we got all 64 check blocks, and 64 of the data blocks, we have to reconstruct the other 64 data blocks, so we are using 150% of the original segment size, temporarily. However, the blocks are (possibly temporarily) in the store as well, making it 250%, plus a bit for the healing blocks. This will be alleviated by probabilistic caching when it eventually is implemented. > blocks take up in the DataStore, or four, if your web browser makes a > copy on a different partition first. This doesn't even include the Yeah, that is annoying. Not much we can do about it. And I understand their reasons (which are basically that the file might change on every restart). > final copy of the decoded file. I don't want to have to have 4 GB of > free space just to decode 1 GB of data. I would much rather give the > extra 3 GB to my DataStore. Well, if you want the datastore to use a large amount of space, and don't need oodles of temp space, set tempInStore=true to enable the temp files to be accounted as part of the store, so that the space used by them is used by the store when not doing big splitfile fetches. > > I was thinking it would be better to do what other programs do for > partial downloads: .dat files. An in-place reconstruction of the file > could take place after enough data for each segment is downloaded. If > the user can tell Freenet where they want to store the file after it > isfile downloaded, the dat file could be constructed in the same place > so that the file doesn't need to be copied across partitions in the > end. The DAT file would also be able to store data that is deleted from > the node, so that the data store could actually be smaller than the size > of the SplitFile being downloaded. Resuming downloads would also be > quicker because FEC wouldn't have to be run on already-decoded segments > again. It would make very little difference since we stream the buckets anyway. It would also mean that broken non-redundant splitfiles that don't specify a block size would not be fetchable. > > The ultimate way of doing this would be to allow a user to transparently > mount the data store and access a SplitFile directly. This way, a user > wouldn't have to give up extra space on their drive (which could be used > by the data store itself, anyway), AND the node would have a cached copy > of the blocks to use. This has obvious problems though. Umm, this is not possible, because a splitfile is not actually in your datastore as a single file. I can see what you are getting at though, but it's not necessary or useful especially once we implement probabilistic caching.
-- Matthew Toseland [EMAIL PROTECTED]/[EMAIL PROTECTED] Full time freenet hacker. http://freenetproject.org/ Freenet Distribution Node (temporary) at http://80-192-4-23.cable.ubr09.na.blueyonder.co.uk:8889/BjbP6Lfu-lo/ ICTHUS.
pgp00000.pgp
Description: PGP signature
