The current work on the download manager is 80% written; I just have to write the scheduler code. After that, I'd have to do persistence, although as I explain below I may not implement this immediately.
The objectives are: - To be able to have large numbers of downloads queued/running, with priorities, in the node, so that we can efficiently implement FUQID like functionality, and later prefetch and USK updating, and so that we don't get major problems with heavy fproxy usage etc. - To start a fetch at every opportunity (these occur with a frequency determined by load balancing). - To have an architecture which is reasonably amenable to persistence so that we can save unfinished downloads to disk for long-term FUQID-style retrying downloads. One thing this means is storing the state on the objects rather than on the threads. - That the fetch we start at each opportunity is randomly chosen from the possibilities at the current retry and priority level, so that we can have consistency, and therefore best performance. I have coded inserts and requests for the new async architecture; it should not take long to code the client request scheduler, and wire it all in so that I can test it. The mid-term problem is this: We have 32kB blocks, and we have to do FUQID-style functionality in the node because FCPv2 is high level; the result is that if we have 20GB of splitfiles queued (which is entirely possible, and may even be common), and we've fetched all the manifests (this is a little less likely), we have 655,360 blocks queued. Each of these needs an object including its key (64+ bytes), its retry level, its callback, etc. If we can pack all this into 512 bytes, it's still 300MB of RAM. Possible approaches: - 1. It's the user's problem! If he wants to queue 20GB of requests he can buy a DIMM! - 2. Make sure we don't fetch the splitfile metadata for a file while we have over a certain number of blocks in RAM. The problem with this is that some files may just not be retrievable. - 3. Compress the data in RAM. I estimate we can achieve a *significant* reduction in memory usage by clever coding - a factor of 4, say. - 4. Use hibernate or a similar object persistence framework. Keep the objects in a database rather than in RAM (most of the time). Cost: 3.5MB on the freenet-ext.jar. - 5. Use BDB directly to save the queued requests data to a database. Cost: we already have BDB, #4 might or might not be easier. I do think that it's worth the cost; we will have a world class downloader back-end, superior IMHO to Azuerus for example. As long as we, and others, exploit this in the UI, this should be an important selling point. And from the above options, it's clear that we don't have to solve the queues-use-up-lots-of-RAM problem immediately. -- Matthew J Toseland - toad at amphibian.dyndns.org Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: <https://emu.freenetproject.org/pipermail/tech/attachments/20060124/1b82dc87/attachment.pgp>
