The DB4O branch has finally been merged! Actually it was merged on Thursday, but I've been going through the diffs to make sure I haven't done anything catastrophic. The db4o branch was a project to make Freenet use a database (db4o) for tracking the status of persistent requests and inserts, thus making requests restart instantly on restarting the node, and radically reducing memory usage for large queues.
Both of these aims have been achieved; the main catch is slightly greater disk access. Downloads take around 400K static memory usage per queued gigabyte, uploads don't have any static memory usage but in any case you need 80-100M or so of "scratch space". Uploads do however require significant memory when they are being started, so in the default 192MB memory limit, you should be able to start a 4GB insert (and then another one, and then another one), but not anything much larger than that, and/or queue a huge quantity of downloads. More memory does help performance, of course, but in 192M you should be able to do almost anything. The diff merging db4o was over 35,000 lines of code (not all of this additions), bringing the total (not including contrib, but including plugins) to over 298,000. The project was started on May 20th last year. Other notable changes include persistent blob temp files: persistent 32K temporary buckets are allocated from a single large file, persistent-temp-<port number>/persistent-blob.tmp. This will automatically grow and shrink as needed, although it does not at present defragment itself. One advantage is that we use much fewer file descriptors, which was getting to be a problem on some nodes; another is that enumerating the files in the persistent temp directory was causing out of memory on startup on nodes with small amounts of memory and 4GB inserts queued. But we may make it optional if people don't like it / if it doesn't work well. HOW YOU CAN HELP ============ The db4o branch was extensively tested before merging, but it is likely that I have missed critical bugs; if you can, please download the new build and test it out, report any bugs you find via https://bugs.freenetproject.org/ or IRC (#freenet or #freenet-testers on irc.freenode.net). Check the known bugs below first. There is considerable work to do before we can release a new stable build (1207) including db4o. Known issues ======== MIGRATING OLD DOWNLOADS/UPLOADS Old downloads and uploads from downloads.dat.gz should generally be transparently imported to the new system, and inserts should keep their existing progress. However, non-global persistent requests may be broken. I think Frost uses, or at least can use, non-global persistent requests, so this will need to be fixed before releasing a new build. PERSISTENCE=REBOOT Persistence=reboot (FCP requests which persist only until the node restarts) may be broken. IMHO this isn't used much, but it will need to be fixed at some point. GARBAGE COLLECTION: STRUCTURES Because db4o does not provide any form of garbage collection, we have to remove structures manually when e.g. requests finish. I have spent considerable time debugging such leaks, however if there ARE any such leaks, it will be very difficult to either recover the space (which should be minimal in the vast majority of cases), or be sure that there isn't any incriminating data from past requests still in the database. Details for the paranoid or the curious: You can however examine the contents of your database to see whether there are any leaks by setting your log level details to freenet.node.node:DEBUG,freenet.client.async.ClientRequestScheduler:DEBUG. You will then get a dump of the count of each type of object in the wrapper.log on startup, and logs/ will have the details of each individual object (search for "DATABASE:"). If you can reproduce such a leak let me know. Note that a few objects are just overhead and are to be expected, even if nothing is queued, also a few objects (FCPClient, a few arrays/lists/maps iirc; if you investigate you'll see a pattern) are the result of FCP clients, and again are not of any great concern. However, if you run a request, and then delete it, and after restarting there are leaking FreenetURIs or ClientCHKs or any other kind of key, there is a serious problem, which I need to know about. (Note that this may be caused by an ERROR e.g. a NullPointerException, check the logs). GARBAGE COLLECTION: TEMP FILES Right now there is no automatic collection of old temp files. Temporary files are removed when a request is removed and/or when they are no longer needed, but if there is a bug (and there always are), it is possible we could have an on-disk space leak. I have tried to eliminate this, it doesn't happen in my testing, but it is possible especially if something breaks. An unclean shutdown at the wrong point *should* not cause such temp space leaks, as we record the list of buckets to delete before committing a transaction. At some point, not necessarily before releasing a new build, it is planned to implement an automatic garbage collector for temp files (sadly doing this for structures would be a rather larger project, but for temp buckets it's not difficult). -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20090404/257ff8b7/attachment.pgp>
