On 11/01/10 16:30, Matthew Toseland wrote: > On Sunday 10 January 2010 11:35:47 Ximin Luo wrote: >> http://github.com/freenet/plugin-Library-staging >> >> If anyone has some free time, please check out the above repo and run the >> test: >> >> $ git clone git://github.com/freenet/plugin-Library-staging.git $ cd >> plugin-Library-staging $ ant junit >> >> Open up ./test/plugins/Library/index/BIndexTest.java and fiddle about with >> the numbers at the top, and run the test with different settings. >> BIndexTest is the first test to run; you can just hit ctrl-C after it's >> done to skip the rest. >> >> What you're testing is an algorithm to write to a on-freenet b-tree. Ie. >> most of the data is on freenet, and you're just pulling/pushing in >> (approximately) the minimum nodes needed to update it, plus a new root >> node. >> >> This would be useful for search indexes (which are huge), and could also >> be useful for other things that need to store a huge on-freenet data >> structure. Obviously, it needs a shit load of testing first, so any help >> would be appreciated. > > IMHO any such on-freenet data structure should be single owner, to avoid > concurrent updates / finding the latest version problems, and the data > should all be duplicated locally so it can be easily reinserted.
The only thing that is "owned" is the USK root pointer to the rest of the data structure. The data nodes are currently stored as CHK files, so they are effectively immutable, which means concurrency isn't a problem. Multiple root USKs from different owners could point to the same CHK nodes, making fork operations trivial. > However, this is very important stuff, because It will allow us to generate > indexes gradually during spidering and much more efficiently (no waiting > several days while doing hideous numbers of seeks while it writes the > index). There may be other uses too in the medium term, as infinity0 hints > at, e.g. published databases etc. > > > ------------------------------------------------------------------------ > > _______________________________________________ Devl mailing list > Devl at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
