On Monday 11 January 2010 16:49:30 Ximin Luo wrote: > On 11/01/10 16:30, Matthew Toseland wrote: > > On Sunday 10 January 2010 11:35:47 Ximin Luo wrote: > >> http://github.com/freenet/plugin-Library-staging > >> > >> If anyone has some free time, please check out the above repo and run the > >> test: > >> > >> $ git clone git://github.com/freenet/plugin-Library-staging.git $ cd > >> plugin-Library-staging $ ant junit > >> > >> Open up ./test/plugins/Library/index/BIndexTest.java and fiddle about with > >> the numbers at the top, and run the test with different settings. > >> BIndexTest is the first test to run; you can just hit ctrl-C after it's > >> done to skip the rest. > >> > >> What you're testing is an algorithm to write to a on-freenet b-tree. Ie. > >> most of the data is on freenet, and you're just pulling/pushing in > >> (approximately) the minimum nodes needed to update it, plus a new root > >> node. > >> > >> This would be useful for search indexes (which are huge), and could also > >> be useful for other things that need to store a huge on-freenet data > >> structure. Obviously, it needs a shit load of testing first, so any help > >> would be appreciated. > > > > IMHO any such on-freenet data structure should be single owner, to avoid > > concurrent updates / finding the latest version problems, and the data > > should all be duplicated locally so it can be easily reinserted. > > The only thing that is "owned" is the USK root pointer to the rest of the data > structure. The data nodes are currently stored as CHK files, so they are > effectively immutable, which means concurrency isn't a problem. Multiple root > USKs from different owners could point to the same CHK nodes, making fork > operations trivial.
Right. But each root has an owner. And a forked index will have both a different root and some set of different sub-paths. The point is we can't have a single linear USK space with private key known by many people, because this will be chaotic. The correct model is to fork and merge a la DSCM's or wiki-over-DSCM's. > > > However, this is very important stuff, because It will allow us to generate > > indexes gradually during spidering and much more efficiently (no waiting > > several days while doing hideous numbers of seeks while it writes the > > index). There may be other uses too in the medium term, as infinity0 hints > > at, e.g. published databases etc. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20100119/b201f983/attachment.pgp>
