On Monday 19 May 2008 11:34, Matthew Toseland wrote: > On Sunday 18 May 2008 19:44, Ian Clarke wrote: > > > > Looking at the manual, it looks like Perst operates at a lower level > > than db4o - you need to manually create and maintain indexes. This is > > closer to the Java collections API, which could be good because its > > more familiar, but it leaves more opportunities for the programmer to > > screw up, which is bad. > > IMHO we will get better performance and fewer screwups from a more familiar > API. > > > > My preference for db4o is based mainly on my familiarity with it, the > > fact that it is more widely used, and the fact that it superficially > > appears to have a more credible company behind it. > > > > None of these is a solid justification for choosing one option over > > the other, but they all suggest that db4o is the way to go. > > We should, as bback suggests, test Perst, see what happens when it runs out of > disk space. I will look into it soon.
Completed testing. Perst fails gracefully on running out of disk space in between writes. The database is readable after the failed commit, and if I take away the bigfile occupying all the disk space, we can even write to it. How about we go with Perst? > > > > Ian. > > > > On Sun, May 18, 2008 at 11:08 AM, <bbackde at googlemail.com> wrote: > > > I know I repeat myself, but if you consider db4o you should also > > > consider perst as an option. > > > Toad, can you do your 'disk full test' with perst to compare it against > db4o? > > > > > > Perst and db4o seem to provide the same things. But according to > > > http://www.garret.ru/~knizhnik/perstbench.html > > > perst is much faster. > > > > > > No, I don't get money for advertising perst, but I made good > > > experiences with it. > > > > > > On Sun, May 18, 2008 at 12:46 PM, Daniel Cheng > > > <j16sdiz+freenet at gmail.com> wrote: > > >> On Sun, May 18, 2008 at 12:27 PM, Florent Daigni?re > > >> <nextgens at freenetproject.org> wrote: > > >>> * Matthew Toseland <toad at amphibian.dyndns.org> [2008-05-17 19:00:13]: > > >>> > > >>>> On Saturday 17 May 2008 00:29, Matthew Toseland wrote: > > >>>> > Ian and I have eventually come to the conclusion that we should > include > > >>>> db4o, > > >>>> > and use it for our various persistence needs. I eventually reached > the > > >>>> > conclusion that while we can do most of what we need to do with > simple > > >>>> > flatfile databases, there are big chunks that will require a real > database > > >>>> of > > >>>> > some kind (even if it's only a persistent hash table). db4o has > various > > >>>> > advantages: > > >>>> > - Robust in real-world use. See for example this testimonial from a > company > > >>>> > who used it on cell phones: > > >>>> > http://www.db4o.com/about/customers/success/mandalait.aspx > > >>>> > BDBJE has not met our expectations in this regard. It seems very > sensitive > > >>>> to > > >>>> > unusual situations - in particular, it will spontaneously corrupt and > lose > > >>>> > all data on running out of disk space. > > >>>> > - True object database: no SQL, simple and powerful queries, etc. > > >>>> > - Transparent or manual activation of objects from storage. > > >>>> > - 800K jar, so not big enough to be a problem. > > >>>> > - Mature and actively maintained. > > >>>> > - Allows for future expansion (e.g. passive requests will need to > store a > > >>>> fair > > >>>> > amount of persistent data). > > >>>> > - Much more flexible than the hand-coded solution I was thinking of. > We can > > >>>> > persistent the entire queue (not just the splitfiles), if it's useful > to do > > >>>> > that. > > >>>> > - Transactions (although this requires some juggling of in-memory > objects on > > >>>> > rollback). > > >>>> > > > >>>> > Tasks: > > >>>> > - Add db4o to freenet-ext.jar. > > >>>> > - Think about using it for the datastore. We don't want to have two > > >>>> databases! > > >>>> > Sdiz's new datastore may be the One True Store, or it may not be. If > it's > > >>>> > not, we don't want to keep BDBJE: we could build a db4o-based store, > with or > > >>>> > without LRU replacement. It would have the advantage of filling up > more > > >>>> > quickly than sdiz's store. It should require reconstructing less > frequently > > >>>> > than BDBJE! > > >>>> > - Migrate the client layer, including splitfiles, pendingKeys, and so > on, to > > >>>> > be persisted via db4o. Of course there will be latency here when > objects are > > >>>> > not cached, so we will need to cache a few request choices in advance > for > > >>>> > each RequestStarter. And we will need to devise some way to deal with > > >>>> > requests that don't want to be persisted - presumably we'd keep them > in RAM. > > >>>> > > > >>>> It turns out that db4o does indeed unrecoverably self-corrupt when it > runs out > > >>>> of disk space. (Thanks nextgens for getting me to test this!) > > >>>> > > >>>> http://amphibian.dyndns.org/bdb4o-test.log > > >>>> > > >>> > > >>> muhahahaha. > > >>> > > >>> Last time I checked the bdb database was recoverable... Okay > > >>> it lost some^wmost of the data in the process but at least it did > > >>> attempt to recover! > > >> > > >> Both BDB (the original C version) and BDB-JE have very bad history on > > >> recovering. > > >> Why should we lost some data when there are better alternative that > don't? > > >> > > >>>> We will therefore have to keep a fallback. IMHO for the client layer > the > > >>>> fallback should be downloads.dat.gz. We are careful not to lose that > when we > > >>>> run out of disk space, and it should only contain what is needed to > restart > > >>>> requests from the beginning (in practice a lot will come from the > store). > > >>>> > > >>> ... > > >>> > > >>> While we are at it, what's wrong with bdb-je's persistence framework > > >>> again ? > > >>> http://www.oracle.com/database/berkeley-db/je/index.html > > >>> > > >> > > >> Another reason to get rid of BDB-JE is memory usage and performance. > > >> > > >> db4o is usable on J2ME and scalable to terabytes database. I think > > >> this say a lot. > > >> > > >>>> I apologise if the above was presented as a fait accompli, any input on > > >>>> databases would be appreciated. On Friday, me and Ian spent a long time > > >>>> debating the issue, first and foremost of whether we should even have a > > >>>> database; I was initially in favour of not having one at all, or using > jdbm's > > >>>> persistent hashtable class (HTree). > > >>>> > > >>>> Personally I think if we have a database it should be a native object > database > > >>>> i.e. either Perst or db4o. It also should be robust, low overhead, > mature, > > >>>> open source etc. I will start implementing the new client layer with > db4o > > >>>> soon, unless convinced to use something else in the meantime. But it > seems > > >>>> that with BDBJE (which isn't a native object database), you can lose > the > > >>>> database even by an unclean shutdown... can anyone confirm this from > > >>>> experience? Or is it only out of disk space and memory corruption that > causes > > >>>> this? > > >>> > > >>> I'm still not convinced that we need a database... as our requirements > > >>> are completely different from their typical use-cases... but well, your > > >>> immediate concern is to store persistent requests to disk, right? What > > >>> about using Hibernate or javax.persistence (from EE) to do that ? > > >>> > > >> eeeeeee.... > > >> Hibernate is just ORM -- You need a sql backend for that. > > >> (I am not oppose to the idea of using a sql backend, but then we have > > >> to decide which one to use) > > >> > > >> javax.persistence have Java5 dependency, and you need a J2EE > > >> container. just too ugly. > > >> > > >> -- > > >> _______________________________________________ > > >> Devl mailing list > > >> Devl at freenetproject.org > > >> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl > > >> > > > > > > > > > > > > -- > > > __________________________________________________ > > > GnuPG key: (0x48DBFA8A) > > > Keyserver: pgpkeys.pca.dfn.de > > > Fingerprint: > > > 477D F057 1BD4 1AE7 8A54 8679 6690 E2EC 48DB FA8A > > > __________________________________________________ > > > _______________________________________________ > > > Devl mailing list > > > Devl at freenetproject.org > > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl > > > > > > > > > > > -- > > Email: ian at uprizer.com > > Cell: +1 512 422 3588 > > Skype: sanity > > _______________________________________________ > > Devl mailing list > > Devl at freenetproject.org > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl > > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20080519/5350c4c3/attachment.pgp>