I know I repeat myself, but if you consider db4o you should also consider perst as an option. Toad, can you do your 'disk full test' with perst to compare it against db4o?
Perst and db4o seem to provide the same things. But according to http://www.garret.ru/~knizhnik/perstbench.html perst is much faster. No, I don't get money for advertising perst, but I made good experiences with it. On Sun, May 18, 2008 at 12:46 PM, Daniel Cheng <j16sdiz+freenet at gmail.com> wrote: > On Sun, May 18, 2008 at 12:27 PM, Florent Daigni?re > <nextgens at freenetproject.org> wrote: >> * Matthew Toseland <toad at amphibian.dyndns.org> [2008-05-17 19:00:13]: >> >>> On Saturday 17 May 2008 00:29, Matthew Toseland wrote: >>> > Ian and I have eventually come to the conclusion that we should include >>> db4o, >>> > and use it for our various persistence needs. I eventually reached the >>> > conclusion that while we can do most of what we need to do with simple >>> > flatfile databases, there are big chunks that will require a real database >>> of >>> > some kind (even if it's only a persistent hash table). db4o has various >>> > advantages: >>> > - Robust in real-world use. See for example this testimonial from a >>> > company >>> > who used it on cell phones: >>> > http://www.db4o.com/about/customers/success/mandalait.aspx >>> > BDBJE has not met our expectations in this regard. It seems very sensitive >>> to >>> > unusual situations - in particular, it will spontaneously corrupt and lose >>> > all data on running out of disk space. >>> > - True object database: no SQL, simple and powerful queries, etc. >>> > - Transparent or manual activation of objects from storage. >>> > - 800K jar, so not big enough to be a problem. >>> > - Mature and actively maintained. >>> > - Allows for future expansion (e.g. passive requests will need to store a >>> fair >>> > amount of persistent data). >>> > - Much more flexible than the hand-coded solution I was thinking of. We >>> > can >>> > persistent the entire queue (not just the splitfiles), if it's useful to >>> > do >>> > that. >>> > - Transactions (although this requires some juggling of in-memory objects >>> > on >>> > rollback). >>> > >>> > Tasks: >>> > - Add db4o to freenet-ext.jar. >>> > - Think about using it for the datastore. We don't want to have two >>> databases! >>> > Sdiz's new datastore may be the One True Store, or it may not be. If it's >>> > not, we don't want to keep BDBJE: we could build a db4o-based store, with >>> > or >>> > without LRU replacement. It would have the advantage of filling up more >>> > quickly than sdiz's store. It should require reconstructing less >>> > frequently >>> > than BDBJE! >>> > - Migrate the client layer, including splitfiles, pendingKeys, and so on, >>> > to >>> > be persisted via db4o. Of course there will be latency here when objects >>> > are >>> > not cached, so we will need to cache a few request choices in advance for >>> > each RequestStarter. And we will need to devise some way to deal with >>> > requests that don't want to be persisted - presumably we'd keep them in >>> > RAM. >>> > >>> It turns out that db4o does indeed unrecoverably self-corrupt when it runs >>> out >>> of disk space. (Thanks nextgens for getting me to test this!) >>> >>> http://amphibian.dyndns.org/bdb4o-test.log >>> >> >> muhahahaha. >> >> Last time I checked the bdb database was recoverable... Okay >> it lost some^wmost of the data in the process but at least it did >> attempt to recover! > > Both BDB (the original C version) and BDB-JE have very bad history on > recovering. > Why should we lost some data when there are better alternative that don't? > >>> We will therefore have to keep a fallback. IMHO for the client layer the >>> fallback should be downloads.dat.gz. We are careful not to lose that when we >>> run out of disk space, and it should only contain what is needed to restart >>> requests from the beginning (in practice a lot will come from the store). >>> >> ... >> >> While we are at it, what's wrong with bdb-je's persistence framework >> again ? >> http://www.oracle.com/database/berkeley-db/je/index.html >> > > Another reason to get rid of BDB-JE is memory usage and performance. > > db4o is usable on J2ME and scalable to terabytes database. I think > this say a lot. > >>> I apologise if the above was presented as a fait accompli, any input on >>> databases would be appreciated. On Friday, me and Ian spent a long time >>> debating the issue, first and foremost of whether we should even have a >>> database; I was initially in favour of not having one at all, or using >>> jdbm's >>> persistent hashtable class (HTree). >>> >>> Personally I think if we have a database it should be a native object >>> database >>> i.e. either Perst or db4o. It also should be robust, low overhead, mature, >>> open source etc. I will start implementing the new client layer with db4o >>> soon, unless convinced to use something else in the meantime. But it seems >>> that with BDBJE (which isn't a native object database), you can lose the >>> database even by an unclean shutdown... can anyone confirm this from >>> experience? Or is it only out of disk space and memory corruption that >>> causes >>> this? >> >> I'm still not convinced that we need a database... as our requirements >> are completely different from their typical use-cases... but well, your >> immediate concern is to store persistent requests to disk, right? What >> about using Hibernate or javax.persistence (from EE) to do that ? >> > eeeeeee.... > Hibernate is just ORM -- You need a sql backend for that. > (I am not oppose to the idea of using a sql backend, but then we have > to decide which one to use) > > javax.persistence have Java5 dependency, and you need a J2EE > container. just too ugly. > > -- > _______________________________________________ > Devl mailing list > Devl at freenetproject.org > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl > -- __________________________________________________ GnuPG key: (0x48DBFA8A) Keyserver: pgpkeys.pca.dfn.de Fingerprint: 477D F057 1BD4 1AE7 8A54 8679 6690 E2EC 48DB FA8A __________________________________________________
