On Sun, May 18, 2008 at 12:27 PM, Florent Daigni?re <nextgens at freenetproject.org> wrote: > * Matthew Toseland <toad at amphibian.dyndns.org> [2008-05-17 19:00:13]: > >> On Saturday 17 May 2008 00:29, Matthew Toseland wrote: >> > Ian and I have eventually come to the conclusion that we should include >> db4o, >> > and use it for our various persistence needs. I eventually reached the >> > conclusion that while we can do most of what we need to do with simple >> > flatfile databases, there are big chunks that will require a real database >> of >> > some kind (even if it's only a persistent hash table). db4o has various >> > advantages: >> > - Robust in real-world use. See for example this testimonial from a company >> > who used it on cell phones: >> > http://www.db4o.com/about/customers/success/mandalait.aspx >> > BDBJE has not met our expectations in this regard. It seems very sensitive >> to >> > unusual situations - in particular, it will spontaneously corrupt and lose >> > all data on running out of disk space. >> > - True object database: no SQL, simple and powerful queries, etc. >> > - Transparent or manual activation of objects from storage. >> > - 800K jar, so not big enough to be a problem. >> > - Mature and actively maintained. >> > - Allows for future expansion (e.g. passive requests will need to store a >> fair >> > amount of persistent data). >> > - Much more flexible than the hand-coded solution I was thinking of. We can >> > persistent the entire queue (not just the splitfiles), if it's useful to do >> > that. >> > - Transactions (although this requires some juggling of in-memory objects >> > on >> > rollback). >> > >> > Tasks: >> > - Add db4o to freenet-ext.jar. >> > - Think about using it for the datastore. We don't want to have two >> databases! >> > Sdiz's new datastore may be the One True Store, or it may not be. If it's >> > not, we don't want to keep BDBJE: we could build a db4o-based store, with >> > or >> > without LRU replacement. It would have the advantage of filling up more >> > quickly than sdiz's store. It should require reconstructing less frequently >> > than BDBJE! >> > - Migrate the client layer, including splitfiles, pendingKeys, and so on, >> > to >> > be persisted via db4o. Of course there will be latency here when objects >> > are >> > not cached, so we will need to cache a few request choices in advance for >> > each RequestStarter. And we will need to devise some way to deal with >> > requests that don't want to be persisted - presumably we'd keep them in >> > RAM. >> > >> It turns out that db4o does indeed unrecoverably self-corrupt when it runs >> out >> of disk space. (Thanks nextgens for getting me to test this!) >> >> http://amphibian.dyndns.org/bdb4o-test.log >> > > muhahahaha. > > Last time I checked the bdb database was recoverable... Okay > it lost some^wmost of the data in the process but at least it did > attempt to recover!
Both BDB (the original C version) and BDB-JE have very bad history on recovering. Why should we lost some data when there are better alternative that don't? >> We will therefore have to keep a fallback. IMHO for the client layer the >> fallback should be downloads.dat.gz. We are careful not to lose that when we >> run out of disk space, and it should only contain what is needed to restart >> requests from the beginning (in practice a lot will come from the store). >> > ... > > While we are at it, what's wrong with bdb-je's persistence framework > again ? > http://www.oracle.com/database/berkeley-db/je/index.html > Another reason to get rid of BDB-JE is memory usage and performance. db4o is usable on J2ME and scalable to terabytes database. I think this say a lot. >> I apologise if the above was presented as a fait accompli, any input on >> databases would be appreciated. On Friday, me and Ian spent a long time >> debating the issue, first and foremost of whether we should even have a >> database; I was initially in favour of not having one at all, or using jdbm's >> persistent hashtable class (HTree). >> >> Personally I think if we have a database it should be a native object >> database >> i.e. either Perst or db4o. It also should be robust, low overhead, mature, >> open source etc. I will start implementing the new client layer with db4o >> soon, unless convinced to use something else in the meantime. But it seems >> that with BDBJE (which isn't a native object database), you can lose the >> database even by an unclean shutdown... can anyone confirm this from >> experience? Or is it only out of disk space and memory corruption that causes >> this? > > I'm still not convinced that we need a database... as our requirements > are completely different from their typical use-cases... but well, your > immediate concern is to store persistent requests to disk, right? What > about using Hibernate or javax.persistence (from EE) to do that ? > eeeeeee.... Hibernate is just ORM -- You need a sql backend for that. (I am not oppose to the idea of using a sql backend, but then we have to decide which one to use) javax.persistence have Java5 dependency, and you need a J2EE container. just too ugly. --
