On Monday 19 May 2008 11:34, Matthew Toseland wrote:
> On Sunday 18 May 2008 19:44, Ian Clarke wrote:
> > 
> > Looking at the manual, it looks like Perst operates at a lower level
> > than db4o - you need to manually create and maintain indexes.  This is
> > closer to the Java collections API, which could be good because its
> > more familiar, but it leaves more opportunities for the programmer to
> > screw up, which is bad.
> 
> IMHO we will get better performance and fewer screwups from a more familiar 
> API.
> > 
> > My preference for db4o is based mainly on my familiarity with it, the
> > fact that it is more widely used, and the fact that it superficially
> > appears to have a more credible company behind it.
> > 
> > None of these is a solid justification for choosing one option over
> > the other, but they all suggest that db4o is the way to go.
> 
> We should, as bback suggests, test Perst, see what happens when it runs out 
of 
> disk space. I will look into it soon.

Completed testing. Perst fails gracefully on running out of disk space in 
between writes. The database is readable after the failed commit, and if I 
take away the bigfile occupying all the disk space, we can even write to it.

How about we go with Perst?
> > 
> > Ian.
> > 
> > On Sun, May 18, 2008 at 11:08 AM,  <bbackde at googlemail.com> wrote:
> > > I know I repeat myself, but if you consider db4o you should also
> > > consider perst as an option.
> > > Toad, can you do your 'disk full test' with perst to compare it against 
> db4o?
> > >
> > > Perst and db4o seem to provide the same things. But according to
> > > http://www.garret.ru/~knizhnik/perstbench.html
> > > perst is much faster.
> > >
> > > No, I don't get money for advertising perst, but I made good
> > > experiences with it.
> > >
> > > On Sun, May 18, 2008 at 12:46 PM, Daniel Cheng
> > > <j16sdiz+freenet at gmail.com> wrote:
> > >> On Sun, May 18, 2008 at 12:27 PM, Florent Daigni?re
> > >> <nextgens at freenetproject.org> wrote:
> > >>> * Matthew Toseland <toad at amphibian.dyndns.org> [2008-05-17 19:00:13]:
> > >>>
> > >>>> On Saturday 17 May 2008 00:29, Matthew Toseland wrote:
> > >>>> > Ian and I have eventually come to the conclusion that we should 
> include
> > >>>> db4o,
> > >>>> > and use it for our various persistence needs. I eventually reached 
> the
> > >>>> > conclusion that while we can do most of what we need to do with 
> simple
> > >>>> > flatfile databases, there are big chunks that will require a real 
> database
> > >>>> of
> > >>>> > some kind (even if it's only a persistent hash table). db4o has 
> various
> > >>>> > advantages:
> > >>>> > - Robust in real-world use. See for example this testimonial from a 
> company
> > >>>> > who used it on cell phones:
> > >>>> > http://www.db4o.com/about/customers/success/mandalait.aspx
> > >>>> > BDBJE has not met our expectations in this regard. It seems very 
> sensitive
> > >>>> to
> > >>>> > unusual situations - in particular, it will spontaneously corrupt 
and 
> lose
> > >>>> > all data on running out of disk space.
> > >>>> > - True object database: no SQL, simple and powerful queries, etc.
> > >>>> > - Transparent or manual activation of objects from storage.
> > >>>> > - 800K jar, so not big enough to be a problem.
> > >>>> > - Mature and actively maintained.
> > >>>> > - Allows for future expansion (e.g. passive requests will need to 
> store a
> > >>>> fair
> > >>>> > amount of persistent data).
> > >>>> > - Much more flexible than the hand-coded solution I was thinking 
of. 
> We can
> > >>>> > persistent the entire queue (not just the splitfiles), if it's 
useful 
> to do
> > >>>> > that.
> > >>>> > - Transactions (although this requires some juggling of in-memory 
> objects on
> > >>>> > rollback).
> > >>>> >
> > >>>> > Tasks:
> > >>>> > - Add db4o to freenet-ext.jar.
> > >>>> > - Think about using it for the datastore. We don't want to have two
> > >>>> databases!
> > >>>> > Sdiz's new datastore may be the One True Store, or it may not be. 
If 
> it's
> > >>>> > not, we don't want to keep BDBJE: we could build a db4o-based 
store, 
> with or
> > >>>> > without LRU replacement. It would have the advantage of filling up 
> more
> > >>>> > quickly than sdiz's store. It should require reconstructing less 
> frequently
> > >>>> > than BDBJE!
> > >>>> > - Migrate the client layer, including splitfiles, pendingKeys, and 
so 
> on, to
> > >>>> > be persisted via db4o. Of course there will be latency here when 
> objects are
> > >>>> > not cached, so we will need to cache a few request choices in 
advance 
> for
> > >>>> > each RequestStarter. And we will need to devise some way to deal 
with
> > >>>> > requests that don't want to be persisted - presumably we'd keep 
them 
> in RAM.
> > >>>> >
> > >>>> It turns out that db4o does indeed unrecoverably self-corrupt when it 
> runs out
> > >>>> of disk space. (Thanks nextgens for getting me to test this!)
> > >>>>
> > >>>> http://amphibian.dyndns.org/bdb4o-test.log
> > >>>>
> > >>>
> > >>> muhahahaha.
> > >>>
> > >>> Last time I checked the bdb database was recoverable... Okay
> > >>> it lost some^wmost of the data in the process but at least it did
> > >>> attempt to recover!
> > >>
> > >> Both BDB (the original C version) and BDB-JE have very bad history on
> > >> recovering.
> > >> Why should we lost some data when there are better alternative that 
> don't?
> > >>
> > >>>> We will therefore have to keep a fallback. IMHO for the client layer 
> the
> > >>>> fallback should be downloads.dat.gz. We are careful not to lose that 
> when we
> > >>>> run out of disk space, and it should only contain what is needed to 
> restart
> > >>>> requests from the beginning (in practice a lot will come from the 
> store).
> > >>>>
> > >>> ...
> > >>>
> > >>> While we are at it, what's wrong with bdb-je's persistence framework
> > >>> again ?
> > >>> http://www.oracle.com/database/berkeley-db/je/index.html
> > >>>
> > >>
> > >> Another reason to get rid of BDB-JE is memory usage and performance.
> > >>
> > >> db4o is usable on J2ME and scalable to terabytes database. I think
> > >> this say a lot.
> > >>
> > >>>> I apologise if the above was presented as a fait accompli, any input 
on
> > >>>> databases would be appreciated. On Friday, me and Ian spent a long 
time
> > >>>> debating the issue, first and foremost of whether we should even have 
a
> > >>>> database; I was initially in favour of not having one at all, or 
using 
> jdbm's
> > >>>> persistent hashtable class (HTree).
> > >>>>
> > >>>> Personally I think if we have a database it should be a native object 
> database
> > >>>> i.e. either Perst or db4o. It also should be robust, low overhead, 
> mature,
> > >>>> open source etc. I will start implementing the new client layer with 
> db4o
> > >>>> soon, unless convinced to use something else in the meantime. But it 
> seems
> > >>>> that with BDBJE (which isn't a native object database), you can lose 
> the
> > >>>> database even by an unclean shutdown... can anyone confirm this from
> > >>>> experience? Or is it only out of disk space and memory corruption 
that 
> causes
> > >>>> this?
> > >>>
> > >>> I'm still not convinced that we need a database... as our requirements
> > >>> are completely different from their typical use-cases... but well, 
your
> > >>> immediate concern is to store persistent requests to disk, right? What
> > >>> about using Hibernate or javax.persistence (from EE) to do that ?
> > >>>
> > >> eeeeeee....
> > >> Hibernate is just ORM -- You need a sql backend for that.
> > >> (I am not oppose to the idea of using a sql backend, but then we have
> > >> to decide which one to use)
> > >>
> > >> javax.persistence have Java5 dependency, and you need a J2EE
> > >> container. just too ugly.
> > >>
> > >> --
> > >> _______________________________________________
> > >> Devl mailing list
> > >> Devl at freenetproject.org
> > >> http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> > >>
> > >
> > >
> > >
> > > --
> > > __________________________________________________
> > > GnuPG key: (0x48DBFA8A)
> > > Keyserver: pgpkeys.pca.dfn.de
> > > Fingerprint:
> > > 477D F057 1BD4 1AE7 8A54 8679 6690 E2EC 48DB FA8A
> > > __________________________________________________
> > > _______________________________________________
> > > Devl mailing list
> > > Devl at freenetproject.org
> > > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> > >
> > 
> > 
> > 
> > -- 
> > Email: ian at uprizer.com
> > Cell: +1 512 422 3588
> > Skype: sanity
> > _______________________________________________
> > Devl mailing list
> > Devl at freenetproject.org
> > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
> > 
> > 
> 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 
<https://emu.freenetproject.org/pipermail/devl/attachments/20080519/5350c4c3/attachment.pgp>

Reply via email to