On Sunday 02 Sep 2012 17:51:49 xor wrote: > On Thursday 30 August 2012 00:40:13 Matthew Toseland wrote: > > Sadly Freetalk/WoT do use rollback so has to > > commit EVERY TIME. > > I oppose to the "sadly". Transaction-based programming has proven to be a > valid approach to solve many issues of traditional "manually undo everything > upon error"-programming.
Lets get one thing clear to begin with: I am not advocating that Freetalk/WoT aggregate transactions by simply not committing. The basic design flaw in Fred's use of the database layer was to assume that object databases work as advertised and you can simply take a big complex in-memory structure and persist it more or less transparently, and then add a load of (de)activation code and reduce memory usage. Freetalk and WoT may be better designed on that level. However they produce more disk I/O. And the reason for this is, mainstream database practice and theory are designed for two cases: 1. Lots of data (absolute amount and transactions/sec) on fast, reliable hardware with professional sysadmins. 2. Tiny amounts of data on commodity hardware. If you have lots of data on commodity disks, it falls down. And note that mainstream use of databases in the second case frequently fails. For example, I have a huge set of bookmarks in a tree on one of my browsers. Every time I create a new category, another one gets renamed. > The approach of fred to not use it is just wrong from a computer-science > perspective. > > The fact that it does not perform so well is an implementation issue of the > database, not of the client code which uses the database. No it is not. The load you put on it requires many seeks. Of course, it might require fewer seeks if it was using a well-designed SQL schema rather than trying to store objects. And I'm only talking about writes here; obviously if everything doesn't fit in memory you need to do seeks on read as well, and they can be very involved given db4o's lack of two-column indexes. However, because of the constant fsync's, we still needs loads of seeks *even if the whole thing fits in the OS disk cache*! > > Also, Freetalk/WoT themselves are CPU/IO hogs due to them using the wrong > algorithms, so we are not even at the point where we can tell whether their > ACID-usage is a problem. There is too much noise generated from the highly > inefficient algorithms which are being used by them, we cannot measure the > performance impact of commit/rollback. That's possible. However, the bottom line is if you have to commit every few seconds, you have to fsync every few seconds. IMHO avoiding that problem, for instance by turning off fsync and making periodic backups, would dramatically improve performance. As regards Freenet I am leaning towards a hand-coded on-disk structure. We don't use queries anyway, and mostly we don't need them; most of the data could be handled as a series of flat-files, and it would be far more robust than db4o, and likely faster too. But the first thing to do is upgrade the database and see whether it 1) breaks even worse or 2) fixes the problems. Past testing suggests #1, but past testing was based on later versions of 7.4, not on the latest 7.12. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/devl/attachments/20120904/07feb275/attachment.pgp>
