On Mon, 2009-12-14, C. Michael Pilato wrote: > Julian. > > I want to draw your attention to something that you may or may not realize > about designing for the BDB backend. The 'trail' objects are (generally) > representative of Berkeley DB transactions -- that part I'm sure you know. > But you might not realize the value of keeping transactions as small as > possible. Berkeley DB will accumulate locks (which I believe are > page-level, not as tight as row-level like you might hope) over the course > of a transaction, releasing those locks only at transaction commit/abort. > Berkeley DB backends are configured to have a maximum number of locks and > lockers allowed, and it's easier than you might think to hit the max-locks > thresholds (especially under high concurrency) and see an error (typically a > "Cannot allocate memory") result from that. > > For example, in the loop below you are writing a bunch of rows to the > `changes' table. Could be 10. Could be 100,000. 100,000 writes and > associated locks might be a problem or it might not. But I use it as a way > to encourage you to think about reducing the amount of work you spend in any > one trail while doing your obliterate work.
Thanks for this, and the subsequent IRC chat; sorry my net connection dropped near the end of that. I suspected something like this, and wrote in a comment in svn_fs_base__begin_obliteration_txn(), "### Does all of the duplication need to be done inside the retry_txn? It is currently inside." I will bear this in mind and may make some changes in this direction. However, my priorities begin with learning how to do obliteration functionally correctly (which I am doing in BDB), and then implementing it in FSFS, and somewhere down the list comes scalability in BDB. I appreciate that obliteration is most needed in large repositories where scalability is essential. > (And yeah, this information should probably live somewhere. Maybe you could > drop it into trail.h as a comment?) OK, dropped into subversion/libsvn_fs_base/trail.h in r890841. - Julian
