I invariably forget to hit Reply-All ... apologies. Begin forwarded message:
> On Feb 24, 2010, at 6:01 PM, Kim Minh Kaplan wrote: > >> Jeff Johnson: >> >>> BTW, can someone describe -- even superficially -- what is >>> being attempted with the PTree store? Any details are welcomed, >>> I'm not yet able to read OCAML code well enough to recognize >>> what type of store is being attempted. >> >> The use of the prefix tree is described in the two articles Set >> Reconciliation with Nearly Optimal Communication Complexity and >> Practical Set Reconciliation. You will find them on the Google code >> page¹. The mathematics are quite involved though. It is the >> information that is needed for two servers to know what keys they need >> to exchange. >> > > Thank you! I've read through those papers about 3 times now ... another > month and I'll think I understand. > > Note that I'm hoping to use the same mechanism for keeping > RPM package repositories in sync, not only server <-> server, but > also server <-> client, using the same efficient gossip protocol. > > I have another usage case distributing i18n msgstr's through servers, > not through traditional PO files as well. > > These are the reasons I'm looking at sks-keyserver. > > >>> The PTree deadlock is easily reproduced, and (with db_stat) a >>> detailed deadlock diagnosis could be attempted. >> >> How would that be? >> > > Deadlock diagnosis is described in chapter 11 here: > > > http://www.oracle.com/technology/documentation/berkeley-db/db/programmer_reference/index.html > > The db_stat I captured (I have the full file if interested) showed > a page self-deadlock: > > ... > 800036f7 WRITE 3 HELD ptree page 1213 > 2 READ 1 WAIT ptree page 1213 > ... > > which (tenatively) I attribute to either > 1) a write cursor isn't being closed correctly in pTreeDB.ml > or (my hunch) > 2) there's a duplicate key that is stored on the same page > > Caveat: these are just very preliminary guesses. I'll dig out a fix > over the next month while attempting to add DB_INIT_LOCK to pTreeDB.ml. > >>> Is there interest in adding DB_INIT_LOCK to the PTree store? >> >> While I do not see any practical interest in doing this, the symptom >> smells like a bug hanging in there that could very well be related to >> the problem Arnold experienced recently. There would definitely be >> interest in correcting it. >> > > You are correct that sks-keyserver (as is) will benefit little from > adding DB_INIT_LOCK. > > However, there's a class of failures from running other sks operations > underneath an active server that likely will benefit from DB_INIT_LOCK imho. > > (different topic) > While you are here ... > > As part of recovering from the deadlock, some of the indices were > damaged (checked using db_verify). The fixup for that is > db_dump ... | db_load ... > However, db_dump has no guarantee of preserving all data, its just "best > effort". > In my case, the data loss showed up with a DB_PAGE_NOTFOUND, which (because it > was easiest) led to a full reload of the database from a dump. > > None of that was hard or bothersome. > > However, DB->associate() to attach a secondary table index to a > primary key store could have regenerated the secondary index > if a DB_CREATE flag was passed. That is likely easier maintenance > than loading from a dump. > > Caveat: the secondary -> primary association must be unique for DB->associate > to be used. > > Is there interest in making the maintenance of the secondary indices more > automatic? > I noted maybe 3-5 problems in mail archives whose root cause might well have > been > data loss in the secondary indices. Removing the secondary index, and adding > DB_CREATE > to regenerate when opening, isn't hard to add. > > Caveat: With DB->associate, all secondary table adds/deletes are done with > a callback, so a large amount of the update code would need to be rearranged > in order to use DB->associate(). > > 73 de Jeff _______________________________________________ Sks-devel mailing list Sks-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/sks-devel
