Jukka, Marcel, you saw this problem for SegmentMK which uses branch-merge extensively, but is this not a problem all distributed MK implementations will run into? After all, branch-merge is part of the MK API. Unless the MK impl uses pessimistic locking from the start, of course. In particular, I wonder about the MongoMK.
Michael On Mar 7, 2013, at 11:06 AM, Jukka Zitting wrote: > Hi, > > There are a few scenarios where the optimistic locking approach used > by the SegmentMK fails in practice: > > 1) A large batch operation while other smaller changes are being committed. > > 2) Lots of concurrent changes being committed against the same journal. > > In scenario 1 the large operation like an import can't complete itself > since while it is rebasing itself and re-applying commit hooks other > smaller operations have already updated the journal, triggering new > rounds of rebasing and hook processing for the large operation until > it bails out with the "System overloaded" exception. > > In scenario 2 the same "System overloaded" exception occurs once there > are too many concurrent changes for the system to keep up with when > using just a single journal. As noted by Marcel and others, this case > comes up pretty quickly in a benchmark that explicitly tries to push > the system to the limit. > > While in scenario 2 the "System overloaded" exception is a valid > alternative to a potentially prolonged wait until the commit can go > through, in scenario 1 it is clearly troublesome. Thus I'd like to > address it, and the solution I have in mind actually works for both > cases: > > When encountering a case where the optimistic locking mechanism can't > push a commit through in say one second, instead of waiting for a > longer while I'd have the SegmentMK fall back to pessimistic locking > where it explicitly acquired a hard lock on the journal and does the > rebase/hook processing one more time while holding that lock. This > guarantees that all commits will go through eventually (unless there's > a conflict or a validation failure), while keeping the benefits of > optimistic locking for most cases. And even for scenario 1 the bulk of > the commit has already been persisted when the pessimistic locking > kicks in, so the critical section should still be much smaller than > with Jackrabbit 2.x where the lock is held also while the change set > is being persisted. > > BR, > > Jukka Zitting
