Dear devs, Recently I dove into some issues that plagued our production systems, where consistency checks regularly showed that some nodes are not quite as they should be. Mainly orphaned nodes (nodes no longer being referenced from their supposed parent), but also missing child nodes (nodes being referenced from a parent, but non-existent in the database) and wrongly located child nodes (nodes referenced as a child from a parent, but the actual node indicates a different parent). There were some others as well, but these were most prominent.
Now there is a lock of additional logic on top of Jackrabbit, so it took some time narrowing down the actual issue, but I've been able to confirm the issue due to clustering. We intensively use set-ups of Jackrabbit that are clustered. Multiple Jackrabbit instances, with a single MySQL database which includes the binary storage (so no NFS needed). We also use no JCR locking and no transactions, and it may be that when using those the issue becomes less that in our case. But I believe that even with using JCR locking and/or transactions there is still the possibility of the above consistencies issues. To show where I think it is going wrong let me go to the logic of the clustering as I think it works. Before a JCR save action, the journal table in the database is consulted. If there are logged actions performed by other Jackrabbit instances in the cluster than these actions are imported/replayed first (SharedItemStateManager.doExternalUpdate). Then the actual changes are written to the database. Where this fails, it that in between the check for external updates and actually writing stuff into the database, there might actually be update performed by another Jackrabbit instance. And in case these happen to be at the same node, we will simply overwrite the changes of the other node. This is not just a failed action, or even lost data, but actually inconsistencies will occur. I've been able to confirm this using a unit test bashing two cluster nodes over jcr-rmi. There are solutions to this, however they come at a hefty price, one of them I've got working. One of my first attempt was to introduce a modification count column in the bundle data table. This field is actually a duplication of the modcount in the bundle itself, but then when updating a bundle you'll update using "UPDATE .. WHERE id = ? AND modcount = 'expected-old-modcount'" You end up with some additional plumbing to remember the old persisted modcount. This does work in part, no changes are overwritten. However because Jackrabbit writes a changelog using individual updates without a transaction, the other actions will go through, while it should be all-or-nothing. The options which works is to then use transactions at this low level, but this is quite expensive. I've tested this, and then all consistency issues go away. There are other options, like locking tables, using a journal, but all these options I think of are quite intrusive also, while a simple fix at code level without creating cluster barriers seems infeasible. But again my current fix isn't that nice either. But perhaps you have other ideas concerning this problem? \Berry
