When we said we want better data integrity, we meant three things: concurrent editing: it is currently possible for person A's edits to some piece of content to overwrite person B's if the copy person A is working from is older than person B's edits. We need some form of optimistic locking.
atomicity: we would like to be able to store a single integrated domain object (e.g. a page, a world, etc.) in a single operation. Because sparsemap writes to each node in a tree as a separate operation, it is possible for something to go wrong halfway through and you can wind up with half a world, half a page, etc. continued discovery of incompleteness/bugs: you can consult JIRA to see the storage-related issues we have encountered. In the current crop, content can be flagged for delete without any deletes actually taking place. We don't have any reason to believe our maintenance burden is going to lighten any time soon, so we agreed in discussions in Ann Arbor to stop leaning on sparsemap for new functionality. We will, of course, continue to fix any serious bugs we find for as long as that code is still in use. With enough work, we could do all these things with sparsemap, but our divided focus on application and storage layer is not sustainable. As Ray pointed out, we are not in the business of writing/maintaining frameworks and infrastructure. We need to be free to concentrate our effort on our domain: teaching, learning, and collaboration. Zach On May 31, 2012, at 11:16 AM, Ray Davis wrote: > On 5/31/12 8:46 AM, Daniel Parry wrote: >> On Thu, May 31, 2012 at 08:30:07AM -0700, Ray Davis wrote: >>> It's probably easiest if you just browse through the JIRA tickets: >>> >>> https://jira.sakaiproject.org/secure/IssueNavigator.jspa?reset=true&mode=hide&jqlQuery=project+%3D+SPARSE >>> >>> Because any storage solution must address both persistence and querying, >>> I would also include the Solr-integration bugs: >>> >>> https://jira.sakaiproject.org/secure/IssueNavigator.jspa?reset=true&mode=hide&jqlQuery=project+%3D+SOLR >>> >>> In both cases, many of the bugs have been fixed since this adventure >>> started in February 2011 while some still have not. >> >> Would it be fair to say that of the handful of unresolved JIRAs in those two >> queues, none of them pertain to data integrity issues? > > No. > > https://jira.sakaiproject.org/browse/SPARSE-178 > https://jira.sakaiproject.org/browse/SPARSE-195 > https://jira.sakaiproject.org/browse/SPARSE-196 > https://jira.sakaiproject.org/browse/SPARSE-197 > https://jira.sakaiproject.org/browse/SOLR-56 > >>> however, is that every time we try to add a feature (or just look more >>> closely at anomalies in our production environment), we've hit new bugs. >> >> Sounds like a functional QA process :) > > Indeed, but QA-ing, researching, and fixing problems in > below-the-business-logic code isn't what our team is paid for. It's what > the teams that support the Spring framework, or Hibernate, or Oracle, or > Jackrabbit, or Django, or so on are paid for. I've had to go there > occasionally, sure, but mostly I'm expected to deliver above those layers. > > Your kilometerage may vary, of course; I can only speak for my own employer. > >>> Which shouldn't surprise anyone. No matter how skilled, a small >>> development team cannot simultaneously deliver new application >>> functionality on schedule and maintain an independently coded simulation >>> of Apache Jackrabbit. >> >> Shouldn't that cease to be a problem if the sparse map content code base is >> effectively frozen? Or does sparse map content need further extending to >> support >> new functionality? > > I try to work from evidence rather than from blind hope. When I look at > where developer and operations time has gone since February 2011, I > don't see problems ceasing. > > Anyway, that's probably enough from me. Maybe other developers or > operations staff or managers could give their point of view? > > Best, > Ray > _______________________________________________ > oae-production mailing list > oae-product...@collab.sakaiproject.org > http://collab.sakaiproject.org/mailman/listinfo/oae-production _______________________________________________ oae-dev mailing list oae-dev@collab.sakaiproject.org http://collab.sakaiproject.org/mailman/listinfo/oae-dev
