Hi Dave. I think your proposal would be useful to have in Clojure, I have thought about something similar since I read about the STM. But I also think there are quite a few difficulties in implementing this in a "sane" way. Actually, the more I think about it, the more it seems that the goal of STM (to aid concurrent programming at a micro-level) is not really compatible with that of traditional transactions (to enable persistent and consistent changes to a shared resource, at an application/system level). Here are some observations:
1. Normal transactions A successful commit on an external transaction can be (and usually is) a side-effect, so it breaks the assumption that transactional operations are side-effect free and can be retried by the STM. Thus Clojure would need to recognize this as a special case and treat it as a "composite" transaction, which can not be retried entirely. I don't know how STM is implemented in Clojure. Maybe if the external commit can be delayed until the end of the transaction, when it is certain that the in-memory operations succeeded, it could work. I think this is the most promising way of implementing this. 2. XA transactions If Clojure's STM works similar to an XA transaction, it would still be difficult to delegate the coordination to a JTA transaction manager. XA-capable resources MUST have stable storage for crash recovery, which definitely isn't the case (and also wouldn't make sense) with the STM. The STM guarantees ACI, external transactions are usually ACID, so there's a conceptual mismatch between the two. If you think about it, why would you want to have transactions between a transient and a durable resource ? Also, XA resources must be prepared to handle the case of an "heuristic commit", when all resources confirmed the "prepare" stage, but not the final "commit". This situation does not exist with pure STM, so one would have to be aware of the risk and settle for less guarantees when using "composite" transactions. Razvan. On Dec 4, 1:38 am, Rich Hickey <[EMAIL PROTECTED]> wrote: > On Dec 3, 3:48 pm, Dave Griffith <[EMAIL PROTECTED]> wrote: > > > > > It often happens that I would like to have gaurantees about the > > consistency of in-memory data structures and external resources. For > > instance, in the last two large systems I've built, the system would > > respond to an external message received via socket, do some complex > > processing based on the contents of the message which modified it's > > internal state, write a log of the message to a database and update > > some other database tables so that the internal state of the system > > was reconstructable in case of crash, and then send some other > > messages to clients via socket. This all had to either work, or not > > work, with full ACID properties for the system as a whole (both the > > database and memory). This was an enormous pain, requiring in-memory > > rollback functionality to be coded by hand for each and every possible > > interaction. > > > It seems like it should be possible to extend the STM semantics of > > Clojure to allow this sort of thing to Just Work. From the user point > > of view, one would call dosync-external, rather than dosync, and pass > > it an external (JTA?) transaction manager along with the operations to > > be executed. If either the in-memory operations or the external > > operations required rollback, then both would be, otherwise both would > > be committed. If committed, any agent sends that the transaction had > > sent would occur. Simple to use, and pretty much exactly what's > > required for a high-reliability message processing engine. Literally > > thousands of lines of code would disappear from each of my last two > > projects. > > > Looking over the current implementation of Clojure, I think I've > > convinced myself of the following facts. > > > 1) This can't be done purely in Clojure code itself, as the necessary > > hooks into the STM don't exist (nor, I would argue, should they). > > > 2) This could be done pretty easily by hacking the Clojure runtime, at > > a (obviously excessive) cost of inducing a hard dependency between the > > core runtime and some particular transaction manager. The STM > > implementation already has enough of a two-phase commit structure to > > make it pretty easy to integrate with any given transaction management > > system. > > > 3) Creating some sort of plugin system so that arbitrary transaction > > managers could be integrated with the Clojure STM runtime is doable, > > but doesn't much fit with the current architecture. > > > 4) If one were to do such a thing, it would be pretty easy to extend > > the STM with features like timeouts and such. > > > Thoughts? Am I missing anything? Has this already been attempted in > > some way I was unable to google for? My temptation is to try to > > implement something as a quick spike and see how it works. > > It's definitely an interesting idea - I don't think anyone is working > on it. I don't have enough knowledge of JTA/XA to do it myself, but > could provide advice about integration with the STM. > > Rich --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~----------~----~----~----~------~----~------~--~---
