Re: [C++/Java Broker] [Transactions]
On Wed, 2016-01-06 at 08:51 +0100, Olivier Mallassi wrote: > Hi all, > > > sorry. > > I have tried to summarize all this content in this table > https://docs.google.com/spreadsheets/d/1JilNwbBSftHxPZymJoaSeSA1uxpp9 > ZwXZyzDM9q6LQA/edit?usp=sharing > > Hope that helps > > oliv/ > > PS: thx for the precisions on the link between on > session/transaction/process. As Gordon pointed out my initial comments about dispatch and transactions were over-simplistic. Dispatch does "link routing" and "message routing" To support all AMQP 1.0 transaction styles we need to extend "link routing" to "session routing" so a client can collect together all the links in a transaction (including the AMQP transaction factory link) and ensure they are all routed to the same domain (broker). Then supporting TX or DTX is simple, we just need to forward the transaction ID info. Dispatch "message routing" deliberately decouples things so that even on a single link, messages may be routed to/from many independent brokers or non-broker endpoints. Simply forwarding AMQP transaction information doesn't make sense there. HOWEVER for DTX there might be another option. Dispatch could use AMQP only to get the XA transaction ID from the local client. We could forward the XA info without any session context and use it in a new session context at the remote end. This would let dispatch participate in arbitrary XA transactions that are not scoped to any single AMQP broker (even ones that include non AMQP resources.) The trick would be where to get the XA transaction from in the first place. It is Friday so this may all be madness. > > > On Tue, Jan 5, 2016 at 12:54 PM, Gordon Sim wrote: > > > On 01/05/2016 11:42 AM, Robbie Gemmell wrote: > > > > > On 4 January 2016 at 19:59, Gordon Sim wrote: > > > > > > > On 12/30/2015 06:37 PM, aconway wrote: > > > > > > > > > I believe dispatch *could* support at least some of the AMQP > > > > > 1.0 TX and > > > > > DTX modes even with message routing, we just haven't gotten > > > > > there yet. > > > > > > > > > > > > > > > > > For local transactions, any transfer or disposition related to > > > > the > > > > transaction needs to be routed to the same process, over the > > > > same > > > > connection[1], as that to which the transaction controller link > > > > is > > > > routed. > > > > > > > > [1] If the multi-ssns-per-txn capability is not supported by > > > > the process > > > > in > > > > question, the same session would have to be used. If the > > > > multi-txns-per-ssn > > > > capability was not supported then you would need a unique > > > > session for > > > > every > > > > distinct transaction. > > > > > > > > > > > To clarify that last bit, I asume you mean every currently active > > > transaction? > > > > > > > Yes, once the transaction is completed the session would be free to > > be > > associated with another one. > > > > > > > > - > > > > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > > For additional commands, e-mail: users-h...@qpid.apache.org > > > > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [C++/Java Broker] [Transactions]
On 01/06/2016 07:51 AM, Olivier Mallassi wrote: I have tried to summarize all this content in this table https://docs.google.com/spreadsheets/d/1JilNwbBSftHxPZymJoaSeSA1uxpp9ZwXZyzDM9q6LQA/edit?usp=sharing Some further clarifications: The c++ qpid::messaging API does not support distributed transactions at all. Such support would need to be added in conjunction with some XA transaction manager. The JMS API defines optional extended interfaces for XA support. These are implemented only for AMQP 0-10 and are not intended for direct invocation by application code but rather as part of the integration into a JEE application server which would be used for running the application code. This JCA integration work for Qpid has not been widely used. The only other way to drive distributed transactions is through the low level AMQP 0-10 APIs in python or c++ which are now considered deprecated for application use (these are what is used for the tests associated with the c++ brokers dtx support). - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [C++/Java Broker] [Transactions]
Hi all, sorry. I have tried to summarize all this content in this table https://docs.google.com/spreadsheets/d/1JilNwbBSftHxPZymJoaSeSA1uxpp9ZwXZyzDM9q6LQA/edit?usp=sharing Hope that helps oliv/ PS: thx for the precisions on the link between on session/transaction/process. On Tue, Jan 5, 2016 at 12:54 PM, Gordon Sim wrote: > On 01/05/2016 11:42 AM, Robbie Gemmell wrote: > >> On 4 January 2016 at 19:59, Gordon Sim wrote: >> >>> On 12/30/2015 06:37 PM, aconway wrote: >>> I believe dispatch *could* support at least some of the AMQP 1.0 TX and DTX modes even with message routing, we just haven't gotten there yet. >>> >>> >>> For local transactions, any transfer or disposition related to the >>> transaction needs to be routed to the same process, over the same >>> connection[1], as that to which the transaction controller link is >>> routed. >>> >>> [1] If the multi-ssns-per-txn capability is not supported by the process >>> in >>> question, the same session would have to be used. If the >>> multi-txns-per-ssn >>> capability was not supported then you would need a unique session for >>> every >>> distinct transaction. >>> >>> >> To clarify that last bit, I asume you mean every currently active >> transaction? >> > > Yes, once the transaction is completed the session would be free to be > associated with another one. > > > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: [C++/Java Broker] [Transactions]
On 01/05/2016 11:42 AM, Robbie Gemmell wrote: On 4 January 2016 at 19:59, Gordon Sim wrote: On 12/30/2015 06:37 PM, aconway wrote: I believe dispatch *could* support at least some of the AMQP 1.0 TX and DTX modes even with message routing, we just haven't gotten there yet. For local transactions, any transfer or disposition related to the transaction needs to be routed to the same process, over the same connection[1], as that to which the transaction controller link is routed. [1] If the multi-ssns-per-txn capability is not supported by the process in question, the same session would have to be used. If the multi-txns-per-ssn capability was not supported then you would need a unique session for every distinct transaction. To clarify that last bit, I asume you mean every currently active transaction? Yes, once the transaction is completed the session would be free to be associated with another one. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [C++/Java Broker] [Transactions]
On 4 January 2016 at 19:59, Gordon Sim wrote: > On 12/30/2015 06:37 PM, aconway wrote: >> >> On Wed, 2015-12-30 at 10:22 +0100, Olivier Mallassi wrote: >>> >>> Hi all >>> >>> again, my apologies for all these questions. >>> >>> I am trying to clarify my understanding around transactions support >>> within >>> qpid and here is my current understanding (based on documentation & >>> JIRA). >>> I have to say I did not test anything yet. >> >> >> Here's what I know: >> >> * qpidd C++ without HA: persistence (linearstore), TX, DTX. > > > Though DTX is only supported over AMQP 0-10 at present. As Rob notes, AMQP > 1.0 does not fully define how distributed transactions are supposed to work. > >> * qpidd C++ with HA: persistence, limited TX, no DTX >> * dispatch link routing: will forward TX/DTX (I think, not tested) > > > I'd be quite surprised if that works, unless there has been specific code > added to allow it. The transaction context is sent as a special delivery > state on the transfer or disposition and the way this is exposed in proton > would require explicit handling to copy the data object associated with a > delivery. From a quick grep of the dispatch code I don't see any reference > to the relevant proton method (pn_disposition_data). > >> * dispatch message routing: no TX/DTX > > > [...] >> >> I believe dispatch *could* support at least some of the AMQP 1.0 TX and >> DTX modes even with message routing, we just haven't gotten there yet. > > > For local transactions, any transfer or disposition related to the > transaction needs to be routed to the same process, over the same > connection[1], as that to which the transaction controller link is routed. > > [1] If the multi-ssns-per-txn capability is not supported by the process in > question, the same session would have to be used. If the multi-txns-per-ssn > capability was not supported then you would need a unique session for every > distinct transaction. > To clarify that last bit, I asume you mean every currently active transaction? Robbie - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [C++/Java Broker] [Transactions]
On 12/30/2015 06:37 PM, aconway wrote: On Wed, 2015-12-30 at 10:22 +0100, Olivier Mallassi wrote: Hi all again, my apologies for all these questions. I am trying to clarify my understanding around transactions support within qpid and here is my current understanding (based on documentation & JIRA). I have to say I did not test anything yet. Here's what I know: * qpidd C++ without HA: persistence (linearstore), TX, DTX. Though DTX is only supported over AMQP 0-10 at present. As Rob notes, AMQP 1.0 does not fully define how distributed transactions are supposed to work. * qpidd C++ with HA: persistence, limited TX, no DTX * dispatch link routing: will forward TX/DTX (I think, not tested) I'd be quite surprised if that works, unless there has been specific code added to allow it. The transaction context is sent as a special delivery state on the transfer or disposition and the way this is exposed in proton would require explicit handling to copy the data object associated with a delivery. From a quick grep of the dispatch code I don't see any reference to the relevant proton method (pn_disposition_data). * dispatch message routing: no TX/DTX [...] I believe dispatch *could* support at least some of the AMQP 1.0 TX and DTX modes even with message routing, we just haven't gotten there yet. For local transactions, any transfer or disposition related to the transaction needs to be routed to the same process, over the same connection[1], as that to which the transaction controller link is routed. [1] If the multi-ssns-per-txn capability is not supported by the process in question, the same session would have to be used. If the multi-txns-per-ssn capability was not supported then you would need a unique session for every distinct transaction. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [C++/Java Broker] [Transactions]
On Wed, 2015-12-30 at 10:22 +0100, Olivier Mallassi wrote: > Hi all > > again, my apologies for all these questions. > > I am trying to clarify my understanding around transactions support > within > qpid and here is my current understanding (based on documentation & > JIRA). > I have to say I did not test anything yet. Here's what I know: * qpidd C++ without HA: persistence (linearstore), TX, DTX. * qpidd C++ with HA: persistence, limited TX, no DTX * dispatch link routing: will forward TX/DTX (I think, not tested) * dispatch message routing: no TX/DTX Notes: persistence even without TX gives you at-least-once guarantee, if a message is acknowledged it will not be lost, even if broker crashes and is recovered. qpidd + HA plugin allows TX with restrictions, see https://svn.apache.o rg/viewvc?view=revision&revision=r1701109 No DTX (not impossible but not trivial) There is another option: you could deploy qpidd *without* the HA plugin as a cold-standby cluster. Use a shared persistent store (e.g. GFS) and pacemaker or other cluster manager to run exactly one qpidd at a time from the same store. This would give you hardware redundancy with full TX and DTX. The down side is recovery time from the persistent store, depends heavily on how much queue data there is to recover. I believe dispatch *could* support at least some of the AMQP 1.0 TX and DTX modes even with message routing, we just haven't gotten there yet. > My use cases all go into the same pattern : (1) writes into a DB & > publishes messages or (2) consumes messages and updates a DB, all of > this > being "atomic" (e.g. if the DB writings fail, messages are not sent > etc...), with Java & C++ clients and I am figuring out if I can rely > on XA > / DTX or if I need to find another implementation. > > Here is my current understanding, any comments are welcomed. > > AMQP 0.10AMQP 1.0Deployment typeC++ API JMS APIC++ API JMS APIHA Java > brokeractive/Passive > cluster of Java brokerslocal transactionokokokokdistributed / XA > transactionsokokokokFederated + HA C++ Brokers- Federated Exchanges + > clustering of CPP Brokers > - persistence (no message loss)local transactionok ( > https://issues.apache.org/jira/browse/QPID-3384) > > but looks redundant with https://issues.apache.org/jira/browse/QPID-4 > 328 > > *not sure the backup are asynchronously updated*?ok ( > https://issues.apache.org/jira/browse/QPID-4710)?distributed / XA > transactionsko (based on > https://issues.apache.org/jira/browse/QPID-4328)???Dispatch > Router- local tx will be supported ( > https://issues.apache.org/jira/browse/DISPATCH-195) > - distributed tx may be supported > > > So as you can see, I am not clear about distributed transactions > support > (AMQP 0.10 or 1.0). Can you help ? > > > Cheers. > > oliv/ - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [C++/Java Broker] [Transactions]
Hi Olivier, I'm presuming your mail was formatted as a nice table when you sent it... but the Apache Mailing list seems to have mangled it to a point where, try as I might, I have no idea what it is saying :-( Would it be possible to resend in a way that will survive the transformation into a plain test mail? (you can see how the mail appears currently by looking here: http://qpid.2158936.n2.nabble.com/C-Java-Broker-Transactions-td7635717.html) In terms of the Java Broker (HA cluster and non HA): Distributed Transactions are supported for AMQP 0-10, local (non distributed) transactions are supported for all protocols. AMQP 0-9 and AMQP 1.0 do not define how XA transactions should work. There was work done on how distributed transactions should be sent using AMQP 1.0, but this was never finalised. When the protocol work is done it will be easy to add distributed transaction support for AMQP 1.0 into the Java Broker. -- Rob On 30 December 2015 at 09:22, Olivier Mallassi wrote: > Hi all > > again, my apologies for all these questions. > > I am trying to clarify my understanding around transactions support within > qpid and here is my current understanding (based on documentation & JIRA). > I have to say I did not test anything yet. > > My use cases all go into the same pattern : (1) writes into a DB & > publishes messages or (2) consumes messages and updates a DB, all of this > being "atomic" (e.g. if the DB writings fail, messages are not sent > etc...), with Java & C++ clients and I am figuring out if I can rely on XA > / DTX or if I need to find another implementation. > > Here is my current understanding, any comments are welcomed. > > AMQP 0.10AMQP 1.0Deployment typeC++ API JMS APIC++ API JMS APIHA Java > brokeractive/Passive > cluster of Java brokerslocal transactionokokokokdistributed / XA > transactionsokokokokFederated + HA C++ Brokers- Federated Exchanges + > clustering of CPP Brokers > - persistence (no message loss)local transactionok ( > https://issues.apache.org/jira/browse/QPID-3384) > > but looks redundant with https://issues.apache.org/jira/browse/QPID-4328 > > *not sure the backup are asynchronously updated*?ok ( > https://issues.apache.org/jira/browse/QPID-4710)?distributed / XA > transactionsko (based on > https://issues.apache.org/jira/browse/QPID-4328)???Dispatch > Router- local tx will be supported ( > https://issues.apache.org/jira/browse/DISPATCH-195) > - distributed tx may be supported > > > So as you can see, I am not clear about distributed transactions support > (AMQP 0.10 or 1.0). Can you help ? > > > Cheers. > > oliv/ >
[C++/Java Broker] [Transactions]
Hi all again, my apologies for all these questions. I am trying to clarify my understanding around transactions support within qpid and here is my current understanding (based on documentation & JIRA). I have to say I did not test anything yet. My use cases all go into the same pattern : (1) writes into a DB & publishes messages or (2) consumes messages and updates a DB, all of this being "atomic" (e.g. if the DB writings fail, messages are not sent etc...), with Java & C++ clients and I am figuring out if I can rely on XA / DTX or if I need to find another implementation. Here is my current understanding, any comments are welcomed. AMQP 0.10AMQP 1.0Deployment typeC++ API JMS APIC++ API JMS APIHA Java brokeractive/Passive cluster of Java brokerslocal transactionokokokokdistributed / XA transactionsokokokokFederated + HA C++ Brokers- Federated Exchanges + clustering of CPP Brokers - persistence (no message loss)local transactionok ( https://issues.apache.org/jira/browse/QPID-3384) but looks redundant with https://issues.apache.org/jira/browse/QPID-4328 *not sure the backup are asynchronously updated*?ok ( https://issues.apache.org/jira/browse/QPID-4710)?distributed / XA transactionsko (based on https://issues.apache.org/jira/browse/QPID-4328)???Dispatch Router- local tx will be supported ( https://issues.apache.org/jira/browse/DISPATCH-195) - distributed tx may be supported So as you can see, I am not clear about distributed transactions support (AMQP 0.10 or 1.0). Can you help ? Cheers. oliv/
