Re: Qpid Java client deadlocks

2015-02-09 Thread Rob Godfrey
>From my point of view, I'm thinking that if an isolated change - such as the one I am implicitly proposing - makes things *better* (in this case solving the issue that Helen is seeing) and doesn't obviously introduce any new issues, then it appears worthwhile. Trying to reengineer the client to g

Re: Qpid Java client deadlocks

2015-02-09 Thread Rajith Muditha Attapattu
On Mon, Feb 9, 2015 at 6:15 AM, Robbie Gemmell wrote: > The main thing I recall from back then is it being hard to > reason about the impact of such changes and the interaction of the > various locks due to the way the client threads can enter the > application space (e.g onMessage, exception lis

Re: Qpid Java client deadlocks

2015-02-09 Thread Robbie Gemmell
Sounds reasonable. Robbie On 9 February 2015 at 13:40, Rob Godfrey wrote: > So looking back to when the MDL acquisition in closed() was first > introduced... it seems to have been for > https://issues.apache.org/jira/browse/QPID-547 which I think really should > only have covered the close() cas

Re: Qpid Java client deadlocks

2015-02-09 Thread Rob Godfrey
So looking back to when the MDL acquisition in closed() was first introduced... it seems to have been for https://issues.apache.org/jira/browse/QPID-547 which I think really should only have covered the close() case (i.e. application initiated closure) rather than the server initiated closure. As

Re: Qpid Java client deadlocks

2015-02-09 Thread Robbie Gemmell
Taking a quick peek it as you said *seems* safe, since the 'closed' call looks like the only place a non-app non-dispatcher thread could get the MDL. I would probably need a long time to really convince myself though. Robbie On 9 February 2015 at 11:47, Rob Godfrey wrote: > Hi Robbie, > > yes -

Re: Qpid Java client deadlocks

2015-02-09 Thread Rob Godfrey
Hi Robbie, yes - my thinking is currently that the acquiring of the MDL in AMQSession.closed() is incorrect. The MDL seems to be there to provide JMS semantics around the interaction between message delivery and other methods on the (JMS) session object. As such it should only really be acquired

Re: Qpid Java client deadlocks

2015-02-09 Thread Robbie Gemmell
Hi Rob, Its a couple of years since I last did any real work on the 0-10 client, but around that time I did review some proposed changes attempting removal of the MDL enough to know they likely introduced new issues. The main thing I recall from back then is it being hard to reason about the impac

Re: Qpid Java client deadlocks

2015-02-08 Thread Rajith Muditha Attapattu
Rob, It's been a bit more than a year since I've last looked at the code. So I don't remember all the details, unless I look at it again. I made a reasonable effort (and other folks too) to address these issues, but ran into other issues that made the code unstable. There are several variations of

Re: Qpid Java client deadlocks

2015-02-08 Thread Rob Godfrey
Hi Rajith, so - I'm no expert on the locking model in the client... Logically I would think that with the message delivery lock (MDL) being a session level lock, and the failover mutex (FM) being connection wide, we should always acquire the MDL before the FM. Where we are currently acquiring the

Re: Qpid Java client deadlocks

2015-02-05 Thread Rajith Muditha Attapattu
John, If you are unable to wait for the AMQP 1.0 JMS client to be completed, may I suggest an interim solution that *may* work for you. This solution was provided for a customer who was completely blocked due to the above mentioned deadlocks. It's not a Qpid supported client. But it may provide r

Re: Qpid Java client deadlocks

2015-02-05 Thread John Buisson
We will definitely need a timeline on this. We are dequeueing messages on 12 app servers per "pod" with 32 threads per server dequeuing messages. We have over 65 pods. This means, at any given time, we have around 25,000 dequeue threads running. While there are isolated systems, we are encounte

Re: Qpid Java client deadlocks

2015-02-05 Thread Rajith Muditha Attapattu
The consensus is, that there is no easy fix, other than rewriting some key pieces of the client. Given that a new JMS client is being actively developed, we have decided not to devote any significant time/resources on the older client. The new JMS client is AMQP 1.0 and the source is here https://

Re: Qpid Java client deadlocks

2015-02-04 Thread Rob Godfrey
To answer the easy question first, 0.32 clients should be totally compatible with an 0.16 broker. In terms of the deadlock issue, I know there are a number of open deadlock JIRAs that Robbie and Rajith have looked at in the past... Hopefully one of them will be able to chip in here and discuss the

Qpid Java client deadlocks

2015-02-04 Thread Helen Kwong
Hi Qpid gurus, We are using 0.16 Java broker and client on 0-10, and we are running into deadlock issues on the client that involve AMQSession's _messageDeliveryLock and AMQConnection's _failoverMutex, where different threads acquire them in different orders. This is leading to major production he