PS: I am using qpid-jms-client 0.8.0
Regards,
Adel
On Fri, Mar 11, 2016 at 11:07 AM, Adel Boutros <sta191...@gmail.com> wrote:
> Hello guys,
>
> I have debugged a bit and made some progress. The problem is with the
> multi-thread handling. The main thread exits before the Failover trial
> thread has tried the 2nd connection URL and is this exiting with a
> connection to the unreachable virtual host
>
> You can see in the logs below what I mean in a failing case:
>
> *2016-03-11 10:46:31 DEBUG FailoverProvider:160 - Initiating initial
> connection attempt task*
> *2016-03-11 10:46:33 DEBUG FailoverProvider:653 - Connection attempt:[1]
> to: amqp://localhost:10352?amqp.vhost=weather in-progress*
> *2016-03-11 10:46:34 DEBUG FailoverProvider:963 - Executing Failover Task:
> create -> JmsConnectionInfo {} (1)*
> *2016-03-11 10:46:34 DEBUG FailoverProvider:761 - Failover: the provider
> reports failure: Received error from remote peer without description
> [condition = amqp:not-found]*
> *2016-03-11 10:46:34 DEBUG FailoverProvider:519 - handling Provider
> failure: Received error from remote peer without description [condition =
> amqp:not-found]*
> *2016-03-11 10:46:34 DEBUG FailoverProvider:653 - Connection attempt:[1]
> to: amqp://localhost:5672?amqp.vhost=weather in-progress*
>
> At the time the Failover trial Thread (*FailoverProvider:653 *) is trying
> to connect (Last line of Log), the main thread has exited.
>
> If I go in debug mode and run the steps one by one allowing the Failover
> trial thread to complete the 2nd connection attempt, the test is successful:
>
> *2016-03-11 11:03:37 DEBUG FailoverProvider:160 - Initiating initial
> connection attempt task*
> *2016-03-11 11:04:05 DEBUG FailoverProvider:653 - Connection attempt:[1]
> to: amqp://localhost:10352?amqp.vhost=weather in-progress*
> *2016-03-11 11:04:18 DEBUG FailoverProvider:963 - Executing Failover Task:
> create -> JmsConnectionInfo {} (1)*
> *2016-03-11 11:04:18 DEBUG FailoverProvider:985 - Request received error:
> Cannot send to a non-connected transport.*
> *2016-03-11 11:04:18 DEBUG FailoverProvider:519 - handling Provider
> failure: Cannot send to a non-connected transport.*
> *2016-03-11 11:04:18 DEBUG FailoverProvider:985 - Request received error:
> Cannot send to a non-connected transport.*
> *2016-03-11 11:04:18 DEBUG FailoverProvider:761 - Failover: the provider
> reports failure: Cannot send to a non-connected transport.*
> *2016-03-11 11:04:18 DEBUG FailoverProvider:519 - handling Provider
> failure: Cannot send to a non-connected transport.*
> *2016-03-11 11:04:18 DEBUG FailoverProvider:519 - handling Provider
> failure: Cannot send to a non-connected transport.*
> *2016-03-11 11:04:40 DEBUG FailoverProvider:653 - Connection attempt:[1]
> to: amqp://localhost:5672?amqp.vhost=weather in-progress*
> *2016-03-11 11:04:41 DEBUG FailoverProvider:963 - Executing Failover Task:
> create -> JmsConnectionInfo {} (1)*
>
> *2016-03-11 11:04:42 INFO JmsConnection:1114 - Connection
> ID::af28891b-adbe-47f1-9da2-ad03f124a215:1 connected to remote Broker:
> amqp://localhost:5672*
> *2016-03-11 11:04:42 DEBUG FailoverProvider:963 - Executing Failover Task:
> create -> JmsSessionInfo {} (2)*
>
> Regards,
> Adel
>
> On Fri, Mar 11, 2016 at 10:17 AM, Adel Boutros <sta191...@gmail.com>
> wrote:
>
>> Hello Tim,
>>
>> Please find below a complete test case:
>>
>> *Code*
>>
>> public class ScalabilityTest {
>>
>> //~
>> ----------------------------------------------------------------------------------------------------------------
>> //~ Methods
>> //~
>> ----------------------------------------------------------------------------------------------------------------
>>
>> @Test
>> public void testScalability() throws JMSException {
>> String brokerUrl =
>> "failover:(amqp://localhost:5672,amqp://localhost:10352)?failover.nested.amqp.vhost=weather";
>> ConnectionFactory connectionFactory = new
>> JmsConnectionFactory(brokerUrl);
>> Connection connection = connectionFactory.createConnection();
>> Session session = connection.createSession(false,
>> Session.AUTO_ACKNOWLEDGE);
>> }
>>
>> @Test
>> public void testScalabilityReversedOrder() throws JMSException {
>> String brokerUrl =
>> "failover:(amqp://localhost:10352,amqp://localhost:5672)?failover.nested.amqp.vhost=weather";
>> ConnectionFactory connectionFactory = new
>> JmsConnectionFactory(brokerUrl);
>> Connection connection = connectionFactory.createConnection();
>> Session session = connection.createSession(false,
>> Session.AUTO_ACKNOWLEDGE);
>> }
>> }
>>
>>
>> *The 2nd test fails when the brokers are configured as follows:*
>>
>> [image: Inline image 1]
>>
>>
>> Regards,
>>
>> Adel
>>
>>
>> On Fri, Mar 11, 2016 at 7:55 AM, Adel Boutros <sta191...@gmail.com>
>> wrote:
>>
>>> Hello Tim,
>>>
>>> The error is happening when trying to create a session with the below
>>> failover URL when the first URL is that of the REPLICATE. I am using the
>>> amqp.vhost option to specify the virtual host, could that be the issue? I
>>> mean the vhost is not present on the REPLICATE machine.
>>>
>>>
>>> failover:(amqp://localhost:5672,amqp://localhost:10352)?failover.nested.amqp.vhost=weather
>>>
>>> I will check it in an hour when I am at work and let you know.
>>>
>>> Regards,
>>> Adel
>>> On Mar 11, 2016 00:28, "Timothy Bish" <tabish...@gmail.com> wrote:
>>>
>>>> On 03/10/2016 06:12 PM, Rob Godfrey wrote:
>>>>
>>>>> On 10 March 2016 at 23:44, Adel Boutros <sta191...@gmail.com> wrote:
>>>>>
>>>>> If I compile while skipping tests and deploy the jar then the broker no
>>>>>> longer crashes.
>>>>>> Nevertheless, the client doesn't try the 2nd URL, instead it fails
>>>>>> with the
>>>>>> below stack. If I put back the MASTER URL before the Replicate URL,
>>>>>> then
>>>>>> everything works fine.
>>>>>>
>>>>>> It seems to me that the order in which the URLs are provided in the
>>>>>> failover shouldn't matter. Am I correct?
>>>>>>
>>>>>> I'm not too familiar with the new AMQP 1.0 JMS client and how it
>>>>> handles
>>>>> failure and failover... Robbie and Tim can hopefully step in here and
>>>>> say
>>>>> why it is not trying the second URL in the connection string you are
>>>>> providing. Ultimately in the broker we'll be implementing a redirect
>>>>> which
>>>>> will tell the client where it should try to reconnect to (i.e. the
>>>>> host/port of the current master) however currently it just responds
>>>>> that
>>>>> the host name (virtual host) is not found at the location. I don't
>>>>> know
>>>>> whether the JMS client handles the redirect either to be fair.
>>>>>
>>>>> -- Rob
>>>>>
>>>>> I recall writing code and a test for redirect so I think it should in
>>>> theory work if one is provided, but I don't have the code in front of me.
>>>>
>>>>
>>>>>
>>>>> javax.jms.InvalidDestinationException: Received error from remote peer
>>>>>> without description [condition = amqp:not-found]
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.qpid.jms.provider.amqp.AmqpSupport.convertToException(AmqpSupport.java:109)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.qpid.jms.provider.amqp.AmqpAbstractResource.remotelyClosed(AmqpAbstractResource.java:115)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.qpid.jms.provider.amqp.AmqpAbstractResource.processRemoteClose(AmqpAbstractResource.java:210)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.qpid.jms.provider.amqp.AmqpProvider.processUpdates(AmqpProvider.java:772)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.qpid.jms.provider.amqp.AmqpProvider.access$1900(AmqpProvider.java:92)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.qpid.jms.provider.amqp.AmqpProvider$17.run(AmqpProvider.java:701)
>>>>>> at
>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>
>>>>>> Regards,
>>>>>> Adel
>>>>>>
>>>>>> On Thu, Mar 10, 2016 at 11:36 PM, Adel Boutros <sta191...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hello Rob,
>>>>>>>
>>>>>>> I have 3 tests failing under *ProtocolEngine_1_0_0Test* when
>>>>>>> compiling
>>>>>>>
>>>>>> *broker-plugins
>>>>>>
>>>>>>> 6.0.2-SNAPSHOT *in Intellij or maven
>>>>>>>
>>>>>>> -
>>>>>>> ProtocolEngine_1_0_0Test.testProtocolEngineWithNoSaslNonTLSandAnon
>>>>>>> -
>>>>>>> ProtocolEngine_1_0_0Test.testProtocolEngineWithNoSaslTLSandExternal
>>>>>>> -
>>>>>>> ProtocolEngine_1_0_0Test.testProtocolEngineWithSaslNonTLSandAnon
>>>>>>>
>>>>>>>
>>>>>>> All 3 tests are failing with the same issue:
>>>>>>>
>>>>>>> Wanted but not invoked:
>>>>>>> virtualHostImpl.registerConnection(<any>);
>>>>>>> -> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0Test.testProtocolEngineWithNoSaslNonTLSandAnon(ProtocolEngine_1_0_0Test.java:176)
>>>>>>
>>>>>>> However, there were other interactions with this mock:
>>>>>>> -> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.qpid.server.protocol.v1_0.Connection_1_0.openReceived(Connection_1_0.java:182)
>>>>>>
>>>>>>> -> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.qpid.server.protocol.v1_0.Connection_1_0.openReceived(Connection_1_0.java:191)
>>>>>>
>>>>>>> Wanted but not invoked:
>>>>>>> virtualHostImpl.registerConnection(<any>);
>>>>>>> -> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.qpid.server.protocol.v1_0.ProtocolEngine_1_0_0Test.testProtocolEngineWithNoSaslNonTLSandAnon(ProtocolEngine_1_0_0Test.java:176)
>>>>>>
>>>>>>> However, there were other interactions with this mock:
>>>>>>> -> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.qpid.server.protocol.v1_0.Connection_1_0.openReceived(Connection_1_0.java:182)
>>>>>>
>>>>>>> -> at
>>>>>>>
>>>>>>>
>>>>>> org.apache.qpid.server.protocol.v1_0.Connection_1_0.openReceived(Connection_1_0.java:191)
>>>>>>
>>>>>>> Can you please advise?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Adel Boutros
>>>>>>>
>>>>>>> On Thu, Mar 10, 2016 at 9:20 PM, Rob Godfrey <
>>>>>>> rob.j.godf...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Apologies for this - you have indeed run into bug :-(
>>>>>>>>
>>>>>>>> I've raised a JIRA -
>>>>>>>> https://issues.apache.org/jira/browse/QPID-7136 .
>>>>>>>>
>>>>>>>> I've committed a fix both to trunk, and the 6.0.x branch which I
>>>>>>>> hope
>>>>>>>>
>>>>>>> will
>>>>>>
>>>>>>> fix this issue... If you want to test it out you can use the 6.0.x
>>>>>>>>
>>>>>>> branch
>>>>>>
>>>>>>> which can be found here:
>>>>>>>>
>>>>>>>> https://svn.apache.org/repos/asf/qpid/java/branches/6.0.x
>>>>>>>>
>>>>>>>>
>>>>>>>> Hope this helps,
>>>>>>>> -- Rob
>>>>>>>>
>>>>>>>> On 10 March 2016 at 20:10, Adel Boutros <sta191...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> When we provide a failover URL where the first URL is that of a
>>>>>>>>>
>>>>>>>> replicate
>>>>>>>>
>>>>>>>>> broker, the connection will trigger an IllegalStateException which
>>>>>>>>> is
>>>>>>>>>
>>>>>>>> not
>>>>>>>>
>>>>>>>>> caught anywhere. Thus making the broker crash and the session
>>>>>>>>> creation
>>>>>>>>>
>>>>>>>> fail
>>>>>>>>
>>>>>>>>> on the client-side.
>>>>>>>>>
>>>>>>>>> Shouldn't the "close" method log the exception silently instead of
>>>>>>>>>
>>>>>>>> crashing
>>>>>>>>
>>>>>>>>> the broker?
>>>>>>>>>
>>>>>>>>> *Code*
>>>>>>>>>
>>>>>>>>> String brokerUrl =
>>>>>>>>> "failover:(amqp://localhost:5672,amqp://localhost:10352)";
>>>>>>>>> ConnectionFactory connectionFactory = new
>>>>>>>>>
>>>>>>>> JmsConnectionFactory(brokerUrl);
>>>>>>>>
>>>>>>>>> Connection connection = connectionFactory.createConnection();
>>>>>>>>> Session session = connection.createSession(false,
>>>>>>>>> Session.AUTO_ACKNOWLEDGE);
>>>>>>>>>
>>>>>>>>> *Test case*
>>>>>>>>> Assuming that the broker on *localhost:5672 *is alive but as a
>>>>>>>>>
>>>>>>>> *REPLICATE*
>>>>>>>>
>>>>>>>>> and the one on* localhost:10352* as the *MASTER*
>>>>>>>>>
>>>>>>>>> *Stack trace*
>>>>>>>>> java.lang.IllegalStateException: The virtual host state of
>>>>>>>>> UNAVAILABLE
>>>>>>>>>
>>>>>>>> does
>>>>>>>>
>>>>>>>>> not permit this operation.
>>>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> org.apache.qpid.server.virtualhost.berkeleydb.BDBHAReplicaVirtualHostImpl.throwUnsupportedForReplica(BDBHAReplicaVirtualHostImpl.java:567)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> org.apache.qpid.server.virtualhost.berkeleydb.BDBHAReplicaVirtualHostImpl.deregisterConnection(BDBHAReplicaVirtualHostImpl.java:562)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> org.apache.qpid.server.protocol.v1_0.Connection_1_0.closed(Connection_1_0.java:282)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0.closed(AMQPConnection_1_0.java:470)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> org.apache.qpid.server.transport.MultiVersionProtocolEngine.closed(MultiVersionProtocolEngine.java:112)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> org.apache.qpid.server.transport.NonBlockingConnection.shutdown(NonBlockingConnection.java:359)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:299)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:108)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:499)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:337)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:86)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:457)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>
>>>>>>> at
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>
>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>>>>
>>>>>>>>> If you invert the order of the URLs to put the master first or
>>>>>>>>> simply
>>>>>>>>>
>>>>>>>> kill
>>>>>>>>
>>>>>>>>> the REPLICATE broker, then there will be no exception triggered.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Adel Boutros
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> Tim Bish
>>>> twitter: @tabish121
>>>> blog: http://timbish.blogspot.com/
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
>>>> For additional commands, e-mail: users-h...@qpid.apache.org
>>>>
>>>>
>>
>
