Hi Ishara,
Please note that I tried this with ActiveMQ. I didn't see the reconnection
attempts in the log rather a rapid generation of logs. The log file grew to
100MB in less than a minute.
On Mon, Aug 19, 2013 at 4:21 PM, Ishara Premadasa <ish...@wso2.com> wrote:
> Hi Kasun,
>
> I first tried the JMS transport reconnection mechanism with AS 5.1.0 and
> then i tried with the newly built AS 5.2.0 pack. However with both these
> packs the reconnection works fine where it waits for the JMSConnection to
> come alive by retrying in exponentially increasing time intervals. Please
> see the AS console log i get in the this situation below.
>
> [2013-08-19 15:28:02,454] INFO {org.wso2.andes.client.AMQConnection} -
> Unable to connect to broker at tcp://localhost:5673
> org.wso2.andes.transport.TransportException: Could not open connection
> at
> org.wso2.andes.transport.network.mina.MinaNetworkTransport$IoConnectorCreator.connect(MinaNetworkTransport.java:201)
> at
> org.wso2.andes.transport.network.mina.MinaNetworkTransport.connect(MinaNetworkTransport.java:71)
> at
> org.wso2.andes.client.AMQConnectionDelegate_8_0.makeBrokerConnection(AMQConnectionDelegate_8_0.java:110)
> at
> org.wso2.andes.client.AMQConnection.makeBrokerConnection(AMQConnection.java:616)
> at org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:398)
> at
> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:315)
> at
> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:38)
>
> at
> org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:824)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.getConnection(ServiceTaskManager.java:706)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.receiveMessage(ServiceTaskManager.java:496)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:420)
> at
> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> [2013-08-19 15:28:02,454] ERROR
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Error acquiring a
> JMS connection to : QueueConnectionFactory using JNDI properties :
> {java.naming.factory.initial=org.wso2.andes.jndi.PropertiesFileInitialContextFactory,
> java.naming.provider.url=repository/conf/jndi.properties,
> transport.jms.ConnectionFactoryType=queue,
> ServiceClass=org.wso2.carbon.mashup.javascript.stubgenerator.JSStubGenerator
> }
> javax.jms.JMSException: Error creating connection: Could not open
> connection
> at
> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:325)
> at
> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:38)
>
> at
> org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:824)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.getConnection(ServiceTaskManager.java:706)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.receiveMessage(ServiceTaskManager.java:496)
> at
> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:420)
> at
> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.wso2.andes.AMQConnectionFailureException: Could not open
> connection
> at org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:475)
> at
> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:315)
> ... 10 more
> Caused by: org.wso2.andes.transport.TransportException: Could not open
> connection
> at
> org.wso2.andes.transport.network.mina.MinaNetworkTransport$IoConnectorCreator.connect(MinaNetworkTransport.java:201)
> at
> org.wso2.andes.transport.network.mina.MinaNetworkTransport.connect(MinaNetworkTransport.java:71)
> at
> org.wso2.andes.client.AMQConnectionDelegate_8_0.makeBrokerConnection(AMQConnectionDelegate_8_0.java:110)
> at
> org.wso2.andes.client.AMQConnection.makeBrokerConnection(AMQConnection.java:616)
> at org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:398)
> ... 11 more
> [2013-08-19 15:28:07,451] ERROR
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection attempt
> : 2 for service : HelloService failed. Next retry in 40 seconds
> [2013-08-19 15:28:07,451] ERROR
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection attempt
> : 2 for service : JavaScriptStubGeneratorService failed. Next retry in 40
> seconds
> [2013-08-19 15:28:47,452] INFO
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection attempt
> : 3 for service : HelloService
> [2013-08-19 15:28:47,452] INFO
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection attempt
> : 3 for service : JavaScriptStubGeneratorService
> [2013-08-19 15:28:52,453] WARN
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Unable to shutdown
> all polling tasks of service : HelloService
> [2013-08-19 15:28:52,453] WARN
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Unable to shutdown
> all polling tasks of service : JavaScriptStubGeneratorService
> [2013-08-19 15:28:52,453] INFO
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
> service : HelloService shutdown
> [2013-08-19 15:28:52,454] INFO
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
> service : JavaScriptStubGeneratorService shutdown
> [2013-08-19 15:28:52,454] INFO
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
> service : HelloService [re-]initialized
> [2013-08-19 15:28:52,454] INFO
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
> service : JavaScriptStubGeneratorService [re-]initialized
> [2013-08-19 15:28:53,455] INFO
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection
> attempt: 3 for service: HelloService was successful!
> [2013-08-19 15:28:53,455] INFO
> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection
> attempt: 3 for service: JavaScriptStubGeneratorService was successful!
>
> Therefore the exponential reconnection attempts already work with AS 5.2.0
> as i observed. Please let us know if this is not the error you see in the
> console when MB server is down and AS is trying to reconnect.
>
> Thanks!
> Ishara
>
>
>
>
> On Mon, Aug 19, 2013 at 10:15 AM, Kasun Gajasinghe <kas...@wso2.com>wrote:
>
>> Hi,
>>
>> The error log that repeats is the following. The error logs stop when the
>> activemq server is started.
>>
>> TID: [0] [AS] [2013-08-19 10:12:43,835] ERROR
>> {org.apache.axis2.transport.base.threads.NativeWorkerPool} - Uncaught
>> exception {org.apache.axis2.transport.base.threads.NativeWorkerPool}
>> org.apache.axis2.transport.jms.AxisJMSException: Error acquiring a JMS
>> connection to : QueueConnectionFactory using JNDI properties :
>> {java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory,
>> org.wso2.xkms2.service.crypto.persistence.enabled=true,
>> org.wso2.xkms2.service.crypto.keystore.location=keystore.jks,
>> org.wso2.xkms2.service.crypto.server.key.password=password,
>> org.wso2.xkms2.service.crypto.keystore.password=password,
>> org.wso2.xkms2.service.crypto.default.expriy.interval=365,
>> org.wso2.xkms2.service.crypto.default.private.key.password=password,
>> org.wso2.xkms2.service.crypto.server.cert.aliase=bob,
>> org.wso2.xkms2.service.crypto.issuer.cert.aliase=alice,
>> org.wso2.xkms2.service.crypto.issuer.key.password=password,
>> java.naming.provider.url=tcp://localhost:61616,
>> org.wso2.xkms2.service.crypto.authen.code=secret}
>> at
>> org.apache.axis2.transport.jms.ServiceTaskManager.handleException(ServiceTaskManager.java:1000)
>> at
>> org.apache.axis2.transport.jms.ServiceTaskManager.access$700(ServiceTaskManager.java:50)
>> at
>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:835)
>> at
>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.getConnection(ServiceTaskManager.java:706)
>> at
>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.receiveMessage(ServiceTaskManager.java:496)
>> at
>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:420)
>> at
>> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:662)
>> Caused by: javax.jms.JMSException: Could not connect to broker URL:
>> tcp://localhost:61616. Reason: java.net.ConnectException: Connection refused
>> at
>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
>> at
>> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:280)
>> at
>> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:224)
>> at
>> org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:172)
>> at
>> org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
>> at
>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:824)
>> ... 7 more
>> Caused by: java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>> at java.net.Socket.connect(Socket.java:529)
>> at
>> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:435)
>> at
>> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:401)
>> at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:53)
>> at
>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>> at
>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>> at
>> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
>> at
>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>> at
>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>> at
>> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:260)
>> ... 11 more
>>
>>
>>
>> On Mon, Aug 19, 2013 at 5:38 AM, Ishara Premadasa <ish...@wso2.com>wrote:
>>
>>> Hi Kasun,
>>>
>>> We will be looking into this today and update.
>>> Thanks!
>>>
>>>
>>> On Sun, Aug 18, 2013 at 12:14 PM, Kasun Gajasinghe <kas...@wso2.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> It seems that AS 5.2.0 is not patched with the issue with JMS transport
>>>> where the services with JMS enabled repeatedly try to establish a
>>>> connection with the message broker. This results in GBs of logs in a matter
>>>> of minutes filling up the disk.
>>>>
>>>> As I got to know from Shammi, the solution is to exponentially increase
>>>> the waiting time for the connection. Something like, trying in 1sec, 2sec,
>>>> 4sec, 8, 16, 32 etc. This fix is already impelemented but it seems that it
>>>> didn't get in to AS packs that is about to be released. I tried a setup
>>>> with ActiveMQ, and I can see that it tries to establish a connection with
>>>> the broker every second or so.
>>>>
>>>> Can the MB team look in to this ASAP please? Carbon 4.2.0 is due on
>>>> coming Thursday.
>>>>
>>>> Thanks,
>>>> KasunG
>>>>
>>>> --
>>>> *Kasun Gajasinghe*
>>>> Software Engineer;
>>>> Development Technologies Team, WSO2 Inc.; http://wso2.com
>>>>
>>>>
>>>> ,
>>>> *email: **kasung AT spamfree wso2.com
>>>>
>>>>
>>>> ** cell: **+94 (77) 678-0813*
>>>> *linked-in: *http://lk.linkedin.com/in/gajasinghe
>>>>
>>>>
>>>> *
>>>> *
>>>> *blog: **http://kasunbg.org* <http://kasunbg.org>
>>>>
>>>>
>>>> *
>>>> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
>>>>
>>>>
>>>> *
>>>> *
>>>>
>>>
>>>
>>>
>>> --
>>> Ishara Premasada
>>> Software Engineer,
>>> WSO2 Inc. http://wso2.com/
>>> *Blog : http://isharapremadasa.blogspot.com/
>>> Twitter : https://twitter.com/ishadil
>>> Mobile : +94 714445832*
>>>
>>>
>>>
>>
>>
>> --
>> *Kasun Gajasinghe*
>> Software Engineer;
>> Development Technologies Team, WSO2 Inc.; http://wso2.com
>>
>>
>> ,
>> *email: **kasung AT spamfree wso2.com
>>
>>
>> ** cell: **+94 (77) 678-0813*
>> *linked-in: *http://lk.linkedin.com/in/gajasinghe
>>
>>
>> *
>> *
>> *blog: **http://kasunbg.org* <http://kasunbg.org>
>>
>>
>> *
>> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
>>
>>
>> *
>> *
>>
>
>
>
> --
> Ishara Premasada
> Software Engineer,
> WSO2 Inc. http://wso2.com/
> *Blog : http://isharapremadasa.blogspot.com/
> Twitter : https://twitter.com/ishadil
> Mobile : +94 714445832*
>
>
>
--
*Kasun Gajasinghe*
Software Engineer;
Development Technologies Team, WSO2 Inc.; http://wso2.com
,
*email: **kasung AT spamfree wso2.com
** cell: **+94 (77) 678-0813*
*linked-in: *http://lk.linkedin.com/in/gajasinghe
*
*
*blog: **http://kasunbg.org* <http://kasunbg.org>
*
twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
*
*
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
