On Tue, Aug 20, 2013 at 9:42 AM, Ishara Premadasa <ish...@wso2.com> wrote:
>
>
>
> On Mon, Aug 19, 2013 at 11:29 PM, Kasun Gajasinghe <kas...@wso2.com>wrote:
>
>>
>>
>>> However the only incident i have seen a rapid error log thrown
>>> continuosly in the MB or ActiveMQ integration scenario is, when you start
>>> AS for the first time, if you hadn't start the broker (MB/ActiveMQ etc.)
>>> before starting AS it will throw a continuous error log as there is no
>>> JMSConnection available. Apart from that if the broker fails in middle,
>>> after successful start up, the reconnection should happen fine.
>>>
>>>
>>>
>> Yes... This is what I was doing. We should handle this scenario too,
>> right?
>>
>
> @Shammi/Malinga
>
> WDYT? Should we provide some notification and stop the server start if
> user forgets to start JMS provider (broker) in prior to start AS, ESB etc.?
> IMO as our documentation clearly says that it is needed to start the broker
> first in each integrating scenarios, i am not sure whether it is needed
> to handle this.
>
> Hi Ishara,
We do not need to stop any functionality on the unavailability of the
broker. What we should do is , we need to wait and check whether the broker
is there time to time. This waiting time should be grow up exponentially
by making its max limit to one hour.
Thanks
Shammi
>
>>
>>>
>>>
>>> On Mon, Aug 19, 2013 at 4:44 PM, Kasun Gajasinghe <kas...@wso2.com>wrote:
>>>
>>>> Hi Ishara,
>>>>
>>>> Please note that I tried this with ActiveMQ. I didn't see the
>>>> reconnection attempts in the log rather a rapid generation of logs. The log
>>>> file grew to 100MB in less than a minute.
>>>>
>>>>
>>>> On Mon, Aug 19, 2013 at 4:21 PM, Ishara Premadasa <ish...@wso2.com>wrote:
>>>>
>>>>> Hi Kasun,
>>>>>
>>>>> I first tried the JMS transport reconnection mechanism with AS 5.1.0
>>>>> and then i tried with the newly built AS 5.2.0 pack. However with both
>>>>> these packs the reconnection works fine where it waits for the
>>>>> JMSConnection to come alive by retrying in exponentially increasing time
>>>>> intervals. Please see the AS console log i get in the this situation
>>>>> below.
>>>>>
>>>>> [2013-08-19 15:28:02,454] INFO {org.wso2.andes.client.AMQConnection}
>>>>> - Unable to connect to broker at tcp://localhost:5673
>>>>> org.wso2.andes.transport.TransportException: Could not open connection
>>>>> at
>>>>> org.wso2.andes.transport.network.mina.MinaNetworkTransport$IoConnectorCreator.connect(MinaNetworkTransport.java:201)
>>>>> at
>>>>> org.wso2.andes.transport.network.mina.MinaNetworkTransport.connect(MinaNetworkTransport.java:71)
>>>>> at
>>>>> org.wso2.andes.client.AMQConnectionDelegate_8_0.makeBrokerConnection(AMQConnectionDelegate_8_0.java:110)
>>>>> at
>>>>> org.wso2.andes.client.AMQConnection.makeBrokerConnection(AMQConnection.java:616)
>>>>> at
>>>>> org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:398)
>>>>> at
>>>>> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:315)
>>>>> at
>>>>> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:38)
>>>>>
>>>>> at
>>>>> org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:824)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.getConnection(ServiceTaskManager.java:706)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.receiveMessage(ServiceTaskManager.java:496)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:420)
>>>>> at
>>>>> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> [2013-08-19 15:28:02,454] ERROR
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Error acquiring a
>>>>> JMS connection to : QueueConnectionFactory using JNDI properties :
>>>>> {java.naming.factory.initial=org.wso2.andes.jndi.PropertiesFileInitialContextFactory,
>>>>> java.naming.provider.url=repository/conf/jndi.properties,
>>>>> transport.jms.ConnectionFactoryType=queue,
>>>>> ServiceClass=org.wso2.carbon.mashup.javascript.stubgenerator.JSStubGenerator
>>>>> }
>>>>> javax.jms.JMSException: Error creating connection: Could not open
>>>>> connection
>>>>> at
>>>>> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:325)
>>>>> at
>>>>> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:38)
>>>>>
>>>>> at
>>>>> org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:824)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.getConnection(ServiceTaskManager.java:706)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.receiveMessage(ServiceTaskManager.java:496)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:420)
>>>>> at
>>>>> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: org.wso2.andes.AMQConnectionFailureException: Could not
>>>>> open connection
>>>>> at
>>>>> org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:475)
>>>>> at
>>>>> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:315)
>>>>> ... 10 more
>>>>> Caused by: org.wso2.andes.transport.TransportException: Could not open
>>>>> connection
>>>>> at
>>>>> org.wso2.andes.transport.network.mina.MinaNetworkTransport$IoConnectorCreator.connect(MinaNetworkTransport.java:201)
>>>>> at
>>>>> org.wso2.andes.transport.network.mina.MinaNetworkTransport.connect(MinaNetworkTransport.java:71)
>>>>> at
>>>>> org.wso2.andes.client.AMQConnectionDelegate_8_0.makeBrokerConnection(AMQConnectionDelegate_8_0.java:110)
>>>>> at
>>>>> org.wso2.andes.client.AMQConnection.makeBrokerConnection(AMQConnection.java:616)
>>>>> at
>>>>> org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:398)
>>>>> ... 11 more
>>>>> [2013-08-19 15:28:07,451] ERROR
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection
>>>>> attempt
>>>>> : 2 for service : HelloService failed. Next retry in 40 seconds
>>>>> [2013-08-19 15:28:07,451] ERROR
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection
>>>>> attempt
>>>>> : 2 for service : JavaScriptStubGeneratorService failed. Next retry in 40
>>>>> seconds
>>>>> [2013-08-19 15:28:47,452] INFO
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection
>>>>> attempt
>>>>> : 3 for service : HelloService
>>>>> [2013-08-19 15:28:47,452] INFO
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection
>>>>> attempt
>>>>> : 3 for service : JavaScriptStubGeneratorService
>>>>> [2013-08-19 15:28:52,453] WARN
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Unable to shutdown
>>>>> all polling tasks of service : HelloService
>>>>> [2013-08-19 15:28:52,453] WARN
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Unable to shutdown
>>>>> all polling tasks of service : JavaScriptStubGeneratorService
>>>>> [2013-08-19 15:28:52,453] INFO
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
>>>>> service : HelloService shutdown
>>>>> [2013-08-19 15:28:52,454] INFO
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
>>>>> service : JavaScriptStubGeneratorService shutdown
>>>>> [2013-08-19 15:28:52,454] INFO
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
>>>>> service : HelloService [re-]initialized
>>>>> [2013-08-19 15:28:52,454] INFO
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
>>>>> service : JavaScriptStubGeneratorService [re-]initialized
>>>>> [2013-08-19 15:28:53,455] INFO
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection
>>>>> attempt: 3 for service: HelloService was successful!
>>>>> [2013-08-19 15:28:53,455] INFO
>>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection
>>>>> attempt: 3 for service: JavaScriptStubGeneratorService was successful!
>>>>>
>>>>> Therefore the exponential reconnection attempts already work with AS
>>>>> 5.2.0 as i observed. Please let us know if this is not the error you see
>>>>> in
>>>>> the console when MB server is down and AS is trying to reconnect.
>>>>>
>>>>> Thanks!
>>>>> Ishara
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 19, 2013 at 10:15 AM, Kasun Gajasinghe <kas...@wso2.com>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The error log that repeats is the following. The error logs stop when
>>>>>> the activemq server is started.
>>>>>>
>>>>>> TID: [0] [AS] [2013-08-19 10:12:43,835] ERROR
>>>>>> {org.apache.axis2.transport.base.threads.NativeWorkerPool} - Uncaught
>>>>>> exception {org.apache.axis2.transport.base.threads.NativeWorkerPool}
>>>>>> org.apache.axis2.transport.jms.AxisJMSException: Error acquiring a
>>>>>> JMS connection to : QueueConnectionFactory using JNDI properties :
>>>>>> {java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory,
>>>>>> org.wso2.xkms2.service.crypto.persistence.enabled=true,
>>>>>> org.wso2.xkms2.service.crypto.keystore.location=keystore.jks,
>>>>>> org.wso2.xkms2.service.crypto.server.key.password=password,
>>>>>> org.wso2.xkms2.service.crypto.keystore.password=password,
>>>>>> org.wso2.xkms2.service.crypto.default.expriy.interval=365,
>>>>>> org.wso2.xkms2.service.crypto.default.private.key.password=password,
>>>>>> org.wso2.xkms2.service.crypto.server.cert.aliase=bob,
>>>>>> org.wso2.xkms2.service.crypto.issuer.cert.aliase=alice,
>>>>>> org.wso2.xkms2.service.crypto.issuer.key.password=password,
>>>>>> java.naming.provider.url=tcp://localhost:61616,
>>>>>> org.wso2.xkms2.service.crypto.authen.code=secret}
>>>>>> at
>>>>>> org.apache.axis2.transport.jms.ServiceTaskManager.handleException(ServiceTaskManager.java:1000)
>>>>>> at
>>>>>> org.apache.axis2.transport.jms.ServiceTaskManager.access$700(ServiceTaskManager.java:50)
>>>>>> at
>>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:835)
>>>>>> at
>>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.getConnection(ServiceTaskManager.java:706)
>>>>>> at
>>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.receiveMessage(ServiceTaskManager.java:496)
>>>>>> at
>>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:420)
>>>>>> at
>>>>>> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>> Caused by: javax.jms.JMSException: Could not connect to broker URL:
>>>>>> tcp://localhost:61616. Reason: java.net.ConnectException: Connection
>>>>>> refused
>>>>>> at
>>>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
>>>>>> at
>>>>>> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:280)
>>>>>> at
>>>>>> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:224)
>>>>>> at
>>>>>> org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:172)
>>>>>> at
>>>>>> org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
>>>>>> at
>>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:824)
>>>>>> ... 7 more
>>>>>> Caused by: java.net.ConnectException: Connection refused
>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>>>>>> at
>>>>>> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
>>>>>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>>>>>> at java.net.Socket.connect(Socket.java:529)
>>>>>> at
>>>>>> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:435)
>>>>>> at
>>>>>> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:401)
>>>>>> at
>>>>>> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:53)
>>>>>> at
>>>>>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>>>>>> at
>>>>>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>>>>>> at
>>>>>> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
>>>>>> at
>>>>>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>>>>>> at
>>>>>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>>>>>> at
>>>>>> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:260)
>>>>>> ... 11 more
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 19, 2013 at 5:38 AM, Ishara Premadasa <ish...@wso2.com>wrote:
>>>>>>
>>>>>>> Hi Kasun,
>>>>>>>
>>>>>>> We will be looking into this today and update.
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Aug 18, 2013 at 12:14 PM, Kasun Gajasinghe
>>>>>>> <kas...@wso2.com>wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> It seems that AS 5.2.0 is not patched with the issue with JMS
>>>>>>>> transport where the services with JMS enabled repeatedly try to
>>>>>>>> establish
>>>>>>>> a connection with the message broker. This results in GBs of logs in a
>>>>>>>> matter of minutes filling up the disk.
>>>>>>>>
>>>>>>>> As I got to know from Shammi, the solution is to exponentially
>>>>>>>> increase the waiting time for the connection. Something like, trying in
>>>>>>>> 1sec, 2sec, 4sec, 8, 16, 32 etc. This fix is already impelemented but
>>>>>>>> it
>>>>>>>> seems that it didn't get in to AS packs that is about to be released. I
>>>>>>>> tried a setup with ActiveMQ, and I can see that it tries to establish a
>>>>>>>> connection with the broker every second or so.
>>>>>>>>
>>>>>>>> Can the MB team look in to this ASAP please? Carbon 4.2.0 is due on
>>>>>>>> coming Thursday.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> KasunG
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Kasun Gajasinghe*
>>>>>>>> Software Engineer;
>>>>>>>> Development Technologies Team, WSO2 Inc.; http://wso2.com
>>>>>>>>
>>>>>>>>
>>>>>>>> ,
>>>>>>>> *email: **kasung AT spamfree wso2.com
>>>>>>>>
>>>>>>>>
>>>>>>>> ** cell: **+94 (77) 678-0813*
>>>>>>>> *linked-in: *http://lk.linkedin.com/in/gajasinghe
>>>>>>>>
>>>>>>>>
>>>>>>>> *
>>>>>>>> *
>>>>>>>> *blog: **http://kasunbg.org* <http://kasunbg.org>
>>>>>>>>
>>>>>>>>
>>>>>>>> *
>>>>>>>> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
>>>>>>>>
>>>>>>>>
>>>>>>>> *
>>>>>>>> *
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Ishara Premasada
>>>>>>> Software Engineer,
>>>>>>> WSO2 Inc. http://wso2.com/
>>>>>>> *Blog : http://isharapremadasa.blogspot.com/
>>>>>>> Twitter : https://twitter.com/ishadil
>>>>>>> Mobile : +94 714445832*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Kasun Gajasinghe*
>>>>>> Software Engineer;
>>>>>> Development Technologies Team, WSO2 Inc.; http://wso2.com
>>>>>>
>>>>>>
>>>>>> ,
>>>>>> *email: **kasung AT spamfree wso2.com
>>>>>>
>>>>>>
>>>>>> ** cell: **+94 (77) 678-0813*
>>>>>> *linked-in: *http://lk.linkedin.com/in/gajasinghe
>>>>>>
>>>>>>
>>>>>> *
>>>>>> *
>>>>>> *blog: **http://kasunbg.org* <http://kasunbg.org>
>>>>>>
>>>>>>
>>>>>> *
>>>>>> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
>>>>>>
>>>>>>
>>>>>> *
>>>>>> *
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Ishara Premasada
>>>>> Software Engineer,
>>>>> WSO2 Inc. http://wso2.com/
>>>>> *Blog : http://isharapremadasa.blogspot.com/
>>>>> Twitter : https://twitter.com/ishadil
>>>>> Mobile : +94 714445832*
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Kasun Gajasinghe*
>>>> Software Engineer;
>>>> Development Technologies Team, WSO2 Inc.; http://wso2.com
>>>>
>>>>
>>>> ,
>>>> *email: **kasung AT spamfree wso2.com
>>>>
>>>>
>>>> ** cell: **+94 (77) 678-0813*
>>>> *linked-in: *http://lk.linkedin.com/in/gajasinghe
>>>>
>>>>
>>>> *
>>>> *
>>>> *blog: **http://kasunbg.org* <http://kasunbg.org>
>>>>
>>>>
>>>> *
>>>> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
>>>>
>>>>
>>>> *
>>>> *
>>>>
>>>
>>>
>>>
>>> --
>>> Ishara Premasada
>>> Software Engineer,
>>> WSO2 Inc. http://wso2.com/
>>> *Blog : http://isharapremadasa.blogspot.com/
>>> Twitter : https://twitter.com/ishadil
>>> Mobile : +94 714445832*
>>>
>>>
>>>
>>
>>
>> --
>> *Kasun Gajasinghe*
>> Software Engineer;
>> Development Technologies Team, WSO2 Inc.; http://wso2.com
>>
>>
>> ,
>> *email: **kasung AT spamfree wso2.com
>>
>>
>> ** cell: **+94 (77) 678-0813*
>> *linked-in: *http://lk.linkedin.com/in/gajasinghe
>>
>>
>> *
>> *
>> *blog: **http://kasunbg.org* <http://kasunbg.org>
>>
>>
>> *
>> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
>>
>>
>> *
>> *
>>
>
>
>
> --
> Ishara Premasada
> Software Engineer,
> WSO2 Inc. http://wso2.com/
> *Blog : http://isharapremadasa.blogspot.com/
> Twitter : https://twitter.com/ishadil
> Mobile : +94 714445832*
>
>
>
--
Best Regards,*
Shammi Jayasinghe*
Associate Tech Lead
WSO2, Inc.; http://wso2.com,
mobile: +94 71 4493085
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
