On Mon, Aug 19, 2013 at 11:29 PM, Kasun Gajasinghe <kas...@wso2.com> wrote:
>
>
>> However the only incident i have seen a rapid error log thrown
>> continuosly in the MB or ActiveMQ integration scenario is, when you start
>> AS for the first time, if you hadn't start the broker (MB/ActiveMQ etc.)
>> before starting AS it will throw a continuous error log as there is no
>> JMSConnection available. Apart from that if the broker fails in middle,
>> after successful start up, the reconnection should happen fine.
>>
>>
>>
> Yes... This is what I was doing. We should handle this scenario too,
> right?
>
@Shammi/Malinga
WDYT? Should we provide some notification and stop the server start if user
forgets to start JMS provider (broker) in prior to start AS, ESB etc.? IMO
as our documentation clearly says that it is needed to start the broker
first in each integrating scenarios, i am not sure whether it is needed
to handle this.
>
>>
>>
>> On Mon, Aug 19, 2013 at 4:44 PM, Kasun Gajasinghe <kas...@wso2.com>wrote:
>>
>>> Hi Ishara,
>>>
>>> Please note that I tried this with ActiveMQ. I didn't see the
>>> reconnection attempts in the log rather a rapid generation of logs. The log
>>> file grew to 100MB in less than a minute.
>>>
>>>
>>> On Mon, Aug 19, 2013 at 4:21 PM, Ishara Premadasa <ish...@wso2.com>wrote:
>>>
>>>> Hi Kasun,
>>>>
>>>> I first tried the JMS transport reconnection mechanism with AS 5.1.0
>>>> and then i tried with the newly built AS 5.2.0 pack. However with both
>>>> these packs the reconnection works fine where it waits for the
>>>> JMSConnection to come alive by retrying in exponentially increasing time
>>>> intervals. Please see the AS console log i get in the this situation below.
>>>>
>>>> [2013-08-19 15:28:02,454] INFO {org.wso2.andes.client.AMQConnection}
>>>> - Unable to connect to broker at tcp://localhost:5673
>>>> org.wso2.andes.transport.TransportException: Could not open connection
>>>> at
>>>> org.wso2.andes.transport.network.mina.MinaNetworkTransport$IoConnectorCreator.connect(MinaNetworkTransport.java:201)
>>>> at
>>>> org.wso2.andes.transport.network.mina.MinaNetworkTransport.connect(MinaNetworkTransport.java:71)
>>>> at
>>>> org.wso2.andes.client.AMQConnectionDelegate_8_0.makeBrokerConnection(AMQConnectionDelegate_8_0.java:110)
>>>> at
>>>> org.wso2.andes.client.AMQConnection.makeBrokerConnection(AMQConnection.java:616)
>>>> at
>>>> org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:398)
>>>> at
>>>> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:315)
>>>> at
>>>> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:38)
>>>>
>>>> at
>>>> org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
>>>> at
>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:824)
>>>> at
>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.getConnection(ServiceTaskManager.java:706)
>>>> at
>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.receiveMessage(ServiceTaskManager.java:496)
>>>> at
>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:420)
>>>> at
>>>> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> [2013-08-19 15:28:02,454] ERROR
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Error acquiring a
>>>> JMS connection to : QueueConnectionFactory using JNDI properties :
>>>> {java.naming.factory.initial=org.wso2.andes.jndi.PropertiesFileInitialContextFactory,
>>>> java.naming.provider.url=repository/conf/jndi.properties,
>>>> transport.jms.ConnectionFactoryType=queue,
>>>> ServiceClass=org.wso2.carbon.mashup.javascript.stubgenerator.JSStubGenerator
>>>> }
>>>> javax.jms.JMSException: Error creating connection: Could not open
>>>> connection
>>>> at
>>>> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:325)
>>>> at
>>>> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:38)
>>>>
>>>> at
>>>> org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
>>>> at
>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:824)
>>>> at
>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.getConnection(ServiceTaskManager.java:706)
>>>> at
>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.receiveMessage(ServiceTaskManager.java:496)
>>>> at
>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:420)
>>>> at
>>>> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> Caused by: org.wso2.andes.AMQConnectionFailureException: Could not open
>>>> connection
>>>> at
>>>> org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:475)
>>>> at
>>>> org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:315)
>>>> ... 10 more
>>>> Caused by: org.wso2.andes.transport.TransportException: Could not open
>>>> connection
>>>> at
>>>> org.wso2.andes.transport.network.mina.MinaNetworkTransport$IoConnectorCreator.connect(MinaNetworkTransport.java:201)
>>>> at
>>>> org.wso2.andes.transport.network.mina.MinaNetworkTransport.connect(MinaNetworkTransport.java:71)
>>>> at
>>>> org.wso2.andes.client.AMQConnectionDelegate_8_0.makeBrokerConnection(AMQConnectionDelegate_8_0.java:110)
>>>> at
>>>> org.wso2.andes.client.AMQConnection.makeBrokerConnection(AMQConnection.java:616)
>>>> at
>>>> org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:398)
>>>> ... 11 more
>>>> [2013-08-19 15:28:07,451] ERROR
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection attempt
>>>> : 2 for service : HelloService failed. Next retry in 40 seconds
>>>> [2013-08-19 15:28:07,451] ERROR
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection attempt
>>>> : 2 for service : JavaScriptStubGeneratorService failed. Next retry in 40
>>>> seconds
>>>> [2013-08-19 15:28:47,452] INFO
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection attempt
>>>> : 3 for service : HelloService
>>>> [2013-08-19 15:28:47,452] INFO
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection attempt
>>>> : 3 for service : JavaScriptStubGeneratorService
>>>> [2013-08-19 15:28:52,453] WARN
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Unable to shutdown
>>>> all polling tasks of service : HelloService
>>>> [2013-08-19 15:28:52,453] WARN
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Unable to shutdown
>>>> all polling tasks of service : JavaScriptStubGeneratorService
>>>> [2013-08-19 15:28:52,453] INFO
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
>>>> service : HelloService shutdown
>>>> [2013-08-19 15:28:52,454] INFO
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
>>>> service : JavaScriptStubGeneratorService shutdown
>>>> [2013-08-19 15:28:52,454] INFO
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
>>>> service : HelloService [re-]initialized
>>>> [2013-08-19 15:28:52,454] INFO
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Task manager for
>>>> service : JavaScriptStubGeneratorService [re-]initialized
>>>> [2013-08-19 15:28:53,455] INFO
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection
>>>> attempt: 3 for service: HelloService was successful!
>>>> [2013-08-19 15:28:53,455] INFO
>>>> {org.apache.axis2.transport.jms.ServiceTaskManager} - Reconnection
>>>> attempt: 3 for service: JavaScriptStubGeneratorService was successful!
>>>>
>>>> Therefore the exponential reconnection attempts already work with AS
>>>> 5.2.0 as i observed. Please let us know if this is not the error you see in
>>>> the console when MB server is down and AS is trying to reconnect.
>>>>
>>>> Thanks!
>>>> Ishara
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 19, 2013 at 10:15 AM, Kasun Gajasinghe <kas...@wso2.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> The error log that repeats is the following. The error logs stop when
>>>>> the activemq server is started.
>>>>>
>>>>> TID: [0] [AS] [2013-08-19 10:12:43,835] ERROR
>>>>> {org.apache.axis2.transport.base.threads.NativeWorkerPool} - Uncaught
>>>>> exception {org.apache.axis2.transport.base.threads.NativeWorkerPool}
>>>>> org.apache.axis2.transport.jms.AxisJMSException: Error acquiring a JMS
>>>>> connection to : QueueConnectionFactory using JNDI properties :
>>>>> {java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory,
>>>>> org.wso2.xkms2.service.crypto.persistence.enabled=true,
>>>>> org.wso2.xkms2.service.crypto.keystore.location=keystore.jks,
>>>>> org.wso2.xkms2.service.crypto.server.key.password=password,
>>>>> org.wso2.xkms2.service.crypto.keystore.password=password,
>>>>> org.wso2.xkms2.service.crypto.default.expriy.interval=365,
>>>>> org.wso2.xkms2.service.crypto.default.private.key.password=password,
>>>>> org.wso2.xkms2.service.crypto.server.cert.aliase=bob,
>>>>> org.wso2.xkms2.service.crypto.issuer.cert.aliase=alice,
>>>>> org.wso2.xkms2.service.crypto.issuer.key.password=password,
>>>>> java.naming.provider.url=tcp://localhost:61616,
>>>>> org.wso2.xkms2.service.crypto.authen.code=secret}
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager.handleException(ServiceTaskManager.java:1000)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager.access$700(ServiceTaskManager.java:50)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:835)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.getConnection(ServiceTaskManager.java:706)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.receiveMessage(ServiceTaskManager.java:496)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.run(ServiceTaskManager.java:420)
>>>>> at
>>>>> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: javax.jms.JMSException: Could not connect to broker URL:
>>>>> tcp://localhost:61616. Reason: java.net.ConnectException: Connection
>>>>> refused
>>>>> at
>>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
>>>>> at
>>>>> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:280)
>>>>> at
>>>>> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:224)
>>>>> at
>>>>> org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:172)
>>>>> at
>>>>> org.apache.axis2.transport.jms.JMSUtils.createConnection(JMSUtils.java:629)
>>>>> at
>>>>> org.apache.axis2.transport.jms.ServiceTaskManager$MessageListenerTask.createConnection(ServiceTaskManager.java:824)
>>>>> ... 7 more
>>>>> Caused by: java.net.ConnectException: Connection refused
>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>>>>> at
>>>>> java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
>>>>> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>>>>> at java.net.Socket.connect(Socket.java:529)
>>>>> at
>>>>> org.apache.activemq.transport.tcp.TcpTransport.connect(TcpTransport.java:435)
>>>>> at
>>>>> org.apache.activemq.transport.tcp.TcpTransport.doStart(TcpTransport.java:401)
>>>>> at
>>>>> org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:53)
>>>>> at
>>>>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>>>>> at
>>>>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>>>>> at
>>>>> org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:72)
>>>>> at
>>>>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>>>>> at
>>>>> org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:57)
>>>>> at
>>>>> org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:260)
>>>>> ... 11 more
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Aug 19, 2013 at 5:38 AM, Ishara Premadasa <ish...@wso2.com>wrote:
>>>>>
>>>>>> Hi Kasun,
>>>>>>
>>>>>> We will be looking into this today and update.
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> On Sun, Aug 18, 2013 at 12:14 PM, Kasun Gajasinghe
>>>>>> <kas...@wso2.com>wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> It seems that AS 5.2.0 is not patched with the issue with JMS
>>>>>>> transport where the services with JMS enabled repeatedly try to
>>>>>>> establish
>>>>>>> a connection with the message broker. This results in GBs of logs in a
>>>>>>> matter of minutes filling up the disk.
>>>>>>>
>>>>>>> As I got to know from Shammi, the solution is to exponentially
>>>>>>> increase the waiting time for the connection. Something like, trying in
>>>>>>> 1sec, 2sec, 4sec, 8, 16, 32 etc. This fix is already impelemented but it
>>>>>>> seems that it didn't get in to AS packs that is about to be released. I
>>>>>>> tried a setup with ActiveMQ, and I can see that it tries to establish a
>>>>>>> connection with the broker every second or so.
>>>>>>>
>>>>>>> Can the MB team look in to this ASAP please? Carbon 4.2.0 is due on
>>>>>>> coming Thursday.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> KasunG
>>>>>>>
>>>>>>> --
>>>>>>> *Kasun Gajasinghe*
>>>>>>> Software Engineer;
>>>>>>> Development Technologies Team, WSO2 Inc.; http://wso2.com
>>>>>>>
>>>>>>>
>>>>>>> ,
>>>>>>> *email: **kasung AT spamfree wso2.com
>>>>>>>
>>>>>>>
>>>>>>> ** cell: **+94 (77) 678-0813*
>>>>>>> *linked-in: *http://lk.linkedin.com/in/gajasinghe
>>>>>>>
>>>>>>>
>>>>>>> *
>>>>>>> *
>>>>>>> *blog: **http://kasunbg.org* <http://kasunbg.org>
>>>>>>>
>>>>>>>
>>>>>>> *
>>>>>>> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
>>>>>>>
>>>>>>>
>>>>>>> *
>>>>>>> *
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Ishara Premasada
>>>>>> Software Engineer,
>>>>>> WSO2 Inc. http://wso2.com/
>>>>>> *Blog : http://isharapremadasa.blogspot.com/
>>>>>> Twitter : https://twitter.com/ishadil
>>>>>> Mobile : +94 714445832*
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Kasun Gajasinghe*
>>>>> Software Engineer;
>>>>> Development Technologies Team, WSO2 Inc.; http://wso2.com
>>>>>
>>>>>
>>>>> ,
>>>>> *email: **kasung AT spamfree wso2.com
>>>>>
>>>>>
>>>>> ** cell: **+94 (77) 678-0813*
>>>>> *linked-in: *http://lk.linkedin.com/in/gajasinghe
>>>>>
>>>>>
>>>>> *
>>>>> *
>>>>> *blog: **http://kasunbg.org* <http://kasunbg.org>
>>>>>
>>>>>
>>>>> *
>>>>> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
>>>>>
>>>>>
>>>>> *
>>>>> *
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ishara Premasada
>>>> Software Engineer,
>>>> WSO2 Inc. http://wso2.com/
>>>> *Blog : http://isharapremadasa.blogspot.com/
>>>> Twitter : https://twitter.com/ishadil
>>>> Mobile : +94 714445832*
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Kasun Gajasinghe*
>>> Software Engineer;
>>> Development Technologies Team, WSO2 Inc.; http://wso2.com
>>>
>>>
>>> ,
>>> *email: **kasung AT spamfree wso2.com
>>>
>>>
>>> ** cell: **+94 (77) 678-0813*
>>> *linked-in: *http://lk.linkedin.com/in/gajasinghe
>>>
>>>
>>> *
>>> *
>>> *blog: **http://kasunbg.org* <http://kasunbg.org>
>>>
>>>
>>> *
>>> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
>>>
>>>
>>> *
>>> *
>>>
>>
>>
>>
>> --
>> Ishara Premasada
>> Software Engineer,
>> WSO2 Inc. http://wso2.com/
>> *Blog : http://isharapremadasa.blogspot.com/
>> Twitter : https://twitter.com/ishadil
>> Mobile : +94 714445832*
>>
>>
>>
>
>
> --
> *Kasun Gajasinghe*
> Software Engineer;
> Development Technologies Team, WSO2 Inc.; http://wso2.com
>
>
> ,
> *email: **kasung AT spamfree wso2.com
>
>
> ** cell: **+94 (77) 678-0813*
> *linked-in: *http://lk.linkedin.com/in/gajasinghe
>
>
> *
> *
> *blog: **http://kasunbg.org* <http://kasunbg.org>
>
>
> *
> twitter: **http://twitter.com/kasunbg* <http://twitter.com/kasunbg>
>
>
> *
> *
>
--
Ishara Premasada
Software Engineer,
WSO2 Inc. http://wso2.com/
*Blog : http://isharapremadasa.blogspot.com/
Twitter : https://twitter.com/ishadil
Mobile : +94 714445832*
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
