Hi Sajini,
As we discussed some time back, what we did was increased values of below
configurations. Apart from that we increased poolOptions but I don't think
that helped that much.
<reconnectPolicy>
* <baseDelayMs>100</baseDelayMs>*
<policyName>ConstantReconnectionPolicy</policyName>
</reconnectPolicy>
<socketOptions>
* <connectTimeoutMillis>200</connectTimeoutMillis>*
<keepAlive>true</keepAlive>
*<readTimeoutMillis>200</readTimeoutMillis>*
<tcpNoDelay>true</tcpNoDelay>
</socketOptions>
I have bold the important configurations which affect on load handling.
Cheers,
Dhanuka
*Dhanuka Ranasinghe*
Senior Software Engineer
WSO2 Inc. ; http://wso2.com
lean . enterprise . middleware
phone : +94 715381915
On Thu, Oct 9, 2014 at 1:37 AM, Sajini De Silva <saj...@wso2.com> wrote:
> Hi,
>
> I increased the values in poolOptions and socketOptions in the
> configuration. The cause for this exception IMO was setting a low value to
> connectTimeoutMillis option.
>
> @Asitha
>
> Will verify this after doing more tests on this.
>
> Thank you,
> Sajini.
>
> On Mon, Oct 6, 2014 at 11:36 AM, Asitha Nanayakkara <asi...@wso2.com>
> wrote:
>
>> Hi Sajini,
>>
>> Can we take this data source configuration as the default configuration
>> for MB? If so we need to update the master-datasources.xml in the pack [1]
>> I guess.
>>
>> [1]
>> https://github.com/wso2-dev/product-mb/blob/master/modules/distribution/src/main/conf/master-datasources.xml
>>
>> Thanks
>>
>> On Mon, Oct 6, 2014 at 9:35 AM, Bhathiya Jayasekara <bhath...@wso2.com>
>> wrote:
>>
>>> Hi Sajini,
>>>
>>> Can you please point which part of above configuration fixed the issue?
>>>
>>> Thanks,
>>> Bhathiya
>>>
>>> On Mon, Oct 6, 2014 at 9:30 AM, Sajini De Silva <saj...@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> This issue was fixed with following configurations.
>>>>
>>>> *master-datasources.xml in MB node:*
>>>>
>>>> <!-- external Cassandra data source. -->
>>>> <datasource>
>>>> <name>WSO2_CASSANDRA_DB</name>
>>>> <description>The datasource used for cassandra</description>
>>>> <jndiConfig>
>>>> <name>CassandraRepo</name>
>>>> </jndiConfig>
>>>> <definition type="CASSANDRA">
>>>> <configuration>
>>>> <async>false</async>
>>>> <clusterName>TestCluster</clusterName>
>>>> <compression>SNAPPY</compression>
>>>> <concurrency>100</concurrency>
>>>> <username>admin</username>
>>>> <password encrypted="true">admin</password>
>>>> <port>9042</port>
>>>> <maxConnections>100</maxConnections>
>>>>
>>>> <hosts>
>>>> <host>127.0.0.1</host>
>>>> </hosts>
>>>> <loadBalancePolicy>
>>>> <exclusionThreshold>2.5</exclusionThreshold>
>>>> <latencyAware>true</latencyAware>
>>>> <minMeasure>100</minMeasure>
>>>> <policyName>RoundRobinPolicy</policyName>
>>>> <retryPeriod>10</retryPeriod>
>>>> <scale>2</scale>
>>>> </loadBalancePolicy>
>>>>
>>>> <poolOptions>
>>>>
>>>> <coreConnectionsForLocal>1000</coreConnectionsForLocal>
>>>>
>>>> <coreConnectionsForRemote>1000</coreConnectionsForRemote>
>>>>
>>>> <maxConnectionsForLocal>1000</maxConnectionsForLocal>
>>>>
>>>> <maxConnectionsForRemote>1000</maxConnectionsForRemote>
>>>>
>>>> <maxSimultaneousRequestsForLocal>1000</maxSimultaneousRequestsForLocal>
>>>>
>>>> <maxSimultaneousRequestsForRemote>1000</maxSimultaneousRequestsForRemote>
>>>>
>>>> <minSimultaneousRequestsForLocal>1000</minSimultaneousRequestsForLocal>
>>>>
>>>> <minSimultaneousRequestsForRemote>1000</minSimultaneousRequestsForRemote>
>>>> </poolOptions>
>>>>
>>>> <reconnectPolicy>
>>>> <baseDelayMs>3000</baseDelayMs>
>>>>
>>>> <policyName>ConstantReconnectionPolicy</policyName>
>>>> </reconnectPolicy>
>>>> <socketOptions>
>>>>
>>>> <connectTimeoutMillis>10000</connectTimeoutMillis>
>>>> <keepAlive>true</keepAlive>
>>>> <readTimeoutMillis>15000</readTimeoutMillis>
>>>> <tcpNoDelay>true</tcpNoDelay>
>>>> </socketOptions>
>>>>
>>>> </configuration>
>>>>
>>>> </definition>
>>>> </datasource>
>>>>
>>>>
>>>>
>>>> *cassandra-env.sh in cassandra node:*
>>>>
>>>> system_memory_in_mb="10240"
>>>> system_cpu_cores="6"
>>>>
>>>> Tested with an SSD hard drive.
>>>>
>>>> Thank you,
>>>> Sajini.
>>>>
>>>> On Fri, Oct 3, 2014 at 12:23 PM, Sajini De Silva <saj...@wso2.com>
>>>> wrote:
>>>>
>>>>> Hi Prabath,
>>>>>
>>>>> This is the error we got while deleting messages.
>>>>>
>>>>> [2014-10-03 12:16:55,059] ERROR
>>>>> {org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl} - Error while
>>>>> deleting messages
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All
>>>>> host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
>>>>> read))
>>>>> at
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
>>>>> at
>>>>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269)
>>>>> at
>>>>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183)
>>>>> at com.datastax.driver.core.Session.execute(Session.java:111)
>>>>> at
>>>>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207)
>>>>> at
>>>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299)
>>>>> at
>>>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327)
>>>>> at
>>>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203)
>>>>> at
>>>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30)
>>>>> at
>>>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19)
>>>>> at
>>>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by:
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
>>>>> tried for query failed (tried: /127.0.0.1 (Timeout during read))
>>>>> at
>>>>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106)
>>>>> at
>>>>> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177)
>>>>> ... 3 more
>>>>> Oct 3, 2014 12:16:55 PM com.lmax.disruptor.FatalExceptionHandler
>>>>> handleEventException
>>>>> SEVERE: Exception processing: 12095
>>>>> org.wso2.andes.kernel.AndesAckData@5341cc77
>>>>> org.wso2.andes.kernel.AndesException:
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
>>>>> tried for query failed (tried: /127.0.0.1 (Timeout during read))
>>>>> at
>>>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:303)
>>>>> at
>>>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327)
>>>>> at
>>>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203)
>>>>> at
>>>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30)
>>>>> at
>>>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19)
>>>>> at
>>>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by:
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
>>>>> tried for query failed (tried: /127.0.0.1 (Timeout during read))
>>>>> at
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
>>>>> at
>>>>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269)
>>>>> at
>>>>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183)
>>>>> at com.datastax.driver.core.Session.execute(Session.java:111)
>>>>> at
>>>>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207)
>>>>> at
>>>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299)
>>>>> ... 8 more
>>>>> Caused by:
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
>>>>> tried for query failed (tried: /127.0.0.1 (Timeout during read))
>>>>> at
>>>>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106)
>>>>> at
>>>>> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177)
>>>>> ... 3 more
>>>>> Exception in thread "pool-25-thread-1" java.lang.RuntimeException:
>>>>> org.wso2.andes.kernel.AndesException:
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
>>>>> tried for query failed (tried: /127.0.0.1 (Timeout during read))
>>>>> at
>>>>> com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
>>>>> at
>>>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: org.wso2.andes.kernel.AndesException:
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
>>>>> tried for query failed (tried: /127.0.0.1 (Timeout during read))
>>>>> at
>>>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:303)
>>>>> at
>>>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327)
>>>>> at
>>>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203)
>>>>> at
>>>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30)
>>>>> at
>>>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19)
>>>>> at
>>>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
>>>>> ... 3 more
>>>>> Caused by:
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
>>>>> tried for query failed (tried: /127.0.0.1 (Timeout during read))
>>>>> at
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
>>>>> at
>>>>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269)
>>>>> at
>>>>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183)
>>>>> at com.datastax.driver.core.Session.execute(Session.java:111)
>>>>> at
>>>>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207)
>>>>> at
>>>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299)
>>>>> ... 8 more
>>>>> Caused by:
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
>>>>> tried for query failed (tried: /127.0.0.1 (Timeout during read))
>>>>> at
>>>>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106)
>>>>> at
>>>>> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177)
>>>>> ... 3 more
>>>>>
>>>>>
>>>>> We tried this scenario increasing the number of threads in thread pool
>>>>> to 1000, but same error occurred.
>>>>>
>>>>> Thank you,
>>>>> Sajini.
>>>>>
>>>>>
>>>>> On Wed, Oct 1, 2014 at 5:22 PM, Sajini De Silva <saj...@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm getting following exception in MB when doing load testing.
>>>>>>
>>>>>> SEVERE: Exception processing: 81
>>>>>> org.wso2.andes.kernel.distrupter.CassandraDataEvent@2d524d15
>>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All
>>>>>> host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
>>>>>> read))
>>>>>> at
>>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
>>>>>> at
>>>>>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269)
>>>>>> at
>>>>>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183)
>>>>>> at com.datastax.driver.core.Session.execute(Session.java:111)
>>>>>> at
>>>>>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207)
>>>>>> at
>>>>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.storeMessagePart(CQLBasedMessageStoreImpl.java:121)
>>>>>> at
>>>>>> org.wso2.andes.kernel.distrupter.AlternatingCassandraWriter.onEvent(AlternatingCassandraWriter.java:81)
>>>>>> at
>>>>>> org.wso2.andes.kernel.distrupter.AlternatingCassandraWriter.onEvent(AlternatingCassandraWriter.java:38)
>>>>>> at
>>>>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>
>>>>>> Does anyone knows the cause for this error?
>>>>>>
>>>>>> --
>>>>>> Sajini De SIlva
>>>>>> Software Engineer; WSO2 Inc.; http://wso2.com ,
>>>>>> Email: saj...@wso2.com
>>>>>> Blog: http://sajinid.blogspot.com/
>>>>>> Git hub profile: https://github.com/sajinidesilva
>>>>>>
>>>>>> Phone: +94 712797729
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sajini De SIlva
>>>>> Software Engineer; WSO2 Inc.; http://wso2.com ,
>>>>> Email: saj...@wso2.com
>>>>> Blog: http://sajinid.blogspot.com/
>>>>> Git hub profile: https://github.com/sajinidesilva
>>>>>
>>>>> Phone: +94 712797729
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Sajini De SIlva
>>>> Software Engineer; WSO2 Inc.; http://wso2.com ,
>>>> Email: saj...@wso2.com
>>>> Blog: http://sajinid.blogspot.com/
>>>> Git hub profile: https://github.com/sajinidesilva
>>>>
>>>> Phone: +94 712797729
>>>>
>>>>
>>>> _______________________________________________
>>>> Dev mailing list
>>>> Dev@wso2.org
>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>
>>>>
>>>
>>>
>>> --
>>> *Bhathiya Jayasekara*
>>> *Software Engineer,*
>>> *WSO2 inc., http://wso2.com <http://wso2.com>*
>>>
>>> *Phone: +94715478185 <%2B94715478185>*
>>> *LinkedIn: http://www.linkedin.com/in/bhathiyaj
>>> <http://www.linkedin.com/in/bhathiyaj>*
>>> *Twitter: https://twitter.com/bhathiyax <https://twitter.com/bhathiyax>*
>>> *Blog: http://movingaheadblog.blogspot.com
>>> <http://movingaheadblog.blogspot.com/>*
>>>
>>> _______________________________________________
>>> Dev mailing list
>>> Dev@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>> *Asitha Nanayakkara*
>> Software Engineer
>> WSO2, Inc. http://wso2.com/
>> Mob: + 94 77 85 30 682
>>
>>
>
>
> --
> Sajini De SIlva
> Software Engineer; WSO2 Inc.; http://wso2.com ,
> Email: saj...@wso2.com
> Blog: http://sajinid.blogspot.com/
> Git hub profile: https://github.com/sajinidesilva
>
> Phone: +94 712797729
>
>
> _______________________________________________
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
