On Mon, Oct 6, 2014 at 9:35 AM, Bhathiya Jayasekara <bhath...@wso2.com>
wrote:
> Hi Sajini,
>
> Can you please point which part of above configuration fixed the issue?
>
+1. It would also be helpful if you could explain what really caused the
above reported exceptions too.
Cheers,
Prabath
>
> Thanks,
> Bhathiya
>
> On Mon, Oct 6, 2014 at 9:30 AM, Sajini De Silva <saj...@wso2.com> wrote:
>
>> Hi,
>>
>> This issue was fixed with following configurations.
>>
>> *master-datasources.xml in MB node:*
>>
>> <!-- external Cassandra data source. -->
>> <datasource>
>> <name>WSO2_CASSANDRA_DB</name>
>> <description>The datasource used for cassandra</description>
>> <jndiConfig>
>> <name>CassandraRepo</name>
>> </jndiConfig>
>> <definition type="CASSANDRA">
>> <configuration>
>> <async>false</async>
>> <clusterName>TestCluster</clusterName>
>> <compression>SNAPPY</compression>
>> <concurrency>100</concurrency>
>> <username>admin</username>
>> <password encrypted="true">admin</password>
>> <port>9042</port>
>> <maxConnections>100</maxConnections>
>>
>> <hosts>
>> <host>127.0.0.1</host>
>> </hosts>
>> <loadBalancePolicy>
>> <exclusionThreshold>2.5</exclusionThreshold>
>> <latencyAware>true</latencyAware>
>> <minMeasure>100</minMeasure>
>> <policyName>RoundRobinPolicy</policyName>
>> <retryPeriod>10</retryPeriod>
>> <scale>2</scale>
>> </loadBalancePolicy>
>>
>> <poolOptions>
>>
>> <coreConnectionsForLocal>1000</coreConnectionsForLocal>
>>
>> <coreConnectionsForRemote>1000</coreConnectionsForRemote>
>>
>> <maxConnectionsForLocal>1000</maxConnectionsForLocal>
>>
>> <maxConnectionsForRemote>1000</maxConnectionsForRemote>
>>
>> <maxSimultaneousRequestsForLocal>1000</maxSimultaneousRequestsForLocal>
>>
>> <maxSimultaneousRequestsForRemote>1000</maxSimultaneousRequestsForRemote>
>>
>> <minSimultaneousRequestsForLocal>1000</minSimultaneousRequestsForLocal>
>>
>> <minSimultaneousRequestsForRemote>1000</minSimultaneousRequestsForRemote>
>> </poolOptions>
>>
>> <reconnectPolicy>
>> <baseDelayMs>3000</baseDelayMs>
>>
>> <policyName>ConstantReconnectionPolicy</policyName>
>> </reconnectPolicy>
>> <socketOptions>
>> <connectTimeoutMillis>10000</connectTimeoutMillis>
>> <keepAlive>true</keepAlive>
>> <readTimeoutMillis>15000</readTimeoutMillis>
>> <tcpNoDelay>true</tcpNoDelay>
>> </socketOptions>
>>
>> </configuration>
>>
>> </definition>
>> </datasource>
>>
>>
>>
>> *cassandra-env.sh in cassandra node:*
>>
>> system_memory_in_mb="10240"
>> system_cpu_cores="6"
>>
>> Tested with an SSD hard drive.
>>
>> Thank you,
>> Sajini.
>>
>> On Fri, Oct 3, 2014 at 12:23 PM, Sajini De Silva <saj...@wso2.com> wrote:
>>
>>> Hi Prabath,
>>>
>>> This is the error we got while deleting messages.
>>>
>>> [2014-10-03 12:16:55,059] ERROR
>>> {org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl} - Error while
>>> deleting messages
>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All
>>> host(s) tried for query failed (tried: /127.0.0.1 (Timeout during read))
>>> at
>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
>>> at
>>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269)
>>> at
>>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183)
>>> at com.datastax.driver.core.Session.execute(Session.java:111)
>>> at
>>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207)
>>> at
>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299)
>>> at
>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327)
>>> at
>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203)
>>> at
>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30)
>>> at
>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19)
>>> at
>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
>>> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
>>> read))
>>> at
>>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106)
>>> at
>>> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177)
>>> ... 3 more
>>> Oct 3, 2014 12:16:55 PM com.lmax.disruptor.FatalExceptionHandler
>>> handleEventException
>>> SEVERE: Exception processing: 12095
>>> org.wso2.andes.kernel.AndesAckData@5341cc77
>>> org.wso2.andes.kernel.AndesException:
>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
>>> tried for query failed (tried: /127.0.0.1 (Timeout during read))
>>> at
>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:303)
>>> at
>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327)
>>> at
>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203)
>>> at
>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30)
>>> at
>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19)
>>> at
>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
>>> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
>>> read))
>>> at
>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
>>> at
>>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269)
>>> at
>>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183)
>>> at com.datastax.driver.core.Session.execute(Session.java:111)
>>> at
>>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207)
>>> at
>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299)
>>> ... 8 more
>>> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
>>> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
>>> read))
>>> at
>>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106)
>>> at
>>> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177)
>>> ... 3 more
>>> Exception in thread "pool-25-thread-1" java.lang.RuntimeException:
>>> org.wso2.andes.kernel.AndesException:
>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
>>> tried for query failed (tried: /127.0.0.1 (Timeout during read))
>>> at
>>> com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
>>> at
>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:662)
>>> Caused by: org.wso2.andes.kernel.AndesException:
>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
>>> tried for query failed (tried: /127.0.0.1 (Timeout during read))
>>> at
>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:303)
>>> at
>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327)
>>> at
>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203)
>>> at
>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30)
>>> at
>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19)
>>> at
>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
>>> ... 3 more
>>> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
>>> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
>>> read))
>>> at
>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
>>> at
>>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269)
>>> at
>>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183)
>>> at com.datastax.driver.core.Session.execute(Session.java:111)
>>> at
>>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207)
>>> at
>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299)
>>> ... 8 more
>>> Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException:
>>> All host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
>>> read))
>>> at
>>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106)
>>> at
>>> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177)
>>> ... 3 more
>>>
>>>
>>> We tried this scenario increasing the number of threads in thread pool
>>> to 1000, but same error occurred.
>>>
>>> Thank you,
>>> Sajini.
>>>
>>>
>>> On Wed, Oct 1, 2014 at 5:22 PM, Sajini De Silva <saj...@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm getting following exception in MB when doing load testing.
>>>>
>>>> SEVERE: Exception processing: 81
>>>> org.wso2.andes.kernel.distrupter.CassandraDataEvent@2d524d15
>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All
>>>> host(s) tried for query failed (tried: /127.0.0.1 (Timeout during
>>>> read))
>>>> at
>>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
>>>> at
>>>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269)
>>>> at
>>>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183)
>>>> at com.datastax.driver.core.Session.execute(Session.java:111)
>>>> at
>>>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207)
>>>> at
>>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.storeMessagePart(CQLBasedMessageStoreImpl.java:121)
>>>> at
>>>> org.wso2.andes.kernel.distrupter.AlternatingCassandraWriter.onEvent(AlternatingCassandraWriter.java:81)
>>>> at
>>>> org.wso2.andes.kernel.distrupter.AlternatingCassandraWriter.onEvent(AlternatingCassandraWriter.java:38)
>>>> at
>>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> Does anyone knows the cause for this error?
>>>>
>>>> --
>>>> Sajini De SIlva
>>>> Software Engineer; WSO2 Inc.; http://wso2.com ,
>>>> Email: saj...@wso2.com
>>>> Blog: http://sajinid.blogspot.com/
>>>> Git hub profile: https://github.com/sajinidesilva
>>>>
>>>> Phone: +94 712797729
>>>>
>>>>
>>>
>>>
>>> --
>>> Sajini De SIlva
>>> Software Engineer; WSO2 Inc.; http://wso2.com ,
>>> Email: saj...@wso2.com
>>> Blog: http://sajinid.blogspot.com/
>>> Git hub profile: https://github.com/sajinidesilva
>>>
>>> Phone: +94 712797729
>>>
>>>
>>
>>
>> --
>> Sajini De SIlva
>> Software Engineer; WSO2 Inc.; http://wso2.com ,
>> Email: saj...@wso2.com
>> Blog: http://sajinid.blogspot.com/
>> Git hub profile: https://github.com/sajinidesilva
>>
>> Phone: +94 712797729
>>
>>
>> _______________________________________________
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> *Bhathiya Jayasekara*
> *Software Engineer,*
> *WSO2 inc., http://wso2.com <http://wso2.com>*
>
> *Phone: +94715478185 <%2B94715478185>*
> *LinkedIn: http://www.linkedin.com/in/bhathiyaj
> <http://www.linkedin.com/in/bhathiyaj>*
> *Twitter: https://twitter.com/bhathiyax <https://twitter.com/bhathiyax>*
> *Blog: http://movingaheadblog.blogspot.com
> <http://movingaheadblog.blogspot.com/>*
>
> _______________________________________________
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>
--
Prabath Abeysekara
Associate Technical Lead, Data TG.
WSO2 Inc.
Email: praba...@wso2.com
Mobile: +94774171471
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
