Hi, I increased the values in poolOptions and socketOptions in the configuration. The cause for this exception IMO was setting a low value to connectTimeoutMillis option.
@Asitha Will verify this after doing more tests on this. Thank you, Sajini. On Mon, Oct 6, 2014 at 11:36 AM, Asitha Nanayakkara <asi...@wso2.com> wrote: > Hi Sajini, > > Can we take this data source configuration as the default configuration > for MB? If so we need to update the master-datasources.xml in the pack [1] > I guess. > > [1] > https://github.com/wso2-dev/product-mb/blob/master/modules/distribution/src/main/conf/master-datasources.xml > > Thanks > > On Mon, Oct 6, 2014 at 9:35 AM, Bhathiya Jayasekara <bhath...@wso2.com> > wrote: > >> Hi Sajini, >> >> Can you please point which part of above configuration fixed the issue? >> >> Thanks, >> Bhathiya >> >> On Mon, Oct 6, 2014 at 9:30 AM, Sajini De Silva <saj...@wso2.com> wrote: >> >>> Hi, >>> >>> This issue was fixed with following configurations. >>> >>> *master-datasources.xml in MB node:* >>> >>> <!-- external Cassandra data source. --> >>> <datasource> >>> <name>WSO2_CASSANDRA_DB</name> >>> <description>The datasource used for cassandra</description> >>> <jndiConfig> >>> <name>CassandraRepo</name> >>> </jndiConfig> >>> <definition type="CASSANDRA"> >>> <configuration> >>> <async>false</async> >>> <clusterName>TestCluster</clusterName> >>> <compression>SNAPPY</compression> >>> <concurrency>100</concurrency> >>> <username>admin</username> >>> <password encrypted="true">admin</password> >>> <port>9042</port> >>> <maxConnections>100</maxConnections> >>> >>> <hosts> >>> <host>127.0.0.1</host> >>> </hosts> >>> <loadBalancePolicy> >>> <exclusionThreshold>2.5</exclusionThreshold> >>> <latencyAware>true</latencyAware> >>> <minMeasure>100</minMeasure> >>> <policyName>RoundRobinPolicy</policyName> >>> <retryPeriod>10</retryPeriod> >>> <scale>2</scale> >>> </loadBalancePolicy> >>> >>> <poolOptions> >>> >>> <coreConnectionsForLocal>1000</coreConnectionsForLocal> >>> >>> <coreConnectionsForRemote>1000</coreConnectionsForRemote> >>> >>> <maxConnectionsForLocal>1000</maxConnectionsForLocal> >>> >>> <maxConnectionsForRemote>1000</maxConnectionsForRemote> >>> >>> <maxSimultaneousRequestsForLocal>1000</maxSimultaneousRequestsForLocal> >>> >>> <maxSimultaneousRequestsForRemote>1000</maxSimultaneousRequestsForRemote> >>> >>> <minSimultaneousRequestsForLocal>1000</minSimultaneousRequestsForLocal> >>> >>> <minSimultaneousRequestsForRemote>1000</minSimultaneousRequestsForRemote> >>> </poolOptions> >>> >>> <reconnectPolicy> >>> <baseDelayMs>3000</baseDelayMs> >>> >>> <policyName>ConstantReconnectionPolicy</policyName> >>> </reconnectPolicy> >>> <socketOptions> >>> >>> <connectTimeoutMillis>10000</connectTimeoutMillis> >>> <keepAlive>true</keepAlive> >>> <readTimeoutMillis>15000</readTimeoutMillis> >>> <tcpNoDelay>true</tcpNoDelay> >>> </socketOptions> >>> >>> </configuration> >>> >>> </definition> >>> </datasource> >>> >>> >>> >>> *cassandra-env.sh in cassandra node:* >>> >>> system_memory_in_mb="10240" >>> system_cpu_cores="6" >>> >>> Tested with an SSD hard drive. >>> >>> Thank you, >>> Sajini. >>> >>> On Fri, Oct 3, 2014 at 12:23 PM, Sajini De Silva <saj...@wso2.com> >>> wrote: >>> >>>> Hi Prabath, >>>> >>>> This is the error we got while deleting messages. >>>> >>>> [2014-10-03 12:16:55,059] ERROR >>>> {org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl} - Error while >>>> deleting messages >>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All >>>> host(s) tried for query failed (tried: /127.0.0.1 (Timeout during >>>> read)) >>>> at >>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64) >>>> at >>>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269) >>>> at >>>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183) >>>> at com.datastax.driver.core.Session.execute(Session.java:111) >>>> at >>>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207) >>>> at >>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299) >>>> at >>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327) >>>> at >>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203) >>>> at >>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30) >>>> at >>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19) >>>> at >>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:662) >>>> Caused by: >>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >>>> tried for query failed (tried: /127.0.0.1 (Timeout during read)) >>>> at >>>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106) >>>> at >>>> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177) >>>> ... 3 more >>>> Oct 3, 2014 12:16:55 PM com.lmax.disruptor.FatalExceptionHandler >>>> handleEventException >>>> SEVERE: Exception processing: 12095 >>>> org.wso2.andes.kernel.AndesAckData@5341cc77 >>>> org.wso2.andes.kernel.AndesException: >>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >>>> tried for query failed (tried: /127.0.0.1 (Timeout during read)) >>>> at >>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:303) >>>> at >>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327) >>>> at >>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203) >>>> at >>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30) >>>> at >>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19) >>>> at >>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:662) >>>> Caused by: >>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >>>> tried for query failed (tried: /127.0.0.1 (Timeout during read)) >>>> at >>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64) >>>> at >>>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269) >>>> at >>>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183) >>>> at com.datastax.driver.core.Session.execute(Session.java:111) >>>> at >>>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207) >>>> at >>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299) >>>> ... 8 more >>>> Caused by: >>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >>>> tried for query failed (tried: /127.0.0.1 (Timeout during read)) >>>> at >>>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106) >>>> at >>>> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177) >>>> ... 3 more >>>> Exception in thread "pool-25-thread-1" java.lang.RuntimeException: >>>> org.wso2.andes.kernel.AndesException: >>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >>>> tried for query failed (tried: /127.0.0.1 (Timeout during read)) >>>> at >>>> com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45) >>>> at >>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:662) >>>> Caused by: org.wso2.andes.kernel.AndesException: >>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >>>> tried for query failed (tried: /127.0.0.1 (Timeout during read)) >>>> at >>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:303) >>>> at >>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.deleteMessages(DurableAsyncStoringManager.java:327) >>>> at >>>> org.wso2.andes.kernel.storemanager.DurableAsyncStoringManager.processAckReceived(DurableAsyncStoringManager.java:203) >>>> at >>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:30) >>>> at >>>> org.wso2.andes.kernel.distrupter.AckHandler.onEvent(AckHandler.java:19) >>>> at >>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113) >>>> ... 3 more >>>> Caused by: >>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >>>> tried for query failed (tried: /127.0.0.1 (Timeout during read)) >>>> at >>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64) >>>> at >>>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269) >>>> at >>>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183) >>>> at com.datastax.driver.core.Session.execute(Session.java:111) >>>> at >>>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207) >>>> at >>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.deleteMessageMetadataFromQueue(CQLBasedMessageStoreImpl.java:299) >>>> ... 8 more >>>> Caused by: >>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) >>>> tried for query failed (tried: /127.0.0.1 (Timeout during read)) >>>> at >>>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:106) >>>> at >>>> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:177) >>>> ... 3 more >>>> >>>> >>>> We tried this scenario increasing the number of threads in thread pool >>>> to 1000, but same error occurred. >>>> >>>> Thank you, >>>> Sajini. >>>> >>>> >>>> On Wed, Oct 1, 2014 at 5:22 PM, Sajini De Silva <saj...@wso2.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm getting following exception in MB when doing load testing. >>>>> >>>>> SEVERE: Exception processing: 81 >>>>> org.wso2.andes.kernel.distrupter.CassandraDataEvent@2d524d15 >>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All >>>>> host(s) tried for query failed (tried: /127.0.0.1 (Timeout during >>>>> read)) >>>>> at >>>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64) >>>>> at >>>>> com.datastax.driver.core.ResultSetFuture.extractCauseFromExecutionException(ResultSetFuture.java:269) >>>>> at >>>>> com.datastax.driver.core.ResultSetFuture.getUninterruptibly(ResultSetFuture.java:183) >>>>> at com.datastax.driver.core.Session.execute(Session.java:111) >>>>> at >>>>> org.wso2.andes.store.cassandra.dao.GenericCQLDAO.batchExecute(GenericCQLDAO.java:207) >>>>> at >>>>> org.wso2.andes.store.cassandra.CQLBasedMessageStoreImpl.storeMessagePart(CQLBasedMessageStoreImpl.java:121) >>>>> at >>>>> org.wso2.andes.kernel.distrupter.AlternatingCassandraWriter.onEvent(AlternatingCassandraWriter.java:81) >>>>> at >>>>> org.wso2.andes.kernel.distrupter.AlternatingCassandraWriter.onEvent(AlternatingCassandraWriter.java:38) >>>>> at >>>>> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:113) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:662) >>>>> >>>>> Does anyone knows the cause for this error? >>>>> >>>>> -- >>>>> Sajini De SIlva >>>>> Software Engineer; WSO2 Inc.; http://wso2.com , >>>>> Email: saj...@wso2.com >>>>> Blog: http://sajinid.blogspot.com/ >>>>> Git hub profile: https://github.com/sajinidesilva >>>>> >>>>> Phone: +94 712797729 >>>>> >>>>> >>>> >>>> >>>> -- >>>> Sajini De SIlva >>>> Software Engineer; WSO2 Inc.; http://wso2.com , >>>> Email: saj...@wso2.com >>>> Blog: http://sajinid.blogspot.com/ >>>> Git hub profile: https://github.com/sajinidesilva >>>> >>>> Phone: +94 712797729 >>>> >>>> >>> >>> >>> -- >>> Sajini De SIlva >>> Software Engineer; WSO2 Inc.; http://wso2.com , >>> Email: saj...@wso2.com >>> Blog: http://sajinid.blogspot.com/ >>> Git hub profile: https://github.com/sajinidesilva >>> >>> Phone: +94 712797729 >>> >>> >>> _______________________________________________ >>> Dev mailing list >>> Dev@wso2.org >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> *Bhathiya Jayasekara* >> *Software Engineer,* >> *WSO2 inc., http://wso2.com <http://wso2.com>* >> >> *Phone: +94715478185 <%2B94715478185>* >> *LinkedIn: http://www.linkedin.com/in/bhathiyaj >> <http://www.linkedin.com/in/bhathiyaj>* >> *Twitter: https://twitter.com/bhathiyax <https://twitter.com/bhathiyax>* >> *Blog: http://movingaheadblog.blogspot.com >> <http://movingaheadblog.blogspot.com/>* >> >> _______________________________________________ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *Asitha Nanayakkara* > Software Engineer > WSO2, Inc. http://wso2.com/ > Mob: + 94 77 85 30 682 > > -- Sajini De SIlva Software Engineer; WSO2 Inc.; http://wso2.com , Email: saj...@wso2.com Blog: http://sajinid.blogspot.com/ Git hub profile: https://github.com/sajinidesilva Phone: +94 712797729
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
