Hi Dilini,

On Thu, Jul 14, 2016 at 2:31 PM, Dilini Gunatilake <dili...@wso2.com> wrote:

> Hi,
>
> @Thushara,
> Thanks for the info. Any recommended number of max_connections to
> configure considering indexing with a high load etc? Then we can add those
> details in the docs to avoid future issues.
>

My personal recommendation,
Increase the mysql active connections to 350. Please note that this is
bounded to server specs.
Now start 2 servers at a time. Wait 10 mins before starting the other 2
WSO2 servers.
Or else you can always start 1 server at a time every 10 mins. Therefore it
will take 30 mins to start all 4 nodes.

Thanks,
Thushara.


>
> @Sumedha,
> I was able to start all the nodes because the number of connections
> reduces after some time.
>
> As Rajith mentioned, the concern is having that number of connections
> created just for a server startup having less than 25 assets.
>
> Thanks,
> Dilini
>
>
> On Thu, Jul 14, 2016 at 12:53 PM, Rajith Vitharana <raji...@wso2.com>
> wrote:
>
>> Hi,
>>
>> On Thu, Jul 14, 2016 at 12:44 PM, Sumedha Rubasinghe <sume...@wso2.com>
>> wrote:
>>
>>> Guys,
>>> Some food for thought.
>>>
>>> 1. A single MySQL Server has a globally defined max_connections figure
>>> (151 as Dilini mentioned)
>>>
>>> So on a clustered setup, when you point to a single MySQL Server from
>>> various WSO2 products, depending on no of keep alive connections specified,
>>> you will at some point will hit too many connections from server side @
>>> some point.
>>>
>> Yeah totally agree, but that much(395) is bit high AFAIU when just
>> starting up a server(without any load in 4 servers, and when it stabilize,
>> it only consumes less than 20 connections without load)
>>
>> @Thushara, Thanks for the info, will check what you have mentioned.
>>
>> Thanks,
>>
>>>
>>> So you have to rethink the deployment and use a different database
>>> server.
>>>
>>> 2. If ES has a tendency to perform too many database calls @ startup,
>>> - You can temporally eliminate that by controlling node start up order
>>>
>>>
>>> On Thu, Jul 14, 2016 at 12:04 PM, Rajith Vitharana <raji...@wso2.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> We faced the same issue in ESB latest when clustering enabled, with 4
>>>> node cluster, number of connections goes to around 395 and gradually comes
>>>> down to 20 or less, even with the exception happened(mysql having default
>>>> max connections 151), it seems it gets stable to 20 or less connections
>>>> after some time. Created a public jira to track this issue at [1]
>>>>
>>>> [1] - https://wso2.org/jira/browse/REGISTRY-3771
>>>>
>>>> Thanks,
>>>>
>>>> On Wed, Jul 13, 2016 at 4:42 PM, Dilini Gunatilake <dili...@wso2.com>
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I have set up a 4 node cluster for ES 2.1.0 Alpha2 pack with mysql
>>>>> 5.7. When I started one store node I got an error [3].
>>>>>
>>>>> Then, I checked the connections in the mysql server and found that the
>>>>> default number of max_connections (151) have been exceeded while the store
>>>>> node is starting. But initially without starting any ES instances there
>>>>> were only around 20 connections. So, what we observed was when starting ES
>>>>> instances substantially high amount of connections will be created and 
>>>>> they
>>>>> will be gradually reduced after sometime. Is this an acceptable behavior?
>>>>>
>>>>> Also, we could find the recommended max_connections to configure in
>>>>> some products [1][2], but couldn't find anything for ES. So, what is the
>>>>> recommended value for ES?
>>>>> For the time being we will increase the max connections to 300 and
>>>>> proceed with testing.
>>>>>
>>>>> Your earliest response is highly appreciated.
>>>>>
>>>>> [1] https://docs.wso2.com/display/AM200/Tuning+Performance
>>>>> [2]
>>>>> https://docs.wso2.com/display/Governance510/Performance+Tuning+Recommendations
>>>>>
>>>>> [3]
>>>>>
>>>>> [2016-07-13 07:50:40,014] ERROR
>>>>> {org.wso2.carbon.registry.core.dataaccess.TransactionManager} -  Failed to
>>>>> start new registry transaction.
>>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
>>>>> Could not create connection to database server. Attempted reconnect 3
>>>>> times. Giving up.
>>>>> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown
>>>>> Source)
>>>>> at
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:387)
>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917)
>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:896)
>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:885)
>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:860)
>>>>> at
>>>>> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2165)
>>>>> at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2090)
>>>>> at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:795)
>>>>> at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:44)
>>>>> at sun.reflect.GeneratedConstructorAccessor76.newInstance(Unknown
>>>>> Source)
>>>>> at
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>>>>> at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400)
>>>>> at
>>>>> com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:327)
>>>>> at
>>>>> org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278)
>>>>> at
>>>>> org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
>>>>> at
>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701)
>>>>> at
>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635)
>>>>> at
>>>>> org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:188)
>>>>> at
>>>>> org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:128)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCTransactionManager.beginTransaction(JDBCTransactionManager.java:80)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.beginTransaction(EmbeddedRegistry.java:443)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:505)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler.get(MountHandler.java:399)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.get(HandlerManager.java:2444)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.handlers.UserDefinedHandlerManager.get(UserDefinedHandlerManager.java:200)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.handlers.HandlerLifecycleManager.get(HandlerLifecycleManager.java:914)
>>>>> at
>>>>> org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.get(EmbeddedRegistry.java:512)
>>>>> at
>>>>> org.wso2.carbon.registry.core.caching.CacheBackedRegistry.get(CacheBackedRegistry.java:203)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry.getInternal(UserRegistry.java:617)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry.access$400(UserRegistry.java:61)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:597)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry$5.run(UserRegistry.java:594)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at
>>>>> org.wso2.carbon.registry.core.session.UserRegistry.get(UserRegistry.java:594)
>>>>> at
>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.createIndexDocument(AsyncIndexer.java:242)
>>>>> at
>>>>> org.wso2.carbon.registry.indexing.AsyncIndexer$IndexingTask.run(AsyncIndexer.java:210)
>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> Caused by:
>>>>> com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Too
>>>>> many connections
>>>>> at sun.reflect.GeneratedConstructorAccessor107.newInstance(Unknown
>>>>> Source)
>>>>> at
>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:404)
>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:387)
>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:917)
>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3878)
>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3814)
>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871)
>>>>> at
>>>>> com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1694)
>>>>> at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1215)
>>>>> at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2255)
>>>>> at
>>>>> com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2106)
>>>>> ... 43 more
>>>>> [2016-07-13 07:50:40,077] ERROR
>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} -  Error while indexing.
>>>>> Resource at path
>>>>> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail"could
>>>>> not be
>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>> Resource does not exist at path
>>>>> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_thumbnail
>>>>> [2016-07-13 07:50:40,076] ERROR
>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} -  Error while indexing.
>>>>> Resource at path "/_system/governance/gadgets/admin/Stacked Line
>>>>> Chart/1.0.0"could not be
>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>> Resource does not exist at path /_system/governance/gadgets/admin/Stacked
>>>>> Line Chart/1.0.0
>>>>> [2016-07-13 07:50:40,074] ERROR
>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} -  Error while indexing.
>>>>> Resource at path
>>>>> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner"could
>>>>> not be
>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>> Resource does not exist at path
>>>>> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_banner
>>>>> [2016-07-13 07:50:40,067] ERROR
>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} -  Error while indexing.
>>>>> Resource at path
>>>>> "/_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner"could
>>>>> not be
>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>> Resource does not exist at path
>>>>> /_system/governance/store/asset_resources/gadget/8c3eda76-ec7f-43e3-bbe8-efa14f85d780/images_banner
>>>>> [2016-07-13 07:50:40,064] ERROR
>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} -  Error while indexing.
>>>>> Resource at path "/_system/governance/gadgets/admin/WSO2 Carbon Commits
>>>>> List Discussion/1.0.0"could not be
>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>> Resource does not exist at path /_system/governance/gadgets/admin/WSO2
>>>>> Carbon Commits List Discussion/1.0.0
>>>>> [2016-07-13 07:50:40,062] ERROR
>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} -  Error while indexing.
>>>>> Resource at path
>>>>> "/_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail"could
>>>>> not be
>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>> Resource does not exist at path
>>>>> /_system/governance/store/asset_resources/site/6b89dd21-0a98-4489-9f0d-fc327ff46bc0/images_thumbnail
>>>>> [2016-07-13 07:50:40,056] ERROR
>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} -  Error while indexing.
>>>>> Resource at path
>>>>> "/_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail"could
>>>>> not be
>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>> Resource does not exist at path
>>>>> /_system/governance/store/asset_resources/gadget/4c4f7877-ff39-4379-9ee4-6c1827732ec6/images_thumbnail
>>>>> [2016-07-13 07:50:40,079] ERROR
>>>>> {org.wso2.carbon.registry.indexing.AsyncIndexer} -  Error while indexing.
>>>>> Resource at path
>>>>> "/_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail"could
>>>>> not be
>>>>> indexedorg.wso2.carbon.registry.core.exceptions.ResourceNotFoundException:
>>>>> Resource does not exist at path
>>>>> /_system/governance/store/asset_resources/gadget/57235acc-ad87-41db-84e7-dadc0bcff8d5/images_thumbnail
>>>>>
>>>>>
>>>>> Thank you.
>>>>> Regards,
>>>>>
>>>>> --
>>>>>
>>>>> *Dilini GunatilakeSoftware Engineer - QA Team*
>>>>> Mobile : +94 (0) 771 162518
>>>>> dili...@wso2.com
>>>>>
>>>>> _______________________________________________
>>>>> Dev mailing list
>>>>> Dev@wso2.org
>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Rajith Vitharana
>>>>
>>>> Software Engineer,
>>>> WSO2 Inc. : wso2.com
>>>> Mobile : +94715883223
>>>> Blog : http://lankavitharana.blogspot.com/
>>>>
>>>> _______________________________________________
>>>> Dev mailing list
>>>> Dev@wso2.org
>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>
>>>>
>>>
>>>
>>> --
>>> /sumedha
>>> m: +94 773017743
>>> b :  bit.ly/sumedha
>>>
>>
>>
>>
>> --
>> Rajith Vitharana
>>
>> Software Engineer,
>> WSO2 Inc. : wso2.com
>> Mobile : +94715883223
>> Blog : http://lankavitharana.blogspot.com/
>>
>
>
>
> --
>
> *Dilini GunatilakeSoftware Engineer - QA Team*
> Mobile : +94771162518
> dili...@wso2.com
>
>
> _______________________________________________
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
Thanks,
Thushara Kasun Ranawaka
Software Engineer
WSO2 Inc.; <http://www.wso2.com>
lean.enterprise.middleware
Mobile : *+94 (0) 773438949 <%2B94%20%280%29%20773438949>*
*thusha...@wso2.com <thusha...@wso2.com>*
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Reply via email to