Hi All Here we have changed the no of max connections to 100,000. Although the error says use a higher wait timeout I think increasing it is not the best option. And we have put the autoReconnect=true in jdbc url. @Prabath : Is there a way of eliminating this problem. Currently when we detect this we restart the server. Is there a mysql setting that we can use to eliminate this?
Thanks & Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 On Mon, Sep 15, 2014 at 12:55 PM, Madhawa Bandara <madh...@wso2.com> wrote: > Hi, > When Appfactory tries to connect to a remote application server instance > on openstack following error[1] can be observed in the appserver log. > > autoReconnect=true is already set and wait_timeout is set as per the > instructions in the error log but that didn't fix the issue. Any help? > > Thanks, > Madhawa > > [1]- > > TID: [0] [TEST-AS] [2014-09-15 06:39:20,569] ERROR > {org.wso2.carbon.registry.core.utils.LogWriter} - Unable to save log > records {org.wso2.carbon.registry.core.utils.LogWriter} > org.wso2.carbon.registry.core.exceptions.RegistryException: Failed to > update log batch records . The last packet successfully received from the > server was 326,264,793 milliseconds ago. The last packet sent successfully > to the server was 326,264,794 milliseconds ago. is longer than the server > configured value of 'wait_timeout'. You should consider either expiring > and/or testing connection validity before use in your application, > increasing the server configured values for client timeouts, or using the > Connector/J connection property 'autoReconnect=true' to avoid this problem. > at > org.wso2.carbon.registry.core.jdbc.dao.JDBCLogsDAO.addLogRecords(JDBCLogsDAO.java:177) > at > org.wso2.carbon.registry.core.jdbc.dao.JDBCLogsDAO.saveLogBatch(JDBCLogsDAO.java:124) > at > org.wso2.carbon.registry.core.utils.LogWriter.logWrite(LogWriter.java:106) > at org.wso2.carbon.registry.core.utils.LogWriter.run(LogWriter.java:91) > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The > last packet successfully received from the server was 326,264,793 > milliseconds ago. The last packet sent successfully to the server was > 326,264,794 milliseconds ago. is longer than the server configured value of > 'wait_timeout'. You should consider either expiring and/or testing > connection validity before use in your application, increasing the server > configured values for client timeouts, or using the Connector/J connection > property 'autoReconnect=true' to avoid this problem. > at sun.reflect.GeneratedConstructorAccessor399.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121) > at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3988) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2598) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2819) > at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:5346) > at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:126) > at > org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109) > at > org.wso2.carbon.ndatasource.rdbms.ConnectionRollbackOnReturnInterceptor.invoke(ConnectionRollbackOnReturnInterceptor.java:51) > at > org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109) > at > org.apache.tomcat.jdbc.pool.interceptor.AbstractCreateStatementInterceptor.invoke(AbstractCreateStatementInterceptor.java:71) > at > org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109) > at > org.apache.tomcat.jdbc.pool.interceptor.ConnectionState.invoke(ConnectionState.java:153) > at > org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109) > at org.apache.tomcat.jdbc.pool.TrapException.invoke(TrapException.java:41) > at > org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109) > at > org.apache.tomcat.jdbc.pool.DisposableConnectionFacade.invoke(DisposableConnectionFacade.java:80) > at com.sun.proxy.$Proxy14.setAutoCommit(Unknown Source) > at > org.wso2.carbon.registry.core.jdbc.dao.JDBCLogsDAO.addLogRecords(JDBCLogsDAO.java:144) > ... 3 more > Caused by: java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) > at java.net.SocketOutputStream.write(SocketOutputStream.java:159) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3969) > ... 23 more > > TID: [0] [TEST-AS] [2014-09-15 06:42:49,608] WARN > {org.wso2.carbon.server.admin.module.handler.AuthenticationHandler} - > Illegal access attempt at [2014-09-15 06:42:49,0608] from IP address > 10.100.5.57 while trying to authenticate access to service > AppFactoryNDataSourceAdmin > {org.wso2.carbon.server.admin.module.handler.AuthenticationHandler} > TID: [0] [TEST-AS] [2014-09-15 06:43:53,307] WARN > {org.wso2.carbon.server.admin.module.handler.AuthenticationHandler} - > Illegal access attempt at [2014-09-15 06:43:53,0307] from IP address > 10.100.5.57 while trying to authenticate access to service > AppFactoryNDataSourceAdmin > {org.wso2.carbon.server.admin.module.handler.AuthenticationHandler} > TID: [0] [TEST-AS] [2014-09-15 06:46:23,672] INFO > {org.wso2.carbon.identity.authenticator.saml2.sso.SAML2SSOAuthenticator} - > SAML2SSOAuthenticator username is : ad...@af.com > {org.wso2.carbon.identity.authenticator.saml2.sso.SAML2SSOAuthenticator} > TID: [0] [TEST-AS] [2014-09-15 06:46:23,673] INFO > {org.wso2.carbon.identity.authenticator.saml2.sso.SAML2SSOAuthenticator} - > Tenant domain in carbon context is : carbon.super > {org.wso2.carbon.identity.authenticator.saml2.sso.SAML2SSOAuthenticator} > TID: [0] [TEST-AS] [2014-09-15 06:46:23,673] INFO > {org.wso2.carbon.identity.authenticator.saml2.sso.SAML2SSOAuthenticator} - > Tenant id in user realm in corbon context is : -1234 > {org.wso2.carbon.identity.authenticator.saml2.sso.SAML2SSOAuthenticator} > TID: [0] [TEST-AS] [2014-09-15 06:46:23,675] FATAL > {org.wso2.carbon.utils.dbcreator.DatabaseCreator} - Failed to create > database tables for registry resource store. The last packet successfully > received from the server was 326,687,906 milliseconds ago. The last packet > sent successfully to the server was 326,687,907 milliseconds ago. is longer > than the server configured value of 'wait_timeout'. You should consider > either expiring and/or testing connection validity before use in your > application, increasing the server configured values for client timeouts, > or using the Connector/J connection property 'autoReconnect=true' to avoid > this problem. {org.wso2.carbon.utils.dbcreator.DatabaseCreator} > com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet > successfully received from the server was 326,687,906 milliseconds ago. > The last packet sent successfully to the server was 326,687,907 > milliseconds ago. is longer than the server configured value of > 'wait_timeout'. You should consider either expiring and/or testing > connection validity before use in your application, increasing the server > configured values for client timeouts, or using the Connector/J connection > property 'autoReconnect=true' to avoid this problem. > at sun.reflect.GeneratedConstructorAccessor399.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at > com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121) > at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3988) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2598) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2819) > at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:5346) > at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.tomcat.jdbc.pool.ProxyConnection.invoke(ProxyConnection.java:126) > at > org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109) > at > org.wso2.carbon.ndatasource.rdbms.ConnectionRollbackOnReturnInterceptor.invoke(ConnectionRollbackOnReturnInterceptor.java:51) > at > org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109) > at > org.apache.tomcat.jdbc.pool.interceptor.AbstractCreateStatementInterceptor.invoke(AbstractCreateStatementInterceptor.java:71) > at > org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109) > at > org.apache.tomcat.jdbc.pool.interceptor.ConnectionState.invoke(ConnectionState.java:153) > at > org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109) > at org.apache.tomcat.jdbc.pool.TrapException.invoke(TrapException.java:41) > at > org.apache.tomcat.jdbc.pool.JdbcInterceptor.invoke(JdbcInterceptor.java:109) > at > org.apache.tomcat.jdbc.pool.DisposableConnectionFacade.invoke(DisposableConnectionFacade.java:80) > at com.sun.proxy.$Proxy14.setAutoCommit(Unknown Source) > at > org.wso2.carbon.utils.dbcreator.DatabaseCreator.createRegistryDatabase(DatabaseCreator.java:59) > at > org.wso2.carbon.registry.core.jdbc.dataaccess.JDBCDataAccessManager.createDatabase(JDBCDataAccessManager.java:122) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.configure(EmbeddedRegistryService.java:189) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService.<init>(EmbeddedRegistryService.java:99) > at > org.wso2.carbon.registry.core.config.RegistryContext.getEmbeddedRegistryService(RegistryContext.java:644) > at > org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler.getRegistry(MountHandler.java:221) > at > org.wso2.carbon.registry.core.jdbc.handlers.builtin.MountHandler.getRegistryContext(MountHandler.java:352) > at > org.wso2.carbon.registry.core.jdbc.handlers.HandlerManager.getRegistryContext(HandlerManager.java:3185) > at > org.wso2.carbon.registry.core.jdbc.handlers.UserDefinedHandlerManager.getRegistryContext(UserDefinedHandlerManager.java:265) > at > org.wso2.carbon.registry.core.jdbc.handlers.HandlerLifecycleManager.getRegistryContext(HandlerLifecycleManager.java:1323) > at > org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.getRegistryContext(EmbeddedRegistry.java:469) > at > org.wso2.carbon.registry.core.caching.CacheBackedRegistry.resourceExists(CacheBackedRegistry.java:243) > at > org.wso2.carbon.registry.core.session.UserRegistry.resourceExists(UserRegistry.java:629) > at > org.wso2.carbon.core.util.KeyStoreManager.getKeyStore(KeyStoreManager.java:136) > at > org.wso2.carbon.identity.authenticator.saml2.sso.util.Util.getX509CredentialImplForTenant(Util.java:144) > at > org.wso2.carbon.identity.authenticator.saml2.sso.SAML2SSOAuthenticator.validateSignature(SAML2SSOAuthenticator.java:308) > at > org.wso2.carbon.identity.authenticator.saml2.sso.SAML2SSOAuthenticator.validateSignature(SAML2SSOAuthenticator.java:273) > at > org.wso2.carbon.identity.authenticator.saml2.sso.SAML2SSOAuthenticator.validateSignature(SAML2SSOAuthenticator.java:250) > at > org.wso2.carbon.identity.authenticator.saml2.sso.SAML2SSOAuthenticator.login(SAML2SSOAuthenticator.java:108) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212) > at > org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117) > at > org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40) > at > org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110) > at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) > at > org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172) > at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146) > at > org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:755) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) > at > org.eclipse.equinox.http.helper.FilterServletAdaptor$FilterChainImpl.doFilter(FilterServletAdaptor.java:56) > at > org.wso2.carbon.mashup.jsservices.custom.ui.CustomUIServletFilter.doFilter(CustomUIServletFilter.java:233) > at > org.eclipse.equinox.http.helper.FilterServletAdaptor.service(FilterServletAdaptor.java:37) > at > org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61) > at > org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128) > at > org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) > at > org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68) > at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:274) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:271) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:536) > at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:306) > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:166) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:299) > at > org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:57) > at > org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:193) > at > org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189) > at java.security.AccessController.doPrivileged(Native Method) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) > at > org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61) > at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:274) > at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:271) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAsPrivileged(Subject.java:536) > at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:306) > at > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:246) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) > at > org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:57) > at > org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:193) > at > org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:189) > at java.security.AccessController.doPrivileged(Native Method) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) > at > org.wso2.carbon.statistics.webapp.RequestIntercepterValve.invoke(RequestIntercepterValve.java:43) > at > org.wso2.carbon.bam.webapp.stat.publisher.WebAppStatisticPublisherValve.invoke(WebAppStatisticPublisherValve.java:104) > at > org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178) > at > org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47) > at > org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56) > at > org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47) > at > org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141) > at > org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936) > at > org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) > at > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) > at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.net.SocketException: Broken pipe > at java.net.SocketOutputStream.socketWrite0(Native Method) > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) > at java.net.SocketOutputStream.write(SocketOutputStream.java:159) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) > at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3969) > > > > -- > Regards, > > *Madhawa Bandara* > Software Engineer > WSO2, Inc. > lean.enterprise.middleware > > Mobile - *+94777487726 <%2B94777487726>* > Blog* - *classdeffound.blogspot.com > > _______________________________________________ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > >
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
