Hi, One small thing to clarify,
On Thu, Aug 1, 2013 at 3:06 PM, Evanthika Amarasiri <[email protected]>wrote: > Hi, > > We have a setup with one manager node & two worker nodes fronted by an > ELB. The setup was working fine then we let the servers idle for sometime > and when accessing the WSDLs it failed and returned the following error at > ELB side. > > However, both the worker nodes were up and running & we did not see any > exceptions at worker consoles. > > > ***************************************************************************************************************************************************************************************** > TID: [0] [ELB] [2013-08-01 14:19:50,037] ERROR > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - No > application members available > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > > ***************************************************************************************************************************************************************************************** > > As per an offline discussion we had with Azeez, he mentioned that this > could be due to a network issue. But then, even if the network issue was > sorted, the members should reconnect but this never happened. We had to > restart the servers to recover the issue. > > NOTE: We have set the following properties in */repository/conf/ > hazelcast.properties* > > hazelcast.max.no.heartbeat.seconds=600 > hazelcast.max.no.master.confirmation.seconds=450 > > Below are the logs printed at ELB side > > > ***************************************************************************************************************************************************************************************** > TID: [0] [ELB] [2013-08-01 12:35:21,719] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Connection time > out after request is read: http-incoming-23 > {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2013-08-01 12:35:21,721] WARN > {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-10: > Connection time out while in state: REQUEST_DONE > {org.apache.synapse.transport.passthru.TargetHandler} > TID: [0] [ELB] [2013-08-01 12:35:22,724] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Failed over to Host:192.168.4.52, Remote Host:null, Port: 4250, > HTTP:10763, HTTPS:10443, Domain: as.qa.420.domain, Sub-domain:mgt, > Active:true > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2013-08-01 12:40:03,866] INFO > {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} - > Member left [2e7c1146-2075-45d3-bf41-570e9cfa1063]: > /192.168.4.52:4250{org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} > TID: [0] [ELB] [2013-08-01 12:40:03,866] INFO > {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} - > Member left [2e7c1146-2075-45d3-bf41-570e9cfa1063]: > /192.168.4.52:4250{org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} > TID: [0] [ELB] [2013-08-01 12:40:07,161] WARN > {org.apache.synapse.transport.passthru.ConnectCallback} - Connection > refused or failed for : > /192.168.4.52:10443{org.apache.synapse.transport.passthru.ConnectCallback} > TID: [0] [ELB] [2013-08-01 12:40:07,169] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Suspended member Host:192.168.4.52, Remote Host:null, Port: 4250, > HTTP:10763, HTTPS:10443, Domain: as.qa.420.domain, Sub-domain:mgt, > Active:true for 10s due to connection failure to that member > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2013-08-01 12:40:07,170] ERROR > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - No > application members available having host name : mgt.as.qa.420.com and > tenant id : 0 > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2013-08-01 12:40:07,619] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Illegal incoming > connection state: REQUEST_DONE . Possibly two send backs are happening for > the same request {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2013-08-01 12:40:07,619] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Trying to write > response body while the handler is in an inconsistent state REQUEST_DONE > {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2013-08-01 12:40:07,620] ERROR > {org.apache.synapse.transport.passthru.SourceHandler} - Unexpected I/O > error: java.nio.channels.ClosedChannelException > {org.apache.synapse.transport.passthru.SourceHandler} > java.nio.channels.ClosedChannelException > at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:249) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:440) > at > org.apache.http.nio.reactor.ssl.SSLIOSession.sendEncryptedData(SSLIOSession.java:331) > at > org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:411) > at > org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:150) > at > org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:181) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:346) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) > at > org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) > at > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604) > at java.lang.Thread.run(Thread.java:722) > TID: [0] [ELB] [2013-08-01 12:40:07,623] WARN > {org.apache.synapse.transport.passthru.ConnectCallback} - Connection > refused or failed for : > /192.168.4.52:10443{org.apache.synapse.transport.passthru.ConnectCallback} > TID: [0] [ELB] [2013-08-01 12:40:07,624] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Suspended member Host:192.168.4.52, Remote Host:null, Port: 4250, > HTTP:10763, HTTPS:10443, Domain: as.qa.420.domain, Sub-domain:mgt, > Active:true for 10s due to connection failure to that member > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2013-08-01 12:40:07,625] ERROR > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - No > application members available having host name : mgt.as.qa.420.com and > tenant id : 0 > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2013-08-01 12:40:07,654] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Illegal incoming > connection state: REQUEST_DONE . Possibly two send backs are happening for > the same request {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2013-08-01 12:40:07,655] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Trying to write > response body while the handler is in an inconsistent state REQUEST_DONE > {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2013-08-01 12:40:07,656] ERROR > {org.apache.synapse.transport.passthru.SourceHandler} - Unexpected I/O > error: java.nio.channels.ClosedChannelException > {org.apache.synapse.transport.passthru.SourceHandler} > java.nio.channels.ClosedChannelException > at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:249) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:440) > at > org.apache.http.nio.reactor.ssl.SSLIOSession.sendEncryptedData(SSLIOSession.java:331) > at > org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:411) > at > org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:150) > at > org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:181) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:346) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) > at > org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) > at > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604) > at java.lang.Thread.run(Thread.java:722) > TID: [0] [ELB] [2013-08-01 12:40:07,657] WARN > {org.apache.synapse.transport.passthru.ConnectCallback} - Connection > refused or failed for : > /192.168.4.52:10443{org.apache.synapse.transport.passthru.ConnectCallback} > TID: [0] [ELB] [2013-08-01 12:40:07,659] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Suspended member Host:192.168.4.52, Remote Host:null, Port: 4250, > HTTP:10763, HTTPS:10443, Domain: as.qa.420.domain, Sub-domain:mgt, > Active:true for 10s due to connection failure to that member > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2013-08-01 12:40:07,660] ERROR > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - No > application members available having host name : mgt.as.qa.420.com and > tenant id : 0 > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > TID: [0] [ELB] [2013-08-01 12:40:07,682] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Illegal incoming > connection state: REQUEST_DONE . Possibly two send backs are happening for > the same request {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2013-08-01 12:40:07,683] WARN > {org.apache.synapse.transport.passthru.SourceHandler} - Trying to write > response body while the handler is in an inconsistent state REQUEST_DONE > {org.apache.synapse.transport.passthru.SourceHandler} > TID: [0] [ELB] [2013-08-01 12:40:07,684] ERROR > {org.apache.synapse.transport.passthru.SourceHandler} - Unexpected I/O > error: java.nio.channels.ClosedChannelException > {org.apache.synapse.transport.passthru.SourceHandler} > java.nio.channels.ClosedChannelException > at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:249) > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:440) > at > org.apache.http.nio.reactor.ssl.SSLIOSession.sendEncryptedData(SSLIOSession.java:331) > at > org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:411) > at > org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:150) > at > org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:181) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:346) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320) > at > org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) > at > org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) > at > org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604) > at java.lang.Thread.run(Thread.java:722) > TID: [0] [ELB] [2013-08-01 12:40:07,687] WARN > {org.apache.synapse.transport.passthru.ConnectCallback} - Connection > refused or failed for : > /192.168.4.52:10443{org.apache.synapse.transport.passthru.ConnectCallback} > TID: [0] [ELB] [2013-08-01 12:40:07,689] INFO > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - > Suspended member Host:192.168.4.52, Remote Host:null, Port: 4250, > HTTP:10763, HTTPS:10443, Domain: as.qa.420.domain, Sub-domain:mgt, > Active:true for 10s due to connection failure to that member > {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} > > Why the Remote Host name is null?. I saw this in the logs earlier as well.. thanks, > Regards, > Evanthika Amarasiri > Technical Lead - Quality Assurance > Management Committee - QA Technologies > Mobile: +94773125935 > * > * > wso2.com Lean Enterprise Middleware > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Supun Malinga, Senior Software Engineer, WSO2 Inc. http://wso2.com http://wso2.org email - [email protected] <[email protected]> mobile - 071 56 91 321
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
