Hi,
We have a setup with one manager node & two worker nodes fronted by an ELB.
The setup was working fine then we let the servers idle for sometime and
when accessing the WSDLs it failed and returned the following error at ELB
side.
However, both the worker nodes were up and running & we did not see any
exceptions at worker consoles.
*****************************************************************************************************************************************************************************************
TID: [0] [ELB] [2013-08-01 14:19:50,037] ERROR
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - No
application members available
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
*****************************************************************************************************************************************************************************************
As per an offline discussion we had with Azeez, he mentioned that this
could be due to a network issue. But then, even if the network issue was
sorted, the members should reconnect but this never happened. We had to
restart the servers to recover the issue.
NOTE: We have set the following properties in */repository/conf/
hazelcast.properties*
hazelcast.max.no.heartbeat.seconds=600
hazelcast.max.no.master.confirmation.seconds=450
Below are the logs printed at ELB side
*****************************************************************************************************************************************************************************************
TID: [0] [ELB] [2013-08-01 12:35:21,719] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Connection time
out after request is read: http-incoming-23
{org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2013-08-01 12:35:21,721] WARN
{org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-10:
Connection time out while in state: REQUEST_DONE
{org.apache.synapse.transport.passthru.TargetHandler}
TID: [0] [ELB] [2013-08-01 12:35:22,724] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Failed over to Host:192.168.4.52, Remote Host:null, Port: 4250,
HTTP:10763, HTTPS:10443, Domain: as.qa.420.domain, Sub-domain:mgt,
Active:true
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2013-08-01 12:40:03,866] INFO
{org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} -
Member left [2e7c1146-2075-45d3-bf41-570e9cfa1063]:
/192.168.4.52:4250{org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent}
TID: [0] [ELB] [2013-08-01 12:40:03,866] INFO
{org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} -
Member left [2e7c1146-2075-45d3-bf41-570e9cfa1063]:
/192.168.4.52:4250{org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent}
TID: [0] [ELB] [2013-08-01 12:40:07,161] WARN
{org.apache.synapse.transport.passthru.ConnectCallback} - Connection
refused or failed for :
/192.168.4.52:10443{org.apache.synapse.transport.passthru.ConnectCallback}
TID: [0] [ELB] [2013-08-01 12:40:07,169] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Suspended member Host:192.168.4.52, Remote Host:null, Port: 4250,
HTTP:10763, HTTPS:10443, Domain: as.qa.420.domain, Sub-domain:mgt,
Active:true for 10s due to connection failure to that member
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2013-08-01 12:40:07,170] ERROR
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - No
application members available having host name : mgt.as.qa.420.com and
tenant id : 0
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2013-08-01 12:40:07,619] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Illegal incoming
connection state: REQUEST_DONE . Possibly two send backs are happening for
the same request {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2013-08-01 12:40:07,619] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Trying to write
response body while the handler is in an inconsistent state REQUEST_DONE
{org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2013-08-01 12:40:07,620] ERROR
{org.apache.synapse.transport.passthru.SourceHandler} - Unexpected I/O
error: java.nio.channels.ClosedChannelException
{org.apache.synapse.transport.passthru.SourceHandler}
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:249)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:440)
at
org.apache.http.nio.reactor.ssl.SSLIOSession.sendEncryptedData(SSLIOSession.java:331)
at
org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:411)
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:150)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:181)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:346)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Thread.java:722)
TID: [0] [ELB] [2013-08-01 12:40:07,623] WARN
{org.apache.synapse.transport.passthru.ConnectCallback} - Connection
refused or failed for :
/192.168.4.52:10443{org.apache.synapse.transport.passthru.ConnectCallback}
TID: [0] [ELB] [2013-08-01 12:40:07,624] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Suspended member Host:192.168.4.52, Remote Host:null, Port: 4250,
HTTP:10763, HTTPS:10443, Domain: as.qa.420.domain, Sub-domain:mgt,
Active:true for 10s due to connection failure to that member
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2013-08-01 12:40:07,625] ERROR
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - No
application members available having host name : mgt.as.qa.420.com and
tenant id : 0
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2013-08-01 12:40:07,654] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Illegal incoming
connection state: REQUEST_DONE . Possibly two send backs are happening for
the same request {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2013-08-01 12:40:07,655] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Trying to write
response body while the handler is in an inconsistent state REQUEST_DONE
{org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2013-08-01 12:40:07,656] ERROR
{org.apache.synapse.transport.passthru.SourceHandler} - Unexpected I/O
error: java.nio.channels.ClosedChannelException
{org.apache.synapse.transport.passthru.SourceHandler}
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:249)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:440)
at
org.apache.http.nio.reactor.ssl.SSLIOSession.sendEncryptedData(SSLIOSession.java:331)
at
org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:411)
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:150)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:181)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:346)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Thread.java:722)
TID: [0] [ELB] [2013-08-01 12:40:07,657] WARN
{org.apache.synapse.transport.passthru.ConnectCallback} - Connection
refused or failed for :
/192.168.4.52:10443{org.apache.synapse.transport.passthru.ConnectCallback}
TID: [0] [ELB] [2013-08-01 12:40:07,659] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Suspended member Host:192.168.4.52, Remote Host:null, Port: 4250,
HTTP:10763, HTTPS:10443, Domain: as.qa.420.domain, Sub-domain:mgt,
Active:true for 10s due to connection failure to that member
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2013-08-01 12:40:07,660] ERROR
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} - No
application members available having host name : mgt.as.qa.420.com and
tenant id : 0
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
TID: [0] [ELB] [2013-08-01 12:40:07,682] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Illegal incoming
connection state: REQUEST_DONE . Possibly two send backs are happening for
the same request {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2013-08-01 12:40:07,683] WARN
{org.apache.synapse.transport.passthru.SourceHandler} - Trying to write
response body while the handler is in an inconsistent state REQUEST_DONE
{org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [ELB] [2013-08-01 12:40:07,684] ERROR
{org.apache.synapse.transport.passthru.SourceHandler} - Unexpected I/O
error: java.nio.channels.ClosedChannelException
{org.apache.synapse.transport.passthru.SourceHandler}
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:249)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:440)
at
org.apache.http.nio.reactor.ssl.SSLIOSession.sendEncryptedData(SSLIOSession.java:331)
at
org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:411)
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:150)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:181)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:346)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
at java.lang.Thread.run(Thread.java:722)
TID: [0] [ELB] [2013-08-01 12:40:07,687] WARN
{org.apache.synapse.transport.passthru.ConnectCallback} - Connection
refused or failed for :
/192.168.4.52:10443{org.apache.synapse.transport.passthru.ConnectCallback}
TID: [0] [ELB] [2013-08-01 12:40:07,689] INFO
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
Suspended member Host:192.168.4.52, Remote Host:null, Port: 4250,
HTTP:10763, HTTPS:10443, Domain: as.qa.420.domain, Sub-domain:mgt,
Active:true for 10s due to connection failure to that member
{org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
Regards,
Evanthika Amarasiri
Technical Lead - Quality Assurance
Management Committee - QA Technologies
Mobile: +94773125935
*
*
wso2.com Lean Enterprise Middleware
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
