Hi Evanthika,
Can you kindly enable wire logs in the ELB in EC2 cluster and provide the
logs when this exception is thrown?
Thanks,
Jeewantha.
On Fri, Aug 16, 2013 at 4:22 PM, Afkham Azeez <az...@wso2.com> wrote:
> It was a configuration issue where the subDomain wasn't properly getting
> set.
>
>
> On Fri, Aug 16, 2013 at 1:07 PM, Jeewantha Dharmaparakrama <
> jeewan...@wso2.com> wrote:
>
>> Hi Folks,
>>
>> LB seems to be unstable when clustered with latest 4.2.0 kernal packs
>> [1]. Though this was first thought of as a PTT issue (see thread@dev -
>> "java.nio.channels.ClosedChannelException
>> thrown time to time on ELB console") there seems to be some inconstancy
>> when nodes are joined to the cluster which we are currently looking into.
>>
>> When a management node is added to the cluster, only the following log
>> message is printed in sout.
>>
>> INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} -
>> Added member: Host:127.0.0.1, Remote Host:null, Port: 3999, HTTP:-1,
>> HTTPS:-1, Domain: null, Sub-domain:null, Active:true
>>
>> This was checked with the packs built on 5th and 9th august [2] and as
>> per discussion with Azeez and Sameera, the products do not get clustered
>> properly. When management console is accessed, a SOAP Fault pops up on the
>> browser as
>>
>> <soapenv:Envelope><soapenv:
>> Body><nullResponse><Timestamp>Thu Aug 15 2013 16:55:05 GMT+0530
>> (IST)</Timestamp><Ack>Failure</Ack><Errors><ShortMessage>Gateway
>> Error</ShortMessage><LongMessage>No application members available
>> </LongMessage><ErrorCode>500
>> </ErrorCode><SeverityCode>Error</SeverityCode><ErrorClassification>RequestError</ErrorClassification></Errors><ServiceName>null</ServiceName><ResponseCode>null</ResponseCode><ContentType>null</ContentType><Version>1.5.1</Version></nullResponse></soapenv:Body></soapenv:Envelope>
>>
>> and with an Exception at the ELB backend
>>
>> [2013-08-15 16:55:05,407] ERROR - TenantAwareLoadBalanceEndpoint No
>> application members available
>> [2013-08-15 16:55:05,909] WARN - SourceHandler Illegal incoming
>> connection state: REQUEST_DONE . Possibly two send backs are happening for
>> the same request
>> [2013-08-15 16:55:05,909] WARN - SourceHandler Trying to write response
>> body while the handler is in an inconsistent state REQUEST_DONE
>> [2013-08-15 16:55:05,910] ERROR - TenantAwareLoadBalanceEndpoint No
>> application members available
>> [2013-08-15 16:57:28,353] ERROR - TenantAwareLoadBalanceEndpoint No
>> application members available
>> [2013-08-15 16:57:28,708] WARN - SourceHandler Illegal incoming
>> connection state: REQUEST_DONE . Possibly two send backs are happening for
>> the same request
>> [2013-08-15 16:57:28,709] WARN - SourceHandler Trying to write response
>> body while the handler is in an inconsistent state REQUEST_DONE
>> [2013-08-15 16:57:28,709] ERROR - TenantAwareLoadBalanceEndpoint No
>> application members available
>> [2013-08-15 16:57:28,710] ERROR - SourceHandler Unexpected I/O error:
>> java.nio.channels.ClosedChannelException
>> java.nio.channels.ClosedChannelException
>> at
>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:135)
>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:326)
>> at
>> org.apache.http.nio.reactor.ssl.SSLIOSession.sendEncryptedData(SSLIOSession.java:331)
>> at
>> org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:411)
>> at
>> org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:150)
>> at
>> org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:181)
>> at
>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:346)
>> at
>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>> at
>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>> at
>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>> at
>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> ELB tries to send a request to the management node but the application
>> member is not found, hence gets a ClosedChannelException.
>>
>> [1] https://wso2.org/jira/browse/LB-74
>> [2]
>> http://ec2-54-224-94-128.compute-1.amazonaws.com/05-08-2013/<http://www.google.com/url?q=http%3A%2F%2Fec2-54-224-94-128.compute-1.amazonaws.com%2F05-08-2013%2F&sa=D&sntz=1&usg=AFQjCNGFLcc1dySI1P2yUGSGveqEpgXIug>
>> http://ec2-54-224-94-128.compute-1.amazonaws.com/09-08-2013/
>>
>> Jeewantha.
>>
>>
>>
>> --
>> Jeewantha Dharmaparakrama
>> Software Engineer; WSO2, Inc.; http://wso2.com/
>> Phone : (+94) 774726790
>> Skype : prasad.jeewantha
>> LinkedIn : http://www.linkedin.com/in/jeewanthad
>> Twitter: https://twitter.com/jeewamp
>> Blog: http://jeewanthad.blogspot.com/
>>
>
>
>
> --
> *Afkham Azeez*
> Director of Architecture; WSO2, Inc.; http://wso2.com
> Member; Apache Software Foundation; http://www.apache.org/
> * <http://www.apache.org/>**
> email: **az...@wso2.com* <az...@wso2.com>* cell: +94 77 3320919
> blog: **http://blog.afkham.org* <http://blog.afkham.org>*
> twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez>
> *
> linked-in: **http://lk.linkedin.com/in/afkhamazeez*
> *
> *
> *Lean . Enterprise . Middleware*
>
--
Jeewantha Dharmaparakrama
Software Engineer; WSO2, Inc.; http://wso2.com/
Phone : (+94) 774726790
Skype : prasad.jeewantha
LinkedIn : http://www.linkedin.com/in/jeewanthad
Twitter: https://twitter.com/jeewamp
Blog: http://jeewanthad.blogspot.com/
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
