On Wed, Aug 21, 2013 at 7:39 AM, Sameera Jayasoma <same...@wso2.com> wrote:
> Can we have an update on the ELB issues?
>
This issue has been dragging for some time and its not been easy to fix it
cause the error is thrown from httpcore-nio runtime. While an HTTP request
being served, the connection gets closed abruptly and the server continues
to use a closed connection, hence the error is thrown. The problem is even
worse since its intermittent and cant get reproduced locally. This can
either be a bug in httpcore-nio or not be a bug at all. When a connection
is broken due to network inconsistencies, it is reasonable for the server
to throw an Exception. But at the same time, httocore can erroneously be
using a closed connection. We cant analyze the run time behaviour either
coz the issue does not get reproduced locally. Using EC2 setup used by QA
team for debugging purposes is not practical. So its a good idea if we can
have a clone of the Ec2 set-up for debugging purposes.
Jeewantha
>
> On Mon, Aug 19, 2013 at 2:41 PM, Jeewantha Dharmaparakrama <
> jeewan...@wso2.com> wrote:
>
>> Hi Evanthika,
>>
>> Can you kindly enable wire logs in the ELB in EC2 cluster and provide the
>> logs when this exception is thrown?
>>
>> Thanks,
>> Jeewantha.
>>
>>
>> On Fri, Aug 16, 2013 at 4:22 PM, Afkham Azeez <az...@wso2.com> wrote:
>>
>>> It was a configuration issue where the subDomain wasn't properly getting
>>> set.
>>>
>>>
>>> On Fri, Aug 16, 2013 at 1:07 PM, Jeewantha Dharmaparakrama <
>>> jeewan...@wso2.com> wrote:
>>>
>>>> Hi Folks,
>>>>
>>>> LB seems to be unstable when clustered with latest 4.2.0 kernal packs
>>>> [1]. Though this was first thought of as a PTT issue (see thread@dev -
>>>> "java.nio.channels.ClosedChannelException thrown time to time on ELB
>>>> console") there seems to be some inconstancy when nodes are joined to
>>>> the cluster which we are currently looking into.
>>>>
>>>> When a management node is added to the cluster, only the following log
>>>> message is printed in sout.
>>>>
>>>> INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} -
>>>> Added member: Host:127.0.0.1, Remote Host:null, Port: 3999, HTTP:-1,
>>>> HTTPS:-1, Domain: null, Sub-domain:null, Active:true
>>>>
>>>> This was checked with the packs built on 5th and 9th august [2] and as
>>>> per discussion with Azeez and Sameera, the products do not get clustered
>>>> properly. When management console is accessed, a SOAP Fault pops up on the
>>>> browser as
>>>>
>>>> <soapenv:Envelope><soapenv:
>>>> Body><nullResponse><Timestamp>Thu Aug 15 2013 16:55:05 GMT+0530
>>>> (IST)</Timestamp><Ack>Failure</Ack><Errors><ShortMessage>Gateway
>>>> Error</ShortMessage><LongMessage>No application members available
>>>> </LongMessage><ErrorCode>500
>>>> </ErrorCode><SeverityCode>Error</SeverityCode><ErrorClassification>RequestError</ErrorClassification></Errors><ServiceName>null</ServiceName><ResponseCode>null</ResponseCode><ContentType>null</ContentType><Version>1.5.1</Version></nullResponse></soapenv:Body></soapenv:Envelope>
>>>>
>>>> and with an Exception at the ELB backend
>>>>
>>>> [2013-08-15 16:55:05,407] ERROR - TenantAwareLoadBalanceEndpoint No
>>>> application members available
>>>> [2013-08-15 16:55:05,909] WARN - SourceHandler Illegal incoming
>>>> connection state: REQUEST_DONE . Possibly two send backs are happening for
>>>> the same request
>>>> [2013-08-15 16:55:05,909] WARN - SourceHandler Trying to write
>>>> response body while the handler is in an inconsistent state REQUEST_DONE
>>>> [2013-08-15 16:55:05,910] ERROR - TenantAwareLoadBalanceEndpoint No
>>>> application members available
>>>> [2013-08-15 16:57:28,353] ERROR - TenantAwareLoadBalanceEndpoint No
>>>> application members available
>>>> [2013-08-15 16:57:28,708] WARN - SourceHandler Illegal incoming
>>>> connection state: REQUEST_DONE . Possibly two send backs are happening for
>>>> the same request
>>>> [2013-08-15 16:57:28,709] WARN - SourceHandler Trying to write
>>>> response body while the handler is in an inconsistent state REQUEST_DONE
>>>> [2013-08-15 16:57:28,709] ERROR - TenantAwareLoadBalanceEndpoint No
>>>> application members available
>>>> [2013-08-15 16:57:28,710] ERROR - SourceHandler Unexpected I/O error:
>>>> java.nio.channels.ClosedChannelException
>>>> java.nio.channels.ClosedChannelException
>>>> at
>>>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:135)
>>>> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:326)
>>>> at
>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.sendEncryptedData(SSLIOSession.java:331)
>>>> at
>>>> org.apache.http.nio.reactor.ssl.SSLIOSession.outboundTransport(SSLIOSession.java:411)
>>>> at
>>>> org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:150)
>>>> at
>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:181)
>>>> at
>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:346)
>>>> at
>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
>>>> at
>>>> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
>>>> at
>>>> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
>>>> at
>>>> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> ELB tries to send a request to the management node but the application
>>>> member is not found, hence gets a ClosedChannelException.
>>>>
>>>> [1] https://wso2.org/jira/browse/LB-74
>>>> [2]
>>>> http://ec2-54-224-94-128.compute-1.amazonaws.com/05-08-2013/<http://www.google.com/url?q=http%3A%2F%2Fec2-54-224-94-128.compute-1.amazonaws.com%2F05-08-2013%2F&sa=D&sntz=1&usg=AFQjCNGFLcc1dySI1P2yUGSGveqEpgXIug>
>>>> http://ec2-54-224-94-128.compute-1.amazonaws.com/09-08-2013/
>>>>
>>>> Jeewantha.
>>>>
>>>>
>>>>
>>>> --
>>>> Jeewantha Dharmaparakrama
>>>> Software Engineer; WSO2, Inc.; http://wso2.com/
>>>> Phone : (+94) 774726790
>>>> Skype : prasad.jeewantha
>>>> LinkedIn : http://www.linkedin.com/in/jeewanthad
>>>> Twitter: https://twitter.com/jeewamp
>>>> Blog: http://jeewanthad.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> *Afkham Azeez*
>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>> Member; Apache Software Foundation; http://www.apache.org/
>>> * <http://www.apache.org/>**
>>> email: **az...@wso2.com* <az...@wso2.com>* cell: +94 77 3320919
>>> blog: **http://blog.afkham.org* <http://blog.afkham.org>*
>>> twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez>
>>> *
>>> linked-in: **http://lk.linkedin.com/in/afkhamazeez*
>>> *
>>> *
>>> *Lean . Enterprise . Middleware*
>>>
>>
>>
>>
>> --
>> Jeewantha Dharmaparakrama
>> Software Engineer; WSO2, Inc.; http://wso2.com/
>> Phone : (+94) 774726790
>> Skype : prasad.jeewantha
>> LinkedIn : http://www.linkedin.com/in/jeewanthad
>> Twitter: https://twitter.com/jeewamp
>> Blog: http://jeewanthad.blogspot.com/
>>
>
>
>
> --
> Sameera Jayasoma,
> Architect,
>
> WSO2, Inc. (http://wso2.com)
> email: same...@wso2.com
> blog: http://sameera.adahas.org
> twitter: https://twitter.com/sameerajayasoma
> flickr: http://www.flickr.com/photos/sameera-jayasoma/collections
>
>
> Lean . Enterprise . Middleware
>
--
Jeewantha Dharmaparakrama
Software Engineer; WSO2, Inc.; http://wso2.com/
Phone : (+94) 774726790
Skype : prasad.jeewantha
LinkedIn : http://www.linkedin.com/in/jeewanthad
Twitter: https://twitter.com/jeewamp
Blog: http://jeewanthad.blogspot.com/
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
