Hi Asanka,
I too have observed this behavior before. So the AS clustering ports get
opened early and joins with the ELB and the rest of the cluster before the
server is fully initialized.
-Nirodha
On Wed, Sep 17, 2014 at 7:16 AM, Asanka Vithanage <asan...@wso2.com> wrote:
>
> Hi All,
>
> I am currently testing on a ELB 211 fronted 1 manager/2 worker AS 521
> cluster. Here when i am following below steps, i am getting below noted
> errors on ELB stack traces.
>
>
> *Steps:*
>
> Setup a 1 manager/2 worker node ELB fronted setup
>
> Invoke SimpleStockQuate service deployed on AS 521(http path) continuously
> using jmeter script(with 1000 jmeter threads and 10 seconds ramp-up period)
>
> Down the worker node using ./wso2server.sh stop command
>
> Startup the downed worker node and check the ELB logs
>
> *Issue:*
>
> Intermittently observed following issue on ELB stack trace, but AS stack
> traces does not show any thing except member left/member join messages
>
>
> Note: I have tried above steps nearly 10 times and out of them 3 times
> observed below error.[Attached ELB log for further reference]
>
> TID: [0] [ELB] [2014-09-17 11:13:55,107] INFO
> {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} -
> Member left [aa7c330e-eb71-488c-a905-79c3a07ba671]: /10.0.0.85:4300
> {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent}
>
> TID: [0] [ELB] [2014-09-17 11:13:55,122] INFO
> {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} -
> Member left [aa7c330e-eb71-488c-a905-79c3a07ba671]: /10.0.0.85:4300
> {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent}
>
> TID: [0] [ELB] [2014-09-17 11:16:12,293] INFO
> {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} -
> Member joined [cd2f1123-f734-456f-bc3f-7ae03ec0142a]: /10.0.0.85:4300
> {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent}
>
> TID: [0] [ELB] [2014-09-17 11:16:12,297] INFO
> {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} -
> Member joined [cd2f1123-f734-456f-bc3f-7ae03ec0142a]: /10.0.0.85:4300
> {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent}
>
> TID: [0] [ELB] [2014-09-17 11:16:15,423] INFO
> {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} -
> Application member Host:10.0.0.85, Remote Host:null, Port: 4300,
> HTTP:9765, HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker,
> Active:true joined application cluster
> {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent}
>
> TID: [0] [ELB] [2014-09-17 11:16:16,139] WARN
> {org.apache.synapse.transport.passthru.TargetHandler} - Connection closed
> by target host before receiving the request
> {org.apache.synapse.transport.passthru.TargetHandler}
>
> TID: [0] [ELB] [2014-09-17 11:16:16,169] INFO
> {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
> Failed to send message to Member Host:10.0.0.85, Remote Host:null, Port:
> 4300, HTTP:9765, HTTPS:9445, Domain: wso2.as.domain, Sub-domain:worker,
> Active:true . Error Code: 101508
> {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
>
> TID: [0] [ELB] [2014-09-17 11:16:16,170] ERROR
> {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint} -
> Error sending request! Connection timeout after request is read by the
> host as.wso2.com . Error code: 101508 Protocol State: RESPONSE HEAD
> {org.wso2.carbon.lb.endpoint.endpoint.TenantAwareLoadBalanceEndpoint}
>
>
>
> After analyzing AS and ELB stack-traces, i felt that AS worker nodes join
> the cluster before it properly initialized. So when ELB handover the job to
> AS worker nodes, It fails to do the job.
>
>
> Does my observation make sense here or is there any other reason for this
> behavior?
>
>
> WDYT?
>
>
>
> --
> Asanka Vithanage
> Senior Software Engineer -QA
> Mobile: +94 0716286708
> Email: asan...@wso2.com
> WSO2 Inc. www.wso2.com
>
>
> _______________________________________________
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>
--
*Nirodha Gallage*
Senior Software Engineer, QA.
WSO2 Inc.: http://wso2.com/
Mobile: +94716429078
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
