We have 2 remote ESB EI 6.4.0 instances in a cluster sending datas to analytics
worker.
Here is what I can reproduce
> start Analytics worker (EI package 6.4.0) with options -Xms6G -Xmx6G
I tested the worker with both java versions "1.8.0_162" and Java(TM) SE
Runtime Environment (build 1.8.0_192-ea-b02)
> first I send some very low data trafic to ESBs … analytics CPU load keeps
> very low -> ok
> (repeat this 4 or 5 times) sending some more datas to EI-6.4.0 ESBs;
> analytics CPU load grows to 100% when receiving datas from ESBs then go back
> to very low when the trafic is stopped —> pretty good
Running jstat with interval 60s:
Timestamp S0 S1 E O M CCS YGC YGCT FGC
FGCT GCT
36.8 100.00 0.00 31.50 5.00 88.04 80.26 4 0.648 3
0.306 0.955
96.9 100.00 0.00 38.30 5.00 88.04 80.26 4 0.648 3
0.306 0.955
156.9 100.00 0.00 40.12 5.00 88.04 80.26 4 0.648 3
0.306 0.955
> now sending lot of trafic to EI-6.4.0
> analytics CPU load grows to 100%
> works ok for some minutes
> then problems are coming: first, the drift connection is lost on both EI
> ESB (the worker no more responding)
[2019-03-14 10:47:35,457] [-1] []
[DataBridge-ReconnectionService-pool-5-thread-1] WARN
{org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup} - No receiver is
reachable at reconnection, will try to reconnect every 30 sec
[2019-03-14 10:48:58,795] [-1] []
[DataBridge-tcp://10.1.3.12:7612-pool-7-thread-1] ERROR
{org.wso2.carbon.databridge.agent.endpoint.DataEndpoint} - Unable to send
events to the endpoint.
org.wso2.carbon.databridge.agent.exception.DataEndpointException: Cannot send
Events
at
org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.send(ThriftDataEndpoint.java:88)
at
org.wso2.carbon.databridge.agent.endpoint.DataEndpoint$EventPublisher.publish(DataEndpoint.java:314)
> I stop the trafic to EI-6.4.0, analytics CPU load do NOT decrease anymore,
> it stays around 100%
[cid:ACBD3C07-86BE-48B2-B648-3DAF9DB58DCF]
> ESB cannot reconnect anymore to the worker
[2019-03-14 11:14:06,337] [-1] []
[DataBridge-ReconnectionService-pool-5-thread-1] WARN
{org.wso2.carbon.databridge.agent.endpoint.DataEndpointGroup} - No receiver is
reachable at reconnection, will try to reconnect every 30 sec
[2019-03-14 11:14:36,384] [-1] []
[DataBridge-tcp://10.1.3.12:7612-pool-7-thread-1] ERROR
{org.wso2.carbon.databridge.agent.endpoint.DataEndpoint} - Unable to send
events to the endpoint.
org.wso2.carbon.databridge.agent.exception.DataEndpointException: Cannot send
Events
at
org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.send(ThriftDataEndpoint.java:88)
at
org.wso2.carbon.databridge.agent.endpoint.DataEndpoint$EventPublisher.publish(DataEndpoint.java:314)
at
org.wso2.carbon.databridge.agent.endpoint.DataEndpoint$EventPublisher.run(DataEndpoint.java:272)
at java.lang.Thread.run(Thread.java:748)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketTimeoutException: Read timed out
In this ugling state jstat with interval 60s gives
Timestamp S0 S1 E O M CCS YGC YGCT FGC
FGCT GCT
4837.0 0.00 9.22 10.78 45.72 88.60 80.09 27 10.479 3
0.306 10.785
4897.0 0.00 9.22 37.64 45.72 88.60 80.09 27 10.479 3
0.306 10.785
4957.0 0.00 9.22 59.97 45.72 88.60 80.09 27 10.479 3
0.306 10.785
5017.0 0.00 9.22 85.25 45.72 88.60 80.09 27 10.479 3
0.306 10.785
5077.0 9.60 0.00 9.03 47.27 88.65 80.09 28 10.728 3
0.306 11.034
5137.0 9.60 0.00 32.90 47.27 88.65 80.09 28 10.728 3
0.306 11.034
5197.0 9.60 0.00 55.66 47.27 88.65 80.09 28 10.728 3
0.306 11.034
5257.0 9.60 0.00 76.92 47.27 88.65 80.09 28 10.728 3
0.306 11.034
Reading this JSTAT information I understand the GC is not working so much that
it could explain the 100% CPU load.
The drift connections from both ESB keep lost, BUT the interface at
https://worker_host:7443/stores/query' keeps responding (slow but it is
responding)
Bernard
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
