> > Hi, > > Hi Isuru, Please find my comments below..
> This is regarding the analytics for US Election 2016 Tweets. The ESB uses > Twitter Connector to find tweets with some specific hashtags and sends the > tweets as events to CEP via HTTP. The CEP version is 4.0.0. > > The CEP receives the events via the Tomcat HTTP Connector (port 9763). As > mentioned in $subject, the CEP fails to accept requests as there are no > worker threads to handle the requests. > > I have attached a thread dump and I analyzed it using the ThreadLogic > application [1]. All 250 HTTP worker threads (http-nio-9763-exec-*) are in > "PARKING" state and following is a part of stack trace in each thread. > > "http-nio-9763-exec-102" #810 daemon prio=5 os_prio=0 > tid=0x00007f7678010800 nid=0x710 waiting on condition [0x00007f75da48f000] > *java.lang.Thread.State: WAITING (parking)* > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000005cc6bd6a8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > at > java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) > > > > > * at > org.wso2.carbon.event.input.adapter.http.HTTPEventAdapter$1.rejectedExecution(HTTPEventAdapter.java:99) > at > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) > at > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) > at > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) > at > org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet.doPost(HTTPMessageServlet.java:177)* > at javax.servlet.http.HttpServlet.service(HttpServlet.java:646) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > at > org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61) > > > The HTTPMessageServlet submits a HTTPRequestProcessor [2] to an executor > service and the executor service rejects it as the worker queue is full. > However in the RejectedExecutionHandler, the task is put back to same > queue [3]. Here the thread gets parked while waiting for some condition. > This is why the Tomcat HTTP connector can no longer process any requests. > > Yes, above implementation is done purposefully.. We had a requirement where we need to block the HTTP requests when there is no thread (or space in the queue) rather dropping the events. That is why above HTTP adapter is implemented in such way.. Because of above implementation, there will not be an event loss at receiver level. > Then I checked the thread pool used in [2] and found out that all 100 > threads (pool-75-thread-*) in that pool are also in "PARKING" state. > Following is the stack trace. > > > "pool-75-thread-100" #758 prio=5 os_prio=0 tid=0x00007f7634017800 > nid=0x6ba runnable [0x00007f75dd8c4000] > * java.lang.Thread.State: TIMED_WAITING (parking)* > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338) > at > com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136) > at > com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105) > at *com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)* > at > *org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:74)* > at > org.wso2.siddhi.core.stream.input.SingleStreamEntryValve.send(SingleStreamEntryValve.java:99) > at > org.wso2.siddhi.core.stream.input.InputHandler.send(InputHandler.java:49) > at > org.wso2.carbon.event.processor.core.internal.listener.SiddhiInputEventDispatcher.sendEvent(SiddhiInputEventDispatcher.java:39) > at > org.wso2.carbon.event.processor.core.internal.listener.AbstractSiddhiInputEventDispatcher.consumeEvent(AbstractSiddhiInputEventDispatcher.java:92) > at > org.wso2.carbon.event.stream.core.internal.EventJunction.sendEvent(EventJunction.java:142) > at > org.wso2.carbon.event.receiver.core.internal.management.InputEventDispatcher.onEvent(InputEventDispatcher.java:27) > at > org.wso2.carbon.event.receiver.core.internal.EventReceiver.sendEvent(EventReceiver.java:256) > at > org.wso2.carbon.event.receiver.core.internal.EventReceiver.processMappedEvent(EventReceiver.java:200) > at > org.wso2.carbon.event.receiver.core.internal.EventReceiver$MappedEventSubscription.onEvent(EventReceiver.java:307) > at > org.wso2.carbon.event.input.adapter.core.internal.InputAdapterRuntime.onEvent(InputAdapterRuntime.java:109) > at > org.wso2.carbon.event.input.adapter.http.HTTPMessageServlet$HTTPRequestProcessor.run(HTTPMessageServlet.java:210) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > > The call to RingBuffer.next method in SingleStreamEntryValve.send [4] > puts the thread into the PARKING state. > > We need to figure out the reason for this. I really appreciate if you > could let us know what we should check further and how to avoid this kind > of behaviour with CEP. > Now, let me explain about this issue.. In above usecase CEP is not running with default functionalities or components.. Above CEP instance contains few extensions (which is extended).. When analyzing the issue, found that a specific extension is causing this behavior.. That extension consumes considerable amount of processing time.. And it eventually become as the reason for distributor to get full. Removing the extension solves the issue.. It seems, above extension is a must for the usecase. Then, I have asked to divide the execution plan in to few parts to reduce the load for the distruptor (and to reduce the processing time of the single execution plan). After this change, system is back to normal and haven't seen such behavior.. Let's monitor the system over the weekend and see... > > I think it's better to include some metrics in CEP to see how the CEP > engine is behaving, especially some metrics related to Disruptor. Do we > have such metrics in the latest CEP release? > > No, we don't have such metrics specifically for Distruptor.. But, I believe we can use Siddhi metrics.. Thanks, Mohan > Thanks! > > Best Regards, > > [1] https://java.net/projects/threadlogic > [2] > https://github.com/wso2/carbon-analytics-common/blob/v5.0.3/components/event-receiver/event-input-adapters/org.wso2.carbon.event.input.adapter.http/src/main/java/org/wso2/carbon/event/input/adapter/http/HTTPMessageServlet.java#L177 > [3] > https://github.com/wso2/carbon-analytics-common/blob/v5.0.3/components/event-receiver/event-input-adapters/org.wso2.carbon.event.input.adapter.http/src/main/java/org/wso2/carbon/event/input/adapter/http/HTTPEventAdapter.java#L99 > [4] > https://github.com/wso2/siddhi/blob/v3.0.2/modules/siddhi-core/src/main/java/org/wso2/siddhi/core/stream/input/SingleStreamEntryValve.java#L74 > > -- > Isuru Perera > Associate Technical Lead | WSO2, Inc. | http://wso2.com/ > Lean . Enterprise . Middleware > > about.me/chrishantha > Contact: +IsuruPereraWSO2 <https://www.google.com/+IsuruPereraWSO2/about> > -- *V. Mohanadarshan* *Senior Software Engineer,* *Data Technologies Team,* *WSO2, Inc. http://wso2.com <http://wso2.com> * *lean.enterprise.middleware.* email: mo...@wso2.com phone:(+94) 771117673
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
