Pierre, thanks I will try to replicate the scenario to see what is
preventing CASes from timing out in the application client.
JC
On Thu, Mar 31, 2011 at 11:03 AM, Pierre Pujalon <ppuja...@yahoo.fr> wrote:
> JC, I have this timeout setted already then. The callback only receives
> the timeout exception for the first CAS.
> This only occurs when I have a delegated annotator.
> When I configure my application with aggregates without any delegate
> annotator, the callback receives timeout from all CASes.
>
> Pierre
>
>
> Le 31/03/2011 16:40, Jaroslaw Cwiklik a écrit :
>
> Pierre, to set a timeout on the aggregate do the following in the
>> application client:
>>
>> //create Asynchronous Client API
>> uimaAsEngine = new BaseUIMAAsynchronousEngine_impl();
>> //create Map to pass server URI and Endpoint parameters
>> Map<String,Object> appCtx = new HashMap<String,Object>();
>> ...
>> appCtx.put(UimaAsynchronousEngine.Timeout, 500); // 500ms timeout on
>> Process
>> CAS
>> uimaAsEngine.initialize(appCtx);
>> ...
>>
>> This should help with recovery of the CASes with invalid reply queue.
>> NOTE: if you listener plugged in to the uimaASEngine you will receive
>> callback with Timeout exception.
>> In your code you need to decide whether to retry CAS or not.
>>
>> JC
>>
>>
>> On Thu, Mar 31, 2011 at 9:55 AM, Pierre Pujalon<ppuja...@yahoo.fr>
>> wrote:
>>
>> JC
>>>
>>> I have a 15000ms process timeout for the UIMAEngine and I have a 15000ms
>>> timeout for the remoteDelegate
>>>
>>> I haven't found how to set a timeout for the aggregate to be honest.
>>>
>>> You are right I have this exception in the console
>>> javax.jms.InvalidDestinationException: Cannot publish to a deleted
>>> Destination: temp-queue://ID:${brokerUrl}-55450-1301578320924-0:0:1
>>> at
>>> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1696)
>>> at
>>>
>>> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:231)
>>> at
>>>
>>> org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:300)
>>> at
>>>
>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:574)
>>>
>>> at
>>>
>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:507)
>>> at
>>>
>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsOutputChannel.java:1366)
>>> at
>>>
>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemoteEndpoint(JmsOutputChannel.java:1527)
>>> at
>>>
>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendReply(JmsOutputChannel.java:689)
>>> at
>>>
>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.sendReplyToRemoteClient(AggregateAnalysisEngineController_impl.java:2033)
>>> at
>>>
>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.replyToClient(AggregateAnalysisEngineController_impl.java:2170)
>>> at
>>>
>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.finalStep(AggregateAnalysisEngineController_impl.java:1734)
>>> at
>>>
>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2317)
>>>
>>> at
>>>
>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.process(AggregateAnalysisEngineController_impl.java:1229)
>>> at
>>>
>>> org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.java:118)
>>> at
>>>
>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTimerAndProcess(ProcessResponseHandler.java:108)
>>> at
>>>
>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProcessResponseWithCASReference(ProcessResponseHandler.java:387)
>>> at
>>>
>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(ProcessResponseHandler.java:656)
>>> at
>>> org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:149)
>>> at
>>>
>>> org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle(ProcessRequestHandler_impl.java:973)
>>> at
>>>
>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessage(UimaVmMessageListener.java:107)
>>> at
>>>
>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(UimaVmMessageDispatcher.java:70)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> at
>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> at java.lang.Thread.run(Thread.java:619)
>>>
>>>
>>> Thanks
>>> Pierre
>>>
>>> Le 31/03/2011 15:43, Jaroslaw Cwiklik a écrit :
>>>
>>> Pierre, since the aggregate received the three CASes and these are never
>>>
>>>> returned to the application client I suspect that a temp reply queue
>>>> associated with the three CASes is no longer valid. Check your aggregate
>>>> log
>>>> for exceptions. Perhaps the aggregate is trying to return the CASes but
>>>> it
>>>> fails on send since the temp reply queue no longer exists. Just to be
>>>> clear
>>>> do you define Process timeout in your application client? It wasnt clear
>>>> to
>>>> me if the timeout of 15000ms is in the aggregate client or the
>>>> application
>>>> client.
>>>>
>>>> JC
>>>>
>>>> On Thu, Mar 31, 2011 at 9:24 AM, Pierre Pujalon<ppuja...@yahoo.fr>
>>>> wrote:
>>>>
>>>> JC,
>>>>
>>>>> I had to work on this topic and add instrumentation a bit to try to
>>>>> have
>>>>> a
>>>>> clearer vision. The basic idea is to test robustness of the processing
>>>>> when
>>>>> the ActiveMQ have a downtime (generally only for milliseconds). This
>>>>> scenario is a real life scenario.
>>>>>
>>>>> I have an application client using the aggregate client with
>>>>> UimaAsynchronousEngine. I have a pool of 4 CASes, a timeout set to
>>>>> 15000ms
>>>>> in the client and not CASMultiplier in the aggregate.
>>>>>
>>>>> When JMS gets down and up. I receive in my CallbackListener only one
>>>>> exception out of the four CASes (the 15000ms timeout) . The three
>>>>> others
>>>>> never trigger any timeout.
>>>>> Actually, it seems that thoses CASes are being processed in the
>>>>> aggregate
>>>>> as I have the trace of the annotators in the aggregate log. But they
>>>>> never
>>>>> go back to the reply queue.
>>>>>
>>>>> I traced the CASId that the engine sends and receives in the listener.
>>>>> Let's say I have id1, Id2, id3 and id4.
>>>>>
>>>>> As the JMS gets down and up, one of the case (id1) is released . The
>>>>> application client send another CAS . As this very point, the method
>>>>> sendCas
>>>>> returns a CASId id5 but in the CallbackListener the onBeforeMessageSend
>>>>> status tells me the CASid is id4. It is confirmed with the aggregate
>>>>> which
>>>>> logs the message :
>>>>> Duplicate Request With Cas Reference Id: (id4) Received. Ignoring
>>>>> Duplicate.
>>>>>
>>>>> The CasPool in the UIMAEngines is full and the client stalls.
>>>>> Restarting the UIMAAsynchronousEngine is the only solution to continue
>>>>> processing, the ag
>>>>> My guess is that it comes from the UIMAAsynchronousEngine.
>>>>> I hope it is clear
>>>>>
>>>>> Pierre
>>>>> PS : I set the timeout to 5 seconds, and I have a total different
>>>>> scenario
>>>>> which is no better :)
>>>>>
>>>>>
>>>>>
>>>>> Le 29/03/2011 18:56, Jaroslaw Cwiklik a écrit :
>>>>>
>>>>> Pierre, do you declare a timeout on Process in your aggregate
>>>>> deployment
>>>>>
>>>>> descriptor for the remote delegate? If not, perhaps CASes were sent to
>>>>>> the
>>>>>> remote delegate and subsequently dropped because the delegate was not
>>>>>> unable
>>>>>> to deliver them to the aggregate client. A timeout would help to
>>>>>> recover
>>>>>> these CASes.
>>>>>>
>>>>>> Do you know where you are running out of CASes? Is it in application
>>>>>> client
>>>>>> or aggregate client? Use jConsole to attach to each jvm and examine
>>>>>> the
>>>>>> threads. Do you use a Cas Multiplier in the aggregate? How big is its
>>>>>> pool?
>>>>>>
>>>>>> -JC
>>>>>>
>>>>>> On Tue, Mar 29, 2011 at 12:03 PM, Pierre Pujalon<ppuja...@yahoo.fr>
>>>>>> wrote:
>>>>>>
>>>>>> JC,
>>>>>>
>>>>>> I tested the work around and a listener to the delegate is indeed
>>>>>>> recreated. The CAS that were being processed pass through all
>>>>>>> annotators.
>>>>>>> But now I have other issues, but it is not the same each time I tried
>>>>>>> .Either the client is losing connection, or get block on the getCAS
>>>>>>> method
>>>>>>> as if cas process during the JMS down time were not release or the
>>>>>>> aggregate
>>>>>>> or the aggregate client get stalled...
>>>>>>> I need to dig a little bit into this because it does not make many
>>>>>>> sense
>>>>>>> right now...
>>>>>>>
>>>>>>> What bother me is that if I don't make the annotator delegated, the
>>>>>>> aggregate client always recover clean with this scenario.
>>>>>>>
>>>>>>> Anyway Thanks
>>>>>>> Pierre
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Le 28/03/2011 18:07, Jaroslaw Cwiklik a écrit :
>>>>>>>
>>>>>>> Pierre, in my last posting I meant maxRetries=1, not retryCount=1.
>>>>>>>
>>>>>>> JC
>>>>>>>
>>>>>>>> On Mon, Mar 28, 2011 at 12:05 PM, Jaroslaw Cwiklik<uim...@gmail.com
>>>>>>>> >
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Pierre, I've identified the bug in UIMA AS code that causes the
>>>>>>>> problem
>>>>>>>>
>>>>>>>> described. As a workaround, set the retryCount=1 in the error
>>>>>>>>
>>>>>>>>> handling
>>>>>>>>> configuration for the delegate defined in your deployment
>>>>>>>>> descriptor.
>>>>>>>>> Here
>>>>>>>>> is an example:
>>>>>>>>>
>>>>>>>>> <remoteAnalysisEngine key="NoOp">
>>>>>>>>> <inputQueue endpoint="NoOpAnnotatorQueue"
>>>>>>>>> brokerURL="${BrokerURL}"/>
>>>>>>>>> <serializer method="xmi"/>
>>>>>>>>> <asyncAggregateErrorConfiguration>
>>>>>>>>> <processCasErrors maxRetries="1"
>>>>>>>>> timeout="15000" thresholdCount="1" thresholdAction="terminate" />
>>>>>>>>> </asyncAggregateErrorConfiguration>
>>>>>>>>> </remoteAnalysisEngine>
>>>>>>>>>
>>>>>>>>> This should force creation of a new listener on the delegate reply
>>>>>>>>> queue.
>>>>>>>>>
>>>>>>>>> The listener recovery should be done with no retry being set and I
>>>>>>>>> will
>>>>>>>>> make necessary changes in the code to support that use case.
>>>>>>>>>
>>>>>>>>> Thanks for finding the bug!
>>>>>>>>>
>>>>>>>>> JC
>>>>>>>>>
>>>>>>>>> On Tue, Mar 22, 2011 at 1:12 PM, Jaroslaw Cwiklik<uim...@gmail.com
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Pierre, this appears to be a bug. In the scenario you've
>>>>>>>>>> described
>>>>>>>>>>
>>>>>>>>> the
>>>>>>>>>
>>>>>>>>> uima as aggregate client should detect a lost connection (which it
>>>>>>>>>
>>>>>>>>>> does)
>>>>>>>>>> and
>>>>>>>>>> create a new one on a subsequent process CAS request destined to
>>>>>>>>>> the
>>>>>>>>>> remote
>>>>>>>>>> delegate. I will attempt to re-create your scenario and see if
>>>>>>>>>> this
>>>>>>>>>> is
>>>>>>>>>> in
>>>>>>>>>> fact a bug.
>>>>>>>>>>
>>>>>>>>>> Thanks, JC
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Mar 22, 2011 at 7:11 AM, Pierre Pujalon<ppuja...@yahoo.fr
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>> I'm currently using UIMA-AS 2.3.1rc5 and I am having issues with
>>>>>>>>>>
>>>>>>>>>>> asynchronous delegated ..
>>>>>>>>>>>
>>>>>>>>>>> I have a chain of several annotators. One of them is an
>>>>>>>>>>> asynchronous
>>>>>>>>>>> delegate as it is very resource consuming.
>>>>>>>>>>> So I have one UIMA service with a simpleChainDescriptor and
>>>>>>>>>>> several
>>>>>>>>>>> UIMA
>>>>>>>>>>> services with myDelegateAnnotatorDescriptor.
>>>>>>>>>>>
>>>>>>>>>>> When the activeMQ broker fails and restart, all my UIMA services
>>>>>>>>>>> resfresh succesfully their JMS connection. But it appears that
>>>>>>>>>>> the
>>>>>>>>>>> service
>>>>>>>>>>> holding the processing chain (simpleChainDescriptor) itself does
>>>>>>>>>>> not
>>>>>>>>>>> refresh the JMS connection to the delegate JMS queue.
>>>>>>>>>>> When a new CAS is processed, I have the following exception in
>>>>>>>>>>> the
>>>>>>>>>>> console !!!! and the processing of the cas stop silently and
>>>>>>>>>>> passes
>>>>>>>>>>> to
>>>>>>>>>>> the
>>>>>>>>>>> next cas.
>>>>>>>>>>> If I restart the service holding the chain, everything go back
>>>>>>>>>>> to
>>>>>>>>>>> normal.
>>>>>>>>>>>
>>>>>>>>>>> Is this a bug or am I missing something?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Pierre
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ATTENTION: Uima AS Service:simpleChainDescriptor Listener
>>>>>>>>>>> Established
>>>>>>>>>>> Connection to Broker: tcp://brokerurl:61616
>>>>>>>>>>> 22 mars 2011 11:11:00
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer
>>>>>>>>>>> handleListenerSetupFailure
>>>>>>>>>>> ATTENTION: Uima AS Service:simpleChainDescriptor Listener
>>>>>>>>>>> Established
>>>>>>>>>>> Connection to Broker: tcp://brokerurl:61616
>>>>>>>>>>> 22 mars 2011 11:11:00
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel
>>>>>>>>>>> invalidateConnectionAndEndpoints
>>>>>>>>>>> INFO: Controller simpleChainDescriptor Invalidating JMS
>>>>>>>>>>> Connection
>>>>>>>>>>> To
>>>>>>>>>>> Broker tcp://brokerurl:61616 and Closing Sessions To Delegates
>>>>>>>>>>> 22 mars 2011 11:11:01
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl
>>>>>>>>>>> send
>>>>>>>>>>> ATTENTION: Service: simpleChainDescriptor Runtime Exception
>>>>>>>>>>> 22 mars 2011 11:11:01
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl
>>>>>>>>>>> send
>>>>>>>>>>> ATTENTION:
>>>>>>>>>>> org.apache.uima.aae.error.DelegateConnectionLostException:
>>>>>>>>>>> Controller:simpleChainDescriptor Lost Connection to
>>>>>>>>>>> Delegate:myDelegateAnnotatorDescriptor
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:536)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:507)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsOutputChannel.java:1366)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemoteEndpoint(JmsOutputChannel.java:1527)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.serializeCasAndSend(JmsOutputChannel.java:658)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendRequest(JmsOutputChannel.java:610)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatch(AggregateAnalysisEngineController_impl.java:2381)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatchProcessRequest(AggregateAnalysisEngineController_impl.java:2421)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.simpleStep(AggregateAnalysisEngineController_impl.java:1294)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2302)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.process(AggregateAnalysisEngineController_impl.java:1229)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.java:118)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTimerAndProcess(ProcessResponseHandler.java:108)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProcessResponseWithCASReference(ProcessResponseHandler.java:387)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(ProcessResponseHandler.java:656)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:149)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle(ProcessRequestHandler_impl.java:973)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessage(UimaVmMessageListener.java:107)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(UimaVmMessageDispatcher.java:70)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>>>>> 22 mars 2011 11:11:01
>>>>>>>>>>> org.apache.uima.aae.error.handler.ProcessCasErrorHandler
>>>>>>>>>>> handleError
>>>>>>>>>>> ATTENTION: Service: simpleChainDescriptor Runtime Exception
>>>>>>>>>>> 22 mars 2011 11:11:01
>>>>>>>>>>> org.apache.uima.aae.error.handler.ProcessCasErrorHandler
>>>>>>>>>>> handleError
>>>>>>>>>>> ATTENTION:
>>>>>>>>>>> org.apache.uima.aae.error.DelegateConnectionLostException:
>>>>>>>>>>> Controller:simpleChainDescriptor Lost Connection to
>>>>>>>>>>> Delegate:myDelegateAnnotatorDescriptor
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:536)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:507)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsOutputChannel.java:1366)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemoteEndpoint(JmsOutputChannel.java:1527)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.serializeCasAndSend(JmsOutputChannel.java:658)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendRequest(JmsOutputChannel.java:610)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatch(AggregateAnalysisEngineController_impl.java:2381)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatchProcessRequest(AggregateAnalysisEngineController_impl.java:2421)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.simpleStep(AggregateAnalysisEngineController_impl.java:1294)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2302)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.process(AggregateAnalysisEngineController_impl.java:1229)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.java:118)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTimerAndProcess(ProcessResponseHandler.java:108)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProcessResponseWithCASReference(ProcessResponseHandler.java:387)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(ProcessResponseHandler.java:656)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:149)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle(ProcessRequestHandler_impl.java:973)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessage(UimaVmMessageListener.java:107)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(UimaVmMessageDispatcher.java:70)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
