Pierre, do you declare a timeout on Process in your aggregate deployment
descriptor for the remote delegate? If not, perhaps CASes were sent to the
remote delegate and subsequently dropped because the delegate was not unable
to deliver them to the aggregate client. A timeout would help to recover
these CASes.
Do you know where you are running out of CASes? Is it in application client
or aggregate client? Use jConsole to attach to each jvm and examine the
threads. Do you use a Cas Multiplier in the aggregate? How big is its pool?
-JC
On Tue, Mar 29, 2011 at 12:03 PM, Pierre Pujalon <ppuja...@yahoo.fr> wrote:
> JC,
> I tested the work around and a listener to the delegate is indeed
> recreated. The CAS that were being processed pass through all annotators.
> But now I have other issues, but it is not the same each time I tried
> .Either the client is losing connection, or get block on the getCAS method
> as if cas process during the JMS down time were not release or the aggregate
> or the aggregate client get stalled...
> I need to dig a little bit into this because it does not make many sense
> right now...
>
> What bother me is that if I don't make the annotator delegated, the
> aggregate client always recover clean with this scenario.
>
> Anyway Thanks
> Pierre
>
>
>
> Le 28/03/2011 18:07, Jaroslaw Cwiklik a écrit :
>
> Pierre, in my last posting I meant maxRetries=1, not retryCount=1.
>>
>> JC
>>
>> On Mon, Mar 28, 2011 at 12:05 PM, Jaroslaw Cwiklik<uim...@gmail.com>
>> wrote:
>>
>> Pierre, I've identified the bug in UIMA AS code that causes the problem
>>> described. As a workaround, set the retryCount=1 in the error handling
>>> configuration for the delegate defined in your deployment descriptor.
>>> Here
>>> is an example:
>>>
>>> <remoteAnalysisEngine key="NoOp">
>>> <inputQueue endpoint="NoOpAnnotatorQueue"
>>> brokerURL="${BrokerURL}"/>
>>> <serializer method="xmi"/>
>>> <asyncAggregateErrorConfiguration>
>>> <processCasErrors maxRetries="1"
>>> timeout="15000" thresholdCount="1" thresholdAction="terminate" />
>>> </asyncAggregateErrorConfiguration>
>>> </remoteAnalysisEngine>
>>>
>>> This should force creation of a new listener on the delegate reply queue.
>>>
>>> The listener recovery should be done with no retry being set and I will
>>> make necessary changes in the code to support that use case.
>>>
>>> Thanks for finding the bug!
>>>
>>> JC
>>>
>>> On Tue, Mar 22, 2011 at 1:12 PM, Jaroslaw Cwiklik<uim...@gmail.com
>>> >wrote:
>>>
>>> Pierre, this appears to be a bug. In the scenario you've described the
>>>> uima as aggregate client should detect a lost connection (which it does)
>>>> and
>>>> create a new one on a subsequent process CAS request destined to the
>>>> remote
>>>> delegate. I will attempt to re-create your scenario and see if this is
>>>> in
>>>> fact a bug.
>>>>
>>>> Thanks, JC
>>>>
>>>>
>>>> On Tue, Mar 22, 2011 at 7:11 AM, Pierre Pujalon<ppuja...@yahoo.fr
>>>> >wrote:
>>>>
>>>> Hi,
>>>>> I'm currently using UIMA-AS 2.3.1rc5 and I am having issues with
>>>>> asynchronous delegated ..
>>>>>
>>>>> I have a chain of several annotators. One of them is an asynchronous
>>>>> delegate as it is very resource consuming.
>>>>> So I have one UIMA service with a simpleChainDescriptor and several
>>>>> UIMA
>>>>> services with myDelegateAnnotatorDescriptor.
>>>>>
>>>>> When the activeMQ broker fails and restart, all my UIMA services
>>>>> resfresh succesfully their JMS connection. But it appears that the
>>>>> service
>>>>> holding the processing chain (simpleChainDescriptor) itself does not
>>>>> refresh the JMS connection to the delegate JMS queue.
>>>>> When a new CAS is processed, I have the following exception in the
>>>>> console !!!! and the processing of the cas stop silently and passes to
>>>>> the
>>>>> next cas.
>>>>> If I restart the service holding the chain, everything go back to
>>>>> normal.
>>>>>
>>>>> Is this a bug or am I missing something?
>>>>>
>>>>> Thanks,
>>>>> Pierre
>>>>>
>>>>>
>>>>> ATTENTION: Uima AS Service:simpleChainDescriptor Listener Established
>>>>> Connection to Broker: tcp://brokerurl:61616
>>>>> 22 mars 2011 11:11:00
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer
>>>>> handleListenerSetupFailure
>>>>> ATTENTION: Uima AS Service:simpleChainDescriptor Listener Established
>>>>> Connection to Broker: tcp://brokerurl:61616
>>>>> 22 mars 2011 11:11:00
>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel
>>>>> invalidateConnectionAndEndpoints
>>>>> INFO: Controller simpleChainDescriptor Invalidating JMS Connection To
>>>>> Broker tcp://brokerurl:61616 and Closing Sessions To Delegates
>>>>> 22 mars 2011 11:11:01
>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl send
>>>>> ATTENTION: Service: simpleChainDescriptor Runtime Exception
>>>>> 22 mars 2011 11:11:01
>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl send
>>>>> ATTENTION:
>>>>> org.apache.uima.aae.error.DelegateConnectionLostException:
>>>>> Controller:simpleChainDescriptor Lost Connection to
>>>>> Delegate:myDelegateAnnotatorDescriptor
>>>>> at
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:536)
>>>>> at
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:507)
>>>>> at
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsOutputChannel.java:1366)
>>>>> at
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemoteEndpoint(JmsOutputChannel.java:1527)
>>>>> at
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.serializeCasAndSend(JmsOutputChannel.java:658)
>>>>> at
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendRequest(JmsOutputChannel.java:610)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatch(AggregateAnalysisEngineController_impl.java:2381)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatchProcessRequest(AggregateAnalysisEngineController_impl.java:2421)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.simpleStep(AggregateAnalysisEngineController_impl.java:1294)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2302)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.process(AggregateAnalysisEngineController_impl.java:1229)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.java:118)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTimerAndProcess(ProcessResponseHandler.java:108)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProcessResponseWithCASReference(ProcessResponseHandler.java:387)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(ProcessResponseHandler.java:656)
>>>>> at
>>>>> org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:149)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle(ProcessRequestHandler_impl.java:973)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessage(UimaVmMessageListener.java:107)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(UimaVmMessageDispatcher.java:70)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>> 22 mars 2011 11:11:01
>>>>> org.apache.uima.aae.error.handler.ProcessCasErrorHandler handleError
>>>>> ATTENTION: Service: simpleChainDescriptor Runtime Exception
>>>>> 22 mars 2011 11:11:01
>>>>> org.apache.uima.aae.error.handler.ProcessCasErrorHandler handleError
>>>>> ATTENTION:
>>>>> org.apache.uima.aae.error.DelegateConnectionLostException:
>>>>> Controller:simpleChainDescriptor Lost Connection to
>>>>> Delegate:myDelegateAnnotatorDescriptor
>>>>> at
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:536)
>>>>> at
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:507)
>>>>> at
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsOutputChannel.java:1366)
>>>>> at
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemoteEndpoint(JmsOutputChannel.java:1527)
>>>>> at
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.serializeCasAndSend(JmsOutputChannel.java:658)
>>>>> at
>>>>>
>>>>> org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendRequest(JmsOutputChannel.java:610)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatch(AggregateAnalysisEngineController_impl.java:2381)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatchProcessRequest(AggregateAnalysisEngineController_impl.java:2421)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.simpleStep(AggregateAnalysisEngineController_impl.java:1294)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2302)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.process(AggregateAnalysisEngineController_impl.java:1229)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.java:118)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTimerAndProcess(ProcessResponseHandler.java:108)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProcessResponseWithCASReference(ProcessResponseHandler.java:387)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(ProcessResponseHandler.java:656)
>>>>> at
>>>>> org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:149)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle(ProcessRequestHandler_impl.java:973)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessage(UimaVmMessageListener.java:107)
>>>>> at
>>>>>
>>>>> org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(UimaVmMessageDispatcher.java:70)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at
>>>>>
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>> at java.lang.Thread.run(Thread.java:619)
>>>>>
>>>>>
>>>>>
>>>>>
