JC, I have this timeout setted already then. The callback only receives
the timeout exception for the first CAS.
This only occurs when I have a delegated annotator.
When I configure my application with aggregates without any delegate
annotator, the callback receives timeout from all CASes.
Pierre
Le 31/03/2011 16:40, Jaroslaw Cwiklik a écrit :
Pierre, to set a timeout on the aggregate do the following in the
application client:
//create Asynchronous Client API
uimaAsEngine = new BaseUIMAAsynchronousEngine_impl();
//create Map to pass server URI and Endpoint parameters
Map<String,Object> appCtx = new HashMap<String,Object>();
...
appCtx.put(UimaAsynchronousEngine.Timeout, 500); // 500ms timeout on Process
CAS
uimaAsEngine.initialize(appCtx);
...
This should help with recovery of the CASes with invalid reply queue.
NOTE: if you listener plugged in to the uimaASEngine you will receive
callback with Timeout exception.
In your code you need to decide whether to retry CAS or not.
JC
On Thu, Mar 31, 2011 at 9:55 AM, Pierre Pujalon<ppuja...@yahoo.fr> wrote:
JC
I have a 15000ms process timeout for the UIMAEngine and I have a 15000ms
timeout for the remoteDelegate
I haven't found how to set a timeout for the aggregate to be honest.
You are right I have this exception in the console
javax.jms.InvalidDestinationException: Cannot publish to a deleted
Destination: temp-queue://ID:${brokerUrl}-55450-1301578320924-0:0:1
at
org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1696)
at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:231)
at
org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:300)
at
org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:574)
at
org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:507)
at
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsOutputChannel.java:1366)
at
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemoteEndpoint(JmsOutputChannel.java:1527)
at
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendReply(JmsOutputChannel.java:689)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.sendReplyToRemoteClient(AggregateAnalysisEngineController_impl.java:2033)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.replyToClient(AggregateAnalysisEngineController_impl.java:2170)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.finalStep(AggregateAnalysisEngineController_impl.java:1734)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2317)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.process(AggregateAnalysisEngineController_impl.java:1229)
at
org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.java:118)
at
org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTimerAndProcess(ProcessResponseHandler.java:108)
at
org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProcessResponseWithCASReference(ProcessResponseHandler.java:387)
at
org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(ProcessResponseHandler.java:656)
at
org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:149)
at
org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle(ProcessRequestHandler_impl.java:973)
at
org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessage(UimaVmMessageListener.java:107)
at
org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(UimaVmMessageDispatcher.java:70)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Thanks
Pierre
Le 31/03/2011 15:43, Jaroslaw Cwiklik a écrit :
Pierre, since the aggregate received the three CASes and these are never
returned to the application client I suspect that a temp reply queue
associated with the three CASes is no longer valid. Check your aggregate
log
for exceptions. Perhaps the aggregate is trying to return the CASes but it
fails on send since the temp reply queue no longer exists. Just to be
clear
do you define Process timeout in your application client? It wasnt clear
to
me if the timeout of 15000ms is in the aggregate client or the application
client.
JC
On Thu, Mar 31, 2011 at 9:24 AM, Pierre Pujalon<ppuja...@yahoo.fr>
wrote:
JC,
I had to work on this topic and add instrumentation a bit to try to have
a
clearer vision. The basic idea is to test robustness of the processing
when
the ActiveMQ have a downtime (generally only for milliseconds). This
scenario is a real life scenario.
I have an application client using the aggregate client with
UimaAsynchronousEngine. I have a pool of 4 CASes, a timeout set to
15000ms
in the client and not CASMultiplier in the aggregate.
When JMS gets down and up. I receive in my CallbackListener only one
exception out of the four CASes (the 15000ms timeout) . The three others
never trigger any timeout.
Actually, it seems that thoses CASes are being processed in the aggregate
as I have the trace of the annotators in the aggregate log. But they
never
go back to the reply queue.
I traced the CASId that the engine sends and receives in the listener.
Let's say I have id1, Id2, id3 and id4.
As the JMS gets down and up, one of the case (id1) is released . The
application client send another CAS . As this very point, the method
sendCas
returns a CASId id5 but in the CallbackListener the onBeforeMessageSend
status tells me the CASid is id4. It is confirmed with the aggregate
which
logs the message :
Duplicate Request With Cas Reference Id: (id4) Received. Ignoring
Duplicate.
The CasPool in the UIMAEngines is full and the client stalls.
Restarting the UIMAAsynchronousEngine is the only solution to continue
processing, the ag
My guess is that it comes from the UIMAAsynchronousEngine.
I hope it is clear
Pierre
PS : I set the timeout to 5 seconds, and I have a total different
scenario
which is no better :)
Le 29/03/2011 18:56, Jaroslaw Cwiklik a écrit :
Pierre, do you declare a timeout on Process in your aggregate deployment
descriptor for the remote delegate? If not, perhaps CASes were sent to
the
remote delegate and subsequently dropped because the delegate was not
unable
to deliver them to the aggregate client. A timeout would help to recover
these CASes.
Do you know where you are running out of CASes? Is it in application
client
or aggregate client? Use jConsole to attach to each jvm and examine the
threads. Do you use a Cas Multiplier in the aggregate? How big is its
pool?
-JC
On Tue, Mar 29, 2011 at 12:03 PM, Pierre Pujalon<ppuja...@yahoo.fr>
wrote:
JC,
I tested the work around and a listener to the delegate is indeed
recreated. The CAS that were being processed pass through all
annotators.
But now I have other issues, but it is not the same each time I tried
.Either the client is losing connection, or get block on the getCAS
method
as if cas process during the JMS down time were not release or the
aggregate
or the aggregate client get stalled...
I need to dig a little bit into this because it does not make many
sense
right now...
What bother me is that if I don't make the annotator delegated, the
aggregate client always recover clean with this scenario.
Anyway Thanks
Pierre
Le 28/03/2011 18:07, Jaroslaw Cwiklik a écrit :
Pierre, in my last posting I meant maxRetries=1, not retryCount=1.
JC
On Mon, Mar 28, 2011 at 12:05 PM, Jaroslaw Cwiklik<uim...@gmail.com>
wrote:
Pierre, I've identified the bug in UIMA AS code that causes the
problem
described. As a workaround, set the retryCount=1 in the error
handling
configuration for the delegate defined in your deployment descriptor.
Here
is an example:
<remoteAnalysisEngine key="NoOp">
<inputQueue endpoint="NoOpAnnotatorQueue"
brokerURL="${BrokerURL}"/>
<serializer method="xmi"/>
<asyncAggregateErrorConfiguration>
<processCasErrors maxRetries="1"
timeout="15000" thresholdCount="1" thresholdAction="terminate" />
</asyncAggregateErrorConfiguration>
</remoteAnalysisEngine>
This should force creation of a new listener on the delegate reply
queue.
The listener recovery should be done with no retry being set and I
will
make necessary changes in the code to support that use case.
Thanks for finding the bug!
JC
On Tue, Mar 22, 2011 at 1:12 PM, Jaroslaw Cwiklik<uim...@gmail.com
wrote:
Pierre, this appears to be a bug. In the scenario you've described
the
uima as aggregate client should detect a lost connection (which it
does)
and
create a new one on a subsequent process CAS request destined to the
remote
delegate. I will attempt to re-create your scenario and see if this
is
in
fact a bug.
Thanks, JC
On Tue, Mar 22, 2011 at 7:11 AM, Pierre Pujalon<ppuja...@yahoo.fr
wrote:
Hi,
I'm currently using UIMA-AS 2.3.1rc5 and I am having issues with
asynchronous delegated ..
I have a chain of several annotators. One of them is an
asynchronous
delegate as it is very resource consuming.
So I have one UIMA service with a simpleChainDescriptor and several
UIMA
services with myDelegateAnnotatorDescriptor.
When the activeMQ broker fails and restart, all my UIMA services
resfresh succesfully their JMS connection. But it appears that the
service
holding the processing chain (simpleChainDescriptor) itself does
not
refresh the JMS connection to the delegate JMS queue.
When a new CAS is processed, I have the following exception in the
console !!!! and the processing of the cas stop silently and passes
to
the
next cas.
If I restart the service holding the chain, everything go back to
normal.
Is this a bug or am I missing something?
Thanks,
Pierre
ATTENTION: Uima AS Service:simpleChainDescriptor Listener
Established
Connection to Broker: tcp://brokerurl:61616
22 mars 2011 11:11:00
org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer
handleListenerSetupFailure
ATTENTION: Uima AS Service:simpleChainDescriptor Listener
Established
Connection to Broker: tcp://brokerurl:61616
22 mars 2011 11:11:00
org.apache.uima.adapter.jms.activemq.JmsOutputChannel
invalidateConnectionAndEndpoints
INFO: Controller simpleChainDescriptor Invalidating JMS Connection
To
Broker tcp://brokerurl:61616 and Closing Sessions To Delegates
22 mars 2011 11:11:01
org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl
send
ATTENTION: Service: simpleChainDescriptor Runtime Exception
22 mars 2011 11:11:01
org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl
send
ATTENTION:
org.apache.uima.aae.error.DelegateConnectionLostException:
Controller:simpleChainDescriptor Lost Connection to
Delegate:myDelegateAnnotatorDescriptor
at
org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:536)
at
org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:507)
at
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsOutputChannel.java:1366)
at
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemoteEndpoint(JmsOutputChannel.java:1527)
at
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.serializeCasAndSend(JmsOutputChannel.java:658)
at
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendRequest(JmsOutputChannel.java:610)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatch(AggregateAnalysisEngineController_impl.java:2381)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatchProcessRequest(AggregateAnalysisEngineController_impl.java:2421)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.simpleStep(AggregateAnalysisEngineController_impl.java:1294)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2302)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.process(AggregateAnalysisEngineController_impl.java:1229)
at
org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.java:118)
at
org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTimerAndProcess(ProcessResponseHandler.java:108)
at
org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProcessResponseWithCASReference(ProcessResponseHandler.java:387)
at
org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(ProcessResponseHandler.java:656)
at
org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:149)
at
org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle(ProcessRequestHandler_impl.java:973)
at
org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessage(UimaVmMessageListener.java:107)
at
org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(UimaVmMessageDispatcher.java:70)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
22 mars 2011 11:11:01
org.apache.uima.aae.error.handler.ProcessCasErrorHandler
handleError
ATTENTION: Service: simpleChainDescriptor Runtime Exception
22 mars 2011 11:11:01
org.apache.uima.aae.error.handler.ProcessCasErrorHandler
handleError
ATTENTION:
org.apache.uima.aae.error.DelegateConnectionLostException:
Controller:simpleChainDescriptor Lost Connection to
Delegate:myDelegateAnnotatorDescriptor
at
org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:536)
at
org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl.send(JmsEndpointConnection_impl.java:507)
at
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.dispatch(JmsOutputChannel.java:1366)
at
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendCasToRemoteEndpoint(JmsOutputChannel.java:1527)
at
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.serializeCasAndSend(JmsOutputChannel.java:658)
at
org.apache.uima.adapter.jms.activemq.JmsOutputChannel.sendRequest(JmsOutputChannel.java:610)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatch(AggregateAnalysisEngineController_impl.java:2381)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.dispatchProcessRequest(AggregateAnalysisEngineController_impl.java:2421)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.simpleStep(AggregateAnalysisEngineController_impl.java:1294)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.executeFlowStep(AggregateAnalysisEngineController_impl.java:2302)
at
org.apache.uima.aae.controller.AggregateAnalysisEngineController_impl.process(AggregateAnalysisEngineController_impl.java:1229)
at
org.apache.uima.aae.handler.HandlerBase.invokeProcess(HandlerBase.java:118)
at
org.apache.uima.aae.handler.input.ProcessResponseHandler.cancelTimerAndProcess(ProcessResponseHandler.java:108)
at
org.apache.uima.aae.handler.input.ProcessResponseHandler.handleProcessResponseWithCASReference(ProcessResponseHandler.java:387)
at
org.apache.uima.aae.handler.input.ProcessResponseHandler.handle(ProcessResponseHandler.java:656)
at
org.apache.uima.aae.handler.HandlerBase.delegate(HandlerBase.java:149)
at
org.apache.uima.aae.handler.input.ProcessRequestHandler_impl.handle(ProcessRequestHandler_impl.java:973)
at
org.apache.uima.aae.spi.transport.vm.UimaVmMessageListener.onMessage(UimaVmMessageListener.java:107)
at
org.apache.uima.aae.spi.transport.vm.UimaVmMessageDispatcher$1.run(UimaVmMessageDispatcher.java:70)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
