Nelson, yes I think there is a problem in the UIMA-AS. First, there is evidence that a service is trying to use a stale temp queue:
tcp://localhost:61616 Reason: javax.jms.InvalidDestinationException: Cannot use a Temporary destination from another Connection. Also, I think there is another problem related to recovery of temp queues in the service. The service cas multiplier uses multiple queues: 1) input queue - receives GetMeta and Process requests from a client 2) temp queue - receives FreeCas requests from a client. This is only created when a service is a Cas Multiplier. This is used to free CASes in a cas multiplier. The input queue listener recovers the connection since it received an input CAS. I think the temp queue is not recovered and this leads to repeated retries. I will work on fixing this today and let you know when this is ready. I will also create a new snapshot for you to try. Thanks for finding these problems. - jerry On Wed, Sep 28, 2016 at 10:12 AM, nelson rivera <nelsonriver...@gmail.com> wrote: > ok, only show you the result of last test. > Now in my test with uima-as-2.9.0, after complete restart of the > Broker UIMA-AS log show > > 09:51:01.731 - 19: > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer. > handleListenerSetupFailure: > WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Recovered > Connection to Broker: tcp://localhost:61616 - Ready to Process Again > 09:51:01.740 - 46: > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer. > handleListenerSetupFailure: > WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Recovered > Connection to Broker: tcp://localhost:61616 - Ready to Process Again > 09:51:01.747 - 15: > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer. > handleListenerSetupFailure: > WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Recovered > Connection to Broker: tcp://localhost:61616 - Ready to Process Again > 09:51:01.773 - 14: > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont > ainer.onException: > WARNING: Service: Aggregate Cluster Analyzer Runtime Exception > 09:51:01.773 - 14: > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont > ainer.onException: > WARNING: Service: Aggregate Cluster Analyzer Runtime Exception > 09:51:01.774 - 14: > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont > ainer.onException: > WARNING: Jms Listener Failed. Endpoint: XClusterAnalyzerAggregate > Managed By: tcp://localhost:61616 Reason: javax.jms.JMSException: > java.io.EOFException > 09:51:01.774 - 14: > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer. > handleListenerSetupFailure: > WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Unable To > Connect To Broker: tcp://localhost:61616 Retrying Until Successful ... > 09:51:01.777 - 19: > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont > ainer.onException: > WARNING: Service: Aggregate Cluster Analyzer Runtime Exception > 09:51:01.777 - 19: > org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont > ainer.onException: > WARNING: Jms Listener Failed. Endpoint: > ID:nelson-H81-M1-43454-1475070490428-1:1:1 Managed By: > tcp://localhost:61616 Reason: javax.jms.InvalidDestinationException: > Cannot use a Temporary destination from another Connection. > > this lines are continue to repeat in the log indefinitely. And after > this have the same problem when send input Cas at the service > > 2016-09-28 9:27 GMT-04:00, Jaroslaw Cwiklik <uim...@gmail.com>: > > Nelson, I will investigate your use case and let you know. > > > > -jerry > > > > On Wed, Sep 28, 2016 at 9:01 AM, nelson rivera <nelsonriver...@gmail.com > > > > wrote: > > > >> Thank you for describe how UIMA-AS works, it is very helpful. But my > >> scenario is not restart the broker while the uima-as service works on > >> input CAS > >> > >> my scenario is > >> 1- Start Broker and UIMA-AS aggregate service > >> > >> 2- Complete Restart of the Broker (without restart the service). > >> UIMA-AS log show the lines that previously i said you: > >> "Jms Listener Failed." .... > >> "The Session is closed." ... > >> "The Consumer is closed." ... > >> > >> I do not see in my uima-as log the lines that you say in your test > >> after broker is bounced: > >> Listeners reconnect: > >> INFO | Successfully refreshed JMS Connection > >> INFO | Successfully refreshed JMS Connection > >> In my case (uima-as 2.8.1) seems to be a problem with reestablishing > >> the connection to the broker after restart. > >> > >> 4- Then in this point, after the broker was restarted completely, > >> uima-as client sends a CAS to UIMA-AS aggregate service. And for this > >> petition UIMA-AS client receive the parent CAS, but not child CAS. > >> > >> In this moment i will trys with uima-as-2.9.0 snapshot > >> > >> > >> 2016-09-27 15:33 GMT-04:00, Jaroslaw Cwiklik <uim...@gmail.com>: > >> > Nelson, in my scenario I do have a Cas Multiplier which produces a > >> > child > >> > CAS for each input CAS. For the purpose of this > >> > test its not important how many internal delegates there are in a > >> pipeline. > >> > The service is configured as a Cas Multiplier > >> > and it does send child CASes when a broker is running. > >> > > >> > The problem is that once the input CAS is received, its reply queue is > >> > bound to a specific temp reply queue. When a broker is > >> > bounced, that queue is not valid anymore and cant be used to return > >> > child > >> > CASes nor the input CAS. To guard against such > >> > cases you need to use timeouts on the client side. > >> > > >> > Like I said in my previous email I dont get how an input CAS can be > >> > returned back to the client after a broker bounce. This should > >> > not happen since its temp queue does not exist in a new broker. > >> > > >> > I am still looking at this scenario as I don't like what I see in the > >> > service log. I think there might be a bug in the UIMA-AS but I am > >> > still investigating. Bug or not, what I described above is how UIMA-AS > >> > works. You should not expect any replies from a service > >> > working on a CAS that was sent through a broker that is bounced before > >> > a > >> > reply. > >> > > >> > -jerry > >> > > >> > > >> > On Tue, Sep 27, 2016 at 3:18 PM, nelson rivera > >> > <nelsonriver...@gmail.com > >> > > >> > wrote: > >> > > >> >> My service is a aggregate analysis engine that receives input CAS > >> >> with the direcction of a directory of documents, the first component > >> >> is a MultiplerCas that for each document generate a child CAS, each > >> >> child CAS continues the flow that are other primitive annotators > >> >> (primitive annotators in c++ that can throw exceptions), then exists > >> >> other MultiplerCas that acts as Merger, waiting the complete > >> >> processing of each child CAS and finally generate a child CAS where > >> >> each child CAS previously processed is a view of CAS, this a child > CAS > >> >> that is composed of view continĂșa el flujo a otro AE, que espera un > >> >> CAS de esta manera. > >> >> I will send you my uima-as descriptor, to understand better my > >> >> service. All delegates are co-located. > >> >> > >> >> i will trys with uima-as-2.9.0 snapshot how you say, to see how it > >> >> behaves > >> >> > >> >> 2016-09-27 14:27 GMT-04:00, Jaroslaw Cwiklik <uim...@gmail.com>: > >> >> > I was able to stage this scenario in which an aggregate service cas > >> >> > multiplier receives a single input CAS and > >> >> > produces a child CAS. > >> >> > > >> >> > Scenario: > >> >> > 1) Service receives input CAS > >> >> > 2) CasMultiplier generates a child CAS > >> >> > 3) AE.process() is called (child CAS) and it sleeps for awhile > >> >> > 4) Broker is shutdown > >> >> > 5) AE.process() awakes from sleep > >> >> > 6) Uima-AS tries to deliver child CAS to a remote client > >> >> > 7) Connection is invalid so the child CAS is dropped > >> >> > 8) Broker is restarted > >> >> > 9) Listeners reconnect: > >> >> > INFO | Successfully refreshed JMS Connection > >> >> > INFO | Successfully refreshed JMS Connection > >> >> > 10) Broker log shows: > >> >> > WARN Service - Async error occurred: > >> >> > org.apache.activemq.DestinationDoesNotExistException: The > >> >> > destination > >> >> > temp-queue://ID:bluejws65-53920-1474997439574-8:1:1 does not > exist. > >> >> > WARN Service - Async error occurred: > >> >> > org.apache.activemq.DestinationDoesNotExistException: The > >> >> > destination > >> >> > temp-queue://ID:bluejws65-53920-1474997439574-8:1:1 does not > exist. > >> >> > > >> >> > The service failed to send a reply.Nothing is delivered to a client > >> >> > as > >> >> > expected. This is the confusing part since you see a parent CAS > >> >> > being > >> >> sent > >> >> > to a client. I just cant imagine how this is possible. > >> >> > > >> >> > When a service receives an input CAS in a message, it extracts a > JMS > >> >> > replyTo object which is a JMS temp queue and stores in an internal > >> >> > cache > >> >> > until it is time to send a reply. Temp queues are transient and > >> >> > unique > >> >> each > >> >> > time one is created. > >> >> > > >> >> > If the broker dies and subsequently restarts, the cached replyTo > >> >> > queue > >> >> > object is invalid (temp queue no longer exist) which leads to > >> >> > "Async error occurred: > >> >> > org.apache.activemq.DestinationDoesNotExistException" as shows > >> >> > above. > >> >> > > >> >> > In the service log I see a bunch of these exceptions: > >> >> > > >> >> > 02:05:46.574 - 56: > >> >> > org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl. > >> >> openChannel: > >> >> > WARNING: Service: Test Aggregate TAE Unable to Open Connection To > >> >> > Broker: > >> >> > tcp://localhost:61616?wireFormat.maxInactivityDuration=0 - > Silently > >> >> > Retrying ... > >> >> > 02:05:46.574 - 56: > >> >> > org.apache.uima.adapter.jms.activemq.JmsEndpointConnection_impl. > >> >> openChannel: > >> >> > WARNING: > >> >> > javax.jms.JMSException: Could not connect to broker URL: > >> >> > tcp://localhost:61616?wireFormat.maxInactivityDuration=0. Reason: > >> >> > java.net.ConnectException: Connection refused > >> >> > > >> >> > This is *after* the service is restarted so there seems to be a > >> problem > >> >> > with reestablishing the connection to a new broker. > >> >> > I am still investigating that part. I just am confused how you are > >> able > >> >> to > >> >> > see a reply in the client. > >> >> > > >> >> > -Jerry > >> >> > > >> >> > > >> >> > On Tue, Sep 27, 2016 at 1:13 PM, Jaroslaw Cwiklik < > uim...@gmail.com> > >> >> wrote: > >> >> > > >> >> >> From what I understand there are two issues: > >> >> >> > >> >> >> 1) Why service JMS listeners are not reconnecting when a broker is > >> >> >> bounced. This > >> >> >> should be done by default. The 2.9.0 snapshot may address this > >> >> >> since > >> >> >> a > >> >> >> listener > >> >> >> uses newer AMQ jars (5.14.0) > >> >> >> > >> >> >> 2) What happens to a child CAS when a Broker disappears and the > >> >> >> service > >> >> >> wants > >> >> >> to deliver it to a client that is not reachable. I need to > >> >> >> investigate > >> >> >> this part. I think > >> >> >> such CAS is dropped but that should be reflected in a log and > >> also > >> >> >> the > >> >> >> input CAS > >> >> >> should be marked as failed. Again something for me to > >> investigate. > >> >> >> > >> >> >> I will try to set up a test to see if I can create the scenario.It > >> >> >> will > >> >> >> take me a bit of time > >> >> >> to set this up but I will let you know what happens. > >> >> >> > >> >> >> -jerry > >> >> >> > >> >> >> On Tue, Sep 27, 2016 at 12:42 PM, Jaroslaw Cwiklik > >> >> >> <uim...@gmail.com > >> > > >> >> >> wrote: > >> >> >> > >> >> >>> Here is the uima-as 2.9.0-SNAPSHOT to try: > >> >> >>> https://repository.apache.org/content/repositories/snapshots > >> >> >>> /org/apache/uima/uima-as/2.9.0-SNAPSHOT/uima-as-2.9.0-201609 > >> >> >>> 23.195233-6.tar.gz > >> >> >>> > >> >> >>> Can you briefly describe your service? Does your service have any > >> >> remote > >> >> >>> delegates or all parts of the pipeline are in the same process? > >> >> >>> Do you see and exceptions in the service log? > >> >> >>> > >> >> >>> -jerry > >> >> >>> > >> >> >>> On Tue, Sep 27, 2016 at 9:07 AM, nelson rivera > >> >> >>> <nelsonriver...@gmail.com> > >> >> >>> wrote: > >> >> >>> > >> >> >>>> I was testing with AMQ broker 5.14.0 and i get the same error. > >> >> >>>> not > >> >> >>>> appears to be broker > >> >> >>>> issues, appears to be uima-as issues. Any other idea, i am > >> >> >>>> stagnant > >> >> >>>> at > >> >> >>>> work > >> >> >>>> > >> >> >>>> 2016-09-22 13:04 GMT-04:00, Jaroslaw Cwiklik <uim...@gmail.com > >: > >> >> >>>> > Can you install AMQ broker 5.14.0 to see if this is related to > >> >> broker > >> >> >>>> > issues. > >> >> >>>> > > >> >> >>>> > -jerry > >> >> >>>> > > >> >> >>>> > On Thu, Sep 22, 2016 at 11:29 AM, nelson rivera < > >> >> >>>> nelsonriver...@gmail.com> > >> >> >>>> > wrote: > >> >> >>>> > > >> >> >>>> >> just like that, the parent CAS comes back but the child CAS > >> >> >>>> >> doesnt. > >> >> >>>> >> > >> >> >>>> >> And the child CAS if processed, because the uima-as log > >> >> >>>> >> shows > >> >> >>>> >> the > >> >> >>>> >> lines of the other annotators primitive of the flow > processing > >> >> >>>> >> the > >> >> >>>> >> child CAS, but not comes back to UIMA-AS client. > >> >> >>>> >> > >> >> >>>> >> uima-as log only shows > >> >> >>>> >> "Jms Listener Failed. Endpoint: > >> >> >>>> >> temp-queue://ID:localhsot-H81-M1-33610-1474481197190-1:1:1 > >> >> >>>> >> Managed > >> >> >>>> By: > >> >> >>>> >> tcp://localhost:61616 Reason: javax.jms.JMSException: > >> >> >>>> >> java.io.EOFException" ........ > >> >> >>>> >> ADVERTENCIA: Jms Listener Failed. Endpoint: > >> >> >>>> >> XClusterAnalyzerAggregate > >> >> >>>> >> Managed By: tcp://localhost:61616 Reason: > >> >> >>>> >> javax.jms.IllegalStateException: The Consumer is > >> >> >>>> >> closed.............. > >> >> >>>> >> > >> >> >>>> >> The CAS is send to service after restart the Broker, and how > >> >> >>>> >> say > >> >> >>>> >> parent CAS comes back but the child CAS doesnt. And if i try > >> >> quiesce > >> >> >>>> >> the Aggregate Service console shows : "Quiescing UIMA-AS > >> Service. > >> >> >>>> >> Remaining Number of CASes to Process:1". I suppose that is > >> >> >>>> >> the > >> >> >>>> >> child > >> >> >>>> >> CAS. > >> >> >>>> >> > >> >> >>>> >> the client if can send more CASes to the service, but not > >> receive > >> >> >>>> >> response, for each CAS send after the one CAS that was send > >> after > >> >> >>>> >> restart broker (for which parent CAS comes back but the > child > >> >> >>>> >> CAS > >> >> >>>> >> doesnt) not comes back parent CAS or child CAS. > >> >> >>>> >> > >> >> >>>> >> More specific: > >> >> >>>> >> scenario 1: > >> >> >>>> >> 1- Start Broker and UIMA-AS aggregate service. Everything is > >> >> >>>> >> fine. > >> >> >>>> >> > >> >> >>>> >> 2- Restart Broker (without restart the service). UIMA-AS log > >> >> >>>> >> show > >> >> >>>> >> : > >> >> >>>> >> "Jms Listener Failed." .... > >> >> >>>> >> 3- Client sends a CAS to UIMA-AS aggregate service.(The > parent > >> >> >>>> >> CAS > >> >> >>>> >> comes back but the child CAS doesnt, to the uima-as client.) > >> >> >>>> >> If i try quiesce the Aggregate Service console shows : > >> "Quiescing > >> >> >>>> >> UIMA-AS Service. Remaining Number of CASes to Process:1" > >> >> >>>> >> > >> >> >>>> >> scenario 2: > >> >> >>>> >> 1- Start Broker and UIMA-AS aggregate service. Everything is > >> >> >>>> >> fine. > >> >> >>>> >> > >> >> >>>> >> 2- Restart Broker (without restart the service). UIMA-AS log > >> >> >>>> >> show > >> >> >>>> >> : > >> >> >>>> >> "Jms Listener Failed." .... > >> >> >>>> >> 3- Client sends a CAS to UIMA-AS aggregate service.(The > parent > >> >> >>>> >> CAS > >> >> >>>> >> comes back but the child CAS doesnt, to the uima-as client) > >> >> >>>> >> 4- Client sends another CAS to UIMA-AS aggregate > service.(not > >> >> comes > >> >> >>>> >> back parent CAS or child CAS , to the uima-as client) > >> >> >>>> >> 5- Client sends another CAS to UIMA-AS aggregate > service.(not > >> >> comes > >> >> >>>> >> back parent CAS or child CAS , to the uima-as client) > >> >> >>>> >> If i try quiesce the Aggregate Service console shows: > >> >> >>>> >> "Quiescing > >> >> >>>> >> UIMA-AS Service. Remaining Number of CASes to Process:5" > >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> > >> >> >>>> >> 2016-09-22 10:50 GMT-04:00, Jaroslaw Cwiklik < > uim...@gmail.com > >> >: > >> >> >>>> >> > When a parent CAS comes back (after broker is restored), > can > >> >> >>>> >> > the > >> >> >>>> client > >> >> >>>> >> > send more CASes to the service and receive responses? I am > >> >> >>>> >> > trying > >> >> >>>> >> > to > >> >> >>>> >> > determine what the real problem is here. > >> >> >>>> >> > > >> >> >>>> >> > 1) JMS Listener not recovering connection > >> >> >>>> >> > 2) UIMA-AS is not handling CAS reply failures when broker > >> >> >>>> connection is > >> >> >>>> >> > stale > >> >> >>>> >> > > >> >> >>>> >> > -jerry > >> >> >>>> >> > > >> >> >>>> >> > On Thu, Sep 22, 2016 at 10:38 AM, Jaroslaw Cwiklik < > >> >> >>>> uim...@gmail.com> > >> >> >>>> >> > wrote: > >> >> >>>> >> > > >> >> >>>> >> >> Strange. You say that the parent CAS comes back but the > >> >> >>>> >> >> child > >> >> CAS > >> >> >>>> >> doesnt. > >> >> >>>> >> >> Are there any exceptions in the service log? Perhaps the > >> >> >>>> >> >> service > >> >> >>>> >> >> is > >> >> >>>> >> >> unable > >> >> >>>> >> >> to send the child CAS due to missing broker. There should > >> >> >>>> >> >> be > >> >> >>>> something > >> >> >>>> >> in > >> >> >>>> >> >> the uima-as log about this. > >> >> >>>> >> >> > >> >> >>>> >> >> -jerry > >> >> >>>> >> >> > >> >> >>>> >> >> On Thu, Sep 22, 2016 at 10:19 AM, nelson rivera > >> >> >>>> >> >> <nelsonriver...@gmail.com> > >> >> >>>> >> >> wrote: > >> >> >>>> >> >> > >> >> >>>> >> >>> Yes, it is supposed to do the service, given a > >> >> >>>> >> >>> CAS it produces a child CAS and and this CAS continue in > >> >> >>>> >> >>> the > >> >> >>>> >> >>> flow, > >> >> >>>> >> >>> that are other primitive annotators. And I expected that > >> >> >>>> >> >>> the > >> >> >>>> >> >>> child > >> >> >>>> >> >>> CAS > >> >> >>>> >> >>> will sent to UIMA-AS client, but not do. > >> >> >>>> >> >>> > >> >> >>>> >> >>> the scenario: > >> >> >>>> >> >>> 1- Start Broker and UIMA-AS aggregate service > >> >> >>>> >> >>> 2- Client sends a CAS to UIMA-AS aggregate service to > >> >> >>>> >> >>> verify > >> >> >>>> >> >>> everything is fine. The process is correct. > >> >> >>>> >> >>> > >> >> >>>> >> >>> 3- Restart Broker (without restart the service). UIMA-AS > >> log > >> >> >>>> show : > >> >> >>>> >> >>> "Jms Listener Failed." .... > >> >> >>>> >> >>> 4- Client sends a CAS to UIMA-AS aggregate service > >> >> >>>> >> >>> > >> >> >>>> >> >>> UIMA-AS client receive the parent CAS, but not child CAS. > >> >> >>>> >> >>> > >> >> >>>> >> >>> Thank for you quick reply > >> >> >>>> >> >>> > >> >> >>>> >> >>> 2016-09-22 9:48 GMT-04:00, Jaroslaw Cwiklik < > >> uim...@gmail.com > >> >> >: > >> >> >>>> >> >>> > Can you clarify if the UIMA-AS service is a Cas > >> Multiplier. > >> >> >>>> Meaning > >> >> >>>> >> >>> given a > >> >> >>>> >> >>> > CAS it produces child CASes > >> >> >>>> >> >>> > which are then sent to UIMA-AS client. Is this what > this > >> >> >>>> service is > >> >> >>>> >> >>> > supposed to do if the broker is running? > >> >> >>>> >> >>> > > >> >> >>>> >> >>> > Can you confirm the scenario: > >> >> >>>> >> >>> > 1) Client sends a CAS to UIMA-AS aggregate service > >> >> >>>> >> >>> > 2) Service receives the CAS > >> >> >>>> >> >>> > 3) Broker dies > >> >> >>>> >> >>> > 4) Broker is restarted > >> >> >>>> >> >>> > > >> >> >>>> >> >>> > Does the UIMA-AS client receive the CAS back in a reply > >> >> >>>> >> >>> > when > >> >> a > >> >> >>>> >> >>> > broker > >> >> >>>> >> >>> > is > >> >> >>>> >> >>> > restarted? > >> >> >>>> >> >>> > > >> >> >>>> >> >>> > -Jerry > >> >> >>>> >> >>> > > >> >> >>>> >> >>> > > >> >> >>>> >> >>> > > >> >> >>>> >> >>> > > >> >> >>>> >> >>> > On Thu, Sep 22, 2016 at 9:33 AM, nelson rivera < > >> >> >>>> >> >>> nelsonriver...@gmail.com> > >> >> >>>> >> >>> > wrote: > >> >> >>>> >> >>> > > >> >> >>>> >> >>> >> I will try the variants that you say. I was deepening > >> >> >>>> >> >>> >> the > >> >> >>>> problem: > >> >> >>>> >> >>> >> > >> >> >>>> >> >>> >> I have a Aggregate AE that contains CasMultiplier, the > >> >> >>>> >> >>> >> original > >> >> >>>> >> >>> >> CAS > >> >> >>>> >> >>> >> is finish when it arrives to this CasMultiplier, this > >> >> >>>> generates > >> >> >>>> >> >>> >> new > >> >> >>>> >> >>> >> CAS and is this CAS that continue in the flow of > >> >> >>>> >> >>> >> Aggregate. > >> >> >>>> >> >>> >> > >> >> >>>> >> >>> >> After restart broker (without restart the service of > >> >> >>>> >> >>> >> annotator) > >> >> >>>> >> >>> >> and > >> >> >>>> >> >>> >> send a CAS to process, the listener associated to the > >> >> client > >> >> >>>> api > >> >> >>>> >> >>> >> uima-as (BaseUIMAAsynchronousEngine_impl) is notified > >> >> >>>> >> >>> >> when > >> >> >>>> the > >> >> >>>> >> >>> >> original CAS finalizes, but the new Cas generated in > >> >> >>>> >> >>> >> the > >> >> >>>> >> >>> >> CasMultiplier, It is not notifies when finalizes. > >> >> >>>> >> >>> >> > >> >> >>>> >> >>> >> with Aggregate AE simple the log is the same, but how > >> >> >>>> >> >>> >> the > >> >> >>>> original > >> >> >>>> >> >>> >> CAS > >> >> >>>> >> >>> >> is notifies into the listener when finalizes, not > have > >> >> >>>> >> >>> >> problem > >> >> >>>> >> >>> >> because is not expected for another new CAS. > >> >> >>>> >> >>> >> > >> >> >>>> >> >>> >> 2016-09-21 17:23 GMT-04:00, Jaroslaw Cwiklik > >> >> >>>> >> >>> >> <uim...@gmail.com > >> >> >>>> >: > >> >> >>>> >> >>> >> > If possible can you download AMQ 5.14.0 (latest) and > >> use > >> >> >>>> >> >>> >> > that > >> >> >>>> >> >>> >> > broker > >> >> >>>> >> >>> >> > instead of the one the uima-as 2.8.1 ships with? > >> >> >>>> >> >>> >> > There > >> >> >>>> >> >>> >> > was > >> >> >>>> >> >>> >> > a > >> >> >>>> >> >>> >> > nasty > >> >> >>>> >> >>> bug > >> >> >>>> >> >>> >> > in > >> >> >>>> >> >>> >> > AMQ related to keep-alive protocol not working. I am > >> not > >> >> >>>> sure if > >> >> >>>> >> >>> >> > this > >> >> >>>> >> >>> >> > is > >> >> >>>> >> >>> >> > the cause but worth trying. > >> >> >>>> >> >>> >> > > >> >> >>>> >> >>> >> > You can download AMQ from here: > >> >> >>>> >> >>> >> > http://activemq.apache.org/download.html > >> >> >>>> >> >>> >> > > >> >> >>>> >> >>> >> > -Jerry > >> >> >>>> >> >>> >> > > >> >> >>>> >> >>> >> > On Wed, Sep 21, 2016 at 4:52 PM, Jaroslaw Cwiklik > >> >> >>>> >> >>> >> > <uim...@gmail.com> > >> >> >>>> >> >>> >> wrote: > >> >> >>>> >> >>> >> > > >> >> >>>> >> >>> >> >> Not sure what is causing this. I need to stage this > >> >> >>>> >> >>> >> >> with > >> >> >>>> >> >>> >> >> UIMA-AS > >> >> >>>> >> >>> >> >> 2.8.1. > >> >> >>>> >> >>> >> I > >> >> >>>> >> >>> >> >> am currently testing with trunk (latest) code. The > >> most > >> >> >>>> current > >> >> >>>> >> >>> >> >> code > >> >> >>>> >> >>> >> does > >> >> >>>> >> >>> >> >> have newer broker. > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> >> I've started a broker and a sample service: > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> >> $UIMA_HOME/bin/deployAsyncService.sh > >> >> >>>> >> >>> >> >> $UIMA_HOME/examples/deploy/as/ > >> >> >>>> >> >>> >> >> Deploy_MeetingDetectorTAE.xml > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> >> After the service started, I stopped the broker, > >> waited > >> >> >>>> >> >>> >> >> a > >> >> >>>> bit, > >> >> >>>> >> and > >> >> >>>> >> >>> >> >> restarted the broker. > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> >> With UIMA-AS from trunk I see the following in the > >> log: > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> >> 04:21:35.496 - 1: org.apache.uima.adapter.jms. > >> activemq. > >> >> >>>> >> >>> >> >> SpringContainerDeployer.doStartListeners: INFO: > >> >> >>>> >> >>> >> >> Controller: > >> >> >>>> >> >>> >> >> Meeting > >> >> >>>> >> >>> >> >> Detector TAE Trying to Start Listener on Endpoint: > >> >> >>>> >> >>> >> >> queue://MeetingDetectorTaeQueue > >> >> >>>> >> >>> >> >> Selector: Command=2000 OR Command=2002 Broker: > >> >> >>>> >> >>> >> >> tcp://localhost:61616 > >> >> >>>> >> >>> >> >> 04:21:35.873 - 1: org.apache.uima.adapter.jms. > >> activemq. > >> >> >>>> >> >>> >> >> SpringContainerDeployer.doStartListeners: INFO: > >> >> >>>> >> >>> >> >> Controller: > >> >> >>>> >> >>> >> >> Meeting > >> >> >>>> >> >>> >> >> Detector TAE Trying to Start Listener on Endpoint: > >> >> >>>> >> >>> >> >> queue://MeetingDetectorTaeQueue > >> >> >>>> >> >>> >> >> Selector: Command=2001 Broker: > tcp://localhost:61616 > >> >> >>>> >> >>> >> >> 04:21:45.621 - 22: > >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. > >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerContainer.onException: > >> >> WARNING: > >> >> >>>> >> Service: > >> >> >>>> >> >>> >> >> Meeting Detector TAE Runtime Exception > >> >> >>>> >> >>> >> >> 04:21:45.622 - 22: > >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. > >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerContainer.onException: > >> >> WARNING: > >> >> >>>> Jms > >> >> >>>> >> >>> Listener > >> >> >>>> >> >>> >> >> Failed. Endpoint: MeetingDetectorTaeQueue Managed > >> >> >>>> >> >>> >> >> By: > >> >> >>>> >> >>> >> >> tcp://localhost:61616 > >> >> >>>> >> >>> >> >> Reason: javax.jms.JMSException: > java.io.EOFException > >> >> >>>> >> >>> >> >> 04:21:45.622 - 22: > >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. > >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerCont > >> >> >>>> ainer.handleListenerSetupFailure: > >> >> >>>> >> >>> >> WARNING: > >> >> >>>> >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener > Unable > >> To > >> >> >>>> Connect > >> >> >>>> >> To > >> >> >>>> >> >>> >> >> Broker: > >> >> >>>> >> >>> >> >> tcp://localhost:61616 Retrying Until Successful ... > >> >> >>>> >> >>> >> >> 04:21:45.638 - 27: > >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. > >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerContainer.onException: > >> >> WARNING: > >> >> >>>> >> Service: > >> >> >>>> >> >>> >> >> Meeting Detector TAE Runtime Exception > >> >> >>>> >> >>> >> >> 04:21:45.639 - 27: > >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. > >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerContainer.onException: > >> >> WARNING: > >> >> >>>> Jms > >> >> >>>> >> >>> Listener > >> >> >>>> >> >>> >> >> Failed. Endpoint: MeetingDetectorTaeQueue Managed > >> >> >>>> >> >>> >> >> By: > >> >> >>>> >> >>> >> >> tcp://localhost:61616 > >> >> >>>> >> >>> >> >> Reason: javax.jms.JMSException: > java.io.EOFException > >> >> >>>> >> >>> >> >> 04:21:45.639 - 27: > >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. > >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerCont > >> >> >>>> ainer.handleListenerSetupFailure: > >> >> >>>> >> >>> >> WARNING: > >> >> >>>> >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener > Unable > >> To > >> >> >>>> Connect > >> >> >>>> >> To > >> >> >>>> >> >>> >> >> Broker: > >> >> >>>> >> >>> >> >> tcp://localhost:61616 Retrying Until Successful ... > >> >> >>>> >> >>> >> >> 04:22:07.439 - 22: > >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. > >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerCont > >> >> >>>> ainer.handleListenerSetupFailure: > >> >> >>>> >> >>> >> WARNING: > >> >> >>>> >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener > >> Recovered > >> >> >>>> >> Connection > >> >> >>>> >> >>> to > >> >> >>>> >> >>> >> >> Broker: tcp://localhost:61616 - Ready to Process > >> >> >>>> >> >>> >> >> Again > >> >> >>>> >> >>> >> >> 04:22:07.454 - 27: > >> >> >>>> >> >>> >> >> org.apache.uima.adapter.jms.activemq. > >> >> >>>> >> >>> >> >> UimaDefaultMessageListenerCont > >> >> >>>> ainer.handleListenerSetupFailure: > >> >> >>>> >> >>> >> WARNING: > >> >> >>>> >> >>> >> >> Uima AS Service:Meeting Detector TAE Listener > >> Recovered > >> >> >>>> >> Connection > >> >> >>>> >> >>> to > >> >> >>>> >> >>> >> >> Broker: tcp://localhost:61616 - Ready to Process > >> >> >>>> >> >>> >> >> Again > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> >> I will repeat the above test with v 2.8.1 > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> >> -Jerry > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> >> On Wed, Sep 21, 2016 at 4:34 PM, nelson rivera < > >> >> >>>> >> >>> >> nelsonriver...@gmail.com> > >> >> >>>> >> >>> >> >> wrote: > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> >>> I'm using UIMA-AS 2.8.1, any idea that what > happen? > >> >> >>>> >> >>> >> >>> > >> >> >>>> >> >>> >> >>> 2016-09-21 16:09 GMT-04:00, Jaroslaw Cwiklik < > >> >> >>>> uim...@gmail.com > >> >> >>>> >> >: > >> >> >>>> >> >>> >> >>> > Which version of UIMA-AS are you using? > >> >> >>>> >> >>> >> >>> > > >> >> >>>> >> >>> >> >>> > -Jerry > >> >> >>>> >> >>> >> >>> > > >> >> >>>> >> >>> >> >>> > On Wed, Sep 21, 2016 at 3:42 PM, nelson rivera < > >> >> >>>> >> >>> >> >>> nelsonriver...@gmail.com> > >> >> >>>> >> >>> >> >>> > wrote: > >> >> >>>> >> >>> >> >>> > > >> >> >>>> >> >>> >> >>> >> When start ActiveMQ and deploys an annotator as > >> >> >>>> >> >>> >> >>> >> service, > >> >> >>>> >> >>> >> >>> >> processing > >> >> >>>> >> >>> >> >>> >> is > >> >> >>>> >> >>> >> >>> >> executed correctly, but when the broker is > >> stopped > >> >> >>>> >> >>> >> >>> >> and > >> >> >>>> >> >>> >> >>> >> subsequently > >> >> >>>> >> >>> >> >>> >> restarted. UIMA-AS log show the lines below > and > >> >> after > >> >> >>>> this > >> >> >>>> >> >>> >> >>> >> the > >> >> >>>> >> >>> >> >>> >> api > >> >> >>>> >> >>> >> >>> >> client uima-as is not notified in the listener > >> when > >> >> >>>> >> >>> >> >>> >> a > >> >> >>>> cas > >> >> >>>> >> >>> >> >>> >> is > >> >> >>>> >> >>> >> >>> >> processed, any more. > >> >> >>>> >> >>> >> >>> >> can help me?: > >> >> >>>> >> >>> >> >>> >> > >> >> >>>> >> >>> >> >>> >> 02:08:25.570 - 14: > >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac > >> >> >>>> tivemq.UimaDefaultMessageListe > >> >> >>>> >> >>> nerCont > >> >> >>>> >> >>> >> >>> >> ainer.onException: > >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Service: Aggregate Cluster > Analyzer > >> >> >>>> >> >>> >> >>> >> Runtime > >> >> >>>> >> >>> Exception > >> >> >>>> >> >>> >> >>> >> 02:08:25.570 - 14: > >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac > >> >> >>>> tivemq.UimaDefaultMessageListe > >> >> >>>> >> >>> nerCont > >> >> >>>> >> >>> >> >>> >> ainer.onException: > >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Jms Listener Failed. Endpoint: > >> >> >>>> >> >>> >> XClusterAnalyzerAggregate > >> >> >>>> >> >>> >> >>> >> Managed By: tcp://localhost:61616 Reason: > >> >> >>>> >> >>> javax.jms.JMSException: > >> >> >>>> >> >>> >> >>> >> java.io.EOFException > >> >> >>>> >> >>> >> >>> >> 02:08:25.570 - 14: > >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac > >> >> >>>> tivemq.UimaDefaultMessageListe > >> >> >>>> >> >>> >> >>> nerContainer. > >> >> >>>> >> >>> >> >>> >> handleListenerSetupFailure: > >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Uima AS Service:Aggregate Cluster > >> >> >>>> >> >>> >> >>> >> Analyzer > >> >> >>>> >> >>> >> >>> >> Listener > >> >> >>>> >> >>> >> >>> >> Unable To Connect To Broker: > >> >> >>>> >> >>> >> >>> >> tcp://localhost:61616 > >> >> >>>> Retrying > >> >> >>>> >> >>> >> >>> >> ... > >> >> >>>> >> >>> >> >>> >> QueueFailure: ADVERTENCIA: Jms Listener Failed. > >> >> >>>> Endpoint: > >> >> >>>> >> >>> >> >>> >> temp-queue://ID:localhost-H81- > >> >> >>>> M1-33610-1474481197190-1:1:1 > >> >> >>>> >> >>> Managed > >> >> >>>> >> >>> >> >>> >> By: > >> >> >>>> >> >>> >> >>> >> tcp://localhost:61616 Reason: > >> >> >>>> >> >>> >> >>> >> javax.jms.JMSException: > >> >> >>>> >> >>> >> >>> >> java.io.EOFException > >> >> >>>> >> >>> >> >>> >> 02:09:02.222 - 14: > >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac > >> >> >>>> tivemq.UimaDefaultMessageListe > >> >> >>>> >> >>> >> >>> nerContainer. > >> >> >>>> >> >>> >> >>> >> handleListenerSetupFailure: > >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Uima AS Service:Aggregate Cluster > >> >> >>>> >> >>> >> >>> >> Analyzer > >> >> >>>> >> >>> >> >>> >> Listener > >> >> >>>> >> >>> >> >>> >> Established Connection to Broker: > >> >> >>>> >> >>> >> >>> >> tcp://localhost:61616 > >> >> >>>> >> >>> >> >>> >> > >> >> >>>> >> >>> >> >>> >> 02:09:02.317 - 14: > >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac > >> >> >>>> tivemq.UimaDefaultMessageListe > >> >> >>>> >> >>> nerCont > >> >> >>>> >> >>> >> >>> >> ainer.onException: > >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Service: Aggregate Cluster > Analyzer > >> >> >>>> >> >>> >> >>> >> Runtime > >> >> >>>> >> >>> Exception > >> >> >>>> >> >>> >> >>> >> 02:09:02.317 - 14: > >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac > >> >> >>>> tivemq.UimaDefaultMessageListe > >> >> >>>> >> >>> nerCont > >> >> >>>> >> >>> >> >>> >> ainer.onException: > >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Jms Listener Failed. Endpoint: > >> >> >>>> >> >>> >> XClusterAnalyzerAggregate > >> >> >>>> >> >>> >> >>> >> Managed By: tcp://localhost:61616 Reason: > >> >> >>>> >> >>> >> >>> >> org.apache.activemq.ConnectionClosedException: > >> The > >> >> >>>> >> connection > >> >> >>>> >> >>> is > >> >> >>>> >> >>> >> >>> >> already closed02:09:02.356 - 15: > >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac > >> >> >>>> tivemq.UimaDefaultMessageListe > >> >> >>>> >> >>> nerCont > >> >> >>>> >> >>> >> >>> >> ainer.onException: > >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Service: Aggregate Cluster > Analyzer > >> >> >>>> >> >>> >> >>> >> Runtime > >> >> >>>> >> >>> Exception > >> >> >>>> >> >>> >> >>> >> 02:09:02.357 - 15: > >> >> >>>> >> >>> >> >>> >> org.apache.uima.adapter.jms.ac > >> >> >>>> tivemq.UimaDefaultMessageListe > >> >> >>>> >> >>> nerCont > >> >> >>>> >> >>> >> >>> >> ainer.onException: > >> >> >>>> >> >>> >> >>> >> ADVERTENCIA: Jms Listener Failed. Endpoint: > >> >> >>>> >> >>> >> XClusterAnalyzerAggregate > >> >> >>>> >> >>> >> >>> >> Managed By: tcp://localhost:61616 Reason: > >> >> >>>> >> >>> >> >>> >> javax.jms.IllegalStateException: The Consumer > is > >> >> >>>> closed. > >> >> >>>> >> >>> >> >>> >> > >> >> >>>> >> >>> >> >>> > > >> >> >>>> >> >>> >> >>> > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> >> > >> >> >>>> >> >>> >> > > >> >> >>>> >> >>> >> > >> >> >>>> >> >>> > > >> >> >>>> >> >>> > >> >> >>>> >> >> > >> >> >>>> >> >> > >> >> >>>> >> > > >> >> >>>> >> > >> >> >>>> > > >> >> >>>> > >> >> >>> > >> >> >>> > >> >> >> > >> >> > > >> >> > >> > > >> > > >