Re: Broken conections after ACTIVEMQ restart

[nelson rivera]

ok, only show you the result of last test.
Now in my test  with uima-as-2.9.0, after complete restart of the
Broker UIMA-AS log show

09:51:01.731 - 19:
org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.handleListenerSetupFailure:
WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Recovered
Connection to Broker: tcp://localhost:61616 - Ready to Process Again
09:51:01.740 - 46:
org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.handleListenerSetupFailure:
WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Recovered
Connection to Broker: tcp://localhost:61616 - Ready to Process Again
09:51:01.747 - 15:
org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.handleListenerSetupFailure:
WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Recovered
Connection to Broker: tcp://localhost:61616 - Ready to Process Again
09:51:01.773 - 14:
org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.onException:
WARNING: Service: Aggregate Cluster Analyzer Runtime Exception
09:51:01.773 - 14:
org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.onException:
WARNING: Service: Aggregate Cluster Analyzer Runtime Exception
09:51:01.774 - 14:
org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.onException:
WARNING: Jms Listener Failed. Endpoint: XClusterAnalyzerAggregate
Managed By: tcp://localhost:61616 Reason: javax.jms.JMSException:
java.io.EOFException
09:51:01.774 - 14:
org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.handleListenerSetupFailure:
WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Unable To
Connect To Broker: tcp://localhost:61616 Retrying Until Successful ...
09:51:01.777 - 19:
org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.onException:
WARNING: Service: Aggregate Cluster Analyzer Runtime Exception
09:51:01.777 - 19:
org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.onException:
WARNING: Jms Listener Failed. Endpoint:
ID:nelson-H81-M1-43454-1475070490428-1:1:1 Managed By:
tcp://localhost:61616 Reason: javax.jms.InvalidDestinationException:
Cannot use a Temporary destination from another Connection.

this lines are continue to repeat in the log indefinitely. And after
this have the same problem when send input Cas at the service

2016-09-28 9:27 GMT-04:00, Jaroslaw Cwiklik :
> Nelson, I will investigate your use case and let you know.
>
> -jerry
>
> On Wed, Sep 28, 2016 at 9:01 AM, nelson rivera 
> wrote:
>
>> Thank you for describe  how UIMA-AS works, it is very helpful. But my
>> scenario is not restart the broker while the uima-as service works on
>> input CAS
>>
>> my scenario is
>> 1- Start Broker and UIMA-AS aggregate service
>>
>> 2- Complete Restart of the Broker  (without restart the service).
>> UIMA-AS log show the lines that previously i said you:
>> "Jms Listener Failed." 
>> "The Session is closed." ...
>> "The Consumer is closed." ...
>>
>> I do not see in my uima-as log the lines that you say in your test
>> after broker is bounced:
>> Listeners reconnect:
>>  INFO | Successfully refreshed JMS Connection
>>  INFO | Successfully refreshed JMS Connection
>> In my case (uima-as 2.8.1) seems to be a problem with reestablishing
>> the connection to the broker after restart.
>>
>> 4- Then in this point, after the broker was restarted completely,
>> uima-as client sends a CAS to UIMA-AS aggregate service. And for this
>> petition UIMA-AS client receive the parent CAS, but not child CAS.
>>
>> In this moment i will trys with uima-as-2.9.0 snapshot
>>
>>
>> 2016-09-27 15:33 GMT-04:00, Jaroslaw Cwiklik :
>> > Nelson, in my scenario I do have a Cas Multiplier which produces a
>> > child
>> > CAS for each input CAS. For the purpose of this
>> > test its not important how many internal delegates there are in a
>> pipeline.
>> > The service is configured as a Cas Multiplier
>> > and it does send child CASes when a broker is running.
>> >
>> > The problem is that once the input CAS is received, its reply queue is
>> > bound to a specific temp reply queue. When a broker is
>> > bounced, that queue is not valid anymore and cant be used to return
>> > child
>> > CASes nor the input CAS. To guard against such
>> > cases you need to use timeouts on the client side.
>> >
>> > Like I said in my previous email I dont get how an input CAS can be
>> > returned back to the client after a broker bounce. This should
>> > not happen since its temp queue does not exist in a new broker.
>> >
>> > I am still looking at this scenario as I don't like what I see in the
>> > service log. I think there might be a bug in the UIMA-AS but I am
>> > still investigating. Bug or not, what I described above is how UIMA-AS
>> > works. You should not expect any replies from a service
>> > working on a CAS that was sent through a broker that 

Re: Broken conections after ACTIVEMQ restart

[Jaroslaw Cwiklik]

Nelson, yes I think there is a problem in the UIMA-AS. First, there is
evidence that a service
is trying to use a stale temp queue:

tcp://localhost:61616 Reason: javax.jms.InvalidDestinationException:
Cannot use a Temporary destination from another Connection.


Also, I think there is another problem related to recovery of temp queues
in the service.
The service cas multiplier uses multiple queues:
1) input queue - receives GetMeta and Process requests from a client
2) temp queue - receives FreeCas requests from a client. This is only
created when a service
is a Cas Multiplier. This is used to free CASes in a cas multiplier.

The input queue listener recovers the connection since it received an input
CAS. I think the
temp queue is not recovered and this leads to repeated retries.

I will work on fixing this today and let you know when this is ready. I
will also create a new
snapshot for you to try.

Thanks for finding these problems.

- jerry


On Wed, Sep 28, 2016 at 10:12 AM, nelson rivera 
wrote:

> ok, only show you the result of last test.
> Now in my test  with uima-as-2.9.0, after complete restart of the
> Broker UIMA-AS log show
>
> 09:51:01.731 - 19:
> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.
> handleListenerSetupFailure:
> WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Recovered
> Connection to Broker: tcp://localhost:61616 - Ready to Process Again
> 09:51:01.740 - 46:
> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.
> handleListenerSetupFailure:
> WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Recovered
> Connection to Broker: tcp://localhost:61616 - Ready to Process Again
> 09:51:01.747 - 15:
> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.
> handleListenerSetupFailure:
> WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Recovered
> Connection to Broker: tcp://localhost:61616 - Ready to Process Again
> 09:51:01.773 - 14:
> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
> ainer.onException:
> WARNING: Service: Aggregate Cluster Analyzer Runtime Exception
> 09:51:01.773 - 14:
> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
> ainer.onException:
> WARNING: Service: Aggregate Cluster Analyzer Runtime Exception
> 09:51:01.774 - 14:
> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
> ainer.onException:
> WARNING: Jms Listener Failed. Endpoint: XClusterAnalyzerAggregate
> Managed By: tcp://localhost:61616 Reason: javax.jms.JMSException:
> java.io.EOFException
> 09:51:01.774 - 14:
> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerContainer.
> handleListenerSetupFailure:
> WARNING: Uima AS Service:Aggregate Cluster Analyzer Listener Unable To
> Connect To Broker: tcp://localhost:61616 Retrying Until Successful ...
> 09:51:01.777 - 19:
> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
> ainer.onException:
> WARNING: Service: Aggregate Cluster Analyzer Runtime Exception
> 09:51:01.777 - 19:
> org.apache.uima.adapter.jms.activemq.UimaDefaultMessageListenerCont
> ainer.onException:
> WARNING: Jms Listener Failed. Endpoint:
> ID:nelson-H81-M1-43454-1475070490428-1:1:1 Managed By:
> tcp://localhost:61616 Reason: javax.jms.InvalidDestinationException:
> Cannot use a Temporary destination from another Connection.
>
> this lines are continue to repeat in the log indefinitely. And after
> this have the same problem when send input Cas at the service
>
> 2016-09-28 9:27 GMT-04:00, Jaroslaw Cwiklik :
> > Nelson, I will investigate your use case and let you know.
> >
> > -jerry
> >
> > On Wed, Sep 28, 2016 at 9:01 AM, nelson rivera  >
> > wrote:
> >
> >> Thank you for describe  how UIMA-AS works, it is very helpful. But my
> >> scenario is not restart the broker while the uima-as service works on
> >> input CAS
> >>
> >> my scenario is
> >> 1- Start Broker and UIMA-AS aggregate service
> >>
> >> 2- Complete Restart of the Broker  (without restart the service).
> >> UIMA-AS log show the lines that previously i said you:
> >> "Jms Listener Failed." 
> >> "The Session is closed." ...
> >> "The Consumer is closed." ...
> >>
> >> I do not see in my uima-as log the lines that you say in your test
> >> after broker is bounced:
> >> Listeners reconnect:
> >>  INFO | Successfully refreshed JMS Connection
> >>  INFO | Successfully refreshed JMS Connection
> >> In my case (uima-as 2.8.1) seems to be a problem with reestablishing
> >> the connection to the broker after restart.
> >>
> >> 4- Then in this point, after the broker was restarted completely,
> >> uima-as client sends a CAS to UIMA-AS aggregate service. And for this
> >> petition UIMA-AS client receive the parent CAS, but not child CAS.
> >>
> >> In this moment i will trys with uima-as-2.9.0 snapshot
> >>
> >>
> >> 2016-09-27 15:33 GMT-04:00, Jaroslaw Cwiklik 

[ANNOUNCE] Apache UIMA Ruta 2.5.0 released

[Peter Klügl]

The Apache UIMA team is pleased to announce the release of
Apache UIMA Ruta (Rule-based Text Annotation), version 2.5.0.

Apache UIMA Ruta is a rule-based script language supported by
Eclipse-based tooling. The language is designed to enable rapid
development of text processing applications within UIMA. A special focus
lies on the intuitive and flexible domain specific language for defining
patterns of annotations. The Eclipse-based tooling,
called the Apache UIMA Ruta Workbench, was created to support the
user and to facilitate every step when writing rules. Both
the rule language and the workbench integrate
smoothly with Apache UIMA.

Major Changes in this Release

UIMA Ruta Language and Analysis Engine:

- Depends on UIMA 2.9.0 and uimaFIT 2.2.0
- New block FOREACH: iterate over pattern and reuse the matched
annotation in contained rules
- Label expressions across inlined rules are supported, behave like
local variables now
- Feature expressions on label expressions are supported
- Setting values of variables by configuration parameter
- Macros support variables
- New type function: typeFromString
- Wordlist initialization with string expressions: simplified
specification of language-dependent dictionary usage
- New maven archetype for typical UIMA Ruta projects
- Improved and faster maven plugins
- Parameters additionalScripts and additionalEngines are now optional
- Removed engine loader concept and parameter completely
- Removed redundant Document type
- Improved datapath support
- Improved resource manager support
- Many bug fixes and improvements

UIMA Ruta Workbench:

- Improved support of classpath-lookup in maven-based projects
- Launch configuration supports various serialization formats (with
preference)
- Bug fixes

This release requires an update of script projects and its descriptors
in the UIMA Ruta Workbench. There are several ways to achieve this.
The recommended way is to right-click on the UIMA Ruta project and
select "UIMA Ruta -> Convert to UIMA Ruta project", which will update
all provided descriptors. Then, select the project and press
"Project -> Clean..." in the menu, which will regenerate all descriptors
of your scripts based on the new templates.
Projects built with the UIMA Ruta Maven Plugin require no manual effort.

For a full list of the changes, please refer to Jira:
http://uima.apache.org/d/ruta-2.5.0/issuesFixed/jira-report.html

More information about UIMA Ruta can be found here:
http://uima.apache.org/ruta.html

 - Peter Klügl, for the Apache UIMA development team






signature.asc
Description: OpenPGP digital signature

Re: Broken conections after ACTIVEMQ restart

[Jaroslaw Cwiklik]

Nelson, I will investigate your use case and let you know.

-jerry

On Wed, Sep 28, 2016 at 9:01 AM, nelson rivera 
wrote:

> Thank you for describe  how UIMA-AS works, it is very helpful. But my
> scenario is not restart the broker while the uima-as service works on
> input CAS
>
> my scenario is
> 1- Start Broker and UIMA-AS aggregate service
>
> 2- Complete Restart of the Broker  (without restart the service).
> UIMA-AS log show the lines that previously i said you:
> "Jms Listener Failed." 
> "The Session is closed." ...
> "The Consumer is closed." ...
>
> I do not see in my uima-as log the lines that you say in your test
> after broker is bounced:
> Listeners reconnect:
>  INFO | Successfully refreshed JMS Connection
>  INFO | Successfully refreshed JMS Connection
> In my case (uima-as 2.8.1) seems to be a problem with reestablishing
> the connection to the broker after restart.
>
> 4- Then in this point, after the broker was restarted completely,
> uima-as client sends a CAS to UIMA-AS aggregate service. And for this
> petition UIMA-AS client receive the parent CAS, but not child CAS.
>
> In this moment i will trys with uima-as-2.9.0 snapshot
>
>
> 2016-09-27 15:33 GMT-04:00, Jaroslaw Cwiklik :
> > Nelson, in my scenario I do have a Cas Multiplier which produces a child
> > CAS for each input CAS. For the purpose of this
> > test its not important how many internal delegates there are in a
> pipeline.
> > The service is configured as a Cas Multiplier
> > and it does send child CASes when a broker is running.
> >
> > The problem is that once the input CAS is received, its reply queue is
> > bound to a specific temp reply queue. When a broker is
> > bounced, that queue is not valid anymore and cant be used to return child
> > CASes nor the input CAS. To guard against such
> > cases you need to use timeouts on the client side.
> >
> > Like I said in my previous email I dont get how an input CAS can be
> > returned back to the client after a broker bounce. This should
> > not happen since its temp queue does not exist in a new broker.
> >
> > I am still looking at this scenario as I don't like what I see in the
> > service log. I think there might be a bug in the UIMA-AS but I am
> > still investigating. Bug or not, what I described above is how UIMA-AS
> > works. You should not expect any replies from a service
> > working on a CAS that was sent through a broker that is bounced before a
> > reply.
> >
> > -jerry
> >
> >
> > On Tue, Sep 27, 2016 at 3:18 PM, nelson rivera  >
> > wrote:
> >
> >> My service  is a aggregate analysis engine that receives input CAS
> >> with the direcction of a directory of documents, the first component
> >> is a MultiplerCas that for each document generate a child CAS, each
> >> child CAS continues the flow that are other primitive annotators
> >> (primitive annotators in c++ that can throw exceptions), then exists
> >> other MultiplerCas that acts as Merger, waiting  the complete
> >> processing of each child CAS and finally generate a child CAS where
> >> each child CAS previously processed is a view of CAS, this a child CAS
> >> that is composed of view continúa el flujo a otro AE, que espera un
> >> CAS de esta manera.
> >> I will send you my uima-as descriptor,  to understand better my
> >> service. All delegates are co-located.
> >>
> >> i will trys with uima-as-2.9.0 snapshot how you say, to see how it
> >> behaves
> >>
> >> 2016-09-27 14:27 GMT-04:00, Jaroslaw Cwiklik :
> >> > I was able to stage this scenario in which an aggregate service cas
> >> > multiplier receives a single input CAS and
> >> > produces a child CAS.
> >> >
> >> > Scenario:
> >> > 1) Service receives input CAS
> >> > 2) CasMultiplier generates a child CAS
> >> > 3) AE.process() is called (child CAS) and it sleeps for awhile
> >> > 4) Broker is shutdown
> >> > 5) AE.process() awakes from sleep
> >> > 6) Uima-AS tries to deliver child CAS to a remote client
> >> > 7) Connection is invalid so the child CAS is dropped
> >> > 8) Broker is restarted
> >> > 9) Listeners reconnect:
> >> >  INFO | Successfully refreshed JMS Connection
> >> >  INFO | Successfully refreshed JMS Connection
> >> > 10) Broker log shows:
> >> > WARN  Service- Async error occurred:
> >> > org.apache.activemq.DestinationDoesNotExistException: The destination
> >> > temp-queue://ID:bluejws65-53920-1474997439574-8:1:1 does not exist.
> >> > WARN  Service- Async error occurred:
> >> > org.apache.activemq.DestinationDoesNotExistException: The destination
> >> > temp-queue://ID:bluejws65-53920-1474997439574-8:1:1 does not exist.
> >> >
> >> > The service failed to send a reply.Nothing is delivered to a client as
> >> > expected. This is the confusing part since you see a parent CAS being
> >> sent
> >> > to a client. I just cant imagine how this is possible.
> >> >
> >> > When a service receives 

Re: Broken conections after ACTIVEMQ restart

[nelson rivera]

Thank you for describe  how UIMA-AS works, it is very helpful. But my
scenario is not restart the broker while the uima-as service works on
input CAS

my scenario is
1- Start Broker and UIMA-AS aggregate service

2- Complete Restart of the Broker  (without restart the service).
UIMA-AS log show the lines that previously i said you:
"Jms Listener Failed." 
"The Session is closed." ...
"The Consumer is closed." ...

I do not see in my uima-as log the lines that you say in your test
after broker is bounced:
Listeners reconnect:
 INFO | Successfully refreshed JMS Connection
 INFO | Successfully refreshed JMS Connection
In my case (uima-as 2.8.1) seems to be a problem with reestablishing
the connection to the broker after restart.

4- Then in this point, after the broker was restarted completely,
uima-as client sends a CAS to UIMA-AS aggregate service. And for this
petition UIMA-AS client receive the parent CAS, but not child CAS.

In this moment i will trys with uima-as-2.9.0 snapshot


2016-09-27 15:33 GMT-04:00, Jaroslaw Cwiklik :
> Nelson, in my scenario I do have a Cas Multiplier which produces a child
> CAS for each input CAS. For the purpose of this
> test its not important how many internal delegates there are in a pipeline.
> The service is configured as a Cas Multiplier
> and it does send child CASes when a broker is running.
>
> The problem is that once the input CAS is received, its reply queue is
> bound to a specific temp reply queue. When a broker is
> bounced, that queue is not valid anymore and cant be used to return child
> CASes nor the input CAS. To guard against such
> cases you need to use timeouts on the client side.
>
> Like I said in my previous email I dont get how an input CAS can be
> returned back to the client after a broker bounce. This should
> not happen since its temp queue does not exist in a new broker.
>
> I am still looking at this scenario as I don't like what I see in the
> service log. I think there might be a bug in the UIMA-AS but I am
> still investigating. Bug or not, what I described above is how UIMA-AS
> works. You should not expect any replies from a service
> working on a CAS that was sent through a broker that is bounced before a
> reply.
>
> -jerry
>
>
> On Tue, Sep 27, 2016 at 3:18 PM, nelson rivera 
> wrote:
>
>> My service  is a aggregate analysis engine that receives input CAS
>> with the direcction of a directory of documents, the first component
>> is a MultiplerCas that for each document generate a child CAS, each
>> child CAS continues the flow that are other primitive annotators
>> (primitive annotators in c++ that can throw exceptions), then exists
>> other MultiplerCas that acts as Merger, waiting  the complete
>> processing of each child CAS and finally generate a child CAS where
>> each child CAS previously processed is a view of CAS, this a child CAS
>> that is composed of view continúa el flujo a otro AE, que espera un
>> CAS de esta manera.
>> I will send you my uima-as descriptor,  to understand better my
>> service. All delegates are co-located.
>>
>> i will trys with uima-as-2.9.0 snapshot how you say, to see how it
>> behaves
>>
>> 2016-09-27 14:27 GMT-04:00, Jaroslaw Cwiklik :
>> > I was able to stage this scenario in which an aggregate service cas
>> > multiplier receives a single input CAS and
>> > produces a child CAS.
>> >
>> > Scenario:
>> > 1) Service receives input CAS
>> > 2) CasMultiplier generates a child CAS
>> > 3) AE.process() is called (child CAS) and it sleeps for awhile
>> > 4) Broker is shutdown
>> > 5) AE.process() awakes from sleep
>> > 6) Uima-AS tries to deliver child CAS to a remote client
>> > 7) Connection is invalid so the child CAS is dropped
>> > 8) Broker is restarted
>> > 9) Listeners reconnect:
>> >  INFO | Successfully refreshed JMS Connection
>> >  INFO | Successfully refreshed JMS Connection
>> > 10) Broker log shows:
>> > WARN  Service- Async error occurred:
>> > org.apache.activemq.DestinationDoesNotExistException: The destination
>> > temp-queue://ID:bluejws65-53920-1474997439574-8:1:1 does not exist.
>> > WARN  Service- Async error occurred:
>> > org.apache.activemq.DestinationDoesNotExistException: The destination
>> > temp-queue://ID:bluejws65-53920-1474997439574-8:1:1 does not exist.
>> >
>> > The service failed to send a reply.Nothing is delivered to a client as
>> > expected. This is the confusing part since you see a parent CAS being
>> sent
>> > to a client. I just cant imagine how this is possible.
>> >
>> > When a service receives an input CAS in a message, it extracts a JMS
>> > replyTo object which is a JMS temp queue and stores in an internal
>> > cache
>> > until it is time to send a reply. Temp queues are transient and unique
>> each
>> > time one is created.
>> >
>> > If the broker dies and subsequently restarts, the cached replyTo queue
>> > object is invalid (temp queue no