UIMA AS client connection to broker lost issue
Hi, right now I am testing the camel uima driver for error handling, for this I have an AE which just throws an exception when it get a CAS. The error handling is just the default, which means that the worker node just reports the error back to the uima as client. Thats fine, the camel route is configured to retry for every in this scenario, thats also fine. But after a while the system just hangs and stops retrying it took me a while to figure out why. The uima driver assumes that every CAS it sends causes a result to be returned, but once in a while it gets back a CAS with a different/new reference id than the one which was sent. This causes the allocated control thread to wait, when this happens to all control threads it just hangs. First of all can it be assumed that for every sent CAS a status comes back with the same reference id (when using timeouts) ? Here is the error I get back in case the reference id does not match the one of the send CAS: org.apache.uima.jms.error.handler.BrokerConnectionException: Unable To Deliver CAS:-7fbfaf13:1250d5b260d:-7ffa To Destination. Connection To Broker failover:(tcp://XXX1:61616,tcp://XXX2:61616)?randomize=false Has Been Lost Jörn
Re: UIMA AS client connection to broker lost issue
Jörn Kottmann wrote: First of all can it be assumed that for every sent CAS a status comes back with the same reference id (when using timeouts) ? If it cannot be assumed, I have to add custom time out handling to the uima camel integration. I am not sure, but right now it hangs once in a while when the worker node is sending back error replies for a CASes. Jörn
Re: UIMA AS client connection to broker lost issue
Hi Jörn, This exception looks like it is not being thrown from the service, but from a failure in the client to send the request. That code was changed very recently as part of fixing client JMS reconnect. Perhaps the Cas ID returned in that situation is just broken? Eddie On Thu, Nov 19, 2009 at 12:09 PM, Jörn Kottmann wrote: > Hi, > > right now I am testing the camel uima driver for error handling, > for this I have an AE which just throws an exception when it get a CAS. > > The error handling is just the default, which means that the worker node > just reports the error back to the uima as client. Thats fine, the camel > route > is configured to retry for every in this scenario, thats also fine. > But after a while the system just hangs and stops retrying it took me a > while to figure out why. > > The uima driver assumes that every CAS it sends causes a result to be > returned, > but once in a while it gets back a CAS with a different/new reference id > than > the one which was sent. This causes the allocated control thread to wait, > when this happens to all control threads it just hangs. > > First of all can it be assumed that for every sent CAS a status comes back > with the same > reference id (when using timeouts) ? > > Here is the error I get back in case the reference id does not match the one > of the send CAS: > org.apache.uima.jms.error.handler.BrokerConnectionException: Unable To > Deliver CAS:-7fbfaf13:1250d5b260d:-7ffa To Destination. Connection To Broker > failover:(tcp://XXX1:61616,tcp://XXX2:61616)?randomize=false Has Been Lost > > Jörn > >
Re: UIMA AS client connection to broker lost issue
Yes, this exception comes from UIMA AS client and it is thrown if the connection to the broker is lost. If the connection to a broker is lost the current code doesnt use onBeforeMessageSend() callback. This method is only called if the connection is ok, right before the send(). The code that detects connection failure is before this. When you get BrokerConnectionException you know that what you've sent previously will not come back. The reply queue has already been deleted. You may assume this and awake all waiting threads and invalidate their states and retry if that is what you want to do. If the broker finally comes online, eventually UIMA AS client will call onBeforeMessageSend() and that will be a clue that things are back to normal. JC On Sun, Nov 22, 2009 at 8:46 AM, Eddie Epstein wrote: > Hi Jörn, > > This exception looks like it is not being thrown from the service, > but from a failure in the client to send the request. That code was > changed very recently as part of fixing client JMS reconnect. > Perhaps the Cas ID returned in that situation is just broken? > > Eddie > > On Thu, Nov 19, 2009 at 12:09 PM, Jörn Kottmann > wrote: > > Hi, > > > > right now I am testing the camel uima driver for error handling, > > for this I have an AE which just throws an exception when it get a CAS. > > > > The error handling is just the default, which means that the worker node > > just reports the error back to the uima as client. Thats fine, the camel > > route > > is configured to retry for every in this scenario, thats also fine. > > But after a while the system just hangs and stops retrying it took me a > > while to figure out why. > > > > The uima driver assumes that every CAS it sends causes a result to be > > returned, > > but once in a while it gets back a CAS with a different/new reference id > > than > > the one which was sent. This causes the allocated control thread to wait, > > when this happens to all control threads it just hangs. > > > > First of all can it be assumed that for every sent CAS a status comes > back > > with the same > > reference id (when using timeouts) ? > > > > Here is the error I get back in case the reference id does not match the > one > > of the send CAS: > > org.apache.uima.jms.error.handler.BrokerConnectionException: Unable To > > Deliver CAS:-7fbfaf13:1250d5b260d:-7ffa To Destination. Connection To > Broker > > failover:(tcp://XXX1:61616,tcp://XXX2:61616)?randomize=false Has Been > Lost > > > > Jörn > > > > >
Re: UIMA AS client connection to broker lost issue
Jaroslaw Cwiklik wrote: Yes, this exception comes from UIMA AS client and it is thrown if the connection to the broker is lost. If the connection to a broker is lost the current code doesnt use onBeforeMessageSend() callback. This method is only called if the connection is ok, right before the send(). The code that detects connection failure is before this. When you get BrokerConnectionException you know that what you've sent previously will not come back. The reply queue has already been deleted. You may assume this and awake all waiting threads and invalidate their states and retry if that is what you want to do. If the broker finally comes online, eventually UIMA AS client will call onBeforeMessageSend() and that will be a clue that things are back to normal. Sorry, have been through a few very busy weeks, since the release is now delayed I would like to fix this problem. To make sure I understand you correctly, a CAS is send via sendCas(...), but it cannot be send because of a broker connection exception, the error is then reported to the status call back listeners entityProcessComplete method. When the status call back listener receives the BrokerConnectionException it should consider all outstanding CASes as failed and retry them. Is that correct ? Jörn
Re: UIMA AS client connection to broker lost issue
That is right. jerry On Mon, Dec 14, 2009 at 9:53 AM, Jörn Kottmann wrote: > Jaroslaw Cwiklik wrote: > >> Yes, this exception comes from UIMA AS client and it is thrown if the >> connection to the broker is lost. If the connection to a broker is lost >> the >> current code doesnt use onBeforeMessageSend() callback. This method is >> only >> called if the connection is ok, right before the send(). The code that >> detects connection failure is before this. When you get >> BrokerConnectionException you know that what you've sent previously will >> not >> come back. The reply queue has already been deleted. You may assume this >> and >> awake all waiting threads and invalidate their states and retry if that is >> what you want to do. If the broker finally comes online, eventually UIMA >> AS >> client will call onBeforeMessageSend() and that will be a clue that things >> are back to normal. >> >> > Sorry, have been through a few very busy weeks, > since the release is now delayed I would like to fix this problem. > > To make sure I understand you correctly, a CAS is send via sendCas(...), > but it cannot be send because of a broker connection exception, > the error is then reported to the status call back listeners > entityProcessComplete > method. > > When the status call back listener receives the BrokerConnectionException > it should consider all outstanding CASes as failed and retry them. > > Is that correct ? > > Jörn > >
Re: UIMA AS client connection to broker lost issue
Jaroslaw Cwiklik wrote: That is right. jerry On Mon, Dec 14, 2009 at 9:53 AM, Jörn Kottmann wrote: Jaroslaw Cwiklik wrote: Yes, this exception comes from UIMA AS client and it is thrown if the connection to the broker is lost. If the connection to a broker is lost the current code doesnt use onBeforeMessageSend() callback. This method is only called if the connection is ok, right before the send(). The code that detects connection failure is before this. When you get BrokerConnectionException you know that what you've sent previously will not come back. The reply queue has already been deleted. You may assume this and awake all waiting threads and invalidate their states and retry if that is what you want to do. If the broker finally comes online, eventually UIMA AS client will call onBeforeMessageSend() and that will be a clue that things are back to normal. Sorry, have been through a few very busy weeks, since the release is now delayed I would like to fix this problem. To make sure I understand you correctly, a CAS is send via sendCas(...), but it cannot be send because of a broker connection exception, the error is then reported to the status call back listeners entityProcessComplete method. When the status call back listener receives the BrokerConnectionException it should consider all outstanding CASes as failed and retry them. Is that correct ? Must the same action be taken if a timed out CAS comes back ? Jörn
Re: UIMA AS client connection to broker lost issue
No. With timeouts the client API will notify the application for each request that eventually fails. Eddie On Thu, Dec 17, 2009 at 9:32 AM, Jörn Kottmann wrote: >>> >>> When the status call back listener receives the BrokerConnectionException >>> it should consider all outstanding CASes as failed and retry them. >>> >>> Is that correct ? >>> > > Must the same action be taken if a timed out CAS comes back ? > > Jörn >
