Some Send Errors Can Cause Undeployment to Wait Forever
-------------------------------------------------------
Key: SM-1850
URL: https://issues.apache.org/activemq/browse/SM-1850
Project: ServiceMix
Issue Type: Bug
Components: servicemix-camel, servicemix-common
Affects Versions: 3.3
Reporter: Robert H. Pollack
If a component causes an exception to be thrown in
{{org.apache.servicemix.common.EndpointDeliveryChannel.send()}}, the exchange
is nevertheless entered into a hashmap called {{knownExchanges}}. Later, when
the component shuts down, the shutdown code in
{{org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown()}} will wait
for this exchange to be processed--but this will never happen. The result is
that the JBI container completely stops.
Reproducing this problem is easy in Camel (which is where I discovered it); it
requires only a one-line Camel route such as this one:
{{from ("timer:dmftimer1?period=60000").to
("jbi:service:alpha:beta:gamma");}}
To reproduce this problem:
# Create a service assembly containing this route.
# Copy the service assembly to the {{hotdeploy}} directory.
# After the exception is thrown, attempt to shut down the assembly (the
simplest way is to control-C the ServiceMix window). The service assembly will
not undeploy and the JBI container will not shut down.
I will attach two zip files to assist in reproducing this error:
# A pre-built service assembly that you can just copy to {{hotdeploy}}.
# Source code and a build script to assist experimentation.
(The build script is an Ant file that requires you to point the property
{{camel-dir}} at a directory that contains the Camel core jar.
The relevant code in {{EndpointDeliveryChannel}} is this:
{noformat}
public void send(MessageExchange exchange) throws MessagingException {
prepareExchange(exchange);
handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
channel.send(exchange);
}
{noformat}
The call to {{handleExchange()}} enters the exchange into the hashmap. But if
{{channel.send()}} throws an exception, the exchange is never removed from the
hashmap.
The relevant code in {{AsyncBaseLifeCycle}} is this:
{noformat}
public void prepareShutdown(Endpoint endpoint) throws InterruptedException {
Set<String> exchanges = getKnownExchanges(endpoint);
synchronized (exchanges) {
if (!exchanges.isEmpty()) {
exchanges.wait();
}
}
}
{noformat}
I do not know if all send problems cause this behavior. I discovered it by
running a Camel application that attempts to send a message to a non-existent
JBI endpoint. When the Camel program does this, I get a stack trace that (in
part) looks like this:
{noformat}
...
Caused by: javax.jbi.messaging.MessagingException: Could not find route for
exchange: InOnly[
id: ID:10.6.10.123-120d97c610d-3:0
status: Active
role: provider
service: {alpha:beta}gamma
in: null
] for service: {alpha:beta}gamma and interface: null
at
org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:297)
at
org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
at
org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:894)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
at
org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:88)
at
org.apache.servicemix.common.endpoints.SimpleEndpoint.send(SimpleEndpoint.java:66)
at
org.apache.servicemix.camel.CamelConsumerEndpoint.process(CamelConsumerEndpoint.java:89)
... 13 more
{noformat}
Later, when attempting to shut down, a thread dump includes the following:
{noformat}
"Thread-16" daemon prio=6 tid=0x2bb34400 nid=0x1398 in Object.wait()
[0x2ddff000..0x2ddffc94]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x25aed9f8> (a java.util.HashSet)
at java.lang.Object.wait(Object.java:485)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown(AsyncBaseLifeCycle.java:657)
- locked <0x25aed9f8> (a java.util.HashSet)
at
org.apache.servicemix.common.DefaultServiceUnit.removeEndpoint(DefaultServiceUnit.java:205)
- locked <0x0826a048> (a
org.apache.servicemix.common.xbean.XBeanServiceUnit)
at
org.apache.servicemix.common.DefaultComponent.removeEndpoint(DefaultComponent.java:301)
- locked <0x08269958> (a
org.apache.servicemix.camel.CamelJbiComponent)
at
org.apache.servicemix.camel.JbiEndpoint$JbiProducer.stop(JbiEndpoint.java:82)
...
{noformat}
My proposed fix is to change the code in {{EndpointDeliveryChannel}} to a
try-catch block, as shown below. But I defer to those who know ServiceMix
better than I to say whether this needs to be more sophisticated. My change,
which I have tested and which works fine for me, is as follows:
{noformat}
public void send(MessageExchange exchange) throws MessagingException {
prepareExchange(exchange);
handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
try {
channel.send(exchange);
} catch (MessagingException me) {
// If something goes wrong with the send, remove this exchange
// from the list of known exchanges or else the caller will
// never be able to undeploy
// (see AsyncBaseLifeCycle.prepareShutdown()).
handleExchange (exchange, false);
throw me;
} catch (Throwable t) {
handleExchange (exchange, false);
throw new MessagingException (t.getMessage(), t);
}
}
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
