In resources/ducc.classes you could specify something like:
Nodepool jobdriver { nodefile jobdriver.nodes ; parent --default-- }
Class JobDriver reserve-base { nodepool = jobdriver; priority = 0;
enforce = false }
And in resources/jobdriver.nodes:
hostA
hostB
Beware that if your network has a disconnect and an inuse jobdriver node
becomes unreachable, jobs using that jobdriver node will be in trouble.
Lou.
On Wed, Apr 1, 2015 at 7:12 AM, Satya Nand Kanodia <satya.kano...@orkash.com
> wrote:
> Hi,
>
> yeah It can be done in the way you are telling. But there is a parameter
> in ducc.properties for specifying broker on any system. Why isn't there any
> way for Job Driver? Because in any network of systems there is no guaranty
> that all will be available at certain time. Either we should have a option
> to specify job driver on the system which has highest availability or If a
> system running a job driver goes down , DUCC Head should allocate a new
> job driver.
>
>
>
> On 03/30/2015 06:02 PM, Lou DeGenaro wrote:
>
>> I don't think forcing the JD node and the broker on the same host will
>> help
>> much if your network has issues since all parts of DUCC communicate thru
>> the network, some via the broker.
>>
>> Nevertheless, if you have only two machines then you could first start
>> DUCC
>> only on the machine that you wish to have the JD+broker. After DUCC is up
>> on the single machine, you could start the DUCC agent on the other
>> machine.
>>
>> Lou.
>>
>> On Mon, Mar 30, 2015 at 12:34 AM, Satya Nand Kanodia <
>> satya.kano...@orkash.com> wrote:
>>
>> Hi,
>>>
>>> Actually I have two systems. One is running broker and other one is
>>> running job driver( I could not find any way to force job driver to be on
>>> any specific system, It automatically goes on 2nd system). I think when
>>> the
>>> connectivity goes, the problem comes because broker and driver are on two
>>> different systems. Is there any way I can force job driver to be on the
>>> same system, which is running broker.
>>>
>>>
>>> On 03/24/2015 06:00 PM, Lou DeGenaro wrote:
>>>
>>> By "serious" I simply meant that if your network is having problems then
>>>> applications that are dependent on the network will be having problems,
>>>> DUCC
>>>> being one such application. Granted, DUCC's Job Driver should not take
>>>> NPE,
>>>> but fixing the NPE may not make your application run much better.
>>>>
>>>> I recommend fixing your network, then it won't matter if you run DUCC
>>>> 1.x
>>>> or DUCC 2.x.
>>>>
>>>> I can make no promises, but my guess is that DUCC 2.0 will be released
>>>> in
>>>> April or May.
>>>>
>>>> If this error is really a problem for you and you need a fix for DUCC
>>>> 1.x,
>>>> I can try to do that - but I'd prefer to leave it as Won't Fix in light
>>>> of
>>>> the re-write that was done for DUCC 2.0.
>>>>
>>>> Lou.
>>>>
>>>> On Tue, Mar 24, 2015 at 12:50 AM, Satya Nand Kanodia <
>>>> satya.kano...@orkash.com> wrote:
>>>>
>>>> Dear Lou,
>>>>
>>>>> It some time happens in weekends. when It happens we also get some
>>>>> exceptions of networking like exception in connection with database etc
>>>>> etc. So upgrading to DUCC 2.0 is a solution or we can do some thing
>>>>> else
>>>>> ?
>>>>> what did you mean by more serious problems?
>>>>>
>>>>> On 03/23/2015 06:26 PM, Lou DeGenaro wrote:
>>>>>
>>>>> A Jira already exists for this issue https://issues.apache.org/
>>>>>
>>>>>> jira/browse/
>>>>>> UIMA-4225. It has been marked "Won't Fix" because the Job Driver and
>>>>>> corresponding Job Process have been re-written for 2.0 version of DUCC
>>>>>> (not
>>>>>> yet released, but available from trunk).
>>>>>>
>>>>>> Are you able to reproduce this error with regularity? I've only seen
>>>>>> it
>>>>>> once before, which resulted in Jira 4225 and its subsequent closure.
>>>>>> It
>>>>>> also seems that you may have more serious problems than just one job
>>>>>> failing. Based on the message:
>>>>>>
>>>>>> 22 Mar 2015 08:49:49,723 14 WARN jms.DefaultJmsMessageListenerC
>>>>>> ontainer Setup of JMS message listener invoker failed for destination
>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover.
>>>>>>
>>>>>> it seems you may have network or broker problems?
>>>>>>
>>>>>> Lou.
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 23, 2015 at 2:16 AM, Satya Nand Kanodia <
>>>>>> satya.kano...@orkash.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> In DUCC when I submit a job , sometimes it gets hung for infinite
>>>>>>> time.
>>>>>>> Today it happened again. A job started with 2 processes on 22 Mar
>>>>>>> 2015
>>>>>>> 04:34:38 had been continuously running till today.
>>>>>>>
>>>>>>> I used this configuration in job file
>>>>>>>
>>>>>>> process_thread_count=3
>>>>>>> process_failures_limit=50
>>>>>>> process_per_item_time_max=5
>>>>>>> process_deployments_max=2
>>>>>>> attach_console=true
>>>>>>> process_jvm_args=-Xms4g -XX\:+UseConcMarkSweepGC
>>>>>>> process_memory_size=7
>>>>>>> classpath_order=ducc-before-user
>>>>>>>
>>>>>>> Finally I had to terminate it from DUCC-MON. Job usually takes 15
>>>>>>> minutes
>>>>>>> max to process documents. I submitted a batch of 433 documents and
>>>>>>> 342
>>>>>>> documents had been processed. There were two errors of cas timed out.
>>>>>>> There
>>>>>>> were connection exceptions in both processes , which lasted for 10
>>>>>>> seconds.
>>>>>>> exception happened at 22 Mar 2015 08:50:42. After that no response
>>>>>>> from
>>>>>>> both processes.
>>>>>>>
>>>>>>> I got these exceptions from jd.out file
>>>>>>>
>>>>>>> These exception in jd.out file started at 05:09:12.
>>>>>>>
>>>>>>> 22 Mar 2015 05:32:12,352 154 ERROR jd.JobDriver waitForProcessStatus
>>>>>>> N/A process status error? java.lang.NullPointerException
>>>>>>>
>>>>>>> At:
>>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>>> JobDriver.java:1081)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(
>>>>>>> JobDriver.java:1123)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>>> java:1585)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>>> WorkItem.java:196)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>>> java:160)
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:471)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 22 Mar 2015 05:32:12,353 154 ERROR jd.JobDriver exception 19524
>>>>>>> processing error? java.lang.NullPointerException
>>>>>>>
>>>>>>> At:
>>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>>> JobDriver.java:1081)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.isNodeFailure(
>>>>>>> JobDriver.java:1092)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>>> java:1586)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>>> WorkItem.java:196)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>>> java:160)
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:471)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver waitForProcessStatus
>>>>>>> N/A process status error? java.lang.NullPointerException
>>>>>>>
>>>>>>> At:
>>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>>> JobDriver.java:1081)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(
>>>>>>> JobDriver.java:1123)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>>> java:1585)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>>> WorkItem.java:196)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>>> java:160)
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:471)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 22 Mar 2015 05:32:12,354 152 ERROR jd.JobDriver exception 19524
>>>>>>> processing error? java.lang.NullPointerException
>>>>>>>
>>>>>>> At:
>>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>>> JobDriver.java:1081)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.isNodeFailure(
>>>>>>> JobDriver.java:1092)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>>> java:1586)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>>> WorkItem.java:196)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>>> java:160)
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:471)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver waitForProcessStatus
>>>>>>> N/A process status error? java.lang.NullPointerException
>>>>>>>
>>>>>>> At:
>>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>>> JobDriver.java:1081)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.waitForProcessStatus(
>>>>>>> JobDriver.java:1123)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>>> java:1585)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>>> WorkItem.java:196)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>>> java:160)
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:471)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 22 Mar 2015 05:32:12,354 60 ERROR jd.JobDriver exception 19524
>>>>>>> processing error? java.lang.NullPointerException
>>>>>>>
>>>>>>> At:
>>>>>>> java.util.TreeMap.getEntry(TreeMap.java:342)
>>>>>>> java.util.TreeMap.get(TreeMap.java:273)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.getProcess(
>>>>>>> JobDriver.java:1081)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.isNodeFailure(
>>>>>>> JobDriver.java:1092)
>>>>>>> org.apache.uima.ducc.jd.JobDriver.exception(JobDriver.
>>>>>>> java:1586)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.exception(
>>>>>>> WorkItem.java:196)
>>>>>>> org.apache.uima.ducc.jd.client.WorkItem.run(WorkItem.
>>>>>>> java:160)
>>>>>>> java.util.concurrent.Executors$RunnableAdapter.
>>>>>>> call(Executors.java:471)
>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 22 Mar 2015 07:10:59,845 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 07:13:09,855 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 07:16:50,007 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>>
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 07:24:35,994 14 WARN jms.EndpointMessageListener
>>>>>>> Execution
>>>>>>> of
>>>>>>> JMS message listener failed. Caused by: [javax.jms.
>>>>>>> IllegalStateException
>>>>>>> - The Session is closed]
>>>>>>> javax.jms.IllegalStateException: The Session is closed
>>>>>>> at org.apache.activemq.ActiveMQSession.checkClosed(
>>>>>>> ActiveMQSession.java:731)
>>>>>>> at org.apache.activemq.ActiveMQSession.getTransacted(
>>>>>>> ActiveMQSession.java:521)
>>>>>>> at org.apache.activemq.pool.PooledSession.getTransacted(
>>>>>>> PooledSession.java:256)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractMessageListenerContain
>>>>>>> er.commitIfNecessary(AbstractMessageListenerContainer.java:571)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractMessageListenerContain
>>>>>>> er.doExecuteListener(AbstractMessageListenerContainer.java:481)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractPollingMessageListener
>>>>>>> Container.doReceiveAndExecute(AbstractPollingMessageListener
>>>>>>> Container.java:325)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractPollingMessageListener
>>>>>>> Container.receiveAndExecute(AbstractPollingMessageListener
>>>>>>> Container.java:263)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.invokeListener(
>>>>>>> DefaultMessageListenerContainer.java:1058)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.executeOngoingLoop(
>>>>>>> DefaultMessageListenerContainer.java:1050)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine
>>>>>>> r.java:947)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 22 Mar 2015 07:24:36,617 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>>
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> The
>>>>>>> Session is closed
>>>>>>> 22 Mar 2015 07:26:31,822 14 WARN jms.EndpointMessageListener
>>>>>>> Execution
>>>>>>> of
>>>>>>> JMS message listener failed. Caused by: [javax.jms.
>>>>>>> IllegalStateException
>>>>>>> - The Session is closed]
>>>>>>> javax.jms.IllegalStateException: The Session is closed
>>>>>>> at org.apache.activemq.ActiveMQSession.checkClosed(
>>>>>>> ActiveMQSession.java:731)
>>>>>>> at org.apache.activemq.ActiveMQSession.getTransacted(
>>>>>>> ActiveMQSession.java:521)
>>>>>>> at org.apache.activemq.pool.PooledSession.getTransacted(
>>>>>>> PooledSession.java:256)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractMessageListenerContain
>>>>>>> er.commitIfNecessary(AbstractMessageListenerContainer.java:571)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractMessageListenerContain
>>>>>>> er.doExecuteListener(AbstractMessageListenerContainer.java:481)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractPollingMessageListener
>>>>>>> Container.doReceiveAndExecute(AbstractPollingMessageListener
>>>>>>> Container.java:325)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractPollingMessageListener
>>>>>>> Container.receiveAndExecute(AbstractPollingMessageListener
>>>>>>> Container.java:263)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.invokeListener(
>>>>>>> DefaultMessageListenerContainer.java:1058)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.executeOngoingLoop(
>>>>>>> DefaultMessageListenerContainer.java:1050)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine
>>>>>>> r.java:947)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 22 Mar 2015 07:26:32,067 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> The
>>>>>>> Session is closed
>>>>>>> 22 Mar 2015 07:28:12,075 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>>
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 07:30:55,001 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Could not refresh JMS Connection for destination 'ducc.orchestrator.
>>>>>>> abbreviated.state'
>>>>>>> - retrying in 5000 ms. Cause: Channel was inactive for too (>30000)
>>>>>>> long:
>>>>>>> tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:29:26,669 14 WARN jms.EndpointMessageListener
>>>>>>> Execution
>>>>>>> of
>>>>>>> JMS message listener failed. Caused by: [javax.jms.
>>>>>>> IllegalStateException
>>>>>>> - The Session is closed]
>>>>>>> javax.jms.IllegalStateException: The Session is closed
>>>>>>> at org.apache.activemq.ActiveMQSession.checkClosed(
>>>>>>> ActiveMQSession.java:731)
>>>>>>> at org.apache.activemq.ActiveMQSession.getTransacted(
>>>>>>> ActiveMQSession.java:521)
>>>>>>> at org.apache.activemq.pool.PooledSession.getTransacted(
>>>>>>> PooledSession.java:256)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractMessageListenerContain
>>>>>>> er.commitIfNecessary(AbstractMessageListenerContainer.java:571)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractMessageListenerContain
>>>>>>> er.doExecuteListener(AbstractMessageListenerContainer.java:481)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractPollingMessageListener
>>>>>>> Container.doReceiveAndExecute(AbstractPollingMessageListener
>>>>>>> Container.java:325)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> AbstractPollingMessageListener
>>>>>>> Container.receiveAndExecute(AbstractPollingMessageListener
>>>>>>> Container.java:263)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.invokeListener(
>>>>>>> DefaultMessageListenerContainer.java:1058)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.executeOngoingLoop(
>>>>>>> DefaultMessageListenerContainer.java:1050)
>>>>>>> at org.springframework.jms.listener.
>>>>>>> DefaultMessageListenerContaine
>>>>>>> r$AsyncMessageListenerInvoker.run(DefaultMessageListenerContaine
>>>>>>> r.java:947)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 22 Mar 2015 08:29:26,671 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> The
>>>>>>> Session is closed
>>>>>>> 22 Mar 2015 08:33:06,678 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:36:16,841 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:37:27,506 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:39:07,876 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:42:08,029 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>>
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:42:39,341 15 ERROR processor.DefaultErrorHandler
>>>>>>> Failed
>>>>>>> delivery for (MessageId: ID-S143-52155-1426979058276-0-2895 on
>>>>>>> ExchangeId: ID-S143-52155-1426979058276-0-2896). Exhausted after
>>>>>>> delivery
>>>>>>> attempt: 1 caught: org.springframework.jms.
>>>>>>> UncategorizedJmsException:
>>>>>>> Uncategorized exception occured during JMS processing; nested
>>>>>>> exception
>>>>>>> is
>>>>>>> javax.jms.JMSException: Channel was inactive for too (>30000) long:
>>>>>>> tcp://
>>>>>>> 192.168.10.144:61617
>>>>>>> org.springframework.jms.UncategorizedJmsException: Uncategorized
>>>>>>> exception occured during JMS processing; nested exception is
>>>>>>> javax.jms.JMSException: Channel was inactive for too (>30000) long:
>>>>>>> tcp://
>>>>>>> 192.168.10.144:61617
>>>>>>> at org.springframework.jms.support.JmsUtils.
>>>>>>> convertJmsAccessException(JmsUtils.java:316)
>>>>>>> at org.springframework.jms.support.JmsAccessor.
>>>>>>> convertJmsAccessException(JmsAccessor.java:168)
>>>>>>> at org.springframework.jms.core.JmsTemplate.execute(
>>>>>>> JmsTemplate.java:469)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.send(JmsConfiguration.java:188)
>>>>>>> at org.apache.camel.component.jms.JmsProducer.doSend(
>>>>>>> JmsProducer.java:398)
>>>>>>> at org.apache.camel.component.
>>>>>>> jms.JmsProducer.processInOnly(
>>>>>>> JmsProducer.java:352)
>>>>>>> at org.apache.camel.component.jms.JmsProducer.process(
>>>>>>> JmsProducer.java:132)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.SendProcessor$2.
>>>>>>> doInAsyncProducer(
>>>>>>> SendProcessor.java:120)
>>>>>>> at org.apache.camel.impl.ProducerCache.doInAsyncProducer(
>>>>>>> ProducerCache.java:292)
>>>>>>> at org.apache.camel.processor.SendProcessor.process(
>>>>>>> SendProcessor.java:115)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> processNext(
>>>>>>> DelegateAsyncProcessor.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.management.InstrumentationProcessor.
>>>>>>> process(
>>>>>>> InstrumentationProcessor.java:73)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> processNext(
>>>>>>> DelegateAsyncProcessor.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.processor.
>>>>>>> interceptor.TraceInterceptor.
>>>>>>> process(TraceInterceptor.java:91)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.RedeliveryErrorHandler.
>>>>>>> processErrorHandler(RedeliveryErrorHandler.java:330)
>>>>>>> at org.apache.camel.processor.RedeliveryErrorHandler.
>>>>>>> process(
>>>>>>> RedeliveryErrorHandler.java:220)
>>>>>>> at org.apache.camel.processor.RouteContextProcessor.
>>>>>>> processNext(
>>>>>>> RouteContextProcessor.java:45)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.processor.interceptor.DefaultChannel.
>>>>>>> process(DefaultChannel.java:303)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.
>>>>>>> java:117)
>>>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.
>>>>>>> java:80)
>>>>>>> at org.apache.camel.processor.RouteContextProcessor.
>>>>>>> processNext(
>>>>>>> RouteContextProcessor.java:45)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.processor.UnitOfWorkProcessor.
>>>>>>> processAsync(
>>>>>>> UnitOfWorkProcessor.java:150)
>>>>>>> at org.apache.camel.processor.
>>>>>>> UnitOfWorkProcessor.process(
>>>>>>> UnitOfWorkProcessor.java:117)
>>>>>>> at org.apache.camel.processor.
>>>>>>> RouteInflightRepositoryProcess
>>>>>>> or.processNext(RouteInflightRepositoryProcessor.java:48)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> processNext(
>>>>>>> DelegateAsyncProcessor.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.management.InstrumentationProcessor.
>>>>>>> process(
>>>>>>> InstrumentationProcessor.java:73)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:86)
>>>>>>> at org.apache.camel.component.timer.TimerConsumer.
>>>>>>> sendTimerExchange(TimerConsumer.java:134)
>>>>>>> at org.apache.camel.component.timer.TimerConsumer$1.run(
>>>>>>> TimerConsumer.java:63)
>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:555)
>>>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>>> Caused by: javax.jms.JMSException: Channel was inactive for too
>>>>>>> (>30000)
>>>>>>> long: tcp://192.168.10.144:61617
>>>>>>> at org.apache.activemq.util.JMSExceptionSupport.create(
>>>>>>> JMSExceptionSupport.java:49)
>>>>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket(
>>>>>>> ActiveMQConnection.java:1377)
>>>>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket(
>>>>>>> ActiveMQConnection.java:1305)
>>>>>>> at org.apache.activemq.ActiveMQSession.send(
>>>>>>> ActiveMQSession.java:1795)
>>>>>>> at org.apache.activemq.ActiveMQMessageProducer.send(
>>>>>>> ActiveMQMessageProducer.java:277)
>>>>>>> at org.apache.activemq.ActiveMQMessageProducer.send(
>>>>>>> ActiveMQMessageProducer.java:212)
>>>>>>> at org.apache.activemq.pool.PooledProducer.send(
>>>>>>> PooledProducer.java:79)
>>>>>>> at org.apache.activemq.pool.PooledProducer.send(
>>>>>>> PooledProducer.java:62)
>>>>>>> at org.springframework.jms.core.JmsTemplate.doSend(
>>>>>>> JmsTemplate.java:592)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.doSend(JmsConfiguration.java:276)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.access$100(JmsConfiguration.java:157)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191)
>>>>>>> at org.springframework.jms.core.JmsTemplate.execute(
>>>>>>> JmsTemplate.java:466)
>>>>>>> ... 41 more
>>>>>>> Caused by: org.apache.activemq.transport.InactivityIOException:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> at org.apache.activemq.transport.
>>>>>>> AbstractInactivityMonitor$4.
>>>>>>> run(AbstractInactivityMonitor.java:209)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 22 Mar 2015 08:42:39,546 15 WARN timer.TimerConsumer Error
>>>>>>> processing
>>>>>>> exchange. Exchange[Message: org.apache.uima.ducc.transport.event.
>>>>>>> JdStateDuccEvent@db25b23]. Caused by: [org.springframework.jms.
>>>>>>> UncategorizedJmsException
>>>>>>> - Uncategorized exception occured during JMS processing; nested
>>>>>>> exception
>>>>>>> is javax.jms.JMSException: Channel was inactive for too (>30000)
>>>>>>> long:
>>>>>>> tcp://192.168.10.144:61617]
>>>>>>> org.springframework.jms.UncategorizedJmsException: Uncategorized
>>>>>>> exception occured during JMS processing; nested exception is
>>>>>>> javax.jms.JMSException: Channel was inactive for too (>30000) long:
>>>>>>> tcp://
>>>>>>> 192.168.10.144:61617
>>>>>>> at org.springframework.jms.support.JmsUtils.
>>>>>>> convertJmsAccessException(JmsUtils.java:316)
>>>>>>> at org.springframework.jms.support.JmsAccessor.
>>>>>>> convertJmsAccessException(JmsAccessor.java:168)
>>>>>>> at org.springframework.jms.core.JmsTemplate.execute(
>>>>>>> JmsTemplate.java:469)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.send(JmsConfiguration.java:188)
>>>>>>> at org.apache.camel.component.jms.JmsProducer.doSend(
>>>>>>> JmsProducer.java:398)
>>>>>>> at org.apache.camel.component.
>>>>>>> jms.JmsProducer.processInOnly(
>>>>>>> JmsProducer.java:352)
>>>>>>> at org.apache.camel.component.jms.JmsProducer.process(
>>>>>>> JmsProducer.java:132)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.SendProcessor$2.
>>>>>>> doInAsyncProducer(
>>>>>>> SendProcessor.java:120)
>>>>>>> at org.apache.camel.impl.ProducerCache.doInAsyncProducer(
>>>>>>> ProducerCache.java:292)
>>>>>>> at org.apache.camel.processor.SendProcessor.process(
>>>>>>> SendProcessor.java:115)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> processNext(
>>>>>>> DelegateAsyncProcessor.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.management.InstrumentationProcessor.
>>>>>>> process(
>>>>>>> InstrumentationProcessor.java:73)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> processNext(
>>>>>>> DelegateAsyncProcessor.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.processor.
>>>>>>> interceptor.TraceInterceptor.
>>>>>>> process(TraceInterceptor.java:91)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.RedeliveryErrorHandler.
>>>>>>> processErrorHandler(RedeliveryErrorHandler.java:330)
>>>>>>> at org.apache.camel.processor.RedeliveryErrorHandler.
>>>>>>> process(
>>>>>>> RedeliveryErrorHandler.java:220)
>>>>>>> at org.apache.camel.processor.RouteContextProcessor.
>>>>>>> processNext(
>>>>>>> RouteContextProcessor.java:45)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.processor.interceptor.DefaultChannel.
>>>>>>> process(DefaultChannel.java:303)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.
>>>>>>> java:117)
>>>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.
>>>>>>> java:80)
>>>>>>> at org.apache.camel.processor.RouteContextProcessor.
>>>>>>> processNext(
>>>>>>> RouteContextProcessor.java:45)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.processor.UnitOfWorkProcessor.
>>>>>>> processAsync(
>>>>>>> UnitOfWorkProcessor.java:150)
>>>>>>> at org.apache.camel.processor.
>>>>>>> UnitOfWorkProcessor.process(
>>>>>>> UnitOfWorkProcessor.java:117)
>>>>>>> at org.apache.camel.processor.
>>>>>>> RouteInflightRepositoryProcess
>>>>>>> or.processNext(RouteInflightRepositoryProcessor.java:48)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:73)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> processNext(
>>>>>>> DelegateAsyncProcessor.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:90)
>>>>>>> at org.apache.camel.management.InstrumentationProcessor.
>>>>>>> process(
>>>>>>> InstrumentationProcessor.java:73)
>>>>>>> at org.apache.camel.util.AsyncProcessorHelper.process(
>>>>>>> AsyncProcessorHelper.java:99)
>>>>>>> at org.apache.camel.processor.DelegateAsyncProcessor.
>>>>>>> process(
>>>>>>> DelegateAsyncProcessor.java:86)
>>>>>>> at org.apache.camel.component.timer.TimerConsumer.
>>>>>>> sendTimerExchange(TimerConsumer.java:134)
>>>>>>> at org.apache.camel.component.timer.TimerConsumer$1.run(
>>>>>>> TimerConsumer.java:63)
>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:555)
>>>>>>> at java.util.TimerThread.run(Timer.java:505)
>>>>>>> Caused by: javax.jms.JMSException: Channel was inactive for too
>>>>>>> (>30000)
>>>>>>> long: tcp://192.168.10.144:61617
>>>>>>> at org.apache.activemq.util.JMSExceptionSupport.create(
>>>>>>> JMSExceptionSupport.java:49)
>>>>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket(
>>>>>>> ActiveMQConnection.java:1377)
>>>>>>> at org.apache.activemq.ActiveMQConnection.syncSendPacket(
>>>>>>> ActiveMQConnection.java:1305)
>>>>>>> at org.apache.activemq.ActiveMQSession.send(
>>>>>>> ActiveMQSession.java:1795)
>>>>>>> at org.apache.activemq.ActiveMQMessageProducer.send(
>>>>>>> ActiveMQMessageProducer.java:277)
>>>>>>> at org.apache.activemq.ActiveMQMessageProducer.send(
>>>>>>> ActiveMQMessageProducer.java:212)
>>>>>>> at org.apache.activemq.pool.PooledProducer.send(
>>>>>>> PooledProducer.java:79)
>>>>>>> at org.apache.activemq.pool.PooledProducer.send(
>>>>>>> PooledProducer.java:62)
>>>>>>> at org.springframework.jms.core.JmsTemplate.doSend(
>>>>>>> JmsTemplate.java:592)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.doSend(JmsConfiguration.java:276)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:215)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate.access$100(JmsConfiguration.java:157)
>>>>>>> at org.apache.camel.component.jms.JmsConfiguration$
>>>>>>> CamelJmsTemplate$3.doInJms(JmsConfiguration.java:191)
>>>>>>> at org.springframework.jms.core.JmsTemplate.execute(
>>>>>>> JmsTemplate.java:466)
>>>>>>> ... 41 more
>>>>>>> Caused by: org.apache.activemq.transport.InactivityIOException:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> at org.apache.activemq.transport.
>>>>>>> AbstractInactivityMonitor$4.
>>>>>>> run(AbstractInactivityMonitor.java:209)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>>>>> ThreadPoolExecutor.java:1145)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>>>>> ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 22 Mar 2015 08:43:19,482 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:44:59,642 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:46:10,729 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:48:05,731 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>> 22 Mar 2015 08:49:49,723 14 WARN jms.DefaultJmsMessageListenerConta
>>>>>>> iner
>>>>>>> Setup of JMS message listener invoker failed for destination
>>>>>>> 'ducc.orchestrator.abbreviated.state' - trying to recover. Cause:
>>>>>>> Channel
>>>>>>> was inactive for too (>30000) long: tcp://192.168.10.144:61617
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Please help me to find out root cause of this and how can I kill a
>>>>>>> job
>>>>>>> or
>>>>>>> start a new one when any job gets stuck after a certain time..
>>>>>>>
>>>>>>> --
>>>>>>> Thanks and Regards,
>>>>>>> Satya Nand Kanodia
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>> Thanks and Regards,
>>>>> Satya Nand Kanodia
>>>>>
>>>>>
>>>>>
>>>>> --
>>> Thanks and Regards,
>>> Satya Nand Kanodia
>>>
>>>
>>>
> --
> Thanks and Regards,
> Satya Nand Kanodia
>
>
