Also.. why will adding a delay fix the issue? How long of a delay do you recommend?
On Mon, May 9, 2011 at 10:12 AM, Tom Howe <tomh...@artcore.com> wrote: > What do you mean by propagates to file and point? > > We do actually have an error handler - the previous route I posted was just > a summary. Below is the full config for the 2 camel contexts in the app. The > first is for the main message routing, the 2nd is for the heartbeat. > > The app sends messages to splunk and a heartbeart to an activeMQ. > > The symptons we are finding is that occasionally, the application > completely stops sending messages. The log which normally shows both > messages being sent suddendly quietens down and only shows the heartbeat > messages (even though they are not arriving either). After a restart it > continues fine. > > The exceptions found in the log that I previously pasted do not coincide > with the time at which it stops so may be a red herring. > > This application runs on quite heavily loaded VMs. We wonder if it could be > due to memory issues as other applications on the box have died due to > running out of memory. > > <camelContext id="main" autoStartup="true" xmlns=" > http://camel.apache.org/schema/spring";> > > <errorHandler id="mr.error" type="DeadLetterChannel" > deadLetterUri="activemq:mr.error"> > <redeliveryPolicy maximumRedeliveries="10" > redeliveryDelay="100" logHandled="true"/> > </errorHandler> > > <errorHandler id="mr.splunk.error" type="DeadLetterChannel" > deadLetterUri="activemq:mr.splunk.error"> > <redeliveryPolicy maximumRedeliveries="10" > redeliveryDelay="100" logHandled="true"/> > </errorHandler> > > <!--direct endpoints--> > <endpoint id="input" uri="${mr.input.endpoint}"/> > <endpoint id="output" uri="${mr.output.endpoint}"/> > > <endpoint id="splunk" uri="${splunk.endpoint}"/> > <!--<endpoint id="splunk.in" uri="${splunk.in.endpoint}"/>--> > > <!-- convert ispy messages to xml and place in general in queue --> > <route autoStartup="true" errorHandlerRef="mr.error" xmlns=" > http://camel.apache.org/schema/spring";> > <from ref="input"/> > <!--<log message="Message passed f=${header.CamelFileName} > #${header.JMSMessageID} from: ${header.JMSDestination} to output"--> > <!--loggingLevel="INFO"/>--> > > <convertBodyTo type="java.lang.String"/> > <bean ref="ispyJsonToXml" method="process"/> > <choice> > <when> > <xpath>/*[starts-with(@name, 'wfe.')]</xpath> > <to uri="direct:splunk.in"/> > </when> > <otherwise> > <log message="sending message to activmq" > loggingLevel="INFO"/> > <to ref="output"/> > </otherwise> > </choice> > </route> > > > <!-- filter and convert all messages for splunk to splunk format > --> > <route autoStartup="true" errorHandlerRef="mr.splunk.error" xmlns=" > http://camel.apache.org/schema/spring";> > <from uri="direct:splunk.in"/> > <choice> > <when> > <!-- convert ispy-events --> > <xpath>/*[local-name() = "event"][./*/*[local-name() = > "ispy-event"]]</xpath> > <bean ref="ispyXmlToSplunk"/> > </when> > <otherwise> > <!-- convert everything else by grabbing envelope > information --> > <bean ref="envelopeXmlToSplunk"/> > </otherwise> > </choice> > <log message="sending message to splunk" loggingLevel="INFO"/> > <to ref="splunk"/> > </route> > > > </camelContext> > > <bean id="createHeartBeat" class="agentsmith.generators.CreateHeartBeat"> > <!--<property name="eventName" > value="${personality}.heartbeat"/>--> > <property name="eventName" ref="heartBeatName"/> > </bean> > > <camelContext id="heartbeat" autoStartup="true" xmlns=" > http://camel.apache.org/schema/spring";> > > <endpoint id="general2.in" camelContextId="heartbeat" > uri="${general.in.endpoint}"/> > > <endpoint id="heartbeattimer" camelContextId="heartbeat" > uri="quartz://mygroup/heartbeat?cron=0+*+*+?+*+*"/> > > <route autoStartup="true" inheritErrorHandler="true" xmlns=" > http://camel.apache.org/schema/spring";> > <from ref="heartbeattimer"/> > <bean ref="createHeartBeat"/> > <to ref="general2.in"/> > </route> > > </camelContext> > > > > > > > > > On Mon, May 9, 2011 at 9:49 AM, Filippo Balicchia <fbalicc...@gmail.com>wrote: > >> If there is no explicit error handling the default error i triggered, >> and the exception is propagates it back to the caller. >> In your case seems to have network error then then error is propagates >> to file and point. >> >> To resolve your problem i just add 'delay' parameter on your >> expression. With this parameter you instruct camel >> to wait milliseconds before the next poll of the file/directory >> >> Cheers. >> >> --Filippo >> >> >> 2011/5/8 Tom Howe <tomh...@artcore.com>: >> > Most of the messages are flowing to splunk fine. A few are failing to >> get >> > through - possibly we are flooding splunk - hence those errors. >> > But what I dont understand is why the process then stops processing any >> more >> > messages. >> > >> > >> > On Sun, May 8, 2011 at 6:07 AM, Filippo Balicchia <fbalicc...@gmail.com >> >wrote: >> > >> >> Does your splunk accept you connection ? >> >> >> >> From stacktrace seems that refused your connection >> >> >> >> >> >> 2011-05-07 12:41:47.042077500 Caused by: java.net.ConnectException: >> >> Connection refused2011-05-07 12:41:47.042078500 at >> >> >> >> >> >> --Filippo >> >> >> >> >> >> 2011/5/7 Tom Howe <tomh...@artcore.com>: >> >> > I have a simple java app that collect files of disk, converts the >> [json] >> >> > contents and then forwards it to a splunk instance over tcp using >> mina. >> >> > >> >> > >> >> > <endpoint id="input" uri="${input.endpoint}"/> >> >> > <endpoint id="output" uri="${output.endpoint}"/> >> >> > >> >> > <route autoStartup="true" xmlns=" >> >> > http://camel.apache.org/schema/spring";> >> >> > <from ref="input"/> >> >> > <convertBodyTo type="java.lang.String"/> >> >> > <bean ref="jsonToSplunk"/> >> >> > <to ref="splunk" /> >> >> > </route> >> >> > >> >> > input.endpoint = >> >> > file:/dir/to/files?delete=true&recursive=true&maxMessagesPerPoll=500 >> >> > output.endpoint = >> >> > >> mina:tcp://splunkhost/?textline=true&textlineDelimiter=UNIX&sync=false >> >> > >> >> > It also emits a heartbeat message every minute over JMS using camel >> >> quartz >> >> > endpoint. >> >> > >> >> > The problem I am having is that after about a day of running, the >> process >> >> > stops picking up files and sending them to splunk and stops sending >> the >> >> > heartbeat JMS messages. However, the log is still showing that it is >> >> > emitting messages. >> >> > >> >> > It's like the camel routes have just stopped working properly but I >> have >> >> no >> >> > idea why. >> >> > >> >> > >> >> > I see a bunch of these exceptions in the log like the following.. >> which >> >> is >> >> > suggesting it is failing to connect to splunk sometimes. >> >> > >> >> > 2011-05-07 12:41:50.457659500 2011-05-07 13:41:50,456 [Camel (main) >> >> thread >> >> > #0 - file:///home/wfe/var/ispy/events] ERROR >> >> > org.apache.camel.processor.DeadLetterChannel - Failed delivery for >> >> > exchangeId: ID-vp-prod-wfe-a-61766-1304769141692-0-95088. Exhausted >> after >> >> > delivery attempt: 11 caught: >> org.apache.mina.common.RuntimeIOException: >> >> > Failed to get the session.. Processed by failure processor: >> >> > sendTo(Endpoint[activemq://splunk.error] InOnly) >> >> > 2011-05-07 12:41:50.457664500 >> org.apache.mina.common.RuntimeIOException: >> >> > Failed to get the session. >> >> > 2011-05-07 12:41:50.457952500 at >> >> > >> >> >> org.apache.mina.common.support.DefaultConnectFuture.getSession(DefaultConnectFuture.java:63) >> >> > 2011-05-07 12:41:50.457953500 at >> >> > >> >> >> org.apache.camel.component.mina.MinaProducer.openConnection(MinaProducer.java:211) >> >> > 2011-05-07 12:41:50.457954500 at >> >> > >> >> >> org.apache.camel.component.mina.MinaProducer.process(MinaProducer.java:80) >> >> > 2011-05-07 12:41:50.457955500 at >> >> > >> >> >> org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50) >> >> > 2011-05-07 12:41:50.457963500 at >> >> > >> >> >> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) >> >> > 2011-05-07 12:41:50.457964500 at >> >> > >> >> >> org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:104) >> >> > ..snip.. >> >> > 2011-05-07 12:41:47.042077500 Caused by: java.net.ConnectException: >> >> > Connection refused2011-05-07 12:41:47.042078500 at >> >> > sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> >> > 2011-05-07 12:41:47.042082500 at >> >> > sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) >> >> > 2011-05-07 12:41:47.042083500 at >> >> > >> >> >> org.apache.mina.transport.socket.nio.SocketConnector.processSessions(SocketConnector.java:293) >> >> > 2011-05-07 12:41:47.042084500 at >> >> > >> >> >> org.apache.mina.transport.socket.nio.SocketConnector.access$800(SocketConnector.java:53) >> >> > 2011-05-07 12:41:47.042085500 at >> >> > >> >> >> org.apache.mina.transport.socket.nio.SocketConnector$Worker.run(SocketConnector.java:383) >> >> > 2011-05-07 12:41:47.042089500 at >> >> > >> >> >> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51) >> >> > 2011-05-07 12:41:47.042090500 ... 3 more >> >> > >> >> > Any idea if this could be causing the routes to stop? >> >> > >> >> > Let me know if I should provide extra information. >> >> > >> >> > Thanks, Tom >> >> > >> >> >> > >> > >
