Hello Tom, sorry for the delay. > What do you mean by propagates to file and point?
In your first mail you don't specified what type of ErrorHandler you use. than i suppose that you use DefaultErrorHandler. When i say propagated back i'll want to say what is written in http://camel.apache.org/defaulterrorhandler.html Please read capter 5 of in camel in action, but pratically if there is no explicit error handling the file was not consume and with the delay param the file is ready to be picked up in the next scheduled poll. --Filippo 2011/5/9 Tom Howe <tomh...@artcore.com>: > Also.. why will adding a delay fix the issue? How long of a delay do you > recommend? > > On Mon, May 9, 2011 at 10:12 AM, Tom Howe <tomh...@artcore.com> wrote: > >> What do you mean by propagates to file and point? >> >> We do actually have an error handler - the previous route I posted was just >> a summary. Below is the full config for the 2 camel contexts in the app. The >> first is for the main message routing, the 2nd is for the heartbeat. >> >> The app sends messages to splunk and a heartbeart to an activeMQ. >> >> The symptons we are finding is that occasionally, the application >> completely stops sending messages. The log which normally shows both >> messages being sent suddendly quietens down and only shows the heartbeat >> messages (even though they are not arriving either). After a restart it >> continues fine. >> >> The exceptions found in the log that I previously pasted do not coincide >> with the time at which it stops so may be a red herring. >> >> This application runs on quite heavily loaded VMs. We wonder if it could be >> due to memory issues as other applications on the box have died due to >> running out of memory. >> >> <camelContext id="main" autoStartup="true" xmlns=" >> http://camel.apache.org/schema/spring";> >> >> <errorHandler id="mr.error" type="DeadLetterChannel" >> deadLetterUri="activemq:mr.error"> >> <redeliveryPolicy maximumRedeliveries="10" >> redeliveryDelay="100" logHandled="true"/> >> </errorHandler> >> >> <errorHandler id="mr.splunk.error" type="DeadLetterChannel" >> deadLetterUri="activemq:mr.splunk.error"> >> <redeliveryPolicy maximumRedeliveries="10" >> redeliveryDelay="100" logHandled="true"/> >> </errorHandler> >> >> <!--direct endpoints--> >> <endpoint id="input" uri="${mr.input.endpoint}"/> >> <endpoint id="output" uri="${mr.output.endpoint}"/> >> >> <endpoint id="splunk" uri="${splunk.endpoint}"/> >> <!--<endpoint id="splunk.in" uri="${splunk.in.endpoint}"/>--> >> >> <!-- convert ispy messages to xml and place in general in queue --> >> <route autoStartup="true" errorHandlerRef="mr.error" xmlns=" >> http://camel.apache.org/schema/spring";> >> <from ref="input"/> >> <!--<log message="Message passed f=${header.CamelFileName} >> #${header.JMSMessageID} from: ${header.JMSDestination} to output"--> >> <!--loggingLevel="INFO"/>--> >> >> <convertBodyTo type="java.lang.String"/> >> <bean ref="ispyJsonToXml" method="process"/> >> <choice> >> <when> >> <xpath>/*[starts-with(@name, 'wfe.')]</xpath> >> <to uri="direct:splunk.in"/> >> </when> >> <otherwise> >> <log message="sending message to activmq" >> loggingLevel="INFO"/> >> <to ref="output"/> >> </otherwise> >> </choice> >> </route> >> >> >> <!-- filter and convert all messages for splunk to splunk format >> --> >> <route autoStartup="true" errorHandlerRef="mr.splunk.error" xmlns=" >> http://camel.apache.org/schema/spring";> >> <from uri="direct:splunk.in"/> >> <choice> >> <when> >> <!-- convert ispy-events --> >> <xpath>/*[local-name() = "event"][./*/*[local-name() = >> "ispy-event"]]</xpath> >> <bean ref="ispyXmlToSplunk"/> >> </when> >> <otherwise> >> <!-- convert everything else by grabbing envelope >> information --> >> <bean ref="envelopeXmlToSplunk"/> >> </otherwise> >> </choice> >> <log message="sending message to splunk" loggingLevel="INFO"/> >> <to ref="splunk"/> >> </route> >> >> >> </camelContext> >> >> <bean id="createHeartBeat" class="agentsmith.generators.CreateHeartBeat"> >> <!--<property name="eventName" >> value="${personality}.heartbeat"/>--> >> <property name="eventName" ref="heartBeatName"/> >> </bean> >> >> <camelContext id="heartbeat" autoStartup="true" xmlns=" >> http://camel.apache.org/schema/spring";> >> >> <endpoint id="general2.in" camelContextId="heartbeat" >> uri="${general.in.endpoint}"/> >> >> <endpoint id="heartbeattimer" camelContextId="heartbeat" >> uri="quartz://mygroup/heartbeat?cron=0+*+*+?+*+*"/> >> >> <route autoStartup="true" inheritErrorHandler="true" xmlns=" >> http://camel.apache.org/schema/spring";> >> <from ref="heartbeattimer"/> >> <bean ref="createHeartBeat"/> >> <to ref="general2.in"/> >> </route> >> >> </camelContext> >> >> >> >> >> >> >> >> >> On Mon, May 9, 2011 at 9:49 AM, Filippo Balicchia >> <fbalicc...@gmail.com>wrote: >> >>> If there is no explicit error handling the default error i triggered, >>> and the exception is propagates it back to the caller. >>> In your case seems to have network error then then error is propagates >>> to file and point. >>> >>> To resolve your problem i just add 'delay' parameter on your >>> expression. With this parameter you instruct camel >>> to wait milliseconds before the next poll of the file/directory >>> >>> Cheers. >>> >>> --Filippo >>> >>> >>> 2011/5/8 Tom Howe <tomh...@artcore.com>: >>> > Most of the messages are flowing to splunk fine. A few are failing to >>> get >>> > through - possibly we are flooding splunk - hence those errors. >>> > But what I dont understand is why the process then stops processing any >>> more >>> > messages. >>> > >>> > >>> > On Sun, May 8, 2011 at 6:07 AM, Filippo Balicchia <fbalicc...@gmail.com >>> >wrote: >>> > >>> >> Does your splunk accept you connection ? >>> >> >>> >> From stacktrace seems that refused your connection >>> >> >>> >> >>> >> 2011-05-07 12:41:47.042077500 Caused by: java.net.ConnectException: >>> >> Connection refused2011-05-07 12:41:47.042078500 at >>> >> >>> >> >>> >> --Filippo >>> >> >>> >> >>> >> 2011/5/7 Tom Howe <tomh...@artcore.com>: >>> >> > I have a simple java app that collect files of disk, converts the >>> [json] >>> >> > contents and then forwards it to a splunk instance over tcp using >>> mina. >>> >> > >>> >> > >>> >> > <endpoint id="input" uri="${input.endpoint}"/> >>> >> > <endpoint id="output" uri="${output.endpoint}"/> >>> >> > >>> >> > <route autoStartup="true" xmlns=" >>> >> > http://camel.apache.org/schema/spring";> >>> >> > <from ref="input"/> >>> >> > <convertBodyTo type="java.lang.String"/> >>> >> > <bean ref="jsonToSplunk"/> >>> >> > <to ref="splunk" /> >>> >> > </route> >>> >> > >>> >> > input.endpoint = >>> >> > file:/dir/to/files?delete=true&recursive=true&maxMessagesPerPoll=500 >>> >> > output.endpoint = >>> >> > >>> mina:tcp://splunkhost/?textline=true&textlineDelimiter=UNIX&sync=false >>> >> > >>> >> > It also emits a heartbeat message every minute over JMS using camel >>> >> quartz >>> >> > endpoint. >>> >> > >>> >> > The problem I am having is that after about a day of running, the >>> process >>> >> > stops picking up files and sending them to splunk and stops sending >>> the >>> >> > heartbeat JMS messages. However, the log is still showing that it is >>> >> > emitting messages. >>> >> > >>> >> > It's like the camel routes have just stopped working properly but I >>> have >>> >> no >>> >> > idea why. >>> >> > >>> >> > >>> >> > I see a bunch of these exceptions in the log like the following.. >>> which >>> >> is >>> >> > suggesting it is failing to connect to splunk sometimes. >>> >> > >>> >> > 2011-05-07 12:41:50.457659500 2011-05-07 13:41:50,456 [Camel (main) >>> >> thread >>> >> > #0 - file:///home/wfe/var/ispy/events] ERROR >>> >> > org.apache.camel.processor.DeadLetterChannel - Failed delivery for >>> >> > exchangeId: ID-vp-prod-wfe-a-61766-1304769141692-0-95088. Exhausted >>> after >>> >> > delivery attempt: 11 caught: >>> org.apache.mina.common.RuntimeIOException: >>> >> > Failed to get the session.. Processed by failure processor: >>> >> > sendTo(Endpoint[activemq://splunk.error] InOnly) >>> >> > 2011-05-07 12:41:50.457664500 >>> org.apache.mina.common.RuntimeIOException: >>> >> > Failed to get the session. >>> >> > 2011-05-07 12:41:50.457952500 at >>> >> > >>> >> >>> org.apache.mina.common.support.DefaultConnectFuture.getSession(DefaultConnectFuture.java:63) >>> >> > 2011-05-07 12:41:50.457953500 at >>> >> > >>> >> >>> org.apache.camel.component.mina.MinaProducer.openConnection(MinaProducer.java:211) >>> >> > 2011-05-07 12:41:50.457954500 at >>> >> > >>> >> >>> org.apache.camel.component.mina.MinaProducer.process(MinaProducer.java:80) >>> >> > 2011-05-07 12:41:50.457955500 at >>> >> > >>> >> >>> org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50) >>> >> > 2011-05-07 12:41:50.457963500 at >>> >> > >>> >> >>> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77) >>> >> > 2011-05-07 12:41:50.457964500 at >>> >> > >>> >> >>> org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:104) >>> >> > ..snip.. >>> >> > 2011-05-07 12:41:47.042077500 Caused by: java.net.ConnectException: >>> >> > Connection refused2011-05-07 12:41:47.042078500 at >>> >> > sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> >> > 2011-05-07 12:41:47.042082500 at >>> >> > sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) >>> >> > 2011-05-07 12:41:47.042083500 at >>> >> > >>> >> >>> org.apache.mina.transport.socket.nio.SocketConnector.processSessions(SocketConnector.java:293) >>> >> > 2011-05-07 12:41:47.042084500 at >>> >> > >>> >> >>> org.apache.mina.transport.socket.nio.SocketConnector.access$800(SocketConnector.java:53) >>> >> > 2011-05-07 12:41:47.042085500 at >>> >> > >>> >> >>> org.apache.mina.transport.socket.nio.SocketConnector$Worker.run(SocketConnector.java:383) >>> >> > 2011-05-07 12:41:47.042089500 at >>> >> > >>> >> >>> org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51) >>> >> > 2011-05-07 12:41:47.042090500 ... 3 more >>> >> > >>> >> > Any idea if this could be causing the routes to stop? >>> >> > >>> >> > Let me know if I should provide extra information. >>> >> > >>> >> > Thanks, Tom >>> >> > >>> >> >>> > >>> >> >> >
