I have a simple java app that collect files of disk, converts the [json]
contents and then forwards it to a splunk instance over tcp using mina.
<endpoint id="input" uri="${input.endpoint}"/>
<endpoint id="output" uri="${output.endpoint}"/>
<route autoStartup="true" xmlns="
http://camel.apache.org/schema/spring";>
<from ref="input"/>
<convertBodyTo type="java.lang.String"/>
<bean ref="jsonToSplunk"/>
<to ref="splunk" />
</route>
input.endpoint =
file:/dir/to/files?delete=true&recursive=true&maxMessagesPerPoll=500
output.endpoint =
mina:tcp://splunkhost/?textline=true&textlineDelimiter=UNIX&sync=false
It also emits a heartbeat message every minute over JMS using camel quartz
endpoint.
The problem I am having is that after about a day of running, the process
stops picking up files and sending them to splunk and stops sending the
heartbeat JMS messages. However, the log is still showing that it is
emitting messages.
It's like the camel routes have just stopped working properly but I have no
idea why.
I see a bunch of these exceptions in the log like the following.. which is
suggesting it is failing to connect to splunk sometimes.
2011-05-07 12:41:50.457659500 2011-05-07 13:41:50,456 [Camel (main) thread
#0 - file:///home/wfe/var/ispy/events] ERROR
org.apache.camel.processor.DeadLetterChannel - Failed delivery for
exchangeId: ID-vp-prod-wfe-a-61766-1304769141692-0-95088. Exhausted after
delivery attempt: 11 caught: org.apache.mina.common.RuntimeIOException:
Failed to get the session.. Processed by failure processor:
sendTo(Endpoint[activemq://splunk.error] InOnly)
2011-05-07 12:41:50.457664500 org.apache.mina.common.RuntimeIOException:
Failed to get the session.
2011-05-07 12:41:50.457952500 at
org.apache.mina.common.support.DefaultConnectFuture.getSession(DefaultConnectFuture.java:63)
2011-05-07 12:41:50.457953500 at
org.apache.camel.component.mina.MinaProducer.openConnection(MinaProducer.java:211)
2011-05-07 12:41:50.457954500 at
org.apache.camel.component.mina.MinaProducer.process(MinaProducer.java:80)
2011-05-07 12:41:50.457955500 at
org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50)
2011-05-07 12:41:50.457963500 at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:77)
2011-05-07 12:41:50.457964500 at
org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:104)
..snip..
2011-05-07 12:41:47.042077500 Caused by: java.net.ConnectException:
Connection refused2011-05-07 12:41:47.042078500 at
sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
2011-05-07 12:41:47.042082500 at
sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
2011-05-07 12:41:47.042083500 at
org.apache.mina.transport.socket.nio.SocketConnector.processSessions(SocketConnector.java:293)
2011-05-07 12:41:47.042084500 at
org.apache.mina.transport.socket.nio.SocketConnector.access$800(SocketConnector.java:53)
2011-05-07 12:41:47.042085500 at
org.apache.mina.transport.socket.nio.SocketConnector$Worker.run(SocketConnector.java:383)
2011-05-07 12:41:47.042089500 at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
2011-05-07 12:41:47.042090500 ... 3 more
Any idea if this could be causing the routes to stop?
Let me know if I should provide extra information.
Thanks, Tom
