On Sun, May 3, 2009 at 12:15 AM, bwtaylor <bryan_w_tay...@yahoo.com> wrote:
>
> I have information that is coming to me in the logs of an application that I
> don't control. I want to take the lines of the log as they are written (like
> tail -f in unix) and process each line with Camel. Unfortunately for me, the
> "current" log file name changes because the app uses log rotation. The
> filenames have a structured pattern like {stub}-{date-time}.log
>
> It looks like Camel 2.0 will allow me to use the stream component to tail a
> file, but it also appears to require the filename to be static.
>
> One solution idea would go like this:
> 1) poll the directory list for files matching the log file pattern
> 2) determine the latest file and pass just this file through an idempotency
> filter
> 3) When a new filename arrives, stop the old stream component based route,
> start a new one
>
> Is something like this possible? Has anybody written a camel Processor that
> dynamically modifies Camel routes? Another idea would be to modify the file
> parameter in place in the existing route built by the stream component. What
> are the pro's and con's of this approache vs dynamic route building?
a)
You could use a Polling Consumer EIP
http://camel.apache.org/polling-consumer.html
Here you can dynamic construct the endpoint to poll, that is based on
the new log file.
And then receive() in a loop until no more data.
However not really what you are looking for, but the polling consumer
EIP unlike the event driven allows you to be in control on when and
what to poll.
b)
About the dynamic routing aspect. Its possible in Camel 1.x but even
better in Camel 2.0 to construct dynamic routes.
You can build you routes and add it to the camel context. This allows
you to add totally new routes, or modify existing to add new EIP
pattern etc.
c)
But its also possible to alter the existing route at runtime, by only
changing URI or other options. Eg the route skeleton is the same, and
you only need to set a new filename on the stream endpoint.
So what you can do is to stop the route, edit the endpoint uri on the
stream endpoint setting the updated filename, and then start the route
again.
See this unit test that does this:
org.apache.camel.impl.StartAndStopRoutesTest
https://svn.apache.org/repos/asf/camel/trunk/camel-core/src/test/java/org/apache/camel/impl/StartAndStopRoutesTest.java
>
> Also, I see the stream component will support a scanStreamDelay parameter,
> which defaults to 0 and specifies the millis between reads. Let's say I use
> this and 17 lines have arrived when it reads. Do these all go in one message
> or 17?
The stream component for files is line by line based, eg you get 17 messages.
> --
> View this message in context:
> http://www.nabble.com/Tailing-a-Rotating-Log-tp23350759p23350759.html
> Sent from the Camel - Users (activemq) mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
