Thanks for the clarification. This does completely explain the situation,
and applying a variant of your solution "a" seems to have things working
much more reliably.
However, I'd like to suggest that you add a "preMove" option as it seems to
be pretty much a requirement for doing clustered seda-style processing from
a file endpoint. I suppose you could use the new noop/idempotent
capabilities that you have added in 2.0, but it's also nice to have
separation between files that have yet to be discovered and files that are
in process and also I'm a little leery of keeping state information like
that around as the file endpoint idempotent data presumably (maybe I'm
wrong?) doesn't persist across restarts, etc.
Also, is there a way to disable the implicit conversion of java.io.File to a
jms bytes message on JMS? This is also not desirable in this situation.
Putting gigabytes of byte data on jms does not work out very well. To get
around this we have to convert the java.io.File to a String prior to putting
it on JMS, and then convert it back to a java.io.File so we can give our
users the ability to use the camel built in transformers to transform to the
input method of their choice (byte[], InputStream, FileReader, java.io.File,
etc.). This works, but it kind of takes away from the "cleanliness" of the
routes.
Thanks for your help--we've found camel to be much superior in many ways to
another animal-named esb product that we're attempting to migrate away from.
Claus Ibsen-2 wrote:
>
> Hi
>
> #1
> The move is executed *AFTER* the routing.
> The idea is that you process the file while it's in the target folder
> (where its dropped) and after processing you can move it to a backup
> folder.
>
> #2
> Ah I speculate that the JMS consumer is faster than the file consumer
> so when you drop a JMS message with the filename pointing at the move
> folder then the JMS consumer in some circumstances be ahead of the
> file consumer and trying to get the file before it's actually moved
> there. Hence #1
>
>
> You could to fix
> =============
> b) use camel to route and move the file using pipes and filters
> from("file://inbox?delete=true").to("file://moved").to("jms:queue:pleasegetthefilenow");
> Note: However this will read the file content and save it as a new
> file, it's not a native File IO move operation
>
> a) move the file yourself and then afterwards send the JMS message.
> from("file://inbox").to("bean:myFileMover").to("jms:queue:pleasegetthefilenow");
>
> Using a POJO bean you can move the file yourself using File rename.
>
>
>
> /Claus Ibsen
> Apache Camel Committer
> Blog: http://davsclaus.blogspot.com/
>
>
>
> On Tue, Dec 2, 2008 at 11:51 PM, Christopher Hammack
> <[EMAIL PROTECTED]> wrote:
>>
>> In addition to the memory leak issue in Camel 1.5.0, I have a few other
>> concerns about the file consumer endpoint--some of which could be a
>> misunderstanding on my part:
>>
>> 1. When using the default move capability (moving a file to .camel)
>> after
>> it has been picked up, the java.io.File object refers to the path BEFORE
>> the
>> move, not after. So in order to actually read the file, my processing
>> code
>> must have knowledge of which path the file was moved to. Is this
>> intentional?
>>
>> 2. Occasionally, especially when the system is under considerable load,
>> the
>> java.io.File object that I get is not available in the moved location,
>> which
>> generates a FileNotFoundException. When I check that location later on,
>> the
>> file is in the correct location. Looking at the code, it seems like the
>> message should not be being propogated back to me prior to the rename
>> occurring, but it is apparently happening. Any thoughts?
>>
>> The use case for this is a very large number of small files is being
>> dropped
>> into a directory. This directory is then being scanned by camel's file
>> endpoint. The files as they are discovered are then moved to the .camel
>> directory, and the filename is put onto a jms endpoint. A clustered set
>> of
>> camel processors then pull the filename off the endpoint and process the
>> file, and then delete it.
>>
>> Any suggestions would be appreciated as right now I'm "stranding" about
>> one
>> out of every 1000 files because the processing checks for the file prior
>> to
>> the file actually being in the moved location.
>>
>> Thanks.
>> --
>> View this message in context:
>> http://www.nabble.com/Concerns-about-File-endpoint-tp20802855s22882p20802855.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context:
http://www.nabble.com/Concerns-about-File-endpoint-tp20802855s22882p20815087.html
Sent from the Camel - Users mailing list archive at Nabble.com.
