I am trying to write a process that will use a file endpoint (camel 2.1.0) to
read from a directory.
I need the process to read a file from the directory and then do some
processing on the contents (namely hitting a REST service for each record in
the file). We have been asked to limit the number of threads that are
hitting the service to 5. So we decided to simply process 5 files at a time
(to avoid concurrency issues reading 1 file and writing to 1 file with 5
threads)
I tried a few different approaches, and I wanted to see if there was a way
to do what I want.
Approach 1:
from("file://incoming").to("seda:filequeue")
from("seda:filequeue").thread(5).process()
Now - this reads in ALL of the files in the directory (places camelLock on
all) and then sends them to the seda endpoint. I saw log messages that
referred to thread 1 through 6. But from what I read on the documentation,
thread() is not necessarily going t limit it at that number.
Approach 2:
from("file://incoming").thread(5).process()
This only processed 5 files at a time - but created camelLocks on all files
in the directory.
So then I tried approach 3:
from("file://incoming").to("seda:filequeue")
from("seda:filequeue?concurrentConsumers=5").process()
Again this seems to work, however it puts a camelLock on all the files
(because they were all processed by the first part of the route, they are
just queued up in the second).
While approach 3 works - what I would really like is to not have the
camelLock placed on the files that are not being processed.
So watching the directory, there would be (at most) 5 files with camelLock
files created at a time, when they finish they are moved to the .camel
directory, and then it starts processing the next file in the directory.
Is that possible? Is there anything I should be sure to do in an error
route so that I "roll back" the camel locks to ensure that unprocessed files
are ready to process the next time the application starts?
--
View this message in context:
http://old.nabble.com/Processing-5-files-at-a-time---Threads--SEDA-%2B-Concurrent-Consumers--tp26960942p26960942.html
Sent from the Camel - Users mailing list archive at Nabble.com.
