Hi,
I had a very similar problem. I wrote a little bean which writes the
last number processed to a file. Another method gives me the last number
processed.
regards, Marco
Am 06.07.2012 11:53, schrieb Christian Lipp:
Do you mean " eagerMaxMessagesPerPoll" ?
I set it to false, but nothing changed.
In the camel code I could only find
contains in GenericFileConsumer
Add, remove in GenericFileOnCompletion
Regards, CL
-----Ursprüngliche Nachricht-----
Von: Sam (Stephen Samuel) [mailto:samspad...@gmail.com]
Gesendet: Freitag, 06. Juli 2012 11:17
An: users@camel.apache.org
Betreff: Re: IdempotentRepository
"This operation is used if the option eager has been enabled."
Take a look at that, that might be why confirm is not being called.
On Fri, Jul 6, 2012 at 7:17 AM, Christian Lipp<christian.l...@xion.at>wrote:
Hello!
I am fetching files from a FTP server (severall GB for the next years).
The files are produced daily in directories which correspond to the
date, like
- 20120501
- 20120502
- ...
I have only read rights and I am not the only consumer. This means
that they keep the last month or so on the server and I fetch on a daily base.
To avoid that I am fetching files twice I want to use an
IdempotentRepository implementation. I don't want to save each file in
a database or in a text file because the service will run for years
and this is just unnecessary data.
What I want to store is the last processed date only. This handles
just the directories and would mean that I need some other strategy
for the files. I could combine this approach with the default in
memory based store. But let just stick to the directories:
I read the directory sorted by file name. The IdempotentRepository is
called by the FtpConsumer with
- start()
- contains() for every directory and file
- add() for files only
and that's it. No stop(), no confirm(). When I have errors, sometimes
remove() is called. Since the repository is called only with String
(the full path) I have not information if I deal with directories or
files. I know it from the structure, but I am not able to implement a
generic solution.
Anyway the idea is:
- Store the LastProcessedDate inside the repository
- contains(): if the path contains an already processed date
(<LastProcessedDate) then I skip it (return true) otherwise return false.
- add(): if add() jumps to the next directory I set the
LastProcessedDate to the directory before
The only problem is the last processed directory: even if it is
finished I do not get the chance to mark it as processed (set
LastProcessedDate to its value).
So finally my questions: do you think this approach makes sense and if
yes: how would you deal with the last processed directory?
If no, how would you solve it?
Thanks and kind regards, Christian
--
-Sam
