Hi
You can likely try to build a custom IdempotentRepository that extends
the Redis, and then in all the methods, you get the key from Camel
which is the absolute file name, which you then use to calculate your
hash and then call the super method with the hash key instead.
eg something like this:
public boolean contains(exchange, key)
string hashKey = ....
return super.contains(exchange, hashKey);
On Wed, Jul 10, 2019 at 9:00 PM Claus Ibsen <claus.ib...@gmail.com> wrote:
>
> Hi
>
> Yeah its a limitation that idempontentKey can only use the information
> from the file itself such as its name, size, modification timestamp
> etc. Not a custom hashcode.
>
> On Tue, Jul 9, 2019 at 7:22 PM Deepak Anand <deepakanand2...@gmail.com> wrote:
> >
> > Thanks Claus for you response.
> > The same files can be present in different sub directory , so I need
> > checksum to identify them.
> > How in the file endpoint I can generate the checksum and add it to the
> > idempotentRepository .
> > May be a silly question but I am new to Camel earlier I was generating the
> > checksum of the file in the processor and adding it to the header and
> > passing it to the idempotent consumer
> >
> > Thanks and Regards
> > Deepak Anand
> >
> >
> > On Tue, Jul 9, 2019 at 6:08 PM Claus Ibsen <claus.ib...@gmail.com> wrote:
> >
> > > Set idempotentRepository on the file endpoint instead and remove
> > > idempotent consumer in the route.
> > >
> > > On Tue, Jul 9, 2019 at 7:01 PM Deepak Anand <deepakanand2...@gmail.com>
> > > wrote:
> > > >
> > > > Hi Devs ,
> > > >
> > > > I have a requirement to read from a directory where there are no of
> > > > files
> > > > in the subdirectories of it . I am using the camel File component for it
> > > .
> > > > I don't want to process the same file which has been processed earlier
> > > > using an idempotentConsumer and using RedisCache for it.
> > > > In the cache I am adding the checksum to identify the duplicate file.
> > > > The
> > > > problem is that after reading more than 1000 files the same files(which
> > > > were read earlier) are being read again and again which I don't want .
> > > > Below is the snippet of the camel route
> > > >
> > > > from("file://" + sourceLoc + "/?recursive=true&noop=true"
> > > >
> > > +"&idempotent=true&idempotentKey=${file:name}-${file:modified}&readLockRemoveOnCommit=false")
> > > > .log("the read file is
> > > > ${file:name}-${file:modified} ")
> > > > .filter()
> > > > .method(SourceMinAgeFilter.class, "accept")
> > > > .process("checksumprocessor")
> > > > .idempotentConsumer(header("fileCheckSum"),
> > > >
> > > > RedisIdempotentRepository.redisIdempotentRepository(stringredisTemplate,
> > > > redisStoragename))
> > > > .eager(true)
> > > > .skipDuplicate(false)
> > > > .choice()
> > > >
> > > > .when(exchangeProperty(Exchange.DUPLICATE_MESSAGE).isEqualTo(true))
> > > > .log("file ${file:onlyname} is a duplicate and hence
> > > > skipped.")
> > > > .otherwise()
> > > > .process("fileprocessor")
> > > > .log("processed file name from Source Folder" +
> > > > ("${file:onlyname}"))
> > > > .log("Total time to process a file : from Source Folder
> > > " +
> > > > (System.currentTimeMillis() - starttime) + " ms").end();
> > > >
> > > > Could you please let me know what needs to be done so that the same file
> > > is
> > > > not read again and again. Since I am using RedisCache it is not getting
> > > > processed as the checksum is same but I don't want to read the file
> > > > again
> > > > that has been processed
> > > >
> > > > Thanks and Regards
> > > > Deepak Anand
> > >
> > >
> > >
> > > --
> > > Claus Ibsen
> > > -----------------
> > > http://davsclaus.com @davsclaus
> > > Camel in Action 2: https://www.manning.com/ibsen2
> > >
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
