Hi Devs, As mentioned above to create a custom IdempotentRepository and generate the file checksum I need the file that is being processed inside the repository. Was going through the documentation and it seems I can implement ExchangeIdempotentRepository and override the below mentioned methods in it boolean *add <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#add(org.apache.camel.Exchange,%20E)>* (Exchange <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> exchange, E <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> key) Adds the key to the repository. boolean *confirm <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#confirm(org.apache.camel.Exchange,%20E)>* (Exchange <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> exchange, E <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> key) Confirms the key, after the exchange has been processed successfully. boolean *contains <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#contains(org.apache.camel.Exchange,%20E)>* (Exchange <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> exchange, E <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> key) Returns true if this repository contains the specified element. boolean *remove <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#remove(org.apache.camel.Exchange,%20E)>* (Exchange <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> exchange, E <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> key) Removes the key from the repository. The issue which I have is how to pass the exchange to the above repository . It seems while using idempotentKey I can only pass the information regarding the file like name, size, modification timestamp. It would be of great help if anyone can suggest how to pass the Exchange <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> exchange, E <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> key both to the IdempotentRepository using idempotentKey
Thanks and Regards Deepak Anand On Thu, Jul 11, 2019 at 5:17 PM Deepak Anand <[email protected]> wrote: > Thanks Claus for your response , will try to implement the way you > suggested > > Regards > Deepak Anand > > > On Wed, Jul 10, 2019 at 8:34 PM Claus Ibsen <[email protected]> wrote: > >> Hi >> >> You can likely try to build a custom IdempotentRepository that extends >> the Redis, and then in all the methods, you get the key from Camel >> which is the absolute file name, which you then use to calculate your >> hash and then call the super method with the hash key instead. >> >> eg something like this: >> >> public boolean contains(exchange, key) >> string hashKey = .... >> return super.contains(exchange, hashKey); >> >> >> On Wed, Jul 10, 2019 at 9:00 PM Claus Ibsen <[email protected]> >> wrote: >> > >> > Hi >> > >> > Yeah its a limitation that idempontentKey can only use the information >> > from the file itself such as its name, size, modification timestamp >> > etc. Not a custom hashcode. >> > >> > On Tue, Jul 9, 2019 at 7:22 PM Deepak Anand <[email protected]> >> wrote: >> > > >> > > Thanks Claus for you response. >> > > The same files can be present in different sub directory , so I need >> > > checksum to identify them. >> > > How in the file endpoint I can generate the checksum and add it to the >> > > idempotentRepository . >> > > May be a silly question but I am new to Camel earlier I was >> generating the >> > > checksum of the file in the processor and adding it to the header and >> > > passing it to the idempotent consumer >> > > >> > > Thanks and Regards >> > > Deepak Anand >> > > >> > > >> > > On Tue, Jul 9, 2019 at 6:08 PM Claus Ibsen <[email protected]> >> wrote: >> > > >> > > > Set idempotentRepository on the file endpoint instead and remove >> > > > idempotent consumer in the route. >> > > > >> > > > On Tue, Jul 9, 2019 at 7:01 PM Deepak Anand < >> [email protected]> >> > > > wrote: >> > > > > >> > > > > Hi Devs , >> > > > > >> > > > > I have a requirement to read from a directory where there are no >> of files >> > > > > in the subdirectories of it . I am using the camel File component >> for it >> > > > . >> > > > > I don't want to process the same file which has been processed >> earlier >> > > > > using an idempotentConsumer and using RedisCache for it. >> > > > > In the cache I am adding the checksum to identify the duplicate >> file. The >> > > > > problem is that after reading more than 1000 files the same >> files(which >> > > > > were read earlier) are being read again and again which I don't >> want . >> > > > > Below is the snippet of the camel route >> > > > > >> > > > > from("file://" + sourceLoc + "/?recursive=true&noop=true" >> > > > > >> > > > >> +"&idempotent=true&idempotentKey=${file:name}-${file:modified}&readLockRemoveOnCommit=false") >> > > > > .log("the read file is >> > > > > ${file:name}-${file:modified} ") >> > > > > .filter() >> > > > > .method(SourceMinAgeFilter.class, "accept") >> > > > > .process("checksumprocessor") >> > > > > .idempotentConsumer(header("fileCheckSum"), >> > > > > >> > > > > >> RedisIdempotentRepository.redisIdempotentRepository(stringredisTemplate, >> > > > > redisStoragename)) >> > > > > .eager(true) >> > > > > .skipDuplicate(false) >> > > > > .choice() >> > > > > >> > > > > >> .when(exchangeProperty(Exchange.DUPLICATE_MESSAGE).isEqualTo(true)) >> > > > > .log("file ${file:onlyname} is a duplicate and >> hence >> > > > > skipped.") >> > > > > .otherwise() >> > > > > .process("fileprocessor") >> > > > > .log("processed file name from Source Folder" + >> > > > > ("${file:onlyname}")) >> > > > > .log("Total time to process a file : from Source >> Folder >> > > > " + >> > > > > (System.currentTimeMillis() - starttime) + " ms").end(); >> > > > > >> > > > > Could you please let me know what needs to be done so that the >> same file >> > > > is >> > > > > not read again and again. Since I am using RedisCache it is not >> getting >> > > > > processed as the checksum is same but I don't want to read the >> file again >> > > > > that has been processed >> > > > > >> > > > > Thanks and Regards >> > > > > Deepak Anand >> > > > >> > > > >> > > > >> > > > -- >> > > > Claus Ibsen >> > > > ----------------- >> > > > http://davsclaus.com @davsclaus >> > > > Camel in Action 2: https://www.manning.com/ibsen2 >> > > > >> > >> > >> > >> > -- >> > Claus Ibsen >> > ----------------- >> > http://davsclaus.com @davsclaus >> > Camel in Action 2: https://www.manning.com/ibsen2 >> >> >> >> -- >> Claus Ibsen >> ----------------- >> http://davsclaus.com @davsclaus >> Camel in Action 2: https://www.manning.com/ibsen2 >> >
