Hi Devs, As mentioned above to create a custom IdempotentRepository and generate the file checksum I need the file that is being processed inside the repository. Was going through the documentation and it seems I can implement ExchangeIdempotentRepository and override the below mentioned methods in it
boolean add(Exchange exchange, E key) boolean confirm(Exchange exchange, E key) boolean contains(Exchange exchange, E key) boolean remove(Exchange exchange, E key) The issue which I have is how to pass the exchange to the above repository . It seems while using idempotentKey I can only pass the information regarding the file like name, size, modification timestamp. It would be of great help if anyone can suggest how to pass the Exchange exchange, E key both to the IdempotentRepository using idempotentKey Thanks and Regards Deepak Anand On Fri, Jul 12, 2019 at 1:58 PM Deepak Anand <[email protected]> wrote: > Hi Devs, > > As mentioned above to create a custom IdempotentRepository and generate > the file checksum I need the file that is being processed inside the > repository. Was going through the documentation and it seems I can > implement ExchangeIdempotentRepository and override the below mentioned > methods in it > boolean *add > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#add(org.apache.camel.Exchange,%20E)>* > (Exchange > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> > exchange, E > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> > key) > Adds the key to the repository. > boolean *confirm > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#confirm(org.apache.camel.Exchange,%20E)>* > (Exchange > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> > exchange, E > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> > key) > Confirms the key, after the exchange has been processed successfully. > boolean *contains > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#contains(org.apache.camel.Exchange,%20E)>* > (Exchange > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> > exchange, E > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> > key) > Returns true if this repository contains the specified element. > boolean *remove > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#remove(org.apache.camel.Exchange,%20E)>* > (Exchange > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> > exchange, E > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> > key) > Removes the key from the repository. > The issue which I have is how to pass the exchange to the above repository > . It seems while using idempotentKey I can only pass the information > regarding the file like name, size, modification timestamp. > It would be of great help if anyone can suggest how to pass the Exchange > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> > exchange, E > <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> > key > both to the IdempotentRepository using idempotentKey > > Thanks and Regards > Deepak Anand > > > On Thu, Jul 11, 2019 at 5:17 PM Deepak Anand <[email protected]> > wrote: > >> Thanks Claus for your response , will try to implement the way you >> suggested >> >> Regards >> Deepak Anand >> >> >> On Wed, Jul 10, 2019 at 8:34 PM Claus Ibsen <[email protected]> >> wrote: >> >>> Hi >>> >>> You can likely try to build a custom IdempotentRepository that extends >>> the Redis, and then in all the methods, you get the key from Camel >>> which is the absolute file name, which you then use to calculate your >>> hash and then call the super method with the hash key instead. >>> >>> eg something like this: >>> >>> public boolean contains(exchange, key) >>> string hashKey = .... >>> return super.contains(exchange, hashKey); >>> >>> >>> On Wed, Jul 10, 2019 at 9:00 PM Claus Ibsen <[email protected]> >>> wrote: >>> > >>> > Hi >>> > >>> > Yeah its a limitation that idempontentKey can only use the information >>> > from the file itself such as its name, size, modification timestamp >>> > etc. Not a custom hashcode. >>> > >>> > On Tue, Jul 9, 2019 at 7:22 PM Deepak Anand <[email protected]> >>> wrote: >>> > > >>> > > Thanks Claus for you response. >>> > > The same files can be present in different sub directory , so I need >>> > > checksum to identify them. >>> > > How in the file endpoint I can generate the checksum and add it to >>> the >>> > > idempotentRepository . >>> > > May be a silly question but I am new to Camel earlier I was >>> generating the >>> > > checksum of the file in the processor and adding it to the header and >>> > > passing it to the idempotent consumer >>> > > >>> > > Thanks and Regards >>> > > Deepak Anand >>> > > >>> > > >>> > > On Tue, Jul 9, 2019 at 6:08 PM Claus Ibsen <[email protected]> >>> wrote: >>> > > >>> > > > Set idempotentRepository on the file endpoint instead and remove >>> > > > idempotent consumer in the route. >>> > > > >>> > > > On Tue, Jul 9, 2019 at 7:01 PM Deepak Anand < >>> [email protected]> >>> > > > wrote: >>> > > > > >>> > > > > Hi Devs , >>> > > > > >>> > > > > I have a requirement to read from a directory where there are no >>> of files >>> > > > > in the subdirectories of it . I am using the camel File >>> component for it >>> > > > . >>> > > > > I don't want to process the same file which has been processed >>> earlier >>> > > > > using an idempotentConsumer and using RedisCache for it. >>> > > > > In the cache I am adding the checksum to identify the duplicate >>> file. The >>> > > > > problem is that after reading more than 1000 files the same >>> files(which >>> > > > > were read earlier) are being read again and again which I don't >>> want . >>> > > > > Below is the snippet of the camel route >>> > > > > >>> > > > > from("file://" + sourceLoc + "/?recursive=true&noop=true" >>> > > > > >>> > > > >>> +"&idempotent=true&idempotentKey=${file:name}-${file:modified}&readLockRemoveOnCommit=false") >>> > > > > .log("the read file is >>> > > > > ${file:name}-${file:modified} ") >>> > > > > .filter() >>> > > > > .method(SourceMinAgeFilter.class, "accept") >>> > > > > .process("checksumprocessor") >>> > > > > .idempotentConsumer(header("fileCheckSum"), >>> > > > > >>> > > > > >>> RedisIdempotentRepository.redisIdempotentRepository(stringredisTemplate, >>> > > > > redisStoragename)) >>> > > > > .eager(true) >>> > > > > .skipDuplicate(false) >>> > > > > .choice() >>> > > > > >>> > > > > >>> .when(exchangeProperty(Exchange.DUPLICATE_MESSAGE).isEqualTo(true)) >>> > > > > .log("file ${file:onlyname} is a duplicate and >>> hence >>> > > > > skipped.") >>> > > > > .otherwise() >>> > > > > .process("fileprocessor") >>> > > > > .log("processed file name from Source Folder" + >>> > > > > ("${file:onlyname}")) >>> > > > > .log("Total time to process a file : from Source >>> Folder >>> > > > " + >>> > > > > (System.currentTimeMillis() - starttime) + " ms").end(); >>> > > > > >>> > > > > Could you please let me know what needs to be done so that the >>> same file >>> > > > is >>> > > > > not read again and again. Since I am using RedisCache it is not >>> getting >>> > > > > processed as the checksum is same but I don't want to read the >>> file again >>> > > > > that has been processed >>> > > > > >>> > > > > Thanks and Regards >>> > > > > Deepak Anand >>> > > > >>> > > > >>> > > > >>> > > > -- >>> > > > Claus Ibsen >>> > > > ----------------- >>> > > > http://davsclaus.com @davsclaus >>> > > > Camel in Action 2: https://www.manning.com/ibsen2 >>> > > > >>> > >>> > >>> > >>> > -- >>> > Claus Ibsen >>> > ----------------- >>> > http://davsclaus.com @davsclaus >>> > Camel in Action 2: https://www.manning.com/ibsen2 >>> >>> >>> >>> -- >>> Claus Ibsen >>> ----------------- >>> http://davsclaus.com @davsclaus >>> Camel in Action 2: https://www.manning.com/ibsen2 >>> >>
