Hi Claus, I tried the approach suggested by you to write a custom Idempotent Repository. I did that and it worked Thanks a lot
Regards Deepak Anand On Fri, Jul 12, 2019 at 2:09 PM Deepak Anand <deepakanand2...@gmail.com> wrote: > Hi Devs, > > As mentioned above to create a custom IdempotentRepository and generate > the file checksum I need the file that is being processed inside the > repository. Was going through the documentation and it seems I can > implement ExchangeIdempotentRepository and override the below mentioned > methods in it > > boolean add(Exchange exchange, E key) > boolean confirm(Exchange exchange, E key) > boolean contains(Exchange exchange, E key) > boolean remove(Exchange exchange, E key) > > The issue which I have is how to pass the exchange to the above repository > . It seems while using idempotentKey I can only pass the information > regarding the file like name, size, modification timestamp. > It would be of great help if anyone can suggest how to pass the Exchange > exchange, E key both to the IdempotentRepository using idempotentKey > > Thanks and Regards > Deepak Anand > > On Fri, Jul 12, 2019 at 1:58 PM Deepak Anand <deepakanand2...@gmail.com> > wrote: > >> Hi Devs, >> >> As mentioned above to create a custom IdempotentRepository and generate >> the file checksum I need the file that is being processed inside the >> repository. Was going through the documentation and it seems I can >> implement ExchangeIdempotentRepository and override the below mentioned >> methods in it >> boolean *add >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#add(org.apache.camel.Exchange,%20E)>* >> (Exchange >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> >> exchange, E >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> >> key) >> Adds the key to the repository. >> boolean *confirm >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#confirm(org.apache.camel.Exchange,%20E)>* >> (Exchange >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> >> exchange, E >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> >> key) >> Confirms the key, after the exchange has been processed successfully. >> boolean *contains >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#contains(org.apache.camel.Exchange,%20E)>* >> (Exchange >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> >> exchange, E >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> >> key) >> Returns true if this repository contains the specified element. >> boolean *remove >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#remove(org.apache.camel.Exchange,%20E)>* >> (Exchange >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> >> exchange, E >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> >> key) >> Removes the key from the repository. >> The issue which I have is how to pass the exchange to the above repository >> . It seems while using idempotentKey I can only pass the information >> regarding the file like name, size, modification timestamp. >> It would be of great help if anyone can suggest how to pass the Exchange >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> >> exchange, E >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> >> key >> both to the IdempotentRepository using idempotentKey >> >> Thanks and Regards >> Deepak Anand >> >> >> On Thu, Jul 11, 2019 at 5:17 PM Deepak Anand <deepakanand2...@gmail.com> >> wrote: >> >>> Thanks Claus for your response , will try to implement the way you >>> suggested >>> >>> Regards >>> Deepak Anand >>> >>> >>> On Wed, Jul 10, 2019 at 8:34 PM Claus Ibsen <claus.ib...@gmail.com> >>> wrote: >>> >>>> Hi >>>> >>>> You can likely try to build a custom IdempotentRepository that extends >>>> the Redis, and then in all the methods, you get the key from Camel >>>> which is the absolute file name, which you then use to calculate your >>>> hash and then call the super method with the hash key instead. >>>> >>>> eg something like this: >>>> >>>> public boolean contains(exchange, key) >>>> string hashKey = .... >>>> return super.contains(exchange, hashKey); >>>> >>>> >>>> On Wed, Jul 10, 2019 at 9:00 PM Claus Ibsen <claus.ib...@gmail.com> >>>> wrote: >>>> > >>>> > Hi >>>> > >>>> > Yeah its a limitation that idempontentKey can only use the information >>>> > from the file itself such as its name, size, modification timestamp >>>> > etc. Not a custom hashcode. >>>> > >>>> > On Tue, Jul 9, 2019 at 7:22 PM Deepak Anand < >>>> deepakanand2...@gmail.com> wrote: >>>> > > >>>> > > Thanks Claus for you response. >>>> > > The same files can be present in different sub directory , so I need >>>> > > checksum to identify them. >>>> > > How in the file endpoint I can generate the checksum and add it to >>>> the >>>> > > idempotentRepository . >>>> > > May be a silly question but I am new to Camel earlier I was >>>> generating the >>>> > > checksum of the file in the processor and adding it to the header >>>> and >>>> > > passing it to the idempotent consumer >>>> > > >>>> > > Thanks and Regards >>>> > > Deepak Anand >>>> > > >>>> > > >>>> > > On Tue, Jul 9, 2019 at 6:08 PM Claus Ibsen <claus.ib...@gmail.com> >>>> wrote: >>>> > > >>>> > > > Set idempotentRepository on the file endpoint instead and remove >>>> > > > idempotent consumer in the route. >>>> > > > >>>> > > > On Tue, Jul 9, 2019 at 7:01 PM Deepak Anand < >>>> deepakanand2...@gmail.com> >>>> > > > wrote: >>>> > > > > >>>> > > > > Hi Devs , >>>> > > > > >>>> > > > > I have a requirement to read from a directory where there are >>>> no of files >>>> > > > > in the subdirectories of it . I am using the camel File >>>> component for it >>>> > > > . >>>> > > > > I don't want to process the same file which has been processed >>>> earlier >>>> > > > > using an idempotentConsumer and using RedisCache for it. >>>> > > > > In the cache I am adding the checksum to identify the duplicate >>>> file. The >>>> > > > > problem is that after reading more than 1000 files the same >>>> files(which >>>> > > > > were read earlier) are being read again and again which I don't >>>> want . >>>> > > > > Below is the snippet of the camel route >>>> > > > > >>>> > > > > from("file://" + sourceLoc + "/?recursive=true&noop=true" >>>> > > > > >>>> > > > >>>> +"&idempotent=true&idempotentKey=${file:name}-${file:modified}&readLockRemoveOnCommit=false") >>>> > > > > .log("the read file is >>>> > > > > ${file:name}-${file:modified} ") >>>> > > > > .filter() >>>> > > > > .method(SourceMinAgeFilter.class, "accept") >>>> > > > > .process("checksumprocessor") >>>> > > > > .idempotentConsumer(header("fileCheckSum"), >>>> > > > > >>>> > > > > >>>> RedisIdempotentRepository.redisIdempotentRepository(stringredisTemplate, >>>> > > > > redisStoragename)) >>>> > > > > .eager(true) >>>> > > > > .skipDuplicate(false) >>>> > > > > .choice() >>>> > > > > >>>> > > > > >>>> .when(exchangeProperty(Exchange.DUPLICATE_MESSAGE).isEqualTo(true)) >>>> > > > > .log("file ${file:onlyname} is a duplicate and >>>> hence >>>> > > > > skipped.") >>>> > > > > .otherwise() >>>> > > > > .process("fileprocessor") >>>> > > > > .log("processed file name from Source Folder" + >>>> > > > > ("${file:onlyname}")) >>>> > > > > .log("Total time to process a file : from >>>> Source Folder >>>> > > > " + >>>> > > > > (System.currentTimeMillis() - starttime) + " ms").end(); >>>> > > > > >>>> > > > > Could you please let me know what needs to be done so that the >>>> same file >>>> > > > is >>>> > > > > not read again and again. Since I am using RedisCache it is not >>>> getting >>>> > > > > processed as the checksum is same but I don't want to read the >>>> file again >>>> > > > > that has been processed >>>> > > > > >>>> > > > > Thanks and Regards >>>> > > > > Deepak Anand >>>> > > > >>>> > > > >>>> > > > >>>> > > > -- >>>> > > > Claus Ibsen >>>> > > > ----------------- >>>> > > > http://davsclaus.com @davsclaus >>>> > > > Camel in Action 2: https://www.manning.com/ibsen2 >>>> > > > >>>> > >>>> > >>>> > >>>> > -- >>>> > Claus Ibsen >>>> > ----------------- >>>> > http://davsclaus.com @davsclaus >>>> > Camel in Action 2: https://www.manning.com/ibsen2 >>>> >>>> >>>> >>>> -- >>>> Claus Ibsen >>>> ----------------- >>>> http://davsclaus.com @davsclaus >>>> Camel in Action 2: https://www.manning.com/ibsen2 >>>> >>>
