On Thu, Jul 18, 2019 at 11:12 AM Deepak Anand <[email protected]> wrote: > > Hi Claus, > > I tried the approach suggested by you to write a custom Idempotent > Repository. I did that and it worked > Thanks a lot >
Ah that is good to know. Other users that face similar problem in the future can find this conversation and do as you did. > Regards > Deepak Anand > > > > > On Fri, Jul 12, 2019 at 2:09 PM Deepak Anand <[email protected]> > wrote: > > > Hi Devs, > > > > As mentioned above to create a custom IdempotentRepository and generate > > the file checksum I need the file that is being processed inside the > > repository. Was going through the documentation and it seems I can > > implement ExchangeIdempotentRepository and override the below mentioned > > methods in it > > > > boolean add(Exchange exchange, E key) > > boolean confirm(Exchange exchange, E key) > > boolean contains(Exchange exchange, E key) > > boolean remove(Exchange exchange, E key) > > > > The issue which I have is how to pass the exchange to the above repository > > . It seems while using idempotentKey I can only pass the information > > regarding the file like name, size, modification timestamp. > > It would be of great help if anyone can suggest how to pass the Exchange > > exchange, E key both to the IdempotentRepository using idempotentKey > > > > Thanks and Regards > > Deepak Anand > > > > On Fri, Jul 12, 2019 at 1:58 PM Deepak Anand <[email protected]> > > wrote: > > > >> Hi Devs, > >> > >> As mentioned above to create a custom IdempotentRepository and generate > >> the file checksum I need the file that is being processed inside the > >> repository. Was going through the documentation and it seems I can > >> implement ExchangeIdempotentRepository and override the below mentioned > >> methods in it > >> boolean *add > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#add(org.apache.camel.Exchange,%20E)>* > >> (Exchange > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> > >> exchange, E > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> > >> key) > >> Adds the key to the repository. > >> boolean *confirm > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#confirm(org.apache.camel.Exchange,%20E)>* > >> (Exchange > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> > >> exchange, E > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> > >> key) > >> Confirms the key, after the exchange has been processed successfully. > >> boolean *contains > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#contains(org.apache.camel.Exchange,%20E)>* > >> (Exchange > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> > >> exchange, E > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> > >> key) > >> Returns true if this repository contains the specified element. > >> boolean *remove > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html#remove(org.apache.camel.Exchange,%20E)>* > >> (Exchange > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> > >> exchange, E > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> > >> key) > >> Removes the key from the repository. > >> The issue which I have is how to pass the exchange to the above repository > >> . It seems while using idempotentKey I can only pass the information > >> regarding the file like name, size, modification timestamp. > >> It would be of great help if anyone can suggest how to pass the Exchange > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/Exchange.html> > >> exchange, E > >> <https://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/ExchangeIdempotentRepository.html> > >> key > >> both to the IdempotentRepository using idempotentKey > >> > >> Thanks and Regards > >> Deepak Anand > >> > >> > >> On Thu, Jul 11, 2019 at 5:17 PM Deepak Anand <[email protected]> > >> wrote: > >> > >>> Thanks Claus for your response , will try to implement the way you > >>> suggested > >>> > >>> Regards > >>> Deepak Anand > >>> > >>> > >>> On Wed, Jul 10, 2019 at 8:34 PM Claus Ibsen <[email protected]> > >>> wrote: > >>> > >>>> Hi > >>>> > >>>> You can likely try to build a custom IdempotentRepository that extends > >>>> the Redis, and then in all the methods, you get the key from Camel > >>>> which is the absolute file name, which you then use to calculate your > >>>> hash and then call the super method with the hash key instead. > >>>> > >>>> eg something like this: > >>>> > >>>> public boolean contains(exchange, key) > >>>> string hashKey = .... > >>>> return super.contains(exchange, hashKey); > >>>> > >>>> > >>>> On Wed, Jul 10, 2019 at 9:00 PM Claus Ibsen <[email protected]> > >>>> wrote: > >>>> > > >>>> > Hi > >>>> > > >>>> > Yeah its a limitation that idempontentKey can only use the information > >>>> > from the file itself such as its name, size, modification timestamp > >>>> > etc. Not a custom hashcode. > >>>> > > >>>> > On Tue, Jul 9, 2019 at 7:22 PM Deepak Anand < > >>>> [email protected]> wrote: > >>>> > > > >>>> > > Thanks Claus for you response. > >>>> > > The same files can be present in different sub directory , so I need > >>>> > > checksum to identify them. > >>>> > > How in the file endpoint I can generate the checksum and add it to > >>>> the > >>>> > > idempotentRepository . > >>>> > > May be a silly question but I am new to Camel earlier I was > >>>> generating the > >>>> > > checksum of the file in the processor and adding it to the header > >>>> and > >>>> > > passing it to the idempotent consumer > >>>> > > > >>>> > > Thanks and Regards > >>>> > > Deepak Anand > >>>> > > > >>>> > > > >>>> > > On Tue, Jul 9, 2019 at 6:08 PM Claus Ibsen <[email protected]> > >>>> wrote: > >>>> > > > >>>> > > > Set idempotentRepository on the file endpoint instead and remove > >>>> > > > idempotent consumer in the route. > >>>> > > > > >>>> > > > On Tue, Jul 9, 2019 at 7:01 PM Deepak Anand < > >>>> [email protected]> > >>>> > > > wrote: > >>>> > > > > > >>>> > > > > Hi Devs , > >>>> > > > > > >>>> > > > > I have a requirement to read from a directory where there are > >>>> no of files > >>>> > > > > in the subdirectories of it . I am using the camel File > >>>> component for it > >>>> > > > . > >>>> > > > > I don't want to process the same file which has been processed > >>>> earlier > >>>> > > > > using an idempotentConsumer and using RedisCache for it. > >>>> > > > > In the cache I am adding the checksum to identify the duplicate > >>>> file. The > >>>> > > > > problem is that after reading more than 1000 files the same > >>>> files(which > >>>> > > > > were read earlier) are being read again and again which I don't > >>>> want . > >>>> > > > > Below is the snippet of the camel route > >>>> > > > > > >>>> > > > > from("file://" + sourceLoc + "/?recursive=true&noop=true" > >>>> > > > > > >>>> > > > > >>>> +"&idempotent=true&idempotentKey=${file:name}-${file:modified}&readLockRemoveOnCommit=false") > >>>> > > > > .log("the read file is > >>>> > > > > ${file:name}-${file:modified} ") > >>>> > > > > .filter() > >>>> > > > > .method(SourceMinAgeFilter.class, "accept") > >>>> > > > > .process("checksumprocessor") > >>>> > > > > .idempotentConsumer(header("fileCheckSum"), > >>>> > > > > > >>>> > > > > > >>>> RedisIdempotentRepository.redisIdempotentRepository(stringredisTemplate, > >>>> > > > > redisStoragename)) > >>>> > > > > .eager(true) > >>>> > > > > .skipDuplicate(false) > >>>> > > > > .choice() > >>>> > > > > > >>>> > > > > > >>>> .when(exchangeProperty(Exchange.DUPLICATE_MESSAGE).isEqualTo(true)) > >>>> > > > > .log("file ${file:onlyname} is a duplicate and > >>>> hence > >>>> > > > > skipped.") > >>>> > > > > .otherwise() > >>>> > > > > .process("fileprocessor") > >>>> > > > > .log("processed file name from Source Folder" + > >>>> > > > > ("${file:onlyname}")) > >>>> > > > > .log("Total time to process a file : from > >>>> Source Folder > >>>> > > > " + > >>>> > > > > (System.currentTimeMillis() - starttime) + " ms").end(); > >>>> > > > > > >>>> > > > > Could you please let me know what needs to be done so that the > >>>> same file > >>>> > > > is > >>>> > > > > not read again and again. Since I am using RedisCache it is not > >>>> getting > >>>> > > > > processed as the checksum is same but I don't want to read the > >>>> file again > >>>> > > > > that has been processed > >>>> > > > > > >>>> > > > > Thanks and Regards > >>>> > > > > Deepak Anand > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > -- > >>>> > > > Claus Ibsen > >>>> > > > ----------------- > >>>> > > > http://davsclaus.com @davsclaus > >>>> > > > Camel in Action 2: https://www.manning.com/ibsen2 > >>>> > > > > >>>> > > >>>> > > >>>> > > >>>> > -- > >>>> > Claus Ibsen > >>>> > ----------------- > >>>> > http://davsclaus.com @davsclaus > >>>> > Camel in Action 2: https://www.manning.com/ibsen2 > >>>> > >>>> > >>>> > >>>> -- > >>>> Claus Ibsen > >>>> ----------------- > >>>> http://davsclaus.com @davsclaus > >>>> Camel in Action 2: https://www.manning.com/ibsen2 > >>>> > >>> -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
