Claus Ibsen created CAMEL-27181:
-----------------------------------

             Summary: camel-azure-storage-datalake - consumer has no 
deleteAfterRead option
                 Key: CAMEL-27181
                 URL: https://issues.apache.org/jira/browse/CAMEL-27181
             Project: Camel
          Issue Type: Improvement
          Components: camel-azure
            Reporter: Claus Ibsen


h3. Problem
The Data Lake consumer re-reads every file of the file system (or of {{path}}) 
at every poll, and it has no option to delete a file once its exchange 
completes. {{DataLakeConsumer.processBatch}} adds an on-completion whose 
{{onComplete}} only logs at TRACE. The only way to consume each file once is 
the Idempotent Consumer EIP, which runs after the consumer has already opened 
the file.

camel-azure-storage-blob has {{deleteAfterRead}} and {{moveAfterRead}} 
({{BlobConfiguration}}); {{DataLakeConfiguration}} has neither.

The Kamelet {{azure-storage-datalake-source}} cannot offer a delete-after-read 
option because of this, while {{azure-storage-blob-source}} can.

h3. Where
{{components/camel-azure/camel-azure-storage-datalake/src/main/java/org/apache/camel/component/azure/storage/datalake/DataLakeConsumer.java}}
 - {{processBatch}} (the {{Synchronization.onComplete}} does nothing).

h3. Suggested fix
Add a consumer option {{deleteAfterRead}} (default false), as in 
camel-azure-storage-blob:
* take the file name when the exchange is created (the route may change the 
{{CamelAzureStorageDataLakeFileName}} header);
* in {{onComplete}}, delete it with {{DataLakeFileClient.deleteWithResponse}} 
({{DataLakeFileClientWrapper.delete}} already exists); a 404 is ignored, other 
failures go to the exception handler;
* nothing is deleted on failure, so the file is read again at the next poll.
A {{moveAfterRead}} (rename to another directory) could follow the same pattern.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to