[
https://issues.apache.org/jira/browse/CAMEL-27181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-27181 started by Claus Ibsen.
-------------------------------------------
> camel-azure-storage-datalake - consumer has no deleteAfterRead option
> ---------------------------------------------------------------------
>
> Key: CAMEL-27181
> URL: https://issues.apache.org/jira/browse/CAMEL-27181
> Project: Camel
> Issue Type: Improvement
> Components: camel-azure
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Minor
> Fix For: 4.24.0
>
>
> h3. Problem
> The Data Lake consumer re-reads every file of the file system (or of
> {{path}}) at every poll, and it has no option to delete a file once its
> exchange completes. {{DataLakeConsumer.processBatch}} adds an on-completion
> whose {{onComplete}} only logs at TRACE. The only way to consume each file
> once is the Idempotent Consumer EIP, which runs after the consumer has
> already opened the file.
> camel-azure-storage-blob has {{deleteAfterRead}} and {{moveAfterRead}}
> ({{BlobConfiguration}}); {{DataLakeConfiguration}} has neither.
> The Kamelet {{azure-storage-datalake-source}} cannot offer a
> delete-after-read option because of this, while {{azure-storage-blob-source}}
> can.
> h3. Where
> {{components/camel-azure/camel-azure-storage-datalake/src/main/java/org/apache/camel/component/azure/storage/datalake/DataLakeConsumer.java}}
> - {{processBatch}} (the {{Synchronization.onComplete}} does nothing).
> h3. Suggested fix
> Add a consumer option {{deleteAfterRead}} (default false), as in
> camel-azure-storage-blob:
> * take the file name when the exchange is created (the route may change the
> {{CamelAzureStorageDataLakeFileName}} header);
> * in {{onComplete}}, delete it with {{DataLakeFileClient.deleteWithResponse}}
> ({{DataLakeFileClientWrapper.delete}} already exists); a 404 is ignored,
> other failures go to the exception handler;
> * nothing is deleted on failure, so the file is read again at the next poll.
> A {{moveAfterRead}} (rename to another directory) could follow the same
> pattern.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)