Claus Ibsen created CAMEL-27181:
-----------------------------------
Summary: camel-azure-storage-datalake - consumer has no
deleteAfterRead option
Key: CAMEL-27181
URL: https://issues.apache.org/jira/browse/CAMEL-27181
Project: Camel
Issue Type: Improvement
Components: camel-azure
Reporter: Claus Ibsen
h3. Problem
The Data Lake consumer re-reads every file of the file system (or of {{path}})
at every poll, and it has no option to delete a file once its exchange
completes. {{DataLakeConsumer.processBatch}} adds an on-completion whose
{{onComplete}} only logs at TRACE. The only way to consume each file once is
the Idempotent Consumer EIP, which runs after the consumer has already opened
the file.
camel-azure-storage-blob has {{deleteAfterRead}} and {{moveAfterRead}}
({{BlobConfiguration}}); {{DataLakeConfiguration}} has neither.
The Kamelet {{azure-storage-datalake-source}} cannot offer a delete-after-read
option because of this, while {{azure-storage-blob-source}} can.
h3. Where
{{components/camel-azure/camel-azure-storage-datalake/src/main/java/org/apache/camel/component/azure/storage/datalake/DataLakeConsumer.java}}
- {{processBatch}} (the {{Synchronization.onComplete}} does nothing).
h3. Suggested fix
Add a consumer option {{deleteAfterRead}} (default false), as in
camel-azure-storage-blob:
* take the file name when the exchange is created (the route may change the
{{CamelAzureStorageDataLakeFileName}} header);
* in {{onComplete}}, delete it with {{DataLakeFileClient.deleteWithResponse}}
({{DataLakeFileClientWrapper.delete}} already exists); a 404 is ignored, other
failures go to the exception handler;
* nothing is deleted on failure, so the file is read again at the next poll.
A {{moveAfterRead}} (rename to another directory) could follow the same pattern.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)