Claus Ibsen created CAMEL-27182:
-----------------------------------
Summary: camel-azure-storage-datalake - missing fileName logs a
WARN at every poll
Key: CAMEL-27182
URL: https://issues.apache.org/jira/browse/CAMEL-27182
Project: Camel
Issue Type: Bug
Components: camel-azure
Reporter: Claus Ibsen
h3. Problem
With {{fileName}} set to a file that does not exist (yet), every poll fails and
the scheduled consumer logs a WARN with a
{{com.azure.storage.blob.models.BlobStorageException}} (status 404,
BlobNotFound), every {{delay}} (500 ms by default).
{{DataLakeConsumer.poll}} means to turn a 404 into an empty poll, but it
catches only {{DataLakeStorageException}}:
{code:java}
} catch (DataLakeStorageException e) {
if (NOT_FOUND == e.getStatusCode()) {
result = 0;
{code}
The read without {{fileDir}} goes through
{{DataLakeFileClientWrapper.openInputStream}} ->
{{DataLakeFileClient.openInputStream}}. In the Azure SDK
(azure-storage-file-datalake 12.26.x) that method calls
{{BlockBlobClient.openInputStream}} and does not translate the exception, so a
missing file throws {{BlobStorageException}}, not {{DataLakeStorageException}}.
The same applies in {{createBatchExchangesFromPath}}: a listed file that is
deleted before it is read (for example by another consumer) is meant to be
skipped with a DEBUG log on 404, but the {{BlobStorageException}} is not caught
there either, so the file is counted as a failure (reported to the exception
handler, or the poll fails when it is the only file).
camel-azure-storage-blob returns an empty poll in this case
({{BlobConsumer.poll}} catches {{BlobStorageException}} 404).
h3. How to reproduce
{noformat}
from("azure-storage-datalake:myaccount/myfs?fileName=missing.csv&dataLakeServiceClient=#client")
.log("${body}");
{noformat}
Each poll logs a WARN with {{BlobStorageException}} ... {{Status code 404}},
nothing else happens. Seen while testing the Kamelet
{{azure-storage-datalake-source}} with its new {{fileName}} option.
h3. Where
*
{{components/camel-azure/camel-azure-storage-datalake/src/main/java/org/apache/camel/component/azure/storage/datalake/DataLakeConsumer.java}}
- {{poll}} (catch of {{DataLakeStorageException}} only)
* same file - {{createBatchExchangesFromPath}} (404 check for a listed file)
h3. Suggested fix
Treat a 404 of either exception type as not found, for example a small helper:
{code:java}
private static boolean isNotFound(Exception e) {
return e instanceof DataLakeStorageException d && d.getStatusCode() == 404
|| e instanceof BlobStorageException b && b.getStatusCode() == 404;
}
{code}
and use it in both places (or wrap the read with
{{DataLakeImplUtils.returnOrConvertException}}-style translation in
{{DataLakeFileClientWrapper.openInputStream}}).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)