Claus Ibsen created CAMEL-25551:
-----------------------------------
Summary: camel-azure-storage-blob, camel-huaweicloud-obs,
camel-alibaba-oss, camel-aws2-s3-vectors - do not consume an object again while
its exchange is in flight
Key: CAMEL-25551
URL: https://issues.apache.org/jira/browse/CAMEL-25551
Project: Camel
Issue Type: Bug
Components: camel-aws2, camel-azure
Reporter: Claus Ibsen
Assignee: Claus Ibsen
Fix For: 4.23.0
These polling consumers list objects and route them asynchronously, and delete
(or move) the object in the on completion of the exchange. They do not track
which objects are in flight, so when routing takes longer than the poll delay
(slow route, threads(), etc.) the next poll lists the same object again and
routes it a second time. The second exchange then fails to delete/move it (404).
* camel-azure-storage-blob BlobConsumer with deleteAfterRead=true or
moveAfterRead=true. With fileDir it is worse: the blob is deleted and
downloaded again to the same local file while the first exchange is still
reading it.
* camel-huaweicloud-obs OBSConsumer with deleteObject / moveObject.
* camel-alibaba-oss OSSConsumer with deleteAfterRead=true.
* camel-aws2-s3-vectors consumer with deleteAfterRead=true (the
processedVectorIds dedup only covers the branch without delete).
The fix is the same as camel-minio (CAMEL-25512) and the in-progress repository
of camel-aws2-s3: remember the objects in flight, skip them on the next poll,
and release them when the exchange completes or fails, when the exchange could
not be created, and for exchanges that are not routed because the consumer is
stopping (closing their streams).
camel-azure-storage-blob also fails the whole poll when one listed blob is
deleted before it is read (404), and drops the exchanges already created
without closing their streams.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
