[
https://issues.apache.org/jira/browse/CAMEL-25789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Claus Ibsen updated CAMEL-25789:
--------------------------------
Fix Version/s: 4.24.0
> camel-aws2-s3 - streaming upload with namingStrategy=random (or timestamp)
> names the first file of every producer keyName alone, so a restart replaces
> the first file of the previous run
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-25789
> URL: https://issues.apache.org/jira/browse/CAMEL-25789
> Project: Camel
> Issue Type: Bug
> Components: camel-aws2
> Reporter: shashank
> Assignee: shashank
> Priority: Major
> Fix For: 4.24.0
>
>
> {{AWS2S3StreamUploadProducer.fileNameToUpload}} (main 7aae27315946, lines
> 500-550) adds the UUID (namingStrategy {{random}}) or the timestamp
> ({{timestamp}}) to {{keyName}} only when {{part > 0}}:
> {code:java}
> case random:
> if (part > 0) {
> ... dynamicKeyName = fileName + "-" + id.toString() + ext;
> } else {
> ... dynamicKeyName = fileName + ext;
> }
> {code}
> {{part}} is the producer's counter of completed uploads
> ({{UploadState.part}}, line 256), which starts at 0 in every new producer. So
> the first file of every producer is named {{keyName}} alone ({{camel.txt}}),
> and only the later ones {{camel-<uuid>.txt}}. When the route or the
> application is started again, the new producer writes its first file to
> {{camel.txt}} again, and S3 replaces the first file of the previous run: its
> messages ({{batchMessageNumber}}, 10 by default, or up to {{batchSize}}) are
> lost at every restart, unless the bucket is versioned. The same happens
> between two producers with the same {{keyName}} (two instances of an
> application, or two routes). Every exchange completed normally.
> The documentation (aws2-s3-streaming.adoc) says that "with the random
> strategy a UUID will be added after keyName", and that restarting from the
> last part "won't be needed when using the random naming strategy" (the
> {{restartingPolicy}} {{lastPart}} only works with {{progressive}}, whose
> {{override}} default replaces the earlier files by design). So a user who
> picks {{random}} to keep the files of every run loses one file per restart.
> {{AWSS3NamingStrategyTest}} pins the name of the first file ({{testFile.txt}}
> for part 0), but nothing explains why the first file should have no random
> part.
> *Fix*: with {{random}} and {{timestamp}} every upload gets the UUID or the
> timestamp, the first one too. {{progressive}} is unchanged. Documentation of
> the three strategies in aws2-s3-streaming.adoc, and a note in the 4.24
> upgrade guide (the first file has a new name).
> *Tests*: new {{AWS2S3StreamUploadRandomNamingRestartTest}} (mocked
> {{S3Client}} that keeps the completed objects by key, like a bucket).
> {{aProducerStartedAgainDoesNotReplaceTheObjectsOfTheFirstOne}} (15 messages,
> stop, a new producer, 15 messages, stop) fails on main: {{every upload must
> be its own object, but the bucket has [camel-7c06...txt, camel-91f0...txt,
> camel.txt] ==> expected: <4> but was: <3>}}.
> {{everyObjectHasTheKeyNameAndARandomPart}} fails on main: {{camel.txt ==>
> expected: <true> but was: <false>}}. {{AWSS3NamingStrategyTest}} now expects
> the random part and the timestamp for part 0. camel-aws2-s3 unit tests: 106
> run, 0 failures.
> *How it was found*: variant analysis of CAMEL-25759 (pattern: a name that has
> to be unique across starts is built from per-start state, so a restart or a
> second instance reuses it), found while working on the timestamp grouping of
> the same producer. Then the JUnit test.
> *Affected*: main, camel-4.23.x, camel-4.22.x, camel-4.18.x and camel-4.14.x
> (the same {{part > 0}} condition; read, not run). The behaviour dates from
> CAMEL-16185. Not run against S3.
> _Filed with Claude Code on behalf of allthingssecurity._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
