Croway opened a new pull request, #29269: URL: https://github.com/apache/camel/pull/29269
JIRA: [CAMEL-26660](https://issues.apache.org/jira/browse/CAMEL-26660) ## What was wrong When the body is an `Iterable`, `Kinesis2Producer` sends it as `PutRecords` requests of 500 records. `PutRecords` is not atomic: Kinesis can write some records of a request and return an error code for the others. - The first partly failed request threw `RuntimeException("Failed to send records N of M")` at once. The next requests were never sent, and the records already written stayed written. - `CamelAwsKinesisRecordCount` / `CamelAwsKinesisFailedRecordCount` were only set on success, so the failed count was always 0. - Any redelivery of the exchange sent the whole batch again from record 0. With a 1200-record batch whose 2nd request had one failed record, `maximumRedeliveries(1)` wrote 999 records twice. - Requests were split by record count only. The SDK 2.57.7 service model limits a `PutRecords` request to 500 records and 10 MiB, partition keys included. 400 records of 100 KiB went out as a single request of about 40 MiB. ## What changed - The batch is split into requests of at most 500 records **and** 10 MiB, counting data plus partition key (UTF-8 bytes). - A record above 10 MiB fails the exchange with an `IllegalArgumentException` before any request is sent. 10 MiB is the `Data` maximum in the service model. A stream can set a lower limit (`MaxRecordSizeInKiB`, 1024 to 10240), and Kinesis still checks that one. - When records of a request fail, only those records are sent again: up to 3 times, after 100, 200 and 400 ms. The results are matched to the records by position, and a failed record is one whose result has an `ErrorCode`. Then the producer goes on with the next requests. - When records are still not written after the last request: - both headers are set; - the exchange fails with a `RuntimeCamelException` (still a `RuntimeException`) that says how many records were written, how many failed, and the first error code. - No new option. The retry count and delay are constants. - The exchange still fails when records are not written. That default is unchanged, and the SQS-style "complete with the failures in the message" behaviour was not adopted. ## Tests There are 4 new tests in `Kinesis2ProducerTest`, with a Mockito `KinesisClient` that fails chosen records: - a failed record in the 2nd request is sent again alone, the remaining requests are sent (`[500, 500, 1, 200]`), and every record is written exactly once; - the 10 MiB split (`[10, 10, 1]`); - an oversized record makes no call; - a record that keeps failing ends in the exception, and both headers are set. - On `main` (with the producer change reverted), all 4 fail: `expected [10, 10, 1] but was [21]`, `Failed to send records 1 of 500`, an NPE instead of `IllegalArgumentException`, and a plain `RuntimeException` thrown after 1 request. - With the fix, all 11 tests pass. The module suite also passes: 61 unit tests, plus the ITs: 10 run, including `KinesisProducerIT`, which sends a batch, and 3 skipped (the KCL IT and the manual Firehose ITs). - The JIRA reproducer now gives `calls=[500, 500, 1, 200] written=1200 dups=0` both without and with `maximumRedeliveries(1)`, and 400 x 100 KiB goes out as `[102, 102, 102, 94]`. ## Docs - In `aws2-kinesis-component.adoc` (and its catalog copy), the Batch Producer section now documents: - the split and the per-record limit; - the retry of the failed records; - the headers set on failure; - that a redelivery of the exchange, or a retry of a request whose outcome is unknown, can still write records twice, because `PutRecords` has no idempotency token. - There is a new entry in the 4.24 upgrade guide. ## Backport This is a candidate for camel-4.22.x and camel-4.18.x: the same code is on both branches. The upgrade-guide entry stays on `main`. _Claude Code on behalf of Federico Mariani_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
