oscerd opened a new pull request, #28952: URL: https://github.com/apache/camel/pull/28952
Fixes [CAMEL-26261](https://issues.apache.org/jira/browse/CAMEL-26261), found while adding the catalog body types (CAMEL-24961). ## 1. The consumer read every object into memory **Was:** `OSSConsumer` read the content of each listed object with `IOUtils.toByteArray` before it created the exchange, so every object of a poll was held in memory as a `byte[]`, and a large object needed as much heap as its size. There was no option to stream the content. **Now:** a new consumer option `includeBody`, as in camel-aws2-s3. With the default `includeBody=true` nothing changes: the body is the `byte[]` of the object. With `includeBody=false` the body is the `InputStream` of the object, which the consumer does not read. The stream is closed when the exchange is done (completed or failed), and also when the exchange is not processed because the consumer stops in the middle of a batch. If building the exchange fails, the stream is closed at once. The headers are the same in both modes. `OSSUtils.mapOssObject` has a new overload for the stream body; the header mapping that the two existing overloads each had a copy of is now in one private method. ## Tests - New `OSSConsumerIncludeBodyTest` (4 tests) with a mocked `OSSClient` whose streams record what was read and whether they were closed: - by default the body is the `byte[]` of the object, and the stream is closed once read; - with `includeBody=false` the body is the stream itself, nothing was read from it before the route, it is open while the route reads it, and it is closed when the exchange is done; - the stream is closed when the exchange fails; - when the route is stopped while the first exchange of a batch of three is processed, the streams of the two exchanges that are not processed are closed too. - Revert check (the `src/main` changes and the generated configurer taken out): 3 of the 4 new tests fail, as the endpoint does not know `includeBody` (`Failed to resolve endpoint ... includeBody=false`). The one that passes is the default `byte[]` body, which did not change. - Module: 26 tests, 0 failures. ## Docs and generated files - Component doc: the consumer section explains `includeBody`, and that stream caching (on by default) still reads the stream into memory unless it is disabled for the route or spools to disk. A new example (Java and YAML) consumes large objects as streams and writes them to files, with stream caching disabled for the route; the YAML was checked against the YAML DSL schema. - Body types: `consumerOut` gives a stream with `includeBody=false`. - The regenerated `alibaba-oss.json` (new option, body types), configurer and URI factory; the JSON and the doc copied into `camel-catalog`. The new option changes the endpoint DSL. - Upgrade guide: none, the default is unchanged. ## Not changed - There is no `autocloseBody` option as in camel-aws2-s3: the stream is always closed when the exchange is done, as the issue proposes. A route that must keep the stream after the exchange (for example to hand it to another thread) cannot. **Open question:** add `autocloseBody` if someone needs it? - The `getObject` operation of the producer still reads the object into a `byte[]`; the issue is about the consumer. - As in camel-aws2-s3, the consumer gets all the objects of a poll (up to `maxMessagesPerPoll`, 10 by default) before it processes them, so with `includeBody=false` that many streams, and HTTP connections, are open at once. The SDK allows 1024 connections by default. Main only, as the other fixes of the CAMEL-24961 follow-ups. _Claude Code on behalf of Andrea Cosentino (oscerd)_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
