oscerd opened a new pull request, #28969: URL: https://github.com/apache/camel/pull/28969
Fixes [CAMEL-26250](https://issues.apache.org/jira/browse/CAMEL-26250), found while adding the catalog body types (CAMEL-24961). ## 1. generateMultimodal dropped a stream, File or byte[] image **Was:** `generateMultimodal` used the body only when it was a `byte[]` with the `CamelGoogleVertexAIMediaMimeType` header, or a `String` without the `CamelGoogleVertexAIPrompt` header. A stream, a `File`, a `GenericFile`, and a `byte[]` without the MIME type header were dropped without an error, so the model got the prompt alone. The `CamelGoogleVertexAIMediaData` header without the MIME type header was dropped the same way. **Now:** when neither the `CamelGoogleVertexAIMediaData` nor the `CamelGoogleVertexAIMediaGcsUri` header is set, a `String` body without the prompt header is still the prompt, and any other body is the media, read with `getMandatoryBody(byte[].class)`: bytes (binary data survives), an `InputStream`, a `File` or a `GenericFile`. The MIME type is the one of the `CamelGoogleVertexAIMediaMimeType` header or, for a `File` or the file of a file consumer, the one of the file name extension, from `org.apache.camel.util.MimeTypeHelper.probeMimeType` (the fixed table that camel-http and camel-attachments use, with no lookup in the OS). Media without a MIME type fails with an `IllegalArgumentException` that names the header, for the body and for the `CamelGoogleVertexAIMediaData` header. A `String` body together with the prompt header is media as well, so it now needs the MIME type header; the message says that a text body is the prompt only without the prompt header. ## 2. rawPredict and streamRawPredict rejected byte[], stream and File bodies **Was:** `buildRawPredictRequestBody` took a `String` or a `Map` only and failed with "Request body must be a JSON String, Map, or plain text prompt" for anything else, so JSON from a file or an HTTP consumer failed. **Now:** a `Map` is written as JSON, as before. Text, bytes, a stream or a file (a payload for `BodyFacts.kindOf`) is read as text with the type converter, and then goes through the checks a `String` went through: JSON is sent as it is (with `anthropic_version` added for publisher `anthropic`), and other text is wrapped as a prompt. A POJO, a `List` or no body still fails, with a message that lists what is accepted: their `toString()` is not a request. ## 3. Six declared headers were never used Decided per header: - `CamelGoogleVertexAISystemInstruction`: implemented. `buildConfig` sets it as the `systemInstruction` of the `GenerateContentConfig`, so it applies to `generateText`, `generateChat`, `generateChatStreaming`, `generateCode` and `generateMultimodal`. - `CamelGoogleVertexAISafetySettings`: implemented in `buildConfig` (`safetySettings`), for the same operations. Its declared type was `List<com.google.cloud.aiplatform.v1.SafetySetting>`, which the GenAI client of these operations cannot take, so the header takes `List<com.google.genai.types.SafetySetting>` and its javaType is corrected. Another element type fails with an `IllegalArgumentException`. - `CamelGoogleVertexAISafetyRatings`: implemented. The safety ratings of the first candidate, read next to the finish reason, or of the prompt when it was blocked. The javaType is corrected to `List<com.google.genai.types.SafetyRating>`. - `CamelGoogleVertexAIContentBlocked`: implemented. `true` when the response has a block reason in its prompt feedback: such a response has no candidate and no text, and the body was null with nothing to say why. An answer that a safety filter stops never reaches the header code: the GenAI SDK fails `response.text()` with `IllegalArgumentException: The response finished unexpectedly with reason SAFETY`, so that exchange fails, as before (a test pins it). - `CamelGoogleVertexAIAnthropicVersion`: implemented. For `rawPredict` and `streamRawPredict` with publisher `anthropic`, the header takes the place of the `anthropicVersion` option, for a JSON request without `anthropic_version`, a `Map` or a plain text prompt. - `CamelGoogleVertexAIChatMessages`: deprecated, not implemented: `@Deprecated(since = "4.24")` and a deprecation note that says it is not used, which the header table shows. A conversation history is a feature (roles, turn order, what to give back), and the declared type `List<com.google.cloud.aiplatform.v1.Content>` is not the `Content` of the GenAI client that `generateChat` uses. The constant stays. ## Tests - New `GoogleVertexAIProducerBodyTest` (21 tests). It runs the producer against a JDK `HttpServer` fake of the Gemini API, behind a real GenAI `Client` (API key mode, with the base URL of the fake), and a `PredictionServiceClient` built on a fake `PredictionServiceStub` (the module has no Mockito). It covers `generateMultimodal` with a byte[] (bytes 0x89, 0xff, 0x00 and 0xfe, compared after the base64 of the request), a stream, a `File`, the `GenericFile` of a file consumer route, a file with an unknown extension, the media data header without a MIME type, and a text body with and without the prompt header; the system instruction and the safety settings in the request; the safety ratings and the content blocked header for an answer, a blocked prompt and a safety-stopped answer; `rawPredict` with JSON in bytes, a stream and a file, a `Map`, a POJO and no body, and the Anthropic version from the option and from the header; `streamRawPredict` with JSON in bytes. - `GoogleVertexAIProducerOptionsTest`: 2 more tests, the system instruction and the safety settings in `buildConfig`, and safety settings of another type. - With the `src/main` changes reverted, 18 of the 23 new tests fail, each on the bug: the media missing from the request, the old "Request body must be..." exception, null headers, the Anthropic version of the option, no exception for a missing MIME type. The 5 that pass guard behaviour that was already right: a byte[] with the MIME type header, a text body as the prompt, a `Map` request, a POJO or no body failing, and the SDK failing a safety-stopped answer. - Module: 36 tests, 0 failures. ## Docs and generated files - Component doc: the body of `generateMultimodal` (which body is the prompt, which is the media, where the MIME type comes from, with a file consumer example in Java and YAML), the MIME type the media data header needs, a new section on the system instruction, the safety settings and a blocked prompt, and the bodies `rawPredict` takes. - `@BodyTypes`: `generateMultimodal` no longer has its own carriers (the default text, bytes, stream and file apply), `rawPredict` and `streamRawPredict` take text, bytes, a stream, a file or a map, and `producerOut` adds null for a blocked prompt. The descriptions say what the code does now. - The regenerated `google-vertexai.json` (headers and body types) is copied into the catalog with the doc. - Upgrade guide entries for 4.24: the body of `generateMultimodal` and the MIME type failure; the headers now applied, the corrected types and the deprecated chat messages header. ## Not changed - The MIME type of the `CamelGoogleVertexAIMediaGcsUri` header still defaults to `application/octet-stream` without the MIME type header. Guessing it from the object name, as for a file body, would be one line; the issue is about the body, so I left it. **Open question:** guess it there too? - The `Content-Type` header of an HTTP consumer, and the `CamelFileName` header when the body is no longer a file, are not used to find the MIME type: either can be left over from an earlier step. The doc says to set the MIME type header, for example from `Content-Type`. - A blocked prompt still completes, with no text (now with `CamelGoogleVertexAIContentBlocked` set to `true`), while the SDK fails a safety-stopped answer. **Open question:** should a blocked prompt fail the exchange too, so both behave the same? - `CamelGoogleVertexAIProjectId` and `CamelGoogleVertexAILocation` are declared as producer headers and are not read either, and `CamelGoogleVertexAIModelId` and `CamelGoogleVertexAIPublisher` are declared as producer input but only set on the response. They are not in the issue, so they are unchanged. - For `rawPredict`, a POJO, a Jackson `JsonNode` or a `List` still fails. Marshalling a POJO to JSON with Jackson would be a new feature. Main only, as the other fixes of the CAMEL-24961 follow-ups. _Claude Code on behalf of Andrea Cosentino (oscerd)_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
