[
https://issues.apache.org/jira/browse/CAMEL-26069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18136516#comment-18136516
]
Claus Ibsen commented on CAMEL-26069:
-------------------------------------
PR: https://github.com/apache/camel/pull/28892
All 5 bugs and the items read in the code are fixed there. The body is
converted to bytes (a Map or JsonNode is marshalled to JSON, no body fails).
The streaming operations fail for image and embedding models, which do not
stream, and they use the model of the request and resolve inference profiles.
New Nova and Jamba stream parsers were added.
Open question: should invokeImageModelStreaming and
invokeEmbeddingsModelStreaming be deprecated for removal, as they only work for
text models?
_Claude Code on behalf of Claus Ibsen (davsclaus)_
> camel-aws-bedrock - the invoke operations send the body as its toString(),
> and the image and embeddings streaming operations give an empty text
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-26069
> URL: https://issues.apache.org/jira/browse/CAMEL-26069
> Project: Camel
> Issue Type: Bug
> Components: camel-aws2
> Affects Versions: 4.18.0, 4.22.0, 4.23.0
> Reporter: Claus Ibsen
> Assignee: Claus Ibsen
> Priority: Major
> Fix For: 4.24.0
>
>
> Found while adding the catalog body types of aws-bedrock (CAMEL-24961).
> h4. 1. The invoke operations send the body as its toString()
> invokeTextModel, invokeImageModel, invokeEmbeddingsModel and the three
> streaming operations build the request of the model with
> {{SdkBytes.fromUtf8String(String.valueOf(exchange.getMessage().getBody()))}}
> ({{BedrockProducer}}, main ed3d719b379c, lines 170, 217, 269 and 567), so no
> type conversion is done, and only a String body works. Checked on
> 4.24.0-SNAPSHOT with {{camel run}} against a fake of the AWS API that logs
> the requests it gets, with the content type headers set:
> * a byte[] of the JSON request is sent as {{[B@4d774bb6}};
> * an InputStream is sent as
> {{org.apache.camel.converter.stream.ByteArrayInputStreamCache@620106cb}};
> * a java.io.File is sent as its path (and a GenericFile from a file consumer
> as {{GenericFile[...]}}, read in the code);
> * a Map is sent as {{\{inputText=Hello\}}};
> * no body is sent as the text {{null}}.
> *Proposed change*: read the body with {{getMandatoryBody(String.class)}}, or
> as bytes with {{SdkBytes.fromByteArray}}.
> h4. 2. invokeImageModelStreaming and invokeEmbeddingsModelStreaming give an
> empty text
> Both call {{invokeTextModelStreaming}} (lines 574-584), whose stream parsers
> read the generated text of each chunk ({{outputText}} for Titan). A chunk of
> an image or of an embedding has no such text. Checked on 4.24.0-SNAPSHOT with
> {{camel run}} against a fake of the AWS API that logs the requests it gets,
> streaming one chunk with {{images}} for {{amazon.titan-image-generator-v1}}
> and one with {{embedding}} for {{amazon.titan-embed-text-v1}}: the body is an
> empty String, with {{CamelAwsBedrockChunkCount}} 1.
> *Proposed change*: parse the image and embedding chunks, or remove these two
> operations if the models do not stream.
> h4. 3. pojoRequest=true without the modelId option fails after the call
> With pojoRequest=true the body is an {{InvokeModelRequest}} that carries its
> model id, but the response is read with the model of the endpoint: {{switch
> (modelId)}} at line 292 fails with {{NullPointerException: Cannot invoke
> "String.hashCode()"}} after the model was called. Checked on 4.24.0-SNAPSHOT
> with {{camel run}} against a fake of the AWS API that logs the requests it
> gets. The streaming operations fail with "Model ID cannot be null"
> ({{StreamParserFactory}}, lines 35-37), although the request carries the
> model (read in the code).
> *Proposed change*: take the model id of the request in POJO mode.
> h4. 4. amazon.titan-embed-text-v2:0 is accepted by invokeTextModel
> The docs say that an embedding model fails fast with invokeTextModel, and
> {{amazon.titan-embed-text-v1}} does, but {{amazon.titan-embed-text-v2:0}} is
> listed with the Titan text models (line 297), so invokeTextModel gives the
> JSON of the embedding as text. Checked on 4.24.0-SNAPSHOT with {{camel run}}
> against a fake of the AWS API that logs the requests it gets.
> h4. 5. No operation fails with a NullPointerException
> The operation option has no default, and with no CamelAwsBedrockOperation
> header, {{switch (determineOperation(exchange))}} at line 80 fails with
> {{NullPointerException: Cannot invoke "...BedrockOperations.ordinal()"
> because the return value of "...determineOperation(...)" is null}}. Checked
> on 4.24.0-SNAPSHOT with {{camel run}} against the AWS emulator of {{camel
> infra run aws}}.
> *Proposed change*: fail with a message that says to set the operation option
> or header.
> h4. Also read in the code, not run
> * The streaming operations do not resolve an inference profile id or an ARN
> to the model ({{StreamParserFactory.getParser}} gets the raw model id, lines
> 610 and 626), so {{us.anthropic.claude-3-haiku-20240307-v1:0}} fails with
> "Unsupported model for streaming". CAMEL-24224 fixed this for invokeTextModel
> only.
> * converseStream never sets the {{CamelAwsBedrockGuardrailTrace}} header:
> {{ConverseStreamHandler.setGuardrailTrace}} is never called, so lines 914-916
> never run. It does not set {{CamelAwsBedrockConverseOutputMessage}} either,
> which the docs list.
> * Header types in the catalog: {{CamelAwsBedrockGuardrailOutput}} is a
> {{List<GuardrailOutputContent>}}, not a GuardrailAssessment;
> {{CamelAwsBedrockGuardrailTrace}} is a {{GuardrailTraceAssessment}}, not the
> enum GuardrailTrace; converseStream reads {{CamelAwsBedrockGuardrailConfig}}
> as a {{GuardrailStreamConfiguration}}, so the GuardrailConfiguration the docs
> name is ignored there.
> * From the documented response formats, not run: AI21 Jamba gives a null body
> with invokeTextModel (line 509 reads {{completions}}, Jamba answers with
> {{choices}}), and the streaming of Jamba and Nova gives an empty text, as
> their chunks do not match the Titan and Claude parsers
> ({{StreamParserFactory}}, lines 65-72).
> _Claude Code on behalf of davsclaus_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
