oscerd opened a new pull request, #29213: URL: https://github.com/apache/camel/pull/29213
Fixes the camel-pinecone part of [CAMEL-26300](https://issues.apache.org/jira/browse/CAMEL-26300), found by the runtime tests of the body types of the data type transformers (CAMEL-25928). CAMEL-26300 is fixed with one PR per component. Fixes the camel-pinecone items of CAMEL-26300 that are still open on main: the `pinecone:embeddings` data type transformer did not store the text that `pinecone:rag` reads, and it failed with a bare `NullPointerException` without the embedding header. ## 1. pinecone:embeddings stores the text that pinecone:rag reads **Was:** the `pinecone:embeddings` transformer did not read the message body, and UPSERT always sent the vector without metadata, so the text was never stored. `pinecone:rag` reads the text from the `text_segment` metadata of each match, so an embeddings-then-rag round trip gave an empty String for every match. **Now:** the transformer puts the text of a LangChain4j `TextSegment` body, or of a `String` that camel-langchain4j-embeddings converts to one, in the `text_segment` field of a new `CamelPineconeMetadata` header (a `com.google.protobuf.Struct`, like the `CamelPineconeQueryFilter` header). The transformer can only hand the text to the producer through a header, as `pgvector:embeddings` does with its `CamelPgVectorTextContent` and `CamelPgVectorMetadata` headers, and `weaviate:embeddings` with the `CamelWeaviateProperties` header. When the route already set that header, its other fields are kept, as `weaviate:embeddings` keeps the properties a route set. UPSERT stores the header as the metadata of the vector, and UPDATE sets its fields in the metadata of the vector, so re-embedding a changed text with UPDATE also updates the stored text. Another body stores no text, as with the other vector database transformers. Without the header, UPSERT and UPDATE send no metadata, as before. `te xt_segment` is also the key the Pinecone embedding store of LangChain4j uses. ## 2. pinecone:embeddings without the embedding header **Was:** a message without the `CamelLangChain4jEmbeddingsVector` header failed with a bare `NullPointerException`. **Now:** it fails with an `IllegalArgumentException` that names the header, with the same message and exception type as `pgvector:embeddings`. ## Tests - `PineconeEmbeddingsDataTypeTransformerTest`: - a `TextSegment` or `String` body gives the `text_segment` metadata, and an Integer or null body gives none; - the text is added to a metadata header the route already set; - a message without the embedding header fails with the `IllegalArgumentException` that names it. - `PineconeVectorDbProducerTest`, against the fake Pinecone (a JDK HttpServer and a gRPC server), whose index now keeps the upserted vectors in memory: - UPSERT stores the header as the metadata of the vector, and UPDATE sends it as the metadata to set; - a round trip of `pinecone:embeddings`, UPSERT, QUERY and `pinecone:rag` gives the stored text back; - the existing UPSERT and UPDATE tests now also check that no metadata is sent without the header. - Revert check, with the producer and the transformer reverted and the new header constant kept: 5 of the 5 new or changed tests fail. Three fail on an assertion: no metadata header, the caller's metadata without the text, and the vector stored without metadata. The round trip gives an empty String instead of the text, and the missing header gives the old `NullPointerException`. The 12 runs of the existing UPSERT and UPDATE tests with the no-metadata check pass both ways, as that behaviour is unchanged. - camel-pinecone: 31 tests, 0 failures. ## Docs and generated files - The body types of `pinecone:embeddings` now say which bodies give a text and where it goes (`pinecone-embeddings.json` and its catalog copy). - The new `CamelPineconeMetadata` header is in `pinecone.json` and its catalog copy. The endpoint DSL gets the header too. - `pinecone-component.adoc` (and its catalog copy) describes the header and the `pinecone:embeddings` transformer. - An upgrade guide entry for 4.24. ## Not changed - `pinecone:rag` returning an empty List for the `QueryResponseWithUnsignedIndices` of a QUERY, the first pinecone item of CAMEL-26300, was already fixed on main by #29063 (CAMEL-26216). - Only the text is stored, not the metadata of the `TextSegment` (for example a file name a document splitter adds). `pinecone:rag` reads only the text. `qdrant:embeddings` and `pgvector:embeddings` store the segment metadata too; storing it here could be a follow-up. Until then, a route can put its own fields in the `CamelPineconeMetadata` header before the transformer. - The header takes a `Struct`, like `CamelPineconeQueryFilter`; a `Map` is not converted. - QUERY and the other operations ignore the header. - Without the `CamelPineconeIndexName` header, the transformer still sets it to `embeddings`, which takes precedence over the `indexName` option of the endpoint. The doc now says so. Changing it would change which index existing routes write to. Main only, as the other fixes of the CAMEL-24961 follow-ups. _Claude Code on behalf of Andrea Cosentino (oscerd)_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
