bamaer opened a new issue, #8441: URL: https://github.com/apache/hop/issues/8441
### What would you like to happen? ### What would you like to happen? ### Background Hop can now split documents (#8415) and store vectors in pgvector (#8422), but nothing in between turns text into a vector. That leaves the retrieval chain broken at its middle step: chunk, **embed**, store. The `AiProvider` metadata type from #8330 already models the providers that serve embedding models, and `AiModelCatalog` can already classify a model as `ModelType.EMBEDDING`. What is missing is a way to call one. ### Proposal An `Embed text` transform that reads a text field, calls the embedding model of a selected `AiProvider`, and adds the vector to the row. One row in, one row out. The embedding is emitted either as a String holding a JSON array or as a field of the Vector value type (#8409), so it feeds pgvector upsert directly in both forms. Since `AiProvider` has no embedding support yet, this includes extending it to expose an embedding model alongside the existing chat model, which benefits any later transform that needs one. ### Options | Option | Notes | |---|---| | AI provider | `AiProvider` metadata object | | Model | Embedding model name, filtered to `ModelType.EMBEDDING` | | Input field | Text to embed | | Output field | Embedding, String or Vector | | Batch size | Embedding calls are per-request billed and rate limited | Rows that fail at the provider go to the error hop rather than stopping the pipeline, so one bad row does not lose a long indexing run. ### Scope The transform, embedding support on `AiProvider`, unit tests, integration tests against a local Ollama model, and a user manual page. Depends on #8330. Composes with #8409 and #8422 but does not require either. ### Issue Priority Priority: 2 ### Issue Component Component: Transforms ### Issue Priority Priority: 2 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
