bamaer opened a new issue, #8511: URL: https://github.com/apache/hop/issues/8511
### What would you like to happen? ### What would you like to happen? ### Background Hop can chunk documents (#8415), embed them (#8441) and store and search vectors (#8422). All of that is retrieval: finding text to hand to a model. What is still missing is the opposite direction, and it is the one closest to what Hop is for. A contract is a renewal date and a value. A support ticket is a severity and a product area. An invoice is a supplier and a total. Today a pipeline can carry that text but cannot turn it into columns, so it cannot be filtered, joined or aggregated. Tika and the Text chunker get the text out of a document. Nothing gets the fields out of the text. ### Proposal A `Structured extract` transform that reads named, typed fields out of a text field using the model of an `AiProvider`. One row in, one row out, with the extracted fields added. The field grid is the whole configuration. Each row becomes three things at once: a property in the JSON schema the model is constrained by, a column on the stream, and the type the answer is read back into. | Column | Meaning | |---|---| | Name | The output field, and the name the model is asked for | | Type | String, Integer, Number, BigNumber, Boolean, Date or Timestamp | | Description | What the field means, in the user's words. Sent to the model | | Required | Whether the model must answer. Optional fields may answer null | | Allowed values | Constrains the answer to a list, which is how to classify | ### How the model is constrained Where the provider supports it, the grid is sent as a JSON schema and the model is constrained to that shape rather than asked for it. Hop checks this per model through `supportedCapabilities()`, so nothing needs configuring. Where it is not supported, the same fields are described in the prompt and the answer is checked on the way back. ### When extraction fails A value that will not convert, `next March` for a Date, is an error naming the field and showing what came back. It is never written as a silent null, because downstream a missing value and a misread value look identical and only one of them is the model's honest answer. With an error hop attached the row is diverted and the run continues. ### Scope The transform, a shared `AiChatModelFactory` so a later rerank or agent transform can reach a chat model too, unit tests, a sample pipeline, an integration test in the shared `ai` project, and a user manual page. Depends on #8330 for the AI provider. Composes with #8415 and #8422 but requires neither: it does no retrieval and needs no vector store. ### Issue Priority Priority: 2 ### Issue Component Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
