bamaer opened a new issue, #8411: URL: https://github.com/apache/hop/issues/8411
### What would you like to happen? ### Background #8409 adds a `Vector` value type. It binds through JDBC using the canonical text form `[0.1,0.2,0.3]`, which pgvector accepts as a literal, so reading and writing vectors already works with no dialect-specific code. What does not work is DDL generation. `IDatabase.getFieldDefinition()` has no case for the vector type, so when Hop generates a `CREATE TABLE` — for example from the SQL button in Table Output — a vector field becomes a text column and the user has to hand-edit it to `vector(1536)`. ### Proposal Audit the dialects under `plugins/databases` (44 at time of writing) and implement `getFieldDefinition()` for those with a native vector or vector-like column type, so generated DDL is correct without manual editing. Candidates to assess, in rough order of how commonly they are paired with embeddings: | Dialect | Native type | |---|---| | PostgreSQL | `vector(n)` via the pgvector extension | | Oracle | `VECTOR` | | SQL Server | `VECTOR` | | MySQL | `VECTOR` | | Snowflake | `VECTOR` | | SingleStore | `VECTOR` | | DuckDB | fixed-size float array | | ClickHouse | float array | | CrateDB | `FLOAT_VECTOR` | | BigQuery | `ARRAY<FLOAT64>` with a vector index | The list and the exact type names are what the audit establishes; availability varies by version, so each dialect needs checking against the version Hop targets rather than assumed. Dialects with no native vector type keep the current behaviour and fall back to text, which is a correct and usable result. ### Open question: where does the dimension come from Most native vector types are parameterised by dimension (`vector(1536)`), but the Vector value type does not currently model dimension. `getFieldDefinition()` therefore has nothing to emit the `n` from. Options include carrying it in the field's `length`, which is how other parameterised types do it, or defaulting and letting the user override. This should be settled before the first dialect is implemented, since it affects the value type's metadata. ### Suggested approach PostgreSQL first, since it is the dialect with a concrete consumer today. The rest can follow individually — each dialect is a small, self-contained change, and contributors who use a particular database are better placed to verify it than a single blanket PR would be. ### Note The UUID value type has no dialect mapping in any of the 44 dialects either, so this is not a regression introduced by the vector type — it is an improvement over the current baseline for parameterised plugin value types generally. ### Issue Priority Priority: 2 ### Issue Component Component: Database -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
