bamaer opened a new issue, #8409:
URL: https://github.com/apache/hop/issues/8409
### What would you like to happen?
### Background
Hop has no way to carry an embedding through a pipeline as a typed value.
Today a vector
has to travel as a String holding `[0.1,0.2,0.3]`, or as Binary. Both mean
every consumer
re-parses the value, the field dialogs show a String that must secretly
contain a
fixed-length float array, and a dimension mismatch surfaces as a database
error rather
than a type error.
Embeddings are becoming an ordinary part of data pipelines, and databases
have followed:
PostgreSQL (pgvector), Oracle, SQL Server, MySQL, Snowflake, SingleStore,
ClickHouse,
CrateDB and DuckDB all now have a native vector or vector-like column type.
### Proposal
Add a `Vector` value type (`ValueMetaVector`) as a plugin under
`plugins/valuetypes`,
alongside the existing UUID type.
- Backing storage is `float[]`.
- The canonical text form is a bracketed, comma-separated list:
`[0.1,0.2,0.3]`. That form
is both valid JSON and the literal syntax pgvector accepts, so a vector
survives a round
trip through a text file, a Data Grid, a JSON document or a database
column with no
conversion step in between.
- JDBC binding uses that text form, so reading and writing vectors works
against a pgvector
column without any per-dialect changes.
- Type id `1536` (the dimension of OpenAI's `text-embedding-3-small`),
following the
existing convention for plugin value types — UUID uses `32`, the number of
digits in a UUID.
### Why a value type rather than a convention on String
A typed vector gives transforms something to bind to, keeps dimension and
intent visible in
the field metadata, and makes an incorrect value a type error instead of a
runtime database
error.
It also has to ship by default rather than as a Marketplace plugin, and that
is a property
of value types specifically: if a transform plugin is missing you lose a
transform, but if a
value type plugin is missing, any pipeline using it will not deserialise. A
vector type is
therefore either part of the distribution or it does not exist — there is no
useful middle
option. This is the same reason the UUID type ships by default.
The cost is one extra entry in field-type dropdowns, the same cost UUID,
Internet Address,
Avro Record and Serializable already carry.
### Scope
This issue covers the value type itself, with unit tests.
Emitting native vector column types from `IDatabase.getFieldDefinition()`
per dialect is
deliberately **not** in scope here — generated DDL will produce a text
column until that is
done, which matches how the UUID type behaves today. That work will be
tracked separately in another ticket.
### Issue Priority
Priority: 2
### Issue Component
Component: Metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
