This is an automated email from the ASF dual-hosted git repository.
mattcasters pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 4eab975be5 Fixes #8409 : Add a Vector value type for embeddings (#8414)
4eab975be5 is described below
commit 4eab975be549b7fbc17736280990968dbdba83bf
Author: Bart Maertens <[email protected]>
AuthorDate: Thu Sep 17 09:31:02 2026 +0200
Fixes #8409 : Add a Vector value type for embeddings (#8414)
* Issue #8409 : Add a Vector value type for embeddings
* Issue #8409 : Address review feedback on the Vector value type
* Issue #8409 : Document the vector dimension contract on the field length
---
assemblies/plugins/pom.xml | 6 +
.../modules/ROOT/pages/value-types.adoc | 6 +-
.../pages/snippets/hop-concepts/data-types.adoc | 36 ++
.../transforms/0114-vector-conversions.hpl | 181 +++++++++
.../transforms/0114-vector-value-type.hpl | 121 ++++++
.../transforms/0116-vector-lazy-conversion.hpl | 153 ++++++++
.../datasets/golden-vector-conversions.csv | 4 +
.../datasets/golden-vector-lazy-conversion.csv | 3 +
.../datasets/golden-vector-value-type.csv | 2 +
integration-tests/transforms/files/vector-lazy.csv | 3 +
.../transforms/main-0114-vector-value-type.hwf | 85 +++++
.../dataset/golden-vector-conversions.json | 15 +
.../dataset/golden-vector-lazy-conversion.json | 15 +
.../metadata/dataset/golden-vector-value-type.json | 23 ++
.../unit-test/0114-vector-conversions UNIT.json | 28 ++
.../unit-test/0114-vector-value-type UNIT.json | 33 ++
.../0116-vector-lazy-conversion UNIT.json | 28 ++
plugins/valuetypes/pom.xml | 1 +
plugins/valuetypes/{ => vector}/pom.xml | 14 +-
.../{pom.xml => vector/src/assembly/assembly.xml} | 35 +-
.../org/apache/hop/vector/ValueMetaVector.java | 422 +++++++++++++++++++++
.../vector/src/main/resources/vector.svg | 10 +
.../src/main/resources/version.xml} | 16 +-
.../org/apache/hop/vector/ValueMetaVectorTest.java | 251 ++++++++++++
24 files changed, 1449 insertions(+), 42 deletions(-)
diff --git a/assemblies/plugins/pom.xml b/assemblies/plugins/pom.xml
index 1c53b8f415..a6d778d8c5 100644
--- a/assemblies/plugins/pom.xml
+++ b/assemblies/plugins/pom.xml
@@ -1638,6 +1638,12 @@
<version>${project.version}</version>
<type>zip</type>
</dependency>
+ <dependency>
+ <groupId>org.apache.hop</groupId>
+ <artifactId>hop-valuetypes-vector</artifactId>
+ <version>${project.version}</version>
+ <type>zip</type>
+ </dependency>
</dependencies>
<!--
diff --git a/docs/hop-dev-manual/modules/ROOT/pages/value-types.adoc
b/docs/hop-dev-manual/modules/ROOT/pages/value-types.adoc
index 8d662644e3..1a0443640f 100644
--- a/docs/hop-dev-manual/modules/ROOT/pages/value-types.adoc
+++ b/docs/hop-dev-manual/modules/ROOT/pages/value-types.adoc
@@ -21,8 +21,8 @@ under the License.
A value type is what Hop knows about one kind of data: how to compare it,
convert it, serialise it across a hop, and write it to a file.
The built-in set is small — string, integer, number, big number, date,
timestamp, boolean, binary — and it is extensible.
-`UUID` and `JSON` are value types.
-A geometry type, a vector type or a money type would be too.
+`UUID`, `JSON` and `Vector` are value types.
+A geometry type or a money type would be too.
== When you need one
@@ -72,7 +72,7 @@ At minimum you will implement conversion to and from the
types Hop already knows
[WARNING]
====
Type ids are plain integers and there is no allocation registry.
-The built-in types use low numbers and the shipped plugins use `20` for Avro
and `32` for UUID.
+The built-in types use low numbers and the shipped plugins use `20` for Avro,
`32` for UUID, `303` for the Neo4j Graph type and `1536` for Vector.
Two plugins that pick the same id will collide, and the failure will not be
obvious.
Until Hop has a convention, pick something distinctive and high, and say in
your documentation which id you took.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-concepts/data-types.adoc
b/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-concepts/data-types.adoc
index d1261b8e5b..4a16b0bd5e 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-concepts/data-types.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-concepts/data-types.adoc
@@ -119,3 +119,39 @@ Arrays are compared element by element; if all compared
elements are equal, the
The JSON type supports all standard JSON values when reading or writing to
MongoDB.
Values that are part of MongoDB’s extended JSON (for example, Date objects)
are not supported.
To fully support MongoDB-specific BSON values, use the String type instead.
+
+=== Vector
+
+The Vector type holds a dense floating point vector, as produced by an
embedding model and consumed by a vector store.
+Values are stored as a `float[]`.
+
+The canonical text form is a bracketed, comma separated list of numbers:
+
+```
+[0.1,0.2,0.3]
+```
+
+That form is both valid JSON and the literal syntax pgvector accepts, so a
vector survives a round trip through a text file, a Data Grid, a JSON document
or a database column without a conversion step in between.
+
+*Conversions*
+
+Vector converts to and from String using the canonical text form above.
+Conversion from other types is not supported: convert to String first if an
intermediate type is involved.
+
+*Databases*
+
+Hop binds a vector to a prepared statement using its text form, which a
PostgreSQL `vector` column (provided by the pgvector extension) accepts
directly.
+No dialect currently emits a native vector column type when Hop generates DDL,
so a generated `CREATE TABLE` produces a text column that has to be adjusted by
hand.
+
+*Dimensions*
+
+The field's `length` carries the number of dimensions, the way it carries the
size of a String.
+A length of zero or less means the dimension is unspecified.
+
+Hop never pads or truncates a vector to match a declared length.
+An embedding resized to fit would still return results from a vector store,
and those results would be wrong, so a mismatch is left for the store to reject.
+
+*Gotchas*
+
+The type does not itself enforce the number of dimensions.
+A vector store will reject a value whose length does not match the column it
is written to, so the dimensions of the embedding model and of the target
column have to be kept aligned by the pipeline author.
diff --git a/integration-tests/transforms/0114-vector-conversions.hpl
b/integration-tests/transforms/0114-vector-conversions.hpl
new file mode 100644
index 0000000000..735240a812
--- /dev/null
+++ b/integration-tests/transforms/0114-vector-conversions.hpl
@@ -0,0 +1,181 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+ <info>
+ <name>0114-vector-conversions</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>A vector survives a String to Vector to String round trip
through the canonical
+ text form. Integer components are normalised to float form, so [1,2,3]
comes back as
+ [1.0,2.0,3.0].</description>
+ <extended_description/>
+ <pipeline_version/>
+ <pipeline_type>Normal</pipeline_type>
+ <pipeline_status>0</pipeline_status>
+ <parameters>
+ </parameters>
+ <capture_transform_performance>N</capture_transform_performance>
+
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+ <created_user>-</created_user>
+ <created_date>2026/09/16 10:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/09/16 10:00:00.000</modified_date>
+ <key_for_session_key/>
+ <is_key_private>N</is_key_private>
+ </info>
+ <notepads>
+ </notepads>
+ <order>
+ <hop> <from>Text</from> <to>To vector</to> <enabled>Y</enabled> </hop>
+ <hop> <from>To vector</from> <to>Back to text</to> <enabled>Y</enabled>
</hop>
+ <hop> <from>Back to text</from> <to>OUTPUT</to> <enabled>Y</enabled> </hop>
+ </order>
+ <transform>
+ <name>Text</name>
+ <type>DataGrid</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <data>
+ <line>
+ <item>[0.1,0.2,0.3]</item>
+ </line>
+ <line>
+ <item>[1,2,3]</item>
+ </line>
+ <line>
+ <item>[-0.5,0.25]</item>
+ </line>
+ </data>
+ <fields>
+ <field>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ <name>embedding</name>
+ <type>String</type>
+ </field>
+ </fields>
+ <attributes/>
+ <GUI>
+ <xloc>96</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>To vector</name>
+ <type>SelectValues</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <fields>
+ <select_unspecified>N</select_unspecified>
+ <meta>
+ <name>embedding</name>
+ <rename>embedding</rename>
+ <type>Vector</type>
+ <length>-2</length>
+ <precision>-2</precision>
+ <conversion_mask/>
+ <date_format_lenient>false</date_format_lenient>
+ <date_format_locale/>
+ <date_format_timezone/>
+ <lenient_string_to_number>false</lenient_string_to_number>
+ <encoding/>
+ <decimal_symbol/>
+ <grouping_symbol/>
+ <currency_symbol/>
+ <storage_type/>
+ </meta>
+ </fields>
+ <attributes/>
+ <GUI>
+ <xloc>288</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>Back to text</name>
+ <type>SelectValues</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <fields>
+ <select_unspecified>N</select_unspecified>
+ <meta>
+ <name>embedding</name>
+ <rename>embedding</rename>
+ <type>String</type>
+ <length>-2</length>
+ <precision>-2</precision>
+ <conversion_mask/>
+ <date_format_lenient>false</date_format_lenient>
+ <date_format_locale/>
+ <date_format_timezone/>
+ <lenient_string_to_number>false</lenient_string_to_number>
+ <encoding/>
+ <decimal_symbol/>
+ <grouping_symbol/>
+ <currency_symbol/>
+ <storage_type/>
+ </meta>
+ </fields>
+ <attributes/>
+ <GUI>
+ <xloc>496</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>OUTPUT</name>
+ <type>Dummy</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ <GUI>
+ <xloc>704</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ </transform>
+ <transform_error_handling>
+ </transform_error_handling>
+ <attributes/>
+</pipeline>
diff --git a/integration-tests/transforms/0114-vector-value-type.hpl
b/integration-tests/transforms/0114-vector-value-type.hpl
new file mode 100644
index 0000000000..c603aa43da
--- /dev/null
+++ b/integration-tests/transforms/0114-vector-value-type.hpl
@@ -0,0 +1,121 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+ <info>
+ <name>0114-vector-value-type</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>The Vector value type is registered and is applied to a
field, so the stream
+ metadata reports the field as type Vector.</description>
+<extended_description/>
+ <pipeline_version/>
+ <pipeline_type>Normal</pipeline_type>
+ <pipeline_status>0</pipeline_status>
+ <parameters>
+ </parameters>
+ <capture_transform_performance>N</capture_transform_performance>
+
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+ <created_user>-</created_user>
+ <created_date>2026/09/16 10:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/09/16 10:00:00.000</modified_date>
+ <key_for_session_key/>
+ <is_key_private>N</is_key_private>
+ </info>
+ <notepads>
+ </notepads>
+ <order>
+ <hop> <from>Vectors</from> <to>Metadata structure of stream</to>
<enabled>Y</enabled> </hop>
+ <hop> <from>Metadata structure of stream</from> <to>OUTPUT</to>
<enabled>Y</enabled> </hop>
+ </order>
+ <transform>
+ <name>Vectors</name>
+ <type>DataGrid</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <data>
+ <line>
+ <item>[0.1,0.2,0.3]</item>
+ </line>
+ <line>
+ <item>[1,2,3]</item>
+ </line>
+ </data>
+ <fields>
+ <field>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ <name>embedding</name>
+ <type>Vector</type>
+ </field>
+ </fields>
+ <attributes/>
+ <GUI>
+ <xloc>96</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>Metadata structure of stream</name>
+ <type>TransformMetaStructure</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <outputRowcount>N</outputRowcount>
+ <rowcountField/>
+ <attributes/>
+ <GUI>
+ <xloc>320</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ </transform>
+ <transform>
+ <name>OUTPUT</name>
+ <type>Dummy</type>
+ <description/>
+ <distribute>Y</distribute>
+ <custom_distribution/>
+ <copies>1</copies>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ <GUI>
+ <xloc>544</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ </transform>
+ <transform_error_handling>
+ </transform_error_handling>
+ <attributes/>
+</pipeline>
diff --git a/integration-tests/transforms/0116-vector-lazy-conversion.hpl
b/integration-tests/transforms/0116-vector-lazy-conversion.hpl
new file mode 100644
index 0000000000..fa65365630
--- /dev/null
+++ b/integration-tests/transforms/0116-vector-lazy-conversion.hpl
@@ -0,0 +1,153 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+ <info>
+ <pipeline_version/>
+ <capture_transform_performance>N</capture_transform_performance>
+
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+ <pipeline_status>0</pipeline_status>
+ <parameters/>
+ <name>0116-vector-lazy-conversion</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>A lazily converted String field converts to Vector and back.
Select Values builds
+ a fresh target whose storage metadata is empty, so the conversion has to
run against the
+ source field.</description>
+ <extended_description/>
+ <created_user>-</created_user>
+ <modified_user>-</modified_user>
+ <created_date>2026/09/16 21:00:00.000</created_date>
+ <modified_date>2026/09/16 21:00:00.000</modified_date>
+ <created_hop_version/>
+ <modified_hop_version>2.20.0-SNAPSHOT</modified_hop_version>
+ </info>
+ <transform>
+ <type>CSVInput</type>
+ <name>Lazy CSV</name>
+ <filename>${PROJECT_HOME}/files/vector-lazy.csv</filename>
+ <filename_field/>
+ <include_filename>N</include_filename>
+ <rownum_field/>
+ <header>Y</header>
+ <separator>,</separator>
+ <enclosure>"</enclosure>
+ <buffer_size>50000</buffer_size>
+ <lazy_conversion>Y</lazy_conversion>
+ <add_filename_result>N</add_filename_result>
+ <parallel>N</parallel>
+ <encoding>UTF-8</encoding>
+ <newline_possible>N</newline_possible>
+ <schemaDefinition/>
+ <ignoreFields>N</ignoreFields>
+ <namingScheme/>
+ <fields>
+ <field>
+ <name>embedding</name>
+ <length>-1</length>
+ <type>Vector</type>
+ <format/>
+ <trim_type>none</trim_type>
+ <precision>-1</precision>
+ <currency/>
+ <decimal/>
+ <group/>
+ </field>
+ </fields>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>128</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>SelectValues</type>
+ <name>Back to text</name>
+ <fields>
+ <select_unspecified>N</select_unspecified>
+ <meta>
+ <name>embedding</name>
+ <rename>embedding</rename>
+ <type>String</type>
+ <length>-2</length>
+ <precision>-2</precision>
+ <storage_type/>
+ <conversion_mask/>
+ <date_format_lenient>N</date_format_lenient>
+ <date_format_locale/>
+ <date_format_timezone/>
+ <lenient_string_to_number>N</lenient_string_to_number>
+ <decimal_symbol/>
+ <grouping_symbol/>
+ <currency_symbol/>
+ <encoding/>
+ </meta>
+ </fields>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>320</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>Dummy</type>
+ <name>OUTPUT</name>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>512</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ <description/>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <order>
+ <hop>
+ <from>Back to text</from>
+ <to>OUTPUT</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Lazy CSV</from>
+ <to>Back to text</to>
+ <enabled>Y</enabled>
+ </hop>
+ </order>
+ <notepads/>
+ <attributes/>
+ <transform_error_handling/>
+</pipeline>
diff --git
a/integration-tests/transforms/datasets/golden-vector-conversions.csv
b/integration-tests/transforms/datasets/golden-vector-conversions.csv
new file mode 100644
index 0000000000..fc1ef51f45
--- /dev/null
+++ b/integration-tests/transforms/datasets/golden-vector-conversions.csv
@@ -0,0 +1,4 @@
+embedding
+"[0.1,0.2,0.3]"
+"[1.0,2.0,3.0]"
+"[-0.5,0.25]"
diff --git
a/integration-tests/transforms/datasets/golden-vector-lazy-conversion.csv
b/integration-tests/transforms/datasets/golden-vector-lazy-conversion.csv
new file mode 100644
index 0000000000..3d9997fde0
--- /dev/null
+++ b/integration-tests/transforms/datasets/golden-vector-lazy-conversion.csv
@@ -0,0 +1,3 @@
+embedding
+"[1.0,2.0,3.0]"
+"[0.5,0.25]"
diff --git a/integration-tests/transforms/datasets/golden-vector-value-type.csv
b/integration-tests/transforms/datasets/golden-vector-value-type.csv
new file mode 100644
index 0000000000..781cc75bbc
--- /dev/null
+++ b/integration-tests/transforms/datasets/golden-vector-value-type.csv
@@ -0,0 +1,2 @@
+Fieldname,Type
+embedding,Vector
diff --git a/integration-tests/transforms/files/vector-lazy.csv
b/integration-tests/transforms/files/vector-lazy.csv
new file mode 100644
index 0000000000..677b049607
--- /dev/null
+++ b/integration-tests/transforms/files/vector-lazy.csv
@@ -0,0 +1,3 @@
+embedding
+"[1,2,3]"
+"[0.5,0.25]"
diff --git a/integration-tests/transforms/main-0114-vector-value-type.hwf
b/integration-tests/transforms/main-0114-vector-value-type.hwf
new file mode 100644
index 0000000000..97b56639ec
--- /dev/null
+++ b/integration-tests/transforms/main-0114-vector-value-type.hwf
@@ -0,0 +1,85 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<workflow>
+ <name>main-0114-vector-value-type</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description/>
+ <extended_description/>
+ <workflow_version/>
+ <created_user>-</created_user>
+ <created_date>2026/09/16 10:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/09/16 10:00:00.000</modified_date>
+ <parameters>
+ </parameters>
+ <actions>
+ <action>
+ <name>START</name>
+ <description/>
+ <type>SPECIAL</type>
+ <attributes/>
+ <repeat>N</repeat>
+ <schedulerType>0</schedulerType>
+ <intervalSeconds>0</intervalSeconds>
+ <intervalMinutes>60</intervalMinutes>
+ <hour>12</hour>
+ <minutes>0</minutes>
+ <weekDay>1</weekDay>
+ <DayOfMonth>1</DayOfMonth>
+ <parallel>N</parallel>
+ <xloc>96</xloc>
+ <yloc>80</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>0114-vector-value-type UNIT</name>
+ <description/>
+ <type>RunPipelineTests</type>
+ <attributes/>
+ <test_names>
+ <test_name>
+ <name>0114-vector-value-type UNIT</name>
+ </test_name>
+ <test_name>
+ <name>0114-vector-conversions UNIT</name>
+ </test_name>
+ <test_name>
+ <name>0116-vector-lazy-conversion UNIT</name>
+ </test_name>
+ </test_names>
+ <parallel>N</parallel>
+ <xloc>256</xloc>
+ <yloc>80</yloc>
+ <attributes_hac/>
+ </action>
+ </actions>
+ <hops>
+ <hop>
+ <from>START</from>
+ <to>0114-vector-value-type UNIT</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>Y</unconditional>
+ </hop>
+ </hops>
+ <notepads>
+ </notepads>
+ <attributes/>
+</workflow>
diff --git
a/integration-tests/transforms/metadata/dataset/golden-vector-conversions.json
b/integration-tests/transforms/metadata/dataset/golden-vector-conversions.json
new file mode 100644
index 0000000000..a3c8b3c992
--- /dev/null
+++
b/integration-tests/transforms/metadata/dataset/golden-vector-conversions.json
@@ -0,0 +1,15 @@
+{
+ "base_filename": "golden-vector-conversions.csv",
+ "name": "golden-vector-conversions",
+ "description": "Canonical text form after a String to Vector to String round
trip.",
+ "dataset_fields": [
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_name": "embedding",
+ "field_format": ""
+ }
+ ]
+}
diff --git
a/integration-tests/transforms/metadata/dataset/golden-vector-lazy-conversion.json
b/integration-tests/transforms/metadata/dataset/golden-vector-lazy-conversion.json
new file mode 100644
index 0000000000..5bdb560210
--- /dev/null
+++
b/integration-tests/transforms/metadata/dataset/golden-vector-lazy-conversion.json
@@ -0,0 +1,15 @@
+{
+ "base_filename": "golden-vector-lazy-conversion.csv",
+ "name": "golden-vector-lazy-conversion",
+ "description": "Canonical text after a lazily converted String goes through
Vector and back.",
+ "dataset_fields": [
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_name": "embedding",
+ "field_format": ""
+ }
+ ]
+}
diff --git
a/integration-tests/transforms/metadata/dataset/golden-vector-value-type.json
b/integration-tests/transforms/metadata/dataset/golden-vector-value-type.json
new file mode 100644
index 0000000000..ccbd2289a8
--- /dev/null
+++
b/integration-tests/transforms/metadata/dataset/golden-vector-value-type.json
@@ -0,0 +1,23 @@
+{
+ "base_filename": "golden-vector-value-type.csv",
+ "name": "golden-vector-value-type",
+ "description": "Stream metadata for a Data Grid field declared as the Vector
value type.",
+ "dataset_fields": [
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_name": "Fieldname",
+ "field_format": ""
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_name": "Type",
+ "field_format": ""
+ }
+ ]
+}
diff --git
a/integration-tests/transforms/metadata/unit-test/0114-vector-conversions
UNIT.json
b/integration-tests/transforms/metadata/unit-test/0114-vector-conversions
UNIT.json
new file mode 100644
index 0000000000..fa9d827f35
--- /dev/null
+++ b/integration-tests/transforms/metadata/unit-test/0114-vector-conversions
UNIT.json
@@ -0,0 +1,28 @@
+{
+ "variableValues": [],
+ "database_replacements": [],
+ "autoOpening": false,
+ "basePath": "",
+ "golden_data_sets": [
+ {
+ "field_mappings": [
+ {
+ "transform_field": "embedding",
+ "data_set_field": "embedding"
+ }
+ ],
+ "field_order": [
+ "embedding"
+ ],
+ "transform_name": "OUTPUT",
+ "data_set_name": "golden-vector-conversions"
+ }
+ ],
+ "input_data_sets": [],
+ "name": "0114-vector-conversions UNIT",
+ "description": "Verifies a vector round trips losslessly through String to
Vector to String, and that integer components are normalised to float form",
+ "trans_test_tweaks": [],
+ "persist_filename": "",
+ "pipeline_filename": "./0114-vector-conversions.hpl",
+ "test_type": "UNIT_TEST"
+}
diff --git
a/integration-tests/transforms/metadata/unit-test/0114-vector-value-type
UNIT.json
b/integration-tests/transforms/metadata/unit-test/0114-vector-value-type
UNIT.json
new file mode 100644
index 0000000000..0f0824a97e
--- /dev/null
+++ b/integration-tests/transforms/metadata/unit-test/0114-vector-value-type
UNIT.json
@@ -0,0 +1,33 @@
+{
+ "variableValues": [],
+ "database_replacements": [],
+ "autoOpening": false,
+ "basePath": "",
+ "golden_data_sets": [
+ {
+ "field_mappings": [
+ {
+ "transform_field": "Fieldname",
+ "data_set_field": "Fieldname"
+ },
+ {
+ "transform_field": "Type",
+ "data_set_field": "Type"
+ }
+ ],
+ "field_order": [
+ "Fieldname",
+ "Type"
+ ],
+ "transform_name": "OUTPUT",
+ "data_set_name": "golden-vector-value-type"
+ }
+ ],
+ "input_data_sets": [],
+ "name": "0114-vector-value-type UNIT",
+ "description": "Verifies the Vector value type is registered and that a
field declared as Vector is reported as such in the stream metadata",
+ "trans_test_tweaks": [],
+ "persist_filename": "",
+ "pipeline_filename": "./0114-vector-value-type.hpl",
+ "test_type": "UNIT_TEST"
+}
diff --git
a/integration-tests/transforms/metadata/unit-test/0116-vector-lazy-conversion
UNIT.json
b/integration-tests/transforms/metadata/unit-test/0116-vector-lazy-conversion
UNIT.json
new file mode 100644
index 0000000000..451b6014c0
--- /dev/null
+++
b/integration-tests/transforms/metadata/unit-test/0116-vector-lazy-conversion
UNIT.json
@@ -0,0 +1,28 @@
+{
+ "variableValues": [],
+ "database_replacements": [],
+ "autoOpening": false,
+ "basePath": "",
+ "golden_data_sets": [
+ {
+ "field_mappings": [
+ {
+ "transform_field": "embedding",
+ "data_set_field": "embedding"
+ }
+ ],
+ "field_order": [
+ "embedding"
+ ],
+ "transform_name": "OUTPUT",
+ "data_set_name": "golden-vector-lazy-conversion"
+ }
+ ],
+ "input_data_sets": [],
+ "name": "0116-vector-lazy-conversion UNIT",
+ "description": "A lazily converted String field converts to Vector and back
through Select Values",
+ "trans_test_tweaks": [],
+ "persist_filename": "",
+ "pipeline_filename": "./0116-vector-lazy-conversion.hpl",
+ "test_type": "UNIT_TEST"
+}
diff --git a/plugins/valuetypes/pom.xml b/plugins/valuetypes/pom.xml
index fb16e7c8c3..433dbd0cb5 100644
--- a/plugins/valuetypes/pom.xml
+++ b/plugins/valuetypes/pom.xml
@@ -30,5 +30,6 @@
<name>Hop Plugins Value Types</name>
<modules>
<module>uuid</module>
+ <module>vector</module>
</modules>
</project>
diff --git a/plugins/valuetypes/pom.xml b/plugins/valuetypes/vector/pom.xml
similarity index 84%
copy from plugins/valuetypes/pom.xml
copy to plugins/valuetypes/vector/pom.xml
index fb16e7c8c3..ca6a709865 100644
--- a/plugins/valuetypes/pom.xml
+++ b/plugins/valuetypes/vector/pom.xml
@@ -14,21 +14,17 @@
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
- ~
-->
<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
<modelVersion>4.0.0</modelVersion>
-
<parent>
<groupId>org.apache.hop</groupId>
- <artifactId>hop-plugins</artifactId>
+ <artifactId>hop-plugins-valuetypes</artifactId>
<version>2.20.0-SNAPSHOT</version>
</parent>
- <artifactId>hop-plugins-valuetypes</artifactId>
- <packaging>pom</packaging>
- <name>Hop Plugins Value Types</name>
- <modules>
- <module>uuid</module>
- </modules>
+ <artifactId>hop-valuetypes-vector</artifactId>
+ <packaging>jar</packaging>
+ <name>Hop Plugins Vector Type</name>
+
</project>
diff --git a/plugins/valuetypes/pom.xml
b/plugins/valuetypes/vector/src/assembly/assembly.xml
similarity index 50%
copy from plugins/valuetypes/pom.xml
copy to plugins/valuetypes/vector/src/assembly/assembly.xml
index fb16e7c8c3..c7dfcb8994 100644
--- a/plugins/valuetypes/pom.xml
+++ b/plugins/valuetypes/vector/src/assembly/assembly.xml
@@ -1,4 +1,3 @@
-<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
@@ -16,19 +15,25 @@
~ limitations under the License.
~
-->
-<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
- <modelVersion>4.0.0</modelVersion>
- <parent>
- <groupId>org.apache.hop</groupId>
- <artifactId>hop-plugins</artifactId>
- <version>2.20.0-SNAPSHOT</version>
- </parent>
+<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.2.0";
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.2.0
http://maven.apache.org/xsd/assembly-2.2.0.xsd";>
+ <id>hop-valuetypes-vector</id>
+ <formats>
+ <format>zip</format>
+ </formats>
+ <baseDirectory>.</baseDirectory>
+ <files>
+ <file>
+ <source>${project.basedir}/src/main/resources/version.xml</source>
+ <outputDirectory>${hop.plugin.libdir}</outputDirectory>
+ <filtered>true</filtered>
+ </file>
+ </files>
- <artifactId>hop-plugins-valuetypes</artifactId>
- <packaging>pom</packaging>
- <name>Hop Plugins Value Types</name>
- <modules>
- <module>uuid</module>
- </modules>
-</project>
+
+ <componentDescriptors>
+
<componentDescriptor>${maven.multiModuleProjectDirectory}/assemblies/shared/hop-plugin-libs.xml</componentDescriptor>
+ </componentDescriptors>
+</assembly>
\ No newline at end of file
diff --git
a/plugins/valuetypes/vector/src/main/java/org/apache/hop/vector/ValueMetaVector.java
b/plugins/valuetypes/vector/src/main/java/org/apache/hop/vector/ValueMetaVector.java
new file mode 100644
index 0000000000..70097c3c97
--- /dev/null
+++
b/plugins/valuetypes/vector/src/main/java/org/apache/hop/vector/ValueMetaVector.java
@@ -0,0 +1,422 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.vector;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.net.SocketTimeoutException;
+import java.nio.charset.Charset;
+import java.nio.charset.StandardCharsets;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Types;
+import java.util.Arrays;
+import org.apache.hop.core.database.DatabaseMeta;
+import org.apache.hop.core.database.IDatabase;
+import org.apache.hop.core.exception.HopDatabaseException;
+import org.apache.hop.core.exception.HopEofException;
+import org.apache.hop.core.exception.HopFileException;
+import org.apache.hop.core.exception.HopValueException;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.value.ValueMetaBase;
+import org.apache.hop.core.row.value.ValueMetaPlugin;
+
+/**
+ * A dense floating point vector, as produced by an embedding model and
consumed by a vector store.
+ *
+ * <p>The canonical text form is a bracketed, comma separated list of numbers:
{@code
+ * [0.1,0.2,0.3]}. That form is both valid JSON and the literal syntax
pgvector accepts, so a vector
+ * survives a round trip through a text file, a Data Grid, a JSON document or
a database column
+ * without a conversion step in between.
+ *
+ * <p>The field's {@code length} carries the number of dimensions, the way it
carries the size of a
+ * String. A length of zero or less means the dimension is unspecified. Hop
never pads or truncates
+ * a vector to fit a declared length: an embedding resized to match would
still retrieve results,
+ * and they would be wrong.
+ */
+@ValueMetaPlugin(
+ id = "1536", // the dimension of OpenAI's text-embedding-3-small
+ name = "Vector",
+ description = "Dense floating point vector (embedding)",
+ image = "vector.svg")
+public class ValueMetaVector extends ValueMetaBase {
+
+ public static final int TYPE_VECTOR = 1536;
+
+ public ValueMetaVector() {
+ super(null, TYPE_VECTOR);
+ }
+
+ public ValueMetaVector(String name) {
+ super(name, TYPE_VECTOR);
+ }
+
+ public ValueMetaVector(ValueMetaVector meta) {
+ super(meta.name, TYPE_VECTOR);
+ }
+
+ @Override
+ public ValueMetaVector clone() {
+ return (ValueMetaVector) super.clone();
+ }
+
+ @Override
+ public Class<?> getNativeDataTypeClass() {
+ return float[].class;
+ }
+
+ @Override
+ public Object convertData(IValueMeta meta2, Object data2) throws
HopValueException {
+ return toVector(meta2, data2);
+ }
+
+ /**
+ * Convert the specified data to a vector. Used internally instead of
convertData() to avoid
+ * upcasts and casts.
+ */
+ private float[] toVector(IValueMeta meta2, Object data2) throws
HopValueException {
+ if (data2 == null) {
+ return null;
+ }
+ // Already a vector? Done.
+ if (data2 instanceof float[] vector) {
+ return vector;
+ }
+ try {
+ switch (meta2.getType()) {
+ case TYPE_VECTOR:
+ switch (meta2.getStorageType()) {
+ case STORAGE_TYPE_NORMAL:
+ // Only reached when the storage type is normal and the data is
still a String.
+ // A float[] returns above.
+ return parse((String) data2);
+ case STORAGE_TYPE_BINARY_STRING:
+ // The Vector field's own lazy bytes: meta2 owns the storage
metadata needed to
+ // decode them, so the conversion has to run on meta2 and not on
this.
+ return (float[]) meta2.convertBinaryStringToNativeType((byte[])
data2);
+ case STORAGE_TYPE_INDEXED:
+ return toVector(this, meta2.getIndex()[(Integer) data2]);
+ default:
+ break;
+ }
+ break;
+ case TYPE_STRING:
+ // getString resolves normal, lazily converted and indexed storage
against meta2's own
+ // metadata. Converting through this Vector instead would use a
storage metadata that a
+ // freshly built target field does not have.
+ return parse(meta2.getString(data2));
+ default:
+ break;
+ }
+ } catch (HopValueException e) {
+ throw e;
+ } catch (ClassCastException e) {
+ throw new HopValueException(
+ this + " : I can't convert the specified value to data type :
Vector", e);
+ }
+ throw new HopValueException(
+ this + " : I can't convert the specified value to data type : Vector");
+ }
+
+ /**
+ * Parse the canonical text form. Surrounding brackets are optional and
whitespace is ignored, so
+ * both {@code [0.1, 0.2]} and {@code 0.1,0.2} are accepted.
+ *
+ * @param string the text to parse, or null
+ * @return the parsed vector, or null when the input is null or blank
+ * @throws HopValueException when the text is not a list of numbers
+ */
+ public static float[] parse(String string) throws HopValueException {
+ if (string == null) {
+ return null;
+ }
+ String trimmed = string.trim();
+ if (trimmed.isEmpty()) {
+ return null;
+ }
+ if (trimmed.startsWith("[") && trimmed.endsWith("]")) {
+ trimmed = trimmed.substring(1, trimmed.length() - 1).trim();
+ }
+ if (trimmed.isEmpty()) {
+ return new float[0];
+ }
+ String[] parts = trimmed.split(",", -1);
+ float[] vector = new float[parts.length];
+ for (int i = 0; i < parts.length; i++) {
+ String part = parts[i].trim();
+ if (part.isEmpty()) {
+ throw new HopValueException(
+ "Empty element at position " + i + " while parsing a vector from
'" + string + "'");
+ }
+ try {
+ vector[i] = Float.parseFloat(part);
+ } catch (NumberFormatException e) {
+ throw new HopValueException(
+ "'" + part + "' at position " + i + " is not a number in vector '"
+ string + "'", e);
+ }
+ }
+ return vector;
+ }
+
+ /**
+ * Render a vector in the canonical text form.
+ *
+ * @param vector the vector, or null
+ * @return the text form, or null when the vector is null
+ */
+ public static String render(float[] vector) {
+ if (vector == null) {
+ return null;
+ }
+ StringBuilder builder = new StringBuilder(vector.length * 12 + 2);
+ builder.append('[');
+ for (int i = 0; i < vector.length; i++) {
+ if (i > 0) {
+ builder.append(',');
+ }
+ builder.append(vector[i]);
+ }
+ builder.append(']');
+ return builder.toString();
+ }
+
+ @Override
+ public int hashCode(Object object) throws HopValueException {
+ float[] vector = toVector(this, object);
+ return vector == null ? 0 : Arrays.hashCode(vector);
+ }
+
+ @Override
+ public Object cloneValueData(Object object) throws HopValueException {
+ if (object == null) {
+ return null;
+ }
+ // Lazily converted values stay in their binary form, as ValueMetaBase
does: materialising
+ // them here would leave a float[] behind a storage type that still says
binary string.
+ if (storageType != STORAGE_TYPE_NORMAL) {
+ return object;
+ }
+ // Unlike the scalar types, a vector is mutable: hand out a copy so that
two rows sharing a
+ // value can not write through each other.
+ if (object instanceof float[] vector) {
+ return vector.clone();
+ }
+ return toVector(this, object);
+ }
+
+ /**
+ * Vectors have no meaningful natural order, but sorting, grouping and
distinct all need a total
+ * order that is stable. This is the lexicographic order of {@link
Arrays#compare(float[],
+ * float[])}: the first differing element decides, and a vector that is a
prefix of another sorts
+ * first.
+ */
+ @Override
+ protected int typeCompare(Object object1, Object object2) throws
HopValueException {
+ float[] vector1 = toVector(this, object1);
+ float[] vector2 = toVector(this, object2);
+ if (vector1 == null && vector2 == null) {
+ return 0;
+ }
+ if (vector1 == null) {
+ return -1;
+ }
+ if (vector2 == null) {
+ return 1;
+ }
+ return Arrays.compare(vector1, vector2);
+ }
+
+ private static float[] toFloats(double[] doubles) {
+ float[] vector = new float[doubles.length];
+ for (int i = 0; i < doubles.length; i++) {
+ vector[i] = (float) doubles[i];
+ }
+ return vector;
+ }
+
+ private static float[] fromSqlArray(java.sql.Array array) throws
HopValueException {
+ try {
+ Object elements = array.getArray();
+ if (elements instanceof float[] floats) {
+ return floats;
+ }
+ if (elements instanceof double[] doubles) {
+ return toFloats(doubles);
+ }
+ if (elements instanceof Number[] numbers) {
+ float[] vector = new float[numbers.length];
+ for (int i = 0; i < numbers.length; i++) {
+ vector[i] = numbers[i] == null ? 0f : numbers[i].floatValue();
+ }
+ return vector;
+ }
+ return parse(String.valueOf(elements));
+ } catch (SQLException e) {
+ throw new HopValueException("Unable to read a vector from a SQL array
value", e);
+ }
+ }
+
+ @Override
+ public String getString(Object object) throws HopValueException {
+ return render(toVector(this, object));
+ }
+
+ @Override
+ public void setPreparedStatementValue(
+ DatabaseMeta databaseMeta, PreparedStatement preparedStatement, int
index, Object data)
+ throws HopDatabaseException {
+ try {
+ float[] vector = toVector(this, data);
+ if (vector == null) {
+ preparedStatement.setNull(index, Types.VARCHAR);
+ return;
+ }
+ // The canonical text form. pgvector accepts it for a vector column, and
a database without
+ // a vector type stores the same text in a character column. Handing the
driver the float[]
+ // instead is not portable: most drivers reject it, and the ones that do
not tend to write a
+ // serialized Java object into the column.
+ preparedStatement.setString(index, render(vector));
+ } catch (Exception e) {
+ throw new HopDatabaseException(
+ "Error setting vector value #"
+ + index
+ + " ["
+ + toStringMeta()
+ + "] on prepared statement",
+ e);
+ }
+ }
+
+ @Override
+ public Object getValueFromResultSet(IDatabase iDatabase, ResultSet
resultSet, int index)
+ throws HopDatabaseException {
+ try {
+ Object object = resultSet.getObject(index + 1);
+ if (object == null) {
+ return null;
+ }
+ if (object instanceof float[] vector) {
+ return vector;
+ }
+ if (object instanceof double[] doubles) {
+ return toFloats(doubles);
+ }
+ if (object instanceof String text) {
+ return parse(text);
+ }
+ if (object instanceof java.sql.Array array) {
+ return fromSqlArray(array);
+ }
+ // pgvector hands back its own object type through getObject(); its
toString() is the
+ // canonical form, which is also what a character column returns.
Anything else would give
+ // Object.toString(), which parse rejects with a clearer message than a
cast would.
+ return parse(object.toString());
+ } catch (SQLException e) {
+ throw new HopDatabaseException(
+ "Unable to get vector value '"
+ + toStringMeta()
+ + "' from database resultset, index "
+ + index,
+ e);
+ } catch (Exception e) {
+ throw new HopDatabaseException("Unable to read vector value", e);
+ }
+ }
+
+ @Override
+ public byte[] getBinaryString(Object object) throws HopValueException {
+ if (isStorageBinaryString() && identicalFormat) {
+ return (byte[]) object;
+ }
+ float[] vector = toVector(this, object);
+ if (vector == null) {
+ return null;
+ }
+ try {
+ String encode = getStringEncoding();
+ Charset charset = encode == null ? StandardCharsets.UTF_8 :
Charset.forName(encode);
+ return render(vector).getBytes(charset);
+ } catch (Exception e) {
+ throw new HopValueException("Unable to get binary string for vector", e);
+ }
+ }
+
+ @Override
+ public void writeData(DataOutputStream outputStream, Object object) throws
HopFileException {
+ // Delegate non-NORMAL cases to the base class
+ if (getStorageType() != STORAGE_TYPE_NORMAL) {
+ super.writeData(outputStream, object);
+ return;
+ }
+ try {
+ // Convert before writing the flag: parse maps blank text to null, so a
value that
+ // getString and hashCode already treat as null has to serialise as null
too.
+ float[] vector = toVector(this, object);
+ outputStream.writeBoolean(vector == null);
+ if (vector != null) {
+ // Length prefixed binary floats rather than the text form: a 1536
element vector is 6 kB
+ // as text and 6 kB of parsing on the way back, against 6 kB of raw
floats here.
+ outputStream.writeInt(vector.length);
+ for (float value : vector) {
+ outputStream.writeFloat(value);
+ }
+ }
+ } catch (IOException e) {
+ throw new HopFileException(this + " : Unable to write value data to
output stream", e);
+ } catch (Exception e) {
+ throw new HopFileException(
+ "Unable to convert data to a vector before writing to output
stream", e);
+ }
+ }
+
+ protected float[] readVector(DataInputStream inputStream) throws IOException
{
+ int length = inputStream.readInt();
+ if (length < 0) {
+ return null;
+ }
+ float[] vector = new float[length];
+ for (int i = 0; i < length; i++) {
+ vector[i] = inputStream.readFloat();
+ }
+ return vector;
+ }
+
+ @Override
+ public Object readData(DataInputStream inputStream)
+ throws HopFileException, SocketTimeoutException {
+ // Delegate non-NORMAL cases to the base class
+ if (getStorageType() != STORAGE_TYPE_NORMAL) {
+ return super.readData(inputStream);
+ }
+ try {
+ if (inputStream.readBoolean()) {
+ return null;
+ }
+ return readVector(inputStream);
+ } catch (EOFException e) {
+ throw new HopEofException(e);
+ } catch (SocketTimeoutException e) {
+ throw e;
+ } catch (IOException e) {
+ throw new HopFileException(this + " : Unable to read vector value data
from input stream", e);
+ }
+ }
+}
diff --git a/plugins/valuetypes/vector/src/main/resources/vector.svg
b/plugins/valuetypes/vector/src/main/resources/vector.svg
new file mode 100644
index 0000000000..5be125b3ae
--- /dev/null
+++ b/plugins/valuetypes/vector/src/main/resources/vector.svg
@@ -0,0 +1,10 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg version="1.1" xmlns="http://www.w3.org/2000/svg"; width="200px"
height="200px" viewBox="0 0 200 200">
+<path fill="#0e3a5a" d="M30,52h14v96H30V52z M156,52h14v96h-14V52z
M30,52h22v12H30V52z M30,136h22v12H30V136z M148,52h22v12h-22V52z
+ M148,136h22v12h-22V136z"/>
+<circle fill="#0e3a5a" cx="76" cy="76" r="9"/>
+<circle fill="#0e3a5a" cx="100" cy="100" r="9"/>
+<circle fill="#0e3a5a" cx="124" cy="124" r="9"/>
+<path fill="none" stroke="#0e3a5a" stroke-width="6" stroke-linecap="round"
d="M70,130l60-60"/>
+<path fill="#0e3a5a" d="M130,70l-18,2l16,16L130,70z"/>
+</svg>
diff --git a/plugins/valuetypes/pom.xml
b/plugins/valuetypes/vector/src/main/resources/version.xml
similarity index 59%
copy from plugins/valuetypes/pom.xml
copy to plugins/valuetypes/vector/src/main/resources/version.xml
index fb16e7c8c3..6be576acae 100644
--- a/plugins/valuetypes/pom.xml
+++ b/plugins/valuetypes/vector/src/main/resources/version.xml
@@ -16,19 +16,5 @@
~ limitations under the License.
~
-->
-<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
- <modelVersion>4.0.0</modelVersion>
- <parent>
- <groupId>org.apache.hop</groupId>
- <artifactId>hop-plugins</artifactId>
- <version>2.20.0-SNAPSHOT</version>
- </parent>
-
- <artifactId>hop-plugins-valuetypes</artifactId>
- <packaging>pom</packaging>
- <name>Hop Plugins Value Types</name>
- <modules>
- <module>uuid</module>
- </modules>
-</project>
+<version>${project.version}</version>
\ No newline at end of file
diff --git
a/plugins/valuetypes/vector/src/test/java/org/apache/hop/vector/ValueMetaVectorTest.java
b/plugins/valuetypes/vector/src/test/java/org/apache/hop/vector/ValueMetaVectorTest.java
new file mode 100644
index 0000000000..1618227b7d
--- /dev/null
+++
b/plugins/valuetypes/vector/src/test/java/org/apache/hop/vector/ValueMetaVectorTest.java
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.vector;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyInt;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.verify;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.nio.charset.StandardCharsets;
+import java.sql.PreparedStatement;
+import org.apache.hop.core.HopClientEnvironment;
+import org.apache.hop.core.database.DatabaseMeta;
+import org.apache.hop.core.exception.HopValueException;
+import org.apache.hop.core.row.IValueMeta;
+import org.apache.hop.core.row.value.ValueMetaFactory;
+import org.apache.hop.core.row.value.ValueMetaString;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.Test;
+
+class ValueMetaVectorTest {
+
+ @BeforeAll
+ static void setUpBeforeClass() throws Exception {
+ HopClientEnvironment.init();
+ }
+
+ @Test
+ void testTypeAndNativeClass() {
+ ValueMetaVector meta = new ValueMetaVector("embedding");
+ assertEquals(ValueMetaVector.TYPE_VECTOR, meta.getType());
+ assertEquals("embedding", meta.getName());
+ assertEquals(float[].class, meta.getNativeDataTypeClass());
+ }
+
+ @Test
+ void testParseCanonicalForm() throws Exception {
+ assertArrayEquals(new float[] {0.1f, 0.2f, 0.3f},
ValueMetaVector.parse("[0.1,0.2,0.3]"));
+ }
+
+ @Test
+ void testParseToleratesWhitespaceAndMissingBrackets() throws Exception {
+ assertArrayEquals(new float[] {0.1f, 0.2f}, ValueMetaVector.parse(" [ 0.1
, 0.2 ] "));
+ assertArrayEquals(new float[] {0.1f, 0.2f}, ValueMetaVector.parse("0.1,
0.2"));
+ }
+
+ @Test
+ void testParseNegativeAndScientificNotation() throws Exception {
+ assertArrayEquals(new float[] {-0.5f, 1.5e-3f},
ValueMetaVector.parse("[-0.5,1.5e-3]"));
+ }
+
+ @Test
+ void testParseNullAndBlankAndEmpty() throws Exception {
+ assertNull(ValueMetaVector.parse(null));
+ assertNull(ValueMetaVector.parse(" "));
+ assertArrayEquals(new float[0], ValueMetaVector.parse("[]"));
+ }
+
+ @Test
+ void testParseRejectsNonNumbers() {
+ assertThrows(HopValueException.class, () ->
ValueMetaVector.parse("[0.1,abc]"));
+ assertThrows(HopValueException.class, () ->
ValueMetaVector.parse("[0.1,,0.2]"));
+ }
+
+ @Test
+ void testRenderRoundTrip() throws Exception {
+ float[] vector = {0.1f, -2.5f, 3.0f};
+ String rendered = ValueMetaVector.render(vector);
+ assertTrue(rendered.startsWith("[") && rendered.endsWith("]"));
+ assertArrayEquals(vector, ValueMetaVector.parse(rendered));
+ assertNull(ValueMetaVector.render(null));
+ }
+
+ @Test
+ void testGetStringUsesCanonicalForm() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("v");
+ assertEquals("[1.0,2.0]", meta.getString(new float[] {1f, 2f}));
+ assertNull(meta.getString(null));
+ }
+
+ @Test
+ void testConvertFromString() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("v");
+ Object converted = meta.convertData(new ValueMetaString("s"), "[1,2,3]");
+ assertInstanceOf(float[].class, converted);
+ assertArrayEquals(new float[] {1f, 2f, 3f}, (float[]) converted);
+ }
+
+ @Test
+ void testConvertPassesThroughVector() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("v");
+ float[] vector = {1f, 2f};
+ assertArrayEquals(vector, (float[]) meta.convertData(meta, vector));
+ }
+
+ @Test
+ void testConvertNullStaysNull() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("v");
+ assertNull(meta.convertData(new ValueMetaString("s"), null));
+ }
+
+ @Test
+ void testCloneValueDataCopiesTheArray() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("v");
+ float[] original = {1f, 2f, 3f};
+ float[] copy = (float[]) meta.cloneValueData(original);
+ assertArrayEquals(original, copy);
+ assertNotSame(original, copy);
+ copy[0] = 99f;
+ assertEquals(1f, original[0], 0.0f);
+ }
+
+ @Test
+ void testCompareOrdersByLengthThenElement() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("v");
+ assertTrue(meta.compare(new float[] {1f}, new float[] {1f, 2f}) < 0);
+ assertTrue(meta.compare(new float[] {1f, 3f}, new float[] {1f, 2f}) > 0);
+ assertEquals(0, meta.compare(new float[] {1f, 2f}, new float[] {1f, 2f}));
+ }
+
+ @Test
+ void testCompareHandlesNulls() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("v");
+ assertEquals(0, meta.compare(null, null));
+ assertTrue(meta.compare(null, new float[] {1f}) < 0);
+ assertTrue(meta.compare(new float[] {1f}, null) > 0);
+ }
+
+ @Test
+ void testHashCodeMatchesContent() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("v");
+ assertEquals(meta.hashCode(new float[] {1f, 2f}), meta.hashCode(new
float[] {1f, 2f}));
+ assertEquals(0, meta.hashCode(null));
+ }
+
+ @Test
+ void testBinaryStringRoundTrip() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("v");
+ byte[] binary = meta.getBinaryString(new float[] {1f, 2f});
+ assertEquals("[1.0,2.0]", new String(binary, StandardCharsets.UTF_8));
+ assertNull(meta.getBinaryString(null));
+ }
+
+ @Test
+ void testWriteAndReadDataRoundTrip() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("v");
+ float[] vector = new float[1536];
+ for (int i = 0; i < vector.length; i++) {
+ vector[i] = i * 0.001f;
+ }
+ ByteArrayOutputStream bytes = new ByteArrayOutputStream();
+ try (DataOutputStream out = new DataOutputStream(bytes)) {
+ meta.writeData(out, vector);
+ }
+ try (DataInputStream in = new DataInputStream(new
ByteArrayInputStream(bytes.toByteArray()))) {
+ assertArrayEquals(vector, (float[]) meta.readData(in));
+ }
+ }
+
+ @Test
+ void testWriteAndReadNull() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("v");
+ ByteArrayOutputStream bytes = new ByteArrayOutputStream();
+ try (DataOutputStream out = new DataOutputStream(bytes)) {
+ meta.writeData(out, null);
+ }
+ try (DataInputStream in = new DataInputStream(new
ByteArrayInputStream(bytes.toByteArray()))) {
+ assertNull(meta.readData(in));
+ }
+ }
+
+ @Test
+ void testBinaryStringStorageConvertsBack() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("v");
+ meta.setStorageType(IValueMeta.STORAGE_TYPE_BINARY_STRING);
+ meta.setStorageMetadata(new ValueMetaString("v"));
+ byte[] binary = "[1,2,3]".getBytes(StandardCharsets.UTF_8);
+ assertArrayEquals(new float[] {1f, 2f, 3f}, (float[])
meta.convertData(meta, binary));
+ }
+
+ @Test
+ void convertsALazilyConvertedStringFieldToVector() throws Exception {
+ // Select Values builds a fresh NORMAL target whose storageMetadata is
null. The conversion
+ // has to run against the source field's metadata, not the target's.
+ ValueMetaString source = new ValueMetaString("embedding");
+ source.setStorageType(IValueMeta.STORAGE_TYPE_BINARY_STRING);
+ source.setStorageMetadata(new ValueMetaString("embedding"));
+ byte[] lazy = "[1,2,3]".getBytes(StandardCharsets.UTF_8);
+
+ ValueMetaVector target = new ValueMetaVector("embedding");
+
+ assertArrayEquals(new float[] {1f, 2f, 3f}, (float[])
target.convertData(source, lazy));
+ }
+
+ @Test
+ void cloneLeavesLazilyConvertedValuesInTheirBinaryForm() throws Exception {
+ ValueMetaVector meta = new ValueMetaVector("embedding");
+ meta.setStorageType(IValueMeta.STORAGE_TYPE_BINARY_STRING);
+ meta.setStorageMetadata(new ValueMetaString("embedding"));
+ byte[] lazy = "[1,2,3]".getBytes(StandardCharsets.UTF_8);
+
+ // Materialising here would leave a float[] behind a binary-string storage
type, which blows
+ // up later in writeData.
+ assertSame(lazy, meta.cloneValueData(lazy));
+ }
+
+ @Test
+ void isRegisteredWithTheValueMetaFactory() throws Exception {
+ assertEquals(ValueMetaVector.TYPE_VECTOR,
ValueMetaFactory.getIdForValueMeta("Vector"));
+ }
+
+ @Test
+ void bindsTheCanonicalTextFormToAPreparedStatement() throws Exception {
+ PreparedStatement statement = mock(PreparedStatement.class);
+
+ new ValueMetaVector("embedding")
+ .setPreparedStatementValue(
+ mock(DatabaseMeta.class), statement, 1, new float[] {0.1f, 0.2f});
+
+ verify(statement).setString(1, "[0.1,0.2]");
+ verify(statement, never()).setObject(anyInt(), any(float[].class));
+ }
+}
