hansva commented on code in PR #9367:
URL: https://github.com/apache/hop/pull/9367#discussion_r4537467763
##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java:
##########
@@ -326,14 +325,41 @@ public void getFields(
}
}
+ /**
+ * Adds an output field. When the input already has a field with that name
the transform writes
+ * its values into that field, so the field takes the transform's type if
the types differ.
+ */
private static void addField(IRowMeta row, IValueMeta field, String origin)
throws HopPluginException {
String fieldName = field.getName();
- if (fieldName == null || fieldName.isEmpty() ||
row.indexOfValue(fieldName) >= 0) {
+ if (fieldName == null || fieldName.isEmpty()) {
return;
}
field.setOrigin(origin);
- row.addValueMeta(field);
+ int index = row.indexOfValue(fieldName);
+ if (index < 0) {
+ row.addValueMeta(field);
+ } else if (row.getValueMeta(index).getType() != field.getType()) {
+ row.setValueMeta(index, field);
Review Comment:
`indexOfValue` matches without regard to case, so an input field
`Chunk_Index` is replaced by an Integer named `chunk_index`. Later transforms
that look up `Chunk_Index` case-sensitively (JSON, Avro writers) lose the
field, with no warning. Maybe only replace on an exact name match, or keep the
existing name?
##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java:
##########
@@ -430,6 +466,38 @@ public void check(
}
}
+ /** Warns when an output field already exists in the input with a different
type. */
+ private void checkOutputFieldTypes(
+ List<ICheckResult> remarks, TransformMeta transformMeta, IRowMeta prev) {
+ if (prev == null) {
+ return;
+ }
+ IRowMeta output = prev.clone();
+ try {
+ getFields(output, transformMeta.getName(), null, null, null, null);
+ } catch (HopTransformException e) {
+ return;
+ }
+ for (String fieldName : getOutputFieldNames()) {
Review Comment:
`getOutputFieldNames()` can contain the same name twice (e.g. index and
count field both set to `n`), which adds the same 'Output field already exists'
warning twice. A `LinkedHashSet` would avoid that.
##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java:
##########
@@ -326,14 +325,41 @@ public void getFields(
}
}
+ /**
+ * Adds an output field. When the input already has a field with that name
the transform writes
+ * its values into that field, so the field takes the transform's type if
the types differ.
+ */
private static void addField(IRowMeta row, IValueMeta field, String origin)
throws HopPluginException {
String fieldName = field.getName();
- if (fieldName == null || fieldName.isEmpty() ||
row.indexOfValue(fieldName) >= 0) {
+ if (fieldName == null || fieldName.isEmpty()) {
return;
}
field.setOrigin(origin);
- row.addValueMeta(field);
+ int index = row.indexOfValue(fieldName);
+ if (index < 0) {
+ row.addValueMeta(field);
+ } else if (row.getValueMeta(index).getType() != field.getType()) {
Review Comment:
Only the type is compared, so an input field of the same type keeps its
storage type, conversion mask and length while the transform writes a plain
String or Long into it. With lazy conversion (e.g. a CSV input with a String
`chunk_text` in BINARY_STRING storage) the row metadata says `byte[]` but the
value is a `java.lang.String`, and a later `getString()`/compare throws
ClassCastException. The new test `anExistingFieldOfTheSameTypeIsKept` locks
that in. I think the output field's metadata should always replace the existing
one (or at least when storage type/mask/length differ).
##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunker.java:
##########
@@ -67,8 +66,21 @@ public boolean init() {
if (!super.init()) {
return false;
}
- data.chunkSize = Const.toInt(resolve(meta.getChunkSize()), -1);
- data.chunkOverlap = Const.toInt(resolve(meta.getChunkOverlap()), -1);
+ String sizeText = resolve(meta.getChunkSize());
Review Comment:
This parses and validates size and overlap again, next to the same rules in
`TextChunkerMeta.check()` (lines 439-458), and the two already disagree on a
negative overlap (error in check, clamped here). One shared helper on the Meta
that both call would keep them in sync.
##########
integration-tests/transforms/main-0126-text-chunker-invalid-overlap.hwf:
##########
@@ -0,0 +1,126 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<workflow>
+ <name>main-0126-text-chunker-invalid-overlap</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>Text chunker: a chunk overlap that is not a number stops the
transform instead of running with overlap 0 (#9285)</description>
+ <extended_description/>
+ <workflow_version/>
+ <created_user>-</created_user>
+ <created_date>2026/11/07 12:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/11/07 12:00:00.000</modified_date>
+ <parameters>
+ </parameters>
+ <actions>
+ <action>
+ <name>START</name>
+ <description/>
+ <type>SPECIAL</type>
+ <attributes/>
+ <repeat>N</repeat>
+ <schedulerType>0</schedulerType>
+ <intervalSeconds>0</intervalSeconds>
+ <intervalMinutes>60</intervalMinutes>
+ <hour>12</hour>
+ <minutes>0</minutes>
+ <weekDay>1</weekDay>
+ <DayOfMonth>1</DayOfMonth>
+ <parallel>N</parallel>
+ <xloc>96</xloc>
+ <yloc>80</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+
<filename>${PROJECT_HOME}/0126-text-chunker-invalid-overlap.hpl</filename>
+ <params_from_previous>N</params_from_previous>
+ <exec_per_row>N</exec_per_row>
+ <clear_rows>N</clear_rows>
+ <clear_files>N</clear_files>
+ <create_parent_folder>N</create_parent_folder>
+ <set_logfile>N</set_logfile>
+ <set_append_logfile>N</set_append_logfile>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <loglevel>Basic</loglevel>
+ <wait_until_finished>Y</wait_until_finished>
+ <parameters>
+ <pass_all_parameters>Y</pass_all_parameters>
+ </parameters>
+ <run_configuration>local</run_configuration>
+ <name>0126-text-chunker-invalid-overlap.hpl</name>
+ <description>Expected to fail</description>
+ <type>PIPELINE</type>
+ <attributes/>
+ <xloc>320</xloc>
+ <yloc>80</yloc>
+ <parallel>N</parallel>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Overlap 'abc' did not fail - Abort</name>
+ <description/>
+ <type>ABORT</type>
+ <attributes/>
+ <message>A chunk overlap of 'abc' did not stop the Text chunker</message>
+ <loglevel>Basic</loglevel>
+ <parallel>N</parallel>
+ <xloc>320</xloc>
+ <yloc>224</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Success</name>
+ <description/>
+ <type>SUCCESS</type>
+ <attributes/>
+ <parallel>N</parallel>
+ <xloc>544</xloc>
+ <yloc>80</yloc>
+ <attributes_hac/>
+ </action>
+ </actions>
+ <hops>
+ <hop>
+ <from>START</from>
+ <to>0126-text-chunker-invalid-overlap.hpl</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>Y</unconditional>
+ </hop>
+ <hop>
+ <from>0126-text-chunker-invalid-overlap.hpl</from>
+ <to>Overlap 'abc' did not fail - Abort</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>0126-text-chunker-invalid-overlap.hpl</from>
Review Comment:
This passes whenever the child pipeline fails, for any reason (missing
input, a broken transform...), so it doesn't prove the overlap is what makes it
fail. Checking the error/log text would make it a real regression test for the
overlap check.
##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java:
##########
@@ -326,14 +325,41 @@ public void getFields(
}
}
+ /**
+ * Adds an output field. When the input already has a field with that name
the transform writes
+ * its values into that field, so the field takes the transform's type if
the types differ.
+ */
private static void addField(IRowMeta row, IValueMeta field, String origin)
throws HopPluginException {
String fieldName = field.getName();
- if (fieldName == null || fieldName.isEmpty() ||
row.indexOfValue(fieldName) >= 0) {
+ if (fieldName == null || fieldName.isEmpty()) {
return;
}
field.setOrigin(origin);
- row.addValueMeta(field);
+ int index = row.indexOfValue(fieldName);
+ if (index < 0) {
+ row.addValueMeta(field);
+ } else if (row.getValueMeta(index).getType() != field.getType()) {
+ row.setValueMeta(index, field);
+ }
+ }
+
+ /**
+ * Parses a resolved chunk size or overlap.
+ *
+ * @param value the resolved value
+ * @param emptyValue the value to use when nothing is configured
+ * @return the number, or {@code null} when the value is not a whole number
+ */
+ static Integer parseNumber(String value, int emptyValue) {
+ if (Utils.isEmpty(value) || value.isBlank()) {
Review Comment:
Nit: `Utils.isEmpty(value) || value.isBlank()` is redundant; `value == null
|| value.isBlank()` covers both.
##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunker.java:
##########
@@ -67,8 +66,21 @@ public boolean init() {
if (!super.init()) {
return false;
}
- data.chunkSize = Const.toInt(resolve(meta.getChunkSize()), -1);
- data.chunkOverlap = Const.toInt(resolve(meta.getChunkOverlap()), -1);
+ String sizeText = resolve(meta.getChunkSize());
+ String overlapText = resolve(meta.getChunkOverlap());
+ Integer size = TextChunkerMeta.parseNumber(sizeText, 0);
+ Integer overlap = TextChunkerMeta.parseNumber(overlapText, 0);
+ if (size == null) {
+ logError(BaseMessages.getString(PKG,
"TextChunker.Validation.ChunkSizeNotANumber", sizeText));
+ return false;
+ }
+ if (overlap == null) {
+ logError(
+ BaseMessages.getString(PKG,
"TextChunker.Validation.OverlapNotANumber", overlapText));
+ return false;
+ }
+ data.chunkSize = size;
+ data.chunkOverlap = overlap;
Review Comment:
A negative overlap still silently becomes 0 a few lines down (line 90-92),
while `check()` now reports it as an error. So `abc` fails in init, but `-5`
(or `${OVERLAP}` resolving to -5) runs with no overlap and no error, which is
the behaviour the PR title says it fixes. Could a negative overlap also log an
error and return false here?
##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java:
##########
@@ -430,6 +466,38 @@ public void check(
}
}
+ /** Warns when an output field already exists in the input with a different
type. */
+ private void checkOutputFieldTypes(
+ List<ICheckResult> remarks, TransformMeta transformMeta, IRowMeta prev) {
+ if (prev == null) {
+ return;
+ }
+ IRowMeta output = prev.clone();
+ try {
+ getFields(output, transformMeta.getName(), null, null, null, null);
+ } catch (HopTransformException e) {
Review Comment:
If `getFields` throws here, the method returns without adding a remark, so a
broken output field configuration gives no check result at all and only fails
at runtime. Could this add an error remark with the exception message instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]