hansva commented on code in PR #9367:
URL: https://github.com/apache/hop/pull/9367#discussion_r4537467763


##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java:
##########
@@ -326,14 +325,41 @@ public void getFields(
     }
   }
 
+  /**
+   * Adds an output field. When the input already has a field with that name 
the transform writes
+   * its values into that field, so the field takes the transform's type if 
the types differ.
+   */
   private static void addField(IRowMeta row, IValueMeta field, String origin)
       throws HopPluginException {
     String fieldName = field.getName();
-    if (fieldName == null || fieldName.isEmpty() || 
row.indexOfValue(fieldName) >= 0) {
+    if (fieldName == null || fieldName.isEmpty()) {
       return;
     }
     field.setOrigin(origin);
-    row.addValueMeta(field);
+    int index = row.indexOfValue(fieldName);
+    if (index < 0) {
+      row.addValueMeta(field);
+    } else if (row.getValueMeta(index).getType() != field.getType()) {
+      row.setValueMeta(index, field);

Review Comment:
   `indexOfValue` matches without regard to case, so an input field 
`Chunk_Index` is replaced by an Integer named `chunk_index`. Later transforms 
that look up `Chunk_Index` case-sensitively (JSON, Avro writers) lose the 
field, with no warning. Maybe only replace on an exact name match, or keep the 
existing name?



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java:
##########
@@ -430,6 +466,38 @@ public void check(
     }
   }
 
+  /** Warns when an output field already exists in the input with a different 
type. */
+  private void checkOutputFieldTypes(
+      List<ICheckResult> remarks, TransformMeta transformMeta, IRowMeta prev) {
+    if (prev == null) {
+      return;
+    }
+    IRowMeta output = prev.clone();
+    try {
+      getFields(output, transformMeta.getName(), null, null, null, null);
+    } catch (HopTransformException e) {
+      return;
+    }
+    for (String fieldName : getOutputFieldNames()) {

Review Comment:
   `getOutputFieldNames()` can contain the same name twice (e.g. index and 
count field both set to `n`), which adds the same 'Output field already exists' 
warning twice. A `LinkedHashSet` would avoid that.



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java:
##########
@@ -326,14 +325,41 @@ public void getFields(
     }
   }
 
+  /**
+   * Adds an output field. When the input already has a field with that name 
the transform writes
+   * its values into that field, so the field takes the transform's type if 
the types differ.
+   */
   private static void addField(IRowMeta row, IValueMeta field, String origin)
       throws HopPluginException {
     String fieldName = field.getName();
-    if (fieldName == null || fieldName.isEmpty() || 
row.indexOfValue(fieldName) >= 0) {
+    if (fieldName == null || fieldName.isEmpty()) {
       return;
     }
     field.setOrigin(origin);
-    row.addValueMeta(field);
+    int index = row.indexOfValue(fieldName);
+    if (index < 0) {
+      row.addValueMeta(field);
+    } else if (row.getValueMeta(index).getType() != field.getType()) {

Review Comment:
   Only the type is compared, so an input field of the same type keeps its 
storage type, conversion mask and length while the transform writes a plain 
String or Long into it. With lazy conversion (e.g. a CSV input with a String 
`chunk_text` in BINARY_STRING storage) the row metadata says `byte[]` but the 
value is a `java.lang.String`, and a later `getString()`/compare throws 
ClassCastException. The new test `anExistingFieldOfTheSameTypeIsKept` locks 
that in. I think the output field's metadata should always replace the existing 
one (or at least when storage type/mask/length differ).



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunker.java:
##########
@@ -67,8 +66,21 @@ public boolean init() {
     if (!super.init()) {
       return false;
     }
-    data.chunkSize = Const.toInt(resolve(meta.getChunkSize()), -1);
-    data.chunkOverlap = Const.toInt(resolve(meta.getChunkOverlap()), -1);
+    String sizeText = resolve(meta.getChunkSize());

Review Comment:
   This parses and validates size and overlap again, next to the same rules in 
`TextChunkerMeta.check()` (lines 439-458), and the two already disagree on a 
negative overlap (error in check, clamped here). One shared helper on the Meta 
that both call would keep them in sync.



##########
integration-tests/transforms/main-0126-text-chunker-invalid-overlap.hwf:
##########
@@ -0,0 +1,126 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<workflow>
+  <name>main-0126-text-chunker-invalid-overlap</name>
+  <name_sync_with_filename>Y</name_sync_with_filename>
+  <description>Text chunker: a chunk overlap that is not a number stops the 
transform instead of running with overlap 0 (#9285)</description>
+  <extended_description/>
+  <workflow_version/>
+  <created_user>-</created_user>
+  <created_date>2026/11/07 12:00:00.000</created_date>
+  <modified_user>-</modified_user>
+  <modified_date>2026/11/07 12:00:00.000</modified_date>
+  <parameters>
+    </parameters>
+  <actions>
+    <action>
+      <name>START</name>
+      <description/>
+      <type>SPECIAL</type>
+      <attributes/>
+      <repeat>N</repeat>
+      <schedulerType>0</schedulerType>
+      <intervalSeconds>0</intervalSeconds>
+      <intervalMinutes>60</intervalMinutes>
+      <hour>12</hour>
+      <minutes>0</minutes>
+      <weekDay>1</weekDay>
+      <DayOfMonth>1</DayOfMonth>
+      <parallel>N</parallel>
+      <xloc>96</xloc>
+      <yloc>80</yloc>
+      <attributes_hac/>
+    </action>
+    <action>
+      
<filename>${PROJECT_HOME}/0126-text-chunker-invalid-overlap.hpl</filename>
+      <params_from_previous>N</params_from_previous>
+      <exec_per_row>N</exec_per_row>
+      <clear_rows>N</clear_rows>
+      <clear_files>N</clear_files>
+      <create_parent_folder>N</create_parent_folder>
+      <set_logfile>N</set_logfile>
+      <set_append_logfile>N</set_append_logfile>
+      <add_date>N</add_date>
+      <add_time>N</add_time>
+      <loglevel>Basic</loglevel>
+      <wait_until_finished>Y</wait_until_finished>
+      <parameters>
+        <pass_all_parameters>Y</pass_all_parameters>
+      </parameters>
+      <run_configuration>local</run_configuration>
+      <name>0126-text-chunker-invalid-overlap.hpl</name>
+      <description>Expected to fail</description>
+      <type>PIPELINE</type>
+      <attributes/>
+      <xloc>320</xloc>
+      <yloc>80</yloc>
+      <parallel>N</parallel>
+      <attributes_hac/>
+    </action>
+    <action>
+      <name>Overlap 'abc' did not fail - Abort</name>
+      <description/>
+      <type>ABORT</type>
+      <attributes/>
+      <message>A chunk overlap of 'abc' did not stop the Text chunker</message>
+      <loglevel>Basic</loglevel>
+      <parallel>N</parallel>
+      <xloc>320</xloc>
+      <yloc>224</yloc>
+      <attributes_hac/>
+    </action>
+    <action>
+      <name>Success</name>
+      <description/>
+      <type>SUCCESS</type>
+      <attributes/>
+      <parallel>N</parallel>
+      <xloc>544</xloc>
+      <yloc>80</yloc>
+      <attributes_hac/>
+    </action>
+  </actions>
+  <hops>
+    <hop>
+      <from>START</from>
+      <to>0126-text-chunker-invalid-overlap.hpl</to>
+      <enabled>Y</enabled>
+      <evaluation>Y</evaluation>
+      <unconditional>Y</unconditional>
+    </hop>
+    <hop>
+      <from>0126-text-chunker-invalid-overlap.hpl</from>
+      <to>Overlap 'abc' did not fail - Abort</to>
+      <enabled>Y</enabled>
+      <evaluation>Y</evaluation>
+      <unconditional>N</unconditional>
+    </hop>
+    <hop>
+      <from>0126-text-chunker-invalid-overlap.hpl</from>

Review Comment:
   This passes whenever the child pipeline fails, for any reason (missing 
input, a broken transform...), so it doesn't prove the overlap is what makes it 
fail. Checking the error/log text would make it a real regression test for the 
overlap check.



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java:
##########
@@ -326,14 +325,41 @@ public void getFields(
     }
   }
 
+  /**
+   * Adds an output field. When the input already has a field with that name 
the transform writes
+   * its values into that field, so the field takes the transform's type if 
the types differ.
+   */
   private static void addField(IRowMeta row, IValueMeta field, String origin)
       throws HopPluginException {
     String fieldName = field.getName();
-    if (fieldName == null || fieldName.isEmpty() || 
row.indexOfValue(fieldName) >= 0) {
+    if (fieldName == null || fieldName.isEmpty()) {
       return;
     }
     field.setOrigin(origin);
-    row.addValueMeta(field);
+    int index = row.indexOfValue(fieldName);
+    if (index < 0) {
+      row.addValueMeta(field);
+    } else if (row.getValueMeta(index).getType() != field.getType()) {
+      row.setValueMeta(index, field);
+    }
+  }
+
+  /**
+   * Parses a resolved chunk size or overlap.
+   *
+   * @param value the resolved value
+   * @param emptyValue the value to use when nothing is configured
+   * @return the number, or {@code null} when the value is not a whole number
+   */
+  static Integer parseNumber(String value, int emptyValue) {
+    if (Utils.isEmpty(value) || value.isBlank()) {

Review Comment:
   Nit: `Utils.isEmpty(value) || value.isBlank()` is redundant; `value == null 
|| value.isBlank()` covers both.



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunker.java:
##########
@@ -67,8 +66,21 @@ public boolean init() {
     if (!super.init()) {
       return false;
     }
-    data.chunkSize = Const.toInt(resolve(meta.getChunkSize()), -1);
-    data.chunkOverlap = Const.toInt(resolve(meta.getChunkOverlap()), -1);
+    String sizeText = resolve(meta.getChunkSize());
+    String overlapText = resolve(meta.getChunkOverlap());
+    Integer size = TextChunkerMeta.parseNumber(sizeText, 0);
+    Integer overlap = TextChunkerMeta.parseNumber(overlapText, 0);
+    if (size == null) {
+      logError(BaseMessages.getString(PKG, 
"TextChunker.Validation.ChunkSizeNotANumber", sizeText));
+      return false;
+    }
+    if (overlap == null) {
+      logError(
+          BaseMessages.getString(PKG, 
"TextChunker.Validation.OverlapNotANumber", overlapText));
+      return false;
+    }
+    data.chunkSize = size;
+    data.chunkOverlap = overlap;

Review Comment:
   A negative overlap still silently becomes 0 a few lines down (line 90-92), 
while `check()` now reports it as an error. So `abc` fails in init, but `-5` 
(or `${OVERLAP}` resolving to -5) runs with no overlap and no error, which is 
the behaviour the PR title says it fixes. Could a negative overlap also log an 
error and return false here?



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java:
##########
@@ -430,6 +466,38 @@ public void check(
     }
   }
 
+  /** Warns when an output field already exists in the input with a different 
type. */
+  private void checkOutputFieldTypes(
+      List<ICheckResult> remarks, TransformMeta transformMeta, IRowMeta prev) {
+    if (prev == null) {
+      return;
+    }
+    IRowMeta output = prev.clone();
+    try {
+      getFields(output, transformMeta.getName(), null, null, null, null);
+    } catch (HopTransformException e) {

Review Comment:
   If `getFields` throws here, the method returns without adding a remark, so a 
broken output field configuration gives no check result at all and only fails 
at runtime. Could this add an error remark with the exception message instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to