This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 0e8ee5589e Text chunker: overlap cuts words, can produce a chunk per
character, wrong positions in split Structure sections, whitespace-only chunks
and split surrogate pairs, fixes #9283 (#9364)
0e8ee5589e is described below
commit 0e8ee5589e1d8105571ae738be12252ced7eb862
Author: Bart Maertens <[email protected]>
AuthorDate: Wed Nov 11 12:39:57 2026 +0100
Text chunker: overlap cuts words, can produce a chunk per character, wrong
positions in split Structure sections, whitespace-only chunks and split
surrogate pairs, fixes #9283 (#9364)
---
.../pages/pipeline/transforms/textchunker.adoc | 10 +-
...hpl => 0115-text-chunker-character-overlap.hpl} | 24 ++--
.../0115-text-chunker-paragraph-separator.hpl | 3 +
...r.hpl => 0115-text-chunker-structure-split.hpl} | 18 +--
.../golden-text-chunker-character-overlap.csv | 14 +++
.../datasets/golden-text-chunker-character.csv | 4 +-
.../golden-text-chunker-paragraph-separator.csv | 2 +
.../golden-text-chunker-structure-split.csv | 16 +++
.../transforms/main-0115-text-chunker.hwf | 6 +
.../golden-text-chunker-character-overlap.json | 39 ++++++
.../golden-text-chunker-structure-split.json | 39 ++++++
.../0115-text-chunker-character-overlap UNIT.json | 43 +++++++
.../0115-text-chunker-structure-split UNIT.json | 43 +++++++
.../transforms/chunker/TextChunkerMeta.java | 15 ++-
.../chunking/CharacterChunkingStrategy.java | 116 +++++++++++++----
.../chunking/ParagraphChunkingStrategy.java | 10 +-
.../chunking/StructureChunkingStrategy.java | 15 ++-
.../transforms/chunker/document/DocumentNode.java | 49 +++++++-
.../chunker/document/HeadingParserSupport.java | 20 ++-
.../chunker/messages/messages_en_US.properties | 4 +-
.../transforms/chunker/TextChunkerMetaTest.java | 33 +++++
.../chunking/CharacterChunkingStrategyTest.java | 140 +++++++++++++++++++--
.../chunking/ParagraphChunkingStrategyTest.java | 58 +++++++++
.../chunking/StructureChunkingStrategyTest.java | 76 +++++++++++
24 files changed, 725 insertions(+), 72 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textchunker.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textchunker.adoc
index cb16bf1a64..e329402eca 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textchunker.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/textchunker.adoc
@@ -36,7 +36,7 @@ The transform has no dependency on an AI provider or a model.
It is plain text p
|Strategy |Behaviour
|Character
-|Split on a fixed character count, backing off to the nearest word boundary so
words are not cut in half. Overlap carries the tail of each chunk into the
next, which keeps a sentence that straddles a boundary retrievable from both
sides.
+|Split on a fixed character count, backing off to the nearest word boundary so
words are not cut in half. Only a word longer than the chunk size is cut, and
never inside a surrogate pair such as an emoji. Overlap carries the last whole
words of each chunk into the next, which keeps a sentence that straddles a
boundary retrievable from both sides.
|Paragraph
|Split on paragraph breaks, packing whole paragraphs up to the chunk size. A
paragraph longer than the chunk size falls back to character splitting. By
default a paragraph break is a blank line (any `\LF`, `\CR` or `\CRLF` line
ending); the optional *Paragraph separator* option replaces it with a literal
separator.
@@ -79,7 +79,7 @@ The Hop-native parsers make a pipeline, a workflow or a
metadata file retrievabl
|Content type|Document format used by the Structure strategy.
|Content type field|Optional field naming the content type per row, for a
stream that mixes formats.
|Chunk size|Maximum chunk size in characters. For Paragraph and Structure this
is a ceiling that triggers the character fallback.
-|Chunk overlap|Characters carried from the end of one chunk into the start of
the next. Character strategy only.
+|Chunk overlap|Maximum number of characters carried from the end of one chunk
into the start of the next. The overlap starts at a word boundary, so it holds
whole words and can be shorter than this value, or empty when no word starts in
that span. Paragraph and Structure only apply it inside a paragraph or section
that is split with the character fallback.
|Include metadata|Add the chunk metadata fields below to the output.
|Chunk index field|Output field for the position of the chunk within its
document.
|Chunk start position field|Output field for the character offset of the chunk
in the source text.
@@ -89,8 +89,10 @@ The Hop-native parsers make a pipeline, a workflow or a
metadata file retrievabl
== Notes
-Chunk overlap applies to the Character strategy. The Paragraph and Structure
strategies split on natural boundaries, where an overlap would duplicate whole
paragraphs or sections between neighbouring chunks.
+Chunk overlap never crosses a paragraph or section boundary. The Paragraph and
Structure strategies split on natural boundaries, where an overlap would
duplicate whole paragraphs or sections between neighbouring chunks. They only
apply it between the pieces of a paragraph or section that is longer than the
chunk size. Every chunk adds text the previous chunk did not hold: when an
overlap would only repeat the previous chunk, the next chunk starts where the
previous one ended.
-The Structure strategy drops chunks that contain only whitespace. The chunk
index and total chunk count cover only the chunks that are emitted. Text that
yields no chunks produces a single row with an empty chunk and a total of 0,
the same as empty text.
+The chunk start position is the character offset in the input text. For a
section of the Structure strategy, the first chunk starts at the section
heading and the chunks a long section is split into start where their own text
is.
+
+All strategies drop chunks that contain only whitespace. The chunk index and
total chunk count cover only the chunks that are emitted. Text that yields no
chunks produces a single row with an empty chunk and a total of 0, the same as
empty text.
Keep *Include metadata* enabled when the chunks are going into a vector store:
the document ID and chunk index are what let a later search merge neighbouring
chunks back into a passage, and re-index a single document without touching the
rest.
diff --git
a/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
b/integration-tests/transforms/0115-text-chunker-character-overlap.hpl
similarity index 79%
copy from integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
copy to integration-tests/transforms/0115-text-chunker-character-overlap.hpl
index e880461384..62000c8dd2 100644
--- a/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
+++ b/integration-tests/transforms/0115-text-chunker-character-overlap.hpl
@@ -19,9 +19,9 @@ limitations under the License.
-->
<pipeline>
<info>
- <name>0115-text-chunker-paragraph-separator</name>
+ <name>0115-text-chunker-character-overlap</name>
<name_sync_with_filename>Y</name_sync_with_filename>
- <description>Paragraph chunking with a custom paragraph separator instead
of blank lines.</description>
+ <description>Character overlap starts at a word, never repeats a whole
chunk, drops whitespace-only chunks and keeps surrogate pairs together
(#9283).</description>
<extended_description/>
<pipeline_version/>
<pipeline_type>Normal</pipeline_type>
@@ -32,9 +32,9 @@ limitations under the License.
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
<created_user>-</created_user>
- <created_date>2026/10/03 12:00:00.000</created_date>
+ <created_date>2026/09/16 12:00:00.000</created_date>
<modified_user>-</modified_user>
- <modified_date>2026/10/03 12:00:00.000</modified_date>
+ <modified_date>2026/09/16 12:00:00.000</modified_date>
<key_for_session_key/>
<is_key_private>N</is_key_private>
</info>
@@ -57,7 +57,16 @@ limitations under the License.
</partitioning>
<data>
<line>
- <item>First part.<PARA>Second part.<PARA>Third part.</item>
+ <item>supercalifragilistic supercalifragilistic supercalifragilistic
supercalifragilistic supercalifragilistic</item>
+ </line>
+ <line>
+ <item>Apache Hop is a data orchestration platform for building
pipelines</item>
+ </line>
+ <line>
+ <item>a
b</item>
+ </line>
+ <line>
+ <item>😀😀😀😀😀😀😀😀😀😀😀😀😀</item>
</line>
</data>
<fields>
@@ -88,10 +97,9 @@ limitations under the License.
</partitioning>
<inputField>content</inputField>
<outputChunkField>chunk_text</outputChunkField>
- <chunkingStrategy>PARAGRAPH</chunkingStrategy>
- <paragraphSeparator><PARA></paragraphSeparator>
+ <chunkingStrategy>CHARACTER</chunkingStrategy>
<chunkSize>25</chunkSize>
- <chunkOverlap>0</chunkOverlap>
+ <chunkOverlap>15</chunkOverlap>
<includeMetadata>Y</includeMetadata>
<chunkIndexField>chunk_index</chunkIndexField>
<chunkStartPosField>chunk_start_position</chunkStartPosField>
diff --git
a/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
b/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
index e880461384..8ec6bcbf07 100644
--- a/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
+++ b/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
@@ -59,6 +59,9 @@ limitations under the License.
<line>
<item>First part.<PARA>Second part.<PARA>Third part.</item>
</line>
+ <line>
+ <item>one<PARA> <PARA>two</item>
+ </line>
</data>
<fields>
<field>
diff --git
a/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
b/integration-tests/transforms/0115-text-chunker-structure-split.hpl
similarity index 79%
copy from integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
copy to integration-tests/transforms/0115-text-chunker-structure-split.hpl
index e880461384..465f223eb7 100644
--- a/integration-tests/transforms/0115-text-chunker-paragraph-separator.hpl
+++ b/integration-tests/transforms/0115-text-chunker-structure-split.hpl
@@ -19,9 +19,9 @@ limitations under the License.
-->
<pipeline>
<info>
- <name>0115-text-chunker-paragraph-separator</name>
+ <name>0115-text-chunker-structure-split</name>
<name_sync_with_filename>Y</name_sync_with_filename>
- <description>Paragraph chunking with a custom paragraph separator instead
of blank lines.</description>
+ <description>A Structure section longer than the chunk size: the chunks
after the first start where their text is, after the heading, and the overlap
starts at a word (#9283).</description>
<extended_description/>
<pipeline_version/>
<pipeline_type>Normal</pipeline_type>
@@ -32,9 +32,9 @@ limitations under the License.
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
<created_user>-</created_user>
- <created_date>2026/10/03 12:00:00.000</created_date>
+ <created_date>2026/09/16 12:00:00.000</created_date>
<modified_user>-</modified_user>
- <modified_date>2026/10/03 12:00:00.000</modified_date>
+ <modified_date>2026/09/16 12:00:00.000</modified_date>
<key_for_session_key/>
<is_key_private>N</is_key_private>
</info>
@@ -57,7 +57,7 @@ limitations under the License.
</partitioning>
<data>
<line>
- <item>First part.<PARA>Second part.<PARA>Third part.</item>
+ <item># Title word001 word002 word003 word004 word005 word006
word007 word008 word009 word010 word011 word012 word013 word014 word015 word016
word017 word018 word019 word020 word021 word022 word023 word024 word025 word026
word027 word028 word029 word030 word031 word032 word033 word034 word035 word036
word037 word038 word039 word040</item>
</line>
</data>
<fields>
@@ -88,10 +88,10 @@ limitations under the License.
</partitioning>
<inputField>content</inputField>
<outputChunkField>chunk_text</outputChunkField>
- <chunkingStrategy>PARAGRAPH</chunkingStrategy>
- <paragraphSeparator><PARA></paragraphSeparator>
- <chunkSize>25</chunkSize>
- <chunkOverlap>0</chunkOverlap>
+ <chunkingStrategy>STRUCTURE</chunkingStrategy>
+ <contentType>MARKDOWN</contentType>
+ <chunkSize>100</chunkSize>
+ <chunkOverlap>20</chunkOverlap>
<includeMetadata>Y</includeMetadata>
<chunkIndexField>chunk_index</chunkIndexField>
<chunkStartPosField>chunk_start_position</chunkStartPosField>
diff --git
a/integration-tests/transforms/datasets/golden-text-chunker-character-overlap.csv
b/integration-tests/transforms/datasets/golden-text-chunker-character-overlap.csv
new file mode 100644
index 0000000000..c0851d54cb
--- /dev/null
+++
b/integration-tests/transforms/datasets/golden-text-chunker-character-overlap.csv
@@ -0,0 +1,14 @@
+chunk_text,chunk_index,chunk_start_position,total_chunks
+"supercalifragilistic ",0,0,5
+"supercalifragilistic ",1,21,5
+"supercalifragilistic ",2,42,5
+"supercalifragilistic ",3,63,5
+"supercalifragilistic",4,84,5
+"Apache Hop is a data ",0,0,4
+"orchestration platform ",1,21,4
+"platform for building ",2,35,4
+"for building pipelines",3,44,4
+"a ",0,0,2
+" b",1,50,2
+"😀😀😀😀😀😀😀😀😀😀😀😀",0,0,2
+"😀😀😀😀😀😀😀😀😀",1,8,2
diff --git
a/integration-tests/transforms/datasets/golden-text-chunker-character.csv
b/integration-tests/transforms/datasets/golden-text-chunker-character.csv
index c979ed665d..c0bbd313f8 100644
--- a/integration-tests/transforms/datasets/golden-text-chunker-character.csv
+++ b/integration-tests/transforms/datasets/golden-text-chunker-character.csv
@@ -1,4 +1,4 @@
chunk_text,chunk_index,chunk_start_position,total_chunks
"Apache Hop is a data orchestration ",0,0,3
-"estration platform for building ",1,25,3
-" building pipelines and workflows.",2,47,3
+"platform for building pipelines and ",1,35,3
+"and workflows.",2,67,3
diff --git
a/integration-tests/transforms/datasets/golden-text-chunker-paragraph-separator.csv
b/integration-tests/transforms/datasets/golden-text-chunker-paragraph-separator.csv
index 5fd8616245..2a423cbb83 100644
---
a/integration-tests/transforms/datasets/golden-text-chunker-paragraph-separator.csv
+++
b/integration-tests/transforms/datasets/golden-text-chunker-paragraph-separator.csv
@@ -2,3 +2,5 @@ chunk_text,chunk_index,chunk_start_position,total_chunks
"First part.",0,0,3
"Second part.",1,17,3
"Third part.",2,35,3
+"one",0,0,2
+"two",1,35,2
diff --git
a/integration-tests/transforms/datasets/golden-text-chunker-structure-split.csv
b/integration-tests/transforms/datasets/golden-text-chunker-structure-split.csv
new file mode 100644
index 0000000000..fcc170f128
--- /dev/null
+++
b/integration-tests/transforms/datasets/golden-text-chunker-structure-split.csv
@@ -0,0 +1,16 @@
+chunk_text,chunk_index,chunk_start_position,total_chunks
+"Title
+
+word001 word002 word003 word004 word005 word006 word007 word008 word009
word010 word011 ",0,0,5
+"Title
+
+word010 word011 word012 word013 word014 word015 word016 word017 word018
word019 word020 ",1,81,5
+"Title
+
+word019 word020 word021 word022 word023 word024 word025 word026 word027
word028 word029 ",2,153,5
+"Title
+
+word028 word029 word030 word031 word032 word033 word034 word035 word036
word037 word038 ",3,225,5
+"Title
+
+word037 word038 word039 word040",4,297,5
diff --git a/integration-tests/transforms/main-0115-text-chunker.hwf
b/integration-tests/transforms/main-0115-text-chunker.hwf
index 00d468a3e4..e1c26aaac9 100644
--- a/integration-tests/transforms/main-0115-text-chunker.hwf
+++ b/integration-tests/transforms/main-0115-text-chunker.hwf
@@ -66,6 +66,12 @@ limitations under the License.
<test_name>
<name>0115-text-chunker-structure-markdown UNIT</name>
</test_name>
+ <test_name>
+ <name>0115-text-chunker-character-overlap UNIT</name>
+ </test_name>
+ <test_name>
+ <name>0115-text-chunker-structure-split UNIT</name>
+ </test_name>
<test_name>
<name>0117-text-chunker-redaction UNIT</name>
</test_name>
diff --git
a/integration-tests/transforms/metadata/dataset/golden-text-chunker-character-overlap.json
b/integration-tests/transforms/metadata/dataset/golden-text-chunker-character-overlap.json
new file mode 100644
index 0000000000..4e44e2cd1d
--- /dev/null
+++
b/integration-tests/transforms/metadata/dataset/golden-text-chunker-character-overlap.json
@@ -0,0 +1,39 @@
+{
+ "base_filename": "golden-text-chunker-character-overlap.csv",
+ "name": "golden-text-chunker-character-overlap",
+ "description": "Character overlap on word boundaries, no repeated chunks, no
whitespace-only chunks, surrogate pairs kept together",
+ "dataset_fields": [
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_name": "chunk_text",
+ "field_format": ""
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_name": "chunk_index",
+ "field_format": "0"
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_name": "chunk_start_position",
+ "field_format": "0"
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_name": "total_chunks",
+ "field_format": "0"
+ }
+ ]
+}
diff --git
a/integration-tests/transforms/metadata/dataset/golden-text-chunker-structure-split.json
b/integration-tests/transforms/metadata/dataset/golden-text-chunker-structure-split.json
new file mode 100644
index 0000000000..efb56b588c
--- /dev/null
+++
b/integration-tests/transforms/metadata/dataset/golden-text-chunker-structure-split.json
@@ -0,0 +1,39 @@
+{
+ "base_filename": "golden-text-chunker-structure-split.csv",
+ "name": "golden-text-chunker-structure-split",
+ "description": "Structure: positions and word-aligned overlap of a section
split with the character fallback",
+ "dataset_fields": [
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 2,
+ "field_precision": -1,
+ "field_name": "chunk_text",
+ "field_format": ""
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_name": "chunk_index",
+ "field_format": "0"
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_name": "chunk_start_position",
+ "field_format": "0"
+ },
+ {
+ "field_comment": "",
+ "field_length": -1,
+ "field_type": 5,
+ "field_precision": 0,
+ "field_name": "total_chunks",
+ "field_format": "0"
+ }
+ ]
+}
diff --git
a/integration-tests/transforms/metadata/unit-test/0115-text-chunker-character-overlap
UNIT.json
b/integration-tests/transforms/metadata/unit-test/0115-text-chunker-character-overlap
UNIT.json
new file mode 100644
index 0000000000..59039144d8
--- /dev/null
+++
b/integration-tests/transforms/metadata/unit-test/0115-text-chunker-character-overlap
UNIT.json
@@ -0,0 +1,43 @@
+{
+ "variableValues": [],
+ "database_replacements": [],
+ "autoOpening": false,
+ "basePath": "",
+ "golden_data_sets": [
+ {
+ "field_mappings": [
+ {
+ "transform_field": "chunk_text",
+ "data_set_field": "chunk_text"
+ },
+ {
+ "transform_field": "chunk_index",
+ "data_set_field": "chunk_index"
+ },
+ {
+ "transform_field": "chunk_start_position",
+ "data_set_field": "chunk_start_position"
+ },
+ {
+ "transform_field": "total_chunks",
+ "data_set_field": "total_chunks"
+ }
+ ],
+ "field_order": [
+ "chunk_text",
+ "chunk_index",
+ "chunk_start_position",
+ "total_chunks"
+ ],
+ "transform_name": "OUTPUT",
+ "data_set_name": "golden-text-chunker-character-overlap"
+ }
+ ],
+ "input_data_sets": [],
+ "name": "0115-text-chunker-character-overlap UNIT",
+ "description": "Character overlap on word boundaries, no repeated chunks, no
whitespace-only chunks, surrogate pairs kept together",
+ "trans_test_tweaks": [],
+ "persist_filename": "",
+ "pipeline_filename": "./0115-text-chunker-character-overlap.hpl",
+ "test_type": "UNIT_TEST"
+}
diff --git
a/integration-tests/transforms/metadata/unit-test/0115-text-chunker-structure-split
UNIT.json
b/integration-tests/transforms/metadata/unit-test/0115-text-chunker-structure-split
UNIT.json
new file mode 100644
index 0000000000..a5eeedfc22
--- /dev/null
+++
b/integration-tests/transforms/metadata/unit-test/0115-text-chunker-structure-split
UNIT.json
@@ -0,0 +1,43 @@
+{
+ "variableValues": [],
+ "database_replacements": [],
+ "autoOpening": false,
+ "basePath": "",
+ "golden_data_sets": [
+ {
+ "field_mappings": [
+ {
+ "transform_field": "chunk_text",
+ "data_set_field": "chunk_text"
+ },
+ {
+ "transform_field": "chunk_index",
+ "data_set_field": "chunk_index"
+ },
+ {
+ "transform_field": "chunk_start_position",
+ "data_set_field": "chunk_start_position"
+ },
+ {
+ "transform_field": "total_chunks",
+ "data_set_field": "total_chunks"
+ }
+ ],
+ "field_order": [
+ "chunk_text",
+ "chunk_index",
+ "chunk_start_position",
+ "total_chunks"
+ ],
+ "transform_name": "OUTPUT",
+ "data_set_name": "golden-text-chunker-structure-split"
+ }
+ ],
+ "input_data_sets": [],
+ "name": "0115-text-chunker-structure-split UNIT",
+ "description": "Structure: positions and word-aligned overlap of a section
split with the character fallback",
+ "trans_test_tweaks": [],
+ "persist_filename": "",
+ "pipeline_filename": "./0115-text-chunker-structure-split.hpl",
+ "test_type": "UNIT_TEST"
+}
diff --git
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java
index addbf66392..4056aaf175 100644
---
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java
+++
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMeta.java
@@ -147,7 +147,10 @@ public class TextChunkerMeta extends
BaseTransformMeta<TextChunker, TextChunkerD
@HopMetadataProperty(key = "chunkSize", injectionKey = "CHUNK_SIZE")
private String chunkSize = "1000";
- /** The number of characters to overlap between chunks (for CHARACTER
strategy). */
+ /**
+ * The maximum number of characters, in whole words, to overlap between
chunks. Paragraph and
+ * Structure only apply it inside a paragraph or section they split with the
character fallback.
+ */
@GuiWidgetElement(
order = "0800",
type = GuiElementType.TEXT,
@@ -421,8 +424,14 @@ public class TextChunkerMeta extends
BaseTransformMeta<TextChunker, TextChunkerD
warning(remarks, transformMeta, "TextChunker.Validation.OverlapWarning");
}
- if (chunkingStrategy == ChunkingStrategyType.PARAGRAPH && resolvedOverlap
> 0) {
- warning(remarks, transformMeta,
"TextChunker.Validation.ParagraphOverlapIgnored");
+ if ((chunkingStrategy == ChunkingStrategyType.PARAGRAPH
+ || chunkingStrategy == ChunkingStrategyType.STRUCTURE)
+ && resolvedOverlap > 0) {
+ remarks.add(
+ new CheckResult(
+ ICheckResult.TYPE_RESULT_COMMENT,
+ BaseMessages.getString(PKG,
"TextChunker.Validation.OverlapOnlyWhenSplit"),
+ transformMeta));
}
if (!Utils.isEmpty(paragraphSeparator) && chunkingStrategy !=
ChunkingStrategyType.PARAGRAPH) {
diff --git
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategy.java
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategy.java
index 7351c93fd5..c32d4c5197 100644
---
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategy.java
+++
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategy.java
@@ -23,6 +23,11 @@ import org.apache.hop.pipeline.transforms.chunker.Chunk;
/**
* Chunking strategy that splits text on fixed character count while
respecting word boundaries.
* This ensures that chunks don't split words in the middle.
+ *
+ * <p>The overlap carries whole words, at most {@code overlap} characters,
from the end of one chunk
+ * into the next. Only a word longer than the chunk size, which has to be cut
anyway, is overlapped
+ * on characters. Every chunk ends past the end of the previous one, chunks
that hold only
+ * whitespace are dropped, and a surrogate pair is never cut in half.
*/
public class CharacterChunkingStrategy implements ChunkingStrategy {
@@ -35,50 +40,117 @@ public class CharacterChunkingStrategy implements
ChunkingStrategy {
}
if (overlap < 0) {
- chunks.add(new Chunk(text, 0, 0, text.length()));
+ if (!text.isBlank()) {
+ chunks.add(new Chunk(text, 0, 0, text.length()));
+ }
return chunks;
}
overlap = Math.min(overlap, maxSize - 1);
+ int textLength = text.length();
int start = 0;
+ int previousEnd = 0;
int chunkIndex = 0;
- int textLength = text.length();
while (start < textLength) {
- int end = Math.min(start + maxSize, textLength);
+ int chunkEnd = chunkEnd(text, start, maxSize);
- if (end == textLength) {
- String chunkContent = text.substring(start, end);
- chunks.add(new Chunk(chunkContent, chunkIndex++, start, end));
- break;
+ if (start < previousEnd && chunkEnd <= previousEnd) {
+ // The overlap would only repeat text the previous chunk already
holds: no new text would
+ // reach this chunk. Continue where the previous chunk ended instead.
+ start = previousEnd;
+ continue;
+ }
+
+ String chunkContent = text.substring(start, chunkEnd);
+ if (!chunkContent.isBlank()) {
+ chunks.add(new Chunk(chunkContent, chunkIndex++, start, chunkEnd));
}
- int separatorPos = -1;
- for (int i = end - 1; i >= start; i--) {
- if (Character.isWhitespace(text.charAt(i))) {
- separatorPos = i;
- break;
- }
+ if (chunkEnd >= textLength) {
+ break;
}
- int chunkEnd = separatorPos >= start ? separatorPos + 1 : end;
+ previousEnd = chunkEnd;
+ start = nextStart(text, start, chunkEnd, overlap);
+ }
+
+ return chunks;
+ }
+
+ /**
+ * The end of the chunk that starts at {@code start}: just after the last
whitespace within {@code
+ * maxSize} characters, or a hard cut when there is none, moved so it does
not split a surrogate
+ * pair.
+ */
+ private static int chunkEnd(String text, int start, int maxSize) {
+ int textLength = text.length();
+ int end = Math.min(start + maxSize, textLength);
+ if (end == textLength) {
+ return end;
+ }
- if (chunkEnd <= start) {
- chunkEnd = Math.min(start + maxSize, textLength);
+ for (int i = end - 1; i >= start; i--) {
+ if (Character.isWhitespace(text.charAt(i))) {
+ return i + 1;
}
+ }
- String chunkContent = text.substring(start, chunkEnd);
- chunks.add(new Chunk(chunkContent, chunkIndex++, start, chunkEnd));
+ if (splitsSurrogatePair(text, end)) {
+ // Keep the pair together: in this chunk if there is room for something
before it,
+ // otherwise in this chunk even though it then holds one character more
than the maximum.
+ return end - 1 > start ? end - 1 : end + 1;
+ }
+ return end;
+ }
- if (chunkEnd >= textLength) {
- break;
+ /**
+ * Where the chunk after {@code [start, chunkEnd)} starts: the first word
that begins in the last
+ * {@code overlap} characters of the chunk, or {@code chunkEnd} when no word
begins there. When
+ * the chunk was cut inside a word that is too long to back off from, the
overlap stays on
+ * characters, as no word boundary is available.
+ */
+ private static int nextStart(String text, int start, int chunkEnd, int
overlap) {
+ if (overlap <= 0) {
+ return chunkEnd;
+ }
+
+ int candidate = Math.max(start + 1, chunkEnd - overlap);
+
+ int wordStart = candidate;
+ if (!Character.isWhitespace(text.charAt(wordStart - 1))) {
+ // Inside a word: skip to its end.
+ while (wordStart < chunkEnd &&
!Character.isWhitespace(text.charAt(wordStart))) {
+ wordStart++;
}
+ }
+ while (wordStart < chunkEnd &&
Character.isWhitespace(text.charAt(wordStart))) {
+ wordStart++;
+ }
+ if (wordStart < chunkEnd) {
+ return wordStart;
+ }
- start = overlap > 0 ? Math.max(start + 1, chunkEnd - overlap) : chunkEnd;
+ boolean cutInsideWord =
+ chunkEnd < text.length()
+ && !Character.isWhitespace(text.charAt(chunkEnd - 1))
+ && !Character.isWhitespace(text.charAt(chunkEnd));
+ if (!cutInsideWord) {
+ return chunkEnd;
+ }
+ if (splitsSurrogatePair(text, candidate)) {
+ return candidate - 1 > start ? candidate - 1 : candidate + 1;
}
+ return candidate;
+ }
- return chunks;
+ /** True when {@code position} falls between the two halves of a surrogate
pair. */
+ private static boolean splitsSurrogatePair(String text, int position) {
+ return position > 0
+ && position < text.length()
+ && Character.isHighSurrogate(text.charAt(position - 1))
+ && Character.isLowSurrogate(text.charAt(position));
}
@Override
diff --git
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategy.java
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategy.java
index 9bb9b949cd..2747e08aeb 100644
---
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategy.java
+++
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategy.java
@@ -29,7 +29,7 @@ import org.apache.hop.pipeline.transforms.chunker.Chunk;
* #setSeparator(String)} when the text uses a custom multi-byte marker
instead. A paragraph that on
* its own exceeds {@code maxSize} is split further with {@link
CharacterChunkingStrategy}, because
* a chunk larger than the limit would be rejected downstream by the embedding
model rather than
- * simply being large.
+ * simply being large. The overlap applies only within that split, never
between paragraphs.
*
* <p>Consecutive paragraphs are packed into one chunk while they fit within
{@code maxSize}, which
* is the usual contract for an embedding chunker: it keeps related text
together and avoids
@@ -103,7 +103,7 @@ public class ParagraphChunkingStrategy implements
ChunkingStrategy {
public List<Chunk> chunk(String text, int maxSize, int overlap) {
List<Chunk> chunks = new ArrayList<>();
- if (text == null || text.isEmpty()) {
+ if (text == null || text.isBlank()) {
return chunks;
}
if (maxSize <= 0) {
@@ -183,8 +183,12 @@ public class ParagraphChunkingStrategy implements
ChunkingStrategy {
return paragraphs;
}
+ /**
+ * Whitespace-only paragraphs are left out, so they never become a chunk on
their own. Packing
+ * still slices the source text, so a chunk that spans one keeps it.
+ */
private static void addParagraph(List<Paragraph> paragraphs, String text,
int start, int end) {
- if (end > start) {
+ if (end > start && !text.substring(start, end).isBlank()) {
paragraphs.add(new Paragraph(text.substring(start, end), start));
}
}
diff --git
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategy.java
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategy.java
index 65347dff12..c5912173b8 100644
---
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategy.java
+++
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategy.java
@@ -70,24 +70,33 @@ public class StructureChunkingStrategy implements
ChunkingStrategy {
sectionText,
chunks.size(),
section.getStartPosition(),
- section.getStartPosition() + sectionText.length()));
+ section.getBodyStartPosition() + section.getBody().length()));
} else {
String breadcrumb = section.breadcrumb();
String prefix = breadcrumb.isEmpty() ? "" : breadcrumb + "\n\n";
String body = section.getBody();
int bodyMax = prefix.isEmpty() ? maxSize : Math.max(1, maxSize -
prefix.length());
List<Chunk> split = fallback.chunk(body, bodyMax, overlap);
+ boolean firstPart = true;
for (Chunk part : split) {
if (part.getContent().isBlank()) {
continue;
}
String content = prefix.isEmpty() ? part.getContent() : prefix +
part.getContent();
+ // The first part starts the section, heading included, like an
unsplit section does.
+ // The others start where their text is in the source: the body is
stripped and follows
+ // the heading, so their offsets are relative to the body, not to
the section.
+ int start =
+ firstPart
+ ? section.getStartPosition()
+ : section.getBodyStartPosition() + part.getStartPosition();
chunks.add(
new Chunk(
content,
chunks.size(),
- section.getStartPosition() + part.getStartPosition(),
- section.getStartPosition() + part.getEndPosition()));
+ start,
+ section.getBodyStartPosition() + part.getEndPosition()));
+ firstPart = false;
}
}
}
diff --git
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/DocumentNode.java
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/DocumentNode.java
index d1d826115d..8980d14a05 100644
---
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/DocumentNode.java
+++
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/DocumentNode.java
@@ -28,12 +28,31 @@ public final class DocumentNode {
private final String title;
private final String body;
private final int startPosition;
+ private final int bodyStartPosition;
private final List<DocumentNode> children;
+ /**
+ * A node whose body starts at {@code startPosition}: a document root, or a
body that is not a
+ * slice of the source text, such as a summary built by a Hop pipeline or
metadata parser.
+ */
public DocumentNode(String title, String body, int startPosition,
List<DocumentNode> children) {
+ this(title, body, startPosition, startPosition, children);
+ }
+
+ /**
+ * @param startPosition where the section starts in the source text, its
heading included
+ * @param bodyStartPosition where {@code body} starts in the source text
+ */
+ public DocumentNode(
+ String title,
+ String body,
+ int startPosition,
+ int bodyStartPosition,
+ List<DocumentNode> children) {
this.title = title != null ? title : "";
this.body = body != null ? body : "";
this.startPosition = Math.max(0, startPosition);
+ this.bodyStartPosition = Math.max(this.startPosition, bodyStartPosition);
this.children = children != null ? List.copyOf(children) : List.of();
}
@@ -57,6 +76,10 @@ public final class DocumentNode {
return startPosition;
}
+ public int getBodyStartPosition() {
+ return bodyStartPosition;
+ }
+
public List<DocumentNode> getChildren() {
return children;
}
@@ -76,7 +99,16 @@ public final class DocumentNode {
}
if (!body.isBlank()) {
- sections.add(new DocumentSection(List.copyOf(currentPath), body.strip(),
startPosition));
+ int leadingWhitespace = 0;
+ while (Character.isWhitespace(body.charAt(leadingWhitespace))) {
+ leadingWhitespace++;
+ }
+ sections.add(
+ new DocumentSection(
+ List.copyOf(currentPath),
+ body.strip(),
+ startPosition,
+ bodyStartPosition + leadingWhitespace));
}
for (DocumentNode child : children) {
@@ -89,11 +121,22 @@ public final class DocumentNode {
private final List<String> path;
private final String body;
private final int startPosition;
+ private final int bodyStartPosition;
public DocumentSection(List<String> path, String body, int startPosition) {
+ this(path, body, startPosition, startPosition);
+ }
+
+ /**
+ * @param startPosition where the section starts in the source text, its
heading included
+ * @param bodyStartPosition where {@code body} starts in the source text
+ */
+ public DocumentSection(
+ List<String> path, String body, int startPosition, int
bodyStartPosition) {
this.path = path != null ? List.copyOf(path) : List.of();
this.body = body != null ? body : "";
this.startPosition = startPosition;
+ this.bodyStartPosition = bodyStartPosition;
}
public List<String> getPath() {
@@ -108,6 +151,10 @@ public final class DocumentNode {
return startPosition;
}
+ public int getBodyStartPosition() {
+ return bodyStartPosition;
+ }
+
public String breadcrumb() {
return String.join(" > ", path);
}
diff --git
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/HeadingParserSupport.java
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/HeadingParserSupport.java
index 226af2d584..fa39b9815e 100644
---
a/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/HeadingParserSupport.java
+++
b/plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/HeadingParserSupport.java
@@ -48,7 +48,7 @@ final class HeadingParserSupport {
}
String preamble = text.substring(0, headings.get(0).start).strip();
- MutableNode root = new MutableNode("", preamble, 0);
+ MutableNode root = new MutableNode("", preamble, 0, skipWhitespace(text,
0));
MutableNode[] stack = new MutableNode[7];
stack[0] = root;
@@ -59,7 +59,8 @@ final class HeadingParserSupport {
String body = text.substring(bodyStart, bodyEnd).strip();
int level = Math.min(Math.max(heading.level, 1), stack.length - 1);
- MutableNode node = new MutableNode(heading.title, body, heading.start);
+ MutableNode node =
+ new MutableNode(heading.title, body, heading.start,
skipWhitespace(text, bodyStart));
stack[level] = node;
for (int j = level + 1; j < stack.length; j++) {
stack[j] = null;
@@ -75,6 +76,15 @@ final class HeadingParserSupport {
return root.toImmutable();
}
+ /** The first position at or after {@code position} that is not whitespace.
*/
+ private static int skipWhitespace(String text, int position) {
+ int result = position;
+ while (result < text.length() &&
Character.isWhitespace(text.charAt(result))) {
+ result++;
+ }
+ return result;
+ }
+
private static final class HeadingMatch {
private final int level;
private final String title;
@@ -93,12 +103,14 @@ final class HeadingParserSupport {
private final String title;
private final String body;
private final int startPosition;
+ private final int bodyStartPosition;
private final List<MutableNode> children = new ArrayList<>();
- private MutableNode(String title, String body, int startPosition) {
+ private MutableNode(String title, String body, int startPosition, int
bodyStartPosition) {
this.title = title;
this.body = body;
this.startPosition = startPosition;
+ this.bodyStartPosition = bodyStartPosition;
}
private DocumentNode toImmutable() {
@@ -106,7 +118,7 @@ final class HeadingParserSupport {
for (MutableNode child : children) {
childNodes.add(child.toImmutable());
}
- return new DocumentNode(title, body, startPosition, childNodes);
+ return new DocumentNode(title, body, startPosition, bodyStartPosition,
childNodes);
}
}
}
diff --git
a/plugins/transforms/textchunker/src/main/resources/org/apache/hop/pipeline/transforms/chunker/messages/messages_en_US.properties
b/plugins/transforms/textchunker/src/main/resources/org/apache/hop/pipeline/transforms/chunker/messages/messages_en_US.properties
index 091974a3eb..37df05af3d 100644
---
a/plugins/transforms/textchunker/src/main/resources/org/apache/hop/pipeline/transforms/chunker/messages/messages_en_US.properties
+++
b/plugins/transforms/textchunker/src/main/resources/org/apache/hop/pipeline/transforms/chunker/messages/messages_en_US.properties
@@ -47,7 +47,7 @@ TextChunker.chunkSize.Label=Chunk size
TextChunker.chunkSize.Tooltip=Maximum size for each chunk in characters
(approximate for paragraphs)
TextChunker.chunkOverlap.Label=Chunk overlap
-TextChunker.chunkOverlap.Tooltip=Number of characters to overlap between
chunks (for continuity)
+TextChunker.chunkOverlap.Tooltip=Maximum number of characters, in whole words,
carried from the end of one chunk into the next (for continuity). Paragraph and
Structure only use it inside a paragraph or section that has to be split
TextChunker.includeMetadata.Label=Include metadata
TextChunker.includeMetadata.Tooltip=Add metadata fields (chunk index,
position, document ID) to output
@@ -85,5 +85,5 @@ TextChunker.Log.EmptyText=No text to chunk, emitting a single
empty chunk row
TextChunker.Validation.OutputChunkFieldRequired=Output chunk field name must
be specified
TextChunker.Validation.SourceDocumentIdFieldNotFound=Source document ID field
''{0}'' not found in the input stream
TextChunker.Validation.ContentTypeFieldNotFound=Content type field ''{0}'' not
found in the input stream
-TextChunker.Validation.ParagraphOverlapIgnored=The Paragraph strategy does not
apply chunk overlap; the configured overlap is ignored
+TextChunker.Validation.OverlapOnlyWhenSplit=The Paragraph and Structure
strategies only apply the chunk overlap inside a paragraph or section that is
longer than the chunk size and has to be split
TextChunker.Validation.ParagraphSeparatorIgnored=The paragraph separator only
applies to the Paragraph strategy; the configured separator is ignored
diff --git
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMetaTest.java
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMetaTest.java
index 3c9e22cece..d1285beaef 100644
---
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMetaTest.java
+++
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/TextChunkerMetaTest.java
@@ -107,6 +107,39 @@ class TextChunkerMetaTest {
"a separator set for the Character strategy must produce a warning");
}
+ /**
+ * Paragraph and Structure apply the overlap only inside a paragraph or
section they split. The
+ * check says so for both, as a comment: it used to warn, for Paragraph
only, that the overlap was
+ * ignored.
+ */
+ @Test
+ void checkExplainsWhereParagraphAndStructureApplyTheOverlap() {
+ for (ChunkingStrategyType type : ChunkingStrategyType.values()) {
+ TextChunkerMeta meta = new TextChunkerMeta();
+ meta.setDefault();
+ meta.setInputField("body");
+ meta.setChunkingStrategy(type);
+ meta.setChunkOverlap("20");
+
+ IRowMeta prev = new RowMeta();
+ prev.addValueMeta(new ValueMetaString("body"));
+
+ List<ICheckResult> remarks = new ArrayList<>();
+ meta.check(remarks, null, new TransformMeta(), prev, null, null, null,
new Variables(), null);
+
+ boolean explained =
+ remarks.stream()
+ .anyMatch(
+ r ->
+ r.getType() == ICheckResult.TYPE_RESULT_COMMENT
+ && r.getText().contains("overlap"));
+ assertEquals(type != ChunkingStrategyType.CHARACTER, explained,
type.name());
+ assertTrue(
+ remarks.stream().noneMatch(r -> r.getType() ==
ICheckResult.TYPE_RESULT_WARNING),
+ type + " with a valid overlap must not produce a warning");
+ }
+ }
+
/** Chunk index, start position and total count are numbers, not strings. */
@Test
void exposesChunkMetadataAsIntegers() throws Exception {
diff --git
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategyTest.java
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategyTest.java
index 1c252a674f..36dc3fc9d1 100644
---
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategyTest.java
+++
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/CharacterChunkingStrategyTest.java
@@ -20,6 +20,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import java.util.Collections;
import java.util.List;
import org.apache.hop.pipeline.transforms.chunker.Chunk;
import org.junit.jupiter.api.BeforeEach;
@@ -82,23 +83,121 @@ public class CharacterChunkingStrategyTest {
String text = "This is a test string for chunking with overlap.";
List<Chunk> chunks = strategy.chunk(text, 15, 5);
- assertTrue(chunks.size() > 1, "Should have multiple chunks");
+ // "test " is the only whole word within the last 5 characters of the
first chunk. No word
+ // starts
+ // within the last 5 characters of the second ("ring ") or third ("king ")
one, so nothing is
+ // carried there.
+ assertEquals(
+ List.of("This is a test ", "test string ", "for chunking ", "with
overlap."),
+ contents(chunks));
+ assertAddressesTheSource(text, chunks);
+ }
+
+ /** #9283: the overlap must start at a word, not in the middle of one. */
+ @Test
+ public void testOverlapStartsAtAWordBoundary() {
+ List<Chunk> chunks = strategy.chunk(LOREM, 100, 20);
- // Verify overlap exists between chunks
- for (int i = 0; i < chunks.size() - 1; i++) {
+ assertTrue(chunks.size() > 3);
+ assertAddressesTheSource(LOREM, chunks);
+ for (int i = 1; i < chunks.size(); i++) {
+ Chunk previous = chunks.get(i - 1);
Chunk current = chunks.get(i);
- Chunk next = chunks.get(i + 1);
+ int start = current.getStartPosition();
+ assertTrue(
+ Character.isWhitespace(LOREM.charAt(start - 1)),
+ "chunk " + i + " starts in the middle of a word: " +
current.getContent());
+ assertTrue(
+ previous.getEndPosition() - start <= 20,
+ "chunk " + i + " overlaps more than the configured 20 characters");
+ }
+ // The overlap window of the first chunk starts inside "incididunt", which
used to be carried
+ // over as "ncididunt". Now the next whole word is.
+ assertTrue(chunks.get(1).getContent().startsWith("ut labore "),
chunks.get(1).getContent());
+ }
- String currentContent = current.getContent();
- String nextContent = next.getContent();
+ /**
+ * #9283: when a chunk backs off to a space and ends up shorter than the
overlap, the next chunk
+ * used to move forward by one character: 228 chunks for 419 characters.
+ */
+ @Test
+ public void testOverlapDoesNotProduceAChunkPerCharacter() {
+ String text = String.join(" ", Collections.nCopies(20,
"supercalifragilistic"));
+ List<Chunk> chunks = strategy.chunk(text, 25, 15);
- // The beginning of the next chunk should overlap with the end of the
current chunk
- String overlapPart = nextContent.substring(0, Math.min(5,
nextContent.length()));
- String endOfCurrent = currentContent.substring(Math.max(0,
currentContent.length() - 5));
+ assertEquals(20, chunks.size());
+ assertAddressesTheSource(text, chunks);
+ for (int i = 1; i < chunks.size(); i++) {
+ assertTrue(chunks.get(i).getEndPosition() > chunks.get(i -
1).getEndPosition());
+
assertTrue(chunks.get(i).getContent().startsWith("supercalifragilistic"));
+ }
+ }
+
+ /** Every chunk has to add text the previous one did not hold. */
+ @Test
+ public void testNoChunkIsContainedInThePreviousOne() {
+ String text = "aaaa bbbb cccc dddd eeee " + "x".repeat(30) + " ffff gggg";
+ List<Chunk> chunks = strategy.chunk(text, 25, 10);
- // At least some overlap should exist
+ assertAddressesTheSource(text, chunks);
+ for (int i = 1; i < chunks.size(); i++) {
assertTrue(
- endOfCurrent.contains(overlapPart.substring(0, Math.min(2,
overlapPart.length()))));
+ chunks.get(i).getEndPosition() > chunks.get(i - 1).getEndPosition(),
+ "chunk " + i + " only repeats the previous chunk: " +
chunks.get(i).getContent());
+ }
+ }
+
+ /** #9283: a run of whitespace longer than the chunk size gave chunks of
only spaces. */
+ @Test
+ public void testWhitespaceOnlyChunksAreDropped() {
+ String text = "a" + " ".repeat(30) + "b";
+ List<Chunk> chunks = strategy.chunk(text, 10, 0);
+
+ assertAddressesTheSource(text, chunks);
+ for (int i = 0; i < chunks.size(); i++) {
+ assertFalse(chunks.get(i).getContent().isBlank(), "chunk " + i + " is
whitespace only");
+ assertEquals(i, chunks.get(i).getIndex());
+ }
+ assertEquals(2, chunks.size());
+ assertTrue(chunks.get(0).getContent().startsWith("a"));
+ assertTrue(chunks.get(1).getContent().endsWith("b"));
+ }
+
+ @Test
+ public void testWhitespaceOnlyTextYieldsNoChunks() {
+ assertTrue(strategy.chunk(" \n\t ", 10, 2).isEmpty());
+ assertTrue(strategy.chunk(" \n\t ", 10, -1).isEmpty());
+ }
+
+ /** #9283: the split was on char positions, so an emoji was cut into two
lone surrogates. */
+ @Test
+ public void testSurrogatePairIsNotCut() {
+ String text = "\uD83D\uDE00\uD83D\uDE01\uD83D\uDE02";
+ List<Chunk> chunks = strategy.chunk(text, 5, 0);
+
+ assertEquals(List.of("\uD83D\uDE00\uD83D\uDE01", "\uD83D\uDE02"),
contents(chunks));
+ assertAddressesTheSource(text, chunks);
+ }
+
+ @Test
+ public void testSurrogatePairIsNotCutWhenItIsTheOnlyCharacter() {
+ String text = "\uD83D\uDE00\uD83D\uDE01";
+ List<Chunk> chunks = strategy.chunk(text, 1, 0);
+
+ assertEquals(List.of("\uD83D\uDE00", "\uD83D\uDE01"), contents(chunks));
+ }
+
+ @Test
+ public void testOverlapInsideALongWordDoesNotCutASurrogatePair() {
+ String text = "\uD83D\uDE00".repeat(20);
+ List<Chunk> chunks = strategy.chunk(text, 9, 3);
+
+ assertAddressesTheSource(text, chunks);
+ for (Chunk chunk : chunks) {
+ assertFalse(Character.isLowSurrogate(chunk.getContent().charAt(0)),
chunk.toString());
+ assertFalse(
+
Character.isHighSurrogate(chunk.getContent().charAt(chunk.getLength() - 1)),
+ chunk.toString());
}
}
@@ -179,6 +278,25 @@ public class CharacterChunkingStrategyTest {
assertEquals(text, reconstructFromChunks(text, chunks));
}
+ private static final String LOREM =
+ "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
tempor incididunt"
+ + " ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis
nostrud exercitation"
+ + " ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis
aute irure dolor in"
+ + " reprehenderit in voluptate velit esse cillum dolore eu fugiat
nulla pariatur.";
+
+ private static List<String> contents(List<Chunk> chunks) {
+ return chunks.stream().map(Chunk::getContent).toList();
+ }
+
+ /** Each chunk is the slice of the source text its positions name, and the
chunks cover it. */
+ private static void assertAddressesTheSource(String text, List<Chunk>
chunks) {
+ for (Chunk chunk : chunks) {
+ assertEquals(
+ text.substring(chunk.getStartPosition(), chunk.getEndPosition()),
chunk.getContent());
+ }
+ assertEquals(text.strip(), reconstructFromChunks(text, chunks).strip());
+ }
+
private static String reconstructFromChunks(String text, List<Chunk> chunks)
{
if (chunks.isEmpty()) {
return "";
diff --git
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategyTest.java
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategyTest.java
index 17826d0f51..37a119431b 100644
---
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategyTest.java
+++
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/ParagraphChunkingStrategyTest.java
@@ -137,6 +137,64 @@ public class ParagraphChunkingStrategyTest {
assertEquals(text, chunks.get(0).getContent());
}
+ /** #9283: a whitespace-only paragraph between two separators came out as a
chunk. */
+ @Test
+ public void testWhitespaceOnlyParagraphsAreNotEmitted() {
+ strategy.setSeparator("---");
+ String text = "one--- ---two";
+
+ List<Chunk> chunks = strategy.chunk(text, 6, 0);
+
+ assertEquals(2, chunks.size());
+ assertEquals("one", chunks.get(0).getContent());
+ assertEquals(0, chunks.get(0).getIndex());
+ assertEquals("two", chunks.get(1).getContent());
+ assertEquals(1, chunks.get(1).getIndex());
+ assertEquals(text.indexOf("two"), chunks.get(1).getStartPosition());
+ }
+
+ @Test
+ public void testWhitespaceOnlyParagraphIsKeptInsideAPackedChunk() {
+ strategy.setSeparator("---");
+ String text = "one--- ---two";
+
+ List<Chunk> chunks = strategy.chunk(text, 100, 0);
+
+ assertEquals(1, chunks.size());
+ assertEquals(text, chunks.get(0).getContent());
+ }
+
+ @Test
+ public void testWhitespaceOnlyTextYieldsNoChunks() {
+ assertTrue(strategy.chunk(" \n\n \t ", 100, 0).isEmpty());
+ strategy.setSeparator("---");
+ assertTrue(strategy.chunk(" ", 100, 0).isEmpty());
+ }
+
+ /** The overlap inside an oversized paragraph starts at a word, the same as
for Character. */
+ @Test
+ public void testOverlapInsideAnOversizedParagraphStartsAtAWord() {
+ String text =
+ "Short one.\n\nLorem ipsum dolor sit amet, consectetur adipiscing
elit, sed do eiusmod"
+ + " tempor incididunt ut labore et dolore magna aliqua. Ut enim ad
minim veniam, quis"
+ + " nostrud exercitation ullamco laboris nisi ut aliquip ex ea
commodo consequat.";
+
+ List<Chunk> chunks = strategy.chunk(text, 100, 20);
+
+ assertTrue(chunks.size() > 3);
+ for (int i = 2; i < chunks.size(); i++) {
+ Chunk chunk = chunks.get(i);
+ assertEquals(
+ text.substring(chunk.getStartPosition(), chunk.getEndPosition()),
chunk.getContent());
+ assertTrue(
+ Character.isWhitespace(text.charAt(chunk.getStartPosition() - 1)),
+ "chunk " + i + " starts mid-word: " + chunk.getContent());
+ assertTrue(
+ chunk.getStartPosition() < chunks.get(i - 1).getEndPosition(),
+ "the overlap still applies inside an oversized paragraph");
+ }
+ }
+
@Test
public void testEmptyParagraphs() {
String text = "First\n\n\n\nSecond";
diff --git
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategyTest.java
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategyTest.java
index 44f5452f83..fc6f30d76c 100644
---
a/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategyTest.java
+++
b/plugins/transforms/textchunker/src/test/java/org/apache/hop/pipeline/transforms/chunker/chunking/StructureChunkingStrategyTest.java
@@ -97,6 +97,82 @@ class StructureChunkingStrategyTest {
assertTrue(chunks.get(chunks.size() - 1).getContent().contains("beta"));
}
+ /**
+ * #9283: the chunks a long section is split into reported the heading
position plus their offset
+ * in the stripped body, so every chunk after the first was off by the
length of the heading.
+ */
+ @Test
+ void positionsOfASplitSectionPointAtTheirTextInTheSource() {
+ StringBuilder body = new StringBuilder();
+ for (int i = 1; i <= 40; i++) {
+ body.append(String.format("word%03d ", i));
+ }
+ String text = "# Title\n\n" + body.toString().strip();
+
+ List<Chunk> chunks = strategy.chunk(text, 100, 0, ContentType.MARKDOWN);
+
+ assertTrue(chunks.size() > 2);
+ assertEquals(0, chunks.get(0).getStartPosition(), "the first chunk starts
at the heading");
+ for (int i = 1; i < chunks.size(); i++) {
+ Chunk chunk = chunks.get(i);
+ String partText = chunk.getContent().substring("Title\n\n".length());
+ assertEquals(
+ partText,
+ text.substring(chunk.getStartPosition(), chunk.getEndPosition()),
+ "chunk " + i + " does not address its text in the source");
+ }
+ assertEquals(text.indexOf("word012"), chunks.get(1).getStartPosition());
+ }
+
+ @Test
+ void positionsOfASplitSectionSkipBlankLinesAfterTheHeading() {
+ String text = "## A\n\n\n " + "alpha beta gamma delta
".repeat(10).strip();
+
+ List<Chunk> chunks = strategy.chunk(text, 40, 0, ContentType.MARKDOWN);
+
+ assertTrue(chunks.size() > 2);
+ for (int i = 1; i < chunks.size(); i++) {
+ Chunk chunk = chunks.get(i);
+ assertEquals(
+ chunk.getContent().substring("A\n\n".length()),
+ text.substring(chunk.getStartPosition(), chunk.getEndPosition()));
+ }
+ }
+
+ /** An unsplit section keeps starting at its heading. */
+ @Test
+ void anUnsplitSectionStartsAtItsHeading() {
+ String text = "# Title\n\nIntro line.\n\n## Section A\n\nBody A.";
+
+ List<Chunk> chunks = strategy.chunk(text, 200, 0, ContentType.MARKDOWN);
+
+ assertEquals(2, chunks.size());
+ assertEquals(0, chunks.get(0).getStartPosition());
+ assertEquals(text.indexOf("## Section A"),
chunks.get(1).getStartPosition());
+ assertEquals(text.length(), chunks.get(1).getEndPosition());
+ }
+
+ /** The overlap inside a split section starts at a word, the same as for
Character. */
+ @Test
+ void overlapInsideASplitSectionStartsAtAWord() {
+ StringBuilder body = new StringBuilder();
+ for (int i = 1; i <= 40; i++) {
+ body.append(String.format("word%03d ", i));
+ }
+ String text = "# Title\n\n" + body.toString().strip();
+
+ List<Chunk> chunks = strategy.chunk(text, 100, 20, ContentType.MARKDOWN);
+
+ assertTrue(chunks.size() > 2);
+ for (int i = 1; i < chunks.size(); i++) {
+ String partText =
chunks.get(i).getContent().substring("Title\n\n".length());
+ assertTrue(partText.startsWith("word"), "chunk " + i + " starts
mid-word: " + partText);
+ assertTrue(
+ chunks.get(i).getStartPosition() < chunks.get(i -
1).getEndPosition(),
+ "chunk " + i + " does not overlap the previous one");
+ }
+ }
+
@Test
void strategyTypeIsStructure() {
assertEquals(ChunkingStrategyType.STRUCTURE, strategy.getType());