Adalennis opened a new issue, #9283:
URL: https://github.com/apache/hop/issues/9283

   ### Apache Hop version?
   
   2.20.0
   
   ### Java version?
   
   Java 21
   
   ### Operating system
   
   macOS
   
   ### What happened?
   
   Found while testing the Text chunker on 2.20.0 (code unchanged on main).
   
   **1. Overlap starts in the middle of a word**
   The next chunk starts at `chunkEnd - overlap` with no word alignment. Lorem 
ipsum, Character, size 100, overlap 20: chunks 2-4 start with `ncididunt`, 
`ostrud`, `is aute`. The page says words are not cut, and #8415 notes that 
mid-word cuts retrieve poorly. The existing `0115` character golden encodes the 
current behaviour.
   
   **2. Overlap can degenerate to one chunk per character**
   When a chunk backs off to a space and ends up shorter than the overlap, the 
next chunk only moves forward by 1 character. 20 x "supercalifragilistic" (419 
characters), Character, size 25, overlap 15: 228 chunks (`alifragilistic `, 
`lifragilistic `, `ifragilistic `, …). The pipeline check shows no warning.
   
   **3. Paragraph and Structure apply the overlap**
   The page says overlap is "Character strategy only", and the check warns "The 
Paragraph strategy does not apply chunk overlap; the configured overlap is 
ignored". But both strategies pass the overlap to the character fallback for 
oversized paragraphs and sections: with size 100 and overlap 20, Paragraph 
chunks start with `ncididunt`, `ostrud`, `is aute` and Structure chunks with 
`empor incididunt`, `eniam, quis`. Structure gets no check warning.
   
   **4. Positions in split Structure sections ignore the heading**
   `# Title` + blank line + 319 characters of words, Structure, size 100: 
chunk_start_position is 0, 88, 176, 264. The text of chunk 2 (`word011`) starts 
at 97 in the input: every chunk after the first is 9 too low (the length of `# 
Title\n\n`). The position is the heading position plus the offset in the 
stripped body.
   
   **5. Whitespace-only chunks in Character and Paragraph**
   `"a" + 30 spaces + "b"`, Character, size 10: chunks 2 and 3 are 10 spaces 
each. Paragraph with separator `---` and size 6: `one--- ---two` gives `one`, ` 
`, `two`. #8751 removed whitespace-only chunks for Structure only; is that 
intended for the other two strategies?
   
   **6. Character can cut a surrogate pair**
   `😀😁😂`, Character, size 5: chunks `😀😁?` and `?`. The split is on Java char 
positions, so an emoji is cut in half.
   
   ### Issue Priority
   
   Priority: 3
   
   ### Issue Component
   
   Component: Hop Gui


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to