Adalennis opened a new issue, #9282:
URL: https://github.com/apache/hop/issues/9282

   ### Apache Hop version?
   
   2.20.0
   
   ### Java version?
   
   Java 21
   
   ### Operating system
   
   macOS
   
   ### What happened?
   
   Found while testing the Text chunker on 2.20.0 (code unchanged on main). All 
with the Structure strategy, chunk size 1000, overlap 0.
   
   **1. AsciiDoc block delimiters become headings, and the text after them is 
lost**
   In `AsciiDocDocumentParser` the heading pattern `^(=+)\s+(.+)$` lets `\s+` 
match a line break, so a `====` example block delimiter is read as a heading 
and the next line as its title. Those sections have an empty body and are 
dropped.
   
   Input (Content type ASCIIDOC, same result with AUTO):
       = Title / Intro. / == Section / Before the block. / ==== / Example text 
inside the block. / ==== / After the block. / == Next / More text.
   Chunks: `Title`, `Title > Section: Before the block.`, `Title > Next: More 
text.`. "Example text inside the block." and "After the block." are in no chunk.
   
   **2. `#` lines inside a Markdown code fence become headings**
       # Guide / Intro. / ```bash / # download the zip / curl -O … / ``` / 
After the block.
   gives two chunks: `Guide: Intro. ```bash` and `download the zip: curl -O … 
``` After the block.` The fence is split in two and a shell comment becomes a 
section title.
   
   **3. A skipped heading level loses its parent**
   `# Guide` followed by `### Detail` gives the breadcrumb `Detail` instead of 
`Guide > Detail`. When the stack has no entry for the level just above, the 
parser falls back to the root instead of the nearest ancestor.
   
   **4. Closing hashes stay in the title**
   `## Install ##` gives the title `Install ##`. In CommonMark the closing 
sequence is not part of the heading.
   
   **5. Content type AUTO picks the wrong parser**
   - Markdown that mentions `<pipeline>` or `<workflow>`, or that starts with 
`{` (for example a Hugo shortcode `{{< note >}}`), is detected as Hop XML or 
metadata JSON. The parse fails (`[Fatal Error] :1:1: Content is not allowed in 
prolog.` on stderr) and the whole text becomes one chunk with the `#` markers 
still in it.
   - A single line that starts with `= `, such as `= 42`, makes AUTO pick 
AsciiDoc. In `# Guide / The answer: / = 42 / ## Next / More text.` the `#` 
headings are ignored and `42` becomes the only section title.
   
   
   ### Issue Priority
   
   Priority: 3
   
   ### Issue Component
   
   Component: Hop Gui


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to