hansva commented on code in PR #9363:
URL: https://github.com/apache/hop/pull/9363#discussion_r4537354669


##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/AsciiDocDocumentParser.java:
##########
@@ -30,6 +52,123 @@ public ContentType getContentType() {
 
   @Override
   public DocumentNode parse(String text) {
-    return HeadingParserSupport.parse(text, HEADING, String::length);
+    if (text == null || text.isEmpty()) {
+      return DocumentNode.root("");
+    }
+    return HeadingParserSupport.parse(text, scan(text).headings());
+  }
+
+  /** The section titles of an AsciiDoc text, skipping delimited blocks. */
+  static List<Heading> findHeadings(String text) {
+    return scan(text).headings();
+  }
+
+  /**
+   * The {@code [start, end)} offsets of the closed listing ({@code ----}) and 
literal ({@code
+   * ....}) blocks. Used by content detection to ignore {@code #} comments in 
code samples.
+   */
+  static List<int[]> findCodeBlocks(String text) {
+    return scan(text).codeBlocks();
+  }
+
+  /**
+   * True when the first line with content, after comments, attribute entries 
and block attributes,
+   * is a document title ({@code = Title}).
+   */
+  static boolean startsWithDocumentTitle(String text) {
+    boolean inComment = false;
+    for (Line line : HeadingParserSupport.lines(text)) {
+      String content = line.text().strip();
+      if (inComment) {
+        inComment = !isCommentDelimiter(content);
+        continue;
+      }
+      if (isCommentDelimiter(content)) {
+        inComment = true;
+        continue;
+      }
+      if (content.isEmpty()
+          || content.startsWith("//")
+          || ATTRIBUTE_ENTRY.matcher(content).matches()
+          || (content.startsWith("[") && content.endsWith("]"))) {
+        continue;
+      }
+      Matcher heading = HEADING.matcher(line.text());
+      return heading.matches() && heading.group(1).length() == 1;
+    }
+    return false;
+  }
+
+  private static boolean isCommentDelimiter(String line) {
+    return line.length() >= 4 && line.chars().allMatch(c -> c == '/');
+  }
+
+  private record Scan(List<Heading> headings, List<int[]> codeBlocks) {}
+
+  private static Scan scan(String text) {
+    List<Heading> headings = new ArrayList<>();
+    List<int[]> codeBlocks = new ArrayList<>();
+    Deque<String> openBlocks = new ArrayDeque<>();
+    int codeBlockStart = -1;
+
+    for (Line line : HeadingParserSupport.lines(text)) {
+      String content = line.text().stripTrailing();
+      String open = openBlocks.peek();
+      if (open != null && closes(open, content)) {
+        openBlocks.pop();
+        if (isCodeBlock(open)) {
+          codeBlocks.add(new int[] {codeBlockStart, line.end()});
+        }
+        continue;
+      }
+      if (open != null && isVerbatim(open)) {
+        // Nothing nests inside a listing, literal, passthrough or comment 
block.
+        continue;
+      }
+      if (DELIMITER.matcher(content).matches()) {

Review Comment:
   Any line of 4 or more `=` or `-` is pushed as an open block here, and that 
includes the underline of a two-line (setext) title, which Asciidoctor still 
accepts:
   
   ```
   Introduction
   ============
   
   Text
   
   == Install
   
   Steps
   ```
   
   `============` opens an example block that never closes, so `== Install` is 
skipped and the whole document becomes a single chunk (a `----` underline even 
opens a verbatim listing). Before this PR `== Install` was its own section. 
Maybe only treat the line as a delimiter when the previous line is blank (or 
not a single text line), or when a matching closing delimiter follows?



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/MarkdownDocumentParser.java:
##########
@@ -30,6 +46,46 @@ public ContentType getContentType() {
 
   @Override
   public DocumentNode parse(String text) {
-    return HeadingParserSupport.parse(text, HEADING, marker -> 
marker.length());
+    if (text == null || text.isEmpty()) {
+      return DocumentNode.root("");
+    }
+    return HeadingParserSupport.parse(text, findHeadings(text));
+  }
+
+  /** The headings of a Markdown text, skipping fenced code blocks. */
+  static List<Heading> findHeadings(String text) {
+    List<Heading> headings = new ArrayList<>();
+    String openFence = null;
+    for (Line line : HeadingParserSupport.lines(text)) {
+      Matcher fence = FENCE.matcher(line.text());
+      if (openFence != null) {
+        // A fence closes on the same character, at least as long, with 
nothing after it.
+        if (fence.matches()
+            && fence.group(1).charAt(0) == openFence.charAt(0)
+            && fence.group(1).length() >= openFence.length()
+            && fence.group(2).isBlank()) {
+          openFence = null;
+        }
+        continue;
+      }
+      if (fence.matches() && !(fence.group(1).charAt(0) == '`' && 
fence.group(2).contains("`"))) {
+        openFence = fence.group(1);
+        continue;
+      }
+      Heading heading = heading(line);
+      if (heading != null) {
+        headings.add(heading);
+      }
+    }
+    return headings;
+  }
+
+  private static Heading heading(Line line) {
+    Matcher matcher = HEADING.matcher(line.text());
+    if (!matcher.matches()) {
+      return null;
+    }
+    String title = 
CLOSING_SEQUENCE.matcher(matcher.group(2).strip()).replaceFirst("");

Review Comment:
   Stripping the closing sequence can leave an empty title: `## #` or `### ###` 
becomes a section titled `""`, so the breadcrumb ends in `Guide > `. The old 
regex kept `#` as the title. Maybe keep the original text when stripping leaves 
nothing?



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/AsciiDocDocumentParser.java:
##########
@@ -16,12 +16,34 @@
  */
 package org.apache.hop.pipeline.transforms.chunker.document;
 
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Deque;
+import java.util.List;
+import java.util.regex.Matcher;
 import java.util.regex.Pattern;
+import 
org.apache.hop.pipeline.transforms.chunker.document.HeadingParserSupport.Heading;
+import 
org.apache.hop.pipeline.transforms.chunker.document.HeadingParserSupport.Line;
 
-/** Splits AsciiDoc on {@code =} headings and builds a section tree. */
+/**
+ * Splits AsciiDoc on {@code =} section titles and builds a section tree. 
Lines inside a delimited
+ * block ({@code ====}, {@code ----}, {@code ////}, ...) are never section 
titles, and a delimiter
+ * line is never a title either.
+ */
 public final class AsciiDocDocumentParser implements DocumentParser {
 
-  private static final Pattern HEADING = Pattern.compile("(?m)^(=+)\\s+(.+)$");
+  /** A section title: 1-6 {@code =}, then a space or tab and the title, on a 
single line. */
+  private static final Pattern HEADING = Pattern.compile("^(={1,6})[ 
\\t]+(\\S.*)$");
+
+  /**
+   * A delimited block line: 4 or more of the same delimiter character, {@code 
--} (open block), a
+   * {@code ```} fence, or a table delimiter such as {@code |===}.
+   */
+  private static final Pattern DELIMITER =
+      Pattern.compile("^(?:([/\\-.+=*_])\\1{3,}|--|```.*|[|,:!]={3,})$");

Review Comment:
   `DELIMITER` recognises ``` fences but not Markdown `~~~` fences. When auto 
detection runs this scan over a Markdown page with `## Example` followed by a 
`~~~` fence containing `== Section A` and `== Section B`, AsciiDoc wins 2 to 1 
and the page is chunked on lines inside its code sample.



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/hopxml/HopXmlSupport.java:
##########
@@ -81,12 +82,51 @@ public final class HopXmlSupport {
 
   private HopXmlSupport() {}
 
+  /** True when the text is an XML document whose root element is a pipeline 
or a workflow. */
   public static boolean looksLikeHopXml(String text) {
-    if (text == null || text.isBlank()) {
-      return false;
+    String root = rootElementName(text);
+    return "pipeline".equals(root) || "workflow".equals(root);
+  }
+
+  /**
+   * The name of the first element of an XML document, after the XML 
declaration, comments and a
+   * document type declaration. Returns {@code null} when the text does not 
start like XML, for
+   * example Markdown that mentions {@code <pipeline>} in a sentence.
+   */
+  public static String rootElementName(String text) {
+    if (text == null) {
+      return null;
+    }
+    String rest = text.strip();

Review Comment:
   Two things here: a `<!DOCTYPE` with an internal subset (`[ <!ENTITY ...> ]`) 
ends at the first `>` inside the subset, so this returns null and the document 
isn't recognised as Hop XML. And the strip plus a substring after each prolog 
item copies almost the whole document, twice per row (in detect and again in 
`looksLikeHopXml`); tracking an index instead of copying would avoid that for 
large .hpl/.hwf files.



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/DocumentParserRegistry.java:
##########
@@ -56,26 +50,57 @@ public static ContentType detect(String text) {
     if (text == null || text.isBlank()) {
       return ContentType.PLAIN;
     }
-    String trimmed = text.stripLeading();
-    if (trimmed.startsWith("{") || text.contains("Configuration:\n{")) {
+    if (looksLikeJsonObject(text) || text.contains("Configuration:\n{")) {
       return ContentType.METADATA;
     }
-    if (HopXmlSupport.looksLikeHopXml(text)) {
-      if (trimmed.contains("<workflow")) {
-        return ContentType.WORKFLOW;
-      }
-      if (trimmed.contains("<pipeline")) {
-        return ContentType.PIPELINE;
-      }
+    String rootElement = HopXmlSupport.rootElementName(text);
+    if ("workflow".equals(rootElement)) {
+      return ContentType.WORKFLOW;
+    }
+    if ("pipeline".equals(rootElement)) {
+      return ContentType.PIPELINE;
     }
-    // find() rather than matches(): a heading anywhere in the document 
counts, including one on
-    // the very first line, which is the common case for a single-page 
document.
-    if (ASCIIDOC_HEADING.matcher(text).find()) {
+    return detectHeadingSyntax(text);
+  }
+
+  /**
+   * Markdown or AsciiDoc, by the headings each parser would find. An AsciiDoc 
document title
+   * ({@code = Title}) as the first content line settles it. Otherwise the 
syntax with the most
+   * headings wins and AsciiDoc wins a tie. A level 0 {@code = } line further 
down only counts when
+   * there are no Markdown headings: next to them it is more likely text such 
as {@code = 42}.
+   */
+  private static ContentType detectHeadingSyntax(String text) {
+    if (AsciiDocDocumentParser.startsWithDocumentTitle(text)) {
       return ContentType.ASCIIDOC;
     }
-    if (MARKDOWN_HEADING.matcher(text).find()) {
-      return ContentType.MARKDOWN;
+    List<HeadingParserSupport.Heading> asciiDocHeadings = 
AsciiDocDocumentParser.findHeadings(text);

Review Comment:
   Auto detection now walks the full text up to 6 times (title check, 
`findHeadings`, the Markdown scan, `findCodeBlocks`, two full `strip()` 
copies), and the chosen parser scans it again, each pass building a substring 
per line. For pipelines chunking many multi-MB documents with AUTO that adds 
up. `findHeadings` and `findCodeBlocks` could share one scan (see the comment 
on `AsciiDocDocumentParser` line 71), and the strip calls only need to look at 
the leading/trailing characters.



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/AsciiDocDocumentParser.java:
##########
@@ -30,6 +52,123 @@ public ContentType getContentType() {
 
   @Override
   public DocumentNode parse(String text) {
-    return HeadingParserSupport.parse(text, HEADING, String::length);
+    if (text == null || text.isEmpty()) {
+      return DocumentNode.root("");
+    }
+    return HeadingParserSupport.parse(text, scan(text).headings());
+  }
+
+  /** The section titles of an AsciiDoc text, skipping delimited blocks. */
+  static List<Heading> findHeadings(String text) {
+    return scan(text).headings();
+  }
+
+  /**
+   * The {@code [start, end)} offsets of the closed listing ({@code ----}) and 
literal ({@code
+   * ....}) blocks. Used by content detection to ignore {@code #} comments in 
code samples.
+   */
+  static List<int[]> findCodeBlocks(String text) {
+    return scan(text).codeBlocks();

Review Comment:
   `findHeadings()` and `findCodeBlocks()` each run a full `scan()` and keep 
half of the result, and the registry calls both on the same text. Returning the 
`Scan` record (or one method returning both lists) would scan once.



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/DocumentParserRegistry.java:
##########
@@ -56,26 +50,57 @@ public static ContentType detect(String text) {
     if (text == null || text.isBlank()) {
       return ContentType.PLAIN;
     }
-    String trimmed = text.stripLeading();
-    if (trimmed.startsWith("{") || text.contains("Configuration:\n{")) {
+    if (looksLikeJsonObject(text) || text.contains("Configuration:\n{")) {
       return ContentType.METADATA;
     }
-    if (HopXmlSupport.looksLikeHopXml(text)) {
-      if (trimmed.contains("<workflow")) {
-        return ContentType.WORKFLOW;
-      }
-      if (trimmed.contains("<pipeline")) {
-        return ContentType.PIPELINE;
-      }
+    String rootElement = HopXmlSupport.rootElementName(text);
+    if ("workflow".equals(rootElement)) {
+      return ContentType.WORKFLOW;
+    }
+    if ("pipeline".equals(rootElement)) {
+      return ContentType.PIPELINE;
     }
-    // find() rather than matches(): a heading anywhere in the document 
counts, including one on
-    // the very first line, which is the common case for a single-page 
document.
-    if (ASCIIDOC_HEADING.matcher(text).find()) {
+    return detectHeadingSyntax(text);
+  }
+
+  /**
+   * Markdown or AsciiDoc, by the headings each parser would find. An AsciiDoc 
document title
+   * ({@code = Title}) as the first content line settles it. Otherwise the 
syntax with the most
+   * headings wins and AsciiDoc wins a tie. A level 0 {@code = } line further 
down only counts when
+   * there are no Markdown headings: next to them it is more likely text such 
as {@code = 42}.
+   */
+  private static ContentType detectHeadingSyntax(String text) {
+    if (AsciiDocDocumentParser.startsWithDocumentTitle(text)) {
       return ContentType.ASCIIDOC;
     }
-    if (MARKDOWN_HEADING.matcher(text).find()) {
-      return ContentType.MARKDOWN;
+    List<HeadingParserSupport.Heading> asciiDocHeadings = 
AsciiDocDocumentParser.findHeadings(text);
+    long asciiDocSections = asciiDocHeadings.stream().filter(h -> h.level() > 
1).count();
+    List<HeadingParserSupport.Heading> markdownHeadings = 
MarkdownDocumentParser.findHeadings(text);
+    long markdown = markdownHeadings.size();
+    if (asciiDocSections > 0) {
+      // In AsciiDoc a # line in a ---- listing block is a code comment, not a 
Markdown heading.
+      List<int[]> codeBlocks = AsciiDocDocumentParser.findCodeBlocks(text);

Review Comment:
   Only listing (`----`) and literal (`....`) blocks are subtracted from the 
Markdown heading count. `#` lines inside comment (`////`), passthrough (`++++`) 
and other verbatim blocks still count as Markdown headings. E.g. one `== Setup` 
section with a `////` or `++++` block holding two `# ...` lines gives 
markdown=2 against 1 AsciiDoc section, so the fragment is detected as Markdown 
and split on those lines. Subtracting all verbatim blocks from the scan would 
avoid that.



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/AsciiDocDocumentParser.java:
##########
@@ -30,6 +52,123 @@ public ContentType getContentType() {
 
   @Override
   public DocumentNode parse(String text) {
-    return HeadingParserSupport.parse(text, HEADING, String::length);
+    if (text == null || text.isEmpty()) {
+      return DocumentNode.root("");
+    }
+    return HeadingParserSupport.parse(text, scan(text).headings());
+  }
+
+  /** The section titles of an AsciiDoc text, skipping delimited blocks. */
+  static List<Heading> findHeadings(String text) {
+    return scan(text).headings();
+  }
+
+  /**
+   * The {@code [start, end)} offsets of the closed listing ({@code ----}) and 
literal ({@code
+   * ....}) blocks. Used by content detection to ignore {@code #} comments in 
code samples.
+   */
+  static List<int[]> findCodeBlocks(String text) {
+    return scan(text).codeBlocks();
+  }
+
+  /**
+   * True when the first line with content, after comments, attribute entries 
and block attributes,
+   * is a document title ({@code = Title}).
+   */
+  static boolean startsWithDocumentTitle(String text) {

Review Comment:
   This re-implements comment tracking next to `scan()`, and the two disagree: 
`isCommentDelimiter` closes a comment on any run of 4+ `/`, while `scan()` 
needs an exact match. With a `//////` comment that contains a `////` line, one 
sees the first heading after the comment and the other doesn't. Could this 
reuse the same block tracking as `scan()`?



##########
plugins/transforms/textchunker/src/main/java/org/apache/hop/pipeline/transforms/chunker/document/DocumentParserRegistry.java:
##########
@@ -56,26 +50,57 @@ public static ContentType detect(String text) {
     if (text == null || text.isBlank()) {
       return ContentType.PLAIN;
     }
-    String trimmed = text.stripLeading();
-    if (trimmed.startsWith("{") || text.contains("Configuration:\n{")) {
+    if (looksLikeJsonObject(text) || text.contains("Configuration:\n{")) {
       return ContentType.METADATA;
     }
-    if (HopXmlSupport.looksLikeHopXml(text)) {
-      if (trimmed.contains("<workflow")) {
-        return ContentType.WORKFLOW;
-      }
-      if (trimmed.contains("<pipeline")) {
-        return ContentType.PIPELINE;
-      }
+    String rootElement = HopXmlSupport.rootElementName(text);
+    if ("workflow".equals(rootElement)) {
+      return ContentType.WORKFLOW;
+    }
+    if ("pipeline".equals(rootElement)) {
+      return ContentType.PIPELINE;
     }
-    // find() rather than matches(): a heading anywhere in the document 
counts, including one on
-    // the very first line, which is the common case for a single-page 
document.
-    if (ASCIIDOC_HEADING.matcher(text).find()) {
+    return detectHeadingSyntax(text);
+  }
+
+  /**
+   * Markdown or AsciiDoc, by the headings each parser would find. An AsciiDoc 
document title
+   * ({@code = Title}) as the first content line settles it. Otherwise the 
syntax with the most
+   * headings wins and AsciiDoc wins a tie. A level 0 {@code = } line further 
down only counts when
+   * there are no Markdown headings: next to them it is more likely text such 
as {@code = 42}.
+   */
+  private static ContentType detectHeadingSyntax(String text) {
+    if (AsciiDocDocumentParser.startsWithDocumentTitle(text)) {
       return ContentType.ASCIIDOC;
     }
-    if (MARKDOWN_HEADING.matcher(text).find()) {
-      return ContentType.MARKDOWN;
+    List<HeadingParserSupport.Heading> asciiDocHeadings = 
AsciiDocDocumentParser.findHeadings(text);
+    long asciiDocSections = asciiDocHeadings.stream().filter(h -> h.level() > 
1).count();
+    List<HeadingParserSupport.Heading> markdownHeadings = 
MarkdownDocumentParser.findHeadings(text);
+    long markdown = markdownHeadings.size();
+    if (asciiDocSections > 0) {
+      // In AsciiDoc a # line in a ---- listing block is a code comment, not a 
Markdown heading.
+      List<int[]> codeBlocks = AsciiDocDocumentParser.findCodeBlocks(text);
+      markdown =
+          markdownHeadings.stream()
+              .filter(h -> codeBlocks.stream().noneMatch(b -> h.start() >= 
b[0] && h.end() <= b[1]))
+              .count();
+    }
+    if (markdown == 0) {
+      return asciiDocHeadings.isEmpty() ? ContentType.PLAIN : 
ContentType.ASCIIDOC;
+    }
+    return asciiDocSections >= markdown ? ContentType.ASCIIDOC : 
ContentType.MARKDOWN;

Review Comment:
   On a tie AsciiDoc wins, even with Markdown headings present: `# Guide` + 
body + `== Note: tie-break` gives 1 against 1, so `# Guide` ends up in the 
preamble and the text is split on the `== Note` line. The javadoc says these 
lines only count when there are no Markdown headings, and also mentions 'level 
0' while the code checks `level() > 1`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to