This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new 4920528d50 TIKA-5056: write markdown math as a code span, not raw
document text Math went into markdown output as a raw node, so document text
in a math span (#3561)
4920528d50 is described below
commit 4920528d5084136bbcb534f6899fc4797f942a5e
Author: Tim Allison <[email protected]>
AuthorDate: Fri Nov 6 14:59:57 2026 -0500
TIKA-5056: write markdown math as a code span, not raw document text Math
went into markdown output as a raw node, so document text in a math span (#3561)
(VLM output, or a PDF custom type or Word style named "Math") could become
HTML
or links. It is now written as $`...`$, which GitHub and GitLab render as
math
and no renderer turns into markup; only "math inline"/"math display" spans
count as math. A randomized test renders the output and asserts no raw
HTML,
link or image survives.
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
CHANGES.txt | 6 +
.../pages/configuration/inference-recipes.adoc | 2 +-
.../pages/configuration/parsers/vlm-parsers.adoc | 6 +-
docs/modules/ROOT/pages/security.adoc | 11 ++
.../apache/tika/sax/ToMarkdownContentHandler.java | 51 +++++++--
.../tika/sax/ToMarkdownContentHandlerTest.java | 121 ++++++++++++++++++++-
.../parser/vlm/MarkdownToXHTMLEmitterTest.java | 6 +-
7 files changed, 184 insertions(+), 19 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index d228daca63..a5bfadb743 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -2,6 +2,12 @@ Release 4.3.0 - ???
BREAKING CHANGES AND DEPRECATIONS
+ * Markdown output writes math as $`...`$ (a code span, which GitHub and
GitLab
+ render as math) instead of raw $...$, so text in a math span can no
longer become
+ HTML or links. Readers that only know $...$, such as pandoc, show it as
code. Only
+ spans with class "math inline" or "math display" count as math; a PDF
structure
+ type or Word style named "Math" is escaped like other text (TIKA-5056).
+
* tika-ml-junkdetect has a new model and API. JunkDetector wraps a
HashedJunkModel
(junkdetect.bin, format JUNKHNG1; the old model format is not read).
knownScripts(),
scoreWithFeatureComponents() and the per-feature compute* methods are
gone; use
diff --git a/docs/modules/ROOT/pages/configuration/inference-recipes.adoc
b/docs/modules/ROOT/pages/configuration/inference-recipes.adoc
index bdbc77e4af..d1c40f722e 100644
--- a/docs/modules/ROOT/pages/configuration/inference-recipes.adoc
+++ b/docs/modules/ROOT/pages/configuration/inference-recipes.adoc
@@ -68,7 +68,7 @@ into XHTML, so a table is a `<table>` in every output format
and a pipe table in
HTML table if it has merged cells). Body
cells come back as `<th>` and are passed through as such. Math comes back as
LaTeX; Tika keeps it
verbatim, delimiters included, in `<span class="math inline">` or `<span
class="math display">`,
-and writes it unescaped in markdown. The load-time health check reads the
+and writes it in markdown as ``$`...`$``, verbatim inside a code span. The
load-time health check reads the
public `/v1/models`, so a wrong key passes startup and surfaces on the first
request as the
document's parse exception (HTTP 401 in the message).
diff --git a/docs/modules/ROOT/pages/configuration/parsers/vlm-parsers.adoc
b/docs/modules/ROOT/pages/configuration/parsers/vlm-parsers.adoc
index 2b79d7c3ae..0b6de7a831 100644
--- a/docs/modules/ROOT/pages/configuration/parsers/vlm-parsers.adoc
+++ b/docs/modules/ROOT/pages/configuration/parsers/vlm-parsers.adoc
@@ -319,8 +319,10 @@ Superscripts and subscripts are `sup`/`sub` in XHTML and
inline HTML in markdown
(`x<sup>2</sup>`); plain-text output reads `x2`.
TeX math (`$...$`, `$$...$$`, `\(...\)`, `\[...\]`) is kept verbatim,
delimiters included, in
-`<span class="math inline">` or `<span class="math display">`, and written
unescaped in markdown
-output. A `$` followed by a space, or a closing `$` followed by a digit, is
not math, so prices are
+`<span class="math inline">` or `<span class="math display">`. Markdown output
writes it as
+``$`...`$``, which GitHub and GitLab render as math: the code span keeps the
TeX verbatim, and
+whatever the document put in it cannot become HTML or a link. Readers that
only know `$...$`, such
+as pandoc, show it as code. Display math is written the same way, inline. A
`$` followed by a space, or a closing `$` followed by a digit, is not math, so
prices are
left alone. Math containing a backtick or a blank line is not recognized.
[#_per_request_configuration]
diff --git a/docs/modules/ROOT/pages/security.adoc
b/docs/modules/ROOT/pages/security.adoc
index 4a481120e8..f5053db5ba 100644
--- a/docs/modules/ROOT/pages/security.adoc
+++ b/docs/modules/ROOT/pages/security.adoc
@@ -100,6 +100,17 @@ For the upgrade from the former `enableUnsecureFeatures`
flag, which is now spli
`allowPipes` and `allowPerRequestConfig`, see
xref:migration-to-4x/migrating-tika-server-4x.adoc#_pipes_and_async_require_allowpipes_per_request_config_requires_allowperrequestconfig[Migrating
tika-server to 4.x].
+=== Extracted Output Is Document Content
+
+Text, links and image references in Tika's output come from the document, so
treat them like
+any untrusted input before you display them. XHTML output escapes document
text, and markdown
+output escapes it for CommonMark: document text does not become HTML or
markdown structure there.
+Links and images the document itself contains are kept, in both formats, with
their URLs as
+written. If you render the output, sanitize it the way you would any
user-supplied HTML or
+markdown, including URL schemes such as `javascript:`. Markdown dialects with
syntax of their own,
+such as MDX expressions or template-engine tags like `{{ }}`, are outside what
the escaping
+covers.
+
== Known Vulnerabilities
Known CVEs in Apache Tika and their remediation are listed on the
diff --git
a/tika-core/src/main/java/org/apache/tika/sax/ToMarkdownContentHandler.java
b/tika-core/src/main/java/org/apache/tika/sax/ToMarkdownContentHandler.java
index ff4530db80..cd65a3e69c 100644
--- a/tika-core/src/main/java/org/apache/tika/sax/ToMarkdownContentHandler.java
+++ b/tika-core/src/main/java/org/apache/tika/sax/ToMarkdownContentHandler.java
@@ -833,18 +833,42 @@ public class ToMarkdownContentHandler extends
DefaultHandler {
}
/**
- * TeX goes out raw: the renderer's escaping ({@code \\mathcal}, {@code
u\\_0}) breaks
- * math-aware readers. In a table cell the renderer still escapes {@code
|}, which GFM
- * unescapes on read.
+ * TeX goes out as {@code $`...`$}, which GitHub and GitLab render as
math: a code span keeps
+ * it verbatim ({@code \\mathcal}, {@code u_0}) yet cannot become HTML, a
link or syntax of
+ * any other markdown dialect, whatever the document put in it. Text
without TeX delimiters
+ * is escaped like any other text.
*/
private void appendMath(String tex) {
- if (tex.isEmpty()) {
+ String math = collapseLineBreaks(tex).strip();
+ if (math.isEmpty()) {
return;
}
ensureInlineContainer();
- HtmlInline raw = new HtmlInline();
- raw.setLiteral(collapseLineBreaks(tex));
- stack.peek().appendChild(raw);
+ String inner = stripTexDelimiters(math);
+ if (inner == null || inner.isBlank()) {
+ stack.peek().appendChild(new Text(math));
+ return;
+ }
+ Code code = new Code();
+ code.setLiteral(inner.strip());
+ stack.peek().appendChild(new Text("$"));
+ stack.peek().appendChild(code);
+ stack.peek().appendChild(new Text("$"));
+ }
+
+ /** The TeX inside {@code $$...$$}, {@code $...$}, {@code \\(...\\)} or
{@code \\[...\\]}, else null. */
+ private static String stripTexDelimiters(String s) {
+ if (s.length() >= 4 && s.startsWith("$$") && s.endsWith("$$")) {
+ return s.substring(2, s.length() - 2);
+ }
+ if (s.length() >= 2 && s.startsWith("$") && s.endsWith("$")) {
+ return s.substring(1, s.length() - 1);
+ }
+ if (s.length() >= 4 && (s.startsWith("\\(") && s.endsWith("\\)")
+ || s.startsWith("\\[") && s.endsWith("\\]"))) {
+ return s.substring(2, s.length() - 2);
+ }
+ return null;
}
/**
@@ -879,16 +903,25 @@ public class ToMarkdownContentHandler extends
DefaultHandler {
return raw;
}
+ /**
+ * Tika writes {@code math inline} or {@code math display}. Parsers that
turn a document's
+ * style or structure names into a class (PDF custom types, Word styles)
yield one token,
+ * so they cannot produce both.
+ */
private static boolean isMathClass(String cls) {
if (cls == null) {
return false;
}
+ boolean math = false;
+ boolean mode = false;
for (String token : cls.trim().split("\\s+")) {
if (token.equals("math")) {
- return true;
+ math = true;
+ } else if (token.equals("inline") || token.equals("display")) {
+ mode = true;
}
}
- return false;
+ return math && mode;
}
private static HtmlInline newBrInline() {
diff --git
a/tika-core/src/test/java/org/apache/tika/sax/ToMarkdownContentHandlerTest.java
b/tika-core/src/test/java/org/apache/tika/sax/ToMarkdownContentHandlerTest.java
index 57378411b7..22ecf02e0c 100644
---
a/tika-core/src/test/java/org/apache/tika/sax/ToMarkdownContentHandlerTest.java
+++
b/tika-core/src/test/java/org/apache/tika/sax/ToMarkdownContentHandlerTest.java
@@ -26,9 +26,14 @@ import java.io.StringWriter;
import java.nio.charset.StandardCharsets;
import java.util.Locale;
import java.util.Random;
+import java.util.regex.Pattern;
+import org.commonmark.parser.Parser;
+import org.commonmark.renderer.html.HtmlRenderer;
import org.junit.jupiter.api.RepeatedTest;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
import org.xml.sax.Attributes;
import org.xml.sax.ContentHandler;
import org.xml.sax.helpers.AttributesImpl;
@@ -525,11 +530,11 @@ public class ToMarkdownContentHandlerTest {
endElement(handler, "p");
handler.endDocument();
String md = handler.toString();
- assertContains("see u\\_0 and $u_0 \\in \\mathcal{H} * x$", md);
+ assertContains("see u\\_0 and $`u_0 \\in \\mathcal{H} * x`$", md);
}
@Test
- public void testMathSpanInTableCellEscapesOnlyPipes() throws Exception {
+ public void testMathSpanInTableCellEscapesPipes() throws Exception {
ToMarkdownContentHandler handler = new ToMarkdownContentHandler();
handler.startDocument();
script(handler, "table", "tr", "th", "#h1", "/th", "th", "#h2", "/th",
"/tr", "tr", "td");
@@ -538,7 +543,7 @@ public class ToMarkdownContentHandlerTest {
endElement(handler, "span");
script(handler, "/td", "td", "#2", "/td", "/tr", "/table");
handler.endDocument();
- assertContains("|$\\\\|x\\\\|_1$|2|", handler.toString());
+ assertContains("|$`\\\\|x\\\\|_1`$|2|", handler.toString());
}
@Test
@@ -554,6 +559,114 @@ public class ToMarkdownContentHandlerTest {
assertContains("u\\_0\\*", handler.toString());
}
+ @Test
+ public void testMathSpanCannotOpenHtmlOrLinks() throws Exception {
+ String md = mathMarkdown("$a<b <img src=x onerror=alert(1)><!-- c -->
<http://x.example>"
+ + " [x](javascript:alert(1))$");
+ String html = renderHtml(md);
+ assertFalse(html.contains("<img"), html);
+ assertFalse(html.contains("<!--"), html);
+ assertFalse(html.contains("href"), html);
+ assertContains("$`a<b <img", md);
+ }
+
+ @Test
+ public void testBackticksCannotCloseMathCodeSpan() throws Exception {
+ for (String tex : new String[]{"` <img src=x>", "a`b``c```d <img
src=x>", "<img src=x> `"}) {
+ String html = renderHtml(mathMarkdown("$" + tex + "$"));
+ assertContains("<code>" + tex.strip().replace("<",
"<").replace(">", ">") + "</code>",
+ html);
+ assertFalse(html.contains("<img"), html);
+ }
+ }
+
+ @Test
+ public void testUndelimitedMathSpanIsEscaped() throws Exception {
+ String md = mathMarkdown("```u_0 \\", "<img src=x onerror=alert(1)>");
+ assertContains("u\\_0", md);
+ assertFalse(renderHtml(md).contains("<img"), md);
+ assertFalse(renderHtml(md).contains("<pre>"), md);
+ }
+
+ /**
+ * Whatever text and math spans hold, the markdown renders to no element a
reader could
+ * act on: no raw HTML, comment, link or image.
+ */
+ @Test
+ public void testRandomMathSpansRenderNoHtmlOrLinks() throws Exception {
+ String[] pieces = {"<", ">", "[", "]", "(", ")", "!", ":", " ", "x",
"img src=x",
+ "javascript:alert(1)", "http://e.x";, "\\", "$", "\\(", "\\)",
"\\[", "\\]", "`",
+ "```", "*", "&", "#", "-", "\"", "|", "<!--", "]:"};
+ Pattern acted = Pattern.compile(
+ "<(?!/?(p|code|em|strong|blockquote|ul|ol|li|pre|h[1-6])>|br
?/?>|hr ?/?>"
+ + "|ol start=\"[0-9]+\">|pre><code
class=\"language-)");
+ long seed = new Random().nextLong();
+ Random r = new Random(seed);
+ for (int iter = 0; iter < 5000; iter++) {
+ ToMarkdownContentHandler handler = new ToMarkdownContentHandler();
+ handler.startDocument();
+ startElement(handler, "p");
+ for (int k = 1 + r.nextInt(3); k > 0; k--) {
+ chars(handler, randomText(r, pieces, r.nextInt(4)));
+ startElement(handler, "span", "class", "math inline");
+ String tex = randomText(r, pieces, 1 + r.nextInt(8));
+ chars(handler, r.nextBoolean() ? "$" + tex + "$" : tex);
+ endElement(handler, "span");
+ }
+ endElement(handler, "p");
+ handler.endDocument();
+ String md = handler.toString();
+ String html = renderHtml(md);
+ assertFalse(acted.matcher(html).find(), "seed=" + seed + " md=" +
md + " html=" + html);
+ }
+ }
+
+ private static String randomText(Random r, String[] pieces, int n) {
+ StringBuilder sb = new StringBuilder();
+ for (int i = 0; i < n; i++) {
+ sb.append(pieces[r.nextInt(pieces.length)]);
+ }
+ return sb.toString();
+ }
+
+ private static String mathMarkdown(String tex) throws Exception {
+ return mathMarkdown(tex, "");
+ }
+
+ private static String mathMarkdown(String tex, String after) throws
Exception {
+ ToMarkdownContentHandler handler = new ToMarkdownContentHandler();
+ handler.startDocument();
+ startElement(handler, "p");
+ startElement(handler, "span", "class", "math inline");
+ chars(handler, tex);
+ endElement(handler, "span");
+ if (!after.isEmpty()) {
+ chars(handler, after);
+ }
+ endElement(handler, "p");
+ handler.endDocument();
+ return handler.toString();
+ }
+
+ private static String renderHtml(String md) {
+ return
HtmlRenderer.builder().build().render(Parser.builder().build().parse(md));
+ }
+
+ // PDF custom structure types and Word style names become a single class
token
+ @ParameterizedTest
+ @ValueSource(strings = {"math", "math_inline", "math-display"})
+ public void testDocumentDerivedMathClassIsEscaped(String cls) throws
Exception {
+ ToMarkdownContentHandler handler = new ToMarkdownContentHandler();
+ handler.startDocument();
+ startElement(handler, "p");
+ startElement(handler, "span", "class", cls);
+ chars(handler, "$u_0$");
+ endElement(handler, "span");
+ endElement(handler, "p");
+ handler.endDocument();
+ assertContains("u\\_0", handler.toString());
+ }
+
@Test
public void testEmphasisEdgeWhitespaceAndEmptyDropped() throws Exception {
ToMarkdownContentHandler handler = new ToMarkdownContentHandler();
@@ -661,7 +774,7 @@ public class ToMarkdownContentHandlerTest {
chars(handler, " is small");
endElement(handler, "p");
handler.endDocument();
- assertContains("$x$ is small", handler.toString());
+ assertContains("$`x`$ is small", handler.toString());
}
@Test
diff --git
a/tika-parsers/tika-parsers-ml/tika-vlm/src/test/java/org/apache/tika/parser/vlm/MarkdownToXHTMLEmitterTest.java
b/tika-parsers/tika-parsers-ml/tika-vlm/src/test/java/org/apache/tika/parser/vlm/MarkdownToXHTMLEmitterTest.java
index 0b167a02d9..4caa166dcd 100644
---
a/tika-parsers/tika-parsers-ml/tika-vlm/src/test/java/org/apache/tika/parser/vlm/MarkdownToXHTMLEmitterTest.java
+++
b/tika-parsers/tika-parsers-ml/tika-vlm/src/test/java/org/apache/tika/parser/vlm/MarkdownToXHTMLEmitterTest.java
@@ -411,9 +411,9 @@ public class MarkdownToXHTMLEmitterTest {
MarkdownToXHTMLEmitter.emit(md, out);
out.endDocument();
String written = out.toString();
- assertContains("norm $\\|\\cdot\\|$ with $u_0 \\in \\mathcal{H}$",
written);
- // GFM unescapes \| in a cell before math, so a renderer sees $\|x\|$
- assertContains("|$\\\\|x\\\\|$|2|", written);
+ assertContains("norm $`\\|\\cdot\\|`$ with $`u_0 \\in \\mathcal{H}`$",
written);
+ // GFM unescapes \| in a cell before the code span, so a renderer sees
\|x\|
+ assertContains("|$`\\\\|x\\\\|`$|2|", written);
}
@Test
