This is an automated email from the ASF dual-hosted git repository.
airborne12 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e59320b9571 [fix](inverted index) update clucene to reject terms
containing an embedded NUL byte (#67179)
e59320b9571 is described below
commit e59320b9571f1ac7796ad6562097730e280b3b60
Author: Jack <[email protected]>
AuthorDate: Thu Aug 27 10:01:20 2026 +0800
[fix](inverted index) update clucene to reject terms containing an embedded
NUL byte (#67179)
Bumps contrib/clucene from abe2b71a1c1 to 08d45e541fe, picking up
apache/doris-thirdparty#368.
Problem
-------
BE dies with SIGSEGV while writing an inverted index when a term
contains a
raw NUL byte in the middle (JSON \u0000 decodes to this). The crash
reproduces
on any table with an inverted index over such data, and restarts in a
loop.
Root cause
----------
In SDocumentWriter.cpp the two paths disagree on what a term is:
* addPosition() hashes and dedups with eq(p->term_, term), i.e. the FULL
string_view length, so "svc\0A" and "svc\0B" are two distinct postings;
* comparePostings() stops at CLUCENE_END_OF_WORD, so the very same two
postings compare EQUAL.
A run of mutually "equal" but distinct postings makes quickSort()'s
partition
loop peel exactly one element per call, so the recursion depth becomes
O(n)
(measured depth/N = 1.000). Each frame is 128 bytes, so ~65k levels
exhaust the
8 MiB thread stack. The stack is so far gone that the kernel cannot
build a
signal frame and force-kills the process (si_code=SI_KERNEL), which is
why no
handler output or stack trace ever reaches the log.
Fix
---
doris-thirdparty#368 rejects such terms in addPosition(), before they
can enter
the postings hash, so the degenerate equal-run can no longer form. The
row now
fails with a clear CLuceneError, which
InvertedIndexColumnWriter::add_document()
already converts into INVERTED_INDEX_CLUCENE_ERROR, instead of taking
the BE down.
Verification
------------
Reproducer: 80k terms shaped "svc\0<i>", untokenized inverted index, 8
MiB stack.
before: SIGSEGV, exit 139
after : 80000 rows rejected, exit 0, 0 postings reach the sort
Performance on clean data (the guard is one compare + a never-taken
branch):
perf instructions, 1M tokens : 10,116,997,256 -> 10,118,997,132
= +2.00 instructions/token (+0.020%)
index output (.fnm/.frq/.nrm/.tii/.tis) : byte-identical
---
contrib/clucene | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/contrib/clucene b/contrib/clucene
index abe2b71a1c1..08d45e541fe 160000
--- a/contrib/clucene
+++ b/contrib/clucene
@@ -1 +1 @@
-Subproject commit abe2b71a1c1afd3120ce39740d2346d07a5c2309
+Subproject commit 08d45e541fef1e48dc4ab00d2d5fde9569b49ee3
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
