airborne12 opened a new pull request, #67179:
URL: https://github.com/apache/doris/pull/67179
Bumps contrib/clucene from abe2b71a1c1 to 08d45e541fe, picking up
apache/doris-thirdparty#368.
Problem
-------
BE dies with SIGSEGV while writing an inverted index when a term contains a
raw NUL byte in the middle (JSON \u0000 decodes to this). The crash
reproduces
on any table with an inverted index over such data, and restarts in a loop.
Root cause
----------
In SDocumentWriter.cpp the two paths disagree on what a term is:
* addPosition() hashes and dedups with eq(p->term_, term), i.e. the FULL
string_view length, so "svc\0A" and "svc\0B" are two distinct postings;
* comparePostings() stops at CLUCENE_END_OF_WORD, so the very same two
postings compare EQUAL.
A run of mutually "equal" but distinct postings makes quickSort()'s partition
loop peel exactly one element per call, so the recursion depth becomes O(n)
(measured depth/N = 1.000). Each frame is 128 bytes, so ~65k levels exhaust
the
8 MiB thread stack. The stack is so far gone that the kernel cannot build a
signal frame and force-kills the process (si_code=SI_KERNEL), which is why no
handler output or stack trace ever reaches the log.
Fix
---
doris-thirdparty#368 rejects such terms in addPosition(), before they can
enter
the postings hash, so the degenerate equal-run can no longer form. The row
now
fails with a clear CLuceneError, which
InvertedIndexColumnWriter::add_document()
already converts into INVERTED_INDEX_CLUCENE_ERROR, instead of taking the BE
down.
Verification
------------
Reproducer: 80k terms shaped "svc\0<i>", untokenized inverted index, 8 MiB
stack.
before: SIGSEGV, exit 139
after : 80000 rows rejected, exit 0, 0 postings reach the sort
Performance on clean data (the guard is one compare + a never-taken branch):
perf instructions, 1M tokens : 10,116,997,256 -> 10,118,997,132
= +2.00 instructions/token (+0.020%)
index output (.fnm/.frq/.nrm/.tii/.tis) : byte-identical
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
