This is an automated email from the ASF dual-hosted git repository.
airborne12 pushed a commit to branch clucene
in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git
The following commit(s) were added to refs/heads/clucene by this push:
new 384edadeefa [fix](index) Recompute destination token counts during
compaction (#420)
384edadeefa is described below
commit 384edadeefa42c77280c197357f1fe2d03089150
Author: arthur.xqw <[email protected]>
AuthorDate: Fri Oct 23 16:18:31 2026 +0800
[fix](index) Recompute destination token counts during compaction (#420)
* [fix](index) Recompute destination token counts during compaction
### What problem does this PR solve?
Issue Number: None
Related PR: Apache Doris integration PR (to be linked)
Problem Summary:
`IndexWriter::indexCompaction` filters deleted documents through the row-ID
translation map but copies the sum of every source field's token count into the
first destination index and writes zero to the remaining destinations.
Consequently BM25 uses stale average document lengths after deletes and
incorrect per-segment statistics after a split.
For two visible documents of lengths 1 and 16 and one deleted document of
length 256, the clean index contains 17 tokens. Native index compaction retains
a token count of 273: on Doris 4.1.5 the visible documents score 0.3069872 /
0.33349273, while rebuilding the same documents scores 0.28530657 / 0.20084909
and reverses their ranking.
Accumulate exact term frequencies from retained postings into each
destination field while merging terms, then write those destination totals
alongside the remapped norms. This also repairs stale source totals when their
postings still retain frequencies. It neither changes BM25's formula nor
changes visibility/statistics before compaction.
An index with norms but without term frequencies cannot supply an exact
count: byte norms are quantized. Reject native compaction for that case before
opening output writers; the companion Doris change selects its existing
raw-column index rebuild path. Indexes without norms continue to compact
without requiring frequencies.
### Release note
Correct BM25 token statistics after inverted-index compaction removes or
redistributes documents.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test (pre-fix SQL reproduction on Doris 4.1.5)
- [ ] Regression test
- Native CLucene ASAN build, using the repository's CMake/`cl_test`
entrypoints.
- `cl_test -v -p IndexCompaction IndexCompress IndexCompressV3
testBlockMaxScoreV3`: 16 tests passed.
- Rebuilt the original `08d45e541fef1e48dc4ab00d2d5fde9569b49ee3`
implementation with the new tests: 2 new tests fail (expected destination token
count 42, actual 315; missing-frequency guard absent); the 4 existing
compaction tests pass.
- Coverage includes deleted long documents, a non-exact byte-norm
length (41 tokens), two source/destination indexes, multiple fields, empty
field values, and indexes with/without norms or frequencies.
- A full Doris master binary has not been built or exercised locally;
the companion PR records its integration validation separately.
- Behavior changed:
- [x] Yes. Compacted norms carry exact destination token totals. Native
compaction rejects norms-bearing indexes without frequencies; callers must
rebuild them.
- Does this need documentation?
- [x] No. Correctness fix; no new setting or on-disk format.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
* [test](index) Cover single-destination compaction after deletes
### What problem does this PR solve?
Issue Number: None
Related PR: https://github.com/apache/doris-thirdparty/pull/420
Problem Summary:
Mirror the ordinary SQL DELETE reproduction at the CLucene boundary. Each
source contains a surviving document and a deleted 256-token document.
Verify that merging into one destination retains two documents and exactly
17 body tokens instead of the 529 source tokens, with independent title
counts.
The old implementation fails this assertion; the existing fix passes it.
### Release note
None (test coverage for the existing compaction fix).
### Check List (For Author)
- Test: 17 CLucene ASAN tests pass. Original implementation fails the three
new compaction regressions; the other 14 selected tests pass.
- Behavior changed: No (test only).
- Does this need documentation: No.
---------
Co-authored-by: xuquanwei <[email protected]>
---
src/core/CLucene/index/IndexWriter.cpp | 41 ++++++---
src/core/CLucene/index/IndexWriter.h | 5 +-
src/test/index/TestIndexCompaction.cpp | 151 ++++++++++++++++++++++++++++++++-
3 files changed, 182 insertions(+), 15 deletions(-)
diff --git a/src/core/CLucene/index/IndexWriter.cpp
b/src/core/CLucene/index/IndexWriter.cpp
index 7ad04859e61..8f9e950331a 100644
--- a/src/core/CLucene/index/IndexWriter.cpp
+++ b/src/core/CLucene/index/IndexWriter.cpp
@@ -1284,7 +1284,7 @@ void
IndexWriter::indexCompaction(std::vector<lucene::store::Directory *> &src_d
// first level vector index is src_index_id
// <std::wstring, ValueArray<uint8_t>> key is field name, value is the
norm of src_doc_id
std::vector<std::map<std::wstring, std::vector<uint8_t>>>
srcFieldNormsMapValues(numIndices);
- std::map<std::wstring, uint64_t> srcFieldTotalTermCountMap;
+ std::vector<std::map<std::wstring, uint64_t>>
destFieldTokenCounts(numDestIndexes);
try {
// check hasProx, indexVersion
@@ -1322,7 +1322,18 @@ void
IndexWriter::indexCompaction(std::vector<lucene::store::Directory *> &src_d
FieldInfo* fi = fieldInfos->fieldInfo(i);
// Is this Field indexed and field need norms ?
if (fi->isIndexed && !fi->omitNorms) {
+ // Without frequencies, byte norms cannot recover exact
lengths
+ // (e.g. 41 tokens encode as 40). The caller must rebuild
such an
+ // index from its source documents instead of publishing
stale totals.
+ if (!fi->hasProx) {
+ _CLTHROWA(CL_ERR_UnsupportedOperation,
+ "Index compaction with norms requires term
frequencies; "
+ "rebuild the index from source documents");
+ }
hasNorms = true;
+ for (auto& counts : destFieldTokenCounts) {
+ counts.emplace(fi->name, 0);
+ }
}
}
}
@@ -1349,8 +1360,6 @@ void
IndexWriter::indexCompaction(std::vector<lucene::store::Directory *> &src_d
srcFieldNormsMapValues[srcIndex][fi->name].emplace_back(
normBuffer.values[j]);
}
- srcFieldTotalTermCountMap[fi->name] +=
- reader->sumTotalTermFreq(fi->name).value_or(0);
}
}
}
@@ -1427,11 +1436,11 @@ void
IndexWriter::indexCompaction(std::vector<lucene::store::Directory *> &src_d
}
/// merge terms
- mergeTerms(hasProx, hasNorms, indexVersion, destFieldNormsMapValues);
+ mergeTerms(hasProx, hasNorms, indexVersion, destFieldNormsMapValues,
destFieldTokenCounts);
/// merge norms if have
if (hasNorms){
- mergeNorms(dest_index_docs, srcFieldTotalTermCountMap,
srcFieldNormsMapValues, normsOutputList);
+ mergeNorms(dest_index_docs, destFieldTokenCounts,
srcFieldNormsMapValues, normsOutputList);
}
/// merge null_bitmap
@@ -1665,7 +1674,8 @@ protected:
};
void IndexWriter::mergeTerms(bool hasProx, bool hasNorms, IndexVersion
indexVersion,
- const std::vector<std::map<std::wstring, std::vector<uint8_t>>>&
destFieldNormsMapValues) {
+ const std::vector<std::map<std::wstring, std::vector<uint8_t>>>&
destFieldNormsMapValues,
+ std::vector<std::map<std::wstring, uint64_t>>& destFieldTokenCounts) {
auto queue = _CLNEW SegmentMergeQueue(readers.size());
auto numSrcIndexes = readers.size();
//std::vector<TermPositions *> postingsList(numSrcIndexes);
@@ -1766,6 +1776,8 @@ void IndexWriter::mergeTerms(bool hasProx, bool hasNorms,
IndexVersion indexVers
std::vector<int32_t> dfs(numDestIndexes, 0);
std::vector<int32_t> lastDocs(numDestIndexes, 0);
+ const bool fieldHasNorms =
!fieldInfos->fieldInfo(smallestTerm->field())->omitNorms;
+ std::vector<uint64_t> termTokenCounts(fieldHasNorms ? numDestIndexes :
0, 0);
while (destPostingQueues->size()) {
if (destPostingQueues->top() != nullptr) {
auto destDoc = destPostingQueues->pop();
@@ -1773,6 +1785,12 @@ void IndexWriter::mergeTerms(bool hasProx, bool
hasNorms, IndexVersion indexVers
auto destDocId = destDoc->destDocId;
auto destFreq = destDoc->destFreq;
auto& descPositions = destDoc->destPositions;
+ // Only postings with a valid destination reach this queue.
Summing
+ // their frequencies excludes deleted documents and attributes
each
+ // token to its destination segment, including after a segment
split.
+ if (fieldHasNorms) {
+ termTokenCounts[destIdx] += destFreq;
+ }
if (infoStream != nullptr) {
for (int i = 0; i < _trans_vec.size(); ++i) {
// find pair < destIdx, destDocId > in _trans_vec[i]
to get the index of the pair
@@ -1882,6 +1900,9 @@ void IndexWriter::mergeTerms(bool hasProx, bool hasNorms,
IndexVersion indexVers
}
for (int i = 0; i < numDestIndexes; ++i) {
+ if (fieldHasNorms) {
+ destFieldTokenCounts[i].at(smallestTerm->field()) +=
termTokenCounts[i];
+ }
if (dfs[i] == 0) {
if (infoStream != nullptr) {
std::string name =
lucene_wcstoutf8string(smallestTerm->text(), smallestTerm->textLength());
@@ -1969,7 +1990,7 @@ void IndexWriter::mergeTerms(bool hasProx, bool hasNorms,
IndexVersion indexVers
}
void IndexWriter::mergeNorms(std::vector<uint32_t> dest_index_docs,
- std::map<std::wstring, uint64_t>
srcFieldTotalTermCountMap,
+ const std::vector<std::map<std::wstring,
uint64_t>>& destFieldTokenCounts,
std::vector<std::map<std::wstring,
std::vector<uint8_t>>> srcFieldNormsMapValues,
std::vector<lucene::store::IndexOutput *>
normsOutputList) {
//Func - Merges the norms for all fields
@@ -2028,11 +2049,7 @@ void IndexWriter::mergeNorms(std::vector<uint32_t>
dest_index_docs,
std::fill(destFieldNormsMap[fieldName].begin(),
destFieldNormsMap[fieldName].end(), 0);
}
- if (i == 0) {
-
normsOutputList[i]->writeLong(srcFieldTotalTermCountMap[fieldName]);
- } else {
- normsOutputList[i]->writeLong(0);
- }
+
normsOutputList[i]->writeLong(destFieldTokenCounts[i].at(fieldName));
auto& destFieldNorms = destFieldNormsMap[fieldName];
normsOutputList[i]->writeBytes(destFieldNorms.data(),
destDocCount);
}
diff --git a/src/core/CLucene/index/IndexWriter.h
b/src/core/CLucene/index/IndexWriter.h
index de1bab67bde..56a9d7157d9 100644
--- a/src/core/CLucene/index/IndexWriter.h
+++ b/src/core/CLucene/index/IndexWriter.h
@@ -329,11 +329,12 @@ public:
void writeFields(lucene::store::Directory* d, std::string segment);
// merge terms and write files
void mergeTerms(bool hasProx, bool hasNorms, IndexVersion indexVersion,
- const std::vector<std::map<std::wstring, std::vector<uint8_t>>>&
destFieldNormsMapValues);
+ const std::vector<std::map<std::wstring, std::vector<uint8_t>>>&
destFieldNormsMapValues,
+ std::vector<std::map<std::wstring, uint64_t>>&
destFieldTokenCounts);
// merge norms and write files
void mergeNorms(
std::vector<uint32_t> dest_index_docs,
- std::map<std::wstring, uint64_t> srcFieldTotalTermCountMap,
+ const std::vector<std::map<std::wstring, uint64_t>>&
destFieldTokenCounts,
std::vector<std::map<std::wstring, std::vector<uint8_t>>>
srcFieldNormsMapValues,
std::vector<lucene::store::IndexOutput*> normsOutputList);
// merge null_bitmap
diff --git a/src/test/index/TestIndexCompaction.cpp
b/src/test/index/TestIndexCompaction.cpp
index 1d49c597886..18e7ff2f4ee 100644
--- a/src/test/index/TestIndexCompaction.cpp
+++ b/src/test/index/TestIndexCompaction.cpp
@@ -21,6 +21,7 @@
#include <vector>
#include "CLucene/debug/mem.h"
#include "test.h"
+#include "store/MockRAMDirectory.h"
#include "CLucene/debug/error.h"
#include "CLucene/index/IndexWriter.h"
#include "CLucene/store/IndexInput.h"
@@ -332,6 +333,151 @@ void TestMergeNullBitmapIgnoreDoc(CuTest *tc) {
+static void writeNormsIndex(RAMDirectory& dir,
+ const std::vector<std::pair<std::string,
std::string>>& documents,
+ bool norms = true, bool frequencies = true) {
+ lucene::analysis::SimpleAnalyzer<char> analyzer;
+ analyzer.set_stopwords(nullptr);
+ lucene::util::SStringReader<char> bodyReader;
+ lucene::util::SStringReader<char> titleReader;
+ lucene::document::Document doc;
+ auto* body = _CLNEW lucene::document::Field(
+ _T("body"), lucene::document::Field::STORE_NO |
lucene::document::Field::INDEX_TOKENIZED);
+ auto* title = _CLNEW lucene::document::Field(
+ _T("title"), lucene::document::Field::STORE_NO |
lucene::document::Field::INDEX_TOKENIZED);
+ for (auto* field : {body, title}) {
+ field->setOmitNorms(!norms);
+ field->setOmitTermFreqAndPositions(!frequencies);
+ doc.add(*field);
+ }
+ IndexWriter writer(&dir, &analyzer, true);
+ writer.setMaxBufferedDocs(-1);
+ writer.setRAMBufferSizeMB(512);
+ writer.setUseCompoundFile(false);
+ for (const auto& values : documents) {
+ bodyReader.init(values.first.data(), values.first.size(), false);
+ titleReader.init(values.second.data(), values.second.size(), false);
+ body->setValue(analyzer.tokenStream(body->name(), &bodyReader), true);
+ title->setValue(analyzer.tokenStream(title->name(), &titleReader),
true);
+ writer.addDocument(&doc);
+ }
+ writer.close();
+}
+
+static std::string repeatedToken(const char* token, size_t count) {
+ std::string text;
+ for (size_t i = 0; i < count; ++i) {
+ text += token;
+ text += ' ';
+ }
+ return text;
+}
+
+void TestCompactionTokenCounts(CuTest* tc) {
+ RAMDirectory source0;
+ RAMDirectory source1;
+ writeNormsIndex(source0, {{"alpha", "one two"},
+ {repeatedToken("deleted", 257),
repeatedToken("gone", 73)},
+ {"alpha alpha " + repeatedToken("omega", 14),
""}});
+ // Length 41 is not exactly representable in a byte norm. The token total
+ // must come from the retained postings, not from decoding the norms.
+ writeNormsIndex(source1, {{"alpha " + repeatedToken("omega", 40), "one"},
+ {"", "one two three"}});
+ // Compaction closes its output directories. The test directory keeps the
+ // files available for reopening, as filesystem directories do in Doris.
+ MockRAMDirectory destination0;
+ MockRAMDirectory destination1;
+ destination0.setRandomIOExceptionRate(0.0, 0);
+ destination1.setRandomIOExceptionRate(0.0, 0);
+ RAMDirectory scratch;
+ lucene::analysis::SimpleAnalyzer<char> analyzer;
+ IndexWriter writer(&scratch, &analyzer, true);
+ std::vector<lucene::store::Directory*> sources {&source0, &source1};
+ std::vector<lucene::store::Directory*> destinations {&destination0,
&destination1};
+ std::vector<std::vector<std::pair<uint32_t, uint32_t>>> mapping {
+ {{0, 0}, {UINT32_MAX, UINT32_MAX}, {1, 0}}, {{0, 1}, {1, 1}}};
+ writer.indexCompaction(sources, destinations, mapping, {2, 2});
+ writer.close();
+ std::vector<uint64_t> bodyCounts;
+ std::vector<uint64_t> titleCounts;
+ for (auto* directory : destinations) {
+ auto* reader = IndexReader::open(directory);
+ bodyCounts.push_back(reader->sumTotalTermFreq(_T("body")).value());
+ titleCounts.push_back(reader->sumTotalTermFreq(_T("title")).value());
+ reader->close();
+ _CLDELETE(reader);
+ }
+ CuAssertIntEquals(tc, _T("first destination body tokens"), 42,
bodyCounts[0]);
+ CuAssertIntEquals(tc, _T("second destination body tokens"), 16,
bodyCounts[1]);
+ CuAssertIntEquals(tc, _T("first destination title tokens"), 3,
titleCounts[0]);
+ CuAssertIntEquals(tc, _T("second destination title tokens"), 3,
titleCounts[1]);
+}
+
+void TestCompactionDeletedDocumentsSingleDestination(CuTest* tc) {
+ // Match the ordinary DELETE regression: each source has a surviving
document
+ // and a deleted 256-token document. The destination has 17 tokens, not
529.
+ const std::string deleted = repeatedToken("alpha", 240) +
repeatedToken("omega", 16);
+ RAMDirectory source0;
+ RAMDirectory source1;
+ writeNormsIndex(source0, {{"alpha", "one"}, {deleted, "gone"}});
+ writeNormsIndex(source1,
+ {{"alpha alpha " + repeatedToken("omega", 14), "two"},
{deleted, "gone"}});
+ MockRAMDirectory destination;
+ destination.setRandomIOExceptionRate(0.0, 0);
+ RAMDirectory scratch;
+ lucene::analysis::SimpleAnalyzer<char> analyzer;
+ IndexWriter writer(&scratch, &analyzer, true);
+ std::vector<lucene::store::Directory*> sources {&source0, &source1};
+ std::vector<std::vector<std::pair<uint32_t, uint32_t>>> mapping {
+ {{0, 0}, {UINT32_MAX, UINT32_MAX}}, {{0, 1}, {UINT32_MAX,
UINT32_MAX}}};
+ writer.indexCompaction(sources, {&destination}, mapping, {2});
+ writer.close();
+ auto* reader = IndexReader::open(&destination);
+ const auto bodyTokens = reader->sumTotalTermFreq(_T("body")).value();
+ const auto titleTokens = reader->sumTotalTermFreq(_T("title")).value();
+ const auto documents = reader->maxDoc();
+ reader->close();
+ _CLDELETE(reader);
+ CuAssertIntEquals(tc, _T("surviving documents"), 2, documents);
+ CuAssertIntEquals(tc, _T("surviving body tokens"), 17, bodyTokens);
+ CuAssertIntEquals(tc, _T("surviving title tokens"), 2, titleTokens);
+}
+
+void TestCompactionWithoutFrequencies(CuTest* tc) {
+ for (bool norms : {false, true}) {
+ RAMDirectory source;
+ writeNormsIndex(source, {{"alpha alpha", "one one"}, {"deleted",
"gone"}},
+ norms, false);
+ MockRAMDirectory destination;
+ destination.setRandomIOExceptionRate(0.0, 0);
+ RAMDirectory scratch;
+ lucene::analysis::SimpleAnalyzer<char> analyzer;
+ IndexWriter writer(&scratch, &analyzer, true);
+ std::vector<lucene::store::Directory*> sources {&source};
+ std::vector<std::vector<std::pair<uint32_t, uint32_t>>> mapping {
+ {{0, 0}, {UINT32_MAX, UINT32_MAX}}};
+ bool rejected = false;
+ try {
+ writer.indexCompaction(sources, {&destination}, mapping, {1});
+ } catch (CLuceneError& error) {
+ CuAssertIntEquals(tc, _T("missing frequencies require a rebuild"),
+ CL_ERR_UnsupportedOperation, error.number());
+ rejected = true;
+ }
+ writer.close();
+ CuAssertIntEquals(tc, _T("only norms need exact frequencies"), norms,
rejected);
+ if (!norms) {
+ auto* reader = IndexReader::open(&destination);
+ const auto tokens = reader->sumTotalTermFreq(_T("body"));
+ const auto documents = reader->maxDoc();
+ reader->close();
+ _CLDELETE(reader);
+ CuAssertIntEquals(tc, _T("norms remain omitted"), false,
tokens.has_value());
+ CuAssertIntEquals(tc, _T("retained document"), 1, documents);
+ }
+ }
+}
+
CuSuite* testIndexCompaction() {
CuSuite* suite = CuSuiteNew(_T("CLucene Index Compaction Test"));
@@ -339,6 +485,9 @@ CuSuite* testIndexCompaction() {
SUITE_ADD_TEST(suite, TestMergeNullBitmapEmptySrc);
SUITE_ADD_TEST(suite, TestMergeNullBitmapEmptyIndexSrcBitmapValues);
SUITE_ADD_TEST(suite, TestMergeNullBitmapIgnoreDoc);
+ SUITE_ADD_TEST(suite, TestCompactionTokenCounts);
+ SUITE_ADD_TEST(suite, TestCompactionDeletedDocumentsSingleDestination);
+ SUITE_ADD_TEST(suite, TestCompactionWithoutFrequencies);
return suite;
-}
\ No newline at end of file
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
