airborne12 commented on code in PR #67538:
URL: https://github.com/apache/doris/pull/67538#discussion_r3985299471
##########
be/src/storage/index/snii/snii_index_reader.cpp:
##########
@@ -402,6 +408,33 @@ Status execute_snii_query(const
::doris::snii::reader::LogicalIndexReader& logic
max_expansions);
emitted_to_sink = true;
break;
+ case InvertedIndexQueryType::LIKE_GRAM_QUERY:
+ case InvertedIndexQueryType::REGEXP_GRAM_QUERY: {
+ // Compile against the same physical dictionary that supplies the
postings. Current
+ // policies can differ from the scheme that was used when this segment
was written.
+ const auto& scheme = logical_reader.gram_scheme();
Review Comment:
Fixed in c2722622203. `SniiIndexReader` now resolves the scheme the current
analyzer cuts with for every analyzed query type (MATCH_ANY / MATCH_ALL /
MATCH_PHRASE / MATCH_PHRASE_PREFIX / MATCH_PHRASE_EDGE) -- the caller's
provider, else the analyzer named by the index properties -- compares it with
the segment's persisted scheme once the logical reader is open, and returns
`INVERTED_INDEX_EVALUATE_SKIPPED` on a mismatch so the scalar predicate
answers. Analyzed queries on a gram-family index no longer enter the result
cache, whose key carries no scheme identity.
Tests:
`LikeGramBindingTest.MatchIsSkippedWhenTheCurrentSchemeDiffersFromTheSegments`
(a dense-4 analyzer over a dense-3 segment is skipped; the segment's own scheme
answers the query), and `test_gram_policy_recovery` now runs MATCH_ANY /
MATCH_ALL with the index on and off on the recovered table and on the mixed
dense-3 / dense-4 table.
##########
be/src/storage/index/snii/snii_index_writer.cpp:
##########
@@ -44,13 +46,97 @@
SniiIndexColumnWriter::SniiIndexColumnWriter(IndexFileWriter* index_file_writer,
_index_meta(index_meta),
_is_char(value_type == FieldType::OLAP_FIELD_TYPE_CHAR) {}
+// Gram-family detection (Rulings R21/R22): the scheme comes only from the
single analyzer
+// provider the writer created itself -- the same provider both produces the
actual tokenizer and
+// answers "am I gram family?", so the two cannot drift. A built-in analyzer
+// (standard/english/...) goes through BuiltinAnalyzerProvider and the base
class default, which
+// is always nullopt, and the policy manager is never consulted (consulting it
would throw
+// "Policy not found" and make every built-in-analyzer index impossible to
build).
+void SniiIndexColumnWriter::_apply_gram_family_scheme(
+ const inverted_index::AnalyzerProviderPtr& analyzer_provider) {
+ if (analyzer_provider != nullptr) {
+ _gram_scheme = analyzer_provider->gram_scheme();
+ }
+ // An index-level char_filter is wrapped around the reader by the writer
itself
+ // (create_reader) and is invisible to the provider: once one exists, the
stored term is no
+ // longer equal to GramExtractor.extract(raw column value), breaking the
row invariant the
+ // query side (phase C) relies on, so this is treated as "not gram family"
(fail-safe, for
+ // the same reason as R22).
+ if (!_analyzer_config.char_filter_map.empty()) {
+ _gram_scheme.reset();
+ }
+ DCHECK(!_gram_scheme.has_value() || _should_analyzer);
+ if (!_gram_scheme.has_value() || !_has_positions) {
+ return;
+ }
+ // R15: a gram-family hit forces a degradation to docs-only (the gram
index does not support
+ // phrase positions), and it has to happen before SpimiTermBuffer is fixed
by _has_positions.
+ LOG(INFO) << "gram-family analyzer forces docs-only index, ignoring
support_phrase for index "
+ << _index_meta->index_id();
+ _has_positions = false;
+ _config = ::doris::snii::format::IndexConfig::kDocsOnly;
+}
+
+// Arms the density solve for this segment. The configured density stays as
the fallback: it is
+// what a segment gets when the feature is off, when the sample carries no
window of the
+// promised length, or when nothing at all is written.
+void SniiIndexColumnWriter::_arm_density_calibration() {
+ if (!_gram_scheme.has_value() ||
!config::enable_gram_index_adaptive_density) {
Review Comment:
Fixed in 2cc0776c891: calibration is armed only for `SPARSE` schemes. A
dense writer cuts rows as they arrive and charges no sample. Test:
`GramDensityCalibrationTest.DenseSchemesAreNotCalibrated`.
##########
be/src/storage/index/snii/snii_index_writer.cpp:
##########
@@ -211,6 +281,20 @@ Status SniiIndexColumnWriter::add_values(const std::string
/*name*/, const void*
}
const auto* v = reinterpret_cast<const Slice*>(values);
for (size_t i = 0; i < count; ++i) {
+ if (_density_calibrating) {
+ // Held back, not dropped: this row is tokenized once the rate is
known. Feeding
+ // the solver here costs one linear pass over the row and retains
nothing of it.
+ _density_solver->observe(std::string_view(v->data, v->size));
+ _density_sample.emplace_back(_rid, std::string(v->data, v->size));
Review Comment:
Fixed in 2cc0776c891: each held-back row is charged its payload plus
`sizeof(std::pair<uint32_t, std::string>)`, so the cap is reached by row count
as well as by bytes (400 empty rows finish a 4 KiB budget), and the solver's
256 KiB histogram is charged to the memory reporter for exactly as long as it
is resident. The monotonic-queue scratch is reused across rows and bounded by
the longest row, so it is not charged separately. Tests:
`GramDensityCalibrationTest.EmptyRowsStillReachTheSampleCap`,
`CharPaddingIsNeitherEvidenceNorRetained`.
##########
be/src/storage/index/snii/snii_index_writer.cpp:
##########
@@ -211,6 +281,20 @@ Status SniiIndexColumnWriter::add_values(const std::string
/*name*/, const void*
}
const auto* v = reinterpret_cast<const Slice*>(values);
for (size_t i = 0; i < count; ++i) {
+ if (_density_calibrating) {
+ // Held back, not dropped: this row is tokenized once the rate is
known. Feeding
+ // the solver here costs one linear pass over the row and retains
nothing of it.
+ _density_solver->observe(std::string_view(v->data, v->size));
Review Comment:
Fixed in 2cc0776c891: a CHAR value is observed and retained up to its first
NUL, exactly as `_add_value_tokens` tokenizes it, and the solver folds ASCII
case before hashing when the scheme has `lower_case`, as `GramExtractor` folds
the row. Tests: `GramDensityTest.FoldsCaseWhenTheSchemeDoes`,
`GramDensityCalibrationTest.CharPaddingIsNeitherEvidenceNorRetained`,
`GramDensityCalibrationTest.FoldingSchemesFoldTheEvidence`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
