github-actions[bot] commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4078677851
##########
be/src/storage/index/inverted/tokenizer/ngram/ngram_tokenizer.cpp:
##########
@@ -91,6 +96,9 @@ void NGramTokenizer::reset() {
_char_buffer = nullptr;
_char_offset = 0;
_char_length = _in->read((const void**)&_char_buffer, 0,
static_cast<int32_t>(_in->size()));
+ if (_char_length > 0 && !validate_utf8(_char_buffer, _char_length)) {
Review Comment:
[P1] Preserve existing malformed-byte indexing when provenance is disabled.
This validation now runs for every NGram use, although Doris can store
malformed bytes in VARCHAR (the existing UTF-8 regression inserts
`UNHEX('C0AF')` and `UNHEX('FF')`). Before this patch NGram skipped failed code
points and still indexed valid neighbors; now `reset()` throws and
`add_document()` turns the row into a failed index write. The earlier
provenance thread allowed rejection as one offset repair, but applying it
unconditionally changes ordinary, non-provenance behavior; ICUTokenizer has the
same new check. Please keep the prior behavior when source tracking is off and
use a conservative span or a scoped failure only for the offset-aware path,
with an end-to-end writer compatibility case.
##########
be/src/runtime/index_policy/index_policy_mgr.cpp:
##########
@@ -291,29 +377,33 @@ void IndexPolicyMgr::process_filter_configs(
continue;
}
- // Use normalized name for case-insensitive lookup
std::string normalized_filter_name = normalize_name(filter_name);
- if (_name_to_id.contains(normalized_filter_name)) {
+ if (const auto* filter_policy =
find_policy_by_name_locked(filter_name);
+ filter_policy != nullptr) {
// Nested filter policy
- const int64_t filter_policy_id =
_name_to_id.at(normalized_filter_name);
- const auto& filter_policy = _policys.at(filter_policy_id);
- auto type_it = filter_policy.properties.find(PROP_TYPE);
- if (type_it == filter_policy.properties.end()) {
+ if (filter_policy->type != expected_type) {
Review Comment:
[P1] Convert this new error on the direct SEARCH path too. The scoring
helper now catches this wrong-family policy exception, but
`FieldReaderResolver::resolve_with_analyzer_context()` has a separate
`build_analyzer_context()` in `variant_inverted_index_search.cpp` that calls
`create_analyzer_provider()` without a catch;
`FunctionSearch::build_leaf_query()` only `RETURN_IF_ERROR`s that `Status` API.
A replayed analyzer that binds (for example) a CHAR_FILTER policy as
`token_filter` therefore throws instead of returning
`INVERTED_INDEX_ANALYZER_ERROR`. This is a different parallel path from the
earlier scoring thread; please reuse the `Result` helper or wrap the
FieldReaderResolver builder and add a direct SEARCH regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
