airborne12 opened a new pull request, #70352: URL: https://github.com/apache/doris/pull/70352
### What problem does this PR solve? Issue Number: CIR-22263 is background for #69199; no issue closure is requested. The source explicitly states that the tokenized T2 merge hotspot is outside that fix. Related PRs: [#62181](https://github.com/apache/doris/pull/62181), [#66880](https://github.com/apache/doris/pull/66880), [#68607](https://github.com/apache/doris/pull/68607), [#68661](https://github.com/apache/doris/pull/68661), [#68674](https://github.com/apache/doris/pull/68674), [#68701](https://github.com/apache/doris/pull/68701), [#69199](https://github.com/apache/doris/pull/69199), [#69441](https://github.com/apache/doris/pull/69441), [#69442](https://github.com/apache/doris/pull/69442), [#69443](https://github.com/apache/doris/pull/69443), [#69444](https://github.com/apache/doris/pull/69444), [#70007](https://github.com/apache/doris/pull/70007), [#70151](https://github.com/apache/doris/pull/70151) Backport the missing 4.2.x index fixes and their required partition-drop feature. This enables physical inverted-index deletion on selected partitions while retaining the table index definition. It rejects mixed partition ALTER clauses and unfinished BUILD/DROP jobs of the same index before metadata changes. It also preserves array MATCH tokens and positions, enables phrase/scoring behavior for single-term normalizers, fixes NULL-vector memory ownership, validates bundled SNII compatibility data, and reduces shared memory-accounting contention. ### Release note Support `DROP INDEX ... PARTITION/PARTITIONS (...)` for inverted indexes, with mixed-clause and job-conflict guards. Fix array MATCH fallback, normalizer phrase defaults/scoring, and SNII NULL memory accounting. Process memory metrics are coalesced per reporter and can lag by less than 1 MiB while that reporter has live bytes; writer-local accounting remains exact, and totals flush at zero live bytes or destruction. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test - Behavior changed: - [x] Yes. Partition-scoped physical deletion and the release-note behavior above. - Does this need documentation? - [x] Yes. The new partition DROP INDEX syntax needs release documentation; the source #62181 does not link a documentation PR. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label ### Validation - `./build.sh --fe -j8` and `./build.sh --be -j16`: passed with Clang 20.1.8, CMake 3.31.8, JDK 17, ASAN/UBSAN, and target-default PCH. - `./run-fe-ut.sh --run org.apache.doris.alter.SchemaChangeHandlerTest,org.apache.doris.alter.IndexChangeJobTest,org.apache.doris.analysis.InvertedIndexPropertiesTest,org.apache.doris.catalog.IndexTest,org.apache.doris.nereids.parser.NereidsParserTest,org.apache.doris.nereids.trees.expressions.functions.scalar.TokenizeTest`: 169 passed, no failures/errors/skips; every source-added FE case executed. - BE discovery listed 362 selected tests, including all 32 source-added cases. `./run-be-ut.sh --run --filter="SniiStreamedWriterSessionTest.*:SniiWriterNullDocids.*:SniiWriterFailureLatch.*:SniiWriterNorms.*:SniiWriterTest.*:SniiCoreMetadata.*:SniiGoldenCorpus*.*:FunctionMatchTest.*:BlockDocSetTest.*:CollectionStatistics*.*:IndexReaderHelperTest.*:InvertedIndexParserTest.*:MultiSegmentCollectorTest.*:SniiBuildMemoryTracker.*:SniiSpimiGlobalSpill.*:SniiGlobalMemoryLimiter.*:SniiSpimiStreamingWriter.*" -j16`: 361 passed, no failures. The existing `SniiGlobalMemoryLimiter.ProductionReaderReDerivesTheShareAndZeroDisablesIt` self-skipped because the host was under its memory-pressure threshold; none of the 32 new cases skipped. - `SniiGoldenCorpus.WriteOrVerify` ran in bundled verify mode with both golden override variables unset: 513 observations, zero mismatches. No ASAN/UBSAN/leak diagnostic was observed in the executed BE tests. - `./run-regression-test.sh --conf <isolated-conf> --run -d index_p0,inverted_index_p0 -s index_meta,test_tokenize,test_match_keyword_score,test_drop_index_on_partition,test_storage_format_snii,test_custom_normalizer,test_bm25_score,test_match_phrase_prefix_score,test_array_index1,test_array_index2,test_index_match_phrase_edge_arr,test_index_match_regexp_arr -parallel 1 -suiteParallel 1`: 13 suite executions passed, zero failed/fatal/skipped. Two existing files declare `test_array_index2`, so both executed. - The initial private 8G ASAN process limit caused allocator rejections at 8.1G RSS. After increasing only the ignored private runtime configuration to 16G and restarting the private backend, all SQL suites passed. No production memory limit or sanitizer suppression changed. - Full current-target merge-tree clang-format 16 and complete FE reactor Checkstyle passed. The complete PR English check passed with zero data exceptions. Checked head `aeb5e5c4996f04147a5efe91fe8f34334ac8a0a4`, target base `66122209cac78805a96a42f44d2a4a20b428f038`. - Final post-build changes contain shortened comments and explicit SQL fixture BUILD steps only. Compiled executable code and unit-test assertions are unchanged. SQL tests ran on the final fixture. Local-mode integration was exercised; a cloud integration cluster was not run. <details> <summary>Source range-diff review and per-hunk backport audit</summary> ### Range-diff review Each excerpt comes from `git range-diff --creation-factor=10000 <source>^! <pick>^!`. The hunk audit below covers every original source hunk and the final compatibility changes. #62181: ```diff 1: 791c56d85db ! 1: f0aaaa72789 [feature](inverted index) support DROP INDEX ON PARTITION for inverted index (#62181) - | RENAME COLUMN name=identifier newName=identifier #renameColumnClause + | RENAME COLUMN name=qualifiedName TO? newName=identifier #renameColumnClause - } else if (alterOp instanceof DropIndexOp) { ``` Keep the target parser context and legacy AlterClause dispatch. Pass PartitionNamesInfo through DropIndexClause, retain full-table behavior, and run every original feature case. The final regression compatibility commit explicitly builds physical indexes before the original waits and assertions. #66880: ```diff 1: 655b46eb3eb ! 1: 7e9a66f9ae6 [fix](be) Preserve SNII NULL-docid memory accounting (#66880) - // Register with the process-wide memory limiter so writers can honor advisory spill requests. + // Register this writer so the process limiter can request spills from its posting arena. -- // Limiter cannot see. ``` Keep target writer signatures and newer surrounding comments. NULL vectors and reservations move together; the final compatibility commit adapts the array-offset pointer in the source test. #68607: ```diff 1: dbdd07dc7ec ! 1: f746b1163c3 [fix](index) Enforce SNII compatibility corpus and simplify comments (#68607) - // Bigger blocks -> better per-block zstd on the dict region, larger cold - // fetch+decompress unit per dict-block miss. Write side only. --// PROCESS-WIDE share for SNII index-build RAM, as a PERCENT of the process ``` Retain target BKD bulk I/O, cache alignment, PRX policy, and test linkage. Add only the corpus macro to the target CMake block. The obsolete raw accounting-field comment is N-A because #66880 replaces that field. Later #70007 updates accounting code and comments. #68661: ```diff 1: 75d1f03a7cc ! 1: 887af5f734a [fix](inverted index) Preserve tokens from all array elements in MATCH fallback (#68661) - #include <hs/hs.h> -+#include <algorithm> -+#include <cstdint> ``` Keep the target includes, writer APIs, and trailing-slop behavior. Port all array-token/position/NULL handling and the eleven new assertions. The source PR and squash differ in surrounding include context only. #68674: ```diff 1: 2fb774e8cd1 ! 1: 0d1f4bfb3f3 [fix](fe) Check conflicting index change jobs before dropping index metadata (#68674) - } - } -+ // A light DROP INDEX publishes the index metadata before it submits its delete jobs. Run the checks ``` Use AlterClause/CreateIndexClause/DropIndexClause, preserve the target cloud/local light-index type policy, and check conflicts before metadata changes. Partition operations added by #62181 bypass full-table light-drop admission. #68701: ```diff 1: e103c10b567 ! 1: 7474a1908ae [fix](fe) Reject trailing tokens in tokenize properties (#68701) - ## fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/functions/scalar/TokenizeTest.java ## - tokenize::checkLegalityBeforeTypeCoercion); - Assertions.assertEquals("tokenize second argument must be properties format", exception.getMessage()); ``` Place the EOF grammar rule in fe-core rather than fe-sql-parser. The target lacks TokenizeTest, so create it with the source-added case and required fixture imports. #69199: ```diff 1: 7a9d745388b ! 1: e5dcb052356 [fix](inverted index) Align phrase position defaults with analysis (#69199) - query_type == InvertedIndexQueryType::MATCH_PHRASE_PREFIX_QUERY) { - const auto& properties = index_meta->properties(); - if (get_parser_phrase_support_string_from_properties(properties) == ``` Keep target out-of-line scoring helpers, the generalized is_analyzed API, visible-row statistics, and existing keyword/regexp scoring. Represent a single-term normalizer as is_analyzed=false. Map moved FE constants/classes and Java 8 collection APIs; remove one merge-created duplicate assignment. #69441: ```diff 1: f1fd1b432fb ! 1: dad4d04bd9b [improvement](fe) Scan index change jobs once in the light DROP INDEX pre-check (#69441) ``` The implementation patch is unchanged; range-diff reports the added cherry-pick provenance in the commit message. Retained job history is still snapshotted once. #69442: ```diff 1: adecd627ab0 ! 1: 2c59294702b [fix](fe) Reject DROP INDEX ... PARTITION combined with other ALTER clauses (#69442) - // that can reject those jobs before any clause below touches table metadata (processDropColumn - // updates the bloom filter columns in place), so a rejected statement leaves the table unchanged. - checkLightDropIndexPreconditions(db, olapTable, alterOps, enableAddIndexForNewData); ``` Use the legacy DropIndexClause and reject mixed partition work before metadata mutation. Keep all mixed-statement and metadata/job-count assertions; shorten comments. #69443: ```diff 1: 8ebb90998a5 ! 1: f2d288dc143 [fix](fe) Treat an unfinished index change job of the same index as a conflict in either direction (#69443) -+ // An unfinished BUILD INDEX idx_v1 on p1. Build and drop jobs of one index are not ordered, so a -+ // build running after the drop would add the dropped index back to the rowsets. -+ // It is only registered in indexChangeJobs so that the job scheduler does not run or cancel it. ``` Keep the direction-independent predicate and both full-table and partition rejection assertions. The 4.1 target additionally needs an early full-table conflict check because its source pre-check helper is absent. Shorten the test comment. #69444: ```diff 1: 5964f9f2635 ! 1: a420d49fd4a [fix](fe) Run an ALTER that also needs heavy work as a heavy schema change instead of a light index change (#69444) - // TODO sequence mapping do not support light schema yet - if (olapTable.hasColumnSeqMapping() || !sequenceMapping.isEmpty()) { - lightSchemaChange = false; ``` Track heavy work in the target column-operation branches and preserve its dispatch. Source-only sequence-mapping/changelog branches are absent. The test uses target-supported NGRAM_BF with new-data-only indexing disabled instead of unavailable BLOOMFILTER syntax. #70007: ```diff 1: f7396d7229a ! 1: f93980b880b [improvement](be) Remove per-term contention from SNII index build memory accounting (#70007) - assert_ok(writer.add_values(reinterpret_cast<const uint8_t*>(values.data()), values.size())); + assert_ok(writer.add_values("c1", values.data(), values.size())); ``` Adapt only the old writer.add_values test signature. The accounting/publication algorithm is unchanged. A later comment-only commit shortens its comments. #70151: ```diff 1: 380912cc734 ! 1: 4502c778b5e [fix](regression) Pin the VARIANT subcolumn limit in test_match_keyword_score (#70151) ``` The initial implementation patch is unchanged; range-diff reports provenance only. Retain the target JSON-string INSERT fixture and shorten the final explanatory comment. ### Original-source hunk audit Original source paths: F1 `be/src/storage/index/index_file_writer.cpp` F2 `be/src/storage/index/index_file_writer.h` F3 `be/src/storage/index/inverted/snii/snii_index_writer.cpp` F4 `be/src/storage/index/inverted/snii/snii_index_writer.h` F5 `be/src/storage/index/inverted/snii/writer/logical_index_writer.cpp` F6 `be/src/storage/index/inverted/snii/writer/logical_index_writer.h` F7 `be/src/storage/index/inverted/snii/writer/snii_compound_writer.cpp` F8 `be/src/storage/index/inverted/snii/writer/snii_compound_writer.h` F9 `be/test/storage/index/inverted/snii/compaction/snii_streamed_session_test.cpp` F10 `be/test/storage/index/inverted/snii/snii_writer_test.cpp` F11 `be/test/storage/index/inverted/snii/writer/spimi_streaming_writer_test.cpp` F12 `be/src/common/config.cpp` F13 `be/src/common/config.h` F14 `be/src/storage/index/inverted/snii/bkd/bkd_builder.cpp` F15 `be/src/storage/index/inverted/snii/bkd/bkd_builder.h` F16 `be/src/storage/index/inverted/snii/bkd/bkd_format.h` F17 `be/src/storage/index/inverted/snii/bkd/bkd_index_block.cpp` F18 `be/src/storage/index/inverted/snii/bkd/bkd_index_block.h` F19 `be/src/storage/index/inverted/snii/bkd/bkd_reader.cpp` F20 `be/src/storage/index/inverted/snii/bkd/bkd_reader.h` F21 `be/src/storage/index/inverted/snii/bkd/bkd_types.h` F22 `be/src/storage/index/inverted/snii/bkd/leaf_codec.cpp` F23 `be/src/storage/index/inverted/snii/bkd/leaf_codec.h` F24 `be/src/storage/index/inverted/snii/bkd/point_merger.h` F25 `be/src/storage/index/inverted/snii/bkd/point_run.h` F26 `be/src/storage/index/inverted/snii/bkd/point_sorter.h` F27 `be/src/storage/index/inverted/snii/bkd/point_source.h` F28 `be/src/storage/index/inverted/snii/compaction/region_reader.h` F29 `be/src/storage/index/inverted/snii/compaction/term_cursor.h` F30 `be/src/storage/index/inverted/snii/encoding/crc32c.cpp` F31 `be/src/storage/index/inverted/snii/encoding/crc32c.h` F32 `be/src/storage/index/inverted/snii/format/bsbf.cpp` F33 `be/src/storage/index/inverted/snii/format/core_metadata.cpp` F34 `be/src/storage/index/inverted/snii/format/dict_block.h` F35 `be/src/storage/index/inverted/snii/format/dict_entry.cpp` F36 `be/src/storage/index/inverted/snii/format/dict_entry.h` F37 `be/src/storage/index/inverted/snii/format/format_constants.h` F38 `be/src/storage/index/inverted/snii/format/frq_prelude.h` F39 `be/src/storage/index/inverted/snii/format/metadata_directory.cpp` F40 `be/src/storage/index/inverted/snii/format/prx_pod.cpp` F41 `be/src/storage/index/inverted/snii/format/sampled_term_index.h` F42 `be/src/storage/index/inverted/snii/reader/dict_block_cache.h` F43 `be/src/storage/index/inverted/snii/reader/logical_index_reader.cpp` F44 `be/src/storage/index/inverted/snii/snii_bkd_index_reader.h` F45 `be/src/storage/index/inverted/snii/snii_bkd_query.h` F46 `be/src/storage/index/inverted/snii/snii_blob_directory.h` F47 `be/src/storage/index/inverted/snii/snii_doris_adapter.cpp` F48 `be/src/storage/index/inverted/snii/writer/compact_posting_pool.h` F49 `be/src/storage/index/inverted/snii/writer/global_memory_limiter.h` F50 `be/src/storage/index/inverted/snii/writer/snii_build_memory_tracker.h` F51 `be/src/storage/index/inverted/snii/writer/spill_run_codec.h` F52 `be/src/storage/index/inverted/snii/writer/spimi_term_buffer.cpp` F53 `be/src/storage/index/inverted/snii/writer/spimi_term_buffer.h` F54 `be/src/storage/segment/count_on_index_fastpath.h` F55 `be/test/CMakeLists.txt` F56 `be/test/storage/index/inverted/snii/bench/bkd_native_vs_clucene_bench_test.cpp` F57 `be/test/storage/index/inverted/snii/bench/snii_vs_v3_benchmark_test.cpp` F58 `be/test/storage/index/inverted/snii/bkd/bkd_builder_test.cpp` F59 `be/test/storage/index/inverted/snii/bkd/bkd_container_roundtrip_test.cpp` F60 `be/test/storage/index/inverted/snii/bkd/bkd_corruption_test.cpp` F61 `be/test/storage/index/inverted/snii/bkd/bkd_differential_test.cpp` F62 `be/test/storage/index/inverted/snii/bkd/bkd_format_test.cpp` F63 `be/test/storage/index/inverted/snii/bkd/bkd_golden_format_test.cpp` F64 `be/test/storage/index/inverted/snii/bkd/bkd_index_block_test.cpp` F65 `be/test/storage/index/inverted/snii/bkd/bkd_property_test.cpp` F66 `be/test/storage/index/inverted/snii/bkd/bkd_reader_test.cpp` F67 `be/test/storage/index/inverted/snii/bkd/bkd_types_test.cpp` F68 `be/test/storage/index/inverted/snii/bkd/leaf_codec_test.cpp` F69 `be/test/storage/index/inverted/snii/bkd/point_sorter_test.cpp` F70 `be/test/storage/index/inverted/snii/bkd/staged_blob_file_test.cpp` F71 `be/test/storage/index/inverted/snii/compaction/snii_term_cursor_test.cpp` F72 `be/test/storage/index/inverted/snii/encoding/crc32c_test.cpp` F73 `be/test/storage/index/inverted/snii/encoding/pfor_test.cpp` F74 `be/test/storage/index/inverted/snii/format/core_metadata_test.cpp` F75 `be/test/storage/index/inverted/snii/format/dict_block_test.cpp` F76 `be/test/storage/index/inverted/snii/format/frq_pod_test.cpp` F77 `be/test/storage/index/inverted/snii/format/frq_prelude_test.cpp` F78 `be/test/storage/index/inverted/snii/format/prx_pod_test.cpp` F79 `be/test/storage/index/inverted/snii/io/local_file_test.cpp` F80 `be/test/storage/index/inverted/snii/query/byte_skip_test.cpp` F81 `be/test/storage/index/inverted/snii/query/count_query_test.cpp` F82 `be/test/storage/index/inverted/snii/query/phrase_skip_test.cpp` F83 `be/test/storage/index/inverted/snii/query/posting_grouping_test.cpp` F84 `be/test/storage/index/inverted/snii/snii_ann_container_test.cpp` F85 `be/test/storage/index/inverted/snii/snii_bkd_adapter_test.cpp` F86 `be/test/storage/index/inverted/snii/snii_bkd_adapter_types_test.cpp` F87 `be/test/storage/index/inverted/snii/snii_golden_corpus_test.cpp` F88 `be/test/storage/index/inverted/snii/snii_spill_io_test.cpp` F89 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/english_array_phrase.expect` F90 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/english_array_phrase.idx` F91 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/english_docs.expect` F92 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/english_docs.idx` F93 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/english_phrase.expect` F94 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/english_phrase.idx` F95 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/english_phrase_nolower.expect` F96 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/english_phrase_nolower.idx` F97 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/keyword_array_docs.expect` F98 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/keyword_array_docs.idx` F99 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/keyword_docs.expect` F100 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/keyword_docs.idx` F101 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/keyword_phrase.expect` F102 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/keyword_phrase.idx` F103 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/unicode_phrase.expect` F104 `be/test/storage/index/inverted/snii/testdata/compatibility_corpus/unicode_phrase.idx` F105 `be/test/storage/index/inverted/snii/writer/compact_posting_pool_test.cpp` F106 `be/test/storage/index/inverted/snii/writer/logical_index_writer_freq_stats_test.cpp` F107 `be/test/storage/index/inverted/snii/writer/posting_interleave_test.cpp` F108 `be/test/storage/index/inverted/snii/writer/snii_build_memory_tracker_test.cpp` F109 `be/test/storage/index/inverted/snii/writer/snii_compound_writer_blob_test.cpp` F110 `be/test/storage/index/inverted/snii/writer/snii_prx_tier_writer_test.cpp` F111 `be/test/storage/index/inverted/snii/writer/snii_writer_golden_bytes_test.cpp` F112 `be/test/storage/index/inverted/snii/writer/spill_run_codec_test.cpp` F113 `be/test/storage/index/inverted/snii/writer/spimi_global_memory_limiter_test.cpp` F114 `be/test/storage/index/inverted/snii/writer/spimi_run_cap_test.cpp` F115 `be/test/storage/index/inverted/snii/writer/spimi_term_buffer_test.cpp` F116 `be/test/storage/segment/segment_iterator_count_emit_shortcut_test.cpp` F117 `gensrc/proto/snii.proto` F118 `be/src/exprs/function/match.cpp` F119 `be/src/exprs/function/match.h` F120 `be/test/exprs/function/function_match_test.cpp` F121 `fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java` F122 `fe/fe-core/src/test/java/org/apache/doris/alter/SchemaChangeHandlerTest.java` F123 `fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java` F124 `fe/fe-core/src/test/java/org/apache/doris/nereids/parser/NereidsParserTest.java` F125 `fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/functions/scalar/TokenizeTest.java` F126 `fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4` F127 `regression-test/suites/inverted_index_p0/test_tokenize.groovy` F128 `be/src/storage/index/index_reader_helper.h` F129 `be/src/storage/index/inverted/clucene/spi/clucene_index_source.cpp` F130 `be/src/storage/index/inverted/clucene/spi/clucene_postings_cursor.h` F131 `be/src/storage/index/inverted/reader/collection_statistics.cpp` F132 `be/src/storage/index/inverted/reader/collection_statistics.h` F133 `be/test/storage/index/inverted/analysis/inverted_index_parser_test.cpp` F134 `be/test/storage/index/inverted/query/engine/boolean_postings_bench_test.cpp` F135 `be/test/storage/index/inverted/query/engine/multi_segment_collector_test.cpp` F136 `be/test/storage/index/inverted/query/exec/block_doc_set_test.cpp` F137 `be/test/storage/index/inverted/reader/collection_statistics_test.cpp` F138 `be/test/storage/segment/index_reader_helper_test.cpp` F139 `fe/fe-catalog/src/main/java/org/apache/doris/catalog/Index.java` F140 `fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java` F141 `fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java` F142 `fe/fe-core/src/test/java/org/apache/doris/catalog/IndexTest.java` F143 `regression-test/suites/index_p0/test_index_meta.groovy` F144 `fe/fe-core/src/main/java/org/apache/doris/alter/IndexChangeJob.java` F145 `be/src/storage/index/inverted/snii/writer/snii_build_memory_tracker.cpp` F146 `regression-test/suites/inverted_index_p0/test_match_keyword_score.groovy` F147 `fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4` F148 `fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java` F149 `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/DropIndexOp.java` F150 `fe/fe-core/src/test/java/org/apache/doris/alter/IndexChangeJobTest.java` F151 `regression-test/suites/inverted_index_p0/index_change/test_drop_index_on_partition.groovy` P = Ported: the source changed statements are present, including changes already on the target. A = Adapted; numbered explanations identify the target API or retained behavior. V = Ported(verbatim): the complete file equals the original merged source at this pick. Hunk headers below are copied verbatim from gh pr diff. Each source is compared with its own pick commit; later requested fixes can update shared files. Source #66880: `655b46eb3eb3675c4481d5505998505ec836bcdb` → `7e9a66f9ae68801812b997032d63d59dbd3c49b5` |File|Original source hunk header|Result| |---|---|---| |F1|`@@ -325,7 +325,7 @@ void IndexFileWriter::_release_snii_blob_directories() {`|P| |F1|`@@ -348,13 +348,12 @@ Status IndexFileWriter::add_snii_index(const TabletIndex* index_meta, uint32_t d`|P| |F2|`@@ -118,7 +118,7 @@ class IndexFileWriter {`|P| |F3|`@@ -161,6 +161,7 @@ Status SniiIndexColumnWriter::init() {`|P| |F3|`@@ -352,16 +353,24 @@ Status SniiIndexColumnWriter::add_array_values(size_t field_size, const uint8_t*`|P| |F3|`@@ -540,10 +549,7 @@ Status SniiIndexColumnWriter::add_nulls(uint32_t count) {`|P| |F3|`@@ -552,7 +558,6 @@ Status SniiIndexColumnWriter::add_nulls(uint32_t count) {`|P| |F3|`@@ -564,13 +569,14 @@ Status SniiIndexColumnWriter::add_array_nulls(const uint8_t* null_map, size_t nu`|P| |F3|`@@ -583,10 +589,6 @@ Status SniiIndexColumnWriter::finish() {`|P| |F3|`@@ -599,8 +601,10 @@ Status SniiIndexColumnWriter::finish() {`|P| |F3|`@@ -622,12 +626,11 @@ Status SniiIndexColumnWriter::_latch_analysis_failure(Status status) {`|P| |F4|`@@ -113,9 +113,7 @@ class SniiIndexColumnWriter final : public IndexColumnWriter {`|P| |F4|`@@ -187,15 +185,10 @@ class SniiIndexColumnWriter final : public IndexColumnWriter {`|P| |F5|`@@ -522,9 +522,6 @@ uint32_t high_df_floor_for(const SniiIndexInput& in) {`|P| |F6|`@@ -194,7 +194,7 @@ struct FreqStats {`|P| |F6|`@@ -281,7 +281,6 @@ class LogicalIndexWriter {`|P| |F7|`@@ -150,6 +150,31 @@ Status SniiCompoundWriter::inherit(const reader::SniiRewriteSnapshot& snapshot,`|P| |F7|`@@ -174,7 +199,7 @@ Status SniiCompoundWriter::add_logical_index(const SniiIndexInput& in) {`|P| |F8|`@@ -160,10 +160,11 @@ class SniiCompoundWriter {`|P| |F8|`@@ -245,6 +246,7 @@ class SniiCompoundWriter {`|P| |F9|`@@ -500,6 +500,58 @@ TEST(SniiStreamedWriterSessionTest, NullDocidsStorageIsTransferredIntoStreamedWr`|P| |F10|`@@ -31,6 +31,7 @@`|A1| |F10|`@@ -38,8 +39,10 @@`|A1| |F10|`@@ -52,6 +55,7 @@`|A1| |F10|`@@ -505,6 +509,120 @@ TEST(SniiWriterNullDocids, AddNullsGrowsGeometricallyNotQuadratically) {`|A1| |F11|whole file|V| Source #68607: `dbdd07dc7ece048f6320de3f98ae8a3111c414ee` → `f746b1163c35c7760121c5911838082139e06eff` |File|Original source hunk header|Result| |---|---|---| |F12|`@@ -1423,29 +1423,10 @@ DEFINE_mDouble(inverted_index_ram_buffer_size, "512");`|P| |F13|`@@ -1474,45 +1474,11 @@ DECLARE_mInt32(snii_prx_zstd_level_direct_load);`|P| |F14|whole file|V| |F15|whole file|V| |F16|`@@ -137,15 +137,7 @@ inline constexpr uint32_t kPointDocIdBytes = 4;`|P| |F17|`@@ -30,14 +30,7 @@ namespace doris::snii::bkd {`|P| |F17|`@@ -67,9 +60,7 @@ bool resolve_field_type(uint32_t raw, FieldType* type, uint32_t* bytes_per_dim)`|P| |F17|`@@ -80,10 +71,7 @@ constexpr size_t kMaxLeafDirectoryRowBytes = 10 + 5;`|P| |F17|`@@ -96,8 +84,7 @@ Status decode_header(ByteSource* src, BkdIndexHeader* header) {`|P| |F17|`@@ -157,9 +144,7 @@ bool value_order_is_valid(Slice min_value, Slice max_value, Slice splits, size_t`|P| |F17|`@@ -236,7 +221,7 @@ void encode_bkd_index_block(const BkdIndexHeader& header, Slice min_value, Slice`|P| |F17|`@@ -295,7 +280,7 @@ Status BkdIndexBlockReader::decode_payload(Slice payload, uint64_t data_length)`|P| |F18|whole file|V| |F19|`@@ -75,10 +75,7 @@ void radix_sort_u32(std::vector<uint32_t>* values, std::vector<uint32_t>* scratc`|P| |F19|`@@ -92,18 +89,7 @@ Status check_extent(uint64_t offset, uint64_t length, uint64_t file_size, std::s`|P| |F19|`@@ -136,12 +122,7 @@ uint32_t split_upper_bound(Slice splits, uint32_t width, const uint8_t* key) {`|P| |F19|`@@ -197,17 +178,7 @@ bool locate_leaf_window(const BkdIndexBlockReader& block, Slice lower, bool lowe`|P| |F19|`@@ -274,17 +245,13 @@ Status BkdReader::open(io::FileReader* file, const BkdSections& sections,`|P| |F19|`@@ -312,17 +279,14 @@ Status BkdReader::range(Slice lower, bool lower_inclusive, Slice upper, bool upp`|P| |F19|`@@ -340,29 +304,7 @@ Status BkdReader::range(Slice lower, bool lower_inclusive, Slice upper, bool upp`|A2| |F19|`@@ -558,16 +500,12 @@ Status BkdReader::estimate_cardinality(Slice lower, bool lower_inclusive, Slice`|P| |F20|`@@ -35,23 +35,10 @@ namespace roaring {`|P| |F20|`@@ -70,36 +57,10 @@ struct BkdQueryScratch {`|P| |F20|`@@ -108,58 +69,18 @@ class BkdReader {`|P| |F20|`@@ -177,42 +98,27 @@ class BkdReader {`|A3| |F21|whole file|V| |F22|`@@ -29,11 +29,7 @@`|P| |F22|`@@ -43,9 +39,7 @@ namespace {`|P| |F22|`@@ -220,11 +214,7 @@ namespace {`|P| |F22|`@@ -241,11 +231,7 @@ Status decode_leaf_head(ByteSource* src, uint32_t bytes_per_dim, uint32_t expect`|P| |F23|`@@ -25,76 +25,25 @@`|P| |F23|`@@ -145,34 +94,11 @@ struct DecodedLeafBlock {`|P| |F24|whole file|V| |F25|whole file|V| |F26|whole file|V| |F27|whole file|V| |F28|whole file|V| |F29|whole file|V| |F30|whole file|V| |F31|whole file|V| |F32|whole file|V| |F33|`@@ -127,9 +127,7 @@ Status decode_core_pb(const doris::snii::SniiCoreMetadataPB& input, CoreMetadata`|P| |F34|whole file|V| |F35|whole file|V| |F36|whole file|V| |F37|`@@ -50,16 +50,9 @@ enum class SectionType : uint8_t {`|P| |F37|`@@ -165,7 +158,7 @@ inline constexpr uint32_t kDefaultInlineThreshold = 256; // slim encoded bytes `|P| |F38|whole file|V| |F39|whole file|V| |F40|`@@ -1207,7 +1207,7 @@ Status build_prx_window(std::span<const std::vector<uint32_t>> per_doc_positions`|P| |F40|`@@ -1292,7 +1292,7 @@ Status try_build_prx_window_flat(std::span<const uint32_t> positions_flat,`|P| |F41|whole file|V| |F42|whole file|V| |F43|`@@ -382,19 +382,7 @@ Status LogicalIndexReader::dict_block_reader_for_ordinal(`|P| |F44|whole file|V| |F45|whole file|V| |F46|whole file|V| |F47|`@@ -168,10 +168,7 @@ Status DorisSniiFileReader::read_batch(const std::vector<::doris::snii::io::Rang`|P| |F47|`@@ -250,7 +247,7 @@ Status DorisSniiFileReader::read_batch(const std::vector<::doris::snii::io::Rang`|P| |F47|`@@ -295,10 +292,7 @@ Status DorisSniiFileReader::read_batch(const std::vector<::doris::snii::io::Rang`|P| |F3|`@@ -163,21 +163,8 @@ Status SniiIndexColumnWriter::init() {`|P| |F4|`@@ -194,7 +194,7 @@ class SniiIndexColumnWriter final : public IndexColumnWriter {`|N-A4| |F48|whole file|V| |F49|whole file|V| |F5|`@@ -68,8 +68,8 @@ constexpr int kRawFrqRegion = 0;`|P| |F6|`@@ -42,49 +42,7 @@`|P| |F6|`@@ -109,7 +67,7 @@ struct SniiIndexInput {`|P| |F6|`@@ -399,7 +357,7 @@ class LogicalIndexWriter {`|P| |F50|`@@ -25,74 +25,23 @@ class MemTracker;`|A32| |F7|`@@ -623,66 +623,13 @@ Status SniiCompoundWriter::write_tail() {`|A5| |F8|`@@ -37,51 +37,14 @@ namespace doris::snii::reader {`|P| |F51|whole file|V| |F52|`@@ -101,7 +101,7 @@ std::string make_run_path(const std::string& dir) {`|A32| |F52|`@@ -115,14 +115,14 @@ std::atomic<uint64_t> g_dense_rank_inversions {0};`|A32| |F52|`@@ -131,7 +131,7 @@ inline bool g11_prefetch_enabled() {`|A32| |F52|`@@ -314,7 +314,7 @@ SpimiTermBuffer::SpimiTermBuffer(bool has_positions, size_t spill_threshold_byte`|A32| |F52|`@@ -373,17 +373,8 @@ void SpimiTermBuffer::report_arena_delta() {`|A32| |F52|`@@ -405,9 +396,8 @@ uint64_t SpimiTermBuffer::resident_bytes() const {`|A32| |F52|`@@ -504,31 +494,7 @@ void SpimiTermBuffer::accumulate(uint32_t term_id, uint32_t docid, uint32_t pos,`|A32| |F52|`@@ -541,24 +507,7 @@ void SpimiTermBuffer::maybe_spill_after_token() {`|A32| |F52|`@@ -717,7 +666,7 @@ uint32_t SpimiTermBuffer::append_owned_vocab_term(std::string&& term_str) {`|A32| |F53|`@@ -37,11 +37,11 @@ namespace doris::snii::writer {`|A32| |F53|`@@ -102,81 +102,12 @@ struct TermPostings {`|A32| |F53|`@@ -216,23 +147,10 @@ class SpimiTermBuffer {`|A32| |F53|`@@ -242,26 +160,13 @@ class SpimiTermBuffer {`|A32| |F53|`@@ -282,12 +187,12 @@ class SpimiTermBuffer {`|A32| |F53|`@@ -357,13 +262,8 @@ class SpimiTermBuffer {`|A32| |F53|`@@ -378,18 +278,11 @@ class SpimiTermBuffer {`|A32| |F53|`@@ -400,7 +293,7 @@ class SpimiTermBuffer {`|A32| |F53|`@@ -414,23 +307,14 @@ class SpimiTermBuffer {`|A32| |F53|`@@ -497,7 +381,7 @@ class SpimiTermBuffer {`|A32| |F53|`@@ -505,7 +389,7 @@ class SpimiTermBuffer {`|A32| |F53|`@@ -543,7 +427,7 @@ class SpimiTermBuffer {`|A32| |F53|`@@ -551,13 +435,7 @@ void set_bench_disable_g11_prefetch(bool disabled);`|A32| |F54|`@@ -20,33 +20,7 @@`|P| |F54|`@@ -81,35 +55,7 @@ inline bool count_on_index_fastpath_safe(const CountOnIndexFastpathFacts& f) {`|P| |F55|`@@ -145,6 +145,11 @@ endif()`|A6| |F56|`@@ -15,40 +15,7 @@`|P| |F57|`@@ -1178,7 +1178,7 @@ class DISABLED_SniiVsV3BenchmarkTest : public ::testing::Test {`|P| |F57|`@@ -1204,15 +1204,7 @@ class DISABLED_SniiVsV3BenchmarkTest : public ::testing::Test {`|P| |F57|`@@ -1271,7 +1263,7 @@ class DISABLED_SniiVsV3BenchmarkTest : public ::testing::Test {`|P| |F57|`@@ -1358,10 +1350,7 @@ class DISABLED_SniiVsV3BenchmarkTest : public ::testing::Test {`|P| |F58|whole file|V| |F59|whole file|V| |F60|`@@ -15,33 +15,7 @@`|P| |F60|`@@ -445,12 +419,7 @@ TEST(BkdCorruptionTest, EveryByteFlipInBkdDataIsAStatusNeverACrash) {`|P| |F60|`@@ -493,9 +462,7 @@ TEST(BkdCorruptionTest, EveryTruncationOfBkdIndexIsRejected) {`|P| |F60|`@@ -618,8 +585,7 @@ TEST(BkdCorruptionTest, HeaderCountsThatContradictTheDirectoryAreRejected) {`|P| |F60|`@@ -662,9 +628,7 @@ TEST(BkdCorruptionTest, InflatedLeafDirectoryCountIsRejected) {`|P| |F60|`@@ -672,16 +636,7 @@ TEST(BkdCorruptionTest, InflatedLeafDirectoryCountIsRejected) {`|P| |F60|`@@ -794,11 +749,7 @@ TEST(BkdCorruptionTest, SelfConsistentSplitAboveGlobalMaximumIsRejectedBeforeQue`|P| |F60|`@@ -882,8 +833,7 @@ TEST(BkdCorruptionTest, SectionExtentsAreValidatedNotTrusted) {`|P| |F61|whole file|V| |F62|whole file|V| |F63|whole file|V| |F64|whole file|V| |F65|whole file|V| |F66|`@@ -82,10 +82,7 @@ class MemoryFileWriter final : public io::FileWriter {`|P| |F66|`@@ -168,9 +165,7 @@ Status build_container(const std::vector<Point>& points, uint32_t points_per_lea`|P| |F66|`@@ -310,7 +305,7 @@ TEST(BkdReaderTest, OpenReportsTheHeaderAndTheGlobalBounds) {`|P| |F66|`@@ -319,9 +314,7 @@ TEST(BkdReaderTest, OpenRejectsSectionsThatRunPastTheEndOfTheFile) {`|P| |F66|`@@ -496,9 +489,7 @@ TEST(BkdReaderTest, LowerAboveUpperIsAnEmptyIntervalNotAnError) {`|P| |F66|`@@ -517,8 +508,7 @@ TEST(BkdReaderTest, IntervalsEntirelyOutsideTheGlobalBoundsCostNoIo) {`|P| |F66|`@@ -601,9 +591,7 @@ TEST(BkdReaderTest, DamagedLeafBytesDowngradeToAStatusInsteadOfCrashing) {`|P| |F66|`@@ -644,9 +632,7 @@ TEST(BkdReaderTest, ScratchIsReusedAcrossQueriesWithoutChangingResults) {`|P| |F66|`@@ -667,9 +653,7 @@ TEST(BkdReaderTest, ConcurrentQueriesShareOneImmutableReader) {`|P| |F66|`@@ -694,9 +678,7 @@ TEST(BkdReaderTest, ConcurrentQueriesShareOneImmutableReader) {`|P| |F66|`@@ -733,10 +715,7 @@ std::vector<Slice> slices_of(const std::vector<std::vector<uint8_t>>& encoded) {`|P| |F66|`@@ -831,9 +810,7 @@ TEST(BkdReaderTest, LookupManyOnAnEmptyIndexIsEmpty) {`|P| |F66|`@@ -852,15 +829,7 @@ TEST(BkdReaderTest, EstimateReadsNoData) {`|P| |F67|whole file|V| |F68|whole file|V| |F69|whole file|V| |F70|whole file|V| |F71|whole file|V| |F72|whole file|V| |F73|`@@ -102,16 +102,8 @@ TEST(SniiPfor, ExceptionsAtBoundaries) {`|P| |F73|`@@ -156,7 +148,7 @@ uint32_t ref_low_mask(uint8_t w) {`|P| |F73|`@@ -252,7 +244,7 @@ std::vector<std::vector<uint32_t>> representative_inputs() {`|P| |F73|`@@ -272,7 +264,7 @@ TEST(SniiPforTest, RoundTripRandom) {`|P| |F73|`@@ -294,7 +286,7 @@ TEST(SniiPforTest, WidthMatchesLinearScan) {`|P| |F73|`@@ -306,9 +298,8 @@ TEST(SniiPforTest, GoldenByteIdentical) {`|P| |F73|`@@ -334,7 +325,7 @@ TEST(SniiPforTest, EncodeMatchesLegacyReference) {`|P| |F73|`@@ -350,14 +341,14 @@ TEST(SniiPforTest, AllZeros) {`|P| |F73|`@@ -372,7 +363,7 @@ TEST(SniiPforTest, EmptyRun) {`|P| |F73|`@@ -383,7 +374,7 @@ TEST(SniiPforTest, TopBitSet) {`|P| |F73|`@@ -406,7 +397,7 @@ TEST(SniiPforTest, ManyExceptions) {`|P| |F73|`@@ -421,8 +412,7 @@ TEST(SniiPforTest, LargeRunHeapFallback) {`|P| |F74|whole file|V| |F75|`@@ -388,13 +388,7 @@ TEST(SniiDictBlock, RejectsNonMonotonicAnchorOffsets) {`|P| |F76|`@@ -354,14 +354,8 @@ TEST(SniiFrqPod, UncompLenCapRejected) {`|P| |F77|whole file|V| |F78|`@@ -1184,13 +1184,7 @@ TEST(SniiPrxPod, FlatBuilderExtraPositionsRejected) {`|P| |F78|`@@ -1337,9 +1331,6 @@ TEST(SniiPrxPodCounterTest, LargeWindowStillBuildsRawPlaintextOnce) {`|P| |F78|`@@ -1564,15 +1555,7 @@ TEST(SniiPrxPodTest, NullSinkRejected) {`|P| |F79|whole file|V| |F80|whole file|V| |F81|whole file|V| |F82|whole file|V| |F83|whole file|V| |F84|`@@ -15,30 +15,7 @@`|P| |F85|`@@ -15,21 +15,7 @@`|P| |F85|`@@ -357,9 +343,7 @@ TEST_F(SniiBkdAdapterTest, NullRowsBecomeANullBitmapNotPoints) {`|P| |F85|`@@ -441,10 +425,7 @@ TEST_F(SniiBkdAdapterTest, EncodesThroughTheIndexFieldTypeNotTheWidestOne) {`|P| |F85|`@@ -604,12 +585,7 @@ TEST_F(SniiBkdAdapterTest, ReaderIdentifiesAsBkd) {`|P| |F85|`@@ -660,16 +636,7 @@ TEST_F(SniiBkdAdapterTest, WriterFactoryStillRefusesNonIndexableTypes) {`|P| |F86|`@@ -15,32 +15,7 @@`|P| |F87|`@@ -15,26 +15,15 @@`|P| |F87|`@@ -68,6 +57,24 @@ namespace {`|P| |F87|`@@ -162,7 +169,7 @@ struct Sample {`|P| |F87|`@@ -455,16 +462,40 @@ Observation observe(const Sample& sample, const OpenedSample& opened, const Quer`|P| |F87|`@@ -498,6 +529,7 @@ TEST_F(SniiGoldenCorpus, WriteOrVerify) {`|P| |F88|whole file|V| |F89|whole file|V| |F90|whole file|V| |F91|whole file|V| |F92|whole file|V| |F93|whole file|V| |F94|whole file|V| |F95|whole file|V| |F96|whole file|V| |F97|whole file|V| |F98|whole file|V| |F99|whole file|V| |F100|whole file|V| |F101|whole file|V| |F102|whole file|V| |F103|whole file|V| |F104|whole file|V| |F105|whole file|V| |F106|whole file|V| |F107|whole file|V| |F108|`@@ -15,24 +15,7 @@`|A32| |F109|whole file|V| |F110|`@@ -15,19 +15,7 @@`|P| |F111|`@@ -15,33 +15,8 @@`|P| |F112|whole file|V| |F113|`@@ -37,36 +37,7 @@`|A32| |F113|`@@ -599,7 +570,7 @@ TEST(SniiGlobalMemoryLimiter, ProcessPressureBeyondSniiStillReclaimsWhatSniiHold`|A32| |F113|`@@ -781,7 +752,7 @@ TEST(SniiSpimiGlobalSpill, AttachRegistersReportsTrackArenaAndDtorUnregisters) {`|A32| |F113|`@@ -864,7 +835,7 @@ TEST(SniiSpimiGlobalSpill, LimiterFlagsLargestOwnerSpillsSmallBufferSpared) {`|A32| |F114|whole file|V| |F115|whole file|V| |F116|`@@ -15,16 +15,7 @@`|P| |F116|`@@ -128,7 +119,7 @@ struct Fixture {`|P| |F116|`@@ -257,7 +248,7 @@ TEST(CountEmitShortcut, NullableColumnsEmitNotNullDefaults) {`|P| |F116|`@@ -290,7 +281,7 @@ TEST(CountEmitShortcut, MultipleIteratorsEmitIndependently) {`|P| |F117|whole file|V| Source #68661: `75d1f03a7cc0654cc8e48e5318ae2631e96cd980` → `887af5f734a05bbbe99899c3d281a7ecd874c864` |File|Original source hunk header|Result| |---|---|---| |F118|`@@ -19,6 +19,12 @@`|P| |F118|`@@ -44,6 +50,253 @@ const InvertedIndexAnalyzerCtx* get_match_analyzer_ctx(FunctionContext* context)`|P| |F118|`@@ -123,6 +376,7 @@ Status FunctionMatchBase::execute_impl(FunctionContext* context, Block& block,`|P| |F118|`@@ -136,6 +390,7 @@ Status FunctionMatchBase::execute_impl(FunctionContext* context, Block& block,`|P| |F118|`@@ -157,7 +412,7 @@ Status FunctionMatchBase::execute_impl(FunctionContext* context, Block& block,`|P| |F118|`@@ -220,8 +475,9 @@ std::vector<segment_v2::TermInfo> FunctionMatchBase::analyse_query_str_token(`|P| |F118|`@@ -233,6 +489,9 @@ inline std::vector<segment_v2::TermInfo> FunctionMatchBase::analyse_data_token(`|P| |F118|`@@ -241,9 +500,12 @@ inline std::vector<segment_v2::TermInfo> FunctionMatchBase::analyse_data_token(`|P| |F118|`@@ -280,7 +542,8 @@ Status FunctionMatchAny::execute_match(FunctionContext* context, const std::stri`|P| |F118|`@@ -294,21 +557,23 @@ Status FunctionMatchAny::execute_match(FunctionContext* context, const std::stri`|P| |F118|`@@ -320,7 +585,8 @@ Status FunctionMatchAll::execute_match(FunctionContext* context, const std::stri`|P| |F118|`@@ -334,26 +600,32 @@ Status FunctionMatchAll::execute_match(FunctionContext* context, const std::stri`|P| |F118|`@@ -361,12 +633,12 @@ Status FunctionMatchAll::execute_match(FunctionContext* context, const std::stri`|P| |F118|`@@ -380,43 +652,10 @@ Status FunctionMatchPhrase::execute_match(FunctionContext* context, const std::s`|P| |F118|`@@ -428,7 +667,8 @@ Status FunctionMatchPhrasePrefix::execute_match(`|P| |F118|`@@ -442,52 +682,23 @@ Status FunctionMatchPhrasePrefix::execute_match(`|P| |F118|`@@ -527,25 +738,25 @@ Status FunctionMatchRegexp::execute_match(FunctionContext* context, const std::s`|P| |F118|`@@ -560,7 +771,8 @@ Status FunctionMatchPhraseEdge::execute_match(`|P| |F118|`@@ -574,49 +786,11 @@ Status FunctionMatchPhraseEdge::execute_match(`|P| |F119|`@@ -72,12 +72,12 @@ class FunctionMatchBase : public IFunction {`|P| |F119|`@@ -87,8 +87,9 @@ class FunctionMatchBase : public IFunction {`|P| |F119|`@@ -107,12 +108,12 @@ class FunctionMatchAny : public FunctionMatchBase {`|P| |F119|`@@ -122,12 +123,12 @@ class FunctionMatchAll : public FunctionMatchBase {`|P| |F119|`@@ -137,12 +138,12 @@ class FunctionMatchPhrase : public FunctionMatchBase {`|P| |F119|`@@ -152,12 +153,12 @@ class FunctionMatchPhrasePrefix : public FunctionMatchBase {`|P| |F119|`@@ -167,12 +168,12 @@ class FunctionMatchRegexp : public FunctionMatchBase {`|P| |F119|`@@ -182,12 +183,12 @@ class FunctionMatchPhraseEdge : public FunctionMatchBase {`|P| |F120|`@@ -17,6 +17,7 @@`|P| |F120|`@@ -25,8 +26,11 @@`|P| |F120|`@@ -464,6 +468,403 @@ TEST(FunctionMatchTest, array_offset_handling) {`|P| Source #68674: `2fb774e8cd11e737a328cf8f3656453b0259dd82` → `0d1f4bfb3f3bfd2ef231d2ae079f3097c8f50a0c` |File|Original source hunk header|Result| |---|---|---| |F121|`@@ -2388,6 +2388,11 @@ public int getAsInt() {`|A7| |F121|`@@ -2641,21 +2646,7 @@ public int getAsInt() {`|A8| |F121|`@@ -3921,6 +3912,63 @@ public void buildOrDeleteTableInvertedIndices(Database db, OlapTable olapTable,`|A7| |F122|`@@ -1293,6 +1293,96 @@ public void testAddDuplicateInvertedIndexException() throws Exception {`|A9| Source #68701: `e103c10b567cce649b74e23bc7a2880285e6402c` → `7474a1908ae8e2b00e38d855a8675623eb7dad24` |File|Original source hunk header|Result| |---|---|---| |F123|`@@ -337,7 +337,7 @@ public DataType parseDataType(String dataType) {`|P| |F124|`@@ -99,6 +99,26 @@`|P| |F125|`@@ -82,4 +82,14 @@ public void testTokenizeRejectsMalformedProperties() {`|A10| |F126|`@@ -101,6 +101,10 @@ expressionWithEof`|A11| |F127|`@@ -18,6 +18,15 @@`|P| Source #69199: `7a9d745388bd8b71cedcbc91c95747cd4758ab27` → `e5dcb052356d148911494ec7b3ce57811e177bbd` |File|Original source hunk header|Result| |---|---|---| |F128|`@@ -17,6 +17,10 @@`|P| |F128|`@@ -24,6 +28,16 @@ namespace doris::segment_v2 {`|P| |F128|`@@ -59,7 +73,8 @@ class IndexReaderHelper {`|P| |F128|`@@ -96,7 +111,8 @@ class IndexReaderHelper {`|A12| |F128|`@@ -110,7 +126,8 @@ class IndexReaderHelper {`|A12| |F128|`@@ -118,4 +135,4 @@ class IndexReaderHelper {`|P| |F129|`@@ -68,6 +68,7 @@ ClucenePostingsCursor::Partition open_postings(lucene::index::IndexReader* reade`|P| |F130|`@@ -39,12 +39,14 @@ class ClucenePostingsCursor final : public index_query::PostingsCursor,`|A13| |F130|`@@ -87,12 +89,13 @@ class ClucenePostingsCursor final : public index_query::PostingsCursor,`|P| |F130|`@@ -312,6 +315,7 @@ class ClucenePostingsCursor final : public index_query::PostingsCursor,`|A13| |F130|`@@ -376,6 +380,7 @@ class ClucenePostingsCursor final : public index_query::PostingsCursor,`|A13| |F131|`@@ -43,8 +43,10 @@ namespace collection_statistics_detail {`|A14| |F131|`@@ -262,7 +264,10 @@ Status CollectionStatistics::process_segment(const RowsetSharedPtr& rowset,`|A15| |F131|`@@ -332,9 +337,12 @@ Status CollectionStatistics::process_segment(const RowsetSharedPtr& rowset,`|A15| |F131|`@@ -343,7 +351,10 @@ Status CollectionStatistics::process_segment(const RowsetSharedPtr& rowset,`|A16| |F131|`@@ -364,10 +375,11 @@ Status CollectionStatistics::process_segment(const RowsetSharedPtr& rowset,`|A14| |F132|`@@ -86,7 +86,7 @@ class CollectionStatistics {`|A17| |F132|`@@ -121,12 +121,12 @@ struct SniiScoringSegmentStats {`|A17| |F133|`@@ -130,6 +130,29 @@ TEST_F(InvertedIndexParserTest, TestGetParserPhraseSupportStringFromProperties)`|P| |F134|`@@ -202,7 +202,7 @@ uint64_t posting_row_checksum(uint32_t doc, float score, bool scoring) {`|P| |F134|`@@ -255,7 +255,7 @@ PhysicalTruthRows expected_physical_truth(const PostingSpec& spec, uint32_t prof`|P| |F135|`@@ -249,7 +249,8 @@ static void check_native_posting_block(const index_query::PostingsBlock& block,`|P| |F135|`@@ -407,7 +408,7 @@ TEST_F(MultiSegmentCollectorTest, PhraseCandidatesRemainValidAfterWeightDestruct`|P| |F136|whole file|V| |F137|`@@ -414,7 +414,7 @@ class CollectionStatisticsTest : public ::testing::Test {`|P| |F137|`@@ -431,8 +431,12 @@ class CollectionStatisticsTest : public ::testing::Test {`|P| |F137|`@@ -470,7 +474,7 @@ class CollectionStatisticsTest : public ::testing::Test {`|P| |F137|`@@ -486,8 +490,11 @@ class CollectionStatisticsTest : public ::testing::Test {`|P| |F137|`@@ -497,10 +504,15 @@ class CollectionStatisticsTest : public ::testing::Test {`|P| |F137|`@@ -655,6 +667,7 @@ class CollectionStatisticsTest : public ::testing::Test {`|A18| |F137|`@@ -663,7 +676,8 @@ class CollectionStatisticsTest : public ::testing::Test {`|A18| |F137|`@@ -1006,6 +1020,27 @@ TEST_F(CollectionStatisticsTest, SniiScoringUsesPhysicalStatistics) {`|P| |F137|`@@ -1194,6 +1229,16 @@ TEST(CollectionStatisticsSniiScoringTest, ResolveRejectsSegmentWithoutPositions)`|A18| |F10|`@@ -137,6 +137,28 @@ class ScopedGramPolicies {`|P| |F10|`@@ -551,6 +573,35 @@ TEST(SniiWriterNorms, WritesNormsFollowSharedNormsPolicy) {`|P| |F138|`@@ -396,6 +396,49 @@ TEST_F(IndexReaderHelperTest, IsNeedSimilarityScoreWithInvertedIndexQueryTypeTes`|P| |F139|`@@ -75,10 +75,13 @@ public Index(long indexId, String indexName, List<String> columns,`|A19| |F140|`@@ -354,11 +354,20 @@ private static void checkInvertedIndexProperties(Map<String, String> properties,`|P| |F141|`@@ -849,10 +849,11 @@ public void testExactLowercasePolicyKeepsMixedCaseBuiltinNormalizerBinding() {`|A20| |F141|`@@ -860,6 +861,8 @@ public void testExactLowercasePolicyKeepsMixedCaseBuiltinNormalizerBinding() {`|P| |F142|`@@ -24,9 +24,44 @@`|A21| |F143|whole file|V| Source #69441: `f1fd1b432fb4df3089c31e84cdfb6b4cccfe1999` → `dad4d04bd9bd66eb7d938fce0bb85ec23f13d3a6` |File|Original source hunk header|Result| |---|---|---| |F121|`@@ -3960,11 +3960,14 @@ private void checkLightDropIndexPreconditions(Database db, OlapTable olapTable,`|P| |F121|`@@ -3974,20 +3977,26 @@ public boolean hasIndexChangeJobOnPartition(`|P| |F122|`@@ -1328,14 +1328,14 @@ public void testDropIndexRejectedByIndexChangeJobKeepsMetadata() throws Exceptio`|P| Source #69443: `8ebb90998a5837a195b067077613b1ef8f62de90` → `f2d288dc14385ec41cc00724a9b2d82ee7e77100` |File|Original source hunk header|Result| |---|---|---| |F144|whole file|V| |F121|`@@ -3888,7 +3888,7 @@ public void buildOrDeleteTableInvertedIndices(Database db, OlapTable olapTable,`|P| |F121|`@@ -3965,7 +3965,7 @@ private void checkNoIndexChangeJobOnPartitions(Database db, OlapTable olapTable,`|P| |F121|`@@ -3973,13 +3973,11 @@ && isConflictingIndexChangeJob(indexChangeJob, olapTable.getBaseIndexId(), db.ge`|P| |F121|`@@ -3987,12 +3985,12 @@ && isConflictingIndexChangeJob(indexChangeJob, originIndexId, dbId, tableId, alt`|P| |F122|`@@ -1383,6 +1383,58 @@ public void testLightAddAndDropIndexInOneStatementRejected() throws Exception {`|A22| Source #69444: `5964f9f2635bf37337b1514367cd530962fde2b1` → `a420d49fd4a0476bb732c6291677dba7fd7f7a79` |File|Original source hunk header|Result| |---|---|---| |F121|`@@ -2325,6 +2325,9 @@ public void process(String rawSql, List<AlterOp> alterOps, Database db,`|P| |F121|`@@ -2508,10 +2511,12 @@ public int getAsInt() {`|A23| |F121|`@@ -2521,10 +2526,12 @@ public int getAsInt() {`|A23| |F121|`@@ -2533,10 +2540,12 @@ public int getAsInt() {`|A23| |F121|`@@ -2544,15 +2553,18 @@ public int getAsInt() {`|A23| |F121|`@@ -2567,6 +2579,9 @@ public int getAsInt() {`|P| |F121|`@@ -2672,7 +2687,12 @@ public int getAsInt() {`|A24| |F121|`@@ -2681,7 +2701,7 @@ public int getAsInt() {`|P| |F122|`@@ -1485,6 +1485,68 @@ public void testDropIndexOnPartitionsMustBeTheOnlyClause() throws Exception {`|A25| Source #70007: `f7396d7229a26b421b07dde19f3be1dede8a7256` → `f93980b880b5a8a3a25ec402091bff465488585a` |File|Original source hunk header|Result| |---|---|---| |F145|`@@ -18,6 +18,9 @@`|A33| |F145|`@@ -53,34 +56,76 @@ namespace {`|A33| |F50|`@@ -17,6 +17,8 @@`|A33| |F50|`@@ -37,7 +39,28 @@ enum class BuildMemoryPopulation {`|A33| |F52|`@@ -41,6 +41,7 @@`|A33| |F52|`@@ -346,8 +347,10 @@ void SpimiTermBuffer::attach_global_limiter(GlobalMemoryLimiter* limiter) {`|A33| |F52|`@@ -375,9 +378,22 @@ void SpimiTermBuffer::report_arena_delta() {`|A33| |F53|`@@ -287,8 +287,9 @@ class SpimiTermBuffer {`|A33| |F53|`@@ -389,6 +390,10 @@ class SpimiTermBuffer {`|A33| |F85|`@@ -340,12 +340,13 @@ TEST_F(SniiBkdAdapterTest, BuildMemoryIsChargedAndGivenBackAtFinish) {`|A26| |F108|`@@ -23,6 +23,8 @@`|A33| |F108|`@@ -31,6 +33,10 @@`|A33| |F108|`@@ -46,16 +52,16 @@ TEST(SniiBuildMemoryTracker, ReservationsMoveTheTrackerAndReturnToBaseline) {`|A33| |F108|`@@ -66,11 +72,11 @@ TEST(SniiBuildMemoryTracker, LegacyReportPathAlsoMovesTheTracker) {`|A33| |F108|`@@ -84,13 +90,13 @@ TEST(SniiBuildMemoryTracker, SeparateReportersAccumulateIntoOneLine) {`|A33| |F108|`@@ -105,15 +111,15 @@ TEST(SniiBuildMemoryTracker, RegisteredPopulationIsTrackedSeparatelyFromTheObser`|A33| |F108|`@@ -129,8 +135,8 @@ TEST(SniiBuildMemoryTracker, ReporterDestructorDrainsUnbalancedBytes) {`|A33| |F108|`@@ -142,8 +148,8 @@ TEST(SniiBuildMemoryTracker, ReporterDestructorDrainsTheRegisteredSubsetToo) {`|A33| |F108|`@@ -153,11 +159,138 @@ TEST(SniiBuildMemoryTracker, UnregisteredResidueNeverReachesTheDecisionInput) {`|A33| |F113|whole file|V| Source #70151: `380912cc7344e92375c3de58ec37c99602cad013` → `4502c778b5e27409312c57b5ac9a4d1ae96f82d4` |File|Original source hunk header|Result| |---|---|---| |F146|`@@ -99,6 +99,10 @@ suite("test_match_keyword_score", "p0") {`|P| Source #62181: `791c56d85db0648ae8911616a92d138f87ab7b11` → `f0aaaa7278934abf51a3daecee0d0e1b617404f4` |File|Original source hunk header|Result| |---|---|---| |F147|`@@ -352,7 +352,8 @@ supportedDropStatement`|P| |F147|`@@ -764,7 +765,7 @@ alterTableClause`|P| |F121|`@@ -2349,37 +2349,80 @@ public int getAsInt() {`|A27| |F121|`@@ -2409,7 +2452,7 @@ public int getAsInt() {`|P| |F148|`@@ -6287,14 +6287,26 @@ public Command visitBuildIndex(BuildIndexContext ctx) {`|P| |F149|`@@ -18,6 +18,7 @@`|A28| |F149|`@@ -25,6 +26,8 @@`|P| |F149|`@@ -36,12 +39,26 @@ public class DropIndexOp extends AlterTableOp {`|A28| |F149|`@@ -60,6 +77,16 @@ public boolean isAlter() {`|P| |F149|`@@ -73,6 +100,18 @@ public void validate(ConnectContext ctx) throws UserException {`|P| |F149|`@@ -92,6 +131,9 @@ public String toSql() {`|P| |F150|`@@ -30,6 +30,8 @@`|A29| |F150|`@@ -806,4 +808,127 @@ public void testCancelNgramBfBuildIndex() throws UserException {`|A29| |F151|`@@ -0,0 +1,273 @@`|A34| Source #69442: `adecd627ab031189821bc3a3cf2fd036aa2426e1` → `2c59294702be3697feb45773622606778cc99032` |File|Original source hunk header|Result| |---|---|---| |F121|`@@ -2392,6 +2392,14 @@ public int getAsInt() {`|A30| |F122|`@@ -1435,6 +1435,56 @@ public void testDropIndexRejectedByPendingBuildOfSameIndex() throws Exception {`|A31| Adaptation and applicability evidence: 1. Keep source NULL-handoff assertions. The final compatibility commit casts offsets.data() to const uint8_t* for the target add_array_values API. 2. Keep target add_whole_leaves/read_leaves bulk I/O and its comments; do not restore the source single-leaf loop. Production tokens are unchanged by this comment/corpus pick. 3. Keep the target read_leaves extent API and bulk-read explanation. The source single-leaf declaration is obsolete on this branch. 4. Target uses _null_docids_reservation from #66880 and has no _null_docids_charged_bytes field; this hunk only rewrites that obsolete field comment. 5. Keep target cache-alignment behavior and its shorter existing comment: Pad large containers when alignment saves more cache I/O than it writes. No padding code changes. 6. Add SNII_GOLDEN_CORPUS_DIR only. Keep the target test linking and platform guards; the source KUROMOJI/macOS context is absent here. 7. Use List<AlterClause>, DropIndexClause and CreateIndexClause; run the pre-check before metadata mutation. Keep target light-index type support. After #62181, partition drops are excluded with hasPartitionSpec(). 8. Reuse the helper while retaining target cloud support for INVERTED/NGRAM_BF and local support for INVERTED/ANN; do not add source cloud BLOOMFILTER behavior. 9. Keep both source rejection/metadata-preservation tests; shorten the unfinished-job comment. 10. The target lacks this test class. Create it with the source-added trailing-token test and its required imports; do not import unrelated source-baseline tests. 11. Place the identical grammar rule in target fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4. 12. Target scoring overloads are out-of-line. Extend can_score_match() in index_reader_helper.cpp with is_normalizer_only(), retaining newer keyword and regexp scoring. 13. Target already carries has_norms through cursor construction, blocks and partition switches. Keep that implementation and remove a duplicate assignment introduced by the mechanical pick in the final compatibility commit. 14. Keep target generalized is_analyzed argument and validation. Single-term normalizers pass is_analyzed=false; do not replace the newer keyword-scoring API. 15. Apply the normalizer exception at the actual SNII and CLucene process_segment() call sites: should_analyzer(properties) && !is_normalizer_only(properties). Preserve target visible-row/statistics logic. 16. Target already estimates all normless CLucene indexes with doc_count. Retain that generalized keyword/normalizer behavior instead of narrowing it to a single normalizer. 17. Keep target bool is_analyzed API/default and documentation; the source single_term_normalizer parameter is represented by its negation at the call sites. 18. Keep target generalized scoring fixture/API and pass is_analyzed=false in the new single-term-normalizer assertion. Preserve existing keyword and hidden-row tests. 19. Map to fe/fe-core/src/main/java/org/apache/doris/catalog/Index.java. Use target InvertedIndexUtil constants and IndexDef.IndexType; preserve the source phrase-default decision. 20. Use the target Java 8-compatible ImmutableMap/ImmutableList APIs in the new custom-normalizer case. 21. Preserve the existing shadow-column test and JUnit4 fixture. Add the source assertions using Assert, IndexDef.IndexType and Java 8-compatible Guava collections. 22. Keep both full-table and partition rejection cases after #62181; the partition row was restored in the final compatibility commit. Shorten the job-lifecycle comment. 23. Set requiresHeavyChange for the target column operations. Source-only sequence-mapping branches do not exist in branch-4.2 and are not introduced. 24. Use useLightIndexChange for dispatch. The target has no source changelog guard, so preserve the target dispatch around that calculation. 25. Keep both heavy-job assertions and add required imports. For the target grammar, replace source USING BLOOMFILTER with NGRAM_BF and disable new-data-only indexing; both exercise the required rebuild of existing rows. 26. Preserve the target writer.add_values(column, values.data(), values.size()) API and apply the source test/comment change. 27. Use DropIndexClause in the legacy AlterClause loop. Preserve the target full-table light-drop helper and add the physical-only partition path; set isDropIndex on the existing task submission. 28. Use org.apache.doris.info.PartitionNamesInfo and pass it through translateToLegacyAlterClause(). Extend target analysis/DropIndexClause with the same constructor, accessors, validation and SQL rendering. 29. Use the target PartitionNamesInfo package and translate validated operations to AlterClauses before handler calls. Add an assertion that partition names survive translation; preserve existing target SNII/cloud tests. 30. Use AlterClause/DropIndexClause and insert the only-clause guard before metadata mutation; preserve the existing light-drop pre-check. 31. Keep all source mixed-clause rejection and metadata/job-count assertions; shorten the explanatory comment. 32. Apply the #68607 comment cleanup, then the later #70007 accounting changes in this batch. The later implementation updates these comments or test fixtures, so the final file differs from the intermediate #68607 source. The corpus checks and source behavior remain covered. 33. Preserve the #70007 accounting algorithm and shorten its comments to one or two plain sentences. A direct source-to-candidate diff for these production and test files contains comment changes only. 34. Preserve all ten original partition-drop cases and assertions. On the target, CREATE INDEX registers metadata but does not schedule a physical build, so add explicit build_index_on_table calls before each existing build wait. Final compatibility commits remove a duplicate CLucene norms assignment, adapt the array-offset test call, add FE imports, replace Java 9 collection APIs, restore the partition-conflict assertion, and use a supported heavy NGRAM_BF test operation. A final comment-only commit shortens index/accounting comments without changing algorithms or SQL. </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
