leaves12138 commented on code in PR #10494: URL: https://github.com/apache/paimon/pull/10494#discussion_r4237759603
########## paimon-python/pypaimon/table/source/native_full_text_search.py: ########## @@ -0,0 +1,61 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Local REST full-text search delegated to Rust core.""" + +from pypaimon.catalog.table_query_auth import reject_search_under_query_auth +from pypaimon.globalindex.full_text.native_full_text_global_index_reader import NativeFullTextIndexOptions +from pypaimon.globalindex.vector_search_result import DictBasedScoredIndexResult +from pypaimon.read.native_plan import _native_table, _predicate_to_native, native_method_available +from pypaimon.table.file_store_table import FileStoreTable +from pypaimon.write.native_commit import _rest_catalog_supported + + +def try_native_full_text_search(builder): + """Qualify the route before execution; Rust owns scan, filtering and ranking.""" + table = builder._table + reject_search_under_query_auth(table) + if (type(table) is not FileStoreTable + or not table.options.native_read_enabled() + or not table.options.data_evolution_enabled() + or not table.options.row_tracking_enabled() + or table.is_primary_key_table + or table.options.file_format() != 'parquet' + or not native_method_available('Table', 'new_full_text_search_builder') + or not _rest_catalog_supported(table)): + return None + if builder._limit is None or builder._limit <= 0: + raise ValueError('Limit must be positive, set via with_limit()') + # The full-text adapter accepts typed lists and booleans. Reuse its mapping + # before the generic table bridge stringifies options for Rust. + options = {'full-text.' + key: value for key, value in NativeFullTextIndexOptions.from_options( + table.options.options.to_map()).to_native_options().items()} + native = _native_table(table.copy(options)).new_full_text_search_builder() + if builder._field_name is not None and builder._query is not None: + native.with_query(builder._field_name, builder._query) + native.with_limit(builder._limit) + try: + if builder._partition_filter is not None: + native.with_partition_filter(_predicate_to_native(builder._partition_filter)) + if builder._filter is not None: + native.with_filter(_predicate_to_native(builder._filter)) Review Comment: [P2] Preserve Java search exactness when native predicates are folded The new native dispatch passes the original filter through `_predicate_to_native`, but that bridge uses Rust's predicate builder and can change the search exactness contract before full-text ranking. Java `LikeOptimization` requires a nonempty literal/prefix/suffix/substring, so `LIKE ''`, `LIKE '%'` and `LIKE '%%'` remain `Like` leaves; `FilteredRowIdReader.isExact` excludes their indexed candidates when `global-index.filter.refine-from-data=false`. The new classic helper correctly follows that policy. Rust instead rewrites `LIKE ''` to `Eq('')` and the all-wildcard patterns to `IsNotNull`; empty `contains`/`endsWith` literals are also folded to `IsNotNull`. `GlobalIndexScanner::is_exact_predicate` then accepts those normalized leaves as exact. This option defaults to false, so enabling native reads now changes the result set for a supported REST table. The normalization predates this PR, but the new full-text route exposes it. Reproduced on Python head `a604c96fe4b5603133f9eb358eaa06a2ab8837ce` with Rust head `3badea107c3a8f434c82aec239a0b89cc3ad8049`: use the existing REST fixture, DE + row tracking + Parquet, FULL text/scalar modes, `content='paimon'` for all rows, and labels `['', None, 'keep', 'drop', '', 'keep', 'drop', None]`. Build the content full-text index and a label bitmap or BTree index, and search with limit 64: - Refinement disabled, complete scalar coverage: `LIKE '%'` / `LIKE '%%'` / `contains('')` / `endsWith('')` return no rows on classic, but `{0,2,3,4,5,6}` on native. `LIKE ''` returns no rows on classic, but `{0,4}` on native. - After appending labels `['', None, 'keep', 'drop']` as rows 8-11 without rebuilding the indexes: classic returns only `{8,10,11}` for the all-non-null predicates and `{8}` for `LIKE ''`; native additionally retains the indexed rows above. - The focused matrix (2 index families x 5 predicates x complete/partial coverage x refinement off/on) gives 20 failures, all with refinement disabled, and 20 passing refinement-enabled controls. Native execution is wrapped with the existing assertion against a Python full-text scan retry. Please preserve the original Java-aligned candidate-only classification across native conversion, or align the corresponding Rust/FFI predicate normalization for search. Do not relax the classic helper just to match the folded native predicates. Add these empty/all-wildcard cases with both index families and partial coverage; `LIKE '%'` is a normal supported filter, not an invalid input. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
