lxy-9602 commented on code in PR #451:
URL: https://github.com/apache/paimon-cpp/pull/451#discussion_r4345367904
##########
src/paimon/core/operation/internal_read_context.h:
##########
@@ -43,11 +43,31 @@ class RealtimeContext;
// internal read context, contains ReadContext, CoreOptions and TableSchema
class InternalReadContext {
public:
+ struct IndexScoreSource {
+ enum class Type { VECTOR_SEARCH, FULL_TEXT_SEARCH, FULL_TEXT_MATCH };
+
+ bool operator==(const IndexScoreSource& other) const {
+ return type == other.type && field_name == other.field_name;
+ }
+
+ Type type;
+ /// Original table field name, before per-file field mapping.
+ std::string field_name;
+ };
+
Review Comment:
Could we connect `FULL_TEXT_MATCH` to the predicate visitor dispatch?
`VisitFullTextMatch()` is currently declared only on `FileIndexReader`, while
`FunctionVisitor` has no corresponding method and
`PredicateUtils::VisitPredicate()` has no matching case. Consequently, passing
this leaf through the existing visitor path returns `Invalid("invalid
FullTextMatch function in leaf predicate")` without reaching the new method.
Since execution is intentionally deferred, could we either establish the
dispatch now with a default `NotImplemented` implementation, or explicitly
return `NotImplemented` for this predicate and add a TODO describing the
intended dispatch?
##########
src/paimon/core/utils/field_mapping.cpp:
##########
Review Comment:
I just happened to notice this. Please change the comment `/*data field
name*/` above to `/*data_field_name=*/`.
##########
include/paimon/read_context.h:
##########
@@ -346,24 +347,34 @@ class PAIMON_EXPORT ReadContextBuilder {
/// It can significantly improve performance by reducing the amount of data
/// that needs to be read and processed.
///
+ /// All referenced fields, including full-text match fields, must be
projected in the read
+ /// schema. Full-text matches can be combined with scalar predicates using
Review Comment:
Please add a TODO for "All referenced fields must be projected in the read"
##########
include/paimon/predicate/full_text_match.h:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <optional>
+#include <string>
+#include <utility>
+
+#include "paimon/utils/roaring_bitmap64.h"
+#include "paimon/visibility.h"
+
+namespace paimon {
+
+/// A full-text condition returning every matching row without TopK truncation.
+/// Project `_<field>_INDEX_SCORE` in the read schema to return its scores.
+struct PAIMON_EXPORT FullTextMatch {
+ FullTextMatch(const std::string& _field_name, const std::string& _query,
+ std::optional<RoaringBitmap64> _include_row_ids =
std::nullopt)
+ : field_name(_field_name), query(_query),
include_row_ids(std::move(_include_row_ids)) {}
+
+ bool operator==(const FullTextMatch& other) const {
+ return field_name == other.field_name && query == other.query &&
+ include_row_ids == other.include_row_ids;
+ }
+
+ /// Table column containing the indexed text.
+ std::string field_name;
+ /// JSON DSL query with the same syntax and analyzer semantics as
`FullTextSearch::query`.
+ std::string query;
+ /// Optional eligible row ids. File readers use file-local physical row
positions;
+ /// global readers use global row ids. An empty set matches no rows. If not
+ /// set, all rows are searched.
+ std::optional<RoaringBitmap64> include_row_ids;
Review Comment:
In “global readers use global row ids”, does `FullTextMatch` also support
global index?
##########
docs/source/api/file_index.rst:
##########
@@ -21,9 +21,101 @@ File Index
.. _cpp-api-file-index:
+Full-text predicates and root TopK
+=================================
+
+``PredicateBuilder::FullTextMatch`` creates a ``LeafPredicate`` for a full-text
+condition without TopK truncation. It holds a shared ``FullTextMatch`` request,
+whose query uses the same JSON DSL as ``FullTextSearch``. Do not modify a
request
+after passing it to a predicate or read context.
+
+Compose full-text and scalar predicates with the existing
+``PredicateBuilder::And`` and ``PredicateBuilder::Or`` factories. Every field
+referenced by this tree must appear in the read schema; field indices refer to
+positions in that schema. For example, the following constructs
+``body matches "apple" OR price < 10``:
+
+.. code-block:: cpp
+
+ auto request = std::make_shared<paimon::FullTextMatch>(
+ "body", R"({"match":{"query":"apple"}})");
+ PAIMON_ASSIGN_OR_RAISE(
+ std::shared_ptr<paimon::Predicate> text,
+ paimon::PredicateBuilder::FullTextMatch(0, request));
+ auto price = paimon::PredicateBuilder::LessThan(
+ 1, "price", paimon::FieldType::INT, paimon::Literal(10));
+ PAIMON_ASSIGN_OR_RAISE(
+ std::shared_ptr<paimon::Predicate> filter,
+ paimon::PredicateBuilder::Or({text, price}));
+ paimon::ReadContextBuilder builder(table_path);
+ builder.SetReadFieldNames({"body", "price"}).SetPredicate(filter);
+
+``SetVectorSearch`` and ``SetFullTextSearch`` configure a separate root TopK
+request. They are mutually exclusive and require a positive limit. The intended
+semantics are to filter by the complete predicate before ranking; the TopK
search
+field does not need to be projected. A full-text match inside a predicate never
+has a TopK limit.
+
+Score output is requested only through an explicit ``_<field>_INDEX_SCORE``
+projection. ``FullTextMatch``, ``SetVectorSearch`` and ``SetFullTextSearch``
have
+no separate score switch. For example, request the score of a full-text
+predicate by projecting ``_body_INDEX_SCORE``:
+
+.. code-block:: cpp
+
+ auto request = std::make_shared<paimon::FullTextMatch>(
+ "body", R"({"match":{"query":"apple"}})");
+ PAIMON_ASSIGN_OR_RAISE(
+ std::shared_ptr<paimon::Predicate> text,
+ paimon::PredicateBuilder::FullTextMatch(0, request));
+ paimon::ReadContextBuilder builder(table_path);
+ builder.SetReadFieldNames({"body", "_body_INDEX_SCORE"}).SetPredicate(text);
+
+Score fields have type ``float32`` and preserve the source field's exact name,
+including case. Select them with ``SetReadFieldNames`` or ``SetReadSchema``;
+they have no persistent field IDs for ``SetReadFieldIds``. Reading all table
+fields does not implicitly add score fields.
+
+Root TopK searches always need scores internally to rank and merge results
+across files, regardless of the projection. A full-text predicate without TopK
+needs scores only when its score field is projected.
+
+Each projected score must have exactly one search condition as its source.
+The source may be a root TopK request or a full-text predicate. A full-text
+predicate whose score is projected cannot have an OR ancestor, and its field
+cannot occur in another search condition, including the root TopK request.
+An ordinary scalar predicate on the same field is allowed. For example, when
+projecting the score of ``match``, ``match AND (p OR q)`` is valid, while
+``match OR p`` is rejected. Without that score projection, both OR branches and
+repeated full-text search fields are allowed. These checks use the effective
+read projection, after resolving the read schema.
+
+Internally, the read context binds each score output position to a search type
+and the original table field name. The binding identifies a logical source,
+not a request object's identity. Execution must preserve this source when
+copying requests for per-file field mapping or candidate filtering.
+``NeedsScores`` derives execution requirements from the root TopK request and
+these bindings; it does not add another user configuration.
+
Review Comment:
I’m not sure this API documentation needs to describe the internal
implementation details for functions that aren’t actual APIs. Please adjust it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
