github-actions[bot] commented on code in PR #66149:
URL: https://github.com/apache/doris/pull/66149#discussion_r3667991207
##########
be/src/format_v2/parquet/reader/native/column_chunk_reader.cpp:
##########
@@ -1725,22 +1840,77 @@ Status ColumnChunkReader<IN_COLLECTION,
OFFSET_INDEX>::filter_fixed_width_values
selection.total_values);
}
+ VExprSPtrs raw_conjuncts;
+ VExprSPtrs null_map_conjuncts;
+ for (const auto& conjunct : conjuncts) {
+ if (conjunct->can_execute_on_null_map(_field_schema->data_type,
column_id)) {
+ null_map_conjuncts.push_back(conjunct);
+ } else {
+ raw_conjuncts.push_back(conjunct);
+ }
+ }
physical_matches->clear();
+ IColumn::Filter physical_conversion_nulls;
if (selection.selected_values == 0) {
RETURN_IF_ERROR(_page_decoder->skip_values(selection.total_values));
+ } else if (raw_conjuncts.empty()) {
+ RETURN_IF_ERROR(_page_decoder->skip_values(selection.total_values));
+ physical_matches->resize_fill(selection.selected_values, 1);
} else {
- FixedWidthPredicateConsumer consumer(conjuncts,
_field_schema->data_type, column_id,
+ const bool all_raw_binary = std::ranges::all_of(raw_conjuncts,
[&](const auto& conjunct) {
+ return
conjunct->can_execute_on_raw_binary_values(_field_schema->data_type, column_id);
+ });
+ const bool all_raw_fixed = std::ranges::all_of(raw_conjuncts,
[&](const auto& conjunct) {
+ return
conjunct->can_execute_on_raw_fixed_values(_field_schema->data_type, column_id);
+ });
+ if (all_raw_binary &&
+ supports_raw_binary_filter_encoding(_current_encoding,
_metadata.type)) {
+ BinaryPredicateConsumer consumer(raw_conjuncts,
_field_schema->data_type, column_id,
physical_matches,
projected_column);
- RETURN_IF_ERROR(_page_decoder->decode_selected_fixed_values(selection,
consumer));
+ if (_metadata.type == tparquet::Type::BYTE_ARRAY) {
+
RETURN_IF_ERROR(_page_decoder->decode_selected_binary_values(selection,
consumer));
+ } else {
+
RETURN_IF_ERROR(_page_decoder->decode_selected_fixed_values(selection,
consumer));
+ }
+ } else if (all_raw_fixed &&
serde.supports_parquet_raw_predicate(decode_context)) {
+ // Type conversion is fused between the decoder and predicate
sink. Conversion-null
+ // bits travel beside the POD batch, preserving permissive scan
semantics without an
+ // intermediate nullable IColumn.
+ SelectedDecodeSource selected_source(*_page_decoder, selection);
+ FixedWidthPredicateConsumer consumer(raw_conjuncts,
_field_schema->data_type, column_id,
+ physical_matches,
projected_column,
+ &physical_conversion_nulls);
+ RETURN_IF_ERROR(serde.read_parquet_raw_predicate(selected_source,
decode_context,
+
selection.selected_values,
+
enable_strict_mode, consumer));
+ } else {
+ FixedWidthPredicateConsumer consumer(raw_conjuncts,
_field_schema->data_type, column_id,
+ physical_matches,
projected_column);
+
RETURN_IF_ERROR(_page_decoder->decode_selected_fixed_values(selection,
consumer));
+ }
DORIS_CHECK_EQ(physical_matches->size(), selection.selected_values);
}
row_filter->reserve(selected_nulls->size());
size_t physical_row = 0;
- for (const uint8_t is_null : *selected_nulls) {
- row_filter->push_back(is_null != 0 ? 0 :
(*physical_matches)[physical_row++]);
+ for (size_t logical_row = 0; logical_row < selected_nulls->size();
++logical_row) {
+ uint8_t& is_null = (*selected_nulls)[logical_row];
+ if (is_null != 0) {
Review Comment:
[P1] Preserve every TopN row until the first bound
`execute_column()` is explicitly all-pass while `has_value()` is false, but
this direct route can still reject logical NULLs. It maps physical NULLs to
false whenever a raw conjunct exists, and a permissive conversion failure
starts with a false raw match that is not restored when `selected_nulls` is
set. `VTopNPred` admits `NULLS LAST` before bound publication, and
dictionary-ID paths hard-code the physical-NULL rejection too. An all-NULL
input or required invalid DATE can therefore be discarded before it can fill
`LIMIT`, returning too few rows. Please preserve every row until the first
bound (or keep this predicate residual), with optional PLAIN/binary/dictionary
and required invalid-DATE coverage.
##########
be/src/exprs/vtopn_pred.cpp:
##########
@@ -0,0 +1,277 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/vtopn_pred.h"
+
+#include <cstring>
+
+#include "exprs/expr_zonemap_filter.h"
+#include "util/simd/parquet_kernels.h"
+
+namespace doris {
+
+namespace {
+
+using simd::RawComparisonOp;
+
+size_t topn_raw_value_size(PrimitiveType type) {
+ switch (type) {
+#define RETURN_TOPN_RAW_SIZE(TYPE) \
+ case TYPE: \
+ return sizeof(typename PrimitiveTypeTraits<TYPE>::CppType)
+ RETURN_TOPN_RAW_SIZE(TYPE_BOOLEAN);
+ RETURN_TOPN_RAW_SIZE(TYPE_TINYINT);
+ RETURN_TOPN_RAW_SIZE(TYPE_SMALLINT);
+ RETURN_TOPN_RAW_SIZE(TYPE_INT);
+ RETURN_TOPN_RAW_SIZE(TYPE_BIGINT);
+ RETURN_TOPN_RAW_SIZE(TYPE_LARGEINT);
+ RETURN_TOPN_RAW_SIZE(TYPE_FLOAT);
+ RETURN_TOPN_RAW_SIZE(TYPE_DOUBLE);
+ RETURN_TOPN_RAW_SIZE(TYPE_DATE);
+ RETURN_TOPN_RAW_SIZE(TYPE_DATETIME);
+ RETURN_TOPN_RAW_SIZE(TYPE_DATEV2);
+ RETURN_TOPN_RAW_SIZE(TYPE_DATETIMEV2);
+ RETURN_TOPN_RAW_SIZE(TYPE_TIMESTAMPTZ);
+ RETURN_TOPN_RAW_SIZE(TYPE_TIME);
+ RETURN_TOPN_RAW_SIZE(TYPE_TIMEV2);
+ RETURN_TOPN_RAW_SIZE(TYPE_DECIMAL32);
+ RETURN_TOPN_RAW_SIZE(TYPE_DECIMAL64);
+ RETURN_TOPN_RAW_SIZE(TYPE_DECIMALV2);
+ RETURN_TOPN_RAW_SIZE(TYPE_DECIMAL128I);
+ RETURN_TOPN_RAW_SIZE(TYPE_DECIMAL256);
+ RETURN_TOPN_RAW_SIZE(TYPE_IPV4);
+ RETURN_TOPN_RAW_SIZE(TYPE_IPV6);
+#undef RETURN_TOPN_RAW_SIZE
+ default:
+ return 0;
+ }
+}
+
+bool topn_binary_type(PrimitiveType type) {
+ return is_string_type(type) || type == TYPE_VARBINARY;
+}
+
+template <typename T, PrimitiveType PT>
+void execute_topn_raw_comparison(const uint8_t* values, size_t num_values,
const Field& bound,
+ RawComparisonOp op, uint8_t* matches) {
+ simd::raw_compare(values, num_values, bound.get<PT>(), op, matches);
+}
+
+template <typename T>
+bool topn_raw_scalar_matches(const T& value, const T& bound, RawComparisonOp
op) {
+ return op == RawComparisonOp::LE ? value <= bound : value >= bound;
+}
+
+template <PrimitiveType PT>
+void execute_topn_raw_scalar(const uint8_t* values, size_t num_values, const
Field& bound,
+ RawComparisonOp op, uint8_t* matches) {
+ using T = typename PrimitiveTypeTraits<PT>::CppType;
+ const T& typed_bound = bound.get<PT>();
+ for (size_t row = 0; row < num_values; ++row) {
+ T value;
+ std::memcpy(&value, values + row * sizeof(T), sizeof(T));
+ matches[row] &= topn_raw_scalar_matches(value, typed_bound, op) ? 1 :
0;
+ }
+}
+
+bool topn_string_matches(int comparison, RawComparisonOp op) {
+ return op == RawComparisonOp::LE ? comparison <= 0 : comparison >= 0;
+}
+
+} // namespace
+
+bool VTopNPred::can_execute_on_raw_fixed_values(const DataTypePtr& data_type,
int column_id) const {
+ if (_predicate == nullptr || data_type == nullptr || _children.size() != 1
||
+ (_predicate->nulls_first() && data_type->is_nullable())) {
Review Comment:
[P1] Use effective conversion nullability for TopN capability
This checks the internal Parquet type, so a required DATE is considered
non-nullable and `NULLS FIRST` direct filtering is enabled. Required fields are
nevertheless exposed through a nullable scan column, and non-strict conversion
of an invalid DATE can synthesize NULL; ordinary TopN keeps that row for `NULLS
FIRST`. Here the conversion bitmap first sets its raw match to false, and later
marking `selected_nulls` does not restore it. Base this gate on the effective
slot/conversion-result nullability (or retain the residual for
conversion-capable types), and add a required invalid-DATE pushdown
differential before and after bound publication.
##########
be/src/format_v2/parquet/parquet_scan.cpp:
##########
@@ -1925,10 +1988,16 @@ Status
ParquetScanScheduler::read_filter_columns(int64_t batch_rows,
&used_filter));
if (used_filter) {
DORIS_CHECK_EQ(compact_filter.size(),
selected_rows_before);
-
update_counter_if_not_null(_scan_profile.fixed_width_predicate_direct_batches,
- 1);
-
update_counter_if_not_null(_scan_profile.fixed_width_predicate_direct_rows,
+
update_counter_if_not_null(_scan_profile.raw_value_predicate_direct_batches, 1);
Review Comment:
[P2] Attribute direct-path counters to the executed kernel
`used_filter` also means the definition-level-only path succeeded, so this
increments `RawValuePredicateDirect*` even when `raw_conjuncts` is empty and no
value predicate ran; the new NULL-predicate test currently codifies that
attribution. Conversely, `TYPE_VARBINARY` is routed through raw binary TopN but
`is_string_type()` excludes it, so it is counted as fixed-width here. Return
the actual execution kind (definition-only/raw fixed/raw binary/typed) and
update only the matching counters so profiles can show which materialization
work was avoided.
##########
be/src/exprs/vcompound_pred.h:
##########
@@ -484,6 +530,36 @@ class VCompoundPred : public VectorizedFnCall {
}
private:
+ template <typename ExecuteChild>
+ Status _execute_raw_compound(size_t num_values, uint8_t* matches,
+ ExecuteChild&& execute_child) const {
+ if (_op == TExprOpcode::COMPOUND_AND) {
+ for (const auto& child : _children) {
+ RETURN_IF_ERROR(execute_child(child, matches));
+ }
+ return Status::OK();
+ }
+
+ IColumn::Filter combined(num_values, _op == TExprOpcode::COMPOUND_NOT
? 1 : 0);
Review Comment:
[P2] Reuse compound raw masks across decoder fragments
Every raw OR callback allocates/fills `combined` and then allocates/fills a
fresh `child_matches` for each child. Because the decoder invokes this kernel
per selected page/range fragment, an N-child OR performs N+1 row-sized
temporary allocations on every fragment, even for identity/string paths that
otherwise avoid an intermediate value column. This is separate from the SerDe
conversion scratch already reported. Please move these masks to reusable
bounded reader/caller scratch (or keep OR residual) and cover fragmented
multi-page reuse.
##########
be/src/exprs/vtopn_pred.cpp:
##########
@@ -0,0 +1,277 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/vtopn_pred.h"
+
+#include <cstring>
+
+#include "exprs/expr_zonemap_filter.h"
+#include "util/simd/parquet_kernels.h"
+
+namespace doris {
+
+namespace {
+
+using simd::RawComparisonOp;
+
+size_t topn_raw_value_size(PrimitiveType type) {
+ switch (type) {
+#define RETURN_TOPN_RAW_SIZE(TYPE) \
+ case TYPE: \
+ return sizeof(typename PrimitiveTypeTraits<TYPE>::CppType)
+ RETURN_TOPN_RAW_SIZE(TYPE_BOOLEAN);
+ RETURN_TOPN_RAW_SIZE(TYPE_TINYINT);
+ RETURN_TOPN_RAW_SIZE(TYPE_SMALLINT);
+ RETURN_TOPN_RAW_SIZE(TYPE_INT);
+ RETURN_TOPN_RAW_SIZE(TYPE_BIGINT);
+ RETURN_TOPN_RAW_SIZE(TYPE_LARGEINT);
+ RETURN_TOPN_RAW_SIZE(TYPE_FLOAT);
+ RETURN_TOPN_RAW_SIZE(TYPE_DOUBLE);
+ RETURN_TOPN_RAW_SIZE(TYPE_DATE);
+ RETURN_TOPN_RAW_SIZE(TYPE_DATETIME);
+ RETURN_TOPN_RAW_SIZE(TYPE_DATEV2);
+ RETURN_TOPN_RAW_SIZE(TYPE_DATETIMEV2);
+ RETURN_TOPN_RAW_SIZE(TYPE_TIMESTAMPTZ);
+ RETURN_TOPN_RAW_SIZE(TYPE_TIME);
+ RETURN_TOPN_RAW_SIZE(TYPE_TIMEV2);
+ RETURN_TOPN_RAW_SIZE(TYPE_DECIMAL32);
+ RETURN_TOPN_RAW_SIZE(TYPE_DECIMAL64);
+ RETURN_TOPN_RAW_SIZE(TYPE_DECIMALV2);
+ RETURN_TOPN_RAW_SIZE(TYPE_DECIMAL128I);
+ RETURN_TOPN_RAW_SIZE(TYPE_DECIMAL256);
+ RETURN_TOPN_RAW_SIZE(TYPE_IPV4);
+ RETURN_TOPN_RAW_SIZE(TYPE_IPV6);
+#undef RETURN_TOPN_RAW_SIZE
+ default:
+ return 0;
+ }
+}
+
+bool topn_binary_type(PrimitiveType type) {
+ return is_string_type(type) || type == TYPE_VARBINARY;
+}
+
+template <typename T, PrimitiveType PT>
+void execute_topn_raw_comparison(const uint8_t* values, size_t num_values,
const Field& bound,
+ RawComparisonOp op, uint8_t* matches) {
+ simd::raw_compare(values, num_values, bound.get<PT>(), op, matches);
+}
+
+template <typename T>
+bool topn_raw_scalar_matches(const T& value, const T& bound, RawComparisonOp
op) {
+ return op == RawComparisonOp::LE ? value <= bound : value >= bound;
+}
+
+template <PrimitiveType PT>
+void execute_topn_raw_scalar(const uint8_t* values, size_t num_values, const
Field& bound,
+ RawComparisonOp op, uint8_t* matches) {
+ using T = typename PrimitiveTypeTraits<PT>::CppType;
+ const T& typed_bound = bound.get<PT>();
+ for (size_t row = 0; row < num_values; ++row) {
+ T value;
+ std::memcpy(&value, values + row * sizeof(T), sizeof(T));
+ matches[row] &= topn_raw_scalar_matches(value, typed_bound, op) ? 1 :
0;
+ }
+}
+
+bool topn_string_matches(int comparison, RawComparisonOp op) {
+ return op == RawComparisonOp::LE ? comparison <= 0 : comparison >= 0;
+}
+
+} // namespace
+
+bool VTopNPred::can_execute_on_raw_fixed_values(const DataTypePtr& data_type,
int column_id) const {
+ if (_predicate == nullptr || data_type == nullptr || _children.size() != 1
||
+ (_predicate->nulls_first() && data_type->is_nullable())) {
+ return false;
+ }
+ const auto slot = std::dynamic_pointer_cast<VSlotRef>(_children[0]);
+ if (slot == nullptr || slot->column_id() != column_id || slot->data_type()
== nullptr) {
+ return false;
+ }
+ const auto raw_type = remove_nullable(data_type);
+ return remove_nullable(slot->data_type())->equals(*raw_type) &&
+ topn_raw_value_size(raw_type->get_primitive_type()) != 0;
+}
+
+Status VTopNPred::execute_on_raw_fixed_values(const uint8_t* values, size_t
num_values,
+ size_t value_width, const
DataTypePtr& data_type,
+ int column_id, uint8_t* matches)
const {
+ if (!can_execute_on_raw_fixed_values(data_type, column_id)) {
+ return Status::NotSupported("TopN predicate cannot evaluate raw
fixed-width values");
+ }
+ DORIS_CHECK(values != nullptr || num_values == 0);
+ DORIS_CHECK(matches != nullptr || num_values == 0);
+ const auto primitive_type =
remove_nullable(data_type)->get_primitive_type();
+ const size_t expected_width = topn_raw_value_size(primitive_type);
+ if (value_width != expected_width) {
+ return Status::Corruption("Raw TopN width {} does not match expected
{}", value_width,
+ expected_width);
+ }
+
+ // The bound is mutable and may arrive after reader initialization.
Snapshot it once per batch
+ // so every row observes one monotonic TopN frontier without invalidating
cached capability.
+ const Field bound = _predicate->get_value();
+ if (bound.is_null()) {
+ return Status::OK();
+ }
+ if (!expr_zonemap::field_types_compatible(bound.get_type(),
primitive_type)) {
+ return Status::InternalError("TopN bound type {} does not match raw
value type {}",
+ type_to_string(bound.get_type()),
+ type_to_string(primitive_type));
+ }
+ const auto op = _predicate->is_asc() ? RawComparisonOp::LE :
RawComparisonOp::GE;
+ switch (primitive_type) {
+ case TYPE_INT:
+ execute_topn_raw_comparison<int32_t, TYPE_INT>(values, num_values,
bound, op, matches);
+ break;
+ case TYPE_BIGINT:
+ execute_topn_raw_comparison<int64_t, TYPE_BIGINT>(values, num_values,
bound, op, matches);
+ break;
+ case TYPE_FLOAT:
+ execute_topn_raw_comparison<float, TYPE_FLOAT>(values, num_values,
bound, op, matches);
+ break;
+ case TYPE_DOUBLE:
+ execute_topn_raw_comparison<double, TYPE_DOUBLE>(values, num_values,
bound, op, matches);
+ break;
+#define EXECUTE_TOPN_RAW_SCALAR(TYPE)
\
+ case TYPE:
\
+ execute_topn_raw_scalar<TYPE>(values, num_values, bound, op, matches);
\
+ break
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_BOOLEAN);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_TINYINT);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_SMALLINT);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_LARGEINT);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_DATE);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_DATETIME);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_DATEV2);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_DATETIMEV2);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_TIMESTAMPTZ);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_TIME);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_TIMEV2);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_DECIMAL32);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_DECIMAL64);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_DECIMALV2);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_DECIMAL128I);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_DECIMAL256);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_IPV4);
+ EXECUTE_TOPN_RAW_SCALAR(TYPE_IPV6);
+#undef EXECUTE_TOPN_RAW_SCALAR
+ default:
+ return Status::NotSupported("TopN raw fixed-width type {} is
unsupported",
+ type_to_string(primitive_type));
+ }
+ return Status::OK();
+}
+
+bool VTopNPred::can_execute_on_raw_binary_values(const DataTypePtr& data_type,
+ int column_id) const {
+ if (_predicate == nullptr || data_type == nullptr || _children.size() != 1
||
+ (_predicate->nulls_first() && data_type->is_nullable())) {
+ return false;
+ }
+ const auto raw_type = remove_nullable(data_type);
+ const auto raw_primitive_type = raw_type->get_primitive_type();
+ if (!topn_binary_type(raw_primitive_type)) {
+ return false;
+ }
+ const auto slot = std::dynamic_pointer_cast<VSlotRef>(_children[0]);
+ if (slot == nullptr || slot->column_id() != column_id || slot->data_type()
== nullptr) {
+ return false;
+ }
+ const auto slot_primitive_type =
remove_nullable(slot->data_type())->get_primitive_type();
+ return (is_string_type(raw_primitive_type) &&
is_string_type(slot_primitive_type)) ||
+ (raw_primitive_type == TYPE_VARBINARY && slot_primitive_type ==
TYPE_VARBINARY);
+}
+
+Status VTopNPred::execute_on_raw_binary_values(const StringRef* values, size_t
num_values,
+ const DataTypePtr& data_type,
int column_id,
+ uint8_t* matches) const {
+ if (!can_execute_on_raw_binary_values(data_type, column_id)) {
+ return Status::NotSupported("TopN predicate cannot evaluate raw binary
values");
+ }
+ DORIS_CHECK(values != nullptr || num_values == 0);
+ DORIS_CHECK(matches != nullptr || num_values == 0);
+ const Field bound = _predicate->get_value();
+ if (bound.is_null()) {
+ return Status::OK();
+ }
+ const auto primitive_type =
remove_nullable(data_type)->get_primitive_type();
+ StringRef bound_ref;
+ if (is_string_type(primitive_type) && is_string_type(bound.get_type())) {
+ const auto& value = bound.get<TYPE_STRING>();
+ bound_ref = StringRef(value.data(), value.size());
+ } else if (primitive_type == TYPE_VARBINARY && bound.get_type() ==
TYPE_VARBINARY) {
+ bound_ref = bound.get<TYPE_VARBINARY>().to_string_ref();
+ } else {
+ return Status::InternalError("TopN bound type {} does not match raw
binary type {}",
+ type_to_string(bound.get_type()),
+ type_to_string(primitive_type));
+ }
+ const auto op = _predicate->is_asc() ? RawComparisonOp::LE :
RawComparisonOp::GE;
+ for (size_t row = 0; row < num_values; ++row) {
+ matches[row] &= topn_string_matches(values[row].compare(bound_ref),
op) ? 1 : 0;
+ }
+ return Status::OK();
+}
+
+bool VTopNPred::can_evaluate_dictionary_filter() const {
Review Comment:
[P2] Keep late-bound TopN evaluation after dictionary setup
This is considered exactly dictionary-evaluable even before the first bound.
Row-group setup then freezes an all-pass (or older-bound) dictionary bitmap and
removes the TopN residual, so a later bound or tightening is never applied to
any remaining batch in that row group. Monotonic updates make the stale bitmap
correctness-safe for non-NULL values, but they can eliminate the PR's intended
late-bound pruning for a large dictionary-encoded row group. Retain per-batch
TopN evaluation when the bitmap can become stale, or version/rebuild it, with a
bound-after-setup test.
##########
be/src/format_v2/parquet/reader/native_column_reader.cpp:
##########
@@ -747,6 +750,79 @@ Status NativeColumnReader::select_with_fixed_width_filter(
return Status::OK();
}
+Status NativeColumnReader::select_with_runtime_filter(
+ const SelectionVector& selection, uint16_t selected_rows, int64_t
batch_rows,
+ const VExprContextSPtrs& conjuncts, int column_id, MutableColumnPtr*
projected_column,
+ IColumn::Filter* row_filter, bool* used_filter) {
+ DORIS_CHECK(row_filter != nullptr);
+ DORIS_CHECK(used_filter != nullptr);
+ RETURN_IF_ERROR(validate_selected_span(batch_rows));
+ RETURN_IF_ERROR(selection.verify(selected_rows, batch_rows));
+ row_filter->clear();
+ *used_filter = false;
+ if (_nested || conjuncts.empty() || !std::ranges::all_of(conjuncts,
[&](const auto& conjunct) {
+ return conjunct != nullptr && conjunct->root() != nullptr &&
+ conjunct->root()->is_rf_wrapper() &&
+ conjunct->root()->can_execute_on_reader_values(_type,
column_id);
+ })) {
+ return Status::OK();
+ }
+
+ const uint8_t* selection_filter = nullptr;
+ RETURN_IF_ERROR(selection.materialize_filter(selected_rows, batch_rows,
&selection_filter));
+ auto values = _type->create_column();
Review Comment:
[P1] Preserve scan nullability in the typed RF temporary
`values` uses the internal required `_type`, while the normal file block
wraps required Parquet fields in `Nullable`. For a non-strict conversion
failure such as required DATE `INT32_MAX`, ordinary materialization records
NULL and continues; this temporary has no null map, so the same payload returns
`DataQualityError`. The fallback is reachable when reader-value RF evaluation
is supported but raw/dictionary filtering is not, including mixed
dictionary/plain encodings. Materialize with the effective nullable scan type
(or decline this fallback) and add a pushdown-on/off mixed-encoding conversion
test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
