github-actions[bot] commented on code in PR #67441:
URL: https://github.com/apache/doris/pull/67441#discussion_r3912305299
##########
be/test/format_v2/parquet/parquet_statistics_test.cpp:
##########
@@ -1820,6 +1853,105 @@ TEST(NativeParquetStatisticsTest,
TypeDefinedBoundsRequireSupportedColumnOrder)
EXPECT_TRUE(selected_ranges.empty());
}
+TEST(NativeParquetStatisticsTest,
ArrayContainsUsesRepeatedLeafFooterStatisticsOnly) {
+ const auto encode_int32 = [](int32_t value) {
+ std::string bytes(sizeof(value), '\0');
+ memcpy(bytes.data(), &value, sizeof(value));
+ return bytes;
+ };
+
+ const auto leaf_type = std::make_shared<DataTypeInt32>();
+ const auto array_type = std::make_shared<DataTypeArray>(leaf_type);
+ auto root_schema =
std::make_unique<format::parquet::ParquetColumnSchema>();
+ root_schema->kind = format::parquet::ParquetColumnSchemaKind::LIST;
+ root_schema->local_id = 0;
+ root_schema->name = "items";
+ root_schema->type = array_type;
+ root_schema->max_repetition_level = 1;
+ auto leaf_schema =
std::make_unique<format::parquet::ParquetColumnSchema>();
+ leaf_schema->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE;
+ leaf_schema->local_id = 0;
+ leaf_schema->name = "element";
+ leaf_schema->leaf_column_id = 0;
+ leaf_schema->type = leaf_type;
+ leaf_schema->type_descriptor.doris_type = leaf_type;
+ leaf_schema->type_descriptor.physical_type = tparquet::Type::INT32;
+ leaf_schema->max_repetition_level = 1;
+ root_schema->children.push_back(std::move(leaf_schema));
+ std::vector<std::unique_ptr<format::parquet::ParquetColumnSchema>> schema;
+ schema.push_back(std::move(root_schema));
+
+ tparquet::Statistics statistics;
+ statistics.__set_min_value(encode_int32(10));
+ statistics.__set_max_value(encode_int32(20));
+ statistics.__set_null_count(0);
+ tparquet::ColumnMetaData column_metadata;
+ column_metadata.__set_type(tparquet::Type::INT32);
+ column_metadata.__set_num_values(2);
+ column_metadata.__set_statistics(statistics);
+ tparquet::ColumnChunk chunk;
+ chunk.__set_meta_data(column_metadata);
+ tparquet::RowGroup row_group;
+ row_group.__set_columns({chunk});
+ row_group.__set_num_rows(1);
+ tparquet::ColumnOrder order;
+ order.__set_TYPE_ORDER(tparquet::TypeDefinedOrder());
+ tparquet::FileMetaData metadata;
+ metadata.__set_column_orders({order});
+ metadata.__set_row_groups({row_group});
+
+ const auto select_for_value = [&](int32_t value,
+ format::parquet::ParquetPruningStats*
pruning_stats) {
+ format::FileScanRequest request;
Review Comment:
[P1] Exercise and enable the production metadata-safe path
This hand-built request keeps `metadata_pruning_safe_conjunct_count` at its
unlimited default, but production TableReader classifies `array_contains` as
unsafe because `VectorizedFnCall::is_safe_to_execute_on_selected_rows()` does
not include it. ColumnMapper then leaves the safe count at zero, so
`metadata_pruning_conjuncts()` is empty and neither `has_expr_zonemap_filter()`
nor `check_native_statistics()` can reach this feature. Please make the
predicate safely reachable through the real mapper path and add a
TableReader/ColumnMapper integration test; when doing so, also keep the
PageIndex pre-I/O gate from loading indexes for the repeated leaf, since page
pruning intentionally rejects it.
##########
be/src/exprs/function/array/function_array_index.h:
##########
@@ -218,6 +220,21 @@ class FunctionArrayIndex : public IFunction {
return _execute_dispatch(block, arguments, result, input_rows_count);
}
+ ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx,
+ const VExprSPtrs& arguments)
const override {
+ if constexpr (!std::is_same_v<ConcreteAction, ArrayContainsAction>) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ return expr_zonemap::evaluate_array_contains_zonemap(ctx, arguments);
Review Comment:
[P1] Keep this capability from failing non-V2 ZoneMap callers
This shared function capability is also consumed by Segment and the legacy
Parquet reader, not just FileScanner V2. Those callers bind the slot's
top-level `DataTypeArray` (and may have no ZoneMap for the complex wrapper),
whereas `evaluate_array_contains_zonemap()` extracts the element type and calls
`fetch_compatible_slot_type()` before checking `ctx.zone_map()`. The
ARRAY-versus-element mismatch therefore raises an always-on `DORIS_CHECK` and
the scanner reports a failed query for ordinary pushed-down
`array_contains(array_col, literal)` filters. Please scope repeated-element
capability to the V2 row-group context, or return unsupported for a non-element
binding before the invariant-checking helper, and cover Segment and legacy
Parquet callers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
