Gabriel39 commented on code in PR #67784:
URL: https://github.com/apache/doris/pull/67784#discussion_r4056549385
##########
be/src/exprs/create_predicate_function.h:
##########
@@ -130,7 +131,11 @@ inline auto create_minmax_filter(PrimitiveType type, bool
null_aware) {
}
template <size_t N = 0>
-inline auto create_set(PrimitiveType type, bool null_aware) {
+inline HybridSetBase* create_set(PrimitiveType type, bool null_aware) {
Review Comment:
Addressed in 19bb5f231e.
Removed binary support from the shared predicate factory and restored
StringSet. Binary IN/NOT IN now returns NotSupported; the factory rejects
binary storage/runtime predicates. Tests cover IN/NOT IN and both set/min-max
filter factory rejection.
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -185,6 +185,12 @@ const BloomFilterEvalContext::SlotBloomFilter*
BloomFilterEvalContext::slot(int
TExprNode create_texpr_node_from_hybrid_set_value(const void* data, const
PrimitiveType& type,
int precision, int scale) {
+ if (type == TYPE_VARBINARY) {
Review Comment:
Addressed in 19bb5f231e.
Removed binary hybrid-set materialization from the zone-map helper. Binary
IN is unsupported and does not advertise zone-map, dictionary, or Bloom
evaluation capabilities. Added checks for all three capabilities.
##########
be/src/format/arrow/arrow_block_convertor.cpp:
##########
@@ -118,6 +120,94 @@ std::shared_ptr<arrow::DataType> extension_storage_type(
}
}
+bool is_declared_plain_arrow_binding(const DataTypePtr& type,
Review Comment:
Addressed in 19bb5f231e.
Documented the Doris logical type, ordinary SerDe Arrow mapping, and
requested target representation, with the DATETIMEV2 versus TIMESTAMPTZ
timezone example and the recursive nested-type rule.
##########
be/src/format/table/paimon/paimon_arrow_block_convertor.cpp:
##########
@@ -0,0 +1,45 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "format/table/paimon/paimon_arrow_block_convertor.h"
+
+#include <arrow/array/builder_base.h>
+#include <arrow/type.h>
+
+namespace doris::paimon {
+#include "common/compile_check_begin.h"
+
+Status PaimonArrowBlockConvertor::write_column(const std::shared_ptr<const
IDataType>& type,
+ const DataTypeSerDe& serde,
const IColumn& column,
+ const NullMap* null_map,
+ const
std::shared_ptr<arrow::Field>& field,
+ arrow::ArrayBuilder*
array_builder, int64_t start,
+ int64_t end, const
cctz::time_zone& ctz) const {
+ // This adapter dereferences Arrow declarations that are intentionally
forward-declared by
+ // its public header, so keep the complete definitions local to this
implementation file.
+ return serde.write_column_to_paimon_arrow(type, column, null_map,
+
field->WithType(array_builder->type()), array_builder,
+ start, end, ctz);
+}
+
+const PaimonArrowBlockConvertor& paimon_arrow_block_convertor() {
Review Comment:
Addressed in 19bb5f231e.
Removed paimon_arrow_block_convertor() and its static singleton. Each
JniPaimonWriter now owns its PaimonArrowBlockConvertor, and tests construct
their own instances. Future per-writer converter state no longer needs a global
accessor.
##########
be/src/format/transformer/vparquet_transformer.h:
##########
@@ -83,23 +85,26 @@ struct ParquetFileOptions {
TParquetVersion::type parquet_version;
bool parquet_disable_dictionary = false;
bool enable_int96_timestamps = false;
+ // Overrides only INT96 normalization; UTC preserves a wall-clock carrier.
+ std::optional<std::string> int96_timezone = std::nullopt;
};
// a wrapper of parquet output stream
-class VParquetTransformer final : public VFileFormatTransformer {
+class VParquetTransformer : public VFileFormatTransformer {
public:
VParquetTransformer(RuntimeState* state, doris::io::FileWriter*
file_writer,
const VExprContextSPtrs& output_vexpr_ctxs,
std::vector<std::string> column_names, bool
output_object_data,
const ParquetFileOptions& parquet_options,
- const std::string* iceberg_schema_json = nullptr,
- const iceberg::Schema* iceberg_schema = nullptr);
+ std::unique_ptr<ArrowBlockConvertor>
arrow_block_convertor =
Review Comment:
Addressed in 19bb5f231e.
Removed the converter parameter from both Parquet writer constructors. The
generic writer owns its Flight converter, and the Iceberg writer owns its typed
Iceberg converter and selects it internally alongside its schema. The schema
path no longer downcasts an externally supplied converter.
##########
be/src/format_v2/parquet/parquet_timestamp_semantics.h:
##########
@@ -0,0 +1,47 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string>
+
+#include "gen_cpp/PlanNodes_types.h"
+
+namespace doris::format::parquet {
+
+inline constexpr int32_t PARQUET_TIMESTAMP_SEMANTICS_VERSION_1 = 1;
+
+inline std::optional<std::string> get_int96_timezone_override(const
TFileScanRangeParams* params) {
+ if (params == nullptr) {
Review Comment:
Addressed in 19bb5f231e.
Deleted parquet_timestamp_semantics.h and moved timezone selection into
TableReader, its production consumer. Moved the intermediate-protocol test with
it; absent, explicitly empty, explicitly named, and versioned timezone states
remain distinct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
