github-actions[bot] commented on code in PR #68161:
URL: https://github.com/apache/doris/pull/68161#discussion_r4244990911
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -1289,6 +1296,81 @@ public Map<String, String> getScanNodeProperties(
return props;
}
+ static boolean containsTimestampType(DataType type) {
+ if (type instanceof org.apache.paimon.types.TimestampType
+ || type instanceof
org.apache.paimon.types.LocalZonedTimestampType) {
+ return true;
+ }
+ if (type instanceof org.apache.paimon.types.RowType) {
+ return ((org.apache.paimon.types.RowType)
type).getFields().stream()
+ .anyMatch(field -> containsTimestampType(field.type()));
+ }
+ if (type instanceof org.apache.paimon.types.ArrayType) {
+ return containsTimestampType(((org.apache.paimon.types.ArrayType)
type).getElementType());
+ }
+ if (type instanceof org.apache.paimon.types.MapType) {
+ org.apache.paimon.types.MapType mapType =
(org.apache.paimon.types.MapType) type;
+ return containsTimestampType(mapType.getKeyType()) ||
containsTimestampType(mapType.getValueType());
+ }
+ return false;
+ }
+
+ private boolean requiresTimestampPrecisionRepair(Table table,
PaimonTableHandle handle) {
+ Table schemaTable = resolveSchemaDictTable(table, handle);
+ if (!(schemaTable instanceof FileStoreTable)) {
+ return false;
+ }
+ FileStoreTable fileStoreTable = (FileStoreTable) schemaTable;
+ RowType currentType = schemaTable.rowType();
+ SchemaManager schemaManager = fileStoreTable.schemaManager();
+ for (Long schemaId : schemaManager.listAllIds()) {
Review Comment:
[P1] Check timestamp history on both fallback branches. This calls
schemaManager() on the FallbackReadFileStoreTable, which delegates only to the
wrapped branch. If the wrapped branch is TIMESTAMP(4) with no narrower history,
but a fallback-only partition contains a file written while its branch was
TIMESTAMP(6) before narrowing to TIMESTAMP(4), this returns false and both FE
split planning and JNI row filtering receive a timestamp equality. Paimon
1.4.2's precision cast leaves a leading-zero value such as 00:00:01.000001
fractional, so it can reject = 00:00:01.0000 before Doris truncates the value.
Inspect each branch's history when deciding whether to suppress timestamp
pushdown.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -1289,6 +1296,81 @@ public Map<String, String> getScanNodeProperties(
return props;
}
+ static boolean containsTimestampType(DataType type) {
+ if (type instanceof org.apache.paimon.types.TimestampType
+ || type instanceof
org.apache.paimon.types.LocalZonedTimestampType) {
+ return true;
+ }
+ if (type instanceof org.apache.paimon.types.RowType) {
+ return ((org.apache.paimon.types.RowType)
type).getFields().stream()
+ .anyMatch(field -> containsTimestampType(field.type()));
+ }
+ if (type instanceof org.apache.paimon.types.ArrayType) {
+ return containsTimestampType(((org.apache.paimon.types.ArrayType)
type).getElementType());
+ }
+ if (type instanceof org.apache.paimon.types.MapType) {
+ org.apache.paimon.types.MapType mapType =
(org.apache.paimon.types.MapType) type;
+ return containsTimestampType(mapType.getKeyType()) ||
containsTimestampType(mapType.getValueType());
+ }
+ return false;
+ }
+
+ private boolean requiresTimestampPrecisionRepair(Table table,
PaimonTableHandle handle) {
+ Table schemaTable = resolveSchemaDictTable(table, handle);
+ if (!(schemaTable instanceof FileStoreTable)) {
+ return false;
+ }
+ FileStoreTable fileStoreTable = (FileStoreTable) schemaTable;
+ RowType currentType = schemaTable.rowType();
+ SchemaManager schemaManager = fileStoreTable.schemaManager();
+ for (Long schemaId : schemaManager.listAllIds()) {
+ TableSchema historicalSchema = schemaManager.schema(schemaId);
Review Comment:
[P2] Avoid reading every historical schema for unrelated predicates. Both
planScanInternal and getScanNodeProperties call this for any nonempty filter,
so even an integer-only table with WHERE id = 1 lists and reads every schema
file twice per query. SchemaManager.schema(id) performs file I/O, making
planning cost grow with the full schema history on remote stores. Short-circuit
when current fields or pushed predicate cannot involve timestamps, and reuse
the history result between the two planning paths.
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonColumnValue.java:
##########
@@ -195,11 +197,36 @@ public LocalDateTime getDateTime() {
@Override
public LocalDateTime getTimeStampTz() {
- Timestamp ts = record.getTimestamp(idx, dorisType.getPrecision());
+ Timestamp ts = truncateTimestampPrecision(
+ record.getTimestamp(idx, getPaimonTimestampPrecision()));
// Timestamp's local representation is identical to converting its
epoch instant in UTC.
return ts.toLocalDateTime();
}
+ private Timestamp truncateTimestampPrecision(Timestamp timestamp) {
+ if (!(dataType instanceof TimestampType) && !(dataType instanceof
LocalZonedTimestampType)) {
+ return timestamp;
+ }
+ int precision = dorisType.getPrecision();
+ long divisor = 1;
+ for (int i = precision; i < 9; ++i) {
+ divisor *= 10;
+ }
+ LocalDateTime value = timestamp.toLocalDateTime();
+ int truncatedNano = (int) (value.getNano() / divisor * divisor);
+ return Timestamp.fromLocalDateTime(value.withNano(truncatedNano));
+ }
Review Comment:
[P2] Skip timestamp reconstruction when no truncation is needed. The scanner
enables the widened read type for every projected timestamp, including tables
with no precision evolution; both timestamp getters then run this helper for
every non-null row. It converts to LocalDateTime, calls withNano and constructs
a new Timestamp even when the nanoseconds already fit the Doris precision. This
adds allocations and conversion to ordinary JNI timestamp scans. Use the
original value on the aligned fast path, and avoid widening scans that have no
historical narrowing.
##########
be/src/format_v2/expr/cast.cpp:
##########
@@ -24,15 +24,69 @@
#include <ostream>
#include "common/status.h"
+#include "core/assert_cast.h"
#include "core/block/block.h"
#include "core/block/column_with_type_and_name.h"
#include "core/block/columns_with_type_and_name.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_date_or_datetime_v2.h"
+#include "core/data_type/data_type_timestamptz.h"
#include "exprs/function/simple_function_factory.h"
#include "exprs/vexpr_context.h"
#include "exprs/vliteral.h"
namespace doris::format {
+static bool can_truncate_datetimev2_precision(const DataTypePtr& source_type,
+ const DataTypePtr& target_type) {
+ const auto source = remove_nullable(source_type);
+ const auto target = remove_nullable(target_type);
+ const auto primitive = source->get_primitive_type();
+ return primitive == target->get_primitive_type() &&
+ (primitive == TYPE_DATETIMEV2 || primitive == TYPE_TIMESTAMPTZ) &&
+ source->get_scale() > target->get_scale();
+}
+
+static void truncate_datetimev2_precision(ColumnPtr* column, const
DataTypePtr& target_type) {
+ DORIS_CHECK(column != nullptr);
+ auto mutable_column = IColumn::mutate(std::move(*column));
+ IColumn* nested_column = mutable_column.get();
+ if (is_column_nullable(*nested_column)) {
+ nested_column =
static_cast<ColumnNullable*>(nested_column)->get_nested_column_ptr().get();
+ }
+ const auto scale = remove_nullable(target_type)->get_scale();
+ uint32_t divisor = 1;
+ for (uint32_t i = scale; i < 6; ++i) {
+ divisor *= 10;
+ }
+ if (remove_nullable(target_type)->get_primitive_type() == TYPE_DATETIMEV2)
{
+ auto& data = assert_cast<ColumnDateTimeV2&>(*nested_column).get_data();
+ for (auto& value : data) {
+
value.unchecked_set_time_unit<TimeUnit::MICROSECOND>(value.microsecond() /
divisor *
+ divisor);
Review Comment:
[P1] Preserve nanoseconds before native ORC precision truncation. A Paimon
TIMESTAMP(9) or TIMESTAMP_LTZ(9) file evolved to TIMESTAMP(0) can reach this
cast through the native ORC path. ORC's DATETIMEV2/TIMESTAMPTZ decoder first
rounds 00:00:01.999999500 to 00:00:02.000000, so this microsecond-only
truncation returns 00:00:02 rather than Paimon's 00:00:01; an equality
predicate on the expected second misses the row. The new ORC evolution fixture
uses p6 and cannot expose this. Preserve Paimon's truncation before the ORC
nanosecond-to-microsecond rounding, or route p7-p9 narrowed ORC reads through a
path that retains nanoseconds.
##########
fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java:
##########
@@ -1289,6 +1296,81 @@ public Map<String, String> getScanNodeProperties(
return props;
}
+ static boolean containsTimestampType(DataType type) {
+ if (type instanceof org.apache.paimon.types.TimestampType
+ || type instanceof
org.apache.paimon.types.LocalZonedTimestampType) {
+ return true;
+ }
+ if (type instanceof org.apache.paimon.types.RowType) {
+ return ((org.apache.paimon.types.RowType)
type).getFields().stream()
+ .anyMatch(field -> containsTimestampType(field.type()));
+ }
+ if (type instanceof org.apache.paimon.types.ArrayType) {
+ return containsTimestampType(((org.apache.paimon.types.ArrayType)
type).getElementType());
+ }
+ if (type instanceof org.apache.paimon.types.MapType) {
+ org.apache.paimon.types.MapType mapType =
(org.apache.paimon.types.MapType) type;
+ return containsTimestampType(mapType.getKeyType()) ||
containsTimestampType(mapType.getValueType());
+ }
+ return false;
+ }
+
+ private boolean requiresTimestampPrecisionRepair(Table table,
PaimonTableHandle handle) {
+ Table schemaTable = resolveSchemaDictTable(table, handle);
+ if (!(schemaTable instanceof FileStoreTable)) {
+ return false;
+ }
+ FileStoreTable fileStoreTable = (FileStoreTable) schemaTable;
+ RowType currentType = schemaTable.rowType();
+ SchemaManager schemaManager = fileStoreTable.schemaManager();
+ for (Long schemaId : schemaManager.listAllIds()) {
+ TableSchema historicalSchema = schemaManager.schema(schemaId);
+ if (historicalSchema != null
Review Comment:
[P1] Authenticate the new schema-history FileIO reads. Both FE callers reach
this `schema(id)` loop outside `context.executeAuthenticated`, while Paimon's
`listAllIds` and `schema` perform remote FileIO operations. On a fresh filtered
`$ro` scan forced to JNI, the system handle skips authenticated schema
prewarming and the base table can have an unused FileIO. If Doris's plugin
Kerberos principal can read the HDFS table path but Paimon's global login
principal cannot, this loop first opens the HDFS client under the latter
identity and fails with access denied; the earlier authenticated `planSplits`
path could open it under the plugin principal. Wrap the full history lookup at
both call sites in `executeAuthenticated`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
