jacktengg opened a new issue, #66682: URL: https://github.com/apache/doris/issues/66682
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description # [Feature] Track end-to-end `TIMESTAMP_NS` data type support ## Description This issue tracks the work required to make `TIMESTAMP_NS` a first-class Apache Doris data type across its complete lifecycle: SQL syntax, catalog metadata, planning, execution, storage, indexes, data loading, query results, external formats, upgrade compatibility, testing, and documentation. `TIMESTAMP_NS` has the following semantic contract: - It stores a civil timestamp with fixed nanosecond precision. Users write `TIMESTAMP_NS`; a precision parameter such as `TIMESTAMP_NS(9)` is not supported. - Its in-memory and on-disk scalar value is a signed 64-bit count of nanoseconds from Unix epoch `1970-01-01 00:00:00`. - Its representable range is `[1677-09-21 00:12:43.145224192, 2262-04-11 23:47:16.854775807]`. - It is distinct from `DATETIMEV2`: `DATETIMEV2(p)` continues to support only `p = 0..6` and uses the existing packed civil-time representation. - It is distinct from `TIMESTAMPTZ`: `TIMESTAMP_NS` does not preserve a time-zone identifier or offset in the stored value. - Parsing, rounding, range checking, comparison, hashing, and serialization must have identical FE and BE semantics. The implementation spans the following layers: ```text SQL syntax / clients / load inputs | v FE parser -> catalog type and literal -> Nereids analysis and optimization | v Thrift / protobuf contracts | v BE data type -> column/value -> expressions and operators | v segment/rowset -> key coding -> indexes -> compaction/replication | v result protocols / outfile / external ecosystems ``` The foundation is being implemented by [PR #66333](https://github.com/apache/doris/pull/66333), **Add basic TIMESTAMP_NS type support**. The PR was still open when this tracker was drafted; its checkbox should be marked complete only after it is merged. ## Goals - Provide a stable, documented SQL type with exact nanosecond precision and no implicit loss of sub-microsecond data. - Support the full internal OLAP-table lifecycle in duplicate, aggregate, unique-key/MoW, and primary-key usage. - Define every cast and common-type/coercion relationship rather than inheriting `DATETIMEV2` behavior accidentally. - Match FE constant evaluation and BE vectorized execution for literals, casts, functions, predicates, overflow, and rounding. - Preserve chronological ordering across negative and positive epoch values in keys, indexes, partitioning, sorting, joins, and runtime filters. - Make every import, export, protocol, UDF, connector, and external-format path either fully supported or explicitly rejected with a stable diagnostic. - Define rolling-upgrade, downgrade, backup/restore, replication, and cloud behavior before general availability. - Provide FE unit tests, BE unit tests, regression tests, compatibility tests, and user documentation for every supported capability. ## Scope boundary This tracker owns the Doris type contract and the end-to-end behavior of Doris-native tables and execution. Work in external connectors may be implemented in separate issues/PRs, but this tracker must still record the support decision and link the corresponding work. Performance specialization is not required for initial correctness, but `TIMESTAMP_NS` must not permanently fall back to a path with materially worse complexity than other fixed-width date/time types. Benchmarks and optimizations can be split into follow-up PRs after the semantics are stable. ## Progress Status convention: - `[x]`: merged and covered by tests. - `[ ]` with `🚧`: an implementation PR exists but has not merged. - `[ ]`: not complete, not verified, or still needs a linked PR/test. ### Foundation - [ ] 🚧 [PR #66333](https://github.com/apache/doris/pull/66333): add basic `TIMESTAMP_NS` type support. - FE and BE primitive/storage/protocol type identifiers. - FE catalog and Nereids types plus legacy and Nereids literals. - BE `TimeStampNsValue`, fixed-width column/data type, serde, key coding, segment storage, and indexes. - DDL, literal/default-value analysis, partition and bucket metadata, INSERT and Stream Load foundations. - Comparisons, predicates, hashing, grouping, ordering, joins, runtime filters, and basic aggregate foundations. - MySQL text/binary result handling using a string-compatible representation. - Explicit rejection for unsupported Arrow Flight, ORC OUTFILE, and Java UDF paths. - Foundational FE unit, BE unit, and regression tests. Future implementation PRs should be linked under the corresponding checklist item instead of adding an unstructured list at the end of the issue. ## Type contract and SQL surface - [ ] Accept `TIMESTAMP_NS` in CREATE TABLE, CTAS, CREATE TABLE LIKE, view/MV definitions, temporary tables, and prepared DDL. - [ ] Reject `TIMESTAMP_NS(p)`, including `p = 9`, with a clear and stable error. - [ ] Keep `DATETIME(p)` and `DATETIMEV2(p)` limited to `p = 0..6`; do not alias `p = 7..9` to `TIMESTAMP_NS`. - [ ] Define canonical SQL formatting as exactly nine fractional digits where a typed value is rendered. - [ ] Define parsing grammar for date/time separators, compact forms, ISO `T`, leading/trailing whitespace, year zero, and explicit time-zone offsets. - [ ] Reject malformed inputs such as repeated fractional separators, missing fields, invalid calendar values, and trailing garbage consistently in FE and BE. - [ ] Define the 10th fractional digit as the rounding guard digit, including carry to the next second/day/year and overflow at the maximum value. - [ ] Define strict and non-strict behavior for invalid, rounded-out-of-range, and truncated inputs. - [ ] Test the exact minimum, minimum ±1 ns, epoch ±1 ns, ordinary pre/post-epoch values, maximum ±1 ns, and rounding across both bounds. - [ ] Keep `TIMESTAMP_NS`, `DATETIMEV2`, and `TIMESTAMPTZ` visibly distinct in DESCRIBE, SHOW CREATE TABLE, EXPLAIN, error messages, and client metadata. ## FE type system, catalog, and metadata - [ ] Keep Thrift/protobuf/catalog/storage enum numbers append-only and document their compatibility requirements. - [ ] Support serialization/deserialization in edit logs, snapshots, Gson metadata, query plans, and FE-BE RPC messages. - [ ] Support `ScalarType`, legacy `Type`, Nereids `DataType`, type visitors, type width, precision/scale display, and type equality/matching without treating the type as `DATETIMEV2`. - [ ] Validate the type in column definitions, expression result types, aggregate-state parameters, arrays/maps/structs, generated columns, and function signatures. - [ ] Return correct metadata from DESCRIBE, SHOW CREATE TABLE, SHOW DATA TYPES, information_schema, MySQL field packets, and JDBC `DatabaseMetaData`. - [ ] Preserve the type through aliases, slots, subqueries, CTEs, views, UNION outputs, result sinks, and plan serialization. - [ ] Support FE restart, metadata replay, checkpoint/image creation, and follower synchronization with `TIMESTAMP_NS` objects. - [ ] Add an FE capability/version gate so a table or plan using the new enum cannot be sent to an incompatible BE. - [ ] Make downgrade behavior explicit and fail gracefully when old binaries encounter metadata containing `TIMESTAMP_NS`. ## DDL and table models - [ ] Duplicate-key model: support `TIMESTAMP_NS` as key and value columns. - [ ] Aggregate-key model: support legal key/value/aggregation combinations and reject illegal aggregate functions during analysis. - [ ] Unique-key Merge-on-Write model: support key/value columns, delete bitmap, partial update, flexible partial update, and sequence-column behavior. - [ ] Primary-key/point-query paths: support equality lookup and prepared parameters without converting to `DATETIMEV2` or string internally. - [ ] Support nullable and non-nullable columns, NULL ordering, implicit NULL defaults, and explicit literal defaults. - [ ] Decide and implement `DEFAULT CURRENT_TIMESTAMP` semantics; reject precision arguments because `TIMESTAMP_NS` has fixed precision. - [ ] Decide and implement `ON UPDATE CURRENT_TIMESTAMP` if the feature is supported for Doris date/time columns; otherwise reject it explicitly. - [ ] Support ADD/DROP/RENAME/REORDER/MODIFY COLUMN and nullable/default changes. - [ ] Support light and full schema change while preserving every nanosecond. - [ ] Ensure the schema-change compatibility matrix lists only conversions implemented by the BE. - [ ] Support sequence columns, cluster keys, short keys, sort keys, and distribution columns where the corresponding table model permits them. - [ ] Support generated columns whose input or output is `TIMESTAMP_NS`, subject to the cast/function support in this tracker. - [ ] Support single-table rollup/materialized views, asynchronous multi-table MVs, and MTMVs. - [ ] Support truncate, rename, replace table/partition, recycle bin, table clone, and CREATE TABLE LIKE. ## Literal, cast, coercion, and constant evaluation - [ ] Implement one canonical literal parser/rounder shared semantically by legacy FE, Nereids FE, and BE. - [ ] Preserve the 10th guard digit until target-scale rounding is complete; never retain hidden sub-nanosecond state in equality/hash/order. - [ ] Support literal conversion between legacy and Nereids representations without precision loss. - [ ] Support prepared-statement parameters, thrift/protobuf literals, partition literals, default literals, and statistics literals. - [ ] Define and implement explicit casts from/to string, CHAR, VARCHAR, STRING, JSON/JSONB, and VARIANT. - [ ] Define and implement explicit casts from/to DATE, DATEV2, DATETIME, DATETIMEV2, TIMESTAMPTZ, TIME/TIMEV2, and `TIMESTAMP_NS` itself. - [ ] Define and implement explicit casts from/to BOOLEAN, integer types, LARGEINT, FLOAT/DOUBLE, DECIMALV2, and DECIMALV3, or reject each unsupported pair explicitly. - [ ] Define epoch units for numeric casts; do not guess seconds vs. milliseconds vs. nanoseconds from magnitude. - [ ] Make `TIMESTAMP_NS -> TIMESTAMP_NS` a no-op that preserves nullability and constants. - [ ] Define time-zone behavior for casts from offset-bearing strings and `TIMESTAMPTZ`. - [ ] Define loss of precision when casting to microsecond types, including rounding/truncation direction, carries, and boundary underflow/overflow. - [ ] Cover all supported and unsupported casts with strict/non-strict, nullable, constant, non-constant, and vectorized tests. - [ ] Define common/wider types for comparison, UNION/INTERSECT/EXCEPT, CASE, IF, COALESCE, NULLIF, arrays, maps, and structs. - [ ] Do not make `DateTimeV2Type.getWider...` or a legacy date matching rule implicitly absorb `TIMESTAMP_NS`. - [ ] Keep FE-folded, BE-folded, and unfolded results identical, including nested protobuf folding results. - [ ] Audit comparison simplification and cast elimination so synthesized boundary literals always stay in the signed-Int64 nanosecond range. ## Date/time functions and arithmetic Every existing `DATETIMEV2` function must have an explicit `TIMESTAMP_NS` decision: native support with a dedicated signature, support through a documented cast, or analysis-time rejection. It must not match accidentally through `DATETIMEV2` inheritance. - [ ] Current-time functions: `now`, `current_timestamp`, `localtime`, `localtimestamp`, `curdate`, `current_date`, `curtime`, and UTC variants. - [ ] Extractors: `year`, `quarter`, `month`, `week`, `weekofyear`, `day`, `dayofmonth`, `dayofweek`, `dayofyear`, `hour`, `minute`, `second`, `microsecond`, and a nanosecond extractor/API. - [ ] Formatting/parsing: `date_format`, `from_unixtime`, `str_to_date`, `get_format`, and related MySQL-compatible aliases. - [ ] Conversion: `date`, `time`, `timestamp`, `to_date`, `to_monday`, `makedate`, `convert_tz`, `unix_timestamp`, `to_seconds`, and `from_days`/`to_days`. - [ ] Calendar boundaries: `last_day`, `date_trunc`, `date_floor`, `date_ceil`, `month_floor/ceil`, and year/quarter/week/day/hour/minute/second variants. - [ ] Generic arithmetic: `date_add`, `date_sub`, `adddate`, `subdate`, `timestampadd`, `timestampdiff`, `add_time`, `sub_time`, and `+/- INTERVAL`. - [ ] Unit arithmetic: `years_*`, `quarters_*`, `months_*`, `weeks_*`, `days_*`, `hours_*`, `minutes_*`, `seconds_*`, `milliseconds_*`, `microseconds_*`, and `nanoseconds_*` add/sub/diff families. - [ ] Differences: `datediff`, `date_diff`, and all unit-specific diff functions. - [ ] Preserve nanoseconds for calendar operations that do not intentionally truncate the fractional part. - [ ] Define overflow, end-of-month adjustment, leap-year, year-zero, DST/offset, and range-boundary behavior. - [ ] Add dedicated FE signatures returning `TIMESTAMP_NS` only when the function contract requires it; functions returning DATE/DATEV2/TIME/TIMESTAMPTZ must keep their original return type. - [ ] Test constant, nullable, low-cardinality, and ordinary vector columns for every supported function. ## Expressions and relational operators - [ ] Comparisons: `=`, `!=`/`<>`, `<`, `<=`, `>`, `>=`, and `<=>`, with both operand orders where coercion applies. - [ ] Predicates: `IS NULL`, `IS NOT NULL`, `BETWEEN`, `NOT BETWEEN`, `IN`, and `NOT IN`, including NULL elements and constant/non-constant lists. - [ ] Conditional expressions: IF, CASE WHEN, COALESCE, IFNULL, NULLIF, and nullable wrappers. - [ ] Hash and equality semantics must agree for grouping, joins, dictionaries, bloom filters, and distinct aggregation. - [ ] GROUP BY, ORDER BY, TOP-N, LIMIT/OFFSET, DISTINCT, `COUNT(DISTINCT ...)`, multi-column distinct, and aggregate-state combinators. - [ ] Window functions over `TIMESTAMP_NS` partition/order keys, including lead/lag, first/last value, rank family, and range frames. - [ ] Inner/outer/semi/anti/cross joins, null-safe joins, colocate/bucket-shuffle/broadcast joins, and spill paths. - [ ] Runtime filters: IN, bloom, min/max, min, max, bitmap where applicable, local/global, and remote serialization. - [ ] Exchange, shuffle, hash partitioning, local exchange, result merging, and spill/recovery preserve ordering and values. ## Partitioning, bucketing, scan keys, and pruning - [ ] Range partitioning with exact min/max sentinels, open/closed bounds, NULL ownership, MAXVALUE, and the first/last representable date. - [ ] List partitioning with canonical equality/hash behavior after rounding. - [ ] Auto and dynamic partitioning, including generated partition names that remain stable for existing DATE/DATETIME/DATETIMEV2 tables. - [ ] Partition pruning for literals, casts, functions, folded constants, IN lists, and boundary comparisons. - [ ] Hash bucketing with stable signed-epoch hashing and bucket pruning for equality/IN predicates. - [ ] Distribution pruning after constant folding and prepared-parameter binding. - [ ] Short-key encoding, primary/unique key coding, prefix comparisons, and signed ordering across epoch zero. - [ ] Scan-key range construction/splitting, compound keys, fixed-value ranges, and empty/out-of-range ranges. - [ ] Storage predicate parsing and pushdown for comparison, IN, IS NULL, bloom, bitmap, and runtime-filter predicates. - [ ] `DELETE WHERE` conditions and MoW delete predicates with exact nanosecond comparison. ## BE value, column, serde, and storage - [ ] Keep `TimeStampNsValue` a signed Int64 epoch-nanosecond value with documented invariants and no scale argument. - [ ] Validate `ColumnTimeStampNs`, `Field`, `DataTypeTimeStampNs`, mutable/immutable columns, COW behavior, resize/filter/permute/replicate/scatter/gather, and `ColumnVector::get(Field&)`. - [ ] Text, JSON, JSONB, protobuf, MySQL text/binary, Arrow, ORC, and Parquet serdes each have an implemented or explicitly rejected path. - [ ] Segment encoding round trips for PLAIN, BIT_SHUFFLE, FOR, dictionary-related paths where applicable, compression, page seek, and null maps. - [ ] Rowset V2/V3 and cloud rowset metadata store the new field type without colliding with existing enum values. - [ ] Preserve values through memtable sorting, flush, vertical/horizontal compaction, cumulative/base compaction, cold-data compaction, and segment compaction. - [ ] Preserve values through row store, row-column JSONB storage, partial update, flexible partial update, and rowid-based reads. - [ ] Support clone, migration, repair, replica catch-up, tablet report, checksum, and consistency checks. - [ ] Support cloud tablet/rowset creation and commit without changing non-cloud schema serialization. - [ ] Support backup/restore, snapshot, binlog, recycle/recover, and CCR, or record an explicit unsupported limitation. ## Indexes and storage-level filtering - [ ] Zone map: write/read min/max/null state and prune correctly across negative epoch, epoch, and range boundaries. - [ ] Bloom filter index: write/read/probe exact values and distinguish adjacent nanoseconds. - [ ] Inverted index: write/read equality/range predicates and skip unsupported query modes explicitly. - [ ] Bitmap index if allowed for the type; otherwise reject index creation with a clear message. - [ ] N-gram or other string-only indexes must reject `TIMESTAMP_NS` rather than treating it as text. - [ ] Index compaction, schema change, rebuild, clone, and corrupted-page diagnostics. - [ ] Key coder and short-key index maintain chronological signed ordering, especially before 1970. - [ ] Page-level zone maps, segment-level statistics, min/max runtime filters, and scan pruning return no false negatives. ## Statistics and optimizer - [ ] Column min/max, NDV, null count, sample values, histograms, hot values, and partition statistics serialize valid `TIMESTAMP_NS` literals. - [ ] ANALYZE TABLE, automatic statistics collection, statistics cache/replay, and SHOW COLUMN STATS. - [ ] Filter estimation for equality/range/IN/NULL predicates and boundary values. - [ ] Join estimation and histogram comparison use chronological values rather than lossy floating-point or `DATETIMEV2` packed values. - [ ] Top-N optimization, top-N predicate pushdown, late materialization, two-phase read, and sort elimination. - [ ] Materialized-view rewrite preserves the exact type through expression mapping, compensation predicates, and partition mapping. - [ ] Plan cache, SQL cache, prepared statements, and parameterized partition pruning do not reuse a plan with incompatible literal/type metadata. ## Data loading and write paths - [ ] INSERT VALUES, INSERT SELECT, CTAS, multi-table insert, overwrite, group commit, and transactional insert. - [ ] Stream Load for CSV and JSON in strict/non-strict mode, including column mappings and transformations. - [ ] Broker Load, Routine Load, MySQL Load, S3/HDFS load, and TVF-based file ingestion. - [ ] Partial update, flexible partial update, sequence columns, delete signs, and merge-on-write conflict resolution. - [ ] Prepared/binary parameters and JDBC batch inserts. - [ ] Normal values, malformed dates, invalid calendar values, bounds, overflow, time-zone offsets, rounding carry, and NULLs for every load family. - [ ] Error rows, filtered-row accounting, error URLs, transaction rollback, retry/idempotency, and replay preserve the same validation semantics. - [ ] Cloud and non-cloud ingestion paths share the same type metadata and validation behavior. ## Complex types and aggregate state - [ ] ARRAY element support: construction, literal parsing, cast, comparison where allowed, sort, explode, and serialization. - [ ] MAP key/value support: deterministic hash/equality for keys, lookup, construction, and serialization. - [ ] STRUCT fields: literal/construction, field access, cast, comparison where allowed, and serialization. - [ ] JSON/JSONB and VARIANT conversion: define whether values are formatted strings or typed scalars and preserve nine digits. - [ ] Nested combinations such as `ARRAY<STRUCT<TIMESTAMP_NS>>` across storage, shuffle, load, and output. - [ ] Aggregate-state support for min/max, count/count distinct, bitmap/HLL where legal, collect/list/set, topn, histogram, map aggregation, and sequence functions. - [ ] State serialization/merge/finalize remains stable across BE restart and distributed aggregation. ## Materialized views and generated columns - [ ] Synchronous single-table materialized views with `TIMESTAMP_NS` key/value columns and aggregate states. - [ ] Asynchronous multi-table MV creation, refresh, partition mapping, transparent rewrite, and stale-partition compensation. - [ ] MTMV refresh by generated partition name and auto refresh do not depend on changed formatting of existing `DATETIMEV2` partition keys. - [ ] Generated columns returning or consuming `TIMESTAMP_NS`, including folded expressions and schema changes. - [ ] Refresh/rebuild after base-table insert, update, delete, compaction, and restore. ## Query result protocols and export - [ ] MySQL text protocol returns canonical nine-digit text and stable string-compatible metadata. - [ ] MySQL binary/prepared-statement protocol uses metadata consistent with its length-encoded value representation; clients must not decode it as the legacy temporal 0/4/7/11-byte layout. - [ ] Validate JDBC, ODBC, MySQL CLI, prepared statements, and common language drivers. - [ ] HTTP/REST and Arrow Flight SQL either expose a documented nanosecond timestamp mapping or reject the type before execution. - [ ] SELECT INTO OUTFILE for CSV/text, Parquet, ORC, and Arrow, with a support-or-reject test for each format. - [ ] EXPORT jobs, query result files, outfile compression, and round-trip re-import. - [ ] Result-cache and wire serialization preserve exact values and NULLs. ## External catalogs, connectors, and UDFs - [ ] Define mappings for Hive, Iceberg, Hudi, Paimon, JDBC, Elasticsearch, MaxCompute, and other supported external catalogs. - [ ] Define Parquet `TIMESTAMP_NANOS`, ORC timestamp, and Arrow `Timestamp(NANOSECOND, ...)` semantics, including timezone annotations and out-of-range values. - [ ] Define Doris-to-Spark/Flink connector mappings and round-trip behavior. - [ ] External table scans and writes must not silently map `TIMESTAMP_NS` to microsecond `DATETIMEV2`. - [ ] Java UDF/UDAF, Python UDF, RPC UDF, and table-function argument/return support must be implemented deliberately or rejected during FE analysis. - [ ] Document every unsupported connector/format and provide a stable diagnostic rather than a BE crash or corrupt output. ## Compatibility, upgrade, and operations - [ ] Verify that the new type IDs are appended and do not change the encoding or semantics of DATE, DATEV2, DATETIME, DATETIMEV2, TIMEV2, or TIMESTAMPTZ. - [ ] Audit shared parsing, formatting, partition-key, coercion, and date-function code so enabling `TIMESTAMP_NS` does not alter existing types unintentionally. - [ ] Record intentional correctness changes separately and add compatibility tests for their previous and new behavior. - [ ] Support rolling FE/BE upgrade with an explicit minimum-version/capability check before creating or querying `TIMESTAMP_NS` data. - [ ] Make mixed-version RPC, tablet report, clone, compaction, and query scheduling fail safely. - [ ] Document downgrade restrictions and verify that old binaries do not silently reinterpret the new field type. - [ ] Verify backup/restore and CCR across supported versions and reject unsupported cross-version transfers early. - [ ] Verify FE/BE restart, metadata replay, tablet recovery, failed schema change, transaction recovery, and cloud meta-service restart. - [ ] Add diagnostic coverage for unsupported casts, functions, formats, indexes, UDFs, and external mappings. - [ ] Add profile/EXPLAIN visibility sufficient to diagnose partition/bucket/index/runtime-filter pruning. ## Performance and resource usage - [ ] Benchmark scan/filter, sort/top-N, hash/group/join, runtime filter, load, compaction, and serde against other fixed-width 64-bit types. - [ ] Use SIMD/raw fixed-value paths where safe and verify scalar/SIMD result equivalence. - [ ] Avoid per-row string conversion in typed execution, hashing, storage predicates, or shuffle. - [ ] Measure memory use for columns, hash tables, aggregation states, indexes, and spill. - [ ] Validate index selectivity and partition/bucket pruning with large negative- and positive-epoch datasets. ## Test plan ### FE unit tests - [ ] Parser/type validation, fixed precision, metadata serialization, and legacy/Nereids conversion. - [ ] Normal/invalid/min/max/overflow/epoch/offset literals and 10th-digit rounding carries. - [ ] Complete cast and type-coercion matrices, including unsupported pairs and both operand orders. - [ ] Function signatures/return types, FE folding, BE-fold result decoding, and folded/unfolded consistency. - [ ] Partition/list/bucket keys, min/max sentinels, NULL ownership, pruning, auto partitions, and generated names. - [ ] Schema change, defaults/current timestamp, MV/MTMV, statistics, prepared statements, output/UDF/connector rejection. - [ ] Compatibility tests proving unchanged DATE/DATEV2/DATETIME/DATETIMEV2/TIMEV2/TIMESTAMPTZ behavior. ### BE unit tests - [ ] `TimeStampNsValue`, column operations, data type, Field conversion, text/binary/JSONB/protobuf serde, and invalid input. - [ ] Every segment encoding, key coder, short key, signed ordering, row-store representation, and page seeking. - [ ] Zone map, bloom, inverted, and supported bitmap indexes: write, read, equality/range filter, adjacent-nanosecond values, NULLs, and bounds. - [ ] Storage predicates, scan keys, delete conditions, partitioner, hash, runtime filters, and SIMD/raw-value paths. - [ ] Casts and every supported function with constant/nullable/vector input, carries, precision loss, and overflow. - [ ] Aggregates, aggregate-state serialization, sort/top-N, join keys, shuffle, exchange, and spill. - [ ] Compaction/schema-change/partial-update/clone/backup primitives where unit-testable. ### Regression tests - [ ] Separate purpose-based suites under `datatype_p0/timestamp_ns` for literals, storage models, primary key, partition/bucket pruning, indexes, load, casts, functions, expressions, delete/update, group/order/top-N, joins, runtime filters, aggregates, complex types, generated columns, MV/MTMV, output, and compatibility. - [ ] Every suite includes ordinary values, invalid values, min/max, overflow, epoch ±1 ns, values with/without offsets, NULLs, rounding, carry overflow, precision loss, and underflow where applicable. - [ ] Verify EXPLAIN-visible partition/bucket/index/runtime-filter pruning, not only query results. - [ ] Run in both non-cloud and cloud modes for storage/metadata/write/recovery paths. - [ ] Add restart/replay, compaction, schema change, backup/restore, and failure-injection coverage where supported by the regression framework. - [ ] Exercise MySQL text and binary prepared-statement clients plus supported outfile/external-format round trips. ### Quality gates - [ ] New FE and BE code meets the project incremental coverage target. - [ ] FE and BE builds, style checks, FE unit tests, BE unit tests, and affected regression suites pass. - [ ] No sanitizer errors, DCHECK failures, crashes, silent truncation, or inconsistent folded/unfolded results. - [ ] Result files are generated by the regression framework and boundary expectations are reviewed. ## Correctness and compatibility checklist - [ ] FE, BE, load, storage, and client paths use the same parsing and range rules. - [ ] The 10th digit is handled consistently and a rounding carry never wraps signed Int64. - [ ] Equality, order, hash, key coding, zone maps, bloom filters, partitioning, bucketing, joins, and runtime filters agree bit-for-bit. - [ ] Values before 1970 sort chronologically and are not treated as unsigned. - [ ] NULL semantics are correct in predicates, first range partitions, joins, grouping, and indexes. - [ ] Explicit offsets are handled deterministically and session timezone does not silently change persisted values. - [ ] No supported path silently converts nanoseconds to microseconds. - [ ] Existing date/time type syntax, range, formatting, partition naming, casts, functions, and output remain compatible. - [ ] Unsupported paths fail during analysis or return a clear status; they never corrupt data or crash FE/BE. - [ ] Mixed-version, cloud, backup/restore, and replication behavior is documented and tested. ## Documentation - [ ] SQL reference: syntax, fixed precision, range, canonical format, timezone semantics, rounding, NULL/default behavior, and examples. - [ ] Cast/coercion matrix and complete supported-function list. - [ ] DDL/model/index/partition/import/export compatibility matrix. - [ ] Client, protocol, connector, external-format, and UDF support matrix. - [ ] Upgrade, downgrade, backup/restore, CCR, and mixed-version restrictions. - [ ] Known limitations and links to every unfinished subtask. ## Exit criteria - The semantic contract is documented and implemented consistently in FE, BE, storage, load, and result protocols. - All internal OLAP table models and supported DDL/DML operations work end to end with exact nanosecond values. - Cast/coercion behavior for every Doris type pair is implemented or explicitly rejected and fully tested. - Every `DATETIMEV2` date/time function has an explicit, tested `TIMESTAMP_NS` support decision. - Partitioning, bucketing, scan-key/predicate pushdown, indexes, joins, runtime filters, aggregation, sort/top-N, complex types, MVs, and generated columns are verified. - All import/export/client/connector/UDF paths are supported or rejected with documented diagnostics. - Rolling upgrade, downgrade, restart/replay, cloud, backup/restore, replication, compaction, and schema-change behavior is verified. - FE unit, BE unit, regression, compatibility, and failure-recovery tests pass and coverage gates are met. - User-facing and operator documentation is published, and every remaining limitation is linked to a follow-up issue. ## Related work - Foundation PR: [#66333 Add basic TIMESTAMP_NS type support](https://github.com/apache/doris/pull/66333) - Reference tracking-issue structure: [#65418 Track incremental computation with row binlog, Table Stream, and MTMV](https://github.com/apache/doris/issues/65418) ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
