[
https://issues.apache.org/jira/browse/SPARK-56969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-56969:
--------------------------------
Assignee: Max Gekk
> Add SQLConf preview flag for nanosecond-capable timestamp types
> ---------------------------------------------------------------
>
> Key: SPARK-56969
> URL: https://issues.apache.org/jira/browse/SPARK-56969
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
>
> h2. Summary
> Introduce a preview SQL configuration that gates use of
> {{TimestampNTZNanosType(p)}} and {{TimestampLTZNanosType(p)}} ({{p}} in [7,
> 9]) added in [SPARK-56876|https://issues.apache.org/jira/browse/SPARK-56876].
> The flag is *off by default* in production (default {{true}} under
> {{Utils.isTesting}}, following {{spark.sql.timeType.enabled}} /
> {{TIME_TYPE_ENABLED}}).
> h2. Background
> Logical types and parser support exist
> ([SPARK-56876|https://issues.apache.org/jira/browse/SPARK-56876],
> [SPARK-56965|https://issues.apache.org/jira/browse/SPARK-56965]). Physical
> row storage lands in
> [SPARK-56981|https://issues.apache.org/jira/browse/SPARK-56981]. Without
> gating, partially implemented nanos types could surface in schemas, casts, or
> connectors before the SPIP preview milestone is ready.
> {{TimeType}} precedent: {{spark.sql.timeType.enabled}} with
> {{TypeUtils.failUnsupportedDataType}} and {{UNSUPPORTED_TIME_TYPE}}.
> h2. Scope
> h3. 1. SQLConf implementation
> * New entry (name TBD; suggest {{spark.sql.timestampNanos.preview.enabled}}):
> ** {{.internal()}} — preview / unstable
> ** {{.booleanConf}} with {{.createWithDefault(Utils.isTesting)}} (mirror
> {{TIME_TYPE_ENABLED}})
> ** {{.version("4.2.0")}}
> ** {{.doc(...)}} — see *Documentation* below
> * Accessor on {{SQLConf}} (e.g. {{isTimestampNanosPreviewEnabled}})
> h3. 2. Analysis gating
> * Extend {{TypeUtils.failUnsupportedDataType}} to reject schemas/plans that
> *recursively* contain {{TimestampNTZNanosType}} or {{TimestampLTZNanosType}}
> when the flag is disabled
> * Error class {{UNSUPPORTED_TIMESTAMP_NANOS_TYPE}} (or single message
> covering both NTZ/LTZ) in {{error-conditions.json}} and
> {{QueryCompilationErrors}}, analogous to {{UNSUPPORTED_TIME_TYPE}}
> * Error message must name the conf key and how to enable preview (supports
> 12b discoverability)
> h3. 3. Binding policy
> * Register the conf in {{configs-without-binding-policy-exceptions}} if
> required (see {{spark.sql.timeType.enabled}} precedent)
> h3. 4. Documentation (sub-task 12b — same PR)
> *Audience:* operators and advanced users.
> * *{{SQLConf}} {{.doc()}} text* must cover:
> ** What enabling allows (nanos-capable {{TIMESTAMP_NTZ(p)}} /
> {{TIMESTAMP_LTZ(p)}}, {{p}} in 7–9, in schemas and analyzed plans)
> ** Preview / unstable status under
> [SPARK-56822|https://issues.apache.org/jira/browse/SPARK-56822]
> ** Default value and testing default via {{Utils.isTesting}}
> ** That unparameterized {{TIMESTAMP}} / {{TIMESTAMP_NTZ}} / {{TIMESTAMP_LTZ}}
> remain microsecond types
> ** What may still fail when enabled (casts, Parquet read, literals, etc.
> until their JIRAs land)
> * *{{docs/configuration.md}}* — add a row/section for the new key (follow
> existing {{spark.sql.*}} entries):
> ** Key name (exact string as implemented)
> ** Default, version (4.2.0), scope, preview disclaimer
> ** Cross-link to SPIP JIRA or future sql-ref when available
> * *Short enablement note* (minimal user discovery before full sql-ref pass in
> 12c):
> ** One paragraph in {{docs/sql-ref-datatypes.md}} *or* a note in
> {{docs/configuration.md}} intro pointing to the conf
> ** Example: {{SET spark.sql.timestampNanos.preview.enabled=true;}}
> ** Do *not* document capabilities that are not yet merged; list only what
> works at ship time
> h2. Tests
> * Unit: {{SQLConf}} entry exists; default matches {{Utils.isTesting}}
> pattern; readable via {{SQLConf.get}}
> * Unit: {{TypeUtils.failUnsupportedDataType}} throws when flag off and schema
> contains {{TimestampNTZNanosType(9)}} / {{TimestampLTZNanosType(9)}}; passes
> when flag on
> * Unit: error message references the conf key (documentation/discoverability)
> * Regression: existing {{DataTypeSuite}} / types without nanos types
> unaffected
> * Optional: {{AnalysisSuite}} or {{DDLParsingSuite}} — {{CREATE TABLE t (c
> TIMESTAMP_NTZ(9))}} fails with new error when preview off
> ([SPARK-56965|https://issues.apache.org/jira/browse/SPARK-56965] parser
> merged)
> h2. Acceptance criteria
> * With preview *disabled* (default outside tests), any analyzed schema or
> plan that recursively contains {{TimestampNTZNanosType}} or
> {{TimestampLTZNanosType}} fails with a clear error naming
> {{spark.sql.timestampNanos.preview.enabled}} (exact key as implemented)
> * With preview *enabled* (or in tests), {{failUnsupportedDataType}} does not
> block those types (downstream JIRAs may still fail for other reasons)
> * No change to {{TimestampType}}, {{TimestampNTZType}}, or zero-arg
> {{TIMESTAMP}} / {{TIMESTAMP_NTZ}} DDL semantics
> * {{SQLConf}} {{.doc()}} text is sufficient for generated configuration
> reference
> * {{docs/configuration.md}} documents the key with correct default, version,
> and enablement instructions
> * At least one public doc location (configuration.md and/or
> sql-ref-datatypes) lets users find how to turn on preview without reading
> source
> * Conf discoverable via {{SET spark.sql.timestampNanos.preview.enabled}}
> h2. Dependencies
> * Logical types:
> [SPARK-56876|https://issues.apache.org/jira/browse/SPARK-56876]
> * Parser: [SPARK-56965|https://issues.apache.org/jira/browse/SPARK-56965]
> (merged)
> * Recommended before first user-visible Parquet read or cast PRs
> * Physical rows:
> [SPARK-56981|https://issues.apache.org/jira/browse/SPARK-56981] (in progress
> — gating can land independently)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
