Anurag Mantripragada created SPARK-59944:
--------------------------------------------
Summary: Normalize references of generic and upper-case transforms
in CREATE/REPLACE TABLE
Key: SPARK-59944
URL: https://issues.apache.org/jira/browse/SPARK-59944
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Anurag Mantripragada
In CREATE/REPLACE TABLE, PreprocessTableCreation normalizes only
RewritableTransform references to the schema's spelling. A generic
ApplyTransform is not normalized, so in a case-insensitive session ORDERED BY
truncate(4, ID) on a column `id` fails with WRITE_ORDERING_WITH_UNKNOWN_COLUMN,
and PARTITIONED BY behaves the same way.
visitTransform also dispatches on the raw name text, so DAYS(ts) or BUCKET(4,
id) becomes an ApplyTransform instead of a DaysTransform / BucketTransform,
hits the same limitation, and hands the catalog a generic transform named
BUCKET.
Proposed: make ApplyTransform a RewritableTransform (RewritableTransform is
sealed and its only user is PreprocessTableCreation), and consider dispatching
built-in transform names case-insensitively in visitTransform. The second part
also changes PARTITIONED BY, e.g. an upper-case BUCKET would become bucketing
for v1 tables. Found in review of SPARK-34586
(https://github.com/apache/spark/pull/58153).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
