dejankrak-db opened a new pull request, #59640: URL: https://github.com/apache/spark/pull/59640
### What changes were proposed in this pull request? Part 2 of the FileType SPIP ([SPARK-59132](https://issues.apache.org/jira/browse/SPARK-59132)), following the Catalyst type added in #59305. The SPIP requires the `EXTERNAL` keyword when a FILE column is declared in a table, and gives FileType no defined ordering. This PR enforces both at analysis time. Table columns: - A FILE in a table column definition, including one nested in a struct, array or map, must be `FILE EXTERNAL`: `CREATE TABLE`, `REPLACE TABLE`, `ALTER TABLE ADD COLUMNS` and `ALTER TABLE REPLACE COLUMNS` fail with `FILE_TYPE_MISSING_QUALIFIER`. The check is in `CheckAnalysis` for V2 tables, in `PreprocessTableCreation` for V1 tables, and next to the existing `failUnsupportedDataType` call in `AddColumns`/`ReplaceColumns`. - `CREATE TABLE AS SELECT` cannot store a FILE without a qualifier, as one produced by `CAST(... AS FILE)` (`FILE_TYPE_UNKNOWN_WRITE`). - A FILE column cannot be a partition or clustering column of a V2 table (`INVALID_PARTITION_COLUMN_DATA_TYPE`). V1 file source tables already reject it through `PartitioningUtils.canPartitionOn`. Routines: - Functions declare a plain `FILE`, as the SPIP specifies. `CREATE FUNCTION` rejects `FILE EXTERNAL` in its parameter or return types (`FILE_TYPE_IN_ROUTINE_SIGNATURE_UNSUPPORTED_QUALIFIER`), and a FILE in a map key of its return type (`FILE_TYPE_IN_ROUTINE_SIGNATURE_UNSUPPORTED_MAP_KEY`). - The same applies to the return type of UDFs, when they are registered through `spark.udf.register`/`spark.udtf.register` and, for UDFs that are not registered, when `CheckAnalysis` sees them in a query. A non-scalar Python UDF is skipped, as its data type is the schema of the rows it returns. No ordering, following the existing checks for VARIANT: - Map key: `TypeUtils.checkForMapKeyType` (`DATATYPE_MISMATCH.INVALID_MAP_KEY_TYPE`). - Grouping: `ExprUtils.checkValidGroupingExprs` (`GROUP_EXPRESSION_TYPE_IS_NOT_ORDERABLE`). - `INTERSECT`, `EXCEPT` and `DISTINCT`: `CheckAnalysis` and `OperatorWithUncomparableTypeValidator` of the single-pass resolver (`UNSUPPORTED_FEATURE.SET_OPERATION_ON_FILE_TYPE`). - `DISTRIBUTE BY`/`repartition`: `CheckAnalysis` and `RepartitionByExpressionResolver` (`UNSUPPORTED_FEATURE.PARTITION_BY_FILE`). - Hash functions: `HashExpression.checkInputDataTypes` (`DATATYPE_MISMATCH.HASH_FILE_TYPE`), which also guards hashes the planner inserts. - Join keys, comparisons and sorting need no change: FileType is not orderable, so they already fail with `DATATYPE_MISMATCH.INVALID_ORDERING_TYPE`. The `FileType` Scaladoc is updated to describe these rules as current rather than planned. ### Why are the changes needed? The SPIP defines a FILE column of a table as an external file reference, and a bare `FILE` (the unknown mode) as the type of an expression or a UDF result, so a table should never store a bare `FILE`. FileType has no defined ordering, so operations that compare, group or hash values must fail clearly rather than operate on the underlying struct. ### Does this PR introduce _any_ user-facing change? No relative to released versions. The FILE type is only recognized when `spark.sql.fileType.enabled` is true, which is the default only in tests. Relative to master, the statements and operations listed above now fail with the errors listed above when the type is enabled. ### How was this patch tested? Added `FileTypeRestrictionsSuite`, covering each rule for V1 and V2 tables where both apply, nested FILE types, and the cases that are still accepted (`FILE EXTERNAL` columns, a FILE map value, a function declaring a plain `FILE`). Also ran `SparkThrowableSuite`, `AnalysisErrorSuite`, `ComplexTypeSuite`, `HashExpressionsSuite`, `TypeUtilsSuite`, `UDFSuite`, `SQLFunctionSuite`, `VariantSuite`, `DataSourceV2SQLSuiteV1Filter` and `CreateTableAsSelectSuite`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
