dejankrak-db commented on code in PR #59640:
URL: https://github.com/apache/spark/pull/59640#discussion_r4359286798
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:
##########
@@ -998,6 +1021,25 @@ trait CheckAnalysis extends LookupCatalog with
QueryErrorsBase with PlanToString
create.tableSchema.foreach(f =>
TypeUtils.failWithIntervalType(f.dataType))
TypeUtils.failUnsupportedDataType(create.tableSchema, SQLConf.get)
SchemaUtils.checkIndeterminateCollationInSchema(create.tableSchema)
+ create match {
+ // The columns of CREATE TABLE AS SELECT come from its query,
not from DDL.
+ case ctas: V2CreateTableAsSelectPlan =>
+ TypeUtils.checkNoUnknownFileTypeWrite(ctas.query.output)
Review Comment:
Fixed in 85cc86b395f. `ResolveSchemaEvolution` now checks the columns it
would add or retype for a FILE without a qualifier before it calls
`alterTable`, so INSERT, MERGE and the DataFrame writers fail with
`FILE_TYPE_UNKNOWN_WRITE` and the table is not changed. The new test checks
both the error and the unchanged schema.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:
##########
@@ -998,6 +1021,25 @@ trait CheckAnalysis extends LookupCatalog with
QueryErrorsBase with PlanToString
create.tableSchema.foreach(f =>
TypeUtils.failWithIntervalType(f.dataType))
TypeUtils.failUnsupportedDataType(create.tableSchema, SQLConf.get)
SchemaUtils.checkIndeterminateCollationInSchema(create.tableSchema)
+ create match {
+ // The columns of CREATE TABLE AS SELECT come from its query,
not from DDL.
+ case ctas: V2CreateTableAsSelectPlan =>
+ TypeUtils.checkNoUnknownFileTypeWrite(ctas.query.output)
+ case _ =>
+ TypeUtils.checkFileTypeHasQualifier(create.tableSchema)
Review Comment:
Fixed in 85cc86b395f. `CreateTableLikeCommand` (V1) and
`CreateTableLikeExec` (V2) check the copied schema with the same helper, so
`CREATE TABLE t LIKE v` fails with `FILE_TYPE_UNKNOWN_WRITE`. Tested with temp
and persistent view sources and V1 and V2 targets, and that no table is created.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2AlterTableCommands.scala:
##########
@@ -146,6 +147,7 @@ case class ReplaceColumns(
columnsToAdd.foreach { c =>
TypeUtils.failWithIntervalType(c.dataType)
TypeUtils.failUnsupportedDataType(c.dataType, conf)
+ TypeUtils.checkFileTypeHasQualifier(c.dataType)
Review Comment:
Fixed in 85cc86b395f: `AlterColumns` requires `FILE EXTERNAL` in the new
type, as `AddColumns` does, so a VOID column can no longer become a bare FILE.
Added the VOID case to the test, which now uses `checkError`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
