szehon-ho commented on code in PR #54126:
URL: https://github.com/apache/spark/pull/54126#discussion_r3327808578
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Column.java:
##########
@@ -67,13 +67,39 @@ static Column create(
null, null, metadataInJSON);
}
+ /**
+ * Creates a column with a generation expression in SQL string form.
+ *
+ * @since 4.1.0
+ * @deprecated Use
+ * {@link #create(String, DataType, boolean, String, GenerationExpression,
String)} instead.
+ */
+ @Deprecated
static Column create(
String name,
DataType dataType,
boolean nullable,
String comment,
String generationExpression,
String metadataInJSON) {
+ GenerationExpression genExpr = generationExpression != null
+ ? new GenerationExpression(generationExpression) : null;
+ return new ColumnImpl(name, dataType, nullable, comment, null,
+ genExpr, null, metadataInJSON);
+ }
+
+ /**
+ * Creates a column with a generation expression object.
+ *
+ * @since 4.1.0
Review Comment:
Updated to `@since 4.3.0`.
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Column.java:
##########
@@ -123,7 +149,17 @@ static Column create(
* expression compatibility and reject writes as necessary.
*/
@Nullable
- String generationExpression();
+ default String generationExpression() {
+ return columnGenerationExpression() != null ?
columnGenerationExpression().getSql() : null;
+ }
+
+ /**
+ * Returns the generation expression of this table column as an {@link
GenerationExpression}
+ *
+ * @since 4.1.0
Review Comment:
Updated to `@since 4.3.0`.
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Column.java:
##########
@@ -67,13 +67,39 @@ static Column create(
null, null, metadataInJSON);
}
+ /**
+ * Creates a column with a generation expression in SQL string form.
+ *
+ * @since 4.1.0
+ * @deprecated Use
+ * {@link #create(String, DataType, boolean, String, GenerationExpression,
String)} instead.
+ */
+ @Deprecated
static Column create(
String name,
DataType dataType,
boolean nullable,
String comment,
String generationExpression,
String metadataInJSON) {
+ GenerationExpression genExpr = generationExpression != null
+ ? new GenerationExpression(generationExpression) : null;
+ return new ColumnImpl(name, dataType, nullable, comment, null,
+ genExpr, null, metadataInJSON);
+ }
+
+ /**
+ * Creates a column with a generation expression object.
+ *
+ * @since 4.1.0
Review Comment:
Set to `@since 4.3.0`.
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/Column.java:
##########
@@ -123,7 +149,17 @@ static Column create(
* expression compatibility and reject writes as necessary.
*/
@Nullable
- String generationExpression();
+ default String generationExpression() {
+ return columnGenerationExpression() != null ?
columnGenerationExpression().getSql() : null;
+ }
+
+ /**
+ * Returns the generation expression of this table column as an {@link
GenerationExpression}
+ *
+ * @since 4.1.0
+ */
+ @Nullable
+ GenerationExpression columnGenerationExpression();
Review Comment:
Good catch. Changed `columnGenerationExpression()` to a `default` method
returning `null`, so existing `Column` implementations keep compiling. The
standard `ColumnImpl` overrides it.
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/GenerationExpression.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.catalog;
+
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+
+import org.apache.spark.SparkIllegalArgumentException;
+import org.apache.spark.annotation.Evolving;
+import org.apache.spark.sql.connector.expressions.Expression;
+
+/**
+ * A class that represents generation expressions for computed/generated
columns.
+ * <p>
+ * Connectors can define generation expressions using either a SQL string
(Spark SQL dialect) or an
+ * {@link Expression expression} if the generation expression can be expressed
as a supported
+ * connector expression. If both the SQL string and the expression are
provided, Spark first
+ * attempts to convert the given expression to its internal representation. If
the expression
+ * cannot be converted, and a SQL string is provided, Spark will fall back to
parsing the SQL
+ * string.
+ *
+ * @since 4.1.0
+ */
+@Evolving
+public class GenerationExpression {
Review Comment:
Done — extracted a package-private `ColumnExpressionBase`; both
`DefaultValue` and `GenerationExpression` now extend it (sharing the
`sql`/`expression` fields and accessors).
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:
##########
@@ -457,7 +457,27 @@ trait CheckAnalysis extends LookupCatalog with
QueryErrorsBase with PlanToString
WindowResolution.validateResolvedWindowExpression(w)
case s: SubqueryExpression =>
- checkSubqueryExpression(operator, s)
+ // Check if this subquery is inside a generated column expression
Review Comment:
The new branch only intercepts the case where the subquery belongs to a
generated column on a `V2CreateTablePlan`, so we can surface a
generated-column-specific message. Every other operator falls through to the
existing `checkSubqueryExpression(operator, s)`, so general subquery validation
is unchanged. `GeneratedColumnExpression.validate` also rejects any subquery
(`PLAN_EXPRESSION`) as a backstop.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:
##########
@@ -719,6 +739,35 @@ trait CheckAnalysis extends LookupCatalog with
QueryErrorsBase with PlanToString
TypeUtils.failUnsupportedDataType(create.tableSchema, SQLConf.get)
SchemaUtils.checkIndeterminateCollationInSchema(create.tableSchema)
+ // Validate generated column expressions
+ create.columns.foreach { col =>
+ col.generationExpression.foreach { genExpr =>
+ def unsupportedExpressionError(reason: String):
AnalysisException = {
+ new AnalysisException(
+ errorClass = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
+ messageParameters = Map(
+ "fieldName" -> col.name,
+ "expressionStr" -> genExpr.originalSQL,
+ "reason" -> reason
+ )
+ )
+ }
+ // Check for user-defined functions - only built-in functions
are allowed
+ // in generated columns. Traverse the entire expression tree.
+ genExpr.child.foreach {
+ case u: UnresolvedFunction =>
Review Comment:
You're right that unknown functions already fail with `UNRESOLVED_ROUTINE`,
so I removed the redundant check. The remaining check only rejects
`UserDefinedExpression` (Scala/Python/Java/Hive UDFs and v2 catalog functions),
which resolve successfully but aren't built-in; I added a comment explaining
this.
I also looked into surfacing the generated-column-specific message for these
(like we do for subqueries), but it's not feasible here: both errors are thrown
eagerly during resolution, before `CheckAnalysis` runs (unknown functions in
`FunctionResolution.resolveFunction`, persistent SQL UDFs in the
`ResolveSQLFunctions` rule). The subquery case is interceptable only because a
subquery resolves into a valid `SubqueryExpression` and reaches `CheckAnalysis`.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ColumnDefinition.scala:
##########
@@ -273,3 +291,90 @@ case class DefaultValueExpression(
throw QueryCompilationErrors.defaultValueNotConstantError(statement,
colName, originalSQL)
}
}
+
+/**
+ * A wrapper expression to hold the generation expression and its original SQL
text.
+ * The child expression is resolved by the normal analyzer rules through the
expression tree.
+ */
+case class GeneratedColumnExpression(
Review Comment:
Done — moved `GeneratedColumnExpression` to its own file
`GeneratedColumnExpression.scala`.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ColumnDefinition.scala:
##########
@@ -273,3 +291,90 @@ case class DefaultValueExpression(
throw QueryCompilationErrors.defaultValueNotConstantError(statement,
colName, originalSQL)
}
}
+
+/**
+ * A wrapper expression to hold the generation expression and its original SQL
text.
+ * The child expression is resolved by the normal analyzer rules through the
expression tree.
+ */
+case class GeneratedColumnExpression(
+ child: Expression,
+ originalSQL: String)
+ extends UnaryExpression with Unevaluable {
+
+ override def dataType: DataType = child.dataType
+
+ override def stringArgs: Iterator[Any] = Iterator(child, originalSQL)
+
+ override protected def withNewChildInternal(newChild: Expression):
Expression =
+ copy(child = newChild)
+
+ /**
+ * Validate the generation expression and throw an AnalysisException if
invalid.
+ * Validations include:
Review Comment:
Default-value validation mainly checks that the expression is a foldable
constant coercible to the column type (evaluated once at definition time).
Generated-column validation is for an expression evaluated per-row against
other columns, so it additionally enforces: no self-reference, no reference to
other generated columns, determinism, safe up-cast to the column type, no
subqueries, and no non-UTF8 binary collation. See
`GeneratedColumnExpression.validate`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
