[GitHub] [spark] AmplabJenkins commented on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table
AmplabJenkins commented on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-499373828 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11479/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table
AmplabJenkins removed a comment on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-499373824 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table
AmplabJenkins removed a comment on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-499373828 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11479/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table
AmplabJenkins commented on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-499373824 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table
SparkQA commented on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-499372422 **[Test build #106230 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106230/testReport)** for PR 24806 at commit [`85fa370`](https://github.com/apache/spark/commit/85fa370e8e7c4779e23d0695c4b814c4a033fcf4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table
AmplabJenkins removed a comment on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-499371917 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11478/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table
AmplabJenkins commented on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-499371917 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11478/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table
AmplabJenkins commented on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-499371911 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table
AmplabJenkins removed a comment on issue #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#issuecomment-499371911 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#discussion_r290828849 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala ## @@ -434,6 +434,31 @@ class SessionCatalog( externalCatalog.getTable(db, table) } + /** + * Retrieve all metadata of existing permanent tables/views. If no database is specified, + * assume the table/view is in the current database. + * Only the tables/views belong to one same database that can be retrieved are returned. Review comment: `one same database` -> `the same database`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL]
Make HiveMetastoreClient.getTableObjectsByName available in
ExternalCatalog/SessionCatalog API
URL: https://github.com/apache/spark/pull/24774#discussion_r291034950
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##
@@ -434,6 +434,34 @@ class SessionCatalog(
externalCatalog.getTable(db, table)
}
+ /**
+ * Retrieve all metadata of existing permanent tables/views. If no database
is specified,
+ * assume the table/view is in the current database.
+ * Only the tables/views belong to one same database that can be retrieved
are returned.
+ * For example, if none of the requested tables could be retrieved, an empty
list is returned.
+ * There is no guarantee of ordering of the returned tables.
+ */
+ @throws[NoSuchDatabaseException]
+ def getTablesByName(names: Seq[TableIdentifier]): Seq[CatalogTable] = {
+if (names.nonEmpty) {
+ val dbs = names.map(_.database.getOrElse(getCurrentDatabase))
+ if (dbs.distinct.size != 1) {
+val tables = names.map(name => formatTableName(name.table))
+dbs.zip(tables).map { case (d, t) => QualifiedTableName(d, t)}
+throw new AnalysisException(
+ s"Only the tables/views belong to one same database can be
retrieved. Querying " +
+ s"tables/views are ${dbs.zip(tables).map { case (d, t) =>
QualifiedTableName(d, t)}}"
Review comment:
This should be replaced by a variable name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL]
Make HiveMetastoreClient.getTableObjectsByName available in
ExternalCatalog/SessionCatalog API
URL: https://github.com/apache/spark/pull/24774#discussion_r291036823
##
File path:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkMetadataOperationSuite.scala
##
@@ -156,6 +156,7 @@ class SparkMetadataOperationSuite extends
HiveThriftJdbcTest {
withJdbcStatement("table1", "table2") { statement =>
Seq(
+"DROP VIEW IF EXISTS view1",
Review comment:
Weird. If the view is not dropped, why the tables do not exist in your
environment?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL]
Make HiveMetastoreClient.getTableObjectsByName available in
ExternalCatalog/SessionCatalog API
URL: https://github.com/apache/spark/pull/24774#discussion_r291034852
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
##
@@ -434,6 +434,34 @@ class SessionCatalog(
externalCatalog.getTable(db, table)
}
+ /**
+ * Retrieve all metadata of existing permanent tables/views. If no database
is specified,
+ * assume the table/view is in the current database.
+ * Only the tables/views belong to one same database that can be retrieved
are returned.
+ * For example, if none of the requested tables could be retrieved, an empty
list is returned.
+ * There is no guarantee of ordering of the returned tables.
+ */
+ @throws[NoSuchDatabaseException]
+ def getTablesByName(names: Seq[TableIdentifier]): Seq[CatalogTable] = {
+if (names.nonEmpty) {
+ val dbs = names.map(_.database.getOrElse(getCurrentDatabase))
+ if (dbs.distinct.size != 1) {
+val tables = names.map(name => formatTableName(name.table))
+dbs.zip(tables).map { case (d, t) => QualifiedTableName(d, t)}
Review comment:
?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL]
Make HiveMetastoreClient.getTableObjectsByName available in
ExternalCatalog/SessionCatalog API
URL: https://github.com/apache/spark/pull/24774#discussion_r291037716
##
File path:
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala
##
@@ -237,6 +237,25 @@ class VersionsSuite extends SparkFunSuite with Logging {
assert(client.getTableOption("default", "src").isDefined)
}
+test(s"$version: getTablesByName") {
+ assert(client.getTablesByName("default", Seq("src")).head
+== client.getTableOption("default", "src").get)
+}
+
+test(s"$version: getTablesByName when multiple tables") {
+ assert(client.getTablesByName("default", Seq("src", "temporary"))
Review comment:
Also try the invalid names here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL]
Make HiveMetastoreClient.getTableObjectsByName available in
ExternalCatalog/SessionCatalog API
URL: https://github.com/apache/spark/pull/24774#discussion_r291040233
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##
@@ -1091,6 +1114,38 @@ private[hive] object HiveClientImpl {
stats = readHiveStats(properties))
}
+ /**
+ * This is the same process copied from the method `getTable()`
+ * of [[org.apache.hadoop.hive.ql.metadata.Hive]] to do some extra fixes for
non-views.
+ * Methods of extracting multiple [[HiveTable]] like `getRawTablesByName()`
+ * should invoke this before return.
+ */
+ def extraFixesForNonView(tTable: MetaStoreApiTable): MetaStoreApiTable = {
+// For non-views, we need to do some extra fixes
+if (!(HiveTableType.VIRTUAL_VIEW.toString == tTable.getTableType)) {
+ // Fix the non-printable chars
+ val parameters: JMap[String, String] = tTable.getSd.getParameters
+ val sf: String = parameters.get(serdeConstants.SERIALIZATION_FORMAT)
Review comment:
https://github.com/apache/hive/blame/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1342
Do we need to check null here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL]
Make HiveMetastoreClient.getTableObjectsByName available in
ExternalCatalog/SessionCatalog API
URL: https://github.com/apache/spark/pull/24774#discussion_r291037843
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
##
@@ -1091,6 +1114,38 @@ private[hive] object HiveClientImpl {
stats = readHiveStats(properties))
}
+ /**
+ * This is the same process copied from the method `getTable()`
+ * of [[org.apache.hadoop.hive.ql.metadata.Hive]] to do some extra fixes for
non-views.
+ * Methods of extracting multiple [[HiveTable]] like `getRawTablesByName()`
+ * should invoke this before return.
+ */
+ def extraFixesForNonView(tTable: MetaStoreApiTable): MetaStoreApiTable = {
+// For non-views, we need to do some extra fixes
+if (!(HiveTableType.VIRTUAL_VIEW.toString == tTable.getTableType)) {
+ // Fix the non-printable chars
+ val parameters: JMap[String, String] = tTable.getSd.getParameters
+ val sf: String = parameters.get(serdeConstants.SERIALIZATION_FORMAT)
+ if (sf != null) {
+val b: Array[Char] = sf.toCharArray
+if ((b.length == 1) && (b(0) < 10)) { // ^A, ^B, ^C, ^D, \t
+ parameters.put(serdeConstants.SERIALIZATION_FORMAT,
Integer.toString(b(0)))
+}
+ }
+ // Use LazySimpleSerDe for MetadataTypedColumnsetSerDe.
+ // NOTE: LazySimpleSerDe does not support tables with a single column of
col
+ // of type "array". This happens when the table is created using
+ // an earlier version of Hive.
+ if (classOf[MetadataTypedColumnsetSerDe].getName
+== tTable.getSd.getSerdeInfo.getSerializationLib
Review comment:
4-space indent
== needs to be moved to the line 1139
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL]
Make HiveMetastoreClient.getTableObjectsByName available in
ExternalCatalog/SessionCatalog API
URL: https://github.com/apache/spark/pull/24774#discussion_r291037419
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala
##
@@ -84,6 +84,8 @@ private[hive] trait HiveClient {
/** Returns the metadata for the specified table or None if it doesn't
exist. */
def getTableOption(dbName: String, tableName: String): Option[CatalogTable]
+ def getTablesByName(dbName: String, tableNames: Seq[String]):
Seq[CatalogTable]
Review comment:
Add a comment to describe the function?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
gatorsmile commented on a change in pull request #24774: [SPARK-27899][SQL]
Make HiveMetastoreClient.getTableObjectsByName available in
ExternalCatalog/SessionCatalog API
URL: https://github.com/apache/spark/pull/24774#discussion_r291035734
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogSuite.scala
##
@@ -277,6 +277,20 @@ abstract class ExternalCatalogSuite extends SparkFunSuite
with BeforeAndAfterEac
}
}
+ test("get tables by name") {
+assert(newBasicCatalog().getTablesByName("db2", Seq("tbl1", "tbl2"))
+ .map(_.identifier.table) == Seq("tbl1", "tbl2"))
+ }
+
+ test("get tables by name when some tables do not exists") {
+assert(newBasicCatalog().getTablesByName("db2", Seq("tbl1", "tblnotexist"))
Review comment:
See `validNameFormat`. Add a test case when the seq of table names contains
the invalid name.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yeshengm commented on a change in pull request #24802: [SPARK-27949][SQL]Support ANSI SQL grammar `substring(string_expression from n1 [for n2])`
yeshengm commented on a change in pull request #24802:
[SPARK-27949][SQL]Support ANSI SQL grammar `substring(string_expression from n1
[for n2])`
URL: https://github.com/apache/spark/pull/24802#discussion_r291039872
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##
@@ -1333,6 +1333,20 @@ class AstBuilder(conf: SQLConf) extends
SqlBaseBaseVisitor[AnyRef] with Logging
}
}
+ /**
+ * Create a Substring/Substr expression.
+ */
+ override def visitSubstring(ctx: SubstringContext): Expression =
withOrigin(ctx) {
+val field = ctx.field
+val pos = ctx.pos
+val len = ctx.len
+if (len != null) {
+ Substring(expression(field), expression(pos), expression(len))
+} else {
+ new Substring(expression(field), expression(pos))
Review comment:
Whoops, I haven't thought it can be that tricky. Then I don't have a strong
opinion about it. cc @dongjoon-hyun
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on a change in pull request #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table URL: https://github.com/apache/spark/pull/24806#discussion_r291039822 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala ## @@ -374,15 +374,15 @@ class HiveQuerySuite extends HiveComparisonTest with SQLTestUtils with BeforeAnd sql(s"CREATE TABLE test_partition (a STRING) PARTITIONED BY (b BIGINT, c STRING)") sql(s"CREATE TABLE ptest (a STRING, b BIGINT, c STRING)") - val analyzedPlan = sql( + val optimizedPlan = sql( Review comment: The `Cast` operator will be eliminated by optimization rule `ConstantFolding`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] lipzhu commented on a change in pull request #24802: [SPARK-27949][SQL]Support ANSI SQL grammar `substring(string_expression from n1 [for n2])`
lipzhu commented on a change in pull request #24802: [SPARK-27949][SQL]Support
ANSI SQL grammar `substring(string_expression from n1 [for n2])`
URL: https://github.com/apache/spark/pull/24802#discussion_r291038686
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##
@@ -1333,6 +1333,20 @@ class AstBuilder(conf: SQLConf) extends
SqlBaseBaseVisitor[AnyRef] with Logging
}
}
+ /**
+ * Create a Substring/Substr expression.
+ */
+ override def visitSubstring(ctx: SubstringContext): Expression =
withOrigin(ctx) {
+val field = ctx.field
+val pos = ctx.pos
+val len = ctx.len
+if (len != null) {
+ Substring(expression(field), expression(pos), expression(len))
+} else {
+ new Substring(expression(field), expression(pos))
Review comment:
Add a default value for the case class and delete the verbose constructor
may affect `Right` expression. Do we really to do this change in this PR?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-499368616 **[Test build #106229 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106229/testReport)** for PR 24741 at commit [`3ae1816`](https://github.com/apache/spark/commit/3ae181642a6f6c35853488aa2d061607ecf1f712). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-499368163 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11477/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-499368156 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-499368156 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-499368163 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11477/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yeshengm commented on a change in pull request #24802: [SPARK-27949][SQL]Support ANSI SQL grammar `substring(string_expression from n1 [for n2])`
yeshengm commented on a change in pull request #24802:
[SPARK-27949][SQL]Support ANSI SQL grammar `substring(string_expression from n1
[for n2])`
URL: https://github.com/apache/spark/pull/24802#discussion_r291035696
##
File path:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
##
@@ -657,6 +657,8 @@ primaryExpression
| base=primaryExpression '.' fieldName=identifier
#dereference
| '(' expression ')'
#parenthesizedExpression
| EXTRACT '(' field=identifier FROM source=valueExpression ')'
#extract
+| (SUBSTR | SUBSTRING) '(' field=valueExpression (FROM | ',')
pos=valueExpression
Review comment:
Maybe just `str` or something like that?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] lipzhu commented on a change in pull request #24802: [SPARK-27949][SQL]Support ANSI SQL grammar `substring(string_expression from n1 [for n2])`
lipzhu commented on a change in pull request #24802: [SPARK-27949][SQL]Support
ANSI SQL grammar `substring(string_expression from n1 [for n2])`
URL: https://github.com/apache/spark/pull/24802#discussion_r291034877
##
File path:
sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
##
@@ -657,6 +657,8 @@ primaryExpression
| base=primaryExpression '.' fieldName=identifier
#dereference
| '(' expression ')'
#parenthesizedExpression
| EXTRACT '(' field=identifier FROM source=valueExpression ')'
#extract
+| (SUBSTR | SUBSTRING) '(' field=valueExpression (FROM | ',')
pos=valueExpression
Review comment:
How about strExpression?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
AmplabJenkins removed a comment on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499352114 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11476/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
SparkQA commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499352464 **[Test build #106228 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106228/testReport)** for PR 24774 at commit [`873644d`](https://github.com/apache/spark/commit/873644d7a60543a6ef8ad07750219d43d7259f46). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
AmplabJenkins removed a comment on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499352109 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
AmplabJenkins commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499352114 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11476/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
AmplabJenkins commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499352109 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin edited a comment on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
LantaoJin edited a comment on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499350656 After rebased from master, the commit history contains many unrelated commits. I have to create a new local branch and cherry pick the commits from old local branch and then force push it to the remote old branch to clean them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
LantaoJin commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499351527 Looks good now This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
LantaoJin commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499350656 After rebased from master, the commit history contains many unrelated commits. I have to force push from a new branch to clean them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions
SparkQA commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions URL: https://github.com/apache/spark/pull/24749#issuecomment-499349724 **[Test build #106227 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106227/testReport)** for PR 24749 at commit [`e24a12e`](https://github.com/apache/spark/commit/e24a12e6eedea1a6446ba71d6e02aa576b71ca33). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions
AmplabJenkins commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions URL: https://github.com/apache/spark/pull/24749#issuecomment-499349401 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11475/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions
AmplabJenkins commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions URL: https://github.com/apache/spark/pull/24749#issuecomment-499349397 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions
AmplabJenkins removed a comment on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions URL: https://github.com/apache/spark/pull/24749#issuecomment-499349397 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions
AmplabJenkins removed a comment on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions URL: https://github.com/apache/spark/pull/24749#issuecomment-499349401 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11475/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yeshengm commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions
yeshengm commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions URL: https://github.com/apache/spark/pull/24749#issuecomment-499349161 @dongjoon-hyun This is due to a recent PR on qualified column name. Fixed by removing error capture in that syntax. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24809: [WIP][SPARK-21136][SQL] Fix confusing parsing error due to problematic antlr grammer for Hive-style single-from statements
SparkQA commented on issue #24809: [WIP][SPARK-21136][SQL] Fix confusing parsing error due to problematic antlr grammer for Hive-style single-from statements URL: https://github.com/apache/spark/pull/24809#issuecomment-499348435 **[Test build #106226 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106226/testReport)** for PR 24809 at commit [`a5311a7`](https://github.com/apache/spark/commit/a5311a7da14032e98eef6935ec18fa852576dd83). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
AmplabJenkins removed a comment on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499348082 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11474/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
AmplabJenkins removed a comment on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499348076 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24809: [WIP][SPARK-21136][SQL] Fix confusing parsing error due to problematic antlr grammer for Hive-style single-from statements
AmplabJenkins removed a comment on issue #24809: [WIP][SPARK-21136][SQL] Fix confusing parsing error due to problematic antlr grammer for Hive-style single-from statements URL: https://github.com/apache/spark/pull/24809#issuecomment-499348090 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11473/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24809: [WIP][SPARK-21136][SQL] Fix confusing parsing error due to problematic antlr grammer for Hive-style single-from statements
AmplabJenkins removed a comment on issue #24809: [WIP][SPARK-21136][SQL] Fix confusing parsing error due to problematic antlr grammer for Hive-style single-from statements URL: https://github.com/apache/spark/pull/24809#issuecomment-499348084 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing commented on issue #24490: [SPARK-27300][GRAPH] Add Spark Graph modules and dependencies
zsxwing commented on issue #24490: [SPARK-27300][GRAPH] Add Spark Graph modules and dependencies URL: https://github.com/apache/spark/pull/24490#issuecomment-499348099 @s1ck You need to add new modules to https://github.com/apache/spark/blob/76fbe121ab2e829f75389c6fd466afcf2ec3b9d1/project/SparkBuild.scala#L335 This will disable mimi check. Mima doesn't work for new module since it cannot fetch old versions of them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
AmplabJenkins commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499348082 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11474/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24809: [WIP][SPARK-21136][SQL] Fix confusing parsing error due to problematic antlr grammer for Hive-style single-from statements
AmplabJenkins commented on issue #24809: [WIP][SPARK-21136][SQL] Fix confusing parsing error due to problematic antlr grammer for Hive-style single-from statements URL: https://github.com/apache/spark/pull/24809#issuecomment-499348090 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11473/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
AmplabJenkins commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499348076 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zsxwing edited a comment on issue #24490: [SPARK-27300][GRAPH] Add Spark Graph modules and dependencies
zsxwing edited a comment on issue #24490: [SPARK-27300][GRAPH] Add Spark Graph modules and dependencies URL: https://github.com/apache/spark/pull/24490#issuecomment-499348099 @s1ck You need to add new modules to https://github.com/apache/spark/blob/76fbe121ab2e829f75389c6fd466afcf2ec3b9d1/project/SparkBuild.scala#L335 This will disable mimi check. Mima doesn't work for new modules since it cannot fetch old versions of them. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24809: [WIP][SPARK-21136][SQL] Fix confusing parsing error due to problematic antlr grammer for Hive-style single-from statements
AmplabJenkins commented on issue #24809: [WIP][SPARK-21136][SQL] Fix confusing parsing error due to problematic antlr grammer for Hive-style single-from statements URL: https://github.com/apache/spark/pull/24809#issuecomment-499348084 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API
SparkQA commented on issue #24774: [SPARK-27899][SQL] Make HiveMetastoreClient.getTableObjectsByName available in ExternalCatalog/SessionCatalog API URL: https://github.com/apache/spark/pull/24774#issuecomment-499347132 **[Test build #106225 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106225/testReport)** for PR 24774 at commit [`fb7760c`](https://github.com/apache/spark/commit/fb7760c927f53cdbcf81ee6ed224fbc652fdd939). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cryeo commented on a change in pull request #24335: [SPARK-27425][SQL] Add count_if functions
cryeo commented on a change in pull request #24335: [SPARK-27425][SQL] Add
count_if functions
URL: https://github.com/apache/spark/pull/24335#discussion_r291017988
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/CountIf.scala
##
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.aggregate
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.types._
+
+@ExpressionDescription(
+ usage = """
+_FUNC_(expr) - Returns the number of rows that the supplied expression is
non-null and true.
+ """,
+ examples = """
+Examples:
+ > SELECT _FUNC_(col % 2 = 0) FROM VALUES (NULL), (0), (1), (2), (3) AS
tab(col);
+ 2
+ > SELECT _FUNC_(col IS NULL) FROM VALUES (NULL), (0), (1), (2), (3) AS
tab(col);
+ 1
+ """,
+ since = "3.0.0")
+case class CountIf(predicate: Expression) extends UnevaluableAggregate {
+ def child: Expression = Count(
Review comment:
This approach is also used in `UnevaluableAggs` e.g. Every(EveryAgg),
Any(AnyAgg), Some(SomeAgg).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
beliefer removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-498626541 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
beliefer removed a comment on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-498690270 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint
beliefer commented on issue #24792: [SPARK-27943][SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-499343005 @wangyum Help me review this PR, thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-499342189 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-499342195 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11472/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-499342195 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11472/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-499342189 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-499341363 **[Test build #106224 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106224/testReport)** for PR 24741 at commit [`2d05b49`](https://github.com/apache/spark/commit/2d05b4997f9a56d1237d60831588d1fb0c835773). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jzhuge commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs
jzhuge commented on issue #24741: [SPARK-27322][SQL] DataSourceV2: Select from multiple catalogs URL: https://github.com/apache/spark/pull/24741#issuecomment-499341311 Rebased to resolve conflicts with the base branch. And minor fixes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24768: [SPARK-27919][SQL] Add v2 session catalog
dongjoon-hyun commented on issue #24768: [SPARK-27919][SQL] Add v2 session catalog URL: https://github.com/apache/spark/pull/24768#issuecomment-499337974 Thank you, @rdblue . - #24813 is merged. - #24812 seems to need more reviews. I left a few comments there. For this PR, let's come back after #24812 is resolved. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL]
Add extractors for v2 catalog transforms.
URL: https://github.com/apache/spark/pull/24812#discussion_r291011446
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalog/v2/expressions/expressions.scala
##
@@ -111,32 +122,104 @@ private[sql] final case class ApplyTransform(
override def toString: String = describe
}
+/**
+ * Convenience extractor for any Literal.
+ */
+private object Lit {
+ def unapply[T](literal: Literal[T]): Some[(T, DataType)] = {
+Some((literal.value, literal.dataType))
+ }
+}
+
+/**
+ * Convenience extractor for any NamedReference.
+ */
+private object Ref {
+ def unapply(named: NamedReference): Some[Seq[String]] = {
+Some(named.fieldNames)
+ }
+}
+
+/**
+ * Convenience extractor for any Transform.
+ */
+private object NamedTransform {
+ def unapply(transform: Transform): Some[(String, Seq[Expression])] = {
+Some((transform.name, transform.arguments))
+ }
+}
+
private[sql] final case class IdentityTransform(
ref: NamedReference) extends SingleColumnTransform(ref) {
override val name: String = "identity"
override def describe: String = ref.describe
}
+private[sql] object IdentityTransform {
+ def unapply(transform: Transform): Option[FieldReference] = transform match {
+case NamedTransform("identity", Seq(Ref(parts))) =>
+ Some(FieldReference(parts))
+case _ =>
+ None
+ }
+}
Review comment:
If we have `NamedTransform.unapply` and `Ref.unapply`, the others are not
required much like the following.
```scala
- case IdentityTransform(FieldReference(seq)) =>
+ case NamedTransform("identity", Seq(Ref(seq))) =>
```
```scala
- case YearsTransform(FieldReference(seq)) =>
+ case NamedTransform("years", Seq(Ref(seq))) =>
```
Do we need all of them?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL]
Add extractors for v2 catalog transforms.
URL: https://github.com/apache/spark/pull/24812#discussion_r291009964
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalog/v2/expressions/TransformExtractorSuite.scala
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalog.v2.expressions
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst
+import org.apache.spark.sql.types.DataType
+
+class TransformExtractorSuite extends SparkFunSuite {
+ /**
+ * Creates a Literal using an anonymous class.
+ */
+ private def lit[T](literal: T): Literal[T] = new Literal[T] {
+override def value: T = literal
+override def dataType: DataType =
catalyst.expressions.Literal(literal).dataType
+override def describe: String = literal.toString
+ }
+
+ /**
+ * Creates a NamedReference using an anonymous class.
+ */
+ private def ref(names: String*): NamedReference = new NamedReference {
+override def fieldNames: Array[String] = names.toArray
+override def describe: String = names.mkString(".")
+ }
+
+ /**
+ * Creates a Transform using an anonymous class.
+ */
+ def transform(func: String, ref: NamedReference): Transform = new Transform {
+override def name: String = func
+override def references: Array[NamedReference] = Array(ref)
+override def arguments: Array[Expression] = Array(ref)
+override def describe: String = ref.describe
+ }
+
+ /**
+ * Creates a bucket Transform using an anonymous class.
+ */
+ def bucket(numBuckets: Int, ref: NamedReference): Transform = new Transform {
+override def name: String = "bucket"
+override def references: Array[NamedReference] = Array(ref)
+override def arguments: Array[Expression] = Array(lit(numBuckets), ref)
+override def describe: String = ref.describe
+ }
Review comment:
We can also remove this `def bucket` function because this is used only
once. Given the existing functions, this doesn't give much benefits.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL]
Add extractors for v2 catalog transforms.
URL: https://github.com/apache/spark/pull/24812#discussion_r291008966
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalog/v2/expressions/expressions.scala
##
@@ -111,32 +122,104 @@ private[sql] final case class ApplyTransform(
override def toString: String = describe
}
+/**
+ * Convenience extractor for any Literal.
+ */
+private object Lit {
+ def unapply[T](literal: Literal[T]): Some[(T, DataType)] = {
+Some((literal.value, literal.dataType))
+ }
+}
+
+/**
+ * Convenience extractor for any NamedReference.
+ */
+private object Ref {
+ def unapply(named: NamedReference): Some[Seq[String]] = {
Review comment:
`Some` -> `Option`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24777: [SPARK-16692][ML][Python] add MultilabelClassificationEvaluator
SparkQA removed a comment on issue #24777: [SPARK-16692][ML][Python] add MultilabelClassificationEvaluator URL: https://github.com/apache/spark/pull/24777#issuecomment-499322548 **[Test build #106223 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106223/testReport)** for PR 24777 at commit [`7bad5c0`](https://github.com/apache/spark/commit/7bad5c0653e7246e8c68532fcbf24c0a466c6101). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL]
Add extractors for v2 catalog transforms.
URL: https://github.com/apache/spark/pull/24812#discussion_r291008989
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalog/v2/expressions/expressions.scala
##
@@ -111,32 +122,104 @@ private[sql] final case class ApplyTransform(
override def toString: String = describe
}
+/**
+ * Convenience extractor for any Literal.
+ */
+private object Lit {
+ def unapply[T](literal: Literal[T]): Some[(T, DataType)] = {
+Some((literal.value, literal.dataType))
+ }
+}
+
+/**
+ * Convenience extractor for any NamedReference.
+ */
+private object Ref {
+ def unapply(named: NamedReference): Some[Seq[String]] = {
+Some(named.fieldNames)
+ }
+}
+
+/**
+ * Convenience extractor for any Transform.
+ */
+private object NamedTransform {
+ def unapply(transform: Transform): Some[(String, Seq[Expression])] = {
Review comment:
`Some` -> `Option`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24777: [SPARK-16692][ML][Python] add MultilabelClassificationEvaluator
AmplabJenkins removed a comment on issue #24777: [SPARK-16692][ML][Python] add MultilabelClassificationEvaluator URL: https://github.com/apache/spark/pull/24777#issuecomment-499333721 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106223/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24777: [SPARK-16692][ML][Python] add MultilabelClassificationEvaluator
AmplabJenkins removed a comment on issue #24777: [SPARK-16692][ML][Python] add MultilabelClassificationEvaluator URL: https://github.com/apache/spark/pull/24777#issuecomment-499333717 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL]
Add extractors for v2 catalog transforms.
URL: https://github.com/apache/spark/pull/24812#discussion_r291008744
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalog/v2/expressions/expressions.scala
##
@@ -111,32 +122,104 @@ private[sql] final case class ApplyTransform(
override def toString: String = describe
}
+/**
+ * Convenience extractor for any Literal.
+ */
+private object Lit {
+ def unapply[T](literal: Literal[T]): Some[(T, DataType)] = {
Review comment:
`Some` -> `Option`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24777: [SPARK-16692][ML][Python] add MultilabelClassificationEvaluator
AmplabJenkins commented on issue #24777: [SPARK-16692][ML][Python] add MultilabelClassificationEvaluator URL: https://github.com/apache/spark/pull/24777#issuecomment-499333721 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106223/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24777: [SPARK-16692][ML][Python] add MultilabelClassificationEvaluator
AmplabJenkins commented on issue #24777: [SPARK-16692][ML][Python] add MultilabelClassificationEvaluator URL: https://github.com/apache/spark/pull/24777#issuecomment-499333717 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24777: [SPARK-16692][ML][Python] add MultilabelClassificationEvaluator
SparkQA commented on issue #24777: [SPARK-16692][ML][Python] add MultilabelClassificationEvaluator URL: https://github.com/apache/spark/pull/24777#issuecomment-499333561 **[Test build #106223 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106223/testReport)** for PR 24777 at commit [`7bad5c0`](https://github.com/apache/spark/commit/7bad5c0653e7246e8c68532fcbf24c0a466c6101). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class MultilabelClassificationEvaluator (override val uid: String)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL]
Add extractors for v2 catalog transforms.
URL: https://github.com/apache/spark/pull/24812#discussion_r291007975
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalog/v2/expressions/TransformExtractorSuite.scala
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalog.v2.expressions
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst
+import org.apache.spark.sql.types.DataType
+
+class TransformExtractorSuite extends SparkFunSuite {
+ /**
+ * Creates a Literal using an anonymous class.
+ */
+ private def lit[T](literal: T): Literal[T] = new Literal[T] {
+override def value: T = literal
+override def dataType: DataType =
catalyst.expressions.Literal(literal).dataType
+override def describe: String = literal.toString
+ }
+
+ /**
+ * Creates a NamedReference using an anonymous class.
+ */
+ private def ref(names: String*): NamedReference = new NamedReference {
+override def fieldNames: Array[String] = names.toArray
+override def describe: String = names.mkString(".")
+ }
Review comment:
Of course, you can omit the prefix `LogicalExpressions.` with the proper
`import`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL]
Add extractors for v2 catalog transforms.
URL: https://github.com/apache/spark/pull/24812#discussion_r291007769
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalog/v2/expressions/TransformExtractorSuite.scala
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalog.v2.expressions
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst
+import org.apache.spark.sql.types.DataType
+
+class TransformExtractorSuite extends SparkFunSuite {
+ /**
+ * Creates a Literal using an anonymous class.
+ */
+ private def lit[T](literal: T): Literal[T] = new Literal[T] {
+override def value: T = literal
+override def dataType: DataType =
catalyst.expressions.Literal(literal).dataType
+override def describe: String = literal.toString
+ }
+
+ /**
+ * Creates a NamedReference using an anonymous class.
+ */
+ private def ref(names: String*): NamedReference = new NamedReference {
+override def fieldNames: Array[String] = names.toArray
+override def describe: String = names.mkString(".")
+ }
Review comment:
Ditto. Please reuse `LogicalExpressions.reference`.
```scala
-transform("identity", ref("a", "b")) match {
+transform("identity", LogicalExpressions.reference("a.b")) match {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL]
Add extractors for v2 catalog transforms.
URL: https://github.com/apache/spark/pull/24812#discussion_r291007769
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalog/v2/expressions/TransformExtractorSuite.scala
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalog.v2.expressions
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst
+import org.apache.spark.sql.types.DataType
+
+class TransformExtractorSuite extends SparkFunSuite {
+ /**
+ * Creates a Literal using an anonymous class.
+ */
+ private def lit[T](literal: T): Literal[T] = new Literal[T] {
+override def value: T = literal
+override def dataType: DataType =
catalyst.expressions.Literal(literal).dataType
+override def describe: String = literal.toString
+ }
+
+ /**
+ * Creates a NamedReference using an anonymous class.
+ */
+ private def ref(names: String*): NamedReference = new NamedReference {
+override def fieldNames: Array[String] = names.toArray
+override def describe: String = names.mkString(".")
+ }
Review comment:
Ditto. Please reuse `LogicalExpressions.reference`.
```
-transform("identity", ref("a", "b")) match {
+transform("identity", LogicalExpressions.reference("a.b")) match {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL]
Add extractors for v2 catalog transforms.
URL: https://github.com/apache/spark/pull/24812#discussion_r291007263
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalog/v2/expressions/TransformExtractorSuite.scala
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalog.v2.expressions
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst
+import org.apache.spark.sql.types.DataType
+
+class TransformExtractorSuite extends SparkFunSuite {
+ /**
+ * Creates a Literal using an anonymous class.
+ */
+ private def lit[T](literal: T): Literal[T] = new Literal[T] {
+override def value: T = literal
+override def dataType: DataType =
catalyst.expressions.Literal(literal).dataType
+override def describe: String = literal.toString
+ }
Review comment:
It seems that we have this already. Can we reuse
`LogicalExpressions.literal` and remove this?
```scala
-override def arguments: Array[Expression] = Array(lit(numBuckets), ref)
+override def arguments: Array[Expression] =
Array(LogicalExpressions.literal(numBuckets), ref)
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on a change in pull request #24806: [SPARK-27856][SQL]
Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r291005332
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
##
@@ -356,8 +358,28 @@ case class PreprocessTableInsertion(conf: SQLConf)
extends Rule[LogicalPlan] {
s"including ${staticPartCols.size} partition column(s) having
constant value(s).")
}
-val newQuery = DDLPreprocessingUtils.castAndRenameQueryOutput(
- insert.query, expectedColumns, conf)
+val newQuery = if
(conf.getConf(SQLConf.LEGACY_INSERT_TABLE_TYPE_COERCION)) {
+ DDLPreprocessingUtils.castAndRenameQueryOutput(insert.query,
expectedColumns, conf)
+} else {
+ val errors = new mutable.ArrayBuffer[String]()
Review comment:
Here is for the V1 path. `InsertIntoTable` won't be matched in
`ResolveOutputRelation`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL]
Add extractors for v2 catalog transforms.
URL: https://github.com/apache/spark/pull/24812#discussion_r291004072
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalog/v2/expressions/TransformExtractorSuite.scala
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalog.v2.expressions
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst
+import org.apache.spark.sql.types.DataType
+
+class TransformExtractorSuite extends SparkFunSuite {
+ /**
+ * Creates a Literal using an anonymous class.
+ */
+ private def lit[T](literal: T): Literal[T] = new Literal[T] {
+override def value: T = literal
+override def dataType: DataType =
catalyst.expressions.Literal(literal).dataType
+override def describe: String = literal.toString
+ }
+
+ /**
+ * Creates a NamedReference using an anonymous class.
+ */
+ private def ref(names: String*): NamedReference = new NamedReference {
+override def fieldNames: Array[String] = names.toArray
+override def describe: String = names.mkString(".")
+ }
+
+ /**
+ * Creates a Transform using an anonymous class.
+ */
+ def transform(func: String, ref: NamedReference): Transform = new Transform {
+override def name: String = func
+override def references: Array[NamedReference] = Array(ref)
+override def arguments: Array[Expression] = Array(ref)
+override def describe: String = ref.describe
+ }
+
+ /**
+ * Creates a bucket Transform using an anonymous class.
+ */
+ def bucket(numBuckets: Int, ref: NamedReference): Transform = new Transform {
Review comment:
`private def`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
dongjoon-hyun commented on a change in pull request #24812: [SPARK-27965][SQL]
Add extractors for v2 catalog transforms.
URL: https://github.com/apache/spark/pull/24812#discussion_r291004055
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalog/v2/expressions/TransformExtractorSuite.scala
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalog.v2.expressions
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst
+import org.apache.spark.sql.types.DataType
+
+class TransformExtractorSuite extends SparkFunSuite {
+ /**
+ * Creates a Literal using an anonymous class.
+ */
+ private def lit[T](literal: T): Literal[T] = new Literal[T] {
+override def value: T = literal
+override def dataType: DataType =
catalyst.expressions.Literal(literal).dataType
+override def describe: String = literal.toString
+ }
+
+ /**
+ * Creates a NamedReference using an anonymous class.
+ */
+ private def ref(names: String*): NamedReference = new NamedReference {
+override def fieldNames: Array[String] = names.toArray
+override def describe: String = names.mkString(".")
+ }
+
+ /**
+ * Creates a Transform using an anonymous class.
+ */
+ def transform(func: String, ref: NamedReference): Transform = new Transform {
Review comment:
`private def`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util
dongjoon-hyun closed pull request #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util URL: https://github.com/apache/spark/pull/24813 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
AmplabJenkins removed a comment on issue #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms. URL: https://github.com/apache/spark/pull/24812#issuecomment-499327591 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
AmplabJenkins removed a comment on issue #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms. URL: https://github.com/apache/spark/pull/24812#issuecomment-499327592 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106219/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
AmplabJenkins commented on issue #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms. URL: https://github.com/apache/spark/pull/24812#issuecomment-499327592 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106219/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
AmplabJenkins commented on issue #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms. URL: https://github.com/apache/spark/pull/24812#issuecomment-499327591 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
SparkQA removed a comment on issue #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms. URL: https://github.com/apache/spark/pull/24812#issuecomment-499295702 **[Test build #106219 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106219/testReport)** for PR 24812 at commit [`1821443`](https://github.com/apache/spark/commit/1821443196e91f3f1c97362ffa96e6e9519b5754). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms.
SparkQA commented on issue #24812: [SPARK-27965][SQL] Add extractors for v2 catalog transforms. URL: https://github.com/apache/spark/pull/24812#issuecomment-499327285 **[Test build #106219 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106219/testReport)** for PR 24812 at commit [`1821443`](https://github.com/apache/spark/commit/1821443196e91f3f1c97362ffa96e6e9519b5754). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util
AmplabJenkins removed a comment on issue #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util URL: https://github.com/apache/spark/pull/24813#issuecomment-499326841 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106222/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util
AmplabJenkins removed a comment on issue #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util URL: https://github.com/apache/spark/pull/24813#issuecomment-499326834 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util
AmplabJenkins commented on issue #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util URL: https://github.com/apache/spark/pull/24813#issuecomment-499326834 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util
AmplabJenkins commented on issue #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util URL: https://github.com/apache/spark/pull/24813#issuecomment-499326841 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106222/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sandeep-katta commented on a change in pull request #24766: [SPARK-27917][SQL] canonical form of CaseWhen object is incorrect
sandeep-katta commented on a change in pull request #24766: [SPARK-27917][SQL]
canonical form of CaseWhen object is incorrect
URL: https://github.com/apache/spark/pull/24766#discussion_r291002570
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##
@@ -234,6 +234,8 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]]
extends Product {
}
def mapChild(child: Any): Any = child match {
case arg: TreeNode[_] if containsChild(arg) => mapTreeNode(arg)
+ // CaseWhen Case or any tuple type
+ case tuple: (_, _) => (mapChild(tuple._1), mapChild(tuple._2))
Review comment:
I double checked , only the `tuple` and `Option` were missing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util
SparkQA removed a comment on issue #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util URL: https://github.com/apache/spark/pull/24813#issuecomment-499303032 **[Test build #106222 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106222/testReport)** for PR 24813 at commit [`ca848ba`](https://github.com/apache/spark/commit/ca848baa83c7d14c0abbcfad3af22e6a7b05). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util
SparkQA commented on issue #24813: [SPARK-27964][SQL] Move v2 catalog update methods to CatalogV2Util URL: https://github.com/apache/spark/pull/24813#issuecomment-499326456 **[Test build #106222 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106222/testReport)** for PR 24813 at commit [`ca848ba`](https://github.com/apache/spark/commit/ca848baa83c7d14c0abbcfad3af22e6a7b05). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #24806: [SPARK-27856][SQL] Only allow type upcasting when inserting table
gengliangwang commented on a change in pull request #24806: [SPARK-27856][SQL]
Only allow type upcasting when inserting table
URL: https://github.com/apache/spark/pull/24806#discussion_r291000592
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveOutputRelation.scala
##
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute,
NamedExpression, UpCast}
+import org.apache.spark.sql.catalyst.plans.logical.{AppendData, LogicalPlan,
OverwriteByExpression, OverwritePartitionsDynamic, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.DataType
+
+/**
+ * Resolves columns of an output table from the data in a logical plan. This
rule will:
+ *
+ * - Reorder columns when the write is by name
+ * - Insert safe casts when data types do not match
+ * - Insert aliases when column names do not match
+ * - Detect plans that are not compatible with the output table and throw
AnalysisException
+ */
+object ResolveOutputRelation extends Rule[LogicalPlan] {
Review comment:
> move it outside of the Analyzer class, so that we can call its methods.
As per @cloud-fan commented in
https://github.com/apache/spark/pull/24721/files#r287800626
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24767: [SPARK-27918][SQL] Port boolean.sql
AmplabJenkins removed a comment on issue #24767: [SPARK-27918][SQL] Port boolean.sql URL: https://github.com/apache/spark/pull/24767#issuecomment-499323976 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24767: [SPARK-27918][SQL] Port boolean.sql
AmplabJenkins removed a comment on issue #24767: [SPARK-27918][SQL] Port boolean.sql URL: https://github.com/apache/spark/pull/24767#issuecomment-499323980 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106218/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
