[GitHub] [spark] SparkQA commented on issue #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
SparkQA commented on issue #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol URL: https://github.com/apache/spark/pull/25547#issuecomment-523776203 **[Test build #109558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109558/testReport)** for PR 25547 at commit [`58e3b4e`](https://github.com/apache/spark/commit/58e3b4e574467d106a938a6ff5e768149020aa31). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25546: [SPARK-28780][ML][2.3] deprecate LinearSVCModel.setWeightCol
SparkQA commented on issue #25546: [SPARK-28780][ML][2.3] deprecate LinearSVCModel.setWeightCol URL: https://github.com/apache/spark/pull/25546#issuecomment-523776193 **[Test build #109559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109559/testReport)** for PR 25546 at commit [`fe95c7e`](https://github.com/apache/spark/commit/fe95c7e911918f1dfa10fe487a7821662b821237). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
zhengruifeng commented on issue #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol URL: https://github.com/apache/spark/pull/25547#issuecomment-523776134 updated. Thanks for reviewing and explanation! @dongjoon-hyun This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25546: [SPARK-28780][ML][2.3] deprecate LinearSVCModel.setWeightCol
AmplabJenkins removed a comment on issue #25546: [SPARK-28780][ML][2.3] deprecate LinearSVCModel.setWeightCol URL: https://github.com/apache/spark/pull/25546#issuecomment-523775618 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25546: [SPARK-28780][ML][2.3] deprecate LinearSVCModel.setWeightCol
AmplabJenkins removed a comment on issue #25546: [SPARK-28780][ML][2.3] deprecate LinearSVCModel.setWeightCol URL: https://github.com/apache/spark/pull/25546#issuecomment-523775627 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14616/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
AmplabJenkins commented on issue #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol URL: https://github.com/apache/spark/pull/25547#issuecomment-523775569 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14615/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
AmplabJenkins commented on issue #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol URL: https://github.com/apache/spark/pull/25547#issuecomment-523775562 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25546: [SPARK-28780][ML][2.3] deprecate LinearSVCModel.setWeightCol
AmplabJenkins commented on issue #25546: [SPARK-28780][ML][2.3] deprecate LinearSVCModel.setWeightCol URL: https://github.com/apache/spark/pull/25546#issuecomment-523775618 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25546: [SPARK-28780][ML][2.3] deprecate LinearSVCModel.setWeightCol
AmplabJenkins commented on issue #25546: [SPARK-28780][ML][2.3] deprecate LinearSVCModel.setWeightCol URL: https://github.com/apache/spark/pull/25546#issuecomment-523775627 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14616/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
AmplabJenkins removed a comment on issue #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol URL: https://github.com/apache/spark/pull/25547#issuecomment-523775562 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
AmplabJenkins removed a comment on issue #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol URL: https://github.com/apache/spark/pull/25547#issuecomment-523775569 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14615/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on issue #25542: [SPARK-28840][SQL][test-hadoop3.2]conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hi
wangyum commented on issue #25542: [SPARK-28840][SQL][test-hadoop3.2]conf.getClassLoader in SparkSQLCLIDriver should be avoided as it returns the UDFClassLoader which is created by Hive URL: https://github.com/apache/spark/pull/25542#issuecomment-523775344 This issue caused by [HIVE-11878](https://issues.apache.org/jira/browse/HIVE-11878). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
shahidki31 commented on a change in pull request #24715: [SPARK-25474][SQL]
Data source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r316517654
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
##
@@ -650,4 +652,129 @@ class StatisticsCollectionSuite extends
StatisticsCollectionTestBase with Shared
}
}
}
+
+ test("Non-partitioned data source table support fallback to HDFS for size
estimation") {
+withTempDir { dir =>
+ Seq(false, true).foreach { fallBackToHDFS =>
+withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key ->
s"$fallBackToHDFS") {
+ withTable("spark_25474") {
+sql(s"CREATE TABLE spark_25474 (c1 BIGINT) USING PARQUET LOCATION
'${dir.toURI}'")
+
spark.range(5).write.mode(SaveMode.Overwrite).parquet(dir.getCanonicalPath)
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+// Table statistics are always recalculated by FileIndex
+assert(relation.stats.sizeInBytes === getDataSize(dir))
+ }
+}
+ }
+}
+ }
+
+ test("Partitioned data source table support fallback to HDFS for size
estimation") {
+Seq(false, true).foreach { fallBackToHDFS =>
+ withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key ->
s"$fallBackToHDFS") {
+withTempDir { dir =>
+ withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+ assert(relation.stats.sizeInBytes ===
+CommandUtils.getSizeInBytesFallBackToHdfs(spark,
getCatalogTable("spark_25474")))
+} else {
+ assert(relation.stats.sizeInBytes === conf.defaultSizeInBytes)
+}
+ }
+}
+ }
+}
+ }
+
+ test("Partitioned data source table support fallback to HDFS for size
estimation" +
+"with defaultSizeInBytes") {
+val defaultSizeInBytes = 10 * 1024 * 1024
+Seq(false, true).foreach { fallBackToHDFS =>
+ withSQLConf(
+SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> s"$fallBackToHDFS",
+SQLConf.DEFAULT_SIZE_IN_BYTES.key -> s"$defaultSizeInBytes") {
+withTempDir { dir =>
+ withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+ s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+ assert(relation.stats.sizeInBytes ===
+CommandUtils.getSizeInBytesFallBackToHdfs(spark,
getCatalogTable("spark_25474")))
+} else {
+ assert(relation.stats.sizeInBytes === defaultSizeInBytes)
+}
+ }
+}
+ }
+}
+ }
+
+ test("Partitioned data source table stats should be cached") {
+Seq(false, true).foreach { fallBackToHDFS =>
+ withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key ->
s"$fallBackToHDFS") {
+withTempDir { dir =>
+ withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+ s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+ val dataSize =
+CommandUtils.getSizeInBytesFallBackToHdfs(spark,
getCatalogTable("spark_25474"))
+ assert(relation.stats.sizeInBytes === dataSize)
+
+ val qualifiedTableName =
+
QualifiedTableName(spark.sessionState.catalog.getCurrentDatabase, "spark_25474")
+ val logicalRelation =
spark.sessionState.catalog.getCachedTable(qualifiedTableName)
+.asInstanceOf[LogicalRelation]
+ assert(logicalRelation.catalogTable.get.stats.get.sizeInBytes
=== dataSize)
+} else {
+ assert(relation.stats.sizeInBytes === conf.defaultSizeInBytes)
+}
+ }
+}
+ }
+}
+ }
+
+ test("External partitioned data source table does not support fallback to
HDFS " +
[GitHub] [spark] shahidki31 commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
shahidki31 commented on issue #24715: [SPARK-25474][SQL] Data source tables
support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#issuecomment-523773058
```
scala> // enable fallBackToHdfs
scala> spark.sql("set spark.sql.statistics.fallBackToHdfs=true")
res4: org.apache.spark.sql.DataFrame = [key: string, value: string]
scala> spark.sql("EXPLAIN COST SELECT * FROM t3").show(false)
++
|plan
|
++
|== Optimized Logical Plan ==
Relation[c2#1,id#2] parquet, Statistics(sizeInBytes=8.0 EiB)
```
Why is sizeInBytes still 8 EiB, even after enabling the fallback
configuration?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
dongjoon-hyun commented on a change in pull request #25547:
[SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
URL: https://github.com/apache/spark/pull/25547#discussion_r316517662
##
File path:
mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
##
@@ -313,7 +313,8 @@ class LinearSVCModel private[classification] (
setDefault(threshold, 0.0)
@Since("2.2.0")
- def setWeightCol(value: Double): this.type = set(threshold, value)
+ @deprecated("This method is deprecated and will be removed in the future.",
"2.4.4")
Review comment:
Nope. That's radical. We don't remove API in `maintenance` release. `3.0.0`
is natural for this kind of removal. I mean [the semantic
version](https://spark.apache.org/versioning-policy.html),
`major.minor.maintenance`.
> All releases with the same major version number will have API compatibility
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
dongjoon-hyun commented on a change in pull request #25547:
[SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
URL: https://github.com/apache/spark/pull/25547#discussion_r316517662
##
File path:
mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
##
@@ -313,7 +313,8 @@ class LinearSVCModel private[classification] (
setDefault(threshold, 0.0)
@Since("2.2.0")
- def setWeightCol(value: Double): this.type = set(threshold, value)
+ @deprecated("This method is deprecated and will be removed in the future.",
"2.4.4")
Review comment:
Nope. That's radical. We don't remove API in `maintenance` release. `3.0.0`
is natural for this kind of removal. I mean [the semantic
version](https://spark.apache.org/versioning-policy.html),
`major.minor.maintenance`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
dongjoon-hyun commented on a change in pull request #25547:
[SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
URL: https://github.com/apache/spark/pull/25547#discussion_r316517662
##
File path:
mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
##
@@ -313,7 +313,8 @@ class LinearSVCModel private[classification] (
setDefault(threshold, 0.0)
@Since("2.2.0")
- def setWeightCol(value: Double): this.type = set(threshold, value)
+ @deprecated("This method is deprecated and will be removed in the future.",
"2.4.4")
Review comment:
Nope. That's radical. We don't remove API in `maintenance` release. `3.0.0`
is natural for this kind of removal.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
shahidki31 commented on a change in pull request #24715: [SPARK-25474][SQL]
Data source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r316517654
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
##
@@ -650,4 +652,129 @@ class StatisticsCollectionSuite extends
StatisticsCollectionTestBase with Shared
}
}
}
+
+ test("Non-partitioned data source table support fallback to HDFS for size
estimation") {
+withTempDir { dir =>
+ Seq(false, true).foreach { fallBackToHDFS =>
+withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key ->
s"$fallBackToHDFS") {
+ withTable("spark_25474") {
+sql(s"CREATE TABLE spark_25474 (c1 BIGINT) USING PARQUET LOCATION
'${dir.toURI}'")
+
spark.range(5).write.mode(SaveMode.Overwrite).parquet(dir.getCanonicalPath)
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+// Table statistics are always recalculated by FileIndex
+assert(relation.stats.sizeInBytes === getDataSize(dir))
+ }
+}
+ }
+}
+ }
+
+ test("Partitioned data source table support fallback to HDFS for size
estimation") {
+Seq(false, true).foreach { fallBackToHDFS =>
+ withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key ->
s"$fallBackToHDFS") {
+withTempDir { dir =>
+ withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+ assert(relation.stats.sizeInBytes ===
+CommandUtils.getSizeInBytesFallBackToHdfs(spark,
getCatalogTable("spark_25474")))
+} else {
+ assert(relation.stats.sizeInBytes === conf.defaultSizeInBytes)
+}
+ }
+}
+ }
+}
+ }
+
+ test("Partitioned data source table support fallback to HDFS for size
estimation" +
+"with defaultSizeInBytes") {
+val defaultSizeInBytes = 10 * 1024 * 1024
+Seq(false, true).foreach { fallBackToHDFS =>
+ withSQLConf(
+SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> s"$fallBackToHDFS",
+SQLConf.DEFAULT_SIZE_IN_BYTES.key -> s"$defaultSizeInBytes") {
+withTempDir { dir =>
+ withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+ s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+ assert(relation.stats.sizeInBytes ===
+CommandUtils.getSizeInBytesFallBackToHdfs(spark,
getCatalogTable("spark_25474")))
+} else {
+ assert(relation.stats.sizeInBytes === defaultSizeInBytes)
+}
+ }
+}
+ }
+}
+ }
+
+ test("Partitioned data source table stats should be cached") {
+Seq(false, true).foreach { fallBackToHDFS =>
+ withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key ->
s"$fallBackToHDFS") {
+withTempDir { dir =>
+ withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+ s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+ val dataSize =
+CommandUtils.getSizeInBytesFallBackToHdfs(spark,
getCatalogTable("spark_25474"))
+ assert(relation.stats.sizeInBytes === dataSize)
+
+ val qualifiedTableName =
+
QualifiedTableName(spark.sessionState.catalog.getCurrentDatabase, "spark_25474")
+ val logicalRelation =
spark.sessionState.catalog.getCachedTable(qualifiedTableName)
+.asInstanceOf[LogicalRelation]
+ assert(logicalRelation.catalogTable.get.stats.get.sizeInBytes
=== dataSize)
+} else {
+ assert(relation.stats.sizeInBytes === conf.defaultSizeInBytes)
+}
+ }
+}
+ }
+}
+ }
+
+ test("External partitioned data source table does not support fallback to
HDFS " +
[GitHub] [spark] rednaxelafx commented on a change in pull request #20965: [SPARK-21870][SQL] Split aggregation code into small functions
rednaxelafx commented on a change in pull request #20965: [SPARK-21870][SQL]
Split aggregation code into small functions
URL: https://github.com/apache/spark/pull/20965#discussion_r316517404
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
##
@@ -267,29 +302,81 @@ case class HashAggregateExec(
e.aggregateFunction.asInstanceOf[DeclarativeAggregate].mergeExpressions
}
}
-ctx.currentVars = bufVars ++ input
-val boundUpdateExpr = bindReferences(updateExpr, inputAttrs)
-val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExpr)
-val effectiveCodes = subExprs.codes.mkString("\n")
-val aggVals = ctx.withSubExprEliminationExprs(subExprs.states) {
- boundUpdateExpr.map(_.genCode(ctx))
-}
-// aggregate buffer should be updated atomic
-val updates = aggVals.zipWithIndex.map { case (ev, i) =>
+
+if (!conf.codegenSplitAggregateFunc) {
+ ctx.currentVars = bufVars ++ input
+ val boundUpdateExpr = updateExpr.map(BindReferences.bindReference(_,
inputAttrs))
+ val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExpr)
+ val effectiveCodes = subExprs.codes.mkString("\n")
+ val aggVals = ctx.withSubExprEliminationExprs(subExprs.states) {
+boundUpdateExpr.map(_.genCode(ctx))
+ }
+ // aggregate buffer should be updated atomic
+ val updates = aggVals.zipWithIndex.map { case (ev, i) =>
+s"""
+ | ${bufVars(i).isNull} = ${ev.isNull};
+ | ${bufVars(i).value} = ${ev.value};
+ """.stripMargin
+ }
+ s"""
+ | // do aggregate
+ | // common sub-expressions
+ | $effectiveCodes
+ | // evaluate aggregate function
+ | ${evaluateVariables(aggVals)}
+ | // update aggregation buffer
+ | ${updates.mkString("\n").trim}
+ """.stripMargin
+} else {
+ // We need to copy the aggregation buffer to local variables first
because each aggregate
Review comment:
I think I'm starting to understand ... I had a different basic assumption
from your splitting logic.
I started realizing that when reading your sample generated code in the PR
description (BTW that sample code should be updated to match what it currently
generates when you rebase this PR to the latest master)
```java
/* 108 */ // copy aggregation buffer to the local
/* 109 */ boolean agg_localBufIsNull = agg_bufIsNull;
/* 110 */ long agg_localBufValue = agg_bufValue;
/* 111 */ boolean agg_localBufIsNull1 = agg_bufIsNull1;
/* 112 */ double agg_localBufValue1 = agg_bufValue1;
/* 113 */ boolean agg_localBufIsNull2 = agg_bufIsNull2;
/* 114 */ long agg_localBufValue2 = agg_bufValue2;
/* 115 */ // common sub-expressions
/* 116 */
/* 117 */ // process aggregate functions to update aggregation buffer
/* 118 */ agg_doAggregateVal_coalesce(agg_localBufIsNull,
agg_localBufValue, inputadapter_value, inputadapter_isNull);
/* 119 */ agg_doAggregateVal_add(agg_localBufValue1,
inputadapter_isNull1, inputadapter_value1, agg_localBufIsNull1);
/* 120 */ agg_doAggregateVal_add1(inputadapter_isNull2,
inputadapter_value2, agg_localBufIsNull2, agg_localBufValue2);
```
I thought for `SUM(a), AVG(a)` I was going to see `agg_doAggregateVal_sum`
and `agg_doAggregateVal_avg`, but instead what I'm seeing here is a more
fine-grained splitting for each update expression. So instead we get:
- `agg_doAggregateVal_coalesce` for `sum(a)`
- `agg_doAggregateVal_add` and `agg_doAggregateVal_add1` for `avg(a)`
My previous comment in this thread only applies when the splitting boundary
is on per-aggregate-expression granularity, instead of on
per-update-expression-in-aggregate-function granularity.
`kurtosis()` is pretty much the largest declarative aggregate function in
Spark SQL right now. I don't think a single `kurtosis()` would go over 8000
bytes worth of bytecode, so maybe a per-aggregate-expression granularity would
make more sense?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
dongjoon-hyun commented on a change in pull request #24715: [SPARK-25474][SQL]
Data source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r316516876
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
##
@@ -650,4 +652,129 @@ class StatisticsCollectionSuite extends
StatisticsCollectionTestBase with Shared
}
}
}
+
+ test("Non-partitioned data source table support fallback to HDFS for size
estimation") {
+withTempDir { dir =>
+ Seq(false, true).foreach { fallBackToHDFS =>
+withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key ->
s"$fallBackToHDFS") {
+ withTable("spark_25474") {
+sql(s"CREATE TABLE spark_25474 (c1 BIGINT) USING PARQUET LOCATION
'${dir.toURI}'")
+
spark.range(5).write.mode(SaveMode.Overwrite).parquet(dir.getCanonicalPath)
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+// Table statistics are always recalculated by FileIndex
+assert(relation.stats.sizeInBytes === getDataSize(dir))
+ }
+}
+ }
+}
+ }
+
+ test("Partitioned data source table support fallback to HDFS for size
estimation") {
+Seq(false, true).foreach { fallBackToHDFS =>
+ withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key ->
s"$fallBackToHDFS") {
+withTempDir { dir =>
+ withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+ assert(relation.stats.sizeInBytes ===
+CommandUtils.getSizeInBytesFallBackToHdfs(spark,
getCatalogTable("spark_25474")))
+} else {
+ assert(relation.stats.sizeInBytes === conf.defaultSizeInBytes)
+}
+ }
+}
+ }
+}
+ }
+
+ test("Partitioned data source table support fallback to HDFS for size
estimation" +
+"with defaultSizeInBytes") {
+val defaultSizeInBytes = 10 * 1024 * 1024
+Seq(false, true).foreach { fallBackToHDFS =>
+ withSQLConf(
+SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> s"$fallBackToHDFS",
+SQLConf.DEFAULT_SIZE_IN_BYTES.key -> s"$defaultSizeInBytes") {
+withTempDir { dir =>
+ withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+ s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+ assert(relation.stats.sizeInBytes ===
+CommandUtils.getSizeInBytesFallBackToHdfs(spark,
getCatalogTable("spark_25474")))
+} else {
+ assert(relation.stats.sizeInBytes === defaultSizeInBytes)
+}
+ }
+}
+ }
+}
+ }
+
+ test("Partitioned data source table stats should be cached") {
+Seq(false, true).foreach { fallBackToHDFS =>
+ withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key ->
s"$fallBackToHDFS") {
+withTempDir { dir =>
+ withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+ s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+ val dataSize =
+CommandUtils.getSizeInBytesFallBackToHdfs(spark,
getCatalogTable("spark_25474"))
+ assert(relation.stats.sizeInBytes === dataSize)
+
+ val qualifiedTableName =
+
QualifiedTableName(spark.sessionState.catalog.getCurrentDatabase, "spark_25474")
+ val logicalRelation =
spark.sessionState.catalog.getCachedTable(qualifiedTableName)
+.asInstanceOf[LogicalRelation]
+ assert(logicalRelation.catalogTable.get.stats.get.sizeInBytes
=== dataSize)
+} else {
+ assert(relation.stats.sizeInBytes === conf.defaultSizeInBytes)
+}
+ }
+}
+ }
+}
+ }
+
+ test("External partitioned data source table does not support fallback to
HDFS "
[GitHub] [spark] cloud-fan closed pull request #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage
cloud-fan closed pull request #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage URL: https://github.com/apache/spark/pull/25508 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage
cloud-fan commented on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage URL: https://github.com/apache/spark/pull/25508#issuecomment-523769722 thanks, merging to 2.3! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage
AmplabJenkins removed a comment on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage URL: https://github.com/apache/spark/pull/25508#issuecomment-523767499 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109542/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage
AmplabJenkins removed a comment on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage URL: https://github.com/apache/spark/pull/25508#issuecomment-523767489 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #25547: [SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
zhengruifeng commented on a change in pull request #25547:
[SPARK-28780][ML][2.4] deprecate LinearSVCModel.setWeightCol
URL: https://github.com/apache/spark/pull/25547#discussion_r316513289
##
File path:
mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala
##
@@ -313,7 +313,8 @@ class LinearSVCModel private[classification] (
setDefault(threshold, 0.0)
@Since("2.2.0")
- def setWeightCol(value: Double): this.type = set(threshold, value)
+ @deprecated("This method is deprecated and will be removed in the future.",
"2.4.4")
Review comment:
I am not sure but `2.4.5` maybe anothor option?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
dongjoon-hyun commented on a change in pull request #24715: [SPARK-25474][SQL]
Data source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r316513612
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
##
@@ -650,4 +652,129 @@ class StatisticsCollectionSuite extends
StatisticsCollectionTestBase with Shared
}
}
}
+
+ test("Non-partitioned data source table support fallback to HDFS for size
estimation") {
+withTempDir { dir =>
+ Seq(false, true).foreach { fallBackToHDFS =>
+withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key ->
s"$fallBackToHDFS") {
+ withTable("spark_25474") {
+sql(s"CREATE TABLE spark_25474 (c1 BIGINT) USING PARQUET LOCATION
'${dir.toURI}'")
+
spark.range(5).write.mode(SaveMode.Overwrite).parquet(dir.getCanonicalPath)
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+// Table statistics are always recalculated by FileIndex
+assert(relation.stats.sizeInBytes === getDataSize(dir))
+ }
+}
+ }
+}
+ }
+
+ test("Partitioned data source table support fallback to HDFS for size
estimation") {
+Seq(false, true).foreach { fallBackToHDFS =>
+ withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key ->
s"$fallBackToHDFS") {
+withTempDir { dir =>
+ withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+ assert(relation.stats.sizeInBytes ===
+CommandUtils.getSizeInBytesFallBackToHdfs(spark,
getCatalogTable("spark_25474")))
+} else {
+ assert(relation.stats.sizeInBytes === conf.defaultSizeInBytes)
+}
+ }
+}
+ }
+}
+ }
+
+ test("Partitioned data source table support fallback to HDFS for size
estimation" +
+"with defaultSizeInBytes") {
+val defaultSizeInBytes = 10 * 1024 * 1024
+Seq(false, true).foreach { fallBackToHDFS =>
+ withSQLConf(
+SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> s"$fallBackToHDFS",
+SQLConf.DEFAULT_SIZE_IN_BYTES.key -> s"$defaultSizeInBytes") {
+withTempDir { dir =>
+ withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+ s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+ assert(relation.stats.sizeInBytes ===
+CommandUtils.getSizeInBytesFallBackToHdfs(spark,
getCatalogTable("spark_25474")))
+} else {
+ assert(relation.stats.sizeInBytes === defaultSizeInBytes)
+}
+ }
+}
+ }
+}
+ }
+
+ test("Partitioned data source table stats should be cached") {
+Seq(false, true).foreach { fallBackToHDFS =>
+ withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key ->
s"$fallBackToHDFS") {
+withTempDir { dir =>
+ withTable("spark_25474") {
+sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
+ s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
+sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation =
spark.table("spark_25474").queryExecution.analyzed.children.head
+if (fallBackToHDFS) {
+ val dataSize =
+CommandUtils.getSizeInBytesFallBackToHdfs(spark,
getCatalogTable("spark_25474"))
+ assert(relation.stats.sizeInBytes === dataSize)
+
+ val qualifiedTableName =
+
QualifiedTableName(spark.sessionState.catalog.getCurrentDatabase, "spark_25474")
+ val logicalRelation =
spark.sessionState.catalog.getCachedTable(qualifiedTableName)
+.asInstanceOf[LogicalRelation]
+ assert(logicalRelation.catalogTable.get.stats.get.sizeInBytes
=== dataSize)
+} else {
+ assert(relation.stats.sizeInBytes === conf.defaultSizeInBytes)
+}
+ }
+}
+ }
+}
+ }
+
+ test("External partitioned data source table does not support fallback to
HDFS "
[GitHub] [spark] AmplabJenkins commented on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage
AmplabJenkins commented on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage URL: https://github.com/apache/spark/pull/25508#issuecomment-523767499 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109542/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage
AmplabJenkins commented on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage URL: https://github.com/apache/spark/pull/25508#issuecomment-523767489 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage
SparkQA removed a comment on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage URL: https://github.com/apache/spark/pull/25508#issuecomment-523720415 **[Test build #109542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109542/testReport)** for PR 25508 at commit [`b7b7150`](https://github.com/apache/spark/commit/b7b715090c646a9aae81059948d85c6d1977e526). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage
SparkQA commented on issue #25508: [SPARK-28699][CORE][2.3] Fix a corner case for aborting indeterminate stage URL: https://github.com/apache/spark/pull/25508#issuecomment-523767151 **[Test build #109542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109542/testReport)** for PR 25508 at commit [`b7b7150`](https://github.com/apache/spark/commit/b7b715090c646a9aae81059948d85c6d1977e526). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
AmplabJenkins removed a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523766327 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109557/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
SparkQA removed a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523763672 **[Test build #109557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109557/testReport)** for PR 25525 at commit [`4dc6caa`](https://github.com/apache/spark/commit/4dc6caa50765187ab0c535da3ed768b6380ad8cf). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
AmplabJenkins removed a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523766323 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
AmplabJenkins commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523766327 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109557/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
AmplabJenkins commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523766323 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
SparkQA commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523766229 **[Test build #109557 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109557/testReport)** for PR 25525 at commit [`4dc6caa`](https://github.com/apache/spark/commit/4dc6caa50765187ab0c535da3ed768b6380ad8cf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
AmplabJenkins commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523765068 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14614/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
AmplabJenkins removed a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523765068 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14614/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
AmplabJenkins removed a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523765063 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
AmplabJenkins commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523765063 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #24759: [SPARK-27395][SQL] Improve EXPLAIN command
dilipbiswal commented on a change in pull request #24759: [SPARK-27395][SQL]
Improve EXPLAIN command
URL: https://github.com/apache/spark/pull/24759#discussion_r316510915
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/Exchange.scala
##
@@ -77,6 +77,15 @@ case class ReusedExchangeExec(override val output:
Seq[Attribute], child: Exchan
override def outputOrdering: Seq[SortOrder] = {
child.outputOrdering.map(updateAttr(_).asInstanceOf[SortOrder])
}
+
+ override def verboseStringWithOperatorId(): String = {
+val cdgen = ExplainUtils.getCodegenId(this)
+val reuse_op_str = ExplainUtils.getOpId(child)
+s"""
+ |(${ExplainUtils.getOpId(this)}) $nodeName ${cdgen} [Reuses operator
id: $reuse_op_str]
Review comment:
@cloud-fan I think we are okay here. Here is a simple plan involving
`ReusedSubqueryExec`
```
explain formatted select (select avg(a) from s1) + (select avg(a) from s1)
from s1 limit 1;
```
```
== Physical Plan ==
CollectLimit (3)
+- * Project (2)
+- Scan hive default.s1 (1)
(1) Scan hive default.s1
Output: []
(2) Project [codegen id : 1]
Output: [(Subquery scalar-subquery#151 + ReusedSubquery Subquery
scalar-subquery#151) AS (scalarsubquery() + scalarsubquery())#168]
Input : []
(3) CollectLimit
Input: [(scalarsubquery() + scalarsubquery())#168]
= Subqueries =
Subquery:1 Hosting operator id = 2 Hosting Expression = Subquery
scalar-subquery#151
* HashAggregate (7)
+- Exchange (6)
+- * HashAggregate (5)
+- Scan hive default.s1 (4)
(4) Scan hive default.s1
Output: [a#160]
(5) HashAggregate [codegen id : 1]
Input: [a#160]
(6) Exchange
Input: [sum#171, count#172L]
(7) HashAggregate [codegen id : 2]
Input: [sum#171, count#172L]
Subquery:2 Hosting operator id = 2 Hosting Expression = ReusedSubquery
Subquery scalar-subquery#151
```
Please let me know what you think.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
SparkQA commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523764609 **[Test build #109556 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109556/testReport)** for PR 25549 at commit [`f02a30b`](https://github.com/apache/spark/commit/f02a30be5a369467d11b7fc6e0f7d6940daf7589). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523764687 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523764690 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109556/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao edited a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
huaxingao edited a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523557375  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
SparkQA removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523761753 **[Test build #109556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109556/testReport)** for PR 25549 at commit [`f02a30b`](https://github.com/apache/spark/commit/f02a30be5a369467d11b7fc6e0f7d6940daf7589). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523764690 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109556/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523764687 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao edited a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
huaxingao edited a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523555927  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #20965: [SPARK-21870][SQL] Split aggregation code into small functions
maropu commented on a change in pull request #20965: [SPARK-21870][SQL] Split
aggregation code into small functions
URL: https://github.com/apache/spark/pull/20965#discussion_r316510032
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
##
@@ -267,29 +302,81 @@ case class HashAggregateExec(
e.aggregateFunction.asInstanceOf[DeclarativeAggregate].mergeExpressions
}
}
-ctx.currentVars = bufVars ++ input
-val boundUpdateExpr = bindReferences(updateExpr, inputAttrs)
-val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExpr)
-val effectiveCodes = subExprs.codes.mkString("\n")
-val aggVals = ctx.withSubExprEliminationExprs(subExprs.states) {
- boundUpdateExpr.map(_.genCode(ctx))
-}
-// aggregate buffer should be updated atomic
-val updates = aggVals.zipWithIndex.map { case (ev, i) =>
+
+if (!conf.codegenSplitAggregateFunc) {
+ ctx.currentVars = bufVars ++ input
+ val boundUpdateExpr = updateExpr.map(BindReferences.bindReference(_,
inputAttrs))
+ val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExpr)
+ val effectiveCodes = subExprs.codes.mkString("\n")
+ val aggVals = ctx.withSubExprEliminationExprs(subExprs.states) {
+boundUpdateExpr.map(_.genCode(ctx))
+ }
+ // aggregate buffer should be updated atomic
+ val updates = aggVals.zipWithIndex.map { case (ev, i) =>
+s"""
+ | ${bufVars(i).isNull} = ${ev.isNull};
+ | ${bufVars(i).value} = ${ev.value};
+ """.stripMargin
+ }
+ s"""
+ | // do aggregate
Review comment:
Yea, anyway thanks for your carefull checks! I'm currently checking your
comments step-by-step.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao edited a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
huaxingao edited a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523554706  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
SparkQA commented on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523763672 **[Test build #109557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109557/testReport)** for PR 25525 at commit [`4dc6caa`](https://github.com/apache/spark/commit/4dc6caa50765187ab0c535da3ed768b6380ad8cf). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
SparkQA commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523761753 **[Test build #109556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109556/testReport)** for PR 25549 at commit [`f02a30b`](https://github.com/apache/spark/commit/f02a30be5a369467d11b7fc6e0f7d6940daf7589). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523761219 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest
AmplabJenkins removed a comment on issue #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest URL: https://github.com/apache/spark/pull/25550#issuecomment-523761217 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523761225 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14613/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest
AmplabJenkins removed a comment on issue #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest URL: https://github.com/apache/spark/pull/25550#issuecomment-523761222 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14612/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
dilipbiswal commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523761303 Nit : typeo in examples invalided -> invalidated This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest
AmplabJenkins commented on issue #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest URL: https://github.com/apache/spark/pull/25550#issuecomment-523761217 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523761225 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14613/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523761219 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest
AmplabJenkins commented on issue #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest URL: https://github.com/apache/spark/pull/25550#issuecomment-523761222 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14612/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest
dongjoon-hyun commented on issue #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest URL: https://github.com/apache/spark/pull/25550#issuecomment-523760031 Thank you so much for review and approval! 👍 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
AmplabJenkins removed a comment on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network URL: https://github.com/apache/spark/pull/25500#issuecomment-523759770 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
AmplabJenkins removed a comment on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network URL: https://github.com/apache/spark/pull/25500#issuecomment-523759777 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109548/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
HyukjinKwon commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523759859 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523759203 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
AmplabJenkins commented on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network URL: https://github.com/apache/spark/pull/25500#issuecomment-523759777 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109548/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
AmplabJenkins commented on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network URL: https://github.com/apache/spark/pull/25500#issuecomment-523759770 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest
SparkQA commented on issue #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest URL: https://github.com/apache/spark/pull/25550#issuecomment-523759771 **[Test build #109555 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109555/testReport)** for PR 25550 at commit [`39e7ece`](https://github.com/apache/spark/commit/39e7ecec7c03b2cae97b3564952a2dfde4b6a712). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] kevinyu98 commented on a change in pull request #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
kevinyu98 commented on a change in pull request #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#discussion_r316506249 ## File path: docs/sql-ref-syntax-aux-describe-database.md ## @@ -18,5 +18,23 @@ license: | See the License for the specific language governing permissions and limitations under the License. --- +### Description + +Return the metadata of an existing database(name, comment and location). If the database does not exist, +an exception is thrown. + +When `extended` is specified, it also shows the database's properties. Review comment: sure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
SparkQA removed a comment on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network URL: https://github.com/apache/spark/pull/25500#issuecomment-523731949 **[Test build #109548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109548/testReport)** for PR 25500 at commit [`3990fec`](https://github.com/apache/spark/commit/3990fec4bfe91f757872e3d79d059591c9ed1748). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network
SparkQA commented on issue #25500: [SPARK-28778][MESOS] Fixed executors advertised address when running in virtual network URL: https://github.com/apache/spark/pull/25500#issuecomment-523759339 **[Test build #109548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109548/testReport)** for PR 25500 at commit [`3990fec`](https://github.com/apache/spark/commit/3990fec4bfe91f757872e3d79d059591c9ed1748). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] kevinyu98 commented on a change in pull request #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
kevinyu98 commented on a change in pull request #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#discussion_r316506170 ## File path: docs/sql-ref-syntax-aux-describe-database.md ## @@ -18,5 +18,23 @@ license: | See the License for the specific language governing permissions and limitations under the License. --- +### Description + +Return the metadata of an existing database(name, comment and location). If the database does not exist, Review comment: ok. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest
dongjoon-hyun opened a new pull request #25550: [SPARK-28847][TEST] Annotate HiveExternalCatalogVersionsSuite with ExtendedHiveTest URL: https://github.com/apache/spark/pull/25550 ### What changes were proposed in this pull request? This PR aims to annotate `HiveExternalCatalogVersionsSuite` with `ExtendedHiveTest`. ### Why are the changes needed? `HiveExternalCatalogVersionsSuite` is an outstanding test in terms of testing time. This PR aims to allow skipping this test suite when we use `ExtendedHiveTest`.  ### Does this PR introduce any user-facing change? No ### How was this patch tested? Since Jenkins doesn't exclude `ExtendedHiveTest`, there is no difference in Jenkins testing. This PR should be tested by manually by the following. **BEFORE** ``` $ cd sql/hive $ mvn package -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite -Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest ... Run starting. Expected test count is: 1 HiveExternalCatalogVersionsSuite: 22:32:16.218 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load ... ``` **AFTER** ``` $ cd sql/hive $ mvn package -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.HiveExternalCatalogVersionsSuite -Dtest.exclude.tags=org.apache.spark.tags.ExtendedHiveTest ... Run starting. Expected test count is: 0 HiveExternalCatalogVersionsSuite: Run completed in 772 milliseconds. Total number of tests run: 0 Suites: completed 2, aborted 0 Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0 No tests were executed. ... ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523759090 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference
SparkQA removed a comment on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference URL: https://github.com/apache/spark/pull/25541#issuecomment-523756445 **[Test build #109553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109553/testReport)** for PR 25541 at commit [`1f820a4`](https://github.com/apache/spark/commit/1f820a45cae238b67cef65ac7769172ce33d9137). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference
AmplabJenkins removed a comment on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference URL: https://github.com/apache/spark/pull/25541#issuecomment-523758860 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109553/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins removed a comment on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523758024 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference
AmplabJenkins removed a comment on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference URL: https://github.com/apache/spark/pull/25541#issuecomment-523758854 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523759203 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523759090 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference
AmplabJenkins commented on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference URL: https://github.com/apache/spark/pull/25541#issuecomment-523758854 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference
AmplabJenkins commented on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference URL: https://github.com/apache/spark/pull/25541#issuecomment-523758860 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109553/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference
SparkQA commented on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference URL: https://github.com/apache/spark/pull/25541#issuecomment-523758783 **[Test build #109553 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109553/testReport)** for PR 25541 at commit [`1f820a4`](https://github.com/apache/spark/commit/1f820a45cae238b67cef65ac7769172ce33d9137). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24759: [SPARK-27395][SQL] Improve EXPLAIN command
SparkQA commented on issue #24759: [SPARK-27395][SQL] Improve EXPLAIN command URL: https://github.com/apache/spark/pull/24759#issuecomment-523758116 **[Test build #109554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109554/testReport)** for PR 24759 at commit [`b846922`](https://github.com/apache/spark/commit/b846922bcd4172d554f688bd8f713c6837d9c2d0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
AmplabJenkins commented on issue #25549: [SPARK-28828][DOC] Document REFRESH TABLE command URL: https://github.com/apache/spark/pull/25549#issuecomment-523758024 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] kevinyu98 opened a new pull request #25549: [SPARK-28828][DOC] Document REFRESH TABLE command
kevinyu98 opened a new pull request #25549: [SPARK-28828][DOC] Document REFRESH
TABLE command
URL: https://github.com/apache/spark/pull/25549
### What changes were proposed in this pull request?
Document REFRESH TABLE statement in the SQL Reference Guide.
### Why are the changes needed?
Currently there is no documentation in the SPARK SQL to describe how to use
this command, it is to address this issue.
### Does this PR introduce any user-facing change?
Yes.
Before:
There is no documentation for this.
After:
### How was this patch tested?
Using jykll build --serve
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference
AmplabJenkins removed a comment on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference URL: https://github.com/apache/spark/pull/25541#issuecomment-523757614 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24759: [SPARK-27395][SQL] Improve EXPLAIN command
AmplabJenkins removed a comment on issue #24759: [SPARK-27395][SQL] Improve EXPLAIN command URL: https://github.com/apache/spark/pull/24759#issuecomment-523757604 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14611/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24759: [SPARK-27395][SQL] Improve EXPLAIN command
AmplabJenkins commented on issue #24759: [SPARK-27395][SQL] Improve EXPLAIN command URL: https://github.com/apache/spark/pull/24759#issuecomment-523757600 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24759: [SPARK-27395][SQL] Improve EXPLAIN command
AmplabJenkins removed a comment on issue #24759: [SPARK-27395][SQL] Improve EXPLAIN command URL: https://github.com/apache/spark/pull/24759#issuecomment-523757600 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference
AmplabJenkins commented on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference URL: https://github.com/apache/spark/pull/25541#issuecomment-523757623 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14610/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference
AmplabJenkins commented on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference URL: https://github.com/apache/spark/pull/25541#issuecomment-523757614 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference
AmplabJenkins removed a comment on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference URL: https://github.com/apache/spark/pull/25541#issuecomment-523757623 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14610/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24759: [SPARK-27395][SQL] Improve EXPLAIN command
AmplabJenkins commented on issue #24759: [SPARK-27395][SQL] Improve EXPLAIN command URL: https://github.com/apache/spark/pull/24759#issuecomment-523757604 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14611/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao edited a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference
huaxingao edited a comment on issue #25525: [SPARK-28786][DOC][SQL]Document INSERT statement in SQL Reference URL: https://github.com/apache/spark/pull/25525#issuecomment-523557375  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference
SparkQA commented on issue #25541: [SPARK-28831][DOC][SQL]Document CLEAR CACHE statement in SQL Reference URL: https://github.com/apache/spark/pull/25541#issuecomment-523756445 **[Test build #109553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109553/testReport)** for PR 25541 at commit [`1f820a4`](https://github.com/apache/spark/commit/1f820a45cae238b67cef65ac7769172ce33d9137). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] rednaxelafx commented on a change in pull request #20965: [SPARK-21870][SQL] Split aggregation code into small functions
rednaxelafx commented on a change in pull request #20965: [SPARK-21870][SQL]
Split aggregation code into small functions
URL: https://github.com/apache/spark/pull/20965#discussion_r316501527
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala
##
@@ -267,29 +302,81 @@ case class HashAggregateExec(
e.aggregateFunction.asInstanceOf[DeclarativeAggregate].mergeExpressions
}
}
-ctx.currentVars = bufVars ++ input
-val boundUpdateExpr = bindReferences(updateExpr, inputAttrs)
-val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExpr)
-val effectiveCodes = subExprs.codes.mkString("\n")
-val aggVals = ctx.withSubExprEliminationExprs(subExprs.states) {
- boundUpdateExpr.map(_.genCode(ctx))
-}
-// aggregate buffer should be updated atomic
-val updates = aggVals.zipWithIndex.map { case (ev, i) =>
+
+if (!conf.codegenSplitAggregateFunc) {
+ ctx.currentVars = bufVars ++ input
+ val boundUpdateExpr = updateExpr.map(BindReferences.bindReference(_,
inputAttrs))
+ val subExprs =
ctx.subexpressionEliminationForWholeStageCodegen(boundUpdateExpr)
+ val effectiveCodes = subExprs.codes.mkString("\n")
+ val aggVals = ctx.withSubExprEliminationExprs(subExprs.states) {
+boundUpdateExpr.map(_.genCode(ctx))
+ }
+ // aggregate buffer should be updated atomic
+ val updates = aggVals.zipWithIndex.map { case (ev, i) =>
+s"""
+ | ${bufVars(i).isNull} = ${ev.isNull};
+ | ${bufVars(i).value} = ${ev.value};
+ """.stripMargin
+ }
+ s"""
+ | // do aggregate
Review comment:
Thanks! I'm totally fine with either cleaning it up here or in a follow-up.
This is purely cosmetic and wouldn't change the core feature.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25507: [SPARK-28667][SQL] Support InsertInto through the V2SessionCatalog
AmplabJenkins removed a comment on issue #25507: [SPARK-28667][SQL] Support InsertInto through the V2SessionCatalog URL: https://github.com/apache/spark/pull/25507#issuecomment-523753145 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
