[GitHub] [spark] AmplabJenkins removed a comment on pull request #29137: [SPARK-32337][SQL] Show initial plan in AQE plan tree string
AmplabJenkins removed a comment on pull request #29137: URL: https://github.com/apache/spark/pull/29137#issuecomment-663375536 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29212: [SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test
SparkQA commented on pull request #29212: URL: https://github.com/apache/spark/pull/29212#issuecomment-663375756 **[Test build #126474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126474/testReport)** for PR 29212 at commit [`71193f1`](https://github.com/apache/spark/commit/71193f194758de23794a2bc46d16766e334091d2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29137: [SPARK-32337][SQL] Show initial plan in AQE plan tree string
AmplabJenkins commented on pull request #29137: URL: https://github.com/apache/spark/pull/29137#issuecomment-663375536 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29137: [SPARK-32337][SQL] Show initial plan in AQE plan tree string
SparkQA commented on pull request #29137: URL: https://github.com/apache/spark/pull/29137#issuecomment-663374407 **[Test build #126442 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126442/testReport)** for PR 29137 at commit [`1b3a685`](https://github.com/apache/spark/commit/1b3a685c6fc551d806e7064a8f81a67a206e10b1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29137: [SPARK-32337][SQL] Show initial plan in AQE plan tree string
SparkQA removed a comment on pull request #29137: URL: https://github.com/apache/spark/pull/29137#issuecomment-663285576 **[Test build #126442 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126442/testReport)** for PR 29137 at commit [`1b3a685`](https://github.com/apache/spark/commit/1b3a685c6fc551d806e7064a8f81a67a206e10b1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29014: [SPARK-32199][SPARK-32198] Reduce job failures during decommissioning
AmplabJenkins removed a comment on pull request #29014: URL: https://github.com/apache/spark/pull/29014#issuecomment-663369849 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29014: [SPARK-32199][SPARK-32198] Reduce job failures during decommissioning
AmplabJenkins commented on pull request #29014: URL: https://github.com/apache/spark/pull/29014#issuecomment-663369849 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #29204: [SPARK-32412][SQL] Unify error handling for spark thrift server operations
cloud-fan commented on pull request #29204: URL: https://github.com/apache/spark/pull/29204#issuecomment-663369567 cc @juliuszsompolski This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29204: [SPARK-32412][SQL] Unify error handling for spark thrift server operations
AmplabJenkins removed a comment on pull request #29204: URL: https://github.com/apache/spark/pull/29204#issuecomment-663367974 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29202: [SPARK-32406][SQL] Make RESET syntax support single configuration reset
AmplabJenkins removed a comment on pull request #29202: URL: https://github.com/apache/spark/pull/29202#issuecomment-663367930 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29202: [SPARK-32406][SQL] Make RESET syntax support single configuration reset
AmplabJenkins commented on pull request #29202: URL: https://github.com/apache/spark/pull/29202#issuecomment-663367930 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29204: [SPARK-32412][SQL] Unify error handling for spark thrift server operations
AmplabJenkins commented on pull request #29204: URL: https://github.com/apache/spark/pull/29204#issuecomment-663367974 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #29168: [SPARK-32375][SQL] Basic functionality of table catalog v2 for JDBC
MaxGekk commented on pull request #29168: URL: https://github.com/apache/spark/pull/29168#issuecomment-663367114 @cloud-fan Please, take a look at this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29215: [SPARK-32363][PYTHON][BUILD][3.0] Fix flakiness in pip package testing in Jenkins
AmplabJenkins commented on pull request #29215: URL: https://github.com/apache/spark/pull/29215#issuecomment-663366133 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29215: [SPARK-32363][PYTHON][BUILD][3.0] Fix flakiness in pip package testing in Jenkins
AmplabJenkins removed a comment on pull request #29215: URL: https://github.com/apache/spark/pull/29215#issuecomment-663366133 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #29202: [SPARK-32406][SQL] Make RESET syntax support single configuration reset
yaooqinn commented on a change in pull request #29202: URL: https://github.com/apache/spark/pull/29202#discussion_r459871040 ## File path: docs/sql-ref-syntax-aux-conf-mgmt-reset.md ## @@ -21,19 +21,34 @@ license: | ### Description -Reset any runtime configurations specific to the current session which were set via the [SET](sql-ref-syntax-aux-conf-mgmt-set.html) command to their default values. +The RESET command resets runtime configurations specific to the current session which were set via the [SET](sql-ref-syntax-aux-conf-mgmt-set.html) command to their default values. ### Syntax ```sql -RESET +RESET; + +RESET configuration_key; ``` +### Parameters + +* **(none)** + +Reset any runtime configurations specific to the current session which were set via the [SET](sql-ref-syntax-aux-conf-mgmt-set.html) command to their default values. + +* **configuration_key** + +Restore the value of the `configuration_key` to the default value. If the default value is undefined, the `configuration_key` will be removed. + ### Examples ```sql -- Reset any runtime configurations specific to the current session which were set via the SET command to their default values. RESET; + +-- If you start your application with --conf spark.foo=bar and set spark.foo=foobar in runtime, the example below will restore it to 'bar'. If spark.foo is not specified during starting, the example bellow will remove this config from the SQLConf if you set it in runtime. It will do nothing for the nonexistent keys. Review comment: thanks for the suggestions This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on a change in pull request #29202: [SPARK-32406][SQL] Make RESET syntax support single configuration reset
yaooqinn commented on a change in pull request #29202: URL: https://github.com/apache/spark/pull/29202#discussion_r459871190 ## File path: docs/sql-ref-syntax-aux-conf-mgmt-reset.md ## @@ -21,19 +21,34 @@ license: | ### Description -Reset any runtime configurations specific to the current session which were set via the [SET](sql-ref-syntax-aux-conf-mgmt-set.html) command to their default values. +The RESET command resets runtime configurations specific to the current session which were set via the [SET](sql-ref-syntax-aux-conf-mgmt-set.html) command to their default values. ### Syntax ```sql -RESET +RESET; + +RESET configuration_key; ``` +### Parameters + +* **(none)** + +Reset any runtime configurations specific to the current session which were set via the [SET](sql-ref-syntax-aux-conf-mgmt-set.html) command to their default values. + +* **configuration_key** + +Restore the value of the `configuration_key` to the default value. If the default value is undefined, the `configuration_key` will be removed. + ### Examples ```sql -- Reset any runtime configurations specific to the current session which were set via the SET command to their default values. RESET; + +-- If you start your application with --conf spark.foo=bar and set spark.foo=foobar in runtime, the example below will restore it to 'bar'. If spark.foo is not specified during starting, the example bellow will remove this config from the SQLConf if you set it in runtime. It will do nothing for the nonexistent keys. Review comment: thanks for the suggestions This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29215: [SPARK-32363][PYTHON][BUILD][3.0] Fix flakiness in pip package testing in Jenkins
HyukjinKwon commented on pull request #29215: URL: https://github.com/apache/spark/pull/29215#issuecomment-663365338 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28898: [SPARK-32059][SQL] Allow nested schema pruning thru window/sort plans
AmplabJenkins commented on pull request #28898: URL: https://github.com/apache/spark/pull/28898#issuecomment-663364305 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28898: [SPARK-32059][SQL] Allow nested schema pruning thru window/sort plans
AmplabJenkins removed a comment on pull request #28898: URL: https://github.com/apache/spark/pull/28898#issuecomment-663364305 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29202: [SPARK-32406][SQL] Make RESET syntax support single configuration reset
HyukjinKwon commented on pull request #29202: URL: https://github.com/apache/spark/pull/29202#issuecomment-663364200 Looks good This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29202: [SPARK-32406][SQL] Make RESET syntax support single configuration reset
HyukjinKwon commented on a change in pull request #29202:
URL: https://github.com/apache/spark/pull/29202#discussion_r459868527
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala
##
@@ -181,6 +181,21 @@ class SQLConfSuite extends QueryTest with
SharedSparkSession {
}
}
+ test("reset - single configuration") {
+val appId = spark.sparkContext.getConf.getAppId
+sql("RESET spark.app.id")
+assert(spark.conf.get("spark.app.id") === appId, "Should not change spark
core ones")
+sql("SET spark.abc=xyz")
+assert(spark.conf.get("spark.abc") === "xyz")
+sql("RESET spark.abc")
+intercept[NoSuchElementException](spark.conf.get("spark.abc"))
+sql("RESET spark.abc")
+val original = spark.conf.get(SQLConf.GROUP_BY_ORDINAL)
+sql(s"set ${SQLConf.GROUP_BY_ORDINAL.key}=false")
Review comment:
tiny nit: maybe `set` -> `SET` just for consistency.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #29187: [SPARK-32387][SS] Extract UninterruptibleThread runner logic from KafkaOffsetReader
xuanyuanking commented on pull request #29187: URL: https://github.com/apache/spark/pull/29187#issuecomment-663362922 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28649: [SPARK-31829] Check for partition existence before computation
cloud-fan commented on a change in pull request #28649:
URL: https://github.com/apache/spark/pull/28649#discussion_r459867268
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##
@@ -148,6 +148,15 @@ case class InsertIntoHiveTable(
|Table partitions:
${table.partitionColumnNames.mkString(",")}""".stripMargin)
}
+val oldPart = if ( numDynamicPartitions == 0) {
Review comment:
or just make it a lazy val, so that we only compute it when needed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28649: [SPARK-31829] Check for partition existence before computation
cloud-fan commented on a change in pull request #28649:
URL: https://github.com/apache/spark/pull/28649#discussion_r459867319
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##
@@ -166,6 +175,17 @@ case class InsertIntoHiveTable(
if (isDynamic.init.zip(isDynamic.tail).contains((true, false))) {
throw new AnalysisException(ErrorMsg.PARTITION_DYN_STA_ORDER.getMsg)
}
+} else if (numStaticPartitions > 0) {
+ // scalastyle:off
+ // ifNotExists is only valid with static partition, refer to
+ //
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-InsertingdataintoHiveTablesfromqueries
+ // scalastyle:on
+ if (!oldPart.isEmpty && ifPartitionNotExists) {
+val partitionStr =
partitionSpec.map(_.productIterator.mkString("=")).mkString("/")
+logWarning(s"Partition '$partitionStr' already exist, skip running job
to overwrite " +
Review comment:
do we have this warning message before?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28649: [SPARK-31829] Check for partition existence before computation
cloud-fan commented on a change in pull request #28649:
URL: https://github.com/apache/spark/pull/28649#discussion_r459866396
##
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
##
@@ -148,6 +148,15 @@ case class InsertIntoHiveTable(
|Table partitions:
${table.partitionColumnNames.mkString(",")}""".stripMargin)
}
+val oldPart = if ( numDynamicPartitions == 0) {
Review comment:
`numDynamicPartitions == 0 && numStaticPartitions > 0`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29214: [SPARK-31525] [SQL] return an empty list for df.head() when df is empty
AmplabJenkins removed a comment on pull request #29214: URL: https://github.com/apache/spark/pull/29214#issuecomment-663360243 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29214: [SPARK-31525] [SQL] return an empty list for df.head() when df is empty
AmplabJenkins commented on pull request #29214: URL: https://github.com/apache/spark/pull/29214#issuecomment-663360243 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29219: Revert "[SPARK-32253][INFRA] Show errors only for the sbt tests of github actions"
AmplabJenkins removed a comment on pull request #29219: URL: https://github.com/apache/spark/pull/29219#issuecomment-663349908 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29219: Revert "[SPARK-32253][INFRA] Show errors only for the sbt tests of github actions"
SparkQA commented on pull request #29219: URL: https://github.com/apache/spark/pull/29219#issuecomment-663360249 **[Test build #126473 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126473/testReport)** for PR 29219 at commit [`32191ee`](https://github.com/apache/spark/commit/32191ee2327939a2352d65cae3bf1b01bd512fc6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
AmplabJenkins removed a comment on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663360149 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
AmplabJenkins commented on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663360149 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29214: [SPARK-31525] [SQL] return an empty list for df.head() when df is empty
HyukjinKwon commented on a change in pull request #29214: URL: https://github.com/apache/spark/pull/29214#discussion_r459864185 ## File path: python/pyspark/sql/dataframe.py ## @@ -1332,7 +1333,7 @@ def head(self, n=None): """ if n is None: rs = self.head(1) -return rs[0] if rs else None +return rs[0] if rs else [] Review comment: Can you update the migration guide at https://github.com/apache/spark/blob/master/docs/pyspark-migration-guide.md? You can create a new title "Upgrading from PySpark 3.0 to 3.1" This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
SparkQA removed a comment on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663308757 **[Test build #126451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126451/testReport)** for PR 29211 at commit [`fea00dd`](https://github.com/apache/spark/commit/fea00dde227d14bc312f0c0a7733c50def1abf6a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29214: [SPARK-31525] [SQL] return an empty list for df.head() when df is empty
AmplabJenkins removed a comment on pull request #29214: URL: https://github.com/apache/spark/pull/29214#issuecomment-663318988 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29214: [SPARK-31525] [SQL] return an empty list for df.head() when df is empty
HyukjinKwon commented on pull request #29214: URL: https://github.com/apache/spark/pull/29214#issuecomment-663359114 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
SparkQA commented on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663358960 **[Test build #126451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126451/testReport)** for PR 29211 at commit [`fea00dd`](https://github.com/apache/spark/commit/fea00dde227d14bc312f0c0a7733c50def1abf6a). * This patch passes all tests. * This patch **does not merge cleanly**. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29212: [SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test
AmplabJenkins removed a comment on pull request #29212: URL: https://github.com/apache/spark/pull/29212#issuecomment-663358452 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #29188: [SPARK-32179][SPARK-32188][PYTHON][DOCS] Replace and redesign the documentation base
HyukjinKwon edited a comment on pull request #29188: URL: https://github.com/apache/spark/pull/29188#issuecomment-662966732 I believe this is ready for a look or possibly ready to go. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29218: [SPARK-32408][BUILD] Enable crossPaths back to prevent side effects
AmplabJenkins removed a comment on pull request #29218: URL: https://github.com/apache/spark/pull/29218#issuecomment-663358449 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29218: [SPARK-32408][BUILD] Enable crossPaths back to prevent side effects
HyukjinKwon commented on pull request #29218: URL: https://github.com/apache/spark/pull/29218#issuecomment-663358425 I reverted 5e79db6 here and get it back again to show full logs about JUnit tests. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29218: [SPARK-32408][BUILD] Enable crossPaths back to prevent side effects
HyukjinKwon commented on pull request #29218: URL: https://github.com/apache/spark/pull/29218#issuecomment-663358243 I reverted 5e79db6 here and get it back again to confirm IntegratedUDFTestUtils and pandas related tests are not being skipped. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29218: [SPARK-32408][BUILD] Enable crossPaths back to prevent side effects
AmplabJenkins commented on pull request #29218: URL: https://github.com/apache/spark/pull/29218#issuecomment-663358449 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29212: [SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test
AmplabJenkins commented on pull request #29212: URL: https://github.com/apache/spark/pull/29212#issuecomment-663358452 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on pull request #29218: [SPARK-32408][BUILD] Enable crossPaths back to prevent side effects
HyukjinKwon removed a comment on pull request #29218: URL: https://github.com/apache/spark/pull/29218#issuecomment-663358243 I reverted 5e79db6 here and get it back again to confirm IntegratedUDFTestUtils and pandas related tests are not being skipped. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29212: [SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test
HyukjinKwon commented on pull request #29212: URL: https://github.com/apache/spark/pull/29212#issuecomment-663357627 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29217: [SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils
HyukjinKwon commented on pull request #29217: URL: https://github.com/apache/spark/pull/29217#issuecomment-663357397 It is now tested properly: https://github.com/apache/spark/runs/905415145 ``` ... [info] - udf/udf-inner-join.sql - Scalar Pandas UDF (527 milliseconds) ... [info] - udf/udf-special-values.sql - Scalar Pandas UDF (747 milliseconds) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #29152: [SPARK-32356][SQL] Forbid create view with null type
ulysses-you commented on pull request #29152: URL: https://github.com/apache/spark/pull/29152#issuecomment-663355125 @cloud-fan ok may be tomorrow. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] agrawaldevesh commented on a change in pull request #29014: [SPARK-32199][SPARK-32198] Reduce job failures during decommissioning
agrawaldevesh commented on a change in pull request #29014:
URL: https://github.com/apache/spark/pull/29014#discussion_r459859455
##
File path:
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
##
@@ -939,12 +941,40 @@ private[spark] class TaskSchedulerImpl(
override def executorDecommission(
executorId: String, decommissionInfo: ExecutorDecommissionInfo): Unit = {
+synchronized {
+ // The scheduler can get multiple decommission updates from multiple
sources,
+ // and some of those can have isHostDecommissioned false. We merge them
such that
+ // if we heard isHostDecommissioned ever true, then we keep that one
since it is
+ // most likely coming from the cluster manager and thus authoritative
+ val oldDecomInfo = executorsPendingDecommission.get(executorId)
+ if (oldDecomInfo.isEmpty || !oldDecomInfo.get.isHostDecommissioned) {
+executorsPendingDecommission(executorId) = decommissionInfo
+ }
+}
rootPool.executorDecommission(executorId)
backend.reviveOffers()
}
- override def executorLost(executorId: String, reason: ExecutorLossReason):
Unit = {
+ override def getExecutorDecommissionInfo(executorId: String)
+: Option[ExecutorDecommissionInfo] = synchronized {
+ executorsPendingDecommission.get(executorId)
+ }
+
+ override def executorLost(executorId: String, givenReason:
ExecutorLossReason): Unit = {
var failedExecutor: Option[String] = None
+val reason = givenReason match {
+ // Handle executor process loss due to decommissioning
+ case ExecutorProcessLost(message, workerLost, causedByApp) =>
+val executorDecommissionInfo = getExecutorDecommissionInfo(executorId)
+ExecutorProcessLost(
+ message,
+ // Also mark the worker lost if we know that the host was
decommissioned
+ workerLost ||
executorDecommissionInfo.exists(_.isHostDecommissioned),
Review comment:
Indeed that can happen:
First is when a regular executor loss happens without a decommissioning. In
that case, we don't want this logic to kick in.
Second is a race such that a decommissioning happens and somehow the
`DecommissionExecutor` is delayed and it is only processed after the executor
looses heartbeat.
In any case, this whole stuff is kind of best effort: We cannot avoid like
_all_ of the races but as long as we prevent job failures in the common case of
decommissioning, we are okay.
But I totally can see your concern about the executorsPendingDecommission
growing and the executors not being removed from it. I have fixed the code to
handle that situation by not adding anything to executorsPendingDecommission if
the removeExecutor has been called.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
AmplabJenkins removed a comment on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663353438 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126453/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29217: [SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils
SparkQA commented on pull request #29217: URL: https://github.com/apache/spark/pull/29217#issuecomment-663353623 **[Test build #126472 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126472/testReport)** for PR 29217 at commit [`d72c4c1`](https://github.com/apache/spark/commit/d72c4c12bf61aab4dfd93824cef801d7d93e7a4c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wankunde commented on pull request #28944: [SPARK-32128][SQL]import SQLConf.PARTITION_OVERWRITE_VERIFY_PATH config
wankunde commented on pull request #28944: URL: https://github.com/apache/spark/pull/28944#issuecomment-663353487 @dongjoon-hyun @srowen @Ngone51 Could someone help me review this PR? thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
AmplabJenkins removed a comment on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663353436 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
AmplabJenkins commented on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663353436 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
AmplabJenkins removed a comment on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663352393 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
SparkQA commented on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663352431 **[Test build #126453 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126453/testReport)** for PR 29211 at commit [`12e9e23`](https://github.com/apache/spark/commit/12e9e239eec1691095fecf95066ea169745d3eda). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
SparkQA removed a comment on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663310261 **[Test build #126453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126453/testReport)** for PR 29211 at commit [`12e9e23`](https://github.com/apache/spark/commit/12e9e239eec1691095fecf95066ea169745d3eda). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29152: [SPARK-32356][SQL] Forbid create view with null type
SparkQA commented on pull request #29152: URL: https://github.com/apache/spark/pull/29152#issuecomment-663352522 **[Test build #126471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126471/testReport)** for PR 29152 at commit [`d02986c`](https://github.com/apache/spark/commit/d02986cffe43150d6184915eeec1741c51eace83). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
AmplabJenkins commented on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663352393 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
SparkQA removed a comment on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663311804 **[Test build #126456 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126456/testReport)** for PR 29211 at commit [`46fb2f7`](https://github.com/apache/spark/commit/46fb2f7f94738b407fdc17c9b65d73321d7a5085). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
SparkQA commented on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663351645 **[Test build #126456 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126456/testReport)** for PR 29211 at commit [`46fb2f7`](https://github.com/apache/spark/commit/46fb2f7f94738b407fdc17c9b65d73321d7a5085). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 closed pull request #29127: [SPARK-32327][SQL] SHOW TBLPROPERTIES is not allowed on temporary views
imback82 closed pull request #29127: URL: https://github.com/apache/spark/pull/29127 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #29127: [SPARK-32327][SQL] SHOW TBLPROPERTIES is not allowed on temporary views
imback82 commented on pull request #29127: URL: https://github.com/apache/spark/pull/29127#issuecomment-663351139 Yes, I agree. I will close this for now. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on pull request #29212: [SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test
HyukjinKwon removed a comment on pull request #29212: URL: https://github.com/apache/spark/pull/29212#issuecomment-663344096 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29218: [SPARK-32408][BUILD] Enable crossPaths back to prevent side effects
AmplabJenkins removed a comment on pull request #29218: URL: https://github.com/apache/spark/pull/29218#issuecomment-663342691 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29213: [DO-NOT-MERGE] Show Java tests not being ran
AmplabJenkins removed a comment on pull request #29213: URL: https://github.com/apache/spark/pull/29213#issuecomment-663349864 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126452/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29193: [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor
AmplabJenkins removed a comment on pull request #29193: URL: https://github.com/apache/spark/pull/29193#issuecomment-663349786 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126454/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29218: [SPARK-32408][BUILD] Enable crossPaths back to prevent side effects
SparkQA commented on pull request #29218: URL: https://github.com/apache/spark/pull/29218#issuecomment-663349971 **[Test build #126470 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126470/testReport)** for PR 29218 at commit [`7ab3acb`](https://github.com/apache/spark/commit/7ab3acbcd70051dd7a9f1f360c7db8b635f426f7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29212: [SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test
SparkQA commented on pull request #29212: URL: https://github.com/apache/spark/pull/29212#issuecomment-663349930 **[Test build #126469 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126469/testReport)** for PR 29212 at commit [`71193f1`](https://github.com/apache/spark/commit/71193f194758de23794a2bc46d16766e334091d2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29213: [DO-NOT-MERGE] Show Java tests not being ran
AmplabJenkins removed a comment on pull request #29213: URL: https://github.com/apache/spark/pull/29213#issuecomment-663349861 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29219: Revert "[SPARK-32253][INFRA] Show errors only for the sbt tests of github actions"
AmplabJenkins commented on pull request #29219: URL: https://github.com/apache/spark/pull/29219#issuecomment-663349908 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29193: [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost
AmplabJenkins commented on pull request #29193: URL: https://github.com/apache/spark/pull/29193#issuecomment-663349782 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29193: [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor
AmplabJenkins removed a comment on pull request #29193: URL: https://github.com/apache/spark/pull/29193#issuecomment-663349782 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29213: [DO-NOT-MERGE] Show Java tests not being ran
AmplabJenkins commented on pull request #29213: URL: https://github.com/apache/spark/pull/29213#issuecomment-663349861 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29193: [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lo
SparkQA removed a comment on pull request #29193: URL: https://github.com/apache/spark/pull/29193#issuecomment-663310336 **[Test build #126454 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126454/testReport)** for PR 29193 at commit [`e54f221`](https://github.com/apache/spark/commit/e54f22186a1f82ac9704f6644e057dadfc285b86). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29193: [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost
SparkQA commented on pull request #29193: URL: https://github.com/apache/spark/pull/29193#issuecomment-663349299 **[Test build #126454 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126454/testReport)** for PR 29193 at commit [`e54f221`](https://github.com/apache/spark/commit/e54f22186a1f82ac9704f6644e057dadfc285b86). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29213: [DO-NOT-MERGE] Show Java tests not being ran
SparkQA removed a comment on pull request #29213: URL: https://github.com/apache/spark/pull/29213#issuecomment-663310258 **[Test build #126452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126452/testReport)** for PR 29213 at commit [`fccf9d2`](https://github.com/apache/spark/commit/fccf9d2940142c04d5a3d24aeb7b0b2bc0c56d2d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang opened a new pull request #29219: Revert "[SPARK-32253][INFRA] Show errors only for the sbt tests of gi…
gengliangwang opened a new pull request #29219: URL: https://github.com/apache/spark/pull/29219 …thub actions" This reverts commit 026b0b926dfd40038f2cee932f38b917eb25b77e. As @HyukjinKwon pointed out in https://github.com/apache/spark/pull/29133#issuecomment-663339240, there is no JUnit test report after https://github.com/apache/spark/pull/29133. Let's revert https://github.com/apache/spark/pull/29133 for now and find a better solution to improve the log output later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29127: [SPARK-32327][SQL] SHOW TBLPROPERTIES is not allowed on temporary views
viirya commented on pull request #29127: URL: https://github.com/apache/spark/pull/29127#issuecomment-663349227 > Now I don't have a strong opinion. Users can't create temp view with properties and it seems fine if SHOW TBLPROPERTIES returns Nil for temp view. cc @maropu @viirya Sounds ok too. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29213: [DO-NOT-MERGE] Show Java tests not being ran
SparkQA commented on pull request #29213: URL: https://github.com/apache/spark/pull/29213#issuecomment-663349015 **[Test build #126452 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126452/testReport)** for PR 29213 at commit [`fccf9d2`](https://github.com/apache/spark/commit/fccf9d2940142c04d5a3d24aeb7b0b2bc0c56d2d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29107: [SPARK-32308][SQL] Move by-name resolution logic of unionByName from API code to analysis phase
viirya commented on pull request #29107: URL: https://github.com/apache/spark/pull/29107#issuecomment-663348711 Thanks for review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29217: [SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils
AmplabJenkins removed a comment on pull request #29217: URL: https://github.com/apache/spark/pull/29217#issuecomment-663348257 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29217: [SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils
AmplabJenkins commented on pull request #29217: URL: https://github.com/apache/spark/pull/29217#issuecomment-663348257 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] aniket486 commented on pull request #28649: [SPARK-31829] Check for partition existence before computation
aniket486 commented on pull request #28649: URL: https://github.com/apache/spark/pull/28649#issuecomment-663347876 Current implementation calls "[saveAsHiveFile](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L205)" which triggers job to run the "select ... " query only to discard the output [later](https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala#L274) (by not calling externalCatalog.loadPartition)". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29217: [SPARK-32422][SQL][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils
HyukjinKwon commented on pull request #29217: URL: https://github.com/apache/spark/pull/29217#issuecomment-663347022 I reverted 826689a36efbf1b73a0744824080ac01da3a5f60 here and get it back again to confirm `IntegratedUDFTestUtils` and pandas related tests are not being skipped. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29212: [SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test
AmplabJenkins removed a comment on pull request #29212: URL: https://github.com/apache/spark/pull/29212#issuecomment-663345344 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29212: [SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test
AmplabJenkins commented on pull request #29212: URL: https://github.com/apache/spark/pull/29212#issuecomment-663345344 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29198: [SPARK-32401][SQL] Migrate function related commands to new resolution framework
cloud-fan commented on a change in pull request #29198:
URL: https://github.com/apache/spark/pull/29198#discussion_r459850052
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
##
@@ -3650,12 +3652,24 @@ class AstBuilder(conf: SQLConf) extends
SqlBaseBaseVisitor[AnyRef] with Logging
}
}
-val functionIdentifier = visitMultipartIdentifier(ctx.multipartIdentifier)
-CreateFunctionStatement(
- functionIdentifier,
+val nameParts = visitMultipartIdentifier(ctx.multipartIdentifier)
+val isTemp = ctx.TEMPORARY != null
+val func: LogicalPlan = if (isTemp) {
+ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+ // temp func doesn't belong to any catalog and we shouldn't resolve
catalog in the name.
+ if (nameParts.length > 2) {
+throw new AnalysisException(s"Unsupported function name
'${nameParts.quoted}'")
+ }
+ ResolvedFunc(nameParts.asIdentifier)
+} else {
+ UnresolvedFunc(nameParts)
+}
+
+CreateFunction(
+ func,
Review comment:
I'm not sure how to deal with CREATE commands yet. For now, CREATE TABLE
and VIEW still use statement plans. We don't need to do lookup for CREATE
commands, so `UnresolvedFunnc` looks weird here.
Can we keep it unchanged?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
AmplabJenkins removed a comment on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663344241 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126457/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29133: [SPARK-32253][INFRA] Show errors only for the sbt tests of github actions
HyukjinKwon commented on pull request #29133: URL: https://github.com/apache/spark/pull/29133#issuecomment-663344407 Thank you @gengliangwang for investigating this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
AmplabJenkins removed a comment on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663344237 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29217: [SPARK-32422][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils
AmplabJenkins removed a comment on pull request #29217: URL: https://github.com/apache/spark/pull/29217#issuecomment-66333 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
AmplabJenkins commented on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663344237 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29217: [SPARK-32422][TESTS] Use python3 executable instead of python3.6 in IntegratedUDFTestUtils
SparkQA commented on pull request #29217: URL: https://github.com/apache/spark/pull/29217#issuecomment-663344307 **[Test build #126468 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126468/testReport)** for PR 29217 at commit [`c70e29f`](https://github.com/apache/spark/commit/c70e29f83dd1a0deac28653586ed90876a8a9108). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29212: [SPARK-32419][PYTHON][BUILD] Avoid using subshell for Conda env (de)activation in pip packaging test
HyukjinKwon commented on pull request #29212: URL: https://github.com/apache/spark/pull/29212#issuecomment-663344096 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
SparkQA removed a comment on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663313246 **[Test build #126457 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126457/testReport)** for PR 29211 at commit [`5678d28`](https://github.com/apache/spark/commit/5678d284be067b30395606b1bacf5409a79aad53). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29198: [SPARK-32401][SQL] Migrate function related commands to new resolution framework
cloud-fan commented on a change in pull request #29198: URL: https://github.com/apache/spark/pull/29198#discussion_r459849015 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -3610,8 +3610,10 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging case Some(x) => throw new ParseException(s"SHOW $x FUNCTIONS not supported", ctx) } val pattern = Option(ctx.pattern).map(string(_)) -val functionName = Option(ctx.multipartIdentifier).map(visitMultipartIdentifier) -ShowFunctionsStatement(userScope, systemScope, pattern, functionName) +val functionName = Option(ctx.multipartIdentifier) Review comment: or keep it `functionName`, and do ``` ShowFunctions(UnresolvedFunc(functionName), userScope, systemScope, pattern) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29211: [SPARK-31197][CORE] Shutdown executor once we are done decommissioning
SparkQA commented on pull request #29211: URL: https://github.com/apache/spark/pull/29211#issuecomment-663343919 **[Test build #126457 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/126457/testReport)** for PR 29211 at commit [`5678d28`](https://github.com/apache/spark/commit/5678d284be067b30395606b1bacf5409a79aad53). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29198: [SPARK-32401][SQL] Migrate function related commands to new resolution framework
cloud-fan commented on a change in pull request #29198: URL: https://github.com/apache/spark/pull/29198#discussion_r459848829 ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -3610,8 +3610,10 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging case Some(x) => throw new ParseException(s"SHOW $x FUNCTIONS not supported", ctx) } val pattern = Option(ctx.pattern).map(string(_)) -val functionName = Option(ctx.multipartIdentifier).map(visitMultipartIdentifier) -ShowFunctionsStatement(userScope, systemScope, pattern, functionName) +val functionName = Option(ctx.multipartIdentifier) Review comment: nit: maybe `unresolvedFun`? it's not a name anymore. ## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala ## @@ -3610,8 +3610,10 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging case Some(x) => throw new ParseException(s"SHOW $x FUNCTIONS not supported", ctx) } val pattern = Option(ctx.pattern).map(string(_)) -val functionName = Option(ctx.multipartIdentifier).map(visitMultipartIdentifier) -ShowFunctionsStatement(userScope, systemScope, pattern, functionName) +val functionName = Option(ctx.multipartIdentifier) Review comment: nit: maybe `unresolvedFunc`? it's not a name anymore. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29198: [SPARK-32401][SQL] Migrate function related commands to new resolution framework
cloud-fan commented on a change in pull request #29198:
URL: https://github.com/apache/spark/pull/29198#discussion_r459848587
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -1894,7 +1894,7 @@ class Analyzer(
def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp {
// Resolve functions with concrete relations from v2 catalog.
case UnresolvedFunc(multipartIdent) =>
-val funcIdent = parseSessionCatalogFunctionIdentifier(multipartIdent,
"function lookup")
+val funcIdent = parseSessionCatalogFunctionIdentifier(multipartIdent)
Review comment:
If this method is only used here, can we define it here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
