[GitHub] [spark] SparkQA removed a comment on pull request #30488: [SPARK-33071][SPARK-33536][SQL] Avoid changing dataset_id of LogicalPlan in join() to not break DetectAmbiguousSelfJoin
SparkQA removed a comment on pull request #30488: URL: https://github.com/apache/spark/pull/30488#issuecomment-733450217 **[Test build #131728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131728/testReport)** for PR 30488 at commit [`05bca19`](https://github.com/apache/spark/commit/05bca194ca6bcac9d328b686e5d600a7a62b5ac9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30488: [SPARK-33071][SPARK-33536][SQL] Avoid changing dataset_id of LogicalPlan in join() to not break DetectAmbiguousSelfJoin
SparkQA commented on pull request #30488: URL: https://github.com/apache/spark/pull/30488#issuecomment-733533102 **[Test build #131728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131728/testReport)** for PR 30488 at commit [`05bca19`](https://github.com/apache/spark/commit/05bca194ca6bcac9d328b686e5d600a7a62b5ac9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #30478: [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2
wangyum commented on pull request #30478: URL: https://github.com/apache/spark/pull/30478#issuecomment-733527228 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30440: [SPARK-33496][SQL]Improve error message of ANSI explicit cast
HyukjinKwon commented on pull request #30440: URL: https://github.com/apache/spark/pull/30440#issuecomment-733526016 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #30408: [SPARK-33477][SQL] Hive Metastore support filter by date type
HyukjinKwon closed pull request #30408: URL: https://github.com/apache/spark/pull/30408 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30408: [SPARK-33477][SQL] Hive Metastore support filter by date type
HyukjinKwon commented on pull request #30408: URL: https://github.com/apache/spark/pull/30408#issuecomment-733525530 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value
SparkQA removed a comment on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-733433490 **[Test build #131725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131725/testReport)** for PR 30421 at commit [`98986a0`](https://github.com/apache/spark/commit/98986a07f25f15e7e0d16283746ea28f18628c5a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30421: [SPARK-33474][SQL] Support TypeConstructed partition spec value
SparkQA commented on pull request #30421: URL: https://github.com/apache/spark/pull/30421#issuecomment-733520649 **[Test build #131725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131725/testReport)** for PR 30421 at commit [`98986a0`](https://github.com/apache/spark/commit/98986a07f25f15e7e0d16283746ea28f18628c5a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30478: [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2
SparkQA removed a comment on pull request #30478: URL: https://github.com/apache/spark/pull/30478#issuecomment-733467470 **[Test build #131734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131734/testReport)** for PR 30478 at commit [`43d90ca`](https://github.com/apache/spark/commit/43d90cafaf0aa4c8c4a355070d5c71008f6f3ea9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30478: [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2
SparkQA commented on pull request #30478: URL: https://github.com/apache/spark/pull/30478#issuecomment-733520238 **[Test build #131734 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131734/testReport)** for PR 30478 at commit [`43d90ca`](https://github.com/apache/spark/commit/43d90cafaf0aa4c8c4a355070d5c71008f6f3ea9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30408: [SPARK-33477][SQL] Hive Metastore support filter by date type
SparkQA removed a comment on pull request #30408: URL: https://github.com/apache/spark/pull/30408#issuecomment-733471851 **[Test build #131740 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131740/testReport)** for PR 30408 at commit [`29c489a`](https://github.com/apache/spark/commit/29c489ad5f753aaa3551489655073c9f6fc7b0c6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30242: [SPARK-33277][PYSPARK][SQL] Use ContextAwareIterator to stop consuming after the task ends.
SparkQA removed a comment on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-733415179 **[Test build #131717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131717/testReport)** for PR 30242 at commit [`e2cc227`](https://github.com/apache/spark/commit/e2cc2279b9677f4a17d701fca19f1a1b5a9e8037). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30412: [SPARK-33480][SQL] Support char/varchar type
SparkQA removed a comment on pull request #30412: URL: https://github.com/apache/spark/pull/30412#issuecomment-733415048 **[Test build #131716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131716/testReport)** for PR 30412 at commit [`38999b5`](https://github.com/apache/spark/commit/38999b535e78817d2647d186605618438f438220). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
SparkQA removed a comment on pull request #30484: URL: https://github.com/apache/spark/pull/30484#issuecomment-733433369 **[Test build #131724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131724/testReport)** for PR 30484 at commit [`6cbd321`](https://github.com/apache/spark/commit/6cbd32172abf384c5d8a05d09b256fcc78e6e33f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30475: [SPARK-33522][SQL] Improve exception messages while handling UnresolvedTableOrView
SparkQA removed a comment on pull request #30475: URL: https://github.com/apache/spark/pull/30475#issuecomment-733512605 **[Test build #131748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131748/testReport)** for PR 30475 at commit [`68ee277`](https://github.com/apache/spark/commit/68ee277cbde9ecb466b1480af676a2f831e11236). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29066: [SPARK-23889][SQL] DataSourceV2: required sorting and clustering for writes
SparkQA removed a comment on pull request #29066: URL: https://github.com/apache/spark/pull/29066#issuecomment-733418271 **[Test build #131719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131719/testReport)** for PR 29066 at commit [`d47817d`](https://github.com/apache/spark/commit/d47817ddb185041d49af84131a53c33877ec00e4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30486: [SPARK-33530][CORE] Support --archives and spark.archives option natively
HyukjinKwon commented on pull request #30486: URL: https://github.com/apache/spark/pull/30486#issuecomment-733519983 Thank you @dongjoon-hyun! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30486: [SPARK-33530][CORE] Support --archives and spark.archives option natively
dongjoon-hyun commented on pull request #30486: URL: https://github.com/apache/spark/pull/30486#issuecomment-733519344 +1, LGTM (Pending CI) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30408: [SPARK-33477][SQL] Hive Metastore support filter by date type
SparkQA commented on pull request #30408: URL: https://github.com/apache/spark/pull/30408#issuecomment-733516303 **[Test build #131751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131751/testReport)** for PR 30408 at commit [`29c489a`](https://github.com/apache/spark/commit/29c489ad5f753aaa3551489655073c9f6fc7b0c6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30475: [SPARK-33522][SQL] Improve exception messages while handling UnresolvedTableOrView
AmplabJenkins commented on pull request #30475: URL: https://github.com/apache/spark/pull/30475#issuecomment-733515623 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #30387: [SPARK-33443][SQL] LEAD/LAG should support [ IGNORE NULLS | RESPECT NULLS ]
beliefer commented on a change in pull request #30387:
URL: https://github.com/apache/spark/pull/30387#discussion_r530151304
##
File path: sql/core/src/main/scala/org/apache/spark/sql/functions.scala
##
@@ -938,8 +938,40 @@ object functions {
* @group window_funcs
* @since 1.4.0
*/
- def lag(e: Column, offset: Int, defaultValue: Any): Column = withExpr {
-Lag(e.expr, Literal(offset), Literal(defaultValue))
+ def lag(e: Column, offset: Int, defaultValue: Any): Column = {
+lag(e, offset, defaultValue, false)
+ }
+
+ /**
+ * Window function: returns the value that is `offset` rows before the
current row, and
+ * `defaultValue` if there is less than `offset` rows before the current
row. `ignoreNulls`
+ * determines whether null values of row are included in or eliminated from
the calculation.
+ * For example, an `offset` of one will return the previous row at any given
point in the
+ * window partition.
+ *
+ * This is equivalent to the LAG function in SQL.
+ *
+ * @group window_funcs
+ * @since 3.1.0
+ */
+ def lag(columnName: String, offset: Int, defaultValue: Any, ignoreNulls:
Boolean): Column = {
Review comment:
cc @HyukjinKwon too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30475: [SPARK-33522][SQL] Improve exception messages while handling UnresolvedTableOrView
SparkQA commented on pull request #30475: URL: https://github.com/apache/spark/pull/30475#issuecomment-733515605 **[Test build #131748 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131748/testReport)** for PR 30475 at commit [`68ee277`](https://github.com/apache/spark/commit/68ee277cbde9ecb466b1480af676a2f831e11236). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class TruncateTable(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29066: [SPARK-23889][SQL] DataSourceV2: required sorting and clustering for writes
AmplabJenkins commented on pull request #29066: URL: https://github.com/apache/spark/pull/29066#issuecomment-733515306 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #30408: [SPARK-33477][SQL] Hive Metastore support filter by date type
wangyum commented on pull request #30408: URL: https://github.com/apache/spark/pull/30408#issuecomment-733515064 retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29066: [SPARK-23889][SQL] DataSourceV2: required sorting and clustering for writes
SparkQA commented on pull request #29066: URL: https://github.com/apache/spark/pull/29066#issuecomment-733514734 **[Test build #131719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131719/testReport)** for PR 29066 at commit [`d47817d`](https://github.com/apache/spark/commit/d47817ddb185041d49af84131a53c33877ec00e4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public class Distributions ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30478: [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2
AmplabJenkins commented on pull request #30478: URL: https://github.com/apache/spark/pull/30478#issuecomment-733513051 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30289: [SPARK-33141][SQL] Capture SQL configs when creating permanent views
SparkQA commented on pull request #30289: URL: https://github.com/apache/spark/pull/30289#issuecomment-733512958 **[Test build #131749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131749/testReport)** for PR 30289 at commit [`f44e71b`](https://github.com/apache/spark/commit/f44e71b4bbb748f4c7503299fa420d180c56ccaf). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29893: [SPARK-32976][SQL]Support column list in INSERT statement
SparkQA commented on pull request #29893: URL: https://github.com/apache/spark/pull/29893#issuecomment-733512932 **[Test build #131750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131750/testReport)** for PR 29893 at commit [`475e790`](https://github.com/apache/spark/commit/475e790f7eeae80c4123f93c74952eb24f056897). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30479: [WIP][SPARK-33527][SQL] Extend the function of decode so as consistent with mainstream databases
SparkQA commented on pull request #30479: URL: https://github.com/apache/spark/pull/30479#issuecomment-733512705 **[Test build #131747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131747/testReport)** for PR 30479 at commit [`1cf326b`](https://github.com/apache/spark/commit/1cf326bb88c8b743ef629fe6bf062de299ebae9b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30475: [SPARK-33522][SQL] Improve exception messages while handling UnresolvedTableOrView
SparkQA commented on pull request #30475: URL: https://github.com/apache/spark/pull/30475#issuecomment-733512605 **[Test build #131748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131748/testReport)** for PR 30475 at commit [`68ee277`](https://github.com/apache/spark/commit/68ee277cbde9ecb466b1480af676a2f831e11236). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30242: [SPARK-33277][PYSPARK][SQL] Use ContextAwareIterator to stop consuming after the task ends.
AmplabJenkins commented on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-733511510 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
AmplabJenkins commented on pull request #30484: URL: https://github.com/apache/spark/pull/30484#issuecomment-733510794 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30412: [SPARK-33480][SQL] Support char/varchar type
AmplabJenkins commented on pull request #30412: URL: https://github.com/apache/spark/pull/30412#issuecomment-733510497 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30486: [SPARK-33530][CORE] Support --archives and spark.archives option natively
AmplabJenkins commented on pull request #30486: URL: https://github.com/apache/spark/pull/30486#issuecomment-733510493 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30408: [SPARK-33477][SQL] Hive Metastore support filter by date type
AmplabJenkins commented on pull request #30408: URL: https://github.com/apache/spark/pull/30408#issuecomment-733510488 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29066: [SPARK-23889][SQL] DataSourceV2: required sorting and clustering for writes
AmplabJenkins commented on pull request #29066: URL: https://github.com/apache/spark/pull/29066#issuecomment-733510503 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30242: [SPARK-33277][PYSPARK][SQL] Use ContextAwareIterator to stop consuming after the task ends.
AmplabJenkins commented on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-733510490 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30440: [SPARK-33496][SQL]Improve error message of ANSI explicit cast
AmplabJenkins commented on pull request #30440: URL: https://github.com/apache/spark/pull/30440#issuecomment-733510495 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30242: [SPARK-33277][PYSPARK][SQL] Use ContextAwareIterator to stop consuming after the task ends.
SparkQA commented on pull request #30242: URL: https://github.com/apache/spark/pull/30242#issuecomment-733510321 **[Test build #131717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131717/testReport)** for PR 30242 at commit [`e2cc227`](https://github.com/apache/spark/commit/e2cc2279b9677f4a17d701fca19f1a1b5a9e8037). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30484: [SPARK-33532][SQL] Remove unreachable branch in SpecificParquetRecordReaderBase.initialize method
SparkQA commented on pull request #30484: URL: https://github.com/apache/spark/pull/30484#issuecomment-733509871 **[Test build #131724 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131724/testReport)** for PR 30484 at commit [`6cbd321`](https://github.com/apache/spark/commit/6cbd32172abf384c5d8a05d09b256fcc78e6e33f). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] luluorta commented on pull request #30495: [SPARK-33548][WEBUI] display the jvm peak memory usage on the executor ui
luluorta commented on pull request #30495: URL: https://github.com/apache/spark/pull/30495#issuecomment-733508470 cc @cloud-fan Please review this patch, many thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30408: [SPARK-33477][SQL] Hive Metastore support filter by date type
SparkQA commented on pull request #30408: URL: https://github.com/apache/spark/pull/30408#issuecomment-733506446 **[Test build #131740 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131740/testReport)** for PR 30408 at commit [`29c489a`](https://github.com/apache/spark/commit/29c489ad5f753aaa3551489655073c9f6fc7b0c6). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30486: [SPARK-33530][CORE] Support --archives and spark.archives option natively
HyukjinKwon commented on pull request #30486: URL: https://github.com/apache/spark/pull/30486#issuecomment-733506294 @mcg1969 too FYI [conda-pack](https://conda.github.io/conda-pack/). With this change, users can use conda-pack in other cluster modes not only Yarn. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30412: [SPARK-33480][SQL] Support char/varchar type
SparkQA commented on pull request #30412: URL: https://github.com/apache/spark/pull/30412#issuecomment-733505877 **[Test build #131716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131716/testReport)** for PR 30412 at commit [`38999b5`](https://github.com/apache/spark/commit/38999b535e78817d2647d186605618438f438220). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on pull request #30475: [SPARK-33522][SQL] Improve exception messages while handling UnresolvedTableOrView
imback82 commented on pull request #30475: URL: https://github.com/apache/spark/pull/30475#issuecomment-733505521 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #30493: [SPARK-33549][SQL] Remove configuration spark.sql.legacy.allowCastNumericToTimestamp
gengliangwang commented on a change in pull request #30493:
URL: https://github.com/apache/spark/pull/30493#discussion_r530140422
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##
@@ -1311,20 +1311,6 @@ class CastSuite extends CastSuiteBase {
}
}
- test("SPARK-31710: fail casting from numeric to timestamp if it is
forbidden") {
Review comment:
There is a similar test case in
https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala#L844
already
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##
@@ -1311,20 +1311,6 @@ class CastSuite extends CastSuiteBase {
}
}
- test("SPARK-31710: fail casting from numeric to timestamp if it is
forbidden") {
Review comment:
There is a similar test case in
https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala#L844
already
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn removed a comment on pull request #29893: [SPARK-32976][SQL]Support column list in INSERT statement
yaooqinn removed a comment on pull request #29893: URL: https://github.com/apache/spark/pull/29893#issuecomment-733501539 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #29893: [SPARK-32976][SQL]Support column list in INSERT statement
yaooqinn commented on pull request #29893: URL: https://github.com/apache/spark/pull/29893#issuecomment-733501529 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] luluorta commented on a change in pull request #30289: [SPARK-33141][SQL] Capture SQL configs when creating permanent views
luluorta commented on a change in pull request #30289:
URL: https://github.com/apache/spark/pull/30289#discussion_r530133325
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -361,11 +364,34 @@ object ViewHelper {
}
}
+ /**
+ * Convert the view query SQL configs in `properties`.
+ */
+ private def generateQuerySQLConfigs(conf: SQLConf): Map[String, String] = {
+val modifiedConfs = conf.getAllConfs.filter { case (k, _) =>
+ conf.isModifiable(k) && k != SQLConf.MAX_NESTED_VIEW_DEPTH.key
Review comment:
IMHO, we should keep the captured SQL configs as less as possible. It's
hard to precisely captures the configs which only affect the parser and
analyzer, an alternative way is just filtering out the configs that definitely
can NOT affect parsing/analyzing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] luluorta commented on pull request #30289: [SPARK-33141][SQL] Capture SQL configs when creating permanent views
luluorta commented on pull request #30289: URL: https://github.com/apache/spark/pull/30289#issuecomment-733500255 > can we copy-paste more context from JIRA tickets to the PR description? I added more background information to the PR description. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
mridulm commented on pull request #30450: URL: https://github.com/apache/spark/pull/30450#issuecomment-733500240 Given vacation time, I will leave this open so that @otterc @tgravescs and @HeartSaVioR have sufficient time to get to it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30450: [SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client
mridulm commented on a change in pull request #30450:
URL: https://github.com/apache/spark/pull/30450#discussion_r530136636
##
File path:
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala
##
@@ -230,6 +230,37 @@ class YarnClusterSuite extends BaseYarnClusterSuite {
}
}
+ test("running Spark in yarn-cluster mode displays driver log links") {
+val log4jConf = new File(tempDir, "log4j.properties")
+val logOutFile = new File(tempDir, "logs")
+Files.write(
+ s"""log4j.rootCategory=DEBUG,file
+ |log4j.appender.file=org.apache.log4j.FileAppender
+ |log4j.appender.file.file=$logOutFile
+ |log4j.appender.file.layout=org.apache.log4j.PatternLayout
+ |""".stripMargin,
+ log4jConf, StandardCharsets.UTF_8)
+// Since this test is trying to extract log output from the SparkSubmit
process itself,
+// standard options to the Spark process don't take effect. Leverage the
java-opts file which
+// will get picked up for the SparkSubmit process.
+val confDir = new File(tempDir, "conf")
+confDir.mkdir()
+val javaOptsFile = new File(confDir, "java-opts")
+Files.write(s"-Dlog4j.configuration=file://$log4jConf\n", javaOptsFile,
StandardCharsets.UTF_8)
+
+val result = File.createTempFile("result", null, tempDir)
+val finalState = runSpark(clientMode = false,
+ mainClassName(YarnClusterDriver.getClass),
+ appArgs = Seq(result.getAbsolutePath),
+ extraEnv = Map("SPARK_CONF_DIR" -> confDir.getAbsolutePath),
+ extraConf = Map("spark.yarn.includeDriverLogsLink" -> true.toString))
+checkResult(finalState, result)
+val logOutput = Files.toString(logOutFile, StandardCharsets.UTF_8)
+val logFilePattern = raw"""(?s).+\sDriver Logs \(\):
http://.+/\?start=-4096\s.+"""
Review comment:
nit: replace 4096 with regex for number ? Since we are not generating
the url anymore.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30493: [SPARK-33549][SQL] Remove configuration spark.sql.legacy.allowCastNumericToTimestamp
MaxGekk commented on a change in pull request #30493:
URL: https://github.com/apache/spark/pull/30493#discussion_r530136005
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuite.scala
##
@@ -1311,20 +1311,6 @@ class CastSuite extends CastSuiteBase {
}
}
- test("SPARK-31710: fail casting from numeric to timestamp if it is
forbidden") {
Review comment:
> Users just need to set spark.sql.ansi.enabled for the behavior.
Just in case, should we leave the test, and check the cases under ^^ the
config?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] luluorta commented on a change in pull request #30289: [SPARK-33141][SQL] Capture SQL configs when creating permanent views
luluorta commented on a change in pull request #30289:
URL: https://github.com/apache/spark/pull/30289#discussion_r530134316
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -361,11 +379,38 @@ object ViewHelper {
}
}
+ /**
+ * Convert the view query SQL configs in `properties`.
+ */
+ private def generateQuerySQLConfigs(conf: SQLConf): Map[String, String] = {
+val modifiedConfs = conf.getAllConfs.filter { case (k, _) =>
+ conf.isModifiable(k) && !isConfigBlacklisted(k)
+}
+val props = new mutable.HashMap[String, String]
+if (modifiedConfs.nonEmpty) {
+ val confJson = compact(render(JsonProtocol.mapToJson(modifiedConfs)))
+ props.put(VIEW_QUERY_SQL_CONFIGS, confJson)
Review comment:
Thanks for pointing this out. I changed to store one config per table
property entry, each with a "view.sqlConfig." prefix.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] luluorta commented on a change in pull request #30289: [SPARK-33141][SQL] Capture SQL configs when creating permanent views
luluorta commented on a change in pull request #30289:
URL: https://github.com/apache/spark/pull/30289#discussion_r530133390
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
##
@@ -414,6 +433,8 @@ object CatalogTable {
val VIEW_QUERY_OUTPUT_PREFIX = VIEW_PREFIX + "query.out."
val VIEW_QUERY_OUTPUT_NUM_COLUMNS = VIEW_QUERY_OUTPUT_PREFIX + "numCols"
val VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX = VIEW_QUERY_OUTPUT_PREFIX + "col."
+
+ val VIEW_QUERY_SQL_CONFIGS = VIEW_PREFIX + "query.sqlConfigs"
Review comment:
done
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1448,6 +1448,15 @@ object SQLConf {
"must be positive.")
.createWithDefault(100)
+ val APPLY_VIEW_SQL_CONFIGS =
+buildConf("spark.sql.legacy.view.applySQLConfigs")
Review comment:
done
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -334,6 +337,21 @@ case class ShowViewsCommand(
object ViewHelper {
+ private val configPrefixBlacklist = Seq(
Review comment:
done
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -334,6 +337,21 @@ case class ShowViewsCommand(
object ViewHelper {
+ private val configPrefixBlacklist = Seq(
+SQLConf.MAX_NESTED_VIEW_DEPTH.key,
+"spark.sql.optimizer.",
+"spark.sql.codegen.",
+"spark.sql.execution.",
+"spark.sql.shuffle.",
+"spark.sql.adaptive.")
+
+ private def isConfigBlacklisted(key: String): Boolean = {
Review comment:
done
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -380,16 +425,19 @@ object ViewHelper {
// for createViewCommand queryOutput may be different from fieldNames
val queryOutput = analyzedPlan.schema.fieldNames
+val conf = SQLConf.get
Review comment:
done
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -361,11 +379,38 @@ object ViewHelper {
}
}
+ /**
+ * Convert the view query SQL configs in `properties`.
+ */
+ private def generateQuerySQLConfigs(conf: SQLConf): Map[String, String] = {
Review comment:
done
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala
##
@@ -754,4 +754,77 @@ abstract class SQLViewSuite extends QueryTest with
SQLTestUtils {
}
}
}
+
+ test("SPARK-33141: view should be parsed and analyzed with configs set when
creating") {
+withTable("t33141") {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] luluorta commented on a change in pull request #30289: [SPARK-33141][SQL] Capture SQL configs when creating permanent views
luluorta commented on a change in pull request #30289:
URL: https://github.com/apache/spark/pull/30289#discussion_r530133325
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala
##
@@ -361,11 +364,34 @@ object ViewHelper {
}
}
+ /**
+ * Convert the view query SQL configs in `properties`.
+ */
+ private def generateQuerySQLConfigs(conf: SQLConf): Map[String, String] = {
+val modifiedConfs = conf.getAllConfs.filter { case (k, _) =>
+ conf.isModifiable(k) && k != SQLConf.MAX_NESTED_VIEW_DEPTH.key
Review comment:
IMHO, we should keep the captured SQL configs as less as possible.
Because it's hard to precisely captures the configs which only affect the
parser and analyzer, an alternative way is just filtering out the configs that
definitely can NOT affect parsing/analyzing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables
SparkQA removed a comment on pull request #30403: URL: https://github.com/apache/spark/pull/30403#issuecomment-733488765 **[Test build #131745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131745/testReport)** for PR 30403 at commit [`5e7227b`](https://github.com/apache/spark/commit/5e7227ba742cf6af1eb6fc9ce6a810959ab3b94f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables
AmplabJenkins removed a comment on pull request #30403: URL: https://github.com/apache/spark/pull/30403#issuecomment-733492330 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables
AmplabJenkins commented on pull request #30403: URL: https://github.com/apache/spark/pull/30403#issuecomment-733492330 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables
SparkQA commented on pull request #30403: URL: https://github.com/apache/spark/pull/30403#issuecomment-733492307 **[Test build #131745 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131745/testReport)** for PR 30403 at commit [`5e7227b`](https://github.com/apache/spark/commit/5e7227ba742cf6af1eb6fc9ce6a810959ab3b94f). * This patch **fails to build**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ShowColumns(` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733492363 It's a little weird because it passed locally. ``` $ build/sbt "core/testOnly *.FallbackStorageSuite" ... [info] FallbackStorageSuite: [info] - fallback storage APIs - copy/exists (957 milliseconds) [info] - migrate shuffle data to fallback storage (224 milliseconds) [info] - Upload from all decommissioned executors (5 seconds, 232 milliseconds) [info] - Upload multi stages (3 seconds, 67 milliseconds) [info] - Newly added executors should access old data from remote storage (8 seconds, 415 milliseconds) [info] ScalaTest [info] Run completed in 18 seconds, 975 milliseconds. [info] Total number of tests run: 5 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 5, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] luluorta commented on a change in pull request #30289: [SPARK-33141][SQL] Capture SQL configs when creating permanent views
luluorta commented on a change in pull request #30289:
URL: https://github.com/apache/spark/pull/30289#discussion_r530130202
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala
##
@@ -748,4 +748,30 @@ abstract class SQLViewSuite extends QueryTest with
SQLTestUtils {
}
}
}
+
+ test("SPARK-33141 view should be parsed and analyzed with configs set when
creating") {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] luluorta commented on a change in pull request #30289: [SPARK-33141][SQL] Capture SQL configs when creating permanent views
luluorta commented on a change in pull request #30289:
URL: https://github.com/apache/spark/pull/30289#discussion_r530130052
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
##
@@ -771,22 +771,24 @@ class AnalysisSuite extends AnalysisTest with Matchers {
// RuleExecutor only throw exception or log warning when the rule is
supposed to run
// more than once.
val maxIterations = 2
-val conf = new SQLConf().copy(SQLConf.ANALYZER_MAX_ITERATIONS ->
maxIterations)
-val testAnalyzer = new Analyzer(
- new SessionCatalog(new InMemoryCatalog, FunctionRegistry.builtin, conf),
conf)
-
-val plan = testRelation2.select(
- $"a" / Literal(2) as "div1",
- $"a" / $"b" as "div2",
- $"a" / $"c" as "div3",
- $"a" / $"d" as "div4",
- $"e" / $"e" as "div5")
-
-val message = intercept[TreeNodeException[LogicalPlan]] {
- testAnalyzer.execute(plan)
-}.getMessage
-assert(message.startsWith(s"Max iterations ($maxIterations) reached for
batch Resolution, " +
- s"please set '${SQLConf.ANALYZER_MAX_ITERATIONS.key}' to a larger
value."))
+withSQLConf(SQLConf.ANALYZER_MAX_ITERATIONS.key -> maxIterations.toString)
{
+ val conf = new SQLConf().copy(SQLConf.ANALYZER_MAX_ITERATIONS ->
maxIterations)
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] luluorta commented on a change in pull request #30289: [SPARK-33141][SQL] Capture SQL configs when creating permanent views
luluorta commented on a change in pull request #30289:
URL: https://github.com/apache/spark/pull/30289#discussion_r530129632
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -132,9 +132,11 @@ object AnalysisContext {
*/
class Analyzer(
override val catalogManager: CatalogManager,
-conf: SQLConf)
+deprecatedConf: SQLConf)
Review comment:
This change is related to another sub-task of
[SPARK-33138](https://issues.apache.org/jira/browse/SPARK-33138) which makes
internal classes of SparkSession always using active SQLConf. I opened a
seperated PR (https://github.com/apache/spark/pull/30299) and removed this
change here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gaborgsomogyi commented on pull request #30336: [SPARK-33287][SS][UI]Expose state custom metrics information on SS UI
gaborgsomogyi commented on pull request #30336: URL: https://github.com/apache/spark/pull/30336#issuecomment-733490462 Thank you all for taking care! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30470: [SPARK-33495][BUILD] Remove commons-logging.jar's dependency
AmplabJenkins removed a comment on pull request #30470: URL: https://github.com/apache/spark/pull/30470#issuecomment-733490038 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30440: [SPARK-33496][SQL]Improve error message of ANSI explicit cast
AmplabJenkins removed a comment on pull request #30440: URL: https://github.com/apache/spark/pull/30440#issuecomment-733490070 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30470: [SPARK-33495][BUILD] Remove commons-logging.jar's dependency
AmplabJenkins commented on pull request #30470: URL: https://github.com/apache/spark/pull/30470#issuecomment-733490038 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30440: [SPARK-33496][SQL]Improve error message of ANSI explicit cast
AmplabJenkins commented on pull request #30440: URL: https://github.com/apache/spark/pull/30440#issuecomment-733490070 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30440: [SPARK-33496][SQL]Improve error message of ANSI explicit cast
SparkQA removed a comment on pull request #30440: URL: https://github.com/apache/spark/pull/30440#issuecomment-733419129 **[Test build #131721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131721/testReport)** for PR 30440 at commit [`e762162`](https://github.com/apache/spark/commit/e762162311e04c20bb06f9a4735514547050b832). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30440: [SPARK-33496][SQL]Improve error message of ANSI explicit cast
SparkQA commented on pull request #30440: URL: https://github.com/apache/spark/pull/30440#issuecomment-733489487 **[Test build #131721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131721/testReport)** for PR 30440 at commit [`e762162`](https://github.com/apache/spark/commit/e762162311e04c20bb06f9a4735514547050b832). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30289: [SPARK-33141][SQL] Capture SQL configs when creating permanent views
SparkQA commented on pull request #30289: URL: https://github.com/apache/spark/pull/30289#issuecomment-733488853 **[Test build #131746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131746/testReport)** for PR 30289 at commit [`22b7dde`](https://github.com/apache/spark/commit/22b7ddec643960d4dda1046604b10034caf22b94). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30493: [SPARK-33549][SQL] Remove configuration spark.sql.legacy.allowCastNumericToTimestamp
AmplabJenkins removed a comment on pull request #30493: URL: https://github.com/apache/spark/pull/30493#issuecomment-733470894 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables
SparkQA commented on pull request #30403: URL: https://github.com/apache/spark/pull/30403#issuecomment-733488765 **[Test build #131745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131745/testReport)** for PR 30403 at commit [`5e7227b`](https://github.com/apache/spark/commit/5e7227ba742cf6af1eb6fc9ce6a810959ab3b94f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30442: [SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid
SparkQA commented on pull request #30442: URL: https://github.com/apache/spark/pull/30442#issuecomment-733488754 **[Test build #131744 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131744/testReport)** for PR 30442 at commit [`5ab59d4`](https://github.com/apache/spark/commit/5ab59d41cf840f524d37dbb581fabcbbba1c56a4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30493: [SPARK-33549][SQL] Remove configuration spark.sql.legacy.allowCastNumericToTimestamp
SparkQA commented on pull request #30493: URL: https://github.com/apache/spark/pull/30493#issuecomment-733488636 **[Test build #131742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131742/testReport)** for PR 30493 at commit [`6d3e7e2`](https://github.com/apache/spark/commit/6d3e7e2b8bfb8c6d2a21fefc7133b766421fb050). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30486: [SPARK-33530][CORE] Support --archives and spark.archives option natively
SparkQA commented on pull request #30486: URL: https://github.com/apache/spark/pull/30486#issuecomment-733488641 **[Test build #131743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131743/testReport)** for PR 30486 at commit [`15d8ed5`](https://github.com/apache/spark/commit/15d8ed51d99e57403aae3272d9975b96e7735ee2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29994: [DONOTMERGE][WHITESPACE] workflow exercise
AmplabJenkins removed a comment on pull request #29994: URL: https://github.com/apache/spark/pull/29994#issuecomment-733488127 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29994: [DONOTMERGE][WHITESPACE] workflow exercise
AmplabJenkins commented on pull request #29994: URL: https://github.com/apache/spark/pull/29994#issuecomment-733488127 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30495: [SPARK-33548][WEBUI] display the jvm peak memory usage on the executor ui
AmplabJenkins commented on pull request #30495: URL: https://github.com/apache/spark/pull/30495#issuecomment-733487363 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition
AmplabJenkins removed a comment on pull request #30494: URL: https://github.com/apache/spark/pull/30494#issuecomment-733486871 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
AmplabJenkins removed a comment on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733486877 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29066: [SPARK-23889][SQL] DataSourceV2: required sorting and clustering for writes
AmplabJenkins removed a comment on pull request #29066: URL: https://github.com/apache/spark/pull/29066#issuecomment-733486873 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28026: [SPARK-31257][SQL] Unify create table syntax
AmplabJenkins removed a comment on pull request #28026: URL: https://github.com/apache/spark/pull/28026#issuecomment-733486874 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables
AmplabJenkins removed a comment on pull request #30403: URL: https://github.com/apache/spark/pull/30403#issuecomment-733486872 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition
AmplabJenkins commented on pull request #30494: URL: https://github.com/apache/spark/pull/30494#issuecomment-733486871 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables
AmplabJenkins commented on pull request #30403: URL: https://github.com/apache/spark/pull/30403#issuecomment-733486872 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28026: [SPARK-31257][SQL] Unify create table syntax
AmplabJenkins commented on pull request #28026: URL: https://github.com/apache/spark/pull/28026#issuecomment-733486884 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29066: [SPARK-23889][SQL] DataSourceV2: required sorting and clustering for writes
AmplabJenkins commented on pull request #29066: URL: https://github.com/apache/spark/pull/29066#issuecomment-733486883 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
AmplabJenkins commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733486877 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
SparkQA removed a comment on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733450215 **[Test build #131727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131727/testReport)** for PR 30492 at commit [`9395c4b`](https://github.com/apache/spark/commit/9395c4b017ca7b0c60935540a1d445b708ffe75a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30492: [SPARK-33545][CORE] Support Fallback Storage during Worker decommission
SparkQA commented on pull request #30492: URL: https://github.com/apache/spark/pull/30492#issuecomment-733485444 **[Test build #131727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131727/testReport)** for PR 30492 at commit [`9395c4b`](https://github.com/apache/spark/commit/9395c4b017ca7b0c60935540a1d445b708ffe75a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang closed pull request #30353: [SPARK-33431][CORE][SQL][DSTREAM] Use Option.contains instead of instanceOfOption == Some(x)
LuciferYang closed pull request #30353: URL: https://github.com/apache/spark/pull/30353 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29066: [SPARK-23889][SQL] DataSourceV2: required sorting and clustering for writes
SparkQA removed a comment on pull request #29066: URL: https://github.com/apache/spark/pull/29066#issuecomment-733433901 **[Test build #131726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131726/testReport)** for PR 29066 at commit [`ecc6a46`](https://github.com/apache/spark/commit/ecc6a463f3d6c8147e84d7067b7e3d457167f475). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on a change in pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables
imback82 commented on a change in pull request #30403:
URL: https://github.com/apache/spark/pull/30403#discussion_r530121226
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
##
@@ -679,3 +679,30 @@ case class TruncateTable(
partitionSpec: Option[TablePartitionSpec]) extends Command {
override def children: Seq[LogicalPlan] = child :: Nil
}
+
+/**
+ * The logical plan of the CACHE TABLE command.
+ */
+case class CacheTable(
+child: LogicalPlan,
Review comment:
Also, since it's not resolving to catalogs, we should move it out of
`ResolveSessionCatalog`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29066: [SPARK-23889][SQL] DataSourceV2: required sorting and clustering for writes
SparkQA commented on pull request #29066: URL: https://github.com/apache/spark/pull/29066#issuecomment-733480531 **[Test build #131726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131726/testReport)** for PR 29066 at commit [`ecc6a46`](https://github.com/apache/spark/commit/ecc6a463f3d6c8147e84d7067b7e3d457167f475). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition
SparkQA removed a comment on pull request #30494: URL: https://github.com/apache/spark/pull/30494#issuecomment-733470435 **[Test build #131738 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131738/testReport)** for PR 30494 at commit [`1a6628e`](https://github.com/apache/spark/commit/1a6628eb9cc35e51dfda601e11d56288309955c4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30494: [SPARK-33551][SQL] Do not use custom shuffle reader for repartition
SparkQA commented on pull request #30494: URL: https://github.com/apache/spark/pull/30494#issuecomment-733480069 **[Test build #131738 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131738/testReport)** for PR 30494 at commit [`1a6628e`](https://github.com/apache/spark/commit/1a6628eb9cc35e51dfda601e11d56288309955c4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30403: [SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables
SparkQA removed a comment on pull request #30403: URL: https://github.com/apache/spark/pull/30403#issuecomment-733473558 **[Test build #131741 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131741/testReport)** for PR 30403 at commit [`c0e4f3e`](https://github.com/apache/spark/commit/c0e4f3ee4a1760e97ee999aa542fbe7cae72db50). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30486: [SPARK-33530][CORE] Support --archives and spark.archives option natively
HyukjinKwon commented on pull request #30486: URL: https://github.com/apache/spark/pull/30486#issuecomment-733477920 Thanks @maropu and @dongjoon-hyun. I believe I addressed the comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
