[GitHub] [spark] AmplabJenkins removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
AmplabJenkins removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528541908 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110196/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
SparkQA commented on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528544975 **[Test build #110197 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110197/testReport)** for PR 25685 at commit [`cca1348`](https://github.com/apache/spark/commit/cca1348616dffcaf4cb282f5b4cc3d9bf862afd1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
AmplabJenkins commented on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528545161 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110197/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
AmplabJenkins commented on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528545156 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
AmplabJenkins removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528545156 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
SparkQA removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528506257 **[Test build #110197 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110197/testReport)** for PR 25685 at commit [`cca1348`](https://github.com/apache/spark/commit/cca1348616dffcaf4cb282f5b4cc3d9bf862afd1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maryannxue commented on a change in pull request #25644: [SPARK-28940][SQL] Subquery reuse across all subquery levels
maryannxue commented on a change in pull request #25644: [SPARK-28940][SQL]
Subquery reuse across all subquery levels
URL: https://github.com/apache/spark/pull/25644#discussion_r32113
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
##
@@ -1238,7 +1238,10 @@ class DynamicPartitionPruningSuite
val plan = df.queryExecution.executedPlan
val countSubqueryBroadcasts =
-plan.collectInPlanAndSubqueries({ case _: SubqueryBroadcastExec => 1
}).sum
+plan.collectInPlanAndSubqueries({
Review comment:
We have `InSubqueryExec` now in DPP, so better keep it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] alaazbair commented on issue #25682: [SPARK-28842][DOC]Cleanup the formatting/trailing spaces in the K8s integration testing guide
alaazbair commented on issue #25682: [SPARK-28842][DOC]Cleanup the formatting/trailing spaces in the K8s integration testing guide URL: https://github.com/apache/spark/pull/25682#issuecomment-528546021 Could someone please review my PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
AmplabJenkins removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528545161 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110197/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
srowen commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528548702 I added removal of HiveContext -- let's see how this goes with tests. I _think_ that's not controversial? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
SparkQA commented on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528549329 **[Test build #110195 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110195/testReport)** for PR 25685 at commit [`e9a098d`](https://github.com/apache/spark/commit/e9a098d403e871230fdec490da7f91f31ed46916). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
srowen commented on a change in pull request #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#discussion_r321446672 ## File path: python/pyspark/ml/tests/test_image.py ## @@ -74,17 +74,14 @@ def setUpClass(cls): super(ImageFileFormatOnHiveContextTest, cls).setUpClass() cls.hive_available = True # Note that here we enable Hive's support. -cls.spark = None try: -cls.sc._jvm.org.apache.hadoop.hive.conf.HiveConf() +cls.spark = SparkSession.builder.enableHiveSupport().getOrCreate() Review comment: @HyukjinKwon I may need your input on whether this is the right way to 'replace' use of HiveContext; is this test needed anymore even? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
AmplabJenkins commented on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528549682 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110195/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
SparkQA removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528491327 **[Test build #110195 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110195/testReport)** for PR 25685 at commit [`e9a098d`](https://github.com/apache/spark/commit/e9a098d403e871230fdec490da7f91f31ed46916). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
AmplabJenkins commented on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528549676 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
AmplabJenkins removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528549676 Build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog
AmplabJenkins removed a comment on issue #25685: Refactor ALTER TABLE and INSERT INTO for V2 Session Catalog URL: https://github.com/apache/spark/pull/25685#issuecomment-528549682 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110195/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
AmplabJenkins commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528550278 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
AmplabJenkins commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528550283 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15205/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
AmplabJenkins removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528550278 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
AmplabJenkins removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528550283 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15205/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
SparkQA commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528552687 **[Test build #110199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110199/testReport)** for PR 25684 at commit [`a1afcc1`](https://github.com/apache/spark/commit/a1afcc145fde1721d9eb117ececdce83c6da689f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25697: [SPARK-28997][SQL] Add `spark.sql.dialect`
SparkQA removed a comment on issue #25697: [SPARK-28997][SQL] Add `spark.sql.dialect` URL: https://github.com/apache/spark/pull/25697#issuecomment-528440724 **[Test build #110191 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110191/testReport)** for PR 25697 at commit [`3560573`](https://github.com/apache/spark/commit/35605736dbe6e71b62616d4756c24c26f700894e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25697: [SPARK-28997][SQL] Add `spark.sql.dialect`
SparkQA commented on issue #25697: [SPARK-28997][SQL] Add `spark.sql.dialect` URL: https://github.com/apache/spark/pull/25697#issuecomment-528554646 **[Test build #110191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110191/testReport)** for PR 25697 at commit [`3560573`](https://github.com/apache/spark/commit/35605736dbe6e71b62616d4756c24c26f700894e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25697: [SPARK-28997][SQL] Add `spark.sql.dialect`
AmplabJenkins removed a comment on issue #25697: [SPARK-28997][SQL] Add `spark.sql.dialect` URL: https://github.com/apache/spark/pull/25697#issuecomment-528555681 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110191/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25697: [SPARK-28997][SQL] Add `spark.sql.dialect`
AmplabJenkins commented on issue #25697: [SPARK-28997][SQL] Add `spark.sql.dialect` URL: https://github.com/apache/spark/pull/25697#issuecomment-528555674 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25697: [SPARK-28997][SQL] Add `spark.sql.dialect`
AmplabJenkins commented on issue #25697: [SPARK-28997][SQL] Add `spark.sql.dialect` URL: https://github.com/apache/spark/pull/25697#issuecomment-528555681 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110191/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25697: [SPARK-28997][SQL] Add `spark.sql.dialect`
AmplabJenkins removed a comment on issue #25697: [SPARK-28997][SQL] Add `spark.sql.dialect` URL: https://github.com/apache/spark/pull/25697#issuecomment-528555674 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events
SparkQA commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events URL: https://github.com/apache/spark/pull/25702#issuecomment-528562257 **[Test build #110198 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110198/testReport)** for PR 25702 at commit [`52181fb`](https://github.com/apache/spark/commit/52181fb993d10d967d3bb7c23b7498b9e813f632). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events
SparkQA removed a comment on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events URL: https://github.com/apache/spark/pull/25702#issuecomment-528515365 **[Test build #110198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110198/testReport)** for PR 25702 at commit [`52181fb`](https://github.com/apache/spark/commit/52181fb993d10d967d3bb7c23b7498b9e813f632). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events
AmplabJenkins commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events URL: https://github.com/apache/spark/pull/25702#issuecomment-528562593 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110198/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events
AmplabJenkins commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events URL: https://github.com/apache/spark/pull/25702#issuecomment-528562585 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events
AmplabJenkins removed a comment on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events URL: https://github.com/apache/spark/pull/25702#issuecomment-528562585 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events
AmplabJenkins removed a comment on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events URL: https://github.com/apache/spark/pull/25702#issuecomment-528562593 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110198/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mgaido91 commented on issue #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd
mgaido91 commented on issue #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd URL: https://github.com/apache/spark/pull/25643#issuecomment-528567538 @tgravescs all SQL configs which have been set whatever way are applied. > when you do a bunch of dataframe/dataset operations and then do toRdd, it is definitely annoying we never indicate to the user catalyst stuff still happened (like SQL ui isn't present ) yes, I quite agree on this. Unfortunately I don't have a good idea to solve this as the actions are performed outside the "SQL" world, ie. they are performed on the RDD by the user, so it's hard to track them within a SQL execution ID Honestly I have no good idea for that at the moment, might you have one, I think it would be great.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25700: [SPARK-28998][SQL] reorganize the packages of DS v2 interfaces/classes
SparkQA commented on issue #25700: [SPARK-28998][SQL] reorganize the packages of DS v2 interfaces/classes URL: https://github.com/apache/spark/pull/25700#issuecomment-528573596 **[Test build #110193 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110193/testReport)** for PR 25700 at commit [`9b8969f`](https://github.com/apache/spark/commit/9b8969f2737f44fbaddecf72047e49307df2d137). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25700: [SPARK-28998][SQL] reorganize the packages of DS v2 interfaces/classes
SparkQA removed a comment on issue #25700: [SPARK-28998][SQL] reorganize the packages of DS v2 interfaces/classes URL: https://github.com/apache/spark/pull/25700#issuecomment-528454591 **[Test build #110193 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110193/testReport)** for PR 25700 at commit [`9b8969f`](https://github.com/apache/spark/commit/9b8969f2737f44fbaddecf72047e49307df2d137). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25700: [SPARK-28998][SQL] reorganize the packages of DS v2 interfaces/classes
AmplabJenkins commented on issue #25700: [SPARK-28998][SQL] reorganize the packages of DS v2 interfaces/classes URL: https://github.com/apache/spark/pull/25700#issuecomment-528574187 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110193/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25700: [SPARK-28998][SQL] reorganize the packages of DS v2 interfaces/classes
AmplabJenkins commented on issue #25700: [SPARK-28998][SQL] reorganize the packages of DS v2 interfaces/classes URL: https://github.com/apache/spark/pull/25700#issuecomment-528574181 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25700: [SPARK-28998][SQL] reorganize the packages of DS v2 interfaces/classes
AmplabJenkins removed a comment on issue #25700: [SPARK-28998][SQL] reorganize the packages of DS v2 interfaces/classes URL: https://github.com/apache/spark/pull/25700#issuecomment-528574187 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110193/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25700: [SPARK-28998][SQL] reorganize the packages of DS v2 interfaces/classes
AmplabJenkins removed a comment on issue #25700: [SPARK-28998][SQL] reorganize the packages of DS v2 interfaces/classes URL: https://github.com/apache/spark/pull/25700#issuecomment-528574181 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
dilipbiswal commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528576948 LGTM cc @gatorsmile This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd
AmplabJenkins commented on issue #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd URL: https://github.com/apache/spark/pull/25643#issuecomment-528577568 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd
AmplabJenkins commented on issue #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd URL: https://github.com/apache/spark/pull/25643#issuecomment-528577576 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15206/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd
AmplabJenkins removed a comment on issue #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd URL: https://github.com/apache/spark/pull/25643#issuecomment-528577568 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd
AmplabJenkins removed a comment on issue #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd URL: https://github.com/apache/spark/pull/25643#issuecomment-528577576 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15206/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] asfgit closed pull request #25673: [SPARK-28770][CORE][TEST] Fix ReplayListenerSuite tests that sometimes fail
asfgit closed pull request #25673: [SPARK-28770][CORE][TEST] Fix ReplayListenerSuite tests that sometimes fail URL: https://github.com/apache/spark/pull/25673 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] squito commented on issue #25673: [SPARK-28770][CORE][TEST] Fix ReplayListenerSuite tests that sometimes fail
squito commented on issue #25673: [SPARK-28770][CORE][TEST] Fix ReplayListenerSuite tests that sometimes fail URL: https://github.com/apache/spark/pull/25673#issuecomment-528578350 merged to master, thanks everyone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd
SparkQA commented on issue #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd URL: https://github.com/apache/spark/pull/25643#issuecomment-528578531 **[Test build #110200 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110200/testReport)** for PR 25643 at commit [`860b658`](https://github.com/apache/spark/commit/860b658976ede102aeba377eee99f93d308226b0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #25643: [SPARK-28939][SQL] Propagate SQLConf for plans executed by toRdd
viirya commented on a change in pull request #25643: [SPARK-28939][SQL]
Propagate SQLConf for plans executed by toRdd
URL: https://github.com/apache/spark/pull/25643#discussion_r321481486
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecutionRDD.scala
##
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.execution
+
+import java.util.Properties
+
+import scala.collection.JavaConverters._
+
+import org.apache.spark.{Partition, TaskContext}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * It is just a wrapper over `sqlRDD`, which sets and makes effective all the
configs from the
+ * captured `SQLConf`.
+ * Please notice that this means we may miss configurations set after the
creation of this RDD and
+ * before its execution.
+ *
+ * @param sqlRDD the `RDD` generated by the SQL plan
+ * @param conf the `SQLConf` to apply to the execution of the SQL plan
+ */
+class SQLExecutionRDD(
+var sqlRDD: RDD[InternalRow], @transient conf: SQLConf) extends
RDD[InternalRow](sqlRDD) {
Review comment:
Do we need to pass in a SQLConf? I think we just always capture current
SQLConf, so just `private val sqlConfigs = SQLConf.get. getAllConfs`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
maropu commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-528583747 @viirya ok, all the tests passed with `splitAggregateFunc.enabled=false`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-528584592 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15207/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-528584586 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-528584586 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
AmplabJenkins removed a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-528584592 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15207/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
SparkQA commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-528590742 **[Test build #110201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110201/testReport)** for PR 20965 at commit [`ab8de83`](https://github.com/apache/spark/commit/ab8de8328599832b2f535c124aa5cb55a006277e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25670: [SPARK-28869][CORE] Roll over event log files
SparkQA commented on issue #25670: [SPARK-28869][CORE] Roll over event log files URL: https://github.com/apache/spark/pull/25670#issuecomment-528596037 **[Test build #110202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110202/testReport)** for PR 25670 at commit [`9b4d53d`](https://github.com/apache/spark/commit/9b4d53df0372ea0338a3a020f7809a0d5aa0f884). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] asfgit closed pull request #25518: [SPARK-26046][SS] Add StreamingQueryManager.listListeners()
asfgit closed pull request #25518: [SPARK-26046][SS] Add StreamingQueryManager.listListeners() URL: https://github.com/apache/spark/pull/25518 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jose-torres commented on issue #25518: [SPARK-26046][SS] Add StreamingQueryManager.listListeners()
jose-torres commented on issue #25518: [SPARK-26046][SS] Add StreamingQueryManager.listListeners() URL: https://github.com/apache/spark/pull/25518#issuecomment-528596423 Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #25672: [SPARK-28967][CORE] Include cloned version of "properties" to avoid ConcurrentModificationException
HeartSaVioR commented on issue #25672: [SPARK-28967][CORE] Include cloned version of "properties" to avoid ConcurrentModificationException URL: https://github.com/apache/spark/pull/25672#issuecomment-528598145 @felixcheung @srowen Looks like this patch got 2 +1 with test passed. Unless we would want to have more eyes on this, could I ask for looking back this and merging? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
gatorsmile commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528602594 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25693: [SPARK-28989][SQL] Add `spark.sql.ansi.enabled`
dongjoon-hyun commented on issue #25693: [SPARK-28989][SQL] Add `spark.sql.ansi.enabled` URL: https://github.com/apache/spark/pull/25693#issuecomment-528602728 Hi, @gatorsmile , @gengliangwang , @cloud-fan . Thank you for this effort to simplify them in Spark 3.0.0. What will it be the relationship between `spark.sql.dialect` and `spark.sql.ansi.enabled`? And, could you describe the reason why we don't choose `spark.sql.dialect=ansi` instead of `spark.sql.ansi.enabled`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
AmplabJenkins removed a comment on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-523314720 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu edited a comment on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
maropu edited a comment on issue #20965: [SPARK-21870][SQL] Split aggregation
code into small functions
URL: https://github.com/apache/spark/pull/20965#issuecomment-528350052
I checked the TPCDS performance numbers (sf=5) and I couldn't find much
difference with/without this pr:
https://docs.google.com/spreadsheets/d/10eCV0PHeAaWGaXwpKPDDYCSLzsogl4IIiGiitqvw8PY/edit?usp=sharing
Rather, the numbers in q66 didn't change between them. Probably, the other
PRs already have improved it (I didn't dig into it though).
To make sure this pr is still beneficial, I run microbenchmarks below;
```
$ ./bin/spark-shell --master=local[1] --conf spark.driver.memory=8g --conf
spark.sql.shuffle.partitions=1 -v
scala> val numCols = 50
scala> val colExprs = (0 until numCols).map { i => s"id AS _c$i" }
scala> spark.range(500).selectExpr(colExprs:
_*).createOrReplaceTempView("t")
scala> val aggExprs = (0 until numCols).map { i => s"AVG(_c$i)" }
scala> sql("SET spark.sql.codegen.aggregate.splitAggregateFunc.enabled=true")
scala> timer { sql(s"SELECT ${aggExprs.mkString(", ")} FROM
t").write.format("noop").save() }
Elapsed time: 0.997808995s
scala> sql("SET
spark.sql.codegen.aggregate.splitAggregateFunc.enabled=false")
scala> timer { sql(s"SELECT ${aggExprs.mkString(", ")} FROM
t").write.format("noop").save() }
Elapsed time: 25.77200574s
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
maropu commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-528603219 @viirya oh, sorry. I updated the link so that everyone can see it: https://docs.google.com/spreadsheets/d/10eCV0PHeAaWGaXwpKPDDYCSLzsogl4IIiGiitqvw8PY/edit?usp=sharing This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
SparkQA commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528603417 **[Test build #110199 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110199/testReport)** for PR 25684 at commit [`a1afcc1`](https://github.com/apache/spark/commit/a1afcc145fde1721d9eb117ececdce83c6da689f). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
SparkQA removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528552687 **[Test build #110199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110199/testReport)** for PR 25684 at commit [`a1afcc1`](https://github.com/apache/spark/commit/a1afcc145fde1721d9eb117ececdce83c6da689f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
AmplabJenkins commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528603956 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
AmplabJenkins commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528603965 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110199/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25689: [SPARK-28972][DOC]spark.memory.offHeap.size description require to update in document
dongjoon-hyun commented on issue #25689: [SPARK-28972][DOC]spark.memory.offHeap.size description require to update in document URL: https://github.com/apache/spark/pull/25689#issuecomment-528604198 Hi, @PavithraRamachandran . Thank you for your contribution. So, is this for consistency with `spark.executor.memory`? Is this the last piece for the consistency? Could you check the other configurations in this document? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #23956: [SPARK-27042][SS] Close cached Kafka producer in case of task retry
HeartSaVioR commented on issue #23956: [SPARK-27042][SS] Close cached Kafka producer in case of task retry URL: https://github.com/apache/spark/pull/23956#issuecomment-528604212 As we are putting effort on #19096 regarding this issue, I guess we can close this for cleaning up and discuss further on #19096. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
SparkQA commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528604313 **[Test build #110203 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110203/testReport)** for PR 25528 at commit [`59b4f1d`](https://github.com/apache/spark/commit/59b4f1d3efeac15cf14723eace4b74f560601361). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
AmplabJenkins removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528603956 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
AmplabJenkins removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528603965 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110199/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events
dongjoon-hyun commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events URL: https://github.com/apache/spark/pull/25702#issuecomment-528604634 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25654: [SPARK-28912][STREAMING] Fixed MatchError in getCheckpointFiles()
dongjoon-hyun commented on issue #25654: [SPARK-28912][STREAMING] Fixed MatchError in getCheckpointFiles() URL: https://github.com/apache/spark/pull/25654#issuecomment-528606077 Gentle ping, @avkgh . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on issue #19096: [SPARK-21869][SS] A cached Kafka producer should not be closed if any task is using it - adds inuse tracking.
HeartSaVioR commented on issue #19096: [SPARK-21869][SS] A cached Kafka producer should not be closed if any task is using it - adds inuse tracking. URL: https://github.com/apache/spark/pull/19096#issuecomment-528606061 I second on this, as that's in line what I proposed before. Either I can help reviewing or even work on this. We still have some consideration regarding performance and number of connections (as we allow multiple-threads access whereas Apache Commons Pool will prevent this), but even in javadoc of Kafka producer it just says `The producer is thread safe and sharing a single producer instance across threads will generally be faster than having multiple instances.` which doesn't provide any perf. consideration, so I'd rather say let's try this out and measure it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events
AmplabJenkins commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events URL: https://github.com/apache/spark/pull/25702#issuecomment-528607342 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
AmplabJenkins commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528607319 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15209/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events
AmplabJenkins commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events URL: https://github.com/apache/spark/pull/25702#issuecomment-528607351 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15208/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
AmplabJenkins commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528607312 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
SparkQA commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528608033 **[Test build #110203 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110203/testReport)** for PR 25528 at commit [`59b4f1d`](https://github.com/apache/spark/commit/59b4f1d3efeac15cf14723eace4b74f560601361). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
AmplabJenkins commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528608166 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
AmplabJenkins commented on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528608171 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110203/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
AmplabJenkins removed a comment on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528607312 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
SparkQA removed a comment on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528604313 **[Test build #110203 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110203/testReport)** for PR 25528 at commit [`59b4f1d`](https://github.com/apache/spark/commit/59b4f1d3efeac15cf14723eace4b74f560601361). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events
AmplabJenkins removed a comment on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events URL: https://github.com/apache/spark/pull/25702#issuecomment-528607342 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
AmplabJenkins removed a comment on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528608171 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/110203/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events
AmplabJenkins removed a comment on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events URL: https://github.com/apache/spark/pull/25702#issuecomment-528607351 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15208/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
AmplabJenkins removed a comment on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528607319 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15209/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference
AmplabJenkins removed a comment on issue #25528: [SPARK-28802][DOC][SQL]Document DESCRIBE DATABASE statement in SQL Reference URL: https://github.com/apache/spark/pull/25528#issuecomment-528608166 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events
SparkQA commented on issue #25702: [SPARK-29001][CORE] Print event thread stack trace when EventQueue starts to drop events URL: https://github.com/apache/spark/pull/25702#issuecomment-528609010 **[Test build #110204 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110204/testReport)** for PR 25702 at commit [`52181fb`](https://github.com/apache/spark/commit/52181fb993d10d967d3bb7c23b7498b9e813f632). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions
viirya commented on issue #20965: [SPARK-21870][SQL] Split aggregation code into small functions URL: https://github.com/apache/spark/pull/20965#issuecomment-528612083 @maropu Thanks! :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] avkgh commented on issue #25654: [SPARK-28912][STREAMING] Fixed MatchError in getCheckpointFiles()
avkgh commented on issue #25654: [SPARK-28912][STREAMING] Fixed MatchError in getCheckpointFiles() URL: https://github.com/apache/spark/pull/25654#issuecomment-528613352 I will add a unit test when I have the time. I plan to do it within next 2 days. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
AmplabJenkins commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528614095 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
AmplabJenkins commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528614103 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15210/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
AmplabJenkins removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528614103 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/15210/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
AmplabJenkins removed a comment on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528614095 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3
SparkQA commented on issue #25684: [SPARK-28980][CORE][SQL][STREAMING][MLLIB] Remove most items deprecated in Spark 2.2.0 or earlier, for Spark 3 URL: https://github.com/apache/spark/pull/25684#issuecomment-528614772 **[Test build #110205 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110205/testReport)** for PR 25684 at commit [`b23cd76`](https://github.com/apache/spark/commit/b23cd7649b3ae1a45ef9dd77f4d821e63614778d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maryannxue opened a new pull request #25703: [SPARK-29002][SQL] Avoid changing SMJ to BHJ if the build side has a high ratio of empty partitions
maryannxue opened a new pull request #25703: [SPARK-29002][SQL] Avoid changing SMJ to BHJ if the build side has a high ratio of empty partitions URL: https://github.com/apache/spark/pull/25703 ### What changes were proposed in this pull request? This PR aims to avoid AQE regressions by avoiding changing a sort merge join to a broadcast hash join when the expected build plan has a high ratio of empty partitions, in which case sort merge join can actually perform faster. This PR achieves this by adding an internal join hint in order to let the planner know which side has this high ratio of empty partitions and it should avoid planning it as a build plan of a BHJ. Still, it won't affect the other side if the other side qualifies for a build plan of a BHJ. ### Why are the changes needed? It is a performance improvement for AQE. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Added UT. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
