[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
dongjoon-hyun commented on a change in pull request #24068: [SPARK-27105][SQL]
Optimize away exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#discussion_r293216015
##
File path: sql/core/benchmarks/FilterPushdownBenchmark-results.txt
##
@@ -2,669 +2,695 @@
Pushdown for many distinct value case
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 string row (value IS NULL): Best/Avg Time(ms)Rate(M/s) Per
Row(ns) Relative
-
-Parquet Vectorized 11405 / 11485 1.4
725.1 1.0X
-Parquet Vectorized (Pushdown) 675 / 690 23.3
42.9 16.9X
-Native ORC Vectorized 7127 / 7170 2.2
453.1 1.6X
-Native ORC Vectorized (Pushdown) 519 / 541 30.3
33.0 22.0X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-
-Parquet Vectorized 11457 / 11473 1.4
728.4 1.0X
-Parquet Vectorized (Pushdown) 656 / 686 24.0
41.7 17.5X
-Native ORC Vectorized 7328 / 7342 2.1
465.9 1.6X
-Native ORC Vectorized (Pushdown) 539 / 565 29.2
34.2 21.3X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 string row (value = '7864320'): Best/Avg Time(ms)Rate(M/s) Per
Row(ns) Relative
-
-Parquet Vectorized 11878 / 11888 1.3
755.2 1.0X
-Parquet Vectorized (Pushdown) 630 / 654 25.0
40.1 18.9X
-Native ORC Vectorized 7342 / 7362 2.1
466.8 1.6X
-Native ORC Vectorized (Pushdown) 519 / 537 30.3
33.0 22.9X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 string row (value <=> '7864320'): Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
-
-Parquet Vectorized 11423 / 11440 1.4
726.2 1.0X
-Parquet Vectorized (Pushdown) 625 / 643 25.2
39.7 18.3X
-Native ORC Vectorized 7315 / 7335 2.2
465.1 1.6X
-Native ORC Vectorized (Pushdown) 507 / 520 31.0
32.2 22.5X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 string row ('7864320' <= value <= '7864320'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-
-Parquet Vectorized 11440 / 11478 1.4
727.3 1.0X
-Parquet Vectorized (Pushdown) 634 / 652 24.8
40.3 18.0X
-Native ORC Vectorized 7311 / 7324 2.2
464.8 1.6X
-Native ORC Vectorized (Pushdown) 517 / 548 30.4
32.8 22.1X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select all string rows (value IS NOT NULL): Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
-
-Parquet Vectorized 20750 / 20872 0.8
1319.3 1.0X
-Parquet Vectorized (Pushdown) 21002 / 21032 0.7
1335.3 1.0X
-Native ORC Vectorized 16714 / 16742 0.9
1062.6 1.2X
-Native ORC Vectorized (Pushdown)16926 / 16965 0.9
1076.1 1.2X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 int row (value IS NULL):Best/Avg Time(ms)Rate(M/s) Per
Row(ns) Relative
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
dongjoon-hyun commented on a change in pull request #24068: [SPARK-27105][SQL]
Optimize away exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#discussion_r293215498
##
File path: sql/core/benchmarks/FilterPushdownBenchmark-results.txt
##
@@ -2,669 +2,695 @@
Pushdown for many distinct value case
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 string row (value IS NULL): Best/Avg Time(ms)Rate(M/s) Per
Row(ns) Relative
-
-Parquet Vectorized 11405 / 11485 1.4
725.1 1.0X
-Parquet Vectorized (Pushdown) 675 / 690 23.3
42.9 16.9X
-Native ORC Vectorized 7127 / 7170 2.2
453.1 1.6X
-Native ORC Vectorized (Pushdown) 519 / 541 30.3
33.0 22.0X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-
-Parquet Vectorized 11457 / 11473 1.4
728.4 1.0X
-Parquet Vectorized (Pushdown) 656 / 686 24.0
41.7 17.5X
-Native ORC Vectorized 7328 / 7342 2.1
465.9 1.6X
-Native ORC Vectorized (Pushdown) 539 / 565 29.2
34.2 21.3X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 string row (value = '7864320'): Best/Avg Time(ms)Rate(M/s) Per
Row(ns) Relative
-
-Parquet Vectorized 11878 / 11888 1.3
755.2 1.0X
-Parquet Vectorized (Pushdown) 630 / 654 25.0
40.1 18.9X
-Native ORC Vectorized 7342 / 7362 2.1
466.8 1.6X
-Native ORC Vectorized (Pushdown) 519 / 537 30.3
33.0 22.9X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 string row (value <=> '7864320'): Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
-
-Parquet Vectorized 11423 / 11440 1.4
726.2 1.0X
-Parquet Vectorized (Pushdown) 625 / 643 25.2
39.7 18.3X
-Native ORC Vectorized 7315 / 7335 2.2
465.1 1.6X
-Native ORC Vectorized (Pushdown) 507 / 520 31.0
32.2 22.5X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 string row ('7864320' <= value <= '7864320'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-
-Parquet Vectorized 11440 / 11478 1.4
727.3 1.0X
-Parquet Vectorized (Pushdown) 634 / 652 24.8
40.3 18.0X
-Native ORC Vectorized 7311 / 7324 2.2
464.8 1.6X
-Native ORC Vectorized (Pushdown) 517 / 548 30.4
32.8 22.1X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select all string rows (value IS NOT NULL): Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
-
-Parquet Vectorized 20750 / 20872 0.8
1319.3 1.0X
-Parquet Vectorized (Pushdown) 21002 / 21032 0.7
1335.3 1.0X
-Native ORC Vectorized 16714 / 16742 0.9
1062.6 1.2X
-Native ORC Vectorized (Pushdown)16926 / 16965 0.9
1076.1 1.2X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 int row (value IS NULL):Best/Avg Time(ms)Rate(M/s) Per
Row(ns) Relative
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
dongjoon-hyun commented on a change in pull request #24068: [SPARK-27105][SQL]
Optimize away exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#discussion_r293215498
##
File path: sql/core/benchmarks/FilterPushdownBenchmark-results.txt
##
@@ -2,669 +2,695 @@
Pushdown for many distinct value case
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 string row (value IS NULL): Best/Avg Time(ms)Rate(M/s) Per
Row(ns) Relative
-
-Parquet Vectorized 11405 / 11485 1.4
725.1 1.0X
-Parquet Vectorized (Pushdown) 675 / 690 23.3
42.9 16.9X
-Native ORC Vectorized 7127 / 7170 2.2
453.1 1.6X
-Native ORC Vectorized (Pushdown) 519 / 541 30.3
33.0 22.0X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-
-Parquet Vectorized 11457 / 11473 1.4
728.4 1.0X
-Parquet Vectorized (Pushdown) 656 / 686 24.0
41.7 17.5X
-Native ORC Vectorized 7328 / 7342 2.1
465.9 1.6X
-Native ORC Vectorized (Pushdown) 539 / 565 29.2
34.2 21.3X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 string row (value = '7864320'): Best/Avg Time(ms)Rate(M/s) Per
Row(ns) Relative
-
-Parquet Vectorized 11878 / 11888 1.3
755.2 1.0X
-Parquet Vectorized (Pushdown) 630 / 654 25.0
40.1 18.9X
-Native ORC Vectorized 7342 / 7362 2.1
466.8 1.6X
-Native ORC Vectorized (Pushdown) 519 / 537 30.3
33.0 22.9X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 string row (value <=> '7864320'): Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
-
-Parquet Vectorized 11423 / 11440 1.4
726.2 1.0X
-Parquet Vectorized (Pushdown) 625 / 643 25.2
39.7 18.3X
-Native ORC Vectorized 7315 / 7335 2.2
465.1 1.6X
-Native ORC Vectorized (Pushdown) 507 / 520 31.0
32.2 22.5X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 1 string row ('7864320' <= value <= '7864320'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-
-Parquet Vectorized 11440 / 11478 1.4
727.3 1.0X
-Parquet Vectorized (Pushdown) 634 / 652 24.8
40.3 18.0X
-Native ORC Vectorized 7311 / 7324 2.2
464.8 1.6X
-Native ORC Vectorized (Pushdown) 517 / 548 30.4
32.8 22.1X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select all string rows (value IS NOT NULL): Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
-
-Parquet Vectorized 20750 / 20872 0.8
1319.3 1.0X
-Parquet Vectorized (Pushdown) 21002 / 21032 0.7
1335.3 1.0X
-Native ORC Vectorized 16714 / 16742 0.9
1062.6 1.2X
-Native ORC Vectorized (Pushdown)16926 / 16965 0.9
1076.1 1.2X
-
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
-Select 0 int row (value IS NULL):Best/Avg Time(ms)Rate(M/s) Per
Row(ns) Relative
[GitHub] [spark] AmplabJenkins removed a comment on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
AmplabJenkins removed a comment on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501559017 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
AmplabJenkins commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501559017 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
AmplabJenkins commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501559024 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106453/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
AmplabJenkins removed a comment on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501559024 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106453/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #24741: [SPARK-27322][SQL] DataSourceV2 table relation
cloud-fan closed pull request #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
SparkQA removed a comment on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501536110 **[Test build #106453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106453/testReport)** for PR 24826 at commit [`614013e`](https://github.com/apache/spark/commit/614013e0b0e87ef71a082a7ac269244157025aad). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
SparkQA commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501558556 **[Test build #106453 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106453/testReport)** for PR 24826 at commit [`614013e`](https://github.com/apache/spark/commit/614013e0b0e87ef71a082a7ac269244157025aad). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
AmplabJenkins removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501557552 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
AmplabJenkins commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501557557 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106451/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
AmplabJenkins removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501557557 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106451/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
AmplabJenkins commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501557552 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
SparkQA removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501525743 **[Test build #106451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106451/testReport)** for PR 24792 at commit [`9931eb6`](https://github.com/apache/spark/commit/9931eb63c0715ba190717a593ce51b949d5355b2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
SparkQA commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501557174 **[Test build #106451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106451/testReport)** for PR 24792 at commit [`9931eb6`](https://github.com/apache/spark/commit/9931eb63c0715ba190717a593ce51b949d5355b2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
cloud-fan commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501556439 I have only comment about adding more code comments, which can be addressed later. I'm merging it to unblock the DS v2 project, thanks for your hard work @jzhuge @rdblue ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
cloud-fan edited a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501556439 I have only one comment about adding more code comments, which can be addressed later. I'm merging it to unblock the DS v2 project, thanks for your hard work @jzhuge @rdblue ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
dongjoon-hyun commented on a change in pull request #24068: [SPARK-27105][SQL]
Optimize away exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#discussion_r293212240
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala
##
@@ -135,6 +139,34 @@ object FilterPushdownBenchmark extends BenchmarkBase with
SQLHelper {
benchmark.run()
}
+ def filterPushDownBenchmarkWithColumn(
Review comment:
@IvanVergiliev . The following doesn't mean put that into here.
> I think we should definitely have some automated benchmark for this.
Otherwise there's nothing in the codebase exercising the behaviour being
changed, and so nothing to prevent future regressions.
Since this contribution is big, it's worth to have its own benchmark
focusing on filter conversion. Also, the benchmark should have both ORCv1 and
ORCv2 benchmark result.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
dongjoon-hyun commented on a change in pull request #24068: [SPARK-27105][SQL]
Optimize away exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#discussion_r293212240
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala
##
@@ -135,6 +139,34 @@ object FilterPushdownBenchmark extends BenchmarkBase with
SQLHelper {
benchmark.run()
}
+ def filterPushDownBenchmarkWithColumn(
Review comment:
@IvanVergiliev . The following doesn't mean put that into here. Since this
contribution is big, it's worth to have its own benchmark focusing on filter
conversion.
> I think we should definitely have some automated benchmark for this.
Otherwise there's nothing in the codebase exercising the behaviour being
changed, and so nothing to prevent future regressions.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24741: [SPARK-27322][SQL] DataSourceV2 table relation
cloud-fan commented on a change in pull request #24741: [SPARK-27322][SQL]
DataSourceV2 table relation
URL: https://github.com/apache/spark/pull/24741#discussion_r293210583
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##
@@ -731,20 +753,16 @@ class Analyzer(
//and the default database is only used to look up a view);
// 3. Use the currentDb of the SessionCatalog.
private def lookupTableFromCatalog(
+tableIdentifier: TableIdentifier,
u: UnresolvedRelation,
defaultDatabase: Option[String] = None): LogicalPlan = {
- val tableIdentWithDb = u.tableIdentifier.copy(
-database = u.tableIdentifier.database.orElse(defaultDatabase))
+ val tableIdentWithDb = tableIdentifier.copy(
+database = tableIdentifier.database.orElse(defaultDatabase))
try {
catalog.lookupRelation(tableIdentWithDb)
} catch {
-case e: NoSuchTableException =>
- u.failAnalysis(s"Table or view not found:
${tableIdentWithDb.unquotedString}", e)
-// If the database is defined and that database is not found, throw an
AnalysisException.
-// Note that if the database is not defined, it is possible we are
looking up a temp view.
-case e: NoSuchDatabaseException =>
- u.failAnalysis(s"Table or view not found:
${tableIdentWithDb.unquotedString}, the " +
-s"database ${e.db} doesn't exist.", e)
+case _: NoSuchTableException | _: NoSuchDatabaseException =>
+ u
Review comment:
We should add some comments to explain why we need to delay the exception
here. To me it's because we still have a chance to resolve the table relation
with v2 rules.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
cloud-fan commented on a change in pull request #24068: [SPARK-27105][SQL]
Optimize away exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#discussion_r293209902
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala
##
@@ -362,6 +394,13 @@ object FilterPushdownBenchmark extends BenchmarkBase with
SQLHelper {
}
runBenchmark(s"Pushdown benchmark with many filters") {
+ // This benchmark and the next one are similar in that they both test
predicate pushdown
+ // where the filter itself is very large. There have been cases where
the filter conversion
+ // would take minutes to hours for large filters due to it being
implemented with exponential
+ // complexity in the height of the filter tree.
+ // The difference between these two benchmarks is that this one
benchmarks pushdown with a
+ // large string filter (`a AND b AND c ...`), whereas the next one
benchmarks pushdown with
+ // a large Column-based filter (`col(a) || (col(b) || (col(c)...))`).
Review comment:
I still can't get it. Both the string filter and column-based filter will
become an `Expression` in the `Filter` operator. The differences I see are
1. the new benchmark builds a larger filter
2. the new benchmark use `Or` instead of `And`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24857: [MINOR][CORE] Remove an unused variable in SparkSubmt.scala
dongjoon-hyun commented on a change in pull request #24857: [MINOR][CORE]
Remove an unused variable in SparkSubmt.scala
URL: https://github.com/apache/spark/pull/24857#discussion_r293207010
##
File path: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
##
@@ -131,13 +129,6 @@ private[ui] class StagePage(parent: StagesTab, store:
AppStatusStore) extends We
return UIUtils.headerSparkPage(request, stageHeader, content, parent)
}
-val storedTasks = store.taskCount(stageData.stageId, stageData.attemptId)
-val numCompleted = stageData.numCompleteTasks
-val totalTasksNumStr = if (totalTasks == storedTasks) {
- s"$totalTasks"
-} else {
- s"$totalTasks, showing $storedTasks"
-}
Review comment:
@imback82 . Before removing lines, please read the commit history. For
example, this is live code. Please see the following PR.
- https://github.com/apache/spark/pull/22525
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501549531 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106449/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501549526 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501549526 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501549531 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106449/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
SparkQA removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501517370 **[Test build #106449 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106449/testReport)** for PR 24741 at commit [`b8cdf6c`](https://github.com/apache/spark/commit/b8cdf6c22172585b3b3a9452d5e4d2d591ece88e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501549203 **[Test build #106449 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106449/testReport)** for PR 24741 at commit [`b8cdf6c`](https://github.com/apache/spark/commit/b8cdf6c22172585b3b3a9452d5e4d2d591ece88e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24857: [MINOR][CORE] Remove an unused variable in SparkSubmt.scala
dongjoon-hyun commented on a change in pull request #24857: [MINOR][CORE]
Remove an unused variable in SparkSubmt.scala
URL: https://github.com/apache/spark/pull/24857#discussion_r293205670
##
File path:
core/src/main/scala/org/apache/spark/memory/ExecutionMemoryPool.scala
##
@@ -151,7 +151,7 @@ private[memory] class ExecutionMemoryPool(
*/
def releaseMemory(numBytes: Long, taskAttemptId: Long): Unit =
lock.synchronized {
val curMem = memoryForTask.getOrElse(taskAttemptId, 0L)
-var memoryToFree = if (curMem < numBytes) {
+val memoryToFree = if (curMem < numBytes) {
Review comment:
Let's not put the different things in the same PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24857: [MINOR][CORE] Remove an unused variable in SparkSubmt.scala
dongjoon-hyun commented on a change in pull request #24857: [MINOR][CORE]
Remove an unused variable in SparkSubmt.scala
URL: https://github.com/apache/spark/pull/24857#discussion_r293205706
##
File path: core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala
##
@@ -364,7 +364,7 @@ private class DefaultPartitionCoalescer(val balanceSlack:
Double = 0.10)
val partNoLocIter = partitionLocs.partsWithoutLocs.iterator
groupArr.filter(pg => pg.numPartitions == 0).foreach { pg =>
while (partNoLocIter.hasNext && pg.numPartitions == 0) {
- var nxt_part = partNoLocIter.next()
+ val nxt_part = partNoLocIter.next()
Review comment:
ditto.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24857: [MINOR][CORE] Remove an unused variable in SparkSubmt.scala
dongjoon-hyun commented on a change in pull request #24857: [MINOR][CORE]
Remove an unused variable in SparkSubmt.scala
URL: https://github.com/apache/spark/pull/24857#discussion_r293205471
##
File path:
core/src/main/scala/org/apache/spark/deploy/rest/SubmitRestProtocolMessage.scala
##
@@ -46,9 +46,6 @@ private[rest] abstract class SubmitRestProtocolMessage {
val action: String = messageType
var message: String = null
- // For JSON deserialization
- private def setAction(a: String): Unit = { }
-
Review comment:
This was added from the
[beginning](https://github.com/apache/spark/commit/6ec0cdc14390d4dc45acf31040f21e1efc476fc0#diff-fb39e366f633463136727a6b6d5b832fR52)
and the comment seems to mean this is used. Shall we keep the existing one?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] IvanVergiliev commented on issue #24783: [SPARK-27105][SQL][test-hadoop3.2] Optimize away exponential complexity in ORC predicate conversion
IvanVergiliev commented on issue #24783: [SPARK-27105][SQL][test-hadoop3.2] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24783#issuecomment-501548338 @cloud-fan cool, this sounds good to me too! I can also bring my PR back to a state similar to before I merged https://github.com/IvanVergiliev/spark/pull/2/files - with `filter` and `build` in separate functions - and then @gengliangwang can followup with the change to reuse `build` for determining whether leaf nodes are convertible? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] IvanVergiliev commented on a change in pull request #24068: [SPARK-27105][SQL] Optimize away exponential complexity in ORC predicate conversion
IvanVergiliev commented on a change in pull request #24068: [SPARK-27105][SQL]
Optimize away exponential complexity in ORC predicate conversion
URL: https://github.com/apache/spark/pull/24068#discussion_r293204913
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/FilterPushdownBenchmark.scala
##
@@ -362,6 +394,13 @@ object FilterPushdownBenchmark extends BenchmarkBase with
SQLHelper {
}
runBenchmark(s"Pushdown benchmark with many filters") {
+ // This benchmark and the next one are similar in that they both test
predicate pushdown
+ // where the filter itself is very large. There have been cases where
the filter conversion
+ // would take minutes to hours for large filters due to it being
implemented with exponential
+ // complexity in the height of the filter tree.
+ // The difference between these two benchmarks is that this one
benchmarks pushdown with a
+ // large string filter (`a AND b AND c ...`), whereas the next one
benchmarks pushdown with
+ // a large Column-based filter (`col(a) || (col(b) || (col(c)...))`).
Review comment:
@cloud-fan the two go through different code paths. The string-based one was
added in https://github.com/apache/spark/pull/22313 , but it doesn't expose the
slowness when passing a `Column` filter directly. That is, the string-based one
was fast before this PR. The one this PR fixes is specifically when passing in
a `Column` directly to something like `df.filter(Column)`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
AmplabJenkins removed a comment on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501547038 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
AmplabJenkins removed a comment on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501547045 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106450/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
AmplabJenkins commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501547045 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106450/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
AmplabJenkins commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501547038 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
SparkQA removed a comment on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501524364 **[Test build #106450 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106450/testReport)** for PR 24841 at commit [`14203f5`](https://github.com/apache/spark/commit/14203f53604ce0b63a964e8c11288c3f9014792d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
SparkQA commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501546738 **[Test build #106450 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106450/testReport)** for PR 24841 at commit [`14203f5`](https://github.com/apache/spark/commit/14203f53604ce0b63a964e8c11288c3f9014792d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #24699: [SPARK-27666][CORE] Do not release lock while TaskContext already completed
jiangxb1987 commented on issue #24699: [SPARK-27666][CORE] Do not release lock while TaskContext already completed URL: https://github.com/apache/spark/pull/24699#issuecomment-501545350 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #24857: [MINOR][CORE] Remove an unused variable in SparkSubmt.scala
dongjoon-hyun edited a comment on issue #24857: [MINOR][CORE] Remove an unused variable in SparkSubmt.scala URL: https://github.com/apache/spark/pull/24857#issuecomment-501543348 Ur, thank you for the update, but let's remove `unused imports` stuff. You can get reviews later in another PR. It's good to have but sometime it's on the edge due to the intrusiveness. Also, it's beyond the scope of PR title. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24857: [MINOR][CORE] Remove an unused variable in SparkSubmt.scala
dongjoon-hyun commented on issue #24857: [MINOR][CORE] Remove an unused variable in SparkSubmt.scala URL: https://github.com/apache/spark/pull/24857#issuecomment-501543348 Ur, thank you for the update, but let's remove `unused imports` stuff. You can get reviews later in another PR. It's good to have but sometime it's on the edge due to the intrusiveness. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 edited a comment on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
Ngone51 edited a comment on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501538461 @viirya IIUC, executor set up the resources from what the worker assigned to it. For example, worker could "split" its own resources to some separate resource files according to Masters' requirements for executors. Then, executor could set up from corresponding resource file when it starts up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
SparkQA commented on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL URL: https://github.com/apache/spark/pull/24706#issuecomment-501538447 **[Test build #106454 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106454/testReport)** for PR 24706 at commit [`5688cb4`](https://github.com/apache/spark/commit/5688cb47b5171fcb590819c101dacfb73ffde356). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
Ngone51 commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501538461 @viirya IIUC, executor set up the resources from what the worker assigned to it. For example, worker could "split" its own resources to some separate resource files according to Masters' requirements for executors. Then, executors could set up from those resource files when it starts up. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
AmplabJenkins commented on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL URL: https://github.com/apache/spark/pull/24706#issuecomment-501538146 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
AmplabJenkins commented on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL URL: https://github.com/apache/spark/pull/24706#issuecomment-501538151 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11697/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
AmplabJenkins removed a comment on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL URL: https://github.com/apache/spark/pull/24706#issuecomment-501538151 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11697/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
AmplabJenkins removed a comment on issue #24706: [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL URL: https://github.com/apache/spark/pull/24706#issuecomment-501538146 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
AmplabJenkins removed a comment on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501536975 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11696/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
AmplabJenkins removed a comment on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501536973 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
AmplabJenkins commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501536975 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11696/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
AmplabJenkins commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501536973 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
SparkQA commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501536110 **[Test build #106453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106453/testReport)** for PR 24826 at commit [`614013e`](https://github.com/apache/spark/commit/614013e0b0e87ef71a082a7ac269244157025aad). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
HyukjinKwon commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501535952 should be ready for a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs
HyukjinKwon commented on issue #24826: [SPARK-27870][SQL][PYTHON] Add a runtime buffer size configuration for Pandas UDFs URL: https://github.com/apache/spark/pull/24826#issuecomment-501535903 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
SparkQA commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501530960 **[Test build #106452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106452/testReport)** for PR 24792 at commit [`46c12d8`](https://github.com/apache/spark/commit/46c12d8896ef1022ca3e3ee6c2b21a376ae7f378). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
AmplabJenkins removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501530655 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11695/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
AmplabJenkins commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501530655 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11695/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
AmplabJenkins removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501530649 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
AmplabJenkins commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501530649 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
Ngone51 commented on a change in pull request #24841: [SPARK-27369][CORE] Setup
resources when Standalone Worker starts up
URL: https://github.com/apache/spark/pull/24841#discussion_r293187804
##
File path: core/src/test/scala/org/apache/spark/deploy/worker/WorkerSuite.scala
##
@@ -60,7 +60,7 @@ class WorkerSuite extends SparkFunSuite with Matchers with
BeforeAndAfter {
val securityMgr = new SecurityManager(conf)
val rpcEnv = RpcEnv.create("test", "localhost", 12345, conf, securityMgr)
_worker = new Worker(rpcEnv, 5, 20, 1234 * 5,
Array.fill(1)(RpcAddress("1.2.3.4", 1234)),
- "Worker", "/tmp", conf, securityMgr, shuffleServiceSupplier)
+ "Worker", "/tmp", conf, securityMgr, None, Map.empty,
shuffleServiceSupplier)
Review comment:
We need to get Master in to test wether Worker sets up the resources
correctly, since Worker could not start up without a living Master. Maybe, we
could cover this in the following JIRA task, which would sharing resource info
between Master and Worker.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
SparkQA commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501525743 **[Test build #106451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106451/testReport)** for PR 24792 at commit [`9931eb6`](https://github.com/apache/spark/commit/9931eb63c0715ba190717a593ce51b949d5355b2). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
AmplabJenkins removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501525433 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
AmplabJenkins removed a comment on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501525403 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
AmplabJenkins removed a comment on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501525439 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11694/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
AmplabJenkins removed a comment on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501525405 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11693/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
AmplabJenkins commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501525439 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11694/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
AmplabJenkins commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501525433 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
AmplabJenkins commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501525403 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
AmplabJenkins commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501525405 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11693/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
Ngone51 commented on a change in pull request #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#discussion_r293185470 ## File path: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala ## @@ -220,6 +225,38 @@ private[deploy] class Worker( metricsSystem.getServletHandlers.foreach(webUi.attachHandler) } + // TODO if we're starting up multi workers under the same host, discovery script won't work. Review comment: Yeah, technically we can do, but I think it may be more troublesome than `resourceFile` to do it. Document is good, have done in ade97c2. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up
SparkQA commented on issue #24841: [SPARK-27369][CORE] Setup resources when Standalone Worker starts up URL: https://github.com/apache/spark/pull/24841#issuecomment-501524364 **[Test build #106450 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106450/testReport)** for PR 24841 at commit [`14203f5`](https://github.com/apache/spark/commit/14203f53604ce0b63a964e8c11288c3f9014792d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen commented on issue #24791: [K8S] Update Comment for k8s applicationId
srowen commented on issue #24791: [K8S] Update Comment for k8s applicationId URL: https://github.com/apache/spark/pull/24791#issuecomment-501524503 Per https://github.com/apache/spark/pull/24791#discussion_r291568279 this looks incorrect anyway. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] srowen closed pull request #24791: [K8S] Update Comment for k8s applicationId
srowen closed pull request #24791: [K8S] Update Comment for k8s applicationId URL: https://github.com/apache/spark/pull/24791 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #24854: [SPARK-28031][PySpark][TEST] Improve doctest on over function of Column
HyukjinKwon closed pull request #24854: [SPARK-28031][PySpark][TEST] Improve doctest on over function of Column URL: https://github.com/apache/spark/pull/24854 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table
beliefer commented on issue #24792: [SPARK-27953][SQL] Save default constraint with Column into table properties when create Hive table URL: https://github.com/apache/spark/pull/24792#issuecomment-501520317 > Hi, @beliefer . For the umbrella issue, the subtask JIRA ID is enough for the title. OK. Thanks for your reminder. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24854: [SPARK-28031][PySpark][TEST] Improve doctest on over function of Column
HyukjinKwon commented on issue #24854: [SPARK-28031][PySpark][TEST] Improve doctest on over function of Column URL: https://github.com/apache/spark/pull/24854#issuecomment-501520124 Merged to master and branch-2.4. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24779: [SPARK-27929][SQL] Make percentile function receive frq of double
dongjoon-hyun commented on issue #24779: [SPARK-27929][SQL] Make percentile function receive frq of double URL: https://github.com/apache/spark/pull/24779#issuecomment-501518005 @voidbag . - You should put [your explanation](https://github.com/apache/spark/pull/24779#issuecomment-500340705) into PR description. The comments become invisible soon during the reviews. - If then, it can be interpreted like this is not a useful feature for the most users. Do you have any reference for your proposal? > Other db engines don't seem to provide percentile function with frequency, unlike percentile aggregate function of catalyst package in Spark This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
SparkQA commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501517370 **[Test build #106449 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106449/testReport)** for PR 24741 at commit [`b8cdf6c`](https://github.com/apache/spark/commit/b8cdf6c22172585b3b3a9452d5e4d2d591ece88e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on issue #24783: [SPARK-27105][SQL][test-hadoop3.2] Optimize away exponential complexity in ORC predicate conversion
gengliangwang commented on issue #24783: [SPARK-27105][SQL][test-hadoop3.2] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24783#issuecomment-501517200 @dongjoon-hyun Yes, I think so. If it is OK, I am also fine with merging this one directly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501517051 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11692/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501517051 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11692/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
AmplabJenkins removed a comment on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501517044 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation
AmplabJenkins commented on issue #24741: [SPARK-27322][SQL] DataSourceV2 table relation URL: https://github.com/apache/spark/pull/24741#issuecomment-501517044 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24783: [SPARK-27105][SQL][test-hadoop3.2] Optimize away exponential complexity in ORC predicate conversion
dongjoon-hyun commented on issue #24783: [SPARK-27105][SQL][test-hadoop3.2] Optimize away exponential complexity in ORC predicate conversion URL: https://github.com/apache/spark/pull/24783#issuecomment-501516314 Hi, @gengliangwang . Are you going to use this PR for the followup after #24068 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #24791: [K8S] Update Comment for k8s applicationId
dongjoon-hyun edited a comment on issue #24791: [K8S] Update Comment for k8s applicationId URL: https://github.com/apache/spark/pull/24791#issuecomment-501514591 @zhangjiajin . This is not `spark-24791`. Did you create a JIRA ID? You need to use your JIRA id for this. - https://issues.apache.org/jira/browse/SPARK-24791 (Spark Structured Streaming randomly does not process batch) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24791: [K8S] Update Comment for k8s applicationId
dongjoon-hyun commented on issue #24791: [K8S] Update Comment for k8s applicationId URL: https://github.com/apache/spark/pull/24791#issuecomment-501514591 @zhangjiajin . This is not `spark-24791`. Did you create a JIRA ID? - https://issues.apache.org/jira/browse/SPARK-24791 (Spark Structured Streaming randomly does not process batch) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24792: [SPARK-27953][SQL] Add new feature create table could specify column with default constraint
dongjoon-hyun commented on issue #24792: [SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501513881 Hi, @beliefer . For the umbrella issue, the subtask JIRA ID is enough. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #24792: [SPARK-27953][SQL] Add new feature create table could specify column with default constraint
dongjoon-hyun edited a comment on issue #24792: [SPARK-27953][SQL] Add new feature create table could specify column with default constraint URL: https://github.com/apache/spark/pull/24792#issuecomment-501513881 Hi, @beliefer . For the umbrella issue, the subtask JIRA ID is enough for the title. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals
AmplabJenkins removed a comment on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals URL: https://github.com/apache/spark/pull/24593#issuecomment-501512105 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106447/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals
AmplabJenkins commented on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals URL: https://github.com/apache/spark/pull/24593#issuecomment-501512105 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106447/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals
AmplabJenkins commented on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals URL: https://github.com/apache/spark/pull/24593#issuecomment-501512101 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals
AmplabJenkins removed a comment on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals URL: https://github.com/apache/spark/pull/24593#issuecomment-501512101 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals
SparkQA removed a comment on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals URL: https://github.com/apache/spark/pull/24593#issuecomment-501475807 **[Test build #106447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106447/testReport)** for PR 24593 at commit [`93241b3`](https://github.com/apache/spark/commit/93241b30eeb071d575142b26db36c40cad5b93b6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals
SparkQA commented on issue #24593: [SPARK-27692][SQL] Add new optimizer rule to evaluate the deterministic scala udf only once if all inputs are literals URL: https://github.com/apache/spark/pull/24593#issuecomment-501511812 **[Test build #106447 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106447/testReport)** for PR 24593 at commit [`93241b3`](https://github.com/apache/spark/commit/93241b30eeb071d575142b26db36c40cad5b93b6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] imback82 commented on issue #24857: [MINOR][CORE] Remove an unused variable in SparkSubmt.scala
imback82 commented on issue #24857: [MINOR][CORE] Remove an unused variable in SparkSubmt.scala URL: https://github.com/apache/spark/pull/24857#issuecomment-501510491 > * We use `[MINOR]` tag if we don't have SPARK JIRAs. Thanks. I will follow this from the next PR. > * Could you find more instances in `core` module? We usually recommend to do this kind of tasks module by module. I compiled with `-Xlint:unused` for the `core` module. I fixed most of the warnings for unused local variable and unused imports (there were some false positives since scaladoc was referring to the imports). I didn't fix unused private variable, private method, and private default arguments; there are about 50 of them and I can fix them in a separate PR if needed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #24797: Detecting key in map type when value type is complex
dongjoon-hyun commented on issue #24797: Detecting key in map type when value type is complex URL: https://github.com/apache/spark/pull/24797#issuecomment-501508824 Gentle ping, @WitJakuczun . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24749: [WIP][SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions
AmplabJenkins commented on issue #24749: [WIP][SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions URL: https://github.com/apache/spark/pull/24749#issuecomment-501507862 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
