[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19464 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82752/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82752 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82752/testReport)** for PR 19464 at commit [`a6818b6`](https://github.com/apache/spark/commit/a6818b60adef7bec35b002846a3a504ae53dd9f9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19464 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82752/testReport)** for PR 19464 at commit [`a6818b6`](https://github.com/apache/spark/commit/a6818b60adef7bec35b002846a3a504ae53dd9f9). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82726/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82726/testReport)** for PR 19464 at commit [`534d8fb`](https://github.com/apache/spark/commit/534d8fbcd7dfbdc9af06a4d926f6a353f429fce8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82726/testReport)** for PR 19464 at commit [`534d8fb`](https://github.com/apache/spark/commit/534d8fbcd7dfbdc9af06a4d926f6a353f429fce8). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82716/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82716/testReport)** for PR 19464 at commit [`25f98d0`](https://github.com/apache/spark/commit/25f98d0d89e4566339d9ba7701975af4e175c918). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82716 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82716/testReport)** for PR 19464 at commit [`25f98d0`](https://github.com/apache/spark/commit/25f98d0d89e4566339d9ba7701975af4e175c918). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82696/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82696 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82696/testReport)** for PR 19464 at commit [`527b367`](https://github.com/apache/spark/commit/527b367ea482261f6afbb7cdf339495f77c4e7f2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82694/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82694/testReport)** for PR 19464 at commit [`4dcfd83`](https://github.com/apache/spark/commit/4dcfd83612661ce47e8c2c1f33590c61dfe4e473). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82696/testReport)** for PR 19464 at commit [`527b367`](https://github.com/apache/spark/commit/527b367ea482261f6afbb7cdf339495f77c4e7f2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82694/testReport)** for PR 19464 at commit [`4dcfd83`](https://github.com/apache/spark/commit/4dcfd83612661ce47e8c2c1f33590c61dfe4e473). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19464 I can't think of any downside, but it's always safe to avoid behavior changes. LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82672/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82672 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82672/testReport)** for PR 19464 at commit [`31a5d30`](https://github.com/apache/spark/commit/31a5d303f91839124f8957f75d4077be5410524c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82672 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82672/testReport)** for PR 19464 at commit [`31a5d30`](https://github.com/apache/spark/commit/31a5d303f91839124f8957f75d4077be5410524c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82658/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19464 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82658/testReport)** for PR 19464 at commit [`cf0c350`](https://github.com/apache/spark/commit/cf0c350daf12ce80fc781fd17fd15506d83c6d02). * This patch **fails due to an unknown error code, -9**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19464 **[Test build #82658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82658/testReport)** for PR 19464 at commit [`cf0c350`](https://github.com/apache/spark/commit/cf0c350daf12ce80fc781fd17fd15506d83c6d02). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19464 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19464 I think the optimisation by `spark.sql.files.maxPartitionBytes` sql specific conf includes this concept in `FileScanRDD` and it looks already partially doing it in combining input splits. I'd suggest to avoid putting this conf in `FileScanRDD`, for now, if I didn't miss something. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19464 IIUC this issue also existed in `NewHadoopRDD` and `FileScanRDD` (possibly), we'd better also fix them. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19464 Interesting. On the one hand I don't like adding yet another flag that changes behavior, when the user often can't meaningfully decide to set it. There is probably no value in processing an empty partition, sure. Then again it does change behavior slightly, and I wonder if that impacts assumptions that apps rely on somehow. If there's no reason to expect downside, we could do this in Spark 3.x, or make the change now but yes introduce a flag as a safety valve to go back to old behavior, leaving the default to true. But first are there any known impacts to skipping the empty partitions? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19464: [SPARK-22233] [core] Allow user to filter out empty spli...
Github user liutang123 commented on the issue: https://github.com/apache/spark/pull/19464 @kiszk Any other suggestions an can ti PR be merged? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
