[GitHub] spark issue #19634: [SPARK-22412][SQL] Fix incorrect comment in DataSourceSc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19634 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19634: [SPARK-22412][SQL] Fix incorrect comment in DataSourceSc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19634 **[Test build #3973 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3973/testReport)** for PR 19634 at commit [`b8f38a8`](https://github.com/apache/spark/commit/b8f38a8e1645b9109ec2ab7e4684dd8ead47c116). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19634: [SPARK-22412][SQL] Fix incorrect comment in DataSourceSc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19634 **[Test build #3973 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3973/testReport)** for PR 19634 at commit [`b8f38a8`](https://github.com/apache/spark/commit/b8f38a8e1645b9109ec2ab7e4684dd8ead47c116). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19634: [SPARK-22412][SQL] Fix incorrect comment in DataSourceSc...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19634 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19634: [SPARK-22412][SQL] Fix incorrect comment in DataSourceSc...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19634 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19634: [SPARK-22412][SQL] Fix incorrect comment in DataSourceSc...
Github user vgankidi commented on the issue: https://github.com/apache/spark/pull/19634 @gatorsmile I also wanted to discuss if we should consider other bin packing algorithms. According to this http://www.math.unl.edu/~s-sjessie1/203Handouts/Bin%20Packing.pdf, next fit decreasing is the least efficient of all but it is easiest to implement and has O(N) run time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19634: [SPARK-22412][SQL] Fix incorrect comment in DataSourceSc...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19634 @vgankidi Does it help the performance of our file reading? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19634: [SPARK-22412][SQL] Fix incorrect comment in DataSourceSc...
Github user vgankidi commented on the issue: https://github.com/apache/spark/pull/19634 We will end up having fewer combined splits. That reduces the number of files that the job produces and also reduces the number of tasks in the downstream jobs. In some tests I have noticed about 10% reduction in the combined splits. However, the simple implementation of FFD has O(n^2) run time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19634: [SPARK-22412][SQL] Fix incorrect comment in DataSourceSc...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19634 Fewer combined splits might not matter in this case. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
