[GitHub] spark issue #14322: [SPARK-16689] [SQL] FileSourceStrategy: Pruning Partitio...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14322 Yeah, I did not see a noticeable performance difference based on the local tests I did. Based on the outputs of whole-stage code gen, the number of instructions is less. Thus, I think it helps a little bit but it does not affect the overall performance. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14322: [SPARK-16689] [SQL] FileSourceStrategy: Pruning Partitio...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14322 how much benefit can we get by avoiding scan partition columns? Seems that we just parse the directory string to get the partition values, no IO is needed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14322: [SPARK-16689] [SQL] FileSourceStrategy: Pruning Partitio...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14322 cc @marmbrus @cloud-fan @liancheng After history checking, most of codes are done by you. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14322: [SPARK-16689] [SQL] FileSourceStrategy: Pruning Partitio...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14322 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62743/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14322: [SPARK-16689] [SQL] FileSourceStrategy: Pruning Partitio...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14322 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14322: [SPARK-16689] [SQL] FileSourceStrategy: Pruning Partitio...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14322 **[Test build #62743 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62743/consoleFull)** for PR 14322 at commit [`c1ff046`](https://github.com/apache/spark/commit/c1ff0465815f6adefb2b29c2973c9bc63aa13623). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14322: [SPARK-16689] [SQL] FileSourceStrategy: Pruning Partitio...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14322 **[Test build #62743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62743/consoleFull)** for PR 14322 at commit [`c1ff046`](https://github.com/apache/spark/commit/c1ff0465815f6adefb2b29c2973c9bc63aa13623). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14322: [SPARK-16689] [SQL] FileSourceStrategy: Pruning Partitio...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14322
**After the PR changes**, the whole-stage codegen output is like:
```JAVA
== Subtree 1 / 1 ==
*Scan json [value#37L] Format: JSON, InputPaths:
file:/private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-8ac18be7-053f-4498-bf59-5ed87...,
PushedFilters: [], ReadSchema: struct
Generated code:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIterator(references);
/* 003 */ }
/* 004 */
/* 005 */ final class GeneratedIterator extends
org.apache.spark.sql.execution.BufferedRowIterator {
/* 006 */ private Object[] references;
/* 007 */ private org.apache.spark.sql.execution.metric.SQLMetric
scan_numOutputRows;
/* 008 */ private scala.collection.Iterator scan_input;
/* 009 */
/* 010 */ public GeneratedIterator(Object[] references) {
/* 011 */ this.references = references;
/* 012 */ }
/* 013 */
/* 014 */ public void init(int index, scala.collection.Iterator inputs[])
{
/* 015 */ partitionIndex = index;
/* 016 */ this.scan_numOutputRows =
(org.apache.spark.sql.execution.metric.SQLMetric) references[0];
/* 017 */ scan_input = inputs[0];
/* 018 */ }
/* 019 */
/* 020 */ protected void processNext() throws java.io.IOException {
/* 021 */ while (scan_input.hasNext()) {
/* 022 */ InternalRow scan_row = (InternalRow) scan_input.next();
/* 023 */ scan_numOutputRows.add(1);
/* 024 */ append(scan_row);
/* 025 */ if (shouldStop()) return;
/* 026 */ }
/* 027 */ }
/* 028 */ }
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14322: [SPARK-16689] [SQL] FileSourceStrategy: Pruning Partitio...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14322
**Before the PR changes**, the whole-stage codegen output is like:
```JAVA
== Subtree 1 / 1 ==
*Project [value#37L]
+- *Scan json [value#37L,p1#39,p2#40,p3#41] Format: JSON, InputPaths:
file:/private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-f7a4294a-2e1b-4f44-9ebb-1a5eb...,
PushedFilters: [], ReadSchema: struct
Generated code:
/* 001 */ public Object generate(Object[] references) {
/* 002 */ return new GeneratedIterator(references);
/* 003 */ }
/* 004 */
/* 005 */ final class GeneratedIterator extends
org.apache.spark.sql.execution.BufferedRowIterator {
/* 006 */ private Object[] references;
/* 007 */ private org.apache.spark.sql.execution.metric.SQLMetric
scan_numOutputRows;
/* 008 */ private scala.collection.Iterator scan_input;
/* 009 */ private UnsafeRow project_result;
/* 010 */ private
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder project_holder;
/* 011 */ private
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
project_rowWriter;
/* 012 */
/* 013 */ public GeneratedIterator(Object[] references) {
/* 014 */ this.references = references;
/* 015 */ }
/* 016 */
/* 017 */ public void init(int index, scala.collection.Iterator inputs[])
{
/* 018 */ partitionIndex = index;
/* 019 */ this.scan_numOutputRows =
(org.apache.spark.sql.execution.metric.SQLMetric) references[0];
/* 020 */ scan_input = inputs[0];
/* 021 */ project_result = new UnsafeRow(1);
/* 022 */ this.project_holder = new
org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(project_result,
0);
/* 023 */ this.project_rowWriter = new
org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(project_holder,
1);
/* 024 */ }
/* 025 */
/* 026 */ protected void processNext() throws java.io.IOException {
/* 027 */ while (scan_input.hasNext()) {
/* 028 */ InternalRow scan_row = (InternalRow) scan_input.next();
/* 029 */ scan_numOutputRows.add(1);
/* 030 */ boolean scan_isNull4 = scan_row.isNullAt(0);
/* 031 */ long scan_value4 = scan_isNull4 ? -1L :
(scan_row.getLong(0));
/* 032 */ project_rowWriter.zeroOutNullBytes();
/* 033 */
/* 034 */ if (scan_isNull4) {
/* 035 */ project_rowWriter.setNullAt(0);
/* 036 */ } else {
/* 037 */ project_rowWriter.write(0, scan_value4);
/* 038 */ }
/* 039 */ append(project_result);
/* 040 */ if (shouldStop()) return;
/* 041 */ }
/* 042 */ }
/* 043 */ }
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
