Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14322
Yeah, I did not see a noticeable performance difference based on the local
tests I did. Based on the outputs of whole-stage code gen, the number of
instructions is less. Thus, I think it helps a
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/14322
how much benefit can we get by avoiding scan partition columns? Seems that
we just parse the directory string to get the partition values, no IO is needed.
---
If your project is set up for it, y
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14322
cc @marmbrus @cloud-fan @liancheng After history checking, most of codes
are done by you. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14322
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62743/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/14322
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14322
**[Test build #62743 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62743/consoleFull)**
for PR 14322 at commit
[`c1ff046`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/14322
**[Test build #62743 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62743/consoleFull)**
for PR 14322 at commit
[`c1ff046`](https://github.com/apache/spark/commit/c
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14322
**After the PR changes**, the whole-stage codegen output is like:
```JAVA
== Subtree 1 / 1 ==
*Scan json [value#37L] Format: JSON, InputPaths:
file:/private/var/folders/4b/sgmfldk15js4
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/14322
**Before the PR changes**, the whole-stage codegen output is like:
```JAVA
== Subtree 1 / 1 ==
*Project [value#37L]
+- *Scan json [value#37L,p1#39,p2#40,p3#41] Format: JSON, InputPa