Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
> https://issues.apache.org/jira/browse/SPARK-25879
>
> If we select a nested field and a top level field, the schema pruning
will fail. Here is the reproducible test,
> ...
Hi
Github user dbtsai commented on the issue:
https://github.com/apache/spark/pull/21320
cc @viirya
If we select a nested field and a top level field, the schema pruning will
fail. Here is the reproducible test,
```scala
testSchemaPruning("select a single complex
Github user Gauravshah commented on the issue:
https://github.com/apache/spark/pull/21320
backported to 2.3.2 just in case somebody needs it.
https://github.com/Gauravshah/spark/tree/branch-2.3_SPARK-4502 Thanks @mallman
---
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
Hi @Gauravshah. That branch has diverged substantially from whatâs in
master. Right now Iâm preparing a PR to address a problem with the current
implementation in master, but Iâm on holiday
Github user Gauravshah commented on the issue:
https://github.com/apache/spark/pull/21320
@mallman any way I can help pull in rest of the changes from your original
PR (https://github.com/apache/spark/pull/16578) for next release ?
---
Github user dbtsai commented on the issue:
https://github.com/apache/spark/pull/21320
A [PR](https://github.com/apache/spark/pull/22357) fixing the issue I
mentioned above is provided by [Liang-Chi Hsieh](https://github.com/viirya).
Thank you for the quick and clean solution.
---
Github user dbtsai commented on the issue:
https://github.com/apache/spark/pull/21320
Thanks @mallman for schema pruning work which will be a big win in our
pattern of accessing our data.
I'm testing this new feature, and find `where clause` on the selected
nested column can
Github user ajacques commented on the issue:
https://github.com/apache/spark/pull/21320
@mallman Glad to see this got merged in. Thanks for all of your work
pushing through. I'm looking forward to the next phase. Please let me know if I
can help again. I did notice that window
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
Thanks everyone for your contributions, support and patience. It's been a
journey and a half, and I'm excited for the future. I will open a follow-on PR
to address the current known failure
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/21320
Sure, @gatorsmile !
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user IgorBerman commented on the issue:
https://github.com/apache/spark/pull/21320
@mallman, wanted to say huge thanks for your work! this is great step
forward.
---
-
To unsubscribe, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21320
Thanks! Merged to master.
BTW, we can keep thinking whether there are other better solutions for
nested column pruning.
Also cc @dongjoon-hyun If you are interested in the
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95185/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #95185 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95185/testReport)**
for PR 21320 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21320
Seems fine to me too for the similar reasons.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21320
LGTM, as I explained above.
https://github.com/apache/spark/pull/21320#issuecomment-415526369
Thanks for your patience and great work! @mallman Sorry, it takes two years
to merge the
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #95185 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95185/testReport)**
for PR 21320 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
> @mallman Could you remove the changes made in ParquetRowConverter.scala
and also turn off spark.sql.nestedSchemaPruning.enabled by default in this PR?
Done.
---
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21320
The feature has already been developed for almost two years. I am feeling
sorry to merge it in Spark 2.4 release. Personally, I think we should not block
merging this PR to Spark 2.4 release,
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21320
let me take a look this today. Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
@gatorsmile Any concerns about merging this PR at this point?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95041/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #95041 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95041/testReport)**
for PR 21320 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #95041 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95041/testReport)**
for PR 21320 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
@gatorsmile How does this look?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
> Add some test cases when turning on spark.sql.caseSensitive?
Will do.
---
-
To unsubscribe, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21320
Add some test cases when turning on `spark.sql.caseSensitive`?
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94987/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94987 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94987/testReport)**
for PR 21320 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94987 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94987/testReport)**
for PR 21320 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21320
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21320
@mallman Thanks!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
> Try this when spark.sql.nestedSchemaPruning.enabled is on?
I don't think this will be difficult to fix. I'm working on it now and will
add relevant test coverage.
---
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
> Try this when spark.sql.nestedSchemaPruning.enabled is on?
This is a case-sensitivity issue (obviously). I'll get to the root of it.
Thanks.
---
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21320
Try this when `spark.sql.nestedSchemaPruning.enabled` is on?
```SQL
withTable("t1") {
spark.sql(
"""
|Create table t1 (`id` INT,`CoL1` STRING,
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94969/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94969 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94969/testReport)**
for PR 21320 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94971/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94971 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94971/testReport)**
for PR 21320 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94971 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94971/testReport)**
for PR 21320 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94969 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94969/testReport)**
for PR 21320 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94915/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94915 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94915/testReport)**
for PR 21320 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94915 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94915/testReport)**
for PR 21320 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21320
I said either way works fine. It doesn't matter which way we go. Better
close one of them if the approach is the same and both PRs are active.
---
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
> I see no point of leaving this PR open.
I don't agree with you on that point, and I've expressed my view in
https://github.com/apache/spark/pull/21889#issuecomment-413655304.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #4277 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4277/testReport)**
for PR 21320 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #4277 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4277/testReport)**
for PR 21320 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21320
@mallman, can you close this and put some efforts there in
https://github.com/apache/spark/pull/21889? I see no point of leaving this PR
open.
---
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
>> Hello, we've been using your patch at Stripe and we've found something
that looks like a new bug:
>
> Thank you for sharing this, @xinxin-stripe. This is very helpful. I will
investigate
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
> Hello, we've been using your patch at Stripe and we've found something
that looks like a new bug:
Thank you for sharing this, @xinxin-stripe. This is very helpful. I will
investigate and
Github user xinxin-stripe commented on the issue:
https://github.com/apache/spark/pull/21320
Hello, we've been using your patch at Stripe and we've found something that
looks like a new correctness issue:
```
import spark.implicits._
case class Inner(a: String)
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
> @mallman if you're planning on making more code changes, would you be
willing to work on a shared branch or something? I've been working to
incorporate the CR comments.
No, however if
Github user ajacques commented on the issue:
https://github.com/apache/spark/pull/21320
@mallman if you're planning on making more code changes, would you be
willing to work on a shared branch or something? I've been working to
incorporate the CR comments.
---
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
>> the window of opportunity to review syntax and style in this PR closed
long ago.
> Why/when is this window closed? Who closed that?
What I wrote above is a coarse approximation of my
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21320
That's more work for @ajacques though on the other hand. Either way works
fine.
---
-
To unsubscribe, e-mail:
Github user Gauravshah commented on the issue:
https://github.com/apache/spark/pull/21320
Or @ajacques can open a PR to Mallman's branch and he can merge it. Makes
it less work work for him
---
-
To unsubscribe,
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21320
Another way is you rebase @ajacques's commit here and he push some changes
into here since you refuse to address some comments here. I still don't
understand why refuse this though. if that's
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21320
@mallman, can you close this one and put some changes into @ajacques branch
then? no point of opening duplicated changes. Since @ajacques is at least
willing to address other comments including
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21320
> the window of opportunity to review syntax and style in this PR closed
long ago.
Why/when is this window closed? Who closed that?
This project claims a set of specific styles
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
> Then should we keep this one or #21889? shall we deduplicate the efforts?
I requested to open that because this looks going to be inactive per your
comments.
As I stated before, I'll
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21320
Then should we keep this one or https://github.com/apache/spark/pull/21889?
shall we deduplicate the efforts? I requested to open that because this looks
going to be inactive per your comments.
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
> @mallman, can we close this PR? Are you willing to update here or not?
I pushed an update less than a day ago, and I intend to continue pushing
updates as needed.
---
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21320
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21320
@mallman, can we close this PR? Are you willing to update here or not?
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94521/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94521 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94521/testReport)**
for PR 21320 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21320
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94521 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94521/testReport)**
for PR 21320 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94502/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94502 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94502/testReport)**
for PR 21320 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94502 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94502/testReport)**
for PR 21320 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21320
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94476/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #94476 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94476/testReport)**
for PR 21320 at commit
1 - 100 of 258 matches
Mail list logo
