Github user dbtsai commented on the issue:
https://github.com/apache/spark/pull/22357
Thanks all again. Merged into 2.4 branch and master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22357
LGTM from me too.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95950/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95950 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95950/testReport)**
for PR 22357 at commit
Github user dbtsai commented on the issue:
https://github.com/apache/spark/pull/22357
LGTM.
Thank you all for participating the discussion. @cloud-fan and @gatorsmile,
do you have any further comment? If not, I would like to merge it tomorrow into
both master and rc branch
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/22357
> FYI, @mallman I'm working on having ParquetFilter to support
IsNotNull(employer.id) to be pushed into parquet reader.
That would be pretty cool.
---
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/22357
And FYI this is the Jira issue I promised in
https://github.com/apache/spark/pull/22357#issuecomment-419940228
yesterday: https://issues.apache.org/jira/browse/SPARK-25407.
---
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/22357
This LGTM. I'm not going to submit a PR for my approach to this problem.
Thanks @viirya!
---
-
To unsubscribe, e-mail:
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/22357
FYI, the PR I previously mentioned about fixing the use of `withSQLConf` is
#22394.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95950 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95950/testReport)**
for PR 22357 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22357
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95945/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95945 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95945/testReport)**
for PR 22357 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/22357
I have some bad news. The methods `testSchemaPruning` and
`testMixedCasePruning` do not set the configuration settings as expected.
Fixing that reveals 6 failing tests for the mixed case tests. One
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95945 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95945/testReport)**
for PR 22357 at commit
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/22357
@viirya Please amend
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95931/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95931 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95931/testReport)**
for PR 22357 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95931 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95931/testReport)**
for PR 22357 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22357
Can anyone point me out if there are non addressed comments or problems
here? Looks pretty good to me. I think this is rather a bandaid, small and safe
fix to get into branch-2.4.
---
Github user dbtsai commented on the issue:
https://github.com/apache/spark/pull/22357
FYI, @mallman I'm working on having `ParquetFilter` to support
`IsNotNull(employer.id)` to be pushed into parquet reader.
---
-
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/22357
> FYI, per further checking code and discussion with @dbtsai regarding with
predicate pushdown, we know that predicate push down only works for primitive
types on Parquet datasource. So both
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/22357
if recall, parquet reader can have filter pushdown? only not so in spark
parquet data source?
---
-
To unsubscribe, e-mail:
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22357
FYI, per further checking code and discussion with @dbtsai regarding with
predicate pushdown, we know that predicate push down only works for primitive
types on Parquet datasource. So both
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95871/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95871 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95871/testReport)**
for PR 22357 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95884/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95884 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95884/testReport)**
for PR 22357 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22357
Btw, this PR isn't intended to address filter push down for schema pruning.
I do think it should be another one topic.
---
-
To
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22357
I just read @mallman's comment. Thanks for that. Roughly, my two cents:
> IMO, we can get closer to settling the question of relative
performance/behavior by pushing down Parquet reader
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95871 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95871/testReport)**
for PR 22357 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95884 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95884/testReport)**
for PR 22357 at commit
Github user DaimonPl commented on the issue:
https://github.com/apache/spark/pull/22357
@mallman regarding
"But how do we know that pushing down IsNotNull(employer) does not negate
that instruction? "
isn't it pretty obvious that when you read 'employer.id' with
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/22357
> @mallman It will be great that we can have this fix in 2.4 release as
this can dramatically reduce the data being read in many applications which is
the purpose of the original work.
I
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22357
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22357
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95868/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95868 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95868/testReport)**
for PR 22357 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95868 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95868/testReport)**
for PR 22357 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22357
Thanks @dbtsai and @HyukjinKwon. Your comments are addressed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user dbtsai commented on the issue:
https://github.com/apache/spark/pull/22357
cc @beettlle
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user dbtsai commented on the issue:
https://github.com/apache/spark/pull/22357
LGTM except one minor point. Thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user dbtsai commented on the issue:
https://github.com/apache/spark/pull/22357
@mallman It will be great that we can have this fix in 2.4 release as this
can dramatically reduce the data being read in many applications which is the
purpose of the original work.
As
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22357
Thanks! @mallman
For the first query, I think the query plan produced by your WIP patch is
not correct. We don't need to read the `company:struct` from `employer:struct`.
For the
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/22357
I have reconstructed my original patch for this issue, but I've discovered
it will require more work to complete. However, as part of that reconstruction
I've discovered a couple of cases where our
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/22357
Hi @viirya,
Thanks for this PR! I have an alternative implementation which I'd like to
submit for comparison. My implementation was something I removed from my
original patch.
I
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95799/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95799 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95799/testReport)**
for PR 22357 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95799 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95799/testReport)**
for PR 22357 at commit
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22357
retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95787/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95787 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95787/testReport)**
for PR 22357 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95794/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95794 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95794/testReport)**
for PR 22357 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95794 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95794/testReport)**
for PR 22357 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95787 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95787/testReport)**
for PR 22357 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/22357
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95780/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95780 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95780/testReport)**
for PR 22357 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22357
**[Test build #95780 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95780/testReport)**
for PR 22357 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22357
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/22357
cc @dbtsai
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
93 matches
Mail list logo
