Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
> https://issues.apache.org/jira/browse/SPARK-25879
>
> If we select a nested field and a top level field, the schema pruning
will fail. Here is the reproducible test,
> ...
Hi
Github user jainaks commented on the issue:
https://github.com/apache/spark/pull/21320
Hi @mallman, Thanks for this PR. It has huge impact on performance, when
querying the nested parquet schema. I had used the original PR#16578 and found
an issue, that it does not works well when the
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
Hi @jainaks. I can see why your query would not work. In the example you
provide, if you refer to the column as `name.First`, does your query succeed?
---
-
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/28/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3918/
Tes
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #91678 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91678/testReport)**
for PR 21320 at commit
[`8ead76e`](https://github.com/apache/spark/commit/8e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91678/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #91678 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91678/testReport)**
for PR 21320 at commit
[`8ead76e`](https://github.com/apache/spark/commit/8
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3920/
Tes
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #91679 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91679/testReport)**
for PR 21320 at commit
[`89febc8`](https://github.com/apache/spark/commit/89
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/30/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #91679 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91679/testReport)**
for PR 21320 at commit
[`89febc8`](https://github.com/apache/spark/commit/8
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91679/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/33/
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3923/
Tes
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #91684 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91684/testReport)**
for PR 21320 at commit
[`7f67ec0`](https://github.com/apache/spark/commit/7f
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #91684 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91684/testReport)**
for PR 21320 at commit
[`7f67ec0`](https://github.com/apache/spark/commit/7
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91684/
Test FAILed.
---
Github user jainaks commented on the issue:
https://github.com/apache/spark/pull/21320
@mallman It does work fine with "name.First".
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional c
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/21320
@mallman Sorry for the delay. Super busy during the Spark summit. Will
continue the code review in the next few days.
---
-
Github user jainaks commented on the issue:
https://github.com/apache/spark/pull/21320
Hi @mallman ,
I found another major issue after having this fix.
Schema:
a: struct (nullable = true)
||-- b: struct (nullable = true)
|||-- c1: string (nullable = tru
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
Hi @jainaks. Thanks for your report. Do you have the same problem running
your test with this PR?
---
-
To unsubscribe, e-mail:
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
@gatorsmile The last couple of build test failures appear to be entirely
unrelated to this PR. The error message in the one failed test reads
`org.scalatest.exceptions.TestFailedException: Unable to
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
> @mallman It does work fine with "name.First".
@jainaks What is the value of the Spark SQL configuration setting
`spark.sql.caseSensitive` when you run this query? Also, are you querying th
Github user jainaks commented on the issue:
https://github.com/apache/spark/pull/21320
> @jainaks What is the value of the Spark SQL configuration setting
spark.sql.caseSensitive when you run this query? Also, are you querying the
parquet file as part of a Hive metastore table or from
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
@mallman Yes, the issue with window functions is reproducible even with
this PR.
Can you attach a (small) parquet file I can use to test this scenario?
---
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/310/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #92070 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92070/testReport)**
for PR 21320 at commit
[`a255bcb`](https://github.com/apache/spark/commit/a2
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4204/
Tes
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92070/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #92070 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92070/testReport)**
for PR 21320 at commit
[`a255bcb`](https://github.com/apache/spark/commit/a
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test FAILed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
@gatorsmile The last build was killed by SIGKILL. Can you start a new
build, please?
---
-
To unsubscribe, e-mail: reviews-unsub
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/21320
Retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: rev
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/392/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #92191 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92191/testReport)**
for PR 21320 at commit
[`a255bcb`](https://github.com/apache/spark/commit/a2
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #92191 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92191/testReport)**
for PR 21320 at commit
[`a255bcb`](https://github.com/apache/spark/commit/a
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92191/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/444/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #92272 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92272/testReport)**
for PR 21320 at commit
[`cb858f2`](https://github.com/apache/spark/commit/cb
Github user mallman commented on the issue:
https://github.com/apache/spark/pull/21320
@gatorsmile I've removed the changes to the files as you requested. This
removes support for schema pruning on filters of queries. I've pushed the
previous revision to a new branch in our `spark-pub
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21320
**[Test build #92272 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92272/testReport)**
for PR 21320 at commit
[`cb858f2`](https://github.com/apache/spark/commit/c
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional comma
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92272/
Test PASSed.
---
201 - 258 of 258 matches
Mail list logo
