[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2017-06-26 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/14957 OK, I'll close this and just use it in our internal env, thanks all guys's suggestion and review work. Next we may try more complex scenario of this. --- If your project is set up for it, you

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2017-06-18 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14957 @xuanyuanking, let's close this and help review #16578 if you agree on the comments above. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2017-05-22 Thread Gauravshah
Github user Gauravshah commented on the issue: https://github.com/apache/spark/pull/14957 @saulshanabrook looks like #16578 is a superset, trying to invest in that pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2017-05-17 Thread saulshanabrook
Github user saulshanabrook commented on the issue: https://github.com/apache/spark/pull/14957 @xuanyuanking Have you determined if the functionality provided here is superseded by #16578? I am trying to figure out which PR to help out on since I need this feature as well. --- If you

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67781/ Test PASSed. ---

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67781 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67781/consoleFull)** for PR 14957 at commit [`d093c82`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-30 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67781 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67781/consoleFull)** for PR 14957 at commit [`d093c82`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67564/ Test FAILed. ---

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-26 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67564 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67564/consoleFull)** for PR 14957 at commit [`d9aa397`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67564/consoleFull)** for PR 14957 at commit [`d9aa397`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67550 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67550/consoleFull)** for PR 14957 at commit [`5697911`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67550/ Test FAILed. ---

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67550 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67550/consoleFull)** for PR 14957 at commit [`5697911`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67515/ Test PASSed. ---

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67515/consoleFull)** for PR 14957 at commit [`92ed369`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67515 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67515/consoleFull)** for PR 14957 at commit [`92ed369`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67410/ Test PASSed. ---

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67410/consoleFull)** for PR 14957 at commit [`ab8f5ec`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67410/consoleFull)** for PR 14957 at commit [`ab8f5ec`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-21 Thread davies
Github user davies commented on the issue: https://github.com/apache/spark/pull/14957 Together with this one, we should have a optimizer rule that could 1) extract GetStructField (and others) and push that down closer to the data source, or 2) flatten all the nested field in data sou

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67316/ Test PASSed. ---

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67316 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67316/consoleFull)** for PR 14957 at commit [`23465ba`](https://github.com/apache/spark/commit/

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-20 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14957 **[Test build #67316 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67316/consoleFull)** for PR 14957 at commit [`23465ba`](https://github.com/apache/spark/commit/2

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-20 Thread liancheng
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/14957 add to whitelist --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, o

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-20 Thread liancheng
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/14957 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, o

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-10-07 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14957 This would be good to include in Spark 2.1... cc @davies and @liancheng --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-09-09 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/14957 @liancheng @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-09-07 Thread xuanyuanking
Github user xuanyuanking commented on the issue: https://github.com/apache/spark/pull/14957 (Thank you for your comments and help me a lot :) ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not ha

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-09-07 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14957 (BTW, I just left some comments because I like some codes in this path. You can wait for committer's review.) --- If your project is set up for it, you can reply to this email and have your rep

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-09-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14957 Also, it seems you might need to update your PR description. It seems the last commit you just pushed acts differently with your PR description. In addition, maybe you would need to fix the titl

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-09-06 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14957 Could you please check out related tests pass locally? It seems it affects all other data sources. Also, I am not sure of the approach here. Marking nested fields by modifying column na

[GitHub] spark issue #14957: [SPARK-4502][SQL]Support parquet nested struct pruning a...

2016-09-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14957 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat