[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19129 Thank you for review, @gatorsmile , @HyukjinKwon , @maropu . In this issue, I've learned how to track the unused stuff correctly. Thank you again. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19129 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19129 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19129 **[Test build #81525 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81525/testReport)** for PR 19129 at commit [`8e3d8fe`](https://github.com/apache/spark/commit/8e3d8fe26c6bbf15e17a4b80ff8357fe870f2d46). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19129 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81525/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19129 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19129 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81523/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19129 **[Test build #81523 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81523/testReport)** for PR 19129 at commit [`40ed9ff`](https://github.com/apache/spark/commit/40ed9ffc42c976f2efebc417e1258bc73df18236). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19129 **[Test build #81525 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81525/testReport)** for PR 19129 at commit [`8e3d8fe`](https://github.com/apache/spark/commit/8e3d8fe26c6bbf15e17a4b80ff8357fe870f2d46). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19129 **[Test build #81523 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81523/testReport)** for PR 19129 at commit [`40ed9ff`](https://github.com/apache/spark/commit/40ed9ffc42c976f2efebc417e1258bc73df18236). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19129 The PR title resolved two issues under title `[SPARK-15639][SPARK-16321][SQL] Push down filter at RowGroups level for parquet reader` I'll add like the following. Is it enough? ``` - From Spark 2.0.1, `spark.sql.parquet.cacheMetadata` is no longer used. See [SPARK-16321](https://issues.apache.org/jira/browse/SPARK-16321) and [SPARK-15639](https://issues.apache.org/jira/browse/SPARK-16321) for details. ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19129 Thank you! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19129 ` SQL 1.6 to 2.0` sounds good to me. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19129 Or, should I made `Upgrading From Spark SQL 2.2 to 2.3`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19129 It's marked as 2.0.1 and 2.1.0 with the following commit logs. ``` ~/s/spark-2.2:branch-2.0$ git log --oneline | grep SPARK-15639 977fbbfcae [SPARK-15639] [SPARK-16321] [SQL] Push down filter at RowGroups level for parquet reader 91dffcabde Revert "[SPARK-15639][SQL] Try to push down filter at RowGroups level for parquet reader" 7d6bd11964 [SPARK-15639][SQL] Try to push down filter at RowGroups level for parquet reader ``` Which section is proper? - Upgrading From Spark SQL 1.6 to 2.0 - Upgrading From Spark SQL 2.0 to 2.1 I think it's `Upgrading From Spark SQL 1.6 to 2.0`, effectively. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19129 Sure, @gatorsmile . BTW, I searched more and updated the PR description. It's [SPARK-15639](https://github.com/apache/spark/pull/13701) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19129 Please document it in the migration guides. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19129 So, it's removed before 2.0.0. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19129 Wow! Thank you, @HyukjinKwon ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19129 The last code looks removed in https://github.com/apache/spark/commit/678b96e77bf77a64b8df14b19db5a3bb18febfe3 and this option looks introduced in https://github.com/apache/spark/commit/9eb74c7d2cbe127dd4c32bf1a8318497b2fb55b6. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19129 Sure, I'll. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19129 Could you check the change history when we remove the usage of this SQLConf? It sounds like we do not have a test case coverage for this in the past. We did not realize it when removing the usage. We also need to update the migration notes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/19129 Thank you for your review and approval, @HyukjinKwon ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/19129 oh, yea. I got you. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19129 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19129 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81401/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19129 **[Test build #81401 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81401/testReport)** for PR 19129 at commit [`3b305d0`](https://github.com/apache/spark/commit/3b305d067424a83026ad388fb7d50018099070e7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/19129 I roughly checked other options around parquet and I probably found `parquetOutputCommitterClass` in `SQLConf` also is not used now? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19129: [SPARK-13656][SQL] Delete spark.sql.parquet.cacheMetadat...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19129 **[Test build #81401 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81401/testReport)** for PR 19129 at commit [`3b305d0`](https://github.com/apache/spark/commit/3b305d067424a83026ad388fb7d50018099070e7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
