[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19702 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19702 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83728/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19702 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19702 **[Test build #83728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83728/testReport)** for PR 19702 at commit [`d848546`](https://github.com/apache/spark/commit/d8485466835fc62e3d268a98a950f38c408e3d73). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19702 LGTM pending Jenkins --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19702 **[Test build #83728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83728/testReport)** for PR 19702 at commit [`d848546`](https://github.com/apache/spark/commit/d8485466835fc62e3d268a98a950f38c408e3d73). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19702 The only failed test is SQLConfSuite which test the default value of the new config. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19702 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83722/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19702 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19702 **[Test build #83722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83722/testReport)** for PR 19702 at commit [`ba16a5e`](https://github.com/apache/spark/commit/ba16a5e6bf33f0a3b04026e7d4414c4cc7b960ee). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19702 **[Test build #83722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83722/testReport)** for PR 19702 at commit [`ba16a5e`](https://github.com/apache/spark/commit/ba16a5e6bf33f0a3b04026e7d4414c4cc7b960ee). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19702 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83673/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19702 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19702 **[Test build #83673 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83673/testReport)** for PR 19702 at commit [`180c449`](https://github.com/apache/spark/commit/180c4497ff7ca3cb5a5c7e38ce792c76af8cf31d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19702 **[Test build #83673 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83673/testReport)** for PR 19702 at commit [`180c449`](https://github.com/apache/spark/commit/180c4497ff7ca3cb5a5c7e38ce792c76af8cf31d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19702 Will review it tomorrow. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user squito commented on the issue: https://github.com/apache/spark/pull/19702 ok sorry for taking a bit of time to look at this. tl;dr I think what you are proposing here is fine. So to be honest, I'm a bit confused about the parquet versioning -- the types I was mentioned were added here: https://github.com/apache/parquet-format/commit/863875e0be3237c6aa4ed71733d54c91a51deabe#diff-0f9d1b5347959e15259da7ba8f4b6252 to parquet-format v2.4.0 I can't really tell what version of parquet-mr that corresponds to. I can at least see that parquet-mr v1.8.2 doesn't have the new types. I don't even think its master. But IIUC, the latest version of parquet-format introduces new types, independent of the old TIMESTAMP_MILLIS and TIMESTAMP_MICROS. So there won't be any compatibility issues when `isAdjustedToUTC` is added, as that will be in an entirely new type. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19702 Is it available in parquet 1.8.2? that's the version Spark currently use. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user squito commented on the issue: https://github.com/apache/spark/pull/19702 hey thanks for doing this @cloud-fan but I have a small request -- can we get another day to review how this works, especially in connection with somewhat recent changes in parquet to include a [`isAdjustedToUTC`](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L271)? Just want to make sure this doesn't cause problems with resolving with / without time zone in parquet data later on. (don't think it should, just want to take a bit closer look) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19702 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83643/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19702 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19702 **[Test build #83643 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83643/testReport)** for PR 19702 at commit [`e10c806`](https://github.com/apache/spark/commit/e10c8062e3df5b5caa784b0c10ccd92cf56099d2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19702 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19702 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83641/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19702 **[Test build #83641 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83641/testReport)** for PR 19702 at commit [`af62d30`](https://github.com/apache/spark/commit/af62d301ee9d2f3f9ed0a5797110b6388b78f3e6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19702 **[Test build #83643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83643/testReport)** for PR 19702 at commit [`e10c806`](https://github.com/apache/spark/commit/e10c8062e3df5b5caa784b0c10ccd92cf56099d2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19702 LGTM pending tests. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19702 **[Test build #83641 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83641/testReport)** for PR 19702 at commit [`af62d30`](https://github.com/apache/spark/commit/af62d301ee9d2f3f9ed0a5797110b6388b78f3e6). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19702 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19702 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83616/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19702 **[Test build #83616 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83616/testReport)** for PR 19702 at commit [`5ca8bb5`](https://github.com/apache/spark/commit/5ca8bb5904ec85c3c7bb73ab91b1004de5763627). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19702 **[Test build #83616 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83616/testReport)** for PR 19702 at commit [`5ca8bb5`](https://github.com/apache/spark/commit/5ca8bb5904ec85c3c7bb73ab91b1004de5763627). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19702: [SPARK-10365][SQL] Support Parquet logical type TIMESTAM...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19702 cc @liancheng @ueshin @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
