[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14279 @hvanhovell Sure! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/14279 @HyukjinKwon could you backport this 2.0? It has merge conflicts so I cannot cherry pick it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/14279 LGTM - merging to master/2.0. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14279 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14279 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64326/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14279 **[Test build #64326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64326/consoleFull)** for PR 14279 at commit [`af8250e`](https://github.com/apache/spark/commit/af8250e12490c77f02587275eff9aa225e5dcdba). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14279 **[Test build #64326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64326/consoleFull)** for PR 14279 at commit [`af8250e`](https://github.com/apache/spark/commit/af8250e12490c77f02587275eff9aa225e5dcdba). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14279 **[Test build #64065 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64065/consoleFull)** for PR 14279 at commit [`405e723`](https://github.com/apache/spark/commit/405e723215213586eecb676268e00aa2263b6c71). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14279 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14279 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64065/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14279 **[Test build #64065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64065/consoleFull)** for PR 14279 at commit [`405e723`](https://github.com/apache/spark/commit/405e723215213586eecb676268e00aa2263b6c71). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14279 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64061/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14279 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14279 **[Test build #64061 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64061/consoleFull)** for PR 14279 at commit [`7e61674`](https://github.com/apache/spark/commit/7e6167403a743cc0facb9ad7dd43ca5d912c5f25). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14279 **[Test build #64061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64061/consoleFull)** for PR 14279 at commit [`7e61674`](https://github.com/apache/spark/commit/7e6167403a743cc0facb9ad7dd43ca5d912c5f25). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14279 Oh, wait, I think I can handle this. Please ignore the comments I just left. Sorry for noise. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14279 Sigh.. I will proceed but please let me note that this will also bring a breaking change for reading as well not only writing.. It seems there is a backwards compatibility problem with using default date format for `dateFormat` and `timestampFormat` instead of `null` when reading. When it was `null`, it uses `DateTimeUtils.stringToTime` for `TimestampType` and `DateTimeUtils.millisToDays` for `DateType`. It seems okay with `DateTimeUtils.millisToDays` for `DateType` but the problem is with `DateTimeUtils.stringToTime()`. It seems `DateTimeUtils.stringToTime()` for `TimestampType` supports to read several other date formats such as - `2000-01-01T00:00GMT+01:00` - `2000-01-01T00:00+01:00` - `2000-01-01 00:00:00.0` As previous Spark version writes `2000-01-01 00:00:00.0`, setting the default value for `timestampFormat` to ISO8601 format will also break the backwards compatibility. we should explicitly set this for reading as well not only writing for the same timestamps handling between Spark 1.x and Spark 2.x. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14279 Yup, thanks for informing me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14279 BTW I think this is pretty important for 2.0.1 release. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14279 If we are introducing breaking changes to fix the bugs here, let's fix it for real. (definitely a problem if we can't specify dateFormat and timestampFormat separately). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14279 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63954/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14279 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14279 **[Test build #63954 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63954/consoleFull)** for PR 14279 at commit [`2bd9cbf`](https://github.com/apache/spark/commit/2bd9cbf601ea2ec9f95cd82e888dcafa54421715). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14279 If `dateFormat` is for `DateType` and `timestampFormat` for `TimestampType`, I think both can be independently handled although I am worried of backwards compatibility because `dateFormat` for CSV were applied to both `TimestampType` and `DateType` in 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14279 But if dateFormat is set to -MM-dd, then suddenly timestamp output won't show anything past the date part? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14279 Oh, default for `dateFormat` will be `null` and they will use `-MM-dd'T'HH:mm:ss.SSSZZ` for timestamps and `-MM-dd` for dates by default. Test cases for the default for both are added in `CSVSuite` and it is already being tested for JSON in `JsonSuite` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14279 Default to dateFormat for timestampFormat seems wrong? Because in that case all timestamps will be written as -mm-dd only ignoring the rest? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14279 Let me please add `timestampFormat`. As CSV already has `dateFormat` option for both `DateType` and `TimestampType`, Please let me use `dateFormat` for both if `timestampFormat` is not set if it makes sense. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14279 @rxin Ah, I see. Yes, it seems we need for each to allow writing dates and timestamps as the the original format. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14279 **[Test build #63954 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63954/consoleFull)** for PR 14279 at commit [`2bd9cbf`](https://github.com/apache/spark/commit/2bd9cbf601ea2ec9f95cd82e888dcafa54421715). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14279: [SPARK-16216][SQL] Read/write timestamps and dates in IS...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14279 **[Test build #63918 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63918/consoleFull)** for PR 14279 at commit [`4171a3c`](https://github.com/apache/spark/commit/4171a3c53fc78e46646f49f29c77decb32968bf9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
