[ https://issues.apache.org/jira/browse/SPARK-17914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572957#comment-15572957 ]
Oksana Romankova edited comment on SPARK-17914 at 10/13/16 7:42 PM: -------------------------------------------------------------------- Sean, I can't find any evidence of ISO8601 not supporting nanoseconds. All it says that it supports fraction of a second that should be supplied following comma or dot. Different parsing libraries that support ISO8601 have different precision limitations. For instance in Python, datetime.strptime() only supports precision down to microseconds and will throw an exception if nanoseconds were supplied in input string. While it may not be ideal for those who need to be able to retain nanosecond precision after parsing, it is an acceptable behavior. Those who do not need to retain nanosecond precision can catch, or, preemptively, truncate input string. Spark sql DateTimeUtils.stringToTimestamp() doesn't throw, and doesn't truncate properly, which results in incorrect timestamp. In the example above, the acceptable truncation would be: "2016-05-14T15:12:14.0034567Z" -> "2016-05-14 15:12:14.003456" "2016-05-14T15:12:14.000345678Z" -> "2016-05-14 15:12:14.000345" was (Author: oromank...@cardlytics.com): Sean, I can't find any evidence of ISO8601 not supporting nanoseconds. All it says that it supports fraction of a second that should be supplied following comma or dot. Different parsing libraries that support ISO8601 have different precision limitations. For instance in Python, datetime.strptime() only supports precision down to microseconds and will throw an exception if nanoseconds were supplied in input string. While it may not be ideal for those who need to be able to retain nanosecond precision after parsing, it is an acceptable behavior. Those who do not need to retain nanosecond precision can catch, or, preemptively, truncate input string. Spark sql DateTimeUtils.stringToTimestamp() doesn't throw, and doesn't truncate properly, which results in incorrect timestamp. In the example above, the acceptable truncation would be: ``` "2016-05-14T15:12:14.0034567Z" -> "2016-05-14 15:12:14.003456" "2016-05-14T15:12:14.000345678Z" -> "2016-05-14 15:12:14.000345" ``` > Spark SQL casting to TimestampType with nanosecond results in incorrect > timestamp > --------------------------------------------------------------------------------- > > Key: SPARK-17914 > URL: https://issues.apache.org/jira/browse/SPARK-17914 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.6.1 > Reporter: Oksana Romankova > > In some cases when timestamps contain nanoseconds they will be parsed > incorrectly. > Examples: > "2016-05-14T15:12:14.0034567Z" -> "2016-05-14 15:12:14.034567" > "2016-05-14T15:12:14.000345678Z" -> "2016-05-14 15:12:14.345678" > The issue seems to be happening in DateTimeUtils.stringToTimestamp(). It > assumes that only 6 digit fraction of a second will be passed. > With this being the case I would suggest either discarding nanoseconds > automatically, or throw an exception prompting to pre-format timestamps to > microsecond precision first before casting to the Timestamp. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org