[GitHub] spark pull request: Fix datetime parsing in SparkSQL.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8396#discussion_r37761954
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
---
@@ -107,30 +107,21 @@ object DateTimeUtils {
}
def stringToTime(s: String): java.util.Date = {
-if (!s.contains('T')) {
+var indexOfGMT = s.indexOf("GMT");
+if (indexOfGMT != -1) {
+ // timezone with ISO8601
+ val s0 = s.substring(0, indexOfGMT)
+ val s1 = s.substring(indexOfGMT + 3)
+ return stringToTime(s0 + s1)
+} else if (!s.contains('T')) {
// JDBC escape string
if (s.contains(' ')) {
Timestamp.valueOf(s)
} else {
Date.valueOf(s)
}
-} else if (s.endsWith("Z")) {
- // this is zero timezone of ISO8601
- stringToTime(s.substring(0, s.length - 1) + "GMT-00:00")
-} else if (s.indexOf("GMT") == -1) {
- // timezone with ISO8601
- val inset = "+00.00".length
- val s0 = s.substring(0, s.length - inset)
- val s1 = s.substring(s.length - inset, s.length)
- if (s0.substring(s0.lastIndexOf(':')).contains('.')) {
-stringToTime(s0 + "GMT" + s1)
- } else {
-stringToTime(s0 + ".0GMT" + s1)
- }
} else {
- // ISO8601 with GMT insert
- val ISO8601GMT: SimpleDateFormat = new SimpleDateFormat(
"-MM-dd'T'HH:mm:ss.SSSz" )
- ISO8601GMT.parse(s)
+ javax.xml.bind.DatatypeConverter.parseDateTime(s).getTime()
--- End diff --
This might bear explanation -- why does it implementing parsing correctly
and why is GMT special-cased?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Fix datetime parsing in SparkSQL.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8396#issuecomment-134241948 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Fix datetime parsing in SparkSQL.
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8396#issuecomment-134241529 See https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark for how to name pull requests, which would automatically link them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Fix datetime parsing in SparkSQL.
GitHub user kevincox opened a pull request: https://github.com/apache/spark/pull/8396 Fix datetime parsing in SparkSQL. This fixes https://issues.apache.org/jira/browse/SPARK-9794 by using a real ISO8601 parser. (courtesy of the xml component of the standard java library) @angelini You can merge this pull request into a Git repository by running: $ git pull https://github.com/Shopify/spark kevincox-sql-time-parsing Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8396.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8396 commit 5d9a7c5e952240fa02b3022398f4fbc55df137a8 Author: Kevin Cox Date: 2015-08-19T19:04:59Z Fix datetime parsing in SparkSQL. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
