[GitHub] spark issue #16735: [SPARK-19228][SQL] Introduce tryParseDate method to proc...
Github user sergey-rubtsov commented on the issue: https://github.com/apache/spark/pull/16735 I wil try to complete it in this month --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16735: [SPARK-19228][SQL] Introduce tryParseDate method ...
Github user sergey-rubtsov closed the pull request at: https://github.com/apache/spark/pull/16735 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20140: [SPARK-19228][SQL] Introduce tryParseDate method ...
GitHub user sergey-rubtsov opened a pull request:
https://github.com/apache/spark/pull/20140
[SPARK-19228][SQL] Introduce tryParseDate method to process csv date,â¦
⦠add a type-widening rule in findTightestCommonType between DateType and
TimestampType, add java.time.format.DateTimeFormatter to more accurately infer
the type of time, add an end-to-end test case and unit test
## What changes were proposed in this pull request?
By design 'TimestampType' (8 bytes) is larger than 'DateType' (4 bytes).
But when a date is parsed, an option "dateFormat" is ignored and default
date format ("-MM-dd") is using and the date is parsed as timestamp.
This patch fixes that bug.
For other details, please, read the ticket
https://issues.apache.org/jira/browse/SPARK-19228
## How was this patch tested?
Add an end-to-end test case and unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sergey-rubtsov/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20140.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20140
commit d2ed68673082995bee65a3c58be8b71642a60d57
Author: sergei.rubtcov
Date: 2018-01-03T10:50:26Z
[SPARK-19228][SQL] Introduce tryParseDate method to process csv date, add a
type-widening rule in findTightestCommonType between DateType and
TimestampType, add java.time.format.DateTimeFormatter to more accurately infer
the type of time, add an end-to-end test case and unit test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20140: [SPARK-19228][SQL] Introduce tryParseDate method to proc...
Github user sergey-rubtsov commented on the issue: https://github.com/apache/spark/pull/20140 @HyukjinKwon, @gatorsmile could you please help find someone to review this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21363: [SPARK-19228][SQL] Migrate on Java 8 time from Fa...
GitHub user sergey-rubtsov opened a pull request: https://github.com/apache/spark/pull/21363 [SPARK-19228][SQL] Migrate on Java 8 time from FastDateFormat for meet the ISO8601 ## What changes were proposed in this pull request? Add support for inferring DateType and custom "dateFormat" option. Fix an old bug with parse string to SQL's timestamp value in microseconds accuracy. Add a type-widening rule in findTightestCommonType between DateType and TimestampType. ## How was this patch tested? Fix some tests to accord with an internationally agreed way to represent dates. Add an end-to-end test case and unit tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sergey-rubtsov/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/21363.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #21363 commit 65179a2bdd1623fb7f4077cdc316de5a7436c49d Author: sergei.rubtcov Date: 2018-05-18T12:13:05Z [SPARK-19228][SQL] Migrate on Java 8 time from FastDateFormat for meet the ISO8601 and parsing dates in csv correctly. Add support for inferring DateType and custom "dateFormat" option. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21363: [SPARK-19228][SQL] Migrate on Java 8 time from FastDateF...
Github user sergey-rubtsov commented on the issue: https://github.com/apache/spark/pull/21363 Previous pull request https://github.com/apache/spark/pull/20140 is closed --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21363: [SPARK-19228][SQL] Migrate on Java 8 time from FastDateF...
Github user sergey-rubtsov commented on the issue: https://github.com/apache/spark/pull/21363 @HyukjinKwon, please take a look --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20140: [SPARK-19228][SQL] Introduce tryParseDate method ...
Github user sergey-rubtsov closed the pull request at: https://github.com/apache/spark/pull/20140 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21363: [SPARK-19228][SQL] Migrate on Java 8 time from Fa...
Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/21363#discussion_r189255920
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParserSuite.scala
---
@@ -107,20 +107,26 @@ class UnivocityParserSuite extends SparkFunSuite {
assert(parser.makeConverter("_1", BooleanType, options =
options).apply("true") == true)
val timestampsOptions =
- new CSVOptions(Map("timestampFormat" -> "dd/MM/ hh:mm"), "GMT")
+ new CSVOptions(Map("timestampFormat" -> "dd/MM/ HH:mm"), "GMT")
--- End diff --
In accordind to official documentation, this test must not pass, because
"hh" for hours means that hour in am/pm (1-12):
https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html
https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21363: [SPARK-19228][SQL] Migrate on Java 8 time from Fa...
Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/21363#discussion_r189255997
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParserSuite.scala
---
@@ -107,20 +107,26 @@ class UnivocityParserSuite extends SparkFunSuite {
assert(parser.makeConverter("_1", BooleanType, options =
options).apply("true") == true)
val timestampsOptions =
- new CSVOptions(Map("timestampFormat" -> "dd/MM/ hh:mm"), "GMT")
--- End diff --
In accordind to official documentation, this test must not pass, because
"hh" for hours means that hour in am/pm (1-12):
https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html
https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21363: [SPARK-19228][SQL] Migrate on Java 8 time from FastDateF...
Github user sergey-rubtsov commented on the issue: https://github.com/apache/spark/pull/21363 cc @viirya @mgaido91 @dongjoon-hyun --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21363: [SPARK-19228][SQL] Migrate on Java 8 time from Fa...
Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/21363#discussion_r189582909
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchemaSuite.scala
---
@@ -59,13 +59,21 @@ class CSVInferSchemaSuite extends SparkFunSuite {
assert(CSVInferSchema.inferField(IntegerType, textValueOne, options)
== expectedTypeOne)
}
- test("Timestamp field types are inferred correctly via custom data
format") {
-var options = new CSVOptions(Map("timestampFormat" -> "-mm"),
"GMT")
+ test("Timestamp field types are inferred correctly via custom date
format") {
+var options = new CSVOptions(Map("timestampFormat" -> "-MM"),
"GMT")
--- End diff --
"-mm" means years and minutes, this is date format, this is time format
-MM" means years and months, but I do not insist on this change
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21363: [SPARK-19228][SQL] Migrate on Java 8 time from Fa...
Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/21363#discussion_r189583023
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -90,6 +90,7 @@ private[csv] object CSVInferSchema {
// DecimalTypes have different precisions and scales, so we try
to find the common type.
findTightestCommonType(typeSoFar, tryParseDecimal(field,
options)).getOrElse(StringType)
case DoubleType => tryParseDouble(field, options)
+case DateType => tryParseDate(field, options)
--- End diff --
I can do it, but where exactly it should be documented?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21363: [SPARK-19228][SQL] Migrate on Java 8 time from Fa...
Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/21363#discussion_r189583203
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParserSuite.scala
---
@@ -107,20 +107,26 @@ class UnivocityParserSuite extends SparkFunSuite {
assert(parser.makeConverter("_1", BooleanType, options =
options).apply("true") == true)
val timestampsOptions =
- new CSVOptions(Map("timestampFormat" -> "dd/MM/ hh:mm"), "GMT")
+ new CSVOptions(Map("timestampFormat" -> "dd/MM/ HH:mm"), "GMT")
val customTimestamp = "31/01/2015 00:00"
-val expectedTime =
timestampsOptions.timestampFormat.parse(customTimestamp).getTime
+
+val expectedTime = LocalDateTime.parse(customTimestamp,
timestampsOptions.timestampFormatter)
+ .atZone(options.timeZone.toZoneId)
+ .toInstant.toEpochMilli
val castedTimestamp =
- parser.makeConverter("_1", TimestampType, nullable = true, options =
timestampsOptions)
+ parser.makeConverter("_1", TimestampType, nullable = true,
timestampsOptions)
.apply(customTimestamp)
assert(castedTimestamp == expectedTime * 1000L)
-val customDate = "31/01/2015"
val dateOptions = new CSVOptions(Map("dateFormat" -> "dd/MM/"),
"GMT")
-val expectedDate = dateOptions.dateFormat.parse(customDate).getTime
+val customDate = "31/01/2015"
+
+val expectedDate = LocalDate.parse(customDate,
dateOptions.dateFormatter)
+ .atStartOfDay(options.timeZone.toZoneId)
+ .toInstant.toEpochMilli
val castedDate =
- parser.makeConverter("_1", DateType, nullable = true, options =
dateOptions)
--- End diff --
ok
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21363: [SPARK-19228][SQL] Migrate on Java 8 time from Fa...
Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/21363#discussion_r189588135
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -140,14 +141,23 @@ private[csv] object CSVInferSchema {
private def tryParseDouble(field: String, options: CSVOptions): DataType
= {
if ((allCatch opt field.toDouble).isDefined || isInfOrNan(field,
options)) {
DoubleType
+} else {
+ tryParseDate(field, options)
--- End diff --
For example, by mistake we have identical "timestampFormat" and
"dateFormat" options.
Let it be "-MM-dd"
'TimestampType' (8 bytes) is larger than 'DateType' (4 bytes)
So if they can overlap, we need to try parse it as date firstly, because
both of these types are suitable, but you need to try to use a more compact by
default and it will be correct inferring of type
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21363: [SPARK-19228][SQL] Migrate on Java 8 time from FastDateF...
Github user sergey-rubtsov commented on the issue: https://github.com/apache/spark/pull/21363 @HyukjinKwon please, clarify, how to add a configuration to control this behaviour? Do you mean to keep backward compatibility? But it seems, now, we don't handle a Spark "DataType" at all. I can change the behaviour in next way: Hanlde "DataType" if option "dateFormat" was set by customer, if it is not set, ignore it at all, is it okay? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21363: [SPARK-19228][SQL] Migrate on Java 8 time from Fa...
Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/21363#discussion_r189596275
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
---
@@ -143,6 +145,12 @@ object DateTimeUtils {
millisLocal - getOffsetFromLocalMillis(millisLocal, timeZone)
}
+ def dateTimeToMicroseconds(localDateTime: LocalDateTime, timeZone:
TimeZone): Long = {
+val microOfSecond = localDateTime.getLong(ChronoField.MICRO_OF_SECOND)
+val epochSecond =
localDateTime.atZone(timeZone.toZoneId).toInstant.getEpochSecond
+epochSecond * 100L + microOfSecond
--- End diff --
thanks
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21363: [SPARK-19228][SQL] Migrate on Java 8 time from Fa...
Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/21363#discussion_r189597989
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -140,14 +141,23 @@ private[csv] object CSVInferSchema {
private def tryParseDouble(field: String, options: CSVOptions): DataType
= {
if ((allCatch opt field.toDouble).isDefined || isInfOrNan(field,
options)) {
DoubleType
+} else {
+ tryParseDate(field, options)
--- End diff --
At the moment, DateType here is ignored at all, I'm not sure that it was
conceived when the type was created
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20140: [SPARK-19228][SQL] Introduce tryParseDate method to proc...
Github user sergey-rubtsov commented on the issue: https://github.com/apache/spark/pull/20140 @HyukjinKwon changed as you suggested --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16735: [SPARK-19228][SQL] Introduce tryParseDate method to proc...
Github user sergey-rubtsov commented on the issue: https://github.com/apache/spark/pull/16735 @HyukjinKwon yes, sure, I will check it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20140: [SPARK-19228][SQL] Introduce tryParseDate method ...
Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/20140#discussion_r166261313
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
---
@@ -150,6 +151,16 @@ class CSVOptions(
val isCommentSet = this.comment != '\u'
+ def dateFormatter: DateTimeFormatter = {
+DateTimeFormatter.ofPattern(dateFormat.getPattern)
+
.withLocale(Locale.US).withZone(timeZone.toZoneId).withResolverStyle(ResolverStyle.SMART)
+ }
+
+ def timestampFormatter: DateTimeFormatter = {
+DateTimeFormatter.ofPattern(timestampFormat.getPattern)
--- End diff --
DateTimeFormatter is a standard time library from java 8. FastDateFormat
can't properly parse date and timestamp.
I can create some test cases to prove it, but I need many time for that.
Also, FastDateFormat does not meet the ISO8601:
https://en.wikipedia.org/wiki/ISO_8601
Current implementation of CSVInferSchema contains other bugs. For example,
test test("Timestamp field types are inferred correctly via custom date
format") in class CSVInferSchemaSuite must not pass, because timestampFormat
"-mm" is wrong format for year and month. It should be "-MM".
It is better to make refactor of date types and change deprecated types on
new ones for the whole project.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20140: [SPARK-19228][SQL] Introduce tryParseDate method ...
Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/20140#discussion_r166262196
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
---
@@ -150,6 +151,16 @@ class CSVOptions(
val isCommentSet = this.comment != '\u'
+ def dateFormatter: DateTimeFormatter = {
--- End diff --
DateTimeFormatter has the disadvantage. It does not implement Serializable
in contrast to FastDateFormat. That is why I couldn't make it as a val here.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #20140: [SPARK-19228][SQL] Introduce tryParseDate method ...
Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/20140#discussion_r166264802
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -90,7 +90,10 @@ private[csv] object CSVInferSchema {
// DecimalTypes have different precisions and scales, so we try
to find the common type.
findTightestCommonType(typeSoFar, tryParseDecimal(field,
options)).getOrElse(StringType)
case DoubleType => tryParseDouble(field, options)
-case TimestampType => tryParseTimestamp(field, options)
+case DateType => tryParseDate(field, options)
+case TimestampType =>
+ findTightestCommonType(typeSoFar, tryParseTimestamp(field,
options)).getOrElse(
--- End diff --
Sorry, your question is not really clear for me.
We have to try parse object as DateType first, because date always can be
parsed as date and as timestamp (begin of day).
Current implementation of spark ignores dates and it is always parsing them
as timestamps
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16735: [SPARK-19228][SQL] Introduce tryParseDate method ...
Github user sergey-rubtsov closed the pull request at: https://github.com/apache/spark/pull/16735 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16735: [SPARK-19228][SQL] Introduce tryParseDate method ...
GitHub user sergey-rubtsov reopened a pull request:
https://github.com/apache/spark/pull/16735
[SPARK-19228][SQL] Introduce tryParseDate method to process csv date â¦
â¦column with custom format as date
## What changes were proposed in this pull request?
This patch fixes bugs:
1) All the dates parsed as timestamps.
2) Option "dateFormat" is ignored when read csv files with date data.
Instead of this option default date format ("-MM-dd") is using.
For other details, please, read the ticket
https://issues.apache.org/jira/browse/SPARK-19228
## How was this patch tested?
Tested with unit tests only. Add new test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sergey-rubtsov/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16735.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16735
commit 287005809cec5388dcb75a3d99bc0f0461b9bb69
Author: sergei.rubtcov
Date: 2017-03-10T13:03:27Z
[SPARK-19228][SQL] Introduce tryParseDate method to process csv date, add a
type-widening rule in findTightestCommonType between DateType and
TimestampType, add an end-to-end test case
commit 3e250f56ee5f3a5c1ce5542d56670973233e62b7
Author: sergei.rubtcov
Date: 2017-03-13T14:13:17Z
Merge branch 'master' of https://github.com/apache/spark
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16735: [SPARK-19228][SQL] Introduce tryParseDate method to proc...
Github user sergey-rubtsov commented on the issue: https://github.com/apache/spark/pull/16735 Couldn't run tests in CSVSuite locally on my Windows OS, apologize for the possible test fails --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16735: [SPARK-19228][SQL] Introduce tryParseDate method to proc...
Github user sergey-rubtsov commented on the issue: https://github.com/apache/spark/pull/16735 Hi @HyukjinKwon, @gatorsmile Could you take a look, please? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16735: [SPARK-19228][SQL] Introduce tryParseDate method ...
GitHub user sergey-rubtsov opened a pull request: https://github.com/apache/spark/pull/16735 [SPARK-19228][SQL] Introduce tryParseDate method to process csv date ⦠â¦column with custom format as date ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review http://spark.apache.org/contributing.html before opening a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sergey-rubtsov/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/16735.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #16735 commit 8f78e1129efe8277c190edd5016c1e06b5aeef65 Author: Sergey Rubtsov Date: 2017-01-28T20:21:55Z [SPARK-19228][SQL] Introduce tryParseDate method to process csv date column with custom format as date --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #16735: [SPARK-19228][SQL] Introduce tryParseDate method ...
Github user sergey-rubtsov commented on a diff in the pull request:
https://github.com/apache/spark/pull/16735#discussion_r98523879
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
---
@@ -140,12 +137,21 @@ private[csv] object CSVInferSchema {
}
}
+ private def tryParseDate(field: String, options: CSVOptions): DataType =
{
+// This case infers a custom `dateFormat` is set.
+if ((allCatch opt options.dateFormat.parse(field)).isDefined) {
+ DateType
+} else {
+ tryParseTimestamp(field, options)
+}
+ }
+
private def tryParseTimestamp(field: String, options: CSVOptions):
DataType = {
-// This case infers a custom `dataFormat` is set.
+// This case infers a custom `timestampFormat` is set.
if ((allCatch opt options.timestampFormat.parse(field)).isDefined) {
TimestampType
} else if ((allCatch opt DateTimeUtils.stringToTime(field)).isDefined)
{
- // We keep this for backwords competibility.
+ // We keep this for backwards compatibility.
TimestampType
} else {
tryParseBoolean(field, options)
--- End diff --
okey, I will test it again, thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
