[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-220827427
  
Hi @cloud-fan, Could you please take a look?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/12921#discussion_r63140449
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVTypeCastSuite.scala
 ---
@@ -73,10 +73,10 @@ class CSVTypeCastSuite extends SparkFunSuite {
 
   test("String type should always return the same as the input") {
 assert(
-  CSVTypeCast.castTo("", StringType, nullable = true, CSVOptions()) ==
+  CSVTypeCast.castTo("", StringType, nullable = true, 
CSVOptions("nullValue", null)) ==
 UTF8String.fromString(""))
 assert(
-  CSVTypeCast.castTo("", StringType, nullable = false, CSVOptions()) ==
+  CSVTypeCast.castTo("", StringType, nullable = false, 
CSVOptions("nullValue", null)) ==
--- End diff --

Otherwise, `null`s for `StringType` will be lost in the roundtrip of 
reading and writing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-218945091
  
Please excuse my ping @rxin @falaki 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-218359655
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58319/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-218359654
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-218359513
  
**[Test build #58319 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58319/consoleFull)**
 for PR 12921 at commit 
[`75f1cb8`](https://github.com/apache/spark/commit/75f1cb8d7c7e89103b1ac0607512837693f7f699).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-218350413
  
**[Test build #58319 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58319/consoleFull)**
 for PR 12921 at commit 
[`75f1cb8`](https://github.com/apache/spark/commit/75f1cb8d7c7e89103b1ac0607512837693f7f699).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217603504
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217603463
  
**[Test build #58045 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58045/consoleFull)**
 for PR 12921 at commit 
[`1233bd7`](https://github.com/apache/spark/commit/1233bd7ce9b70aa984cc3c77ca11e1dc455e3e7e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class NoSuchPermanentFunctionException(db: String, func: String)`
  * `class NoSuchFunctionException(db: String, func: String)`
  * `case class GetExternalRowField(`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217603505
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58045/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217598946
  
**[Test build #58045 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58045/consoleFull)**
 for PR 12921 at commit 
[`1233bd7`](https://github.com/apache/spark/commit/1233bd7ce9b70aa984cc3c77ca11e1dc455e3e7e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217122385
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57871/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217122383
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217122231
  
**[Test build #57871 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57871/consoleFull)**
 for PR 12921 at commit 
[`f31f69e`](https://github.com/apache/spark/commit/f31f69ef44f7505d2483b5a98175cf94fc29b0ff).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/12921#discussion_r62164414
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVTypeCastSuite.scala
 ---
@@ -73,10 +73,10 @@ class CSVTypeCastSuite extends SparkFunSuite {
 
   test("String type should always return the same as the input") {
 assert(
-  CSVTypeCast.castTo("", StringType, nullable = true, CSVOptions()) ==
+  CSVTypeCast.castTo("", StringType, nullable = true, 
CSVOptions("nullValue", null)) ==
 UTF8String.fromString(""))
 assert(
-  CSVTypeCast.castTo("", StringType, nullable = false, CSVOptions()) ==
+  CSVTypeCast.castTo("", StringType, nullable = false, 
CSVOptions("nullValue", null)) ==
--- End diff --

@falaki I just noticed and thought this test implies `nullValue` does not 
apply for `StringType`. Is this intendedly being exclusive? I thought 
`nullValue` should be applied for all the types equivalently.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217108420
  
**[Test build #57871 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57871/consoleFull)**
 for PR 12921 at commit 
[`f31f69e`](https://github.com/apache/spark/commit/f31f69ef44f7505d2483b5a98175cf94fc29b0ff).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217106453
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217106457
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57856/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217106378
  
**[Test build #57856 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57856/consoleFull)**
 for PR 12921 at commit 
[`33288c8`](https://github.com/apache/spark/commit/33288c8580c7aa3f217b195fc4b32f2d066edd11).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/12921#discussion_r62159750
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala
 ---
@@ -192,59 +192,59 @@ private[csv] object CSVTypeCast {
   nullable: Boolean = true,
   options: CSVOptions = CSVOptions()): Any = {
 
-castType match {
-  case _: ByteType => if (datum == options.nullValue && nullable) null 
else datum.toByte
-  case _: ShortType => if (datum == options.nullValue && nullable) 
null else datum.toShort
-  case _: IntegerType => if (datum == options.nullValue && nullable) 
null else datum.toInt
-  case _: LongType => if (datum == options.nullValue && nullable) null 
else datum.toLong
-  case _: FloatType =>
-if (datum == options.nullValue && nullable) {
-  null
-} else if (datum == options.nanValue) {
-  Float.NaN
-} else if (datum == options.negativeInf) {
-  Float.NegativeInfinity
-} else if (datum == options.positiveInf) {
-  Float.PositiveInfinity
-} else {
-  Try(datum.toFloat)
-
.getOrElse(NumberFormat.getInstance(Locale.getDefault).parse(datum).floatValue())
-}
-  case _: DoubleType =>
-if (datum == options.nullValue && nullable) {
-  null
-} else if (datum == options.nanValue) {
-  Double.NaN
-} else if (datum == options.negativeInf) {
-  Double.NegativeInfinity
-} else if (datum == options.positiveInf) {
-  Double.PositiveInfinity
-} else {
-  Try(datum.toDouble)
-
.getOrElse(NumberFormat.getInstance(Locale.getDefault).parse(datum).doubleValue())
-}
-  case _: BooleanType => datum.toBoolean
-  case dt: DecimalType =>
-if (datum == options.nullValue && nullable) {
-  null
-} else {
-  val value = new BigDecimal(datum.replaceAll(",", ""))
-  Decimal(value, dt.precision, dt.scale)
-}
-  case _: TimestampType if options.dateFormat != null =>
-// This one will lose microseconds parts.
-// See https://issues.apache.org/jira/browse/SPARK-10681.
-options.dateFormat.parse(datum).getTime * 1000L
-  case _: TimestampType =>
-// This one will lose microseconds parts.
-// See https://issues.apache.org/jira/browse/SPARK-10681.
-DateTimeUtils.stringToTime(datum).getTime  * 1000L
-  case _: DateType if options.dateFormat != null =>
-DateTimeUtils.millisToDays(options.dateFormat.parse(datum).getTime)
-  case _: DateType =>
-
DateTimeUtils.millisToDays(DateTimeUtils.stringToTime(datum).getTime)
-  case _: StringType => UTF8String.fromString(datum)
-  case _ => throw new RuntimeException(s"Unsupported type: 
${castType.typeName}")
+if (datum == null || (datum == options.nullValue && nullable)) {
--- End diff --

Simply the logic below was added just like `inferField()`:

```scala
if (datum == null || (datum == options.nullValue && nullable)) {
  null
} else {
 ... 
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217094308
  
**[Test build #57856 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57856/consoleFull)**
 for PR 12921 at commit 
[`33288c8`](https://github.com/apache/spark/commit/33288c8580c7aa3f217b195fc4b32f2d066edd11).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217089404
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217089406
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57847/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217089340
  
**[Test build #57847 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57847/consoleFull)**
 for PR 12921 at commit 
[`2ea702c`](https://github.com/apache/spark/commit/2ea702cdfd4732db5c1da6f750f1d15a1ba30396).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217088643
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57846/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217088641
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217088408
  
**[Test build #57846 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57846/consoleFull)**
 for PR 12921 at commit 
[`f80df8c`](https://github.com/apache/spark/commit/f80df8cec6079cb8eb34b9893065b329847249f0).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217087742
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217087744
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57845/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217087681
  
**[Test build #57845 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57845/consoleFull)**
 for PR 12921 at commit 
[`ef71599`](https://github.com/apache/spark/commit/ef71599130291aa6bb684afb47e88307b38238f1).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217083084
  
**[Test build #57847 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57847/consoleFull)**
 for PR 12921 at commit 
[`2ea702c`](https://github.com/apache/spark/commit/2ea702cdfd4732db5c1da6f750f1d15a1ba30396).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217082924
  
cc @rxin @falaki 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217082691
  
**[Test build #57846 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57846/consoleFull)**
 for PR 12921 at commit 
[`f80df8c`](https://github.com/apache/spark/commit/f80df8cec6079cb8eb34b9893065b329847249f0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/12921#discussion_r62151223
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
 ---
@@ -447,7 +446,7 @@ class CSVSuite extends QueryTest with SharedSQLContext 
with SQLTestUtils {
 
 verifyCars(cars, withHeader = true, checkValues = false)
 val results = cars.collect()
-assert(results(0).toSeq === Array(2012, "Tesla", "S", "null", "null"))
+assert(results(0).toSeq === Array(2012, "Tesla", "S", null, null))
--- End diff --

This is being tested against the data as below:

```
year,make,model,comment,blank
"2012","Tesla","S",null,

1997,Ford,E350,"Go get one now they are going fast",
null,Chevy,Volt

```

Since the header is `year`,`make`,`model`,`comment`,`blank`, this should 
produce the values `2012`,`Tesla`,`S`,`null`,`null`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-217081814
  
**[Test build #57845 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57845/consoleFull)**
 for PR 12921 at commit 
[`ef71599`](https://github.com/apache/spark/commit/ef71599130291aa6bb684afb47e88307b38238f1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[HyukjinKwon]

GitHub user HyukjinKwon opened a pull request:

https://github.com/apache/spark/pull/12921

[SPARK-15143][SPARK-15144][SQL] Add CSV tests with HadoopFsRelationTest and 
support for nullValue for other types

## What changes were proposed in this pull request?

Currently,  `nullValue` option does not work for some types, `BooleanType`, 
`TimestampType`, `DateType` and `StringType`. So, currently there is no way to 
read null for those types. This PR adds the support just like the other types.

Also, CSV data source is not being tested with `HadoopFsRelationTest` as a 
`HadoopFsRelation`. `HadoopFsRelationTest` includes 50 more tests (eg. 
partitioned table tests).

This PR adds two variables, `extraReadOptions` and `extraWriteOptions` in 
`HadoopFsRelationTest` so that the child class gives some options for reading 
and writing. In order to get the tests in `HadoopFsRelationTest` passed, CSV 
data source needs to give options `header` and `inferSchema` as `true` for 
reading and `header` as `true` for writing.


## How was this patch tested?

Unittests in `CSVHadoopFsRelationTest` and edited tests in `CSVSuite`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/spark SPARK-15143-15144

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12921.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12921


commit ef71599130291aa6bb684afb47e88307b38238f1
Author: hyukjinkwon 
Date:   2016-05-05T05:47:07Z

Add CSV tests with HadoopFsRelationTest and support for nullValue for other 
types




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org