GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/17315
[SPARK-19949][SQL] unify bad record handling in CSV and JSON
## What changes were proposed in this pull request?
Currently JSON and CSV have exactly the same logic about handling bad
rec
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106566183
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
---
@@ -391,9 +288,9 @@ class JacksonParser(
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106570171
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala
---
@@ -65,7 +65,7 @@ private[sql] class JSONOptions(
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106593549
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala
---
@@ -65,7 +65,7 @@ private[sql] class JSONOptions(
v
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106648055
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the A
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106655740
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala
---
@@ -233,81 +187,39 @@ class UnivocityParser(
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106653556
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
---
@@ -55,108 +52,6 @@ class JacksonParser(
privat
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106664412
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the A
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106664841
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the A
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106650196
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the A
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106767486
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the A
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106768186
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the A
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106806437
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala
---
@@ -233,81 +187,39 @@ class UnivocityParser(
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106838434
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -1354,9 +1354,8 @@ test_that("column functions", {
# passing option
df <- as.Data
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106838506
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -1354,9 +1354,8 @@ test_that("column functions", {
# passing option
df <- as.Data
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106840034
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -382,11 +383,17 @@ class DataFrameReader private[sql](sparkSession:
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106840126
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Ap
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106840169
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Ap
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106840206
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -1354,9 +1354,8 @@ test_that("column functions", {
# passing option
df <- as.DataF
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106840345
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -382,11 +383,17 @@ class DataFrameReader private[sql](sparkSession:
S
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106840806
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -1354,9 +1354,8 @@ test_that("column functions", {
# passing option
df <- as.Data
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106840886
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
---
@@ -55,108 +52,6 @@ class JacksonParser(
private
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106842646
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
---
@@ -113,8 +113,11 @@ class CSVFileFormat extend
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106847415
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
---
@@ -55,108 +52,6 @@ class JacksonParser(
private
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106847502
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
---
@@ -113,8 +113,11 @@ class CSVFileFormat extends
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r106852242
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala
---
@@ -233,81 +187,39 @@ class UnivocityParser(
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r107011371
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Ap
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r107012038
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -382,11 +383,17 @@ class DataFrameReader private[sql](sparkSession:
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r107013197
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Ap
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r107013316
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Ap
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r107013706
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -435,14 +442,20 @@ class DataFrameReader private[sql](sparkSession:
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r107014069
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala
---
@@ -46,85 +46,39 @@ class UnivocityParser(
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r107014888
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/UnivocityParser.scala
---
@@ -233,81 +187,41 @@ class UnivocityParser(
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r107020700
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Ap
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r107053686
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apa
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r107055187
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Ap
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17315#discussion_r107055987
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
---
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apa
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17315
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is ena
38 matches
Mail list logo
