Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/19339
@HyukjinKwon @viirya Thanks for your reviewing.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/19339#discussion_r141232322
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -420,7 +425,29 @@ def csv(self, path, schema=None, sep=None,
encoding=None, quote=None, escape=Non
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/19339
@HyukjinKwon I has updated this title. Thanks !
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/19339
umm.. I test it fine using Python 3.4 in my local. I'm not sure why did it
test fail sometime... :(
---
-
To unsubscribe, e
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/19339#discussion_r141082457
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -336,6 +336,7 @@ def csv(self, path, schema=None, sep=None,
encoding=None, quote=None, escape=Non
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/19339#discussion_r141080522
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -420,7 +425,29 @@ def csv(self, path, schema=None, sep=None,
encoding=None, quote=None, escape=Non
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/19339#discussion_r141075069
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -420,7 +425,25 @@ def csv(self, path, schema=None, sep=None,
encoding=None, quote=None, escape=Non
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/19339
This is so weird. I run it fine using Python 3.5.2 but it seems to have
some problem using Python 3.4. Let me try Python 3.4 in my local
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/19339
ok, so maybe I create another JIRA for this issue?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/19339
@HyukjinKwon I think your way works fine after fixing a variable name bug
(`_jsqlContext` >> `_jssql_ctx`). Should we need to modify the json part to be
consistent with the cs
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/19339#discussion_r141033740
--- Diff: python/pyspark/sql/readwriter.py ---
@@ -420,7 +425,22 @@ def csv(self, path, schema=None, sep=None,
encoding=None, quote=None, escape=Non
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/19339#discussion_r140779203
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala ---
@@ -456,6 +456,40 @@ class DataFrameReader private[sql](sparkSession
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/19339
@HyukjinKwon @viirya Could you review this PR? Thanks! :)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
GitHub user goldmedal opened a pull request:
https://github.com/apache/spark/pull/19339
[SPARK-22112][PYSPARK] Add an API to create a DataFrame from RDD[String]
storing CSV
## What changes were proposed in this pull request?
We added a method to the scala API for creating
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/19223
Thanks @HyukjinKwon @felixcheung @viirya
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/19223
ok. I got it. Thanks :)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/19223
@HyukjinKwon Thanks for triggering AppVeyor. In normal case, will AppVeyor
be triggered automatically?
---
-
To unsubscribe
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/19223#discussion_r138856179
--- Diff: python/pyspark/sql/functions.py ---
@@ -1921,10 +1921,12 @@ def from_json(col, schema, options={}):
@since(2.1)
def to_json(col
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/19223
@HyukjinKwon @felixcheung @viirya
I has finished those change at your suggestions for this PR and it also
passed all tests. Please take a look when you are available. Thanks
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/19223#discussion_r138800321
--- Diff: python/pyspark/sql/functions.py ---
@@ -1921,10 +1921,12 @@ def from_json(col, schema, options={}):
@since(2.1)
def to_json(col
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/19223#discussion_r138799591
--- Diff: R/pkg/R/functions.R ---
@@ -1715,7 +1717,15 @@ setMethod("to_date",
#'
#' # Converts an array of structs into a JSON array
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/19223#discussion_r138799483
--- Diff:
sql/core/src/test/resources/sql-tests/results/json-functions.sql.out ---
@@ -26,13 +26,13 @@ Extended Usage:
{"time":&
GitHub user goldmedal opened a pull request:
https://github.com/apache/spark/pull/19223
[SPARK-21513][SQL][FOLLOWUP] Allow UDF to_json support converting MapType
to json for PySpark and SparkR
## What changes were proposed in this pull request?
In previous work SPARK-21513, we
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon ok. I got it. Thanks =)
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon OK, I'll work on R and Python. My JIRA id is 'goldmedal', too.
Thanks for your review :)
@viirya Thanks for your mentor and review
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138501335
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -26,20 +26,50 @@ import
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon @viirya
Sorry for updating this PR so late. Please take a look when you are
available. Thanks
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138237482
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -193,14 +223,35 @@ private[sql] class
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138110523
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala
---
@@ -0,0 +1,133 @@
+/*
+ * Licensed
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138094337
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala
---
@@ -0,0 +1,133 @@
+/*
+ * Licensed
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r138087980
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala
---
@@ -0,0 +1,133 @@
+/*
+ * Licensed
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon We have finished the `MapType` and `ArrayType` of `MapType`s
supporting. Please take a look when you are available. Thanks
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137987843
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -193,14 +228,32 @@ private[sql] class
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137987296
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -193,14 +228,32 @@ private[sql] class
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137984767
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -677,14 +696,42 @@ case class StructsToJson
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137980084
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -677,14 +696,27 @@ case class StructsToJson
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137979014
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -677,14 +696,27 @@ case class StructsToJson
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137978728
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -193,14 +228,32 @@ private[sql] class
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137978096
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -677,14 +696,27 @@ case class StructsToJson
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137977643
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala
---
@@ -0,0 +1,106 @@
+/*
+ * Licensed
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137977153
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala
---
@@ -0,0 +1,106 @@
+/*
+ * Licensed
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137977056
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/json/JacksonGeneratorSuite.scala
---
@@ -0,0 +1,106 @@
+/*
+ * Licensed
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137976953
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -193,14 +228,32 @@ private[sql] class
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137975716
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -193,14 +228,32 @@ private[sql] class
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137959771
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -193,14 +228,58 @@ private[sql] class
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137958902
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -193,14 +228,58 @@ private[sql] class
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137710147
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -26,20 +26,50 @@ import
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137288735
--- Diff: sql/core/src/test/resources/sql-tests/results/cross-join.sql.out
---
@@ -128,6 +128,7 @@ two 2 two 2 one 1 two
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137285989
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -669,14 +679,25 @@ case class StructsToJson
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137282128
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -22,24 +22,49 @@ import java.io.Writer
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137273266
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -22,24 +22,50 @@ import java.io.Writer
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137269586
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -127,7 +153,7 @@ private[sql] class
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137048876
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -27,21 +27,45 @@ import
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r137044817
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -27,21 +27,45 @@ import
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r134469528
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -202,5 +203,9 @@ private[sql] class
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r132615092
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala ---
@@ -186,6 +186,18 @@ class JsonFunctionsSuite extends QueryTest
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r131952135
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -659,13 +660,19 @@ case class StructsToJson
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r131951764
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala
---
@@ -202,5 +202,9 @@ private[sql] class
Github user goldmedal commented on a diff in the pull request:
https://github.com/apache/spark/pull/18875#discussion_r131946027
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
---
@@ -659,13 +660,19 @@ case class StructsToJson
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18875
@HyukjinKwon Thanks for the reference! I'll check it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user goldmedal opened a pull request:
https://github.com/apache/spark/pull/18875
[SPARK-21513][SQL] Allow UDF to_json support converting MapType to json
# What changes were proposed in this pull request?
UDF to_json only supports converting `StructType` or `ArrayType
Github user goldmedal closed the pull request at:
https://github.com/apache/spark/pull/18854
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user goldmedal commented on the issue:
https://github.com/apache/spark/pull/18854
@gatorsmile @viirya Thanks a lot. You are right. I close it for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user goldmedal opened a pull request:
https://github.com/apache/spark/pull/18854
[SPARK-21629][SQL][WIP] Fix Or nullability
## What changes were proposed in this pull request?
Override nullable of Or Expression to make sure nullable behavior is
correct.
## How
66 matches
Mail list logo
