Github user patrickmcgloin commented on a diff in the pull request:
https://github.com/apache/spark/pull/21671#discussion_r199342794
--- Diff: python/pyspark/sql/functions.py ---
@@ -2163,9 +2163,9 @@ def json_tuple(col, *fields):
@since(2.1)
def from_json(col, schema, options={}):
"""
- Parses a column containing a JSON string into a :class:`MapType` with
:class:`StringType`
- as keys type, :class:`StructType` or :class:`ArrayType` of
:class:`StructType`\\s with
- the specified schema. Returns `null`, in the case of an unparseable
string.
+ Parses a column containing a JSON string into a :class:`MapType`,
:class:`StructType`
+ or :class:`ArrayType` of :class:`StructType`\\s with the specified
schema. Returns
+ `null`, in the case of an unparseable string.
--- End diff --
No, previously Dates and Times would not work. If you put a Timestamp in
the value of the key would be Long value of the time and when that would be
read back you would get and exception saying String could not be converted to
Long.
I just created the following test for a Boolean as a key and it fails. I
will make a generic solution for all key types with Unit Tests for each one.
WIll report back soon.
java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String
cannot be cast to java.lang.Boolean
'''
test("roundtrip in to_json and from_json - Boolean as key") {
val df = Seq(Map(true -> 1)).toDF("map")
val optionsDate = Map("dateFormat" -> "dd/MM/yyyy")
val schema = MapType(BooleanType, IntegerType)
val readBack = df.select(to_json($"map").as("json"))
.select(from_json($"json", schema).as("map"))
checkAnswer(df, readBack)
}
'''
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
