Takako Shimamoto created SPARK-23043:
----------------------------------------
Summary: Upgrade json4s-jackson to 3.5.3
Key: SPARK-23043
URL: https://issues.apache.org/jira/browse/SPARK-23043
Project: Spark
Issue Type: Improvement
Components: Build
Affects Versions: 2.2.1
Reporter: Takako Shimamoto
Spark tried to upgrade json4s several times, but failed and still use a few
years old version 3.2.11. I hope the situation will improve soon.
h3. Migration from 3.2.11
Changes that affect Spark are as follows.
h4. scalap dependency
Since 3.3, json4s has its own scalap fork and doesn't depend on scala-lang's
scalap.
https://github.com/json4s/json4s/issues/108
Whereas Spark has added a scala-lang's scalap dependency in POM by the
following issue.
https://github.com/apache/spark/pull/480
We need to remove scala-lang's scalap dependency for json4s 3.5.3 (same
solution as in json4s).
h4. The behavior of \ function has changed
Please see https://github.com/json4s/json4s/pull/309
h5. a single element
{code}
val json =
"""[ {
| "jobId" : 0,
| "status" : "SUCCEEDED"
|} ]""".stripMargin
val ast = JsonMethods.parse(json)
{code}
* version 3.2.11
{code}
ast \ "status" // => JString(SUCCEEDED)
ast \\ "status" // => JString(SUCCEEDED)
{code}
* version 3.5.3
{code}
ast \ "status" // => JArray(List(JString(SUCCEEDED)))
ast \\ "status" // => JString(SUCCEEDED)
{code}
h5. more than a single elements
{code}
val json =
"""[ {
| "jobId" : 1,
| "status" : "FAILED"
|}, {
| "jobId" : 0,
| "status" : "SUCCEEDED"
|} ]""".stripMargin
val ast = JsonMethods.parse(json)
{code}
* version 3.2.11
{code}
ast \ "status" // => JArray(List(JString(FAILED), JString(SUCCEEDED)))
ast \\ "status" // => JObject(List((status,JString(FAILED)),
(status,JString(SUCCEEDED))))
{code}
* version 3.5.3
{code}
ast \ "status" // => JArray(List(JString(FAILED), JString(SUCCEEDED)))
ast \\ "status" // => JObject(List((status,JString(FAILED)),
(status,JString(SUCCEEDED))))
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
