Burak Yavuz created SPARK-23515:
-----------------------------------
Summary: JsonProtocol.sparkEventToJson can OOM when jsonifying an
event
Key: SPARK-23515
URL: https://issues.apache.org/jira/browse/SPARK-23515
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.2.1
Reporter: Burak Yavuz
Assignee: Burak Yavuz
{code}
def sparkEventToJson(event: SparkListenerEvent)
{code}
has a fallback method which creates a JSON object by turning an unrecognized
event to Json and then parsing it again. This method materializes the whole
string to parse the json record, which is unnecessary and can cause OOMs as
seen in the stacktrace below:
{code:java}
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at java.lang.StringBuilder.toString(StringBuilder.java:407)
at
com.fasterxml.jackson.core.util.TextBuffer.contentsAsString(TextBuffer.java:356)
at
com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:235)
at
org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:20)
at
org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:42)
at
org.json4s.jackson.JValueDeserializer.deserialize(JValueDeserializer.scala:35)
at
com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3736)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2726)
at org.json4s.jackson.JsonMethods$class.parse(JsonMethods.scala:20)
at org.json4s.jackson.JsonMethods$.parse(JsonMethods.scala:50)
at
org.apache.spark.util.JsonProtocol$.sparkEventToJson(JsonProtocol.scala:103){code}
We should just use the stream parsing to avoid such OOMs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
