Weizhong created SPARK-7339:
-------------------------------
Summary: PySpark shuffle spill memory sometimes are not correct
Key: SPARK-7339
URL: https://issues.apache.org/jira/browse/SPARK-7339
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 1.3.1
Reporter: Weizhong
Priority: Minor
In PySpark we get memory used before and after spill, then use the difference
of these two value as memorySpilled, but if the before value is small than
after value, then we will get a negative value, but this scenario 0 value may
be more reasonable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
