Kostas Sakellis created SPARK-5645:
--------------------------------------
Summary: Track local bytes read for shuffles - update UI
Key: SPARK-5645
URL: https://issues.apache.org/jira/browse/SPARK-5645
Project: Spark
Issue Type: New Feature
Components: Spark Core
Reporter: Kostas Sakellis
Currently we do not track the local bytes read for a shuffle read. The UI only
shows the remote bytes read. This is pretty confusing to the user because:
1) In local mode all shuffle reads are local
2) the shuffle bytes written from the previous stage might not add up if there
are some bytes that are read locally on the shuffle read side
3) With https://github.com/apache/spark/pull/4067 we display the total number
of records so that won't line up with only showing the remote bytes read.
I propose we track the remote and local bytes read separately. In the UI show
the total bytes read and in brackets show the remote bytes read for a shuffle.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
