[jira] [Updated] (SPARK-5645) Track local bytes read for shuffles - update UI
[ https://issues.apache.org/jira/browse/SPARK-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kay Ousterhout updated SPARK-5645: -- Fix Version/s: 1.3.0 Assignee: Kay Ousterhout (was: Kostas Sakellis) Track local bytes read for shuffles - update UI --- Key: SPARK-5645 URL: https://issues.apache.org/jira/browse/SPARK-5645 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Kostas Sakellis Assignee: Kay Ousterhout Fix For: 1.3.0 Currently we do not track the local bytes read for a shuffle read. The UI only shows the remote bytes read. This is pretty confusing to the user because: 1) In local mode all shuffle reads are local 2) the shuffle bytes written from the previous stage might not add up if there are some bytes that are read locally on the shuffle read side 3) With https://github.com/apache/spark/pull/4067 we display the total number of records so that won't line up with only showing the remote bytes read. I propose we track the remote and local bytes read separately. In the UI show the total bytes read and in brackets show the remote bytes read for a shuffle. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-5645) Track local bytes read for shuffles - update UI
[ https://issues.apache.org/jira/browse/SPARK-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated SPARK-5645: -- Assignee: Kostas Sakellis Track local bytes read for shuffles - update UI --- Key: SPARK-5645 URL: https://issues.apache.org/jira/browse/SPARK-5645 Project: Spark Issue Type: New Feature Components: Spark Core Reporter: Kostas Sakellis Assignee: Kostas Sakellis Currently we do not track the local bytes read for a shuffle read. The UI only shows the remote bytes read. This is pretty confusing to the user because: 1) In local mode all shuffle reads are local 2) the shuffle bytes written from the previous stage might not add up if there are some bytes that are read locally on the shuffle read side 3) With https://github.com/apache/spark/pull/4067 we display the total number of records so that won't line up with only showing the remote bytes read. I propose we track the remote and local bytes read separately. In the UI show the total bytes read and in brackets show the remote bytes read for a shuffle. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
