[ https://issues.apache.org/jira/browse/SPARK-9498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-9498. ------------------------------ Resolution: Not A Problem > Some statistical information missed when the driver is out of the cluster > ------------------------------------------------------------------------- > > Key: SPARK-9498 > URL: https://issues.apache.org/jira/browse/SPARK-9498 > Project: Spark > Issue Type: Improvement > Components: Web UI > Affects Versions: 1.3.1, 1.4.0 > Reporter: Liang Lee > > When an application is submited and the driver is out of the Spark cluster, > Some statistical information missed sometimes. > In stage detail inforamtion page, it will display following info when the > driver is in the spark cluster: > Details for Stage 7 > Total task time across all tasks: 37 min > Input Size / Records: 55.8 GB / 60488 > Shuffle write: 26.6 GB / 585242962 > But when the dreive is out of the spark cluster, it will sometimes display > above info, while sometimes not, just like this: > Details for Stage 7 > Total task time across all tasks: 37 min > That is the Input Size and Shuffle data does not display. > I have check the code and find that when the input size is zero then it will > not display. > And the input size is sent by each Executors and collected by Driver. > The problem is that the data that should be repored by Executors, is missed. > But I don't know why. Could anyone help to solve this problem? > -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org