Mark Khaitman created SPARK-5899: ------------------------------------ Summary: Viewing specific stage information which contains thousands of tasks will freak out the driver and CPU cores from where it runs Key: SPARK-5899 URL: https://issues.apache.org/jira/browse/SPARK-5899 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 1.2.1, 1.3.0 Environment: CentOS 7, Spark Standalone Reporter: Mark Khaitman Priority: Critical
If a user ever tries to view specific stage stats, for example, a repartition stage which involves 30000 partitions, the Web UI attempts to load every single task result onto a single webpage, which is completely destroying CPU usage on the driver, which subsequently causes the remaining tasks / jobs to be nearly impossible to complete. Ideally the task results should be paged (if not too much trouble) to prevent this from happening. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org