[ https://issues.apache.org/jira/browse/SPARK-25504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16625095#comment-16625095 ]
shahid commented on SPARK-25504: -------------------------------- According to the code in the AppStatusListener.scala, spark.ui.retainedTasks is a *soft threshold*, not a hard threshold. That means, number of task can display between (spark.ui.retainedTasks - spark.ui.retainedTasks/10 , spark.ui.retainedTasks] In this case number of tasks display between (60000-6000 to 60000], ie. (54000, 60000]. 58100 is in the range which is true. !AppStatusListener.png! > [Spark Job History]If the number of tasks exceeds spark.ui.retainedTask > value, then stage page of UI should display upto the value set in the > retainedTask. > ----------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-25504 > URL: https://issues.apache.org/jira/browse/SPARK-25504 > Project: Spark > Issue Type: Bug > Components: Web UI > Affects Versions: 2.3.1 > Reporter: ABHISHEK KUMAR GUPTA > Priority: Major > Attachments: AppStatusListener.png > > > *Steps* > 1. spark.ui.retainedTask =60000( spark-default.conf of Job History) > 2. Run a job for creating 100 task as below from Spark-shell > val rdd = sc.parallelize(1 to 49000, 49000) > rdd.count > 3. Launch Job History Page > 4. Click on corresponding spark-shell Application ID link in Job History and > Launch Job Page for the application > 5. Click on the Job ID description and Launch Stage Page > 6. Click on the Description of Stage ID to Launch Stage Page of specific > Stage ID > *Atcual Result* > Showing Total number of task above Pagination Panel as *Tasks(49000)* > It is correct as number of task is 49000 created. > Now Execute as below > 1. Run a job for creating 100010 task as below from Spark-shell > val rdd = sc.parallelize(1 to 100100, 100100)) > rdd.count > 2. Launch Job History Page > 3. Click on corresponding spark-shell Application ID link in Job History and > Launch Job Page for the application > 4. Click on the Job ID description and Launch Stage Page > 5. Click on the Description of Stage ID to Launch Stage Page of specific > Stage ID > *Atcual Result* > Showing Total number of task above Pagination Panel as *Tasks(58100, showing > 100100)* > *Expected:* > spark.ui.retainedTask =60000 > So it should show as *Tasks( 60000, Showing 100100)* as > spark.ui.retainedTask =60000 -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org