[GitHub] [spark] AngersZhuuuu commented on pull request #31522: [SPARK-34399][SQL] Add commit duration to SQL tab's graph node.

[GitBox]

AngersZhuuuu commented on pull request #31522:
URL: https://github.com/apache/spark/pull/31522#issuecomment-793672126


   > How often is that? We can also improve the log to make it easier to search 
for a certain job.
   
   another case , we run a MSCK of table, in SQL tab it shows nothing.
   ![截屏2021-03-09 下午6 16 
20](https://user-images.githubusercontent.com/46485123/110455575-92a36980-8103-11eb-9a3a-565d3d4f7c15.png)
   
   But when it slow, we only can know how long it cost to collect path info in 
stage page.
   ![截屏2021-03-09 下午6 15 
55](https://user-images.githubusercontent.com/46485123/110455524-83bcb700-8103-11eb-82d6-8e36eb962197.png)
   
   But after collect path info and partition statistics. It also need to 
interact with hive. sometimes it is slow, user will ask why the job finished 
only cost 2 minutes but SQL's duration is 10 minutes.
   
   These duration metrics also important for such command for Spark admin to 
quick find the reason and reply to user.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org