[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
AngersZh commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-1081357770 Any suggestion? @tgravescs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
AngersZh commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-1033813586 gentle ping @tgravescs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
AngersZh commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-1014073504 ping @tgravescs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
AngersZh commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-1007968668 > at a high level seems fine, I need to take a more detailed look at a few parts. It also needs docs. Doc added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
AngersZh commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-1006317362 > It is great to see that more metrics are being added to History server. Updating the listing.ldb is one of the major factors impacting the History server performance, there are several new metrics tracked for listing.ldb. But there is also another factor, which is generating each individual application levelDB when there is a user request. There are some metrics tracking the LevelDB generating time and loading time in this PR. Can you help add some more metrics, such as how many applications are being parsed to generated their own LevelDBs, and how many LevelDBs are currently stored in Disk, and what is the total size of them. Thanks. Update a change. And I want confirm with you that: 1. If we don't set `spark.history.store.path`, it will use memory to store app info and it's not belong to LevelDB metrics, right? 2. Only when support hybrid store disk backend, then we have in-memory LevelDBs, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
AngersZh commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-1004554015 Gentle ping on this, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
AngersZh commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-999432545 Gentle ping @dongjoon-hyun @HyukjinKwon @tgravescs @HeartSaVioR @srowen -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
AngersZh commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-997564772 Can any committer review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
AngersZh commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-987684859 Any suggestion? Also ping @cloud-fan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
AngersZh commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-973683363 ping @HyukjinKwon also ping @mridulm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #34326: [SPARK-37053][CORE] Add metrics to SparkHistoryServer
AngersZh commented on pull request #34326: URL: https://github.com/apache/spark/pull/34326#issuecomment-971326096 @thejdeep @HyukjinKwon Update pr desc and add unit test to show the result. Hope for your review and can suggest if I have miss some thing that can be added to metrics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
