Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
w41ter merged PR #60520: URL: https://github.com/apache/doris/pull/60520 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
github-actions[bot] commented on PR #60520: URL: https://github.com/apache/doris/pull/60520#issuecomment-3864608992 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
hello-stephen commented on PR #60520: URL: https://github.com/apache/doris/pull/60520#issuecomment-3858767610 # Cloud UT Coverage Report Increment line coverage `89.32% (92/103)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/47502c2c91f15d5dd22233560b10b7b856542ce9_47502c2c91f15d5dd22233560b10b7b856542ce9_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/47502c2c91f15d5dd22233560b10b7b856542ce9_47502c2c91f15d5dd22233560b10b7b856542ce9_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 79.35% (1795/2262) | | Line Coverage | 64.86% (31957/49269) | | Region Coverage | 65.52% (15937/24323) | | Branch Coverage | 56.05% (8472/15114) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
Yukang-Lian commented on PR #60520: URL: https://github.com/apache/doris/pull/60520#issuecomment-3858689224 run cloudut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
Yukang-Lian commented on PR #60520: URL: https://github.com/apache/doris/pull/60520#issuecomment-3858440679 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
Yukang-Lian commented on PR #60520: URL: https://github.com/apache/doris/pull/60520#issuecomment-3857774976 run cloudut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
Yukang-Lian commented on PR #60520: URL: https://github.com/apache/doris/pull/60520#issuecomment-3854802779 run cloudut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
github-actions[bot] commented on PR #60520: URL: https://github.com/apache/doris/pull/60520#issuecomment-3851972759 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
github-actions[bot] commented on PR #60520: URL: https://github.com/apache/doris/pull/60520#issuecomment-3851972422 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
Yukang-Lian commented on PR #60520: URL: https://github.com/apache/doris/pull/60520#issuecomment-3851908670 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
w41ter commented on code in PR #60520:
URL: https://github.com/apache/doris/pull/60520#discussion_r2767387669
##
cloud/src/recycler/recycler_operation_log.cpp:
##
@@ -672,6 +684,53 @@ int InstanceRecycler::recycle_operation_logs() {
size_t recycled_operation_log_data_size = 0;
DORIS_CLOUD_DEFER {
+metrics_context.finish_report();
+
+if (stats_enabled) {
+g_bvar_recycler_oplog_last_round_total_num.put({instance_id_},
+
oplog_stats.total_num.load());
+g_bvar_recycler_oplog_last_round_not_recycled_num.put(
+{instance_id_}, oplog_stats.not_recycled_num.load());
+if (oplog_stats.failed_num.load() > 0) {
+g_bvar_recycler_oplog_recycle_failed_num.put({instance_id_},
+
oplog_stats.failed_num.load());
Review Comment:
Just a thought: should we use `add` rather than `put` here?
##
cloud/src/recycler/recycler_operation_log.cpp:
##
@@ -672,6 +684,53 @@ int InstanceRecycler::recycle_operation_logs() {
size_t recycled_operation_log_data_size = 0;
DORIS_CLOUD_DEFER {
+metrics_context.finish_report();
+
+if (stats_enabled) {
+g_bvar_recycler_oplog_last_round_total_num.put({instance_id_},
+
oplog_stats.total_num.load());
+g_bvar_recycler_oplog_last_round_not_recycled_num.put(
+{instance_id_}, oplog_stats.not_recycled_num.load());
+if (oplog_stats.failed_num.load() > 0) {
+g_bvar_recycler_oplog_recycle_failed_num.put({instance_id_},
+
oplog_stats.failed_num.load());
+}
+// Per-type last round counts (mBvarStatus, overwritten each round)
+g_bvar_recycler_oplog_last_round_recycled_commit_partition_num.put(
+{instance_id_},
oplog_stats.recycled_commit_partition.load());
+g_bvar_recycler_oplog_last_round_recycled_drop_partition_num.put(
+{instance_id_},
oplog_stats.recycled_drop_partition.load());
+g_bvar_recycler_oplog_last_round_recycled_commit_index_num.put(
+{instance_id_}, oplog_stats.recycled_commit_index.load());
+g_bvar_recycler_oplog_last_round_recycled_drop_index_num.put(
+{instance_id_}, oplog_stats.recycled_drop_index.load());
+g_bvar_recycler_oplog_last_round_recycled_update_tablet_num.put(
+{instance_id_}, oplog_stats.recycled_update_tablet.load());
+g_bvar_recycler_oplog_last_round_recycled_compaction_num.put(
+{instance_id_}, oplog_stats.recycled_compaction.load());
+g_bvar_recycler_oplog_last_round_recycled_schema_change_num.put(
+{instance_id_}, oplog_stats.recycled_schema_change.load());
+g_bvar_recycler_oplog_last_round_recycled_commit_txn_num.put(
+{instance_id_}, oplog_stats.recycled_commit_txn.load());
+// Per-type cumulative counts (mBvarIntAdder, accumulated across
rounds)
+g_bvar_recycler_oplog_recycled_commit_partition_num.put(
+{instance_id_},
oplog_stats.recycled_commit_partition.load());
+g_bvar_recycler_oplog_recycled_drop_partition_num.put(
+{instance_id_},
oplog_stats.recycled_drop_partition.load());
+g_bvar_recycler_oplog_recycled_commit_index_num.put(
+{instance_id_}, oplog_stats.recycled_commit_index.load());
+g_bvar_recycler_oplog_recycled_drop_index_num.put(
+{instance_id_}, oplog_stats.recycled_drop_index.load());
+g_bvar_recycler_oplog_recycled_update_tablet_num.put(
+{instance_id_}, oplog_stats.recycled_update_tablet.load());
+g_bvar_recycler_oplog_recycled_compaction_num.put(
+{instance_id_}, oplog_stats.recycled_compaction.load());
+g_bvar_recycler_oplog_recycled_schema_change_num.put(
+{instance_id_}, oplog_stats.recycled_schema_change.load());
+g_bvar_recycler_oplog_recycled_commit_txn_num.put(
+{instance_id_}, oplog_stats.recycled_commit_txn.load());
Review Comment:
`put` vs `add`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: commi
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
doris-robot commented on PR #60520: URL: https://github.com/apache/doris/pull/60520#issuecomment-3850982387 # Cloud UT Coverage Report Increment line coverage `86.09% (99/115)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/fa8e3a2b7c8056bfa775eac6120959c832e3cb74_fa8e3a2b7c8056bfa775eac6120959c832e3cb74_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/fa8e3a2b7c8056bfa775eac6120959c832e3cb74_fa8e3a2b7c8056bfa775eac6120959c832e3cb74_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 79.35% (1794/2261) | | Line Coverage | 64.91% (32019/49332) | | Region Coverage | 65.54% (15950/24337) | | Branch Coverage | 56.16% (8496/15128) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
gavinchou commented on code in PR #60520:
URL: https://github.com/apache/doris/pull/60520#discussion_r2766938420
##
cloud/src/recycler/recycler_operation_log.cpp:
##
@@ -672,6 +684,53 @@ int InstanceRecycler::recycle_operation_logs() {
size_t recycled_operation_log_data_size = 0;
DORIS_CLOUD_DEFER {
+metrics_context.finish_report();
+
+if (stats_enabled) {
Review Comment:
what is the cost to stats all the stats except
`scan_and_statistics_operation_logs()` ?
`scan_and_statistics_operation_logs()` may scan lots of KVs, it may be
turned off on demand, however, the rest stats seems light-weighted, it seems no
harm to collect the stats by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
gavinchou commented on code in PR #60520:
URL: https://github.com/apache/doris/pull/60520#discussion_r2766927177
##
cloud/src/recycler/recycler_operation_log.cpp:
##
@@ -672,6 +684,53 @@ int InstanceRecycler::recycle_operation_logs() {
size_t recycled_operation_log_data_size = 0;
DORIS_CLOUD_DEFER {
+metrics_context.finish_report();
+
+if (stats_enabled) {
Review Comment:
better move to a separate function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
Yukang-Lian commented on PR #60520: URL: https://github.com/apache/doris/pull/60520#issuecomment-3850899217 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [Enhancement](Recycler) Add bvar metrics for operation logs [doris]
hello-stephen commented on PR #60520: URL: https://github.com/apache/doris/pull/60520#issuecomment-3850899048 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
