This is an automated email from the ASF dual-hosted git repository.
wangdan pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pegasus.git
from 50493b1fc chore(asf.yml): update collaborators (#1757)
new 4028c12f8 feat(new_metrics): add replica-level metric entity (#1345)
new 2d7cb40b8 feat(new_metrics): migrate replica-level metrics for write
service (#1351)
new c3f524002 feat(new_metrics): migrate replica-level metrics for
pegasus_server_impl (part 1) (#1374)
new e6ce28b87 feat(new_metrics): migrate replica-level metrics for
pegasus_server_impl (part 2) (#1386)
new f5eafcbf3 feat(new_metrics): migrate replica-level metrics for
capacity_unit_calculator (#1387)
new 522626048 feat(new_metrics): migrate replica-level metrics for replica
class (#1397)
new c263b8b51 feat(new_metrics): migrate replica-level metrics for
pegasus_event_listener (#1407)
new 688b43630 feat(new_metrics): migrate replica-level metrics for
pegasus_mutation_duplicator (#1413)
new b1730fba5 feat(new_metrics): add server-level metric entity (#1415)
new 2d362f1e1 feat(new_metrics): migrate built-in server-level metrics
(#1418)
new 14c8a102d feat(new_metrics): migrate server-level metrics for nfs
(#1421)
new 659743d6d feat(new_metrics): add disk-level metric entity and migrate
disk-level metrics for fs_manager (#1427)
new dbaa01abd feat(new_metrics): add table-level metric entity and migrate
table-level metrics for server_state of meta (#1431)
new e2243751a feat(new_metrics): add partition-level metric entity and
migrate partition-level metrics for greedy_load_balancer of meta (#1435)
new fb5b5b017 feat(new_metrics): migrate server-level metrics for
meta_service (#1437)
new 3e88153a1 feat(new_metrics): add backup-policy-level metric entity and
migrate backup-policy-level metrics for meta_backup_service (#1438)
new 6bd8b9568 feat(new_metrics): migrate partition-level metrics for
partition_guardian (#1440)
new 23debd2d2 feat(new_metrics): migrate replica-level metrics for
pegasus_manual_compact_service (#1443)
new a5c841e2b feat(new_metrics): migrate metrics for replica_stub (part 1)
(#1455)
new 30221094d feat(new_metrics): migrate metrics for replica_stub (part 2)
(#1459)
new 15ab7a1d0 feat(collector): migrate the collector from
pegasus-kv/collector (#1461)
new f959ba4b4 feat(new_metrics): migrate metrics for replica_stub (part 3)
(#1462)
new 43a0b0e8a feat(new_metrics): migrate metrics for replica_stub (part 4)
(#1463)
new a5ae84682 feat(new_metrics): migrate metrics for replica_stub (part 5)
(#1469)
new ec9a1ae62 feat(new_metrics): migrate metrics for replica_stub (part 6)
(#1474)
new 5249fdc9e feat(new_metrics): migrate metrics for replica_stub (part 7)
(#1475)
new c4c835149 feat(new_metrics): migrate metrics for some duplication
class (#1482)
new cbfc0e842 feat(new_metrics): migrate metrics for task queue (#1484)
new ade5adfde refactor(new_metrics): refactor enum definition for metric
types and units (#1491)
new 8abadfc4d feat(new_metrics): migrate metrics for failure detector
(#1502)
new 556c6ac62 feat(new_metrics): migrate metrics for network (#1504)
new 79b4e6aa3 feat(new_metrics): migrate server-level metrics of rocksdb
(#1506)
new 93391cb2f feat: Aggregate table/server level metrics (#1517)
new 568227956 feat(new_metrics): migrate metrics for profiler (#1524)
new 71e516251 fix(new_metrics): profiled tasks are measured by the wrong
metrics (#1528)
new 8e3b1f036 feat(new_metrics): remove all table-level perf-counters for
each replica (#1531)
new 2169d9c43 refactor(new_metrics): remove perf-counters that are still
used in shared log (#1533)
new 50cf91606 feat(new_metrics): migrate metrics for latency tracer (#1537)
new 641e97bfa fix(new_metrics): total_capacity_mb/total_available_mb are
not atomic (#1542)
new e724b62c6 feat(new_metrics): remove http service for perf counters
(#1540)
new dfb50dcc0 feat(new_metrics): remove pegasus_counter_reporter (#1548)
new 83eeb8c4a feat(new_metrics): remove perf counter since shared log has
been deprecated
new 6be5e286f feat(new_metrics): remove deleted header files introduced
in source files for new metrics
new db8e780bc fix(new_metrics): fix unit tests in verifying values of
backup_requests and write_size_exceed_threshold_requests
new fa82854a3 fix(IWYU): fix the suggestions reported by IWYU while rebase
merging branch migrate-metrics-dev into master such as including
absl/strings/string_view.h
The 45 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
...t_admin-cli.yml => lint_and_test_collector.yml} | 33 +-
.github/workflows/lint_and_test_cpp.yaml | 30 +-
.github/workflows/module_labeler_conf.yml | 2 +
.gitignore | 3 +
src/geo/CMakeLists.txt => collector/Makefile | 10 +-
collector/README.md | 28 +
collector/aggregate/aggregatable.go | 92 +++
collector/aggregate/aggregator.go | 173 ++++++
collector/aggregate/aggregator_test.go | 97 +++
collector/aggregate/history.go | 111 ++++
.../aggregate/history_test.go | 34 +-
collector/aggregate/hook.go | 69 +++
collector/aggregate/hook_test.go | 51 ++
collector/aggregate/perf_client.go | 237 ++++++++
collector/aggregate/perf_client_test.go | 49 ++
collector/aggregate/perf_counter_decoder.go | 62 ++
collector/aggregate/perf_counter_decoder_test.go | 61 ++
collector/aggregate/perf_session.go | 91 +++
collector/aggregate/table_stats.go | 134 +++++
collector/avail/detector.go | 152 +++++
.../start_onebox.sh => collector/config.yml | 45 +-
collector/go.mod | 47 ++
{admin-cli => collector}/go.sum | 179 +++---
.../hotspot/algo.go | 8 +-
collector/main.go | 101 ++++
.../metrics/meta_server_metrics.go | 9 +-
collector/metrics/metric_collector.go | 514 ++++++++++++++++
.../metrics/replica_server_metrics.go | 9 +-
collector/sink/falcon_sink.go | 144 +++++
collector/sink/prometheus_sink.go | 110 ++++
collector/sink/sink.go | 53 ++
collector/templates/index.html | 80 +++
collector/usage/usage_recorder.go | 115 ++++
collector/webui/index.go | 84 +++
src/utils/clock.cpp => collector/webui/tables.go | 49 +-
collector/webui/webserver.go | 61 ++
run.sh | 10 +-
src/CMakeLists.txt | 1 -
src/base/test/CMakeLists.txt | 8 +-
src/common/fs_manager.cpp | 114 ++--
src/common/fs_manager.h | 63 +-
src/common/serialization_helper/thrift_helper.h | 1 -
src/common/test/CMakeLists.txt | 1 +
src/failure_detector/failure_detector.cpp | 17 +-
src/failure_detector/failure_detector.h | 4 +-
src/http/builtin_http_calls.cpp | 6 -
src/http/builtin_http_calls.h | 2 -
src/http/perf_counter_http_service.cpp | 66 ---
src/http/test/perf_counter_http_service_test.cpp | 76 ---
src/meta/greedy_load_balancer.cpp | 48 +-
src/meta/greedy_load_balancer.h | 7 -
src/meta/meta_backup_service.cpp | 50 +-
src/meta/meta_backup_service.h | 28 +-
src/meta/meta_service.cpp | 43 +-
src/meta/meta_service.h | 8 +-
src/meta/meta_split_service.cpp | 5 +
src/meta/partition_guardian.cpp | 14 +-
src/meta/partition_guardian.h | 2 -
src/meta/server_state.cpp | 81 +--
src/meta/server_state.h | 15 +-
src/meta/server_state_restore.cpp | 2 +
src/meta/table_metrics.cpp | 389 ++++++++++++
src/meta/table_metrics.h | 257 ++++++++
src/meta/test/backup_test.cpp | 2 +-
src/meta/test/meta_split_service_test.cpp | 4 +
src/meta/test/state_sync_test.cpp | 2 +
src/meta/test/update_configuration_test.cpp | 2 +
src/nfs/nfs_client_impl.cpp | 56 +-
src/nfs/nfs_client_impl.h | 10 +-
src/nfs/nfs_server_impl.cpp | 34 +-
src/nfs/nfs_server_impl.h | 6 +-
src/nfs/test/CMakeLists.txt | 2 +-
src/perf_counter/builtin_counters.cpp | 57 --
src/perf_counter/builtin_counters.h | 33 --
src/perf_counter/perf_counters.cpp | 13 -
src/perf_counter/perf_counters.h | 2 -
src/perf_counter/test/perf_counter_test.cpp | 1 +
src/perf_counter/test/perf_counters_test.cpp | 29 -
src/redis_protocol/proxy/CMakeLists.txt | 1 -
src/redis_protocol/proxy/main.cpp | 3 -
src/replica/backup/cold_backup_context.cpp | 28 +-
src/replica/backup/replica_backup_manager.cpp | 32 +-
src/replica/backup/replica_backup_manager.h | 9 +-
src/replica/bulk_load/replica_bulk_loader.cpp | 62 +-
src/replica/bulk_load/replica_bulk_loader.h | 9 +
src/replica/duplication/duplication_pipeline.cpp | 16 +-
src/replica/duplication/duplication_pipeline.h | 4 +-
src/replica/duplication/duplication_sync_timer.cpp | 7 +-
src/replica/duplication/load_from_private_log.cpp | 73 +--
src/replica/duplication/load_from_private_log.h | 14 +-
src/replica/duplication/mutation_batch.cpp | 21 +-
src/replica/duplication/mutation_batch.h | 4 +-
src/replica/duplication/replica_duplicator.cpp | 15 +-
src/replica/duplication/replica_duplicator.h | 6 +
.../duplication/replica_duplicator_manager.cpp | 16 +-
.../duplication/replica_duplicator_manager.h | 13 +-
.../test/load_from_private_log_test.cpp | 23 +-
src/replica/mutation_log.cpp | 6 -
src/replica/mutation_log.h | 16 +-
src/replica/replica.cpp | 357 +++++++----
src/replica/replica.h | 90 ++-
src/replica/replica_2pc.cpp | 15 +-
src/replica/replica_backup.cpp | 7 +-
.../replica_backup_manager.h => replica_base.cpp} | 53 +-
src/replica/replica_base.h | 21 +-
src/replica/replica_check.cpp | 5 +-
src/replica/replica_chkpt.cpp | 15 +-
src/replica/replica_learn.cpp | 27 +-
src/replica/replica_stub.cpp | 658 +++++++--------------
src/replica/replica_stub.h | 116 +---
src/replica/replica_throttle.cpp | 11 +-
src/replica/replication_app_base.cpp | 11 +-
src/replica/replication_app_base.h | 4 +
src/replica/split/replica_split_manager.cpp | 55 +-
src/replica/split/replica_split_manager.h | 8 +
src/replica/test/replica_disk_test_base.h | 1 -
src/replica/test/replica_test.cpp | 33 +-
src/reporter/CMakeLists.txt | 44 --
src/reporter/pegasus_counter_reporter.cpp | 381 ------------
src/reporter/pegasus_counter_reporter.h | 119 ----
src/reporter/pegasus_io_service.h | 65 --
src/runtime/profiler.cpp | 495 ++++++++--------
src/runtime/profiler_header.h | 69 ++-
src/runtime/rpc/network.cpp | 31 +-
src/runtime/rpc/network.h | 5 +-
src/runtime/rpc/rpc_message.cpp | 2 +-
src/runtime/task/task_queue.cpp | 80 ++-
src/runtime/task/task_queue.h | 19 +-
src/server/CMakeLists.txt | 1 -
src/server/capacity_unit_calculator.cpp | 149 ++---
src/server/capacity_unit_calculator.h | 29 +-
src/server/info_collector_app.cpp | 11 +-
src/server/info_collector_app.h | 1 -
src/server/pegasus_event_listener.cpp | 121 ++--
src/server/pegasus_event_listener.h | 26 +-
src/server/pegasus_manual_compact_service.cpp | 37 +-
src/server/pegasus_manual_compact_service.h | 6 +-
src/server/pegasus_mutation_duplicator.cpp | 33 +-
src/server/pegasus_mutation_duplicator.h | 6 +-
src/server/pegasus_server_impl.cpp | 412 ++++++-------
src/server/pegasus_server_impl.h | 96 +--
src/server/pegasus_server_impl_init.cpp | 406 +++++++------
src/server/pegasus_server_write.cpp | 24 +-
src/server/pegasus_server_write.h | 4 +-
src/server/pegasus_service_app.h | 47 +-
src/server/pegasus_write_service.cpp | 288 ++++-----
src/server/pegasus_write_service.h | 50 +-
src/server/pegasus_write_service_impl.h | 5 +-
src/server/rocksdb_wrapper.cpp | 9 +-
src/server/rocksdb_wrapper.h | 7 +-
src/server/test/CMakeLists.txt | 1 -
.../test/pegasus_mutation_duplicator_test.cpp | 1 +
src/server/test/pegasus_server_impl_test.cpp | 8 +-
src/server/test/pegasus_server_write_test.cpp | 4 +-
src/utils/CMakeLists.txt | 1 +
src/utils/builtin_metrics.cpp | 92 +++
src/utils/{preloadable.h => builtin_metrics.h} | 33 +-
src/utils/clock.cpp | 1 +
src/utils/enum_helper.h | 63 +-
src/utils/latency_tracer.cpp | 178 ++++--
src/utils/latency_tracer.h | 3 -
src/utils/metrics.cpp | 10 +-
src/utils/metrics.h | 405 ++++++++++++-
src/utils/test/enum_helper_test.cpp | 211 +++++++
src/utils/test/metrics_test.cpp | 184 +++++-
src/utils/time_utils.h | 26 +
166 files changed, 7269 insertions(+), 3593 deletions(-)
copy .github/workflows/{lint_and_test_admin-cli.yml =>
lint_and_test_collector.yml} (75%)
copy src/geo/CMakeLists.txt => collector/Makefile (90%)
create mode 100644 collector/README.md
create mode 100644 collector/aggregate/aggregatable.go
create mode 100644 collector/aggregate/aggregator.go
create mode 100644 collector/aggregate/aggregator_test.go
create mode 100644 collector/aggregate/history.go
copy src/runtime/ranger/access_type.cpp => collector/aggregate/history_test.go
(57%)
create mode 100644 collector/aggregate/hook.go
create mode 100644 collector/aggregate/hook_test.go
create mode 100644 collector/aggregate/perf_client.go
create mode 100644 collector/aggregate/perf_client_test.go
create mode 100644 collector/aggregate/perf_counter_decoder.go
create mode 100644 collector/aggregate/perf_counter_decoder_test.go
create mode 100644 collector/aggregate/perf_session.go
create mode 100644 collector/aggregate/table_stats.go
create mode 100644 collector/avail/detector.go
copy docker/pegasus-docker-compose/start_onebox.sh => collector/config.yml
(59%)
mode change 100755 => 100644
create mode 100644 collector/go.mod
copy {admin-cli => collector}/go.sum (87%)
copy src/http/service_version.cpp => collector/hotspot/algo.go (89%)
create mode 100644 collector/main.go
copy src/perf_counter/test/main.cpp =>
collector/metrics/meta_server_metrics.go (82%)
create mode 100644 collector/metrics/metric_collector.go
copy src/perf_counter/test/main.cpp =>
collector/metrics/replica_server_metrics.go (81%)
create mode 100644 collector/sink/falcon_sink.go
create mode 100644 collector/sink/prometheus_sink.go
create mode 100644 collector/sink/sink.go
create mode 100644 collector/templates/index.html
create mode 100644 collector/usage/usage_recorder.go
create mode 100644 collector/webui/index.go
copy src/utils/clock.cpp => collector/webui/tables.go (53%)
create mode 100644 collector/webui/webserver.go
delete mode 100644 src/http/perf_counter_http_service.cpp
delete mode 100644 src/http/test/perf_counter_http_service_test.cpp
create mode 100644 src/meta/table_metrics.cpp
create mode 100644 src/meta/table_metrics.h
delete mode 100644 src/perf_counter/builtin_counters.cpp
delete mode 100644 src/perf_counter/builtin_counters.h
copy src/replica/{backup/replica_backup_manager.h => replica_base.cpp} (50%)
delete mode 100644 src/reporter/CMakeLists.txt
delete mode 100644 src/reporter/pegasus_counter_reporter.cpp
delete mode 100644 src/reporter/pegasus_counter_reporter.h
delete mode 100644 src/reporter/pegasus_io_service.h
create mode 100644 src/utils/builtin_metrics.cpp
copy src/utils/{preloadable.h => builtin_metrics.h} (67%)
create mode 100644 src/utils/test/enum_helper_test.cpp
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
