This is an automated email from the ASF dual-hosted git repository. wusheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/master by this push: new 48c766ee01 Support RocketMQ monitoring (#11758) 48c766ee01 is described below commit 48c766ee011502f2ef441473bf2a1f3511a5a2cc Author: peachisai <2581009...@qq.com> AuthorDate: Sun Feb 18 07:17:20 2024 +0800 Support RocketMQ monitoring (#11758) * Support RocketMQ monitoring * Add swip3 - RocketMQ monitoring * Add RocketMQ e2e cases --------- Co-authored-by: peachisai <497533...@qq.com> Co-authored-by: 吴晟 Wu Sheng <wu.sh...@foxmail.com> Co-authored-by: 邵一鸣 <shaoy...@vanke.com> Co-authored-by: Wan Kai <wankai...@foxmail.com> --- .github/workflows/skywalking.yaml | 2 + docs/en/changes/changes.md | 4 + docs/en/concepts-and-designs/service-hierarchy.md | 46 +- .../backend/backend-rocketmq-monitoring.md} | 44 +- docs/en/swip/SWIP-3.md | 56 +- docs/menu.yml | 2 + .../skywalking/oap/server/core/analysis/Layer.java | 7 +- .../ui/template/UITemplateInitializer.java | 1 + .../src/main/resources/application.yml | 2 +- .../src/main/resources/hierarchy-definition.yml | 8 + .../otel-rules/rocketmq/rocketmq-broker.yaml | 46 ++ .../otel-rules/rocketmq/rocketmq-cluster.yaml | 81 +++ .../otel-rules/rocketmq/rocketmq-topic.yaml | 70 +++ .../resources/ui-initialized-templates/menu.yaml | 5 + .../rocketmq/rocketmq-broker.json | 142 +++++ .../rocketmq/rocketmq-cluster.json | 569 +++++++++++++++++++++ .../rocketmq/rocketmq-root.json | 41 ++ .../rocketmq/rocketmq-topic.json | 391 ++++++++++++++ .../virtual_mq/virtual-mq-service.json | 51 +- test/e2e-v2/cases/rocketmq/docker-compose.yml | 117 +++++ test/e2e-v2/cases/rocketmq/e2e.yaml | 37 ++ test/e2e-v2/cases/rocketmq/expected/endpoint.yml | 19 + test/e2e-v2/cases/rocketmq/expected/instance.yml | 22 + .../expected/metrics-has-value-empty-label.yml | 35 ++ .../expected/metrics-has-value-group-label.yml | 33 ++ .../rocketmq/expected/metrics-has-value-label.yml | 35 ++ .../cases/rocketmq/expected/metrics-has-value.yml | 31 ++ test/e2e-v2/cases/rocketmq/expected/service.yml | 24 + .../cases/rocketmq/otel-collector-config.yaml | 45 ++ test/e2e-v2/cases/rocketmq/rocketmq-cases.yaml | 89 ++++ 30 files changed, 1986 insertions(+), 69 deletions(-) diff --git a/.github/workflows/skywalking.yaml b/.github/workflows/skywalking.yaml index c4042a17eb..7a634f77fb 100644 --- a/.github/workflows/skywalking.yaml +++ b/.github/workflows/skywalking.yaml @@ -650,6 +650,8 @@ jobs: config: test/e2e-v2/cases/mqe/e2e.yaml - name: Pulsar and BookKeeper config: test/e2e-v2/cases/pulsar/e2e.yaml + - name: RocketMQ + config: test/e2e-v2/cases/rocketmq/e2e.yaml - name: UI Menu BanyanDB config: test/e2e-v2/cases/menu/banyandb/e2e.yaml diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index bb6b212119..0cad776e92 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -46,6 +46,10 @@ - APISIX -> K8S_SERVICE - GENERAL -> APISIX * Add Golang as a supported language for RocketMQ. +* Support Apache RocketMQ server monitoring. +* Add Service Hierarchy auto matching layer relationships (upper -> lower) as following: + - ROCKETMQ -> K8S_SERVICE + - VIRTUAL_MQ -> ROCKETMQ #### UI diff --git a/docs/en/concepts-and-designs/service-hierarchy.md b/docs/en/concepts-and-designs/service-hierarchy.md index 7642f81e4d..126eed6c60 100644 --- a/docs/en/concepts-and-designs/service-hierarchy.md +++ b/docs/en/concepts-and-designs/service-hierarchy.md @@ -11,20 +11,22 @@ There 2 ways to detect the connections: If you want to customize it according to your own needs, please refer to [Service Hierarchy Configuration](service-hierarchy-configuration.md). ### Automatically Matching -| Upper layer | Lower layer | Matching rule | -|-------------------|-------------|-------------------------------------------------------------------| -| MESH | MESH_DP | [MESH On MESH_DP](#mesh-on-mesh_dp) | -| MESH | K8S_SERVICE | [MESH On K8S_SERVICE](#mesh-on-k8s_service) | -| MESH_DP | K8S_SERVICE | [MESH_DP On K8S_SERVICE](#mesh_dp-on-k8s_service) | -| GENERAL | K8S_SERVICE | [GENERAL On K8S_SERVICE](#general-on-k8s_service) | -| GENERAL | APISIX | [GENERAL On APISIX](#general-on-apisix) | -| MYSQL | K8S_SERVICE | [MYSQL On K8S_SERVICE](#mysql-on-k8s_service) | -| POSTGRESQL | K8S_SERVICE | [POSTGRESQL On K8S_SERVICE](#postgresql-on-k8s_service) | -| SO11Y_OAP | K8S_SERVICE | [SO11Y_OAP On K8S_SERVICE](#so11y_oap-on-k8s_service) | -| VIRTUAL_DATABASE | MYSQL | [VIRTUAL_DATABASE On MYSQL](#virtual_database-on-mysql) | -| VIRTUAL_DATABASE | POSTGRESQL | [VIRTUAL_DATABASE On POSTGRESQL](#virtual_database-on-postgresql) | -| NGINX | K8S_SERVICE | [NGINX On K8S_SERVICE](#nginx-on-k8s_service) | -| APISIX | K8S_SERVICE | [APISIX On K8S_SERVICE](#apisix-on-k8s_service) | +| Upper layer | Lower layer | Matching rule | +|------------------|-------------|-------------------------------------------------------------------| +| MESH | MESH_DP | [MESH On MESH_DP](#mesh-on-mesh_dp) | +| MESH | K8S_SERVICE | [MESH On K8S_SERVICE](#mesh-on-k8s_service) | +| MESH_DP | K8S_SERVICE | [MESH_DP On K8S_SERVICE](#mesh_dp-on-k8s_service) | +| GENERAL | K8S_SERVICE | [GENERAL On K8S_SERVICE](#general-on-k8s_service) | +| GENERAL | APISIX | [GENERAL On APISIX](#general-on-apisix) | +| MYSQL | K8S_SERVICE | [MYSQL On K8S_SERVICE](#mysql-on-k8s_service) | +| POSTGRESQL | K8S_SERVICE | [POSTGRESQL On K8S_SERVICE](#postgresql-on-k8s_service) | +| SO11Y_OAP | K8S_SERVICE | [SO11Y_OAP On K8S_SERVICE](#so11y_oap-on-k8s_service) | +| VIRTUAL_DATABASE | MYSQL | [VIRTUAL_DATABASE On MYSQL](#virtual_database-on-mysql) | +| VIRTUAL_DATABASE | POSTGRESQL | [VIRTUAL_DATABASE On POSTGRESQL](#virtual_database-on-postgresql) | +| NGINX | K8S_SERVICE | [NGINX On K8S_SERVICE](#nginx-on-k8s_service) | +| APISIX | K8S_SERVICE | [APISIX On K8S_SERVICE](#apisix-on-k8s_service) | +| ROCKETMQ | K8S_SERVICE | [ROCKETMQ On K8S_SERVICE](#rocketmq-on-k8s_service) | +| VIRTUAL_MQ | ROCKETMQ | [VIRTUAL_MQ On K8S_SERVICE](#virtual_mq-on-rocketmq) | - The following sections will describe the **default matching rules** in detail and use the `upper-layer On lower-layer` format. - The example service name are based on SkyWalking [Showcase](https://github.com/apache/skywalking-showcase) default deployment. @@ -126,6 +128,22 @@ If you want to customize it according to your own needs, please refer to [Servic - APISIX.service.name: `APISIX::frontend.sample-services` - K8S_SERVICE.service.name: `skywalking-showcase::frontend.sample-services` +#### ROCKETMQ On K8S_SERVICE +- Rule name: `short-name` +- Groovy script: `{ (u, l) -> u.shortName == l.shortName }` +- Description: ROCKETMQ.service.shortName == K8S_SERVICE.service.shortName +- Matched Example: + - ROCKETMQ.service.name: `rocketmq::rocketmq.skywalking-showcase` + - K8S_SERVICE.service.name: `skywalking-showcase::rocketmq.skywalking-showcase` + +#### VIRTUAL_MQ On ROCKETMQ +- Rule name: `lower-short-name-with-fqdn` +- Groovy script: `{ (u, l) -> u.shortName.substring(0, u.shortName.lastIndexOf(':')) == l.shortName.concat('.svc.cluster.local') }` +- Description: VIRTUAL_MQ.service.shortName remove port == ROCKETMQ.service.shortName with fqdn suffix +- Matched Example: + - VIRTUAL_MQ.service.name: `rocketmq.skywalking-showcase.svc.cluster.local:9876` + - ROCKETMQ.service.name: `rocketmq::rocketmq.skywalking-showcase` + ### Build Through Specific Agents Use agent tech involved(such as eBPF) and deployment tools(such as operator and agent injector) detect the service hierarchy relations. diff --git a/docs/en/swip/SWIP-3.md b/docs/en/setup/backend/backend-rocketmq-monitoring.md similarity index 79% copy from docs/en/swip/SWIP-3.md copy to docs/en/setup/backend/backend-rocketmq-monitoring.md index 3ec40aa094..9904aa7299 100644 --- a/docs/en/swip/SWIP-3.md +++ b/docs/en/setup/backend/backend-rocketmq-monitoring.md @@ -1,13 +1,28 @@ -# Support RocketMQ Monitoring -## Motivation -RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications. Now that Skywalking can monitor OpenTelemetry metrics, I want to add RocketMQ monitoring via the OpenTelemetry Collector, which fetches metrics from the RocketMQ Exporter +# RocketMQ monitoring -## Architecture Graph -There is no significant architecture-level change. +SkyWalking leverages rocketmq-exporter for collecting metrics data from RocketMQ. It leverages OpenTelemetry +Collector to transfer the metrics to +[OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md). -## Proposed Changes -```rocketmq-exporter``` collects metrics from RocketMQ and transport the data to OpenTelemetry collector, using SkyWalking openTelemetry receiver to receive these metrics。 -Provide cluster, broker, and topic dimensions monitoring. +## Data flow + +1. The `rocketmq-exporter` (https://github.com/apache/rocketmq-exporter?tab=readme-ov-file#readme) collects metrics data from RocketMQ, The RocketMQ version is required to be 4.3.2+. +2. OpenTelemetry Collector fetches metrics from rocketmq-exporter via Prometheus Receiver and pushes metrics to + SkyWalking OAP Server via OpenTelemetry gRPC exporter. +3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to + filter/calculate/aggregate and store the results. + +## Setup + +1. Setup [rocketmq-exporter](https://github.com/apache/rocketmq-exporter). +2. Set up [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/getting-started/#docker). The example for OpenTelemetry Collector configuration, refer + to [here](../../../../test/e2e-v2/cases/rocketmq/otel-collector-config.yaml). +3. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md). + +## RocketMQ Monitoring + +RocketMQ monitoring provides multidimensional metrics monitoring of RocketMQ Exporter as `Layer: RocketMQ` `Service` in +the OAP. In each cluster, the broker is represented as `Instance` and the topic is represented as `Endpoint`. ### RocketMQ Cluster Supported Metrics @@ -50,12 +65,9 @@ Provide cluster, broker, and topic dimensions monitoring. | Producer Message Size | Byte/sec | meter_rocketmq_topic_producer_message_size | The max size of messages produced per second. | RocketMQ Exporter | | Consumer Message Size | Byte/sec | meter_rocketmq_topic_consumer_message_size | The max size of messages consumed per second. | RocketMQ Exporter | -## Imported Dependencies libs and their licenses. -No new dependency. - -## Compatibility -no breaking changes. - -## General usage docs +## Customizations -This feature is out of the box. +You can customize your own metrics/expression/dashboard panel. +The metrics definition and expression rules are found +in `otel-rules/rocketmq/rocketmq-cluster.yaml, otel-rules/rocketmq/rocketmq-broker.yaml, otel-rules/rocketmq/rocketmq-topic.yaml`. +The RocketMQ dashboard panel configurations are found in `ui-initialized-templates/rocketmq`. \ No newline at end of file diff --git a/docs/en/swip/SWIP-3.md b/docs/en/swip/SWIP-3.md index 3ec40aa094..b5d3029890 100644 --- a/docs/en/swip/SWIP-3.md +++ b/docs/en/swip/SWIP-3.md @@ -11,21 +11,23 @@ Provide cluster, broker, and topic dimensions monitoring. ### RocketMQ Cluster Supported Metrics -| Monitoring Panel |Unit | Metric Name | Description | Data Source | -|--------------------------------------------|------------|-------------------------------------------------------------------------|--------------------------------------------------------------------------|------------------| -| Messages Produced Today | Count | meter_rocketmq_cluster_messages_produced_today | The number of cluster messages produced today. | RocketMQ Exporter | -| Messages Consumed Today | Count | meter_rocketmq_cluster_messages_consumed_today | The number of cluster messages consumed today. | RocketMQ Exporter | -| Total Producer Tps | Msg/sec | meter_rocketmq_cluster_total_producer_tps | The number of messages produced per second. | RocketMQ Exporter | -| Total Consume Tps | Msg/sec | meter_rocketmq_cluster_total_consumer_tps | The number of messages consumed per second. | RocketMQ Exporter | -| Producer Message Size | Bytes/sec | meter_rocketmq_cluster_producer_message_size | The max size of a message produced per second. | RocketMQ Exporter | -| Consumer Message Size | Bytes/sec | meter_rocketmq_cluster_consumer_message_size | The max size of the consumed message per second. | RocketMQ Exporter | -| Messages Produced Until Yesterday | Count | meter_rocketmq_cluster_messages_produced_until_yesterday | The total number of messages put until 12 o'clock last night. | RocketMQ Exporter | -| Messages Consumed Until Yesterday | Count | meter_rocketmq_cluster_messages_consumed_until_yesterday | The total number of messages read until 12 o'clock last night. | RocketMQ Exporter | -| Max Consumer Latency | ms | meter_rocketmq_cluster_max_consumer_latency | The max number of consumer latency. | RocketMQ Exporter | -| Max CommitLog Disk Ratio | % | meter_rocketmq_cluster_max_commitLog_disk_ratio | The max utilization ratio of the commit log disk. | RocketMQ Exporter | -| CommitLog Disk Ratio | % | meter_rocketmq_cluster_commitLog_disk_ratio | The utilization ratio of the commit log disk per broker IP. | RocketMQ Exporter | -| Pull ThreadPool Queue Head Wait Time | ms | meter_rocketmq_cluster_pull_threadPool_queue_head_wait_time | The wait time in milliseconds for pulling threadPool queue per broker IP. | RocketMQ Exporter | -| Send ThreadPool Queue Head Wait Time | ms | meter_rocketmq_cluster_send_threadPool_queue_head_wait_time | The wait time in milliseconds for sending threadPool queue per broker IP. | RocketMQ Exporter | +| Monitoring Panel |Unit | Metric Name | Description | Data Source | +|--------------------------------------|------------|-------------------------------------------------------------|---------------------------------------------------------------------------|------------------| +| Messages Produced Today | Count | meter_rocketmq_cluster_messages_produced_today | The number of cluster messages produced today. | RocketMQ Exporter | +| Messages Consumed Today | Count | meter_rocketmq_cluster_messages_consumed_today | The number of cluster messages consumed today. | RocketMQ Exporter | +| Total Producer Tps | Msg/sec | meter_rocketmq_cluster_total_producer_tps | The number of messages produced per second. | RocketMQ Exporter | +| Total Consume Tps | Msg/sec | meter_rocketmq_cluster_total_consumer_tps | The number of messages consumed per second. | RocketMQ Exporter | +| Producer Message Size | Bytes/sec | meter_rocketmq_cluster_producer_message_size | The max size of a message produced per second. | RocketMQ Exporter | +| Consumer Message Size | Bytes/sec | meter_rocketmq_cluster_consumer_message_size | The max size of the consumed message per second. | RocketMQ Exporter | +| Messages Produced Until Yesterday | Count | meter_rocketmq_cluster_messages_produced_until_yesterday | The total number of messages put until 12 o'clock last night. | RocketMQ Exporter | +| Messages Consumed Until Yesterday | Count | meter_rocketmq_cluster_messages_consumed_until_yesterday | The total number of messages read until 12 o'clock last night. | RocketMQ Exporter | +| Max Consumer Latency | ms | meter_rocketmq_cluster_max_consumer_latency | The max number of consumer latency. | RocketMQ Exporter | +| Max CommitLog Disk Ratio | % | meter_rocketmq_cluster_max_commitLog_disk_ratio | The max utilization ratio of the commit log disk. | RocketMQ Exporter | +| CommitLog Disk Ratio | % | meter_rocketmq_cluster_commitLog_disk_ratio | The utilization ratio of the commit log disk per broker IP. | RocketMQ Exporter | +| Pull ThreadPool Queue Head Wait Time | ms | meter_rocketmq_cluster_pull_threadPool_queue_head_wait_time | The wait time in milliseconds for pulling threadPool queue per broker IP. | RocketMQ Exporter | +| Send ThreadPool Queue Head Wait Time | ms | meter_rocketmq_cluster_send_threadPool_queue_head_wait_time | The wait time in milliseconds for sending threadPool queue per broker IP. | RocketMQ Exporter | +| Topic Count | Count | meter_rocketmq_cluster_topic_count | The number of topics that received messages from the producer. | RocketMQ Exporter | +| Broker Count | Count | meter_rocketmq_cluster_broker_count | The number of brokers that received messages from the producer. | RocketMQ Exporter | ### RocketMQ Broker Supported Metrics @@ -38,17 +40,19 @@ Provide cluster, broker, and topic dimensions monitoring. ### RocketMQ Topic Supported Metrics -| Monitoring Panel |Unit | Metric Name | Description | Data Source | -|--------------------------------------------|------------|-------------------------------------------------------------------------|-----------------------------------------------------------------------|-------------------| -| Max Producer Message Size | Byte | meter_rocketmq_topic_max_producer_message_size | The maximum number of messages produced. | RocketMQ Exporter | -| Max Consumer Message Size | Byte | meter_rocketmq_topic_max_consumer_message_size | The maximum number of messages consumed. | RocketMQ Exporter | -| Consumer Latency | ms | meter_rocketmq_topic_consumer_latency | Consumption delay time of a consumer group. | RocketMQ Exporter | -| Producer Tps | Msg/sec | meter_rocketmq_topic_producer_tps | The number of messages produced per second. | RocketMQ Exporter | -| Consumer Group Tps | Msg/sec | meter_rocketmq_topic_consumer_group_tps | The number of messages consumed per second per consumer group. | RocketMQ Exporter | -| Producer Offset | Count | meter_rocketmq_topic_producer_offset | The max progress of a topic's production message. | RocketMQ Exporter | -| Consumer Group Offset | Count | meter_rocketmq_topic_consumer_group_offset | The max progress of a topic's consumption message per consumer group. | RocketMQ Exporter | -| Producer Message Size | Byte/sec | meter_rocketmq_topic_producer_message_size | The max size of messages produced per second. | RocketMQ Exporter | -| Consumer Message Size | Byte/sec | meter_rocketmq_topic_consumer_message_size | The max size of messages consumed per second. | RocketMQ Exporter | +| Monitoring Panel | Unit | Metric Name | Description | Data Source | +|---------------------------|-----------|------------------------------------------------------------------|-----------------------------------------------------------------------|-------------------| +| Max Producer Message Size | Byte | meter_rocketmq_topic_max_producer_message_size | The maximum number of messages produced. | RocketMQ Exporter | +| Max Consumer Message Size | Byte | meter_rocketmq_topic_max_consumer_message_size | The maximum number of messages consumed. | RocketMQ Exporter | +| Consumer Latency | ms | meter_rocketmq_topic_consumer_latency | Consumption delay time of a consumer group. | RocketMQ Exporter | +| Producer Tps | Msg/sec | meter_rocketmq_topic_producer_tps | The number of messages produced per second. | RocketMQ Exporter | +| Consumer Group Tps | Msg/sec | meter_rocketmq_topic_consumer_group_tps | The number of messages consumed per second per consumer group. | RocketMQ Exporter | +| Producer Offset | Count | meter_rocketmq_topic_producer_offset | The max progress of a topic's production message. | RocketMQ Exporter | +| Consumer Group Offset | Count | meter_rocketmq_topic_consumer_group_offset | The max progress of a topic's consumption message per consumer group. | RocketMQ Exporter | +| Producer Message Size | Byte/sec | meter_rocketmq_topic_producer_message_size | The max size of messages produced per second. | RocketMQ Exporter | +| Consumer Message Size | Byte/sec | meter_rocketmq_topic_consumer_message_size | The max size of messages consumed per second. | RocketMQ Exporter | +| Consumer Group_Count | Count | meter_rocketmq_topic_consumer_group_count | The number of consumer groups. | RocketMQ Exporter | +| Broker Count | Count | meter_rocketmq_topic_broker_count | The number of topics that received messages from the producer. | RocketMQ Exporter | ## Imported Dependencies libs and their licenses. No new dependency. diff --git a/docs/menu.yml b/docs/menu.yml index 8139edc123..06d5945413 100644 --- a/docs/menu.yml +++ b/docs/menu.yml @@ -260,6 +260,8 @@ catalog: path: "/en/setup/backend/backend-kafka-monitoring" - name: "Pulsar" path: "/en/setup/backend/backend-pulsar-monitoring" + - name: "RocketMQ" + path: "/en/setup/backend/backend-rocketmq-monitoring" - name: "Self Observability" catalog: - name: "OAP self telemetry" diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java index e57583b528..96d15236fd 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java @@ -213,7 +213,12 @@ public enum Layer { /** * Nginx is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server. */ - NGINX(34, true); + NGINX(34, true), + + /** + * A cloud native messaging and streaming platform, making it simple to build event-driven applications. + */ + ROCKETMQ(35, true); private final int value; /** diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java index 1aa7ba7e15..ec7c239928 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java @@ -72,6 +72,7 @@ public class UITemplateInitializer { Layer.PULSAR.name(), Layer.BOOKKEEPER.name(), Layer.NGINX.name(), + Layer.ROCKETMQ.name(), "custom" }; private final UITemplateManagementService uiTemplateManagementService; diff --git a/oap-server/server-starter/src/main/resources/application.yml b/oap-server/server-starter/src/main/resources/application.yml index aceea10fbe..147694a244 100644 --- a/oap-server/server-starter/src/main/resources/application.yml +++ b/oap-server/server-starter/src/main/resources/application.yml @@ -353,7 +353,7 @@ receiver-otel: selector: ${SW_OTEL_RECEIVER:default} default: enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"otlp-metrics,otlp-logs"} - enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:"apisix,nginx/*,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*,aws-gateway/*,redis/*,elasticsearch/*,rabbitmq/*,mongodb/*,kafka/*,pulsar/*,bookkeeper/*"} + enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:"apisix,nginx/*,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*,aws-gateway/*,redis/*,elasticsearch/*,rabbitmq/*,mongodb/*,kafka/*,pulsar/*,bookkeeper/*,rocketmq/*"} receiver-zipkin: selector: ${SW_RECEIVER_ZIPKIN:-} diff --git a/oap-server/server-starter/src/main/resources/hierarchy-definition.yml b/oap-server/server-starter/src/main/resources/hierarchy-definition.yml index 5a7cf115e9..f1d53cbb91 100644 --- a/oap-server/server-starter/src/main/resources/hierarchy-definition.yml +++ b/oap-server/server-starter/src/main/resources/hierarchy-definition.yml @@ -45,10 +45,16 @@ hierarchy: SO11Y_OAP: K8S_SERVICE: short-name + ROCKETMQ: + K8S_SERVICE: short-name + VIRTUAL_DATABASE: MYSQL: lower-short-name-with-fqdn POSTGRESQL: lower-short-name-with-fqdn + VIRTUAL_MQ: + ROCKETMQ: lower-short-name-with-fqdn + # Use Groovy script to define the matching rules, the input parameters are the upper service(u) and the lower service(l) and the return value is a boolean, # which are used to match the relation between the upper service(u) and the lower service(l) on the different layers. auto-matching-rules: @@ -70,11 +76,13 @@ layer-levels: GENERAL: 3 SO11Y_OAP: 3 VIRTUAL_DATABASE: 3 + VIRTUAL_MQ: 3 MYSQL: 2 POSTGRESQL: 2 APISIX: 2 NGINX: 2 + ROCKETMQ: 2 MESH_DP: 1 diff --git a/oap-server/server-starter/src/main/resources/otel-rules/rocketmq/rocketmq-broker.yaml b/oap-server/server-starter/src/main/resources/otel-rules/rocketmq/rocketmq-broker.yaml new file mode 100644 index 0000000000..4a6f721372 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/otel-rules/rocketmq/rocketmq-broker.yaml @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This will parse a textual representation of a duration. The formats +# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS} +# with days considered to be exactly 24 hours. +# <p> +# Examples: +# <pre> +# "PT20.345S" -- parses as "20.345 seconds" +# "PT15M" -- parses as "15 minutes" (where a minute is 60 seconds) +# "PT10H" -- parses as "10 hours" (where an hour is 3600 seconds) +# "P2D" -- parses as "2 days" (where a day is 24 hours or 86400 seconds) +# "P2DT3H4M" -- parses as "2 days, 3 hours and 4 minutes" +# "P-6H3M" -- parses as "-6 hours and +3 minutes" +# "-P6H3M" -- parses as "-6 hours and -3 minutes" +# "-P-6H+3M" -- parses as "+6 hours and -3 minutes" +# </pre> +filter: "{ tags -> tags.job_name == 'rocketmq-monitoring' }" # The OpenTelemetry job name +expSuffix: tag({tags -> tags.cluster = 'rocketmq::' + tags.cluster}).instance(['cluster'], ['broker'], Layer.ROCKETMQ) +metricPrefix: meter_rocketmq_broker +metricsRules: + + - name: produce_tps + exp: rocketmq_broker_tps.sum(['cluster', 'broker']) + + - name: consume_qps + exp: rocketmq_broker_qps.sum(['cluster','broker']) + + - name: producer_message_size + exp: rocketmq_producer_message_size.sum(['cluster','broker']).downsampling(MAX) + + - name: consumer_message_size + exp: rocketmq_consumer_message_size.sum(['cluster','broker']).downsampling(MAX) \ No newline at end of file diff --git a/oap-server/server-starter/src/main/resources/otel-rules/rocketmq/rocketmq-cluster.yaml b/oap-server/server-starter/src/main/resources/otel-rules/rocketmq/rocketmq-cluster.yaml new file mode 100644 index 0000000000..ec61ec5ac4 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/otel-rules/rocketmq/rocketmq-cluster.yaml @@ -0,0 +1,81 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This will parse a textual representation of a duration. The formats +# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS} +# with days considered to be exactly 24 hours. +# <p> +# Examples: +# <pre> +# "PT20.345S" -- parses as "20.345 seconds" +# "PT15M" -- parses as "15 minutes" (where a minute is 60 seconds) +# "PT10H" -- parses as "10 hours" (where an hour is 3600 seconds) +# "P2D" -- parses as "2 days" (where a day is 24 hours or 86400 seconds) +# "P2DT3H4M" -- parses as "2 days, 3 hours and 4 minutes" +# "P-6H3M" -- parses as "-6 hours and +3 minutes" +# "-P6H3M" -- parses as "-6 hours and -3 minutes" +# "-P-6H+3M" -- parses as "+6 hours and -3 minutes" +# </pre> + +filter: "{ tags -> tags.job_name == 'rocketmq-monitoring' }" # The OpenTelemetry job name +expSuffix: tag({tags -> tags.cluster = 'rocketmq::' + tags.cluster}).service(['cluster'], Layer.ROCKETMQ) +metricPrefix: meter_rocketmq_cluster + +metricsRules: + + - name: messages_produced_today + exp: rocketmq_brokeruntime_msg_put_total_today_now.sum((['cluster']))-rocketmq_brokeruntime_msg_puttotal_yesterdaymorning.sum(['cluster']) + + - name: messages_consumed_today + exp: rocketmq_brokeruntime_msg_gettotal_today_now.sum(['cluster'])-rocketmq_brokeruntime_msg_gettotal_yesterdaymorning.sum(['cluster']) + + - name: total_producer_tps + exp: rocketmq_producer_tps.sum(['cluster']) + + - name: total_consumer_tps + exp: rocketmq_consumer_tps.sum(['cluster']) + + - name: producer_message_size + exp: rocketmq_producer_message_size.sum(['cluster']).downsampling(MAX) + + - name: consumer_message_size + exp: rocketmq_consumer_message_size.sum(['cluster']).downsampling(MAX) + + - name: messages_produced_until_yesterday + exp: rocketmq_brokeruntime_msg_puttotal_yesterdaymorning.sum(['cluster']) + + - name: messages_consumed_until_yesterday + exp: rocketmq_brokeruntime_msg_gettotal_yesterdaymorning.sum(['cluster']) + + - name: max_consumer_latency + exp: rocketmq_group_get_latency_by_storetime.max(['cluster','broker','topic','group']) + + - name: max_commitLog_disk_ratio + exp: rocketmq_brokeruntime_commitlog_disk_ratio.max(['cluster','brokerIP'])*100 + + - name: commitLog_disk_ratio + exp: rocketmq_brokeruntime_commitlog_disk_ratio.sum(['cluster','brokerIP'])*100 + + - name: pull_threadPool_queue_head_wait_time + exp: rocketmq_brokeruntime_pull_threadpoolqueue_headwait_timemills.sum(['cluster','brokerIP']) + + - name: send_threadPool_queue_head_wait_time + exp: rocketmq_brokeruntime_send_threadpoolqueue_headwait_timemills.sum(['cluster','brokerIP']) + + - name: topic_count + exp: rocketmq_producer_offset.count(['cluster','topic']) + + - name: broker_count + exp: rocketmq_producer_offset.count(['cluster','broker']) \ No newline at end of file diff --git a/oap-server/server-starter/src/main/resources/otel-rules/rocketmq/rocketmq-topic.yaml b/oap-server/server-starter/src/main/resources/otel-rules/rocketmq/rocketmq-topic.yaml new file mode 100644 index 0000000000..50751da5e3 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/otel-rules/rocketmq/rocketmq-topic.yaml @@ -0,0 +1,70 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This will parse a textual representation of a duration. The formats +# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS} +# with days considered to be exactly 24 hours. +# <p> +# Examples: +# <pre> +# "PT20.345S" -- parses as "20.345 seconds" +# "PT15M" -- parses as "15 minutes" (where a minute is 60 seconds) +# "PT10H" -- parses as "10 hours" (where an hour is 3600 seconds) +# "P2D" -- parses as "2 days" (where a day is 24 hours or 86400 seconds) +# "P2DT3H4M" -- parses as "2 days, 3 hours and 4 minutes" +# "P-6H3M" -- parses as "-6 hours and +3 minutes" +# "-P6H3M" -- parses as "-6 hours and -3 minutes" +# "-P-6H+3M" -- parses as "+6 hours and -3 minutes" +# </pre> + +filter: "{ tags -> tags.job_name == 'rocketmq-monitoring' }" # The OpenTelemetry job name +expSuffix: tag({tags -> tags.cluster = 'rocketmq::' + tags.cluster}).endpoint(['cluster'], ['topic'], Layer.ROCKETMQ) +metricPrefix: meter_rocketmq_topic + +metricsRules: + + - name: max_producer_message_size + exp: rocketmq_producer_message_size.max(['cluster','topic']) + + - name: max_consumer_message_size + exp: rocketmq_consumer_message_size.max(['cluster','topic','group']) + + - name: consumer_latency + exp: rocketmq_group_get_latency_by_storetime.sum(['cluster','topic','group']) + + - name: producer_tps + exp: rocketmq_producer_tps.sum(['cluster','topic']) + + - name: consumer_group_tps + exp: rocketmq_consumer_tps.sum(['cluster','topic']) + + - name: producer_offset + exp: rocketmq_producer_offset.sum(['cluster','topic']).downsampling(MAX) + + - name: consumer_group_offset + exp: rocketmq_consumer_offset.sum(['cluster','topic','group']).downsampling(MAX) + + - name: producer_message_size + exp: rocketmq_producer_message_size.sum(['cluster','topic']).downsampling(MAX) + + - name: consumer_message_size + exp: rocketmq_consumer_message_size.sum(['cluster','topic','group']).downsampling(MAX) + + - name: consumer_group_count + exp: rocketmq_consumer_offset.count(['cluster','topic','group']) + + - name: broker_count + exp: rocketmq_producer_offset.count(['cluster','topic','broker']) + diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml b/oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml index 0b55c28855..b94a062c8f 100644 --- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml @@ -208,6 +208,11 @@ menus: description: Provide pulsar monitoring through OpenTelemetry's Prometheus Receiver. documentLink: https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-pulsar-monitoring/ i18nKey: mq_pulsar + - title: RocketMQ + layer: ROCKETMQ + description: Provide RocketMQ monitoring through OpenTelemetry's Prometheus Receiver. + documentLink: https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-rocketmq-monitoring/ + i18nKey: mq_rocketmq - title: Self Observability icon: self_observability description: Self Observability provides the observabilities for running components and servers from the SkyWalking ecosystem. diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/rocketmq/rocketmq-broker.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/rocketmq/rocketmq-broker.json new file mode 100644 index 0000000000..84dc465e19 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/rocketmq/rocketmq-broker.json @@ -0,0 +1,142 @@ +[ + { + "id": "RocketMQ-Broker", + "configuration": { + "children": [ + { + "x": 0, + "y": 0, + "w": 12, + "h": 14, + "i": "8", + "type": "Widget", + "id": "8", + "metricTypes": [], + "metrics": [], + "moved": false, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + }, + "widget": { + "title": "Produce Tps", + "tips": "The number of broker produces messages per second", + "name": "produce qps" + }, + "metricMode": "Expression", + "metricConfig": [], + "expressions": [ + "meter_rocketmq_broker_produce_tps" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ] + }, + { + "x": 12, + "y": 0, + "w": 12, + "h": 14, + "i": "9", + "type": "Widget", + "id": "9", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "meter_rocketmq_broker_consume_qps" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + }, + "widget": { + "title": "Consume Qps", + "tips": "The number of broker consumes messages per second", + "name": "consume qps" + } + }, + { + "x": 0, + "y": 14, + "w": 12, + "h": 13, + "i": "13", + "type": "Widget", + "id": "13", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "meter_rocketmq_broker_producer_message_size/1024/1024" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "widget": { + "name": "producer_message_size_per_second", + "title": "Producer Message Size(MB)", + "tips": "The size of the messages produced per second" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + } + }, + { + "x": 12, + "y": 14, + "w": 12, + "h": 13, + "i": "14", + "type": "Widget", + "id": "14", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "meter_rocketmq_broker_consumer_message_size/1024/1024" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "widget": { + "name": "consumer message size", + "title": "Consumer Message Size(MB)", + "tips": "The size of the messages consumed per second" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + } + } + ], + "layer": "ROCKETMQ", + "entity": "ServiceInstance", + "name": "RocketMQ-Broker", + "id": "RocketMQ-Broker" + } + } +] diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/rocketmq/rocketmq-cluster.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/rocketmq/rocketmq-cluster.json new file mode 100644 index 0000000000..12dce5b6c5 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/rocketmq/rocketmq-cluster.json @@ -0,0 +1,569 @@ +[ + { + "id": "RocketMQ-Cluster", + "configuration": { + "children": [ + { + "x": 0, + "y": 0, + "w": 24, + "h": 48, + "i": "0", + "type": "Tab", + "children": [ + { + "name": "Overview", + "children": [ + { + "x": 12, + "y": 0, + "w": 6, + "h": 6, + "i": "8", + "type": "Widget", + "id": "0-0-8", + "metricTypes": [], + "metrics": [], + "moved": false, + "graph": { + "type": "Card", + "fontSize": 26, + "textAlign": "center", + "showUnit": true + }, + "widget": { + "title": "Messages Produced Today", + "tips": "The number of the cluster messages produced today", + "name": "messages produced today" + }, + "metricMode": "Expression", + "metricConfig": [], + "expressions": [ + "latest(meter_rocketmq_cluster_messages_produced_today)" + ], + "typesOfMQE": [ + "SINGLE_VALUE" + ] + }, + { + "x": 18, + "y": 0, + "w": 6, + "h": 6, + "i": "9", + "type": "Widget", + "id": "0-0-9", + "metricTypes": [], + "metrics": [], + "moved": false, + "graph": { + "type": "Card", + "fontSize": 26, + "textAlign": "center", + "showUnit": true + }, + "widget": { + "title": "Messages Consumed Today", + "tips": "The number of the cluster messages consumed today", + "name": "messages consumed today" + }, + "metricMode": "Expression", + "metricConfig": [], + "expressions": [ + "latest(meter_rocketmq_cluster_messages_consumed_today)" + ], + "typesOfMQE": [ + "SINGLE_VALUE" + ] + }, + { + "x": 0, + "y": 20, + "w": 12, + "h": 14, + "i": "10", + "type": "Widget", + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + }, + "widget": { + "title": "Total Producer Tps", + "tips": "The number of messages produced per second", + "name": "total producer TPS" + }, + "expressions": [ + "meter_rocketmq_cluster_total_producer_tps" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "metricMode": "Expression", + "id": "0-0-10", + "moved": false, + "metrics": [], + "metricTypes": [], + "metricConfig": [] + }, + { + "x": 12, + "y": 20, + "w": 12, + "h": 14, + "i": "11", + "type": "Widget", + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + }, + "widget": { + "title": "Total Consume Tps", + "tips": "The number of messages consumed per second", + "name": "total consumer TPS" + }, + "expressions": [ + "meter_rocketmq_cluster_total_consumer_tps" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "metricMode": "Expression", + "id": "0-0-11", + "moved": false, + "metrics": [], + "metricTypes": [], + "metricConfig": [] + }, + { + "x": 0, + "y": 48, + "w": 12, + "h": 14, + "i": "12", + "type": "Widget", + "id": "0-0-12", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "meter_rocketmq_cluster_producer_message_size/1024/1024" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "graph": { + "type": "Bar", + "showBackground": true + }, + "widget": { + "name": "producer message size", + "title": "Producer Message Size(MB)", + "tips": "The size of a message produced per second" + } + }, + { + "x": 12, + "y": 48, + "w": 12, + "h": 14, + "i": "13", + "type": "Widget", + "id": "0-0-13", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "meter_rocketmq_cluster_consumer_message_size/1024/1024" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "graph": { + "type": "Bar", + "showBackground": true + }, + "widget": { + "name": "consumer message size", + "title": "Consumer Message Size(MB)", + "tips": "The size of the consumed message per second" + } + }, + { + "x": 0, + "y": 0, + "w": 6, + "h": 6, + "i": "16", + "type": "Widget", + "id": "0-0-16", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "graph": { + "type": "Card", + "fontSize": 26, + "textAlign": "center", + "showUnit": true + }, + "expressions": [ + "latest(meter_rocketmq_cluster_messages_produced_until_yesterday)" + ], + "typesOfMQE": [ + "SINGLE_VALUE" + ], + "widget": { + "name": "messages produced until yesterday", + "title": "Messages Produced Until Yesterday", + "tips": "The total number of messages put until 12 o'clock last night" + } + }, + { + "x": 6, + "y": 0, + "w": 6, + "h": 6, + "i": "17", + "type": "Widget", + "id": "0-0-17", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "graph": { + "type": "Card", + "fontSize": 26, + "textAlign": "center", + "showUnit": true + }, + "expressions": [ + "latest(meter_rocketmq_cluster_messages_consumed_until_yesterday)" + ], + "typesOfMQE": [ + "SINGLE_VALUE" + ], + "widget": { + "name": "messages consumed until yesterday", + "title": "Messages Consumed Until Yesterday", + "tips": "The total number of messages read until 12 o'clock last night" + } + }, + { + "x": 6, + "y": 6, + "w": 6, + "h": 7, + "i": "18", + "type": "Widget", + "id": "0-0-18", + "metricTypes": [], + "metrics": [], + "metricConfig": [ + { + "unit": "ms" + } + ], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "latest(meter_rocketmq_cluster_max_consumer_latency)" + ], + "typesOfMQE": [ + "SINGLE_VALUE" + ], + "graph": { + "type": "Card", + "fontSize": 26, + "textAlign": "center", + "showUnit": true + }, + "widget": { + "name": "max consumer latency", + "title": "Max Consumer Latency", + "tips": "The max number of consumer latency" + } + }, + { + "x": 0, + "y": 6, + "w": 6, + "h": 7, + "i": "19", + "type": "Widget", + "id": "0-0-19", + "metricTypes": [], + "metrics": [], + "metricConfig": [ + { + "unit": "%" + } + ], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "latest(meter_rocketmq_cluster_max_commitLog_disk_ratio)" + ], + "graph": { + "type": "Card", + "fontSize": 26, + "textAlign": "center", + "showUnit": true + }, + "widget": { + "name": "max commitLog disk ratio", + "title": "Max CommitLog Disk Ratio", + "tips": "The max utilization ratio of the commit log disk" + }, + "typesOfMQE": [ + "SINGLE_VALUE" + ] + }, + { + "x": 12, + "y": 6, + "w": 12, + "h": 14, + "i": "21", + "type": "Widget", + "id": "0-0-21", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "graph": { + "type": "Area", + "opacity": 0.4, + "showXAxis": true, + "showYAxis": true + }, + "expressions": [ + "meter_rocketmq_cluster_commitLog_disk_ratio" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "widget": { + "name": "commitLog disk ratio", + "title": "CommitLog Disk Ratio(%)", + "tips": "The utilization ratio of commit log disk per brokerIp" + } + }, + { + "x": 0, + "y": 34, + "w": 12, + "h": 14, + "i": "22", + "type": "Widget", + "id": "0-0-22", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "meter_rocketmq_cluster_pull_threadPool_queue_head_wait_time" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + }, + "widget": { + "name": "pull threadPool queue head wait time", + "title": "Pull ThreadPool Queue Head Wait Time(ms)", + "tips": "The wait time in milliseconds for pulling threadPool queue" + } + }, + { + "x": 12, + "y": 34, + "w": 12, + "h": 14, + "i": "23", + "type": "Widget", + "id": "0-0-23", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "meter_rocketmq_cluster_send_threadPool_queue_head_wait_time" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "widget": { + "name": "send threadPool queue head wait time", + "title": "Send ThreadPool Queue Head Wait Time(ms)", + "tips": "The wait time in milliseconds for sending threadPool queue" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + } + }, + { + "x": 0, + "y": 13, + "w": 6, + "h": 7, + "i": "24", + "type": "Widget", + "id": "0-0-24", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "latest(meter_rocketmq_cluster_topic_count)" + ], + "typesOfMQE": [ + "SINGLE_VALUE" + ], + "widget": { + "name": "topic count", + "title": "Topic Count", + "tips": "The number of topics that received messages from the producer" + }, + "graph": { + "type": "Card", + "fontSize": 26, + "textAlign": "center", + "showUnit": true + } + }, + { + "x": 6, + "y": 13, + "w": 6, + "h": 7, + "i": "25", + "type": "Widget", + "id": "0-0-25", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "latest(meter_rocketmq_cluster_broker_count)" + ], + "typesOfMQE": [ + "SINGLE_VALUE" + ], + "widget": { + "name": "broker count", + "title": "Broker Count", + "tips": "The number of brokers that received messages from the producer" + }, + "graph": { + "type": "Card", + "fontSize": 26, + "textAlign": "center", + "showUnit": true + } + } + ] + }, + { + "name": "Broker", + "children": [ + { + "x": 0, + "y": 0, + "w": 24, + "h": 30, + "i": "0", + "type": "Widget", + "graph": { + "type": "InstanceList", + "dashboardName": "RocketMQ-Broker", + "fontSize": 12 + }, + "metrics": [], + "metricTypes": [], + "metricConfig": [], + "id": "0-1-0" + } + ] + }, + { + "name": "Topic", + "children": [ + { + "x": 0, + "y": 16, + "w": 24, + "h": 48, + "i": "0", + "type": "Widget", + "graph": { + "type": "EndpointList", + "dashboardName": "RocketMQ-Topic", + "fontSize": 12 + }, + "metrics": [], + "metricTypes": [], + "metricConfig": [], + "id": "0-2-0" + } + ] + } + ], + "id": "0", + "activedTabIndex": 0, + "moved": false + } + ], + "layer": "ROCKETMQ", + "entity": "Service", + "name": "RocketMQ-Cluster", + "id": "RocketMQ-Cluster", + "isRoot": false, + "isDefault": true, + "expressions": [ + "avg(meter_rocketmq_cluster_total_producer_tps)", + "avg(meter_rocketmq_cluster_total_consumer_tps)", + "avg(meter_rocketmq_cluster_consumer_message_size)/1024/1024", + "avg(meter_rocketmq_cluster_producer_message_size)/1024/1024" + ], + "expressionsConfig": [ + { + "unit": "messages / s", + "label": "Producer TPS" + }, + { + "unit": "messages / s", + "label": "Consumer TPS" + }, + { + "unit": "MB / s", + "label": "Producer Message Size" + }, + { + "unit": "MB / s", + "label": "Consumer Message Size" + } + ] + } + } +] diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/rocketmq/rocketmq-root.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/rocketmq/rocketmq-root.json new file mode 100644 index 0000000000..579ffec4d1 --- /dev/null +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/rocketmq/rocketmq-root.json @@ -0,0 +1,41 @@ +[{ + "id": "RocketMQ-Root", + "configuration": { + "children": [{ + "x": 0, + "y": 3, + "w": 24, + "h": 29, + "i": "0", + "type": "Widget", + "graph": { + "type": "ServiceList", + "dashboardName": "RocketMQ-Cluster", + "fontSize": 12, + "showXAxis": false, + "showYAxis": false, + "showGroup": true + } + }, { + "x": 0, + "y": 0, + "w": 24, + "h": 3, + "i": "1", + "type": "Text", + "graph": { + "fontColor": "theme", + "backgroundColor": "theme", + "content": "Provide RocketMQ monitoring through OpenTelemetry's Prometheus Receiver", + "fontSize": 14, + "textAlign": "left", + "url": "https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-Rocketmq-monitoring/" + } + }], + "id": "Rocketmq-Root", + "layer": "ROCKETMQ", + "entity": "All", + "name": "Rocketmq-Root", + "isRoot": true + } +}] \ No newline at end of file diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/rocketmq/rocketmq-topic.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/rocketmq/rocketmq-topic.json new file mode 100644 index 0000000000..4e3060433a --- /dev/null +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/rocketmq/rocketmq-topic.json @@ -0,0 +1,391 @@ +[ + { + "id": "RocketMQ-Topic", + "configuration": { + "children": [ + { + "x": 0, + "y": 48, + "w": 12, + "h": 15, + "i": "0", + "type": "Widget", + "id": "0", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "graph": { + "type": "Bar", + "showBackground": true + }, + "expressions": [ + "meter_rocketmq_topic_producer_message_size/1024/1024" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "widget": { + "title": "Producer Message Size(MB)", + "tips": "The size of messages produced per second", + "name": "producer_message_size" + } + }, + { + "x": 12, + "y": 48, + "w": 12, + "h": 15, + "i": "1", + "type": "Widget", + "id": "1", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "graph": { + "type": "Bar", + "showBackground": true + }, + "expressions": [ + "meter_rocketmq_topic_consumer_message_size/1024/1024" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "widget": { + "name": "consumer_message_size", + "title": "Consumer Message Size(MB)", + "tips": "The size of messages consumed per second " + } + }, + { + "x": 6, + "y": 0, + "w": 6, + "h": 9, + "i": "2", + "type": "Widget", + "id": "2", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "metricConfig": [ + { + "unit": "MB" + } + ], + "moved": false, + "expressions": [ + "latest(meter_rocketmq_topic_max_consumer_message_size)/1024/1024" + ], + "typesOfMQE": [ + "SINGLE_VALUE" + ], + "widget": { + "name": "max_consumer_message_size", + "title": "Max Consumer Message Size", + "tips": "The maximum number of messages consumed" + }, + "graph": { + "type": "Card", + "fontSize": 40, + "textAlign": "center", + "showUnit": true + } + }, + { + "x": 0, + "y": 0, + "w": 6, + "h": 9, + "i": "3", + "type": "Widget", + "id": "3", + "metricTypes": [], + "metrics": [], + "metricConfig": [ + { + "unit": "MB" + } + ], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "latest(meter_rocketmq_topic_max_producer_message_size)/1024/1024" + ], + "typesOfMQE": [ + "SINGLE_VALUE" + ], + "widget": { + "name": "max_producer_message_size", + "title": "Max Producer Message Size", + "tips": "The maximum number of messages produced" + }, + "graph": { + "type": "Card", + "fontSize": 40, + "textAlign": "center", + "showUnit": true + } + }, + { + "x": 12, + "y": 35, + "w": 12, + "h": 13, + "i": "5", + "type": "Widget", + "id": "5", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "meter_rocketmq_topic_consumer_latency/1000" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "widget": { + "name": "consumer_latency", + "title": "Consumer Latency(s)", + "tips": "Consumption delay time of a consumer group" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + } + }, + { + "x": 0, + "y": 22, + "w": 12, + "h": 13, + "i": "6", + "type": "Widget", + "id": "6", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "meter_rocketmq_topic_producer_offset" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + }, + "widget": { + "title": "Producer Offset", + "name": "producer_offset", + "tips": "The progress of a topic's production message" + } + }, + { + "x": 12, + "y": 22, + "w": 12, + "h": 13, + "i": "7", + "type": "Widget", + "id": "7", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "meter_rocketmq_topic_consumer_group_offset" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "widget": { + "title": "Consumer Group Offset", + "name": "consumer_group_offset", + "tips": "The progress of a topic's consumption message per consumer group" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + } + }, + { + "x": 12, + "y": 9, + "w": 12, + "h": 13, + "i": "8", + "type": "Widget", + "id": "8", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "meter_rocketmq_topic_consumer_group_tps" + ], + "typesOfMQE": [ + "UNKNOWN" + ], + "widget": { + "title": "Consumer Group Tps", + "name": "consumer_group_tps", + "tips": "The number of messages consumed per second per consumer group" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + } + }, + { + "x": 0, + "y": 9, + "w": 12, + "h": 13, + "i": "9", + "type": "Widget", + "id": "9", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "meter_rocketmq_topic_producer_tps" + ], + "typesOfMQE": [ + "UNKNOWN" + ], + "widget": { + "title": "Producer Tps", + "name": "producer_tps", + "tips": "The number of messages produced per second" + }, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + } + }, + { + "x": 12, + "y": 0, + "w": 6, + "h": 9, + "i": "10", + "type": "Widget", + "id": "10", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "expressions": [ + "latest(meter_rocketmq_topic_consumer_group_count)" + ], + "graph": { + "type": "Card", + "fontSize": 40, + "textAlign": "center", + "showUnit": true + }, + "typesOfMQE": [ + "SINGLE_VALUE" + ], + "widget": { + "name": "consumer group count", + "title": "Consumer Group Count", + "tips": "The number of consumer groups" + } + }, + { + "x": 18, + "y": 0, + "w": 6, + "h": 9, + "i": "11", + "type": "Widget", + "id": "11", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "graph": { + "type": "Card", + "fontSize": 40, + "textAlign": "center", + "showUnit": true + }, + "expressions": [ + "latest(meter_rocketmq_topic_broker_count)" + ], + "typesOfMQE": [ + "UNKNOWN" + ], + "widget": { + "name": "broker count", + "title": "Broker Count", + "tips": "The number of topics that received messages from the producer" + } + }, + { + "x": 0, + "y": 35, + "w": 12, + "h": 13, + "i": "12", + "type": "Widget", + "id": "12", + "metricTypes": [], + "metrics": [], + "metricMode": "Expression", + "moved": false, + "graph": { + "type": "Line", + "step": false, + "smooth": false, + "showSymbol": true, + "showXAxis": true, + "showYAxis": true + }, + "expressions": [ + "meter_rocketmq_topic_producer_offset-meter_rocketmq_topic_consumer_group_offset" + ], + "typesOfMQE": [ + "TIME_SERIES_VALUES" + ], + "widget": { + "name": "backlogged messages", + "title": "Backlogged Messages", + "tips": "The number of backlogged messages in each group" + } + } + ], + "layer": "ROCKETMQ", + "entity": "Endpoint", + "name": "RocketMQ-Topic", + "id": "RocketMQ-Topic" + } + } +] \ No newline at end of file diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/virtual_mq/virtual-mq-service.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/virtual_mq/virtual-mq-service.json index cb43879df6..388e3dae6c 100644 --- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/virtual_mq/virtual-mq-service.json +++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/virtual_mq/virtual-mq-service.json @@ -12,7 +12,7 @@ "i": "1", "type": "Widget", "widget": { - "title": "Consume Traffic (calls / min)" + "title": "Consumer Traffic (calls / min)" }, "graph": { "type": "Line", @@ -32,7 +32,7 @@ "metricConfig": [ { "unit": "calls / min", - "label": "Consume Traffic" + "label": "Consumer Traffic" } ], "associate": [ @@ -64,7 +64,7 @@ "i": "2", "type": "Widget", "widget": { - "title": "Consume Access Successful Rate (%)", + "title": "Consumer Access Successful Rate (%)", "name": "Successful_Rate" }, "graph": { @@ -221,7 +221,7 @@ "i": "5", "type": "Widget", "widget": { - "title": "Produce Traffic (calls / min)" + "title": "Producer Traffic (calls / min)" }, "graph": { "type": "Line", @@ -241,7 +241,7 @@ "metricConfig": [ { "unit": "calls / min", - "label": "Produce Traffic" + "label": "Producer Traffic" } ], "associate": [ @@ -273,7 +273,7 @@ "i": "6", "type": "Widget", "widget": { - "title": "Produce Access Successful Rate (%)", + "title": "Producer Access Successful Rate (%)", "name": "Successful_Rate" }, "graph": { @@ -355,19 +355,19 @@ "label": "Transmission Latency" }, { - "label": "Consume Successful Rate", + "label": "Consumer Successful Rate", "unit": "%" }, { - "label": "Consume Traffic", + "label": "Consumer Traffic", "unit": "calls / min" }, { - "label": "Produce Successful Rate", + "label": "Producer Successful Rate", "unit": "%" }, { - "label": "Produce Traffic", + "label": "Producer Traffic", "unit": "calls / min" } ] @@ -378,7 +378,36 @@ "name": "Virtual-MQ-Service", "id": "Virtual-MQ-Service", "isRoot": false, - "isDefault": true + "isDefault": true, + "expressions": [ + "avg(mq_service_consume_latency)", + "avg(mq_endpoint_produce_sla)/100", + "avg(mq_endpoint_consume_sla)/100", + "avg(mq_service_produce_cpm)", + "avg(mq_service_consume_cpm)" + ], + "expressionsConfig": [ + { + "unit": "ms", + "label": "Consumer Latency" + }, + { + "unit": "%", + "label": "Producer Successful Rate" + }, + { + "unit": "%", + "label": "Consumer Successful Rate" + }, + { + "unit": "calls / min", + "label": "Producer Traffic" + }, + { + "unit": "calls / min", + "label": "Consumer Traffic" + } + ] } } ] diff --git a/test/e2e-v2/cases/rocketmq/docker-compose.yml b/test/e2e-v2/cases/rocketmq/docker-compose.yml new file mode 100644 index 0000000000..31807a60b2 --- /dev/null +++ b/test/e2e-v2/cases/rocketmq/docker-compose.yml @@ -0,0 +1,117 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +version: "3" + +services: + oap: + extends: + file: ../../script/docker-compose/base-compose.yml + service: oap + ports: + - "12800:12800" + networks: + - e2e + + rocketmq-namesrv-1: + image: apache/rocketmq:4.9.4 + hostname: rocketmq-namesrv-1 + ports: + - "9876:9876" + environment: + TZ: Asia/Shanghai + command: sh mqnamesrv + healthcheck: + test: [ "CMD", "nc", "-nz", "127.0.0.1", "9876" ] + interval: 15s + timeout: 10s + retries: 120 + networks: + - e2e + + rocketmq-broker-1: + image: apache/rocketmq:4.9.4 + hostname: rocketmq-broker-1 + ports: + - "10911:10911" + - "10909:10909" + environment: + TZ: Asia/Shanghai + NAMESRV_ADDR: "rocketmq-namesrv-1:9876" + command: sh mqbroker + healthcheck: + test: [ "CMD", "nc", "-nz", "127.0.0.1", "10911" ] + interval: 15s + timeout: 10s + retries: 120 + depends_on: + rocketmq-namesrv-1: + condition: service_healthy + networks: + - e2e + + rocketmq-producer-perf-test: + image: apache/rocketmq:4.9.4 + hostname: rocketmq-producer-perf-test + environment: + TZ: Asia/Shanghai + NAMESRV_ADDR: "rocketmq-namesrv-1:9876" + command: sh tools.sh org.apache.rocketmq.example.quickstart.Producer + depends_on: + rocketmq-broker-1: + condition: service_healthy + networks: + - e2e + + rocketmq-consumer-perf-test: + image: apache/rocketmq:4.9.4 + hostname: rocketmq-consumer-perf-test + environment: + TZ: Asia/Shanghai + NAMESRV_ADDR: "rocketmq-namesrv-1:9876" + command: sh tools.sh org.apache.rocketmq.example.quickstart.Consumer + depends_on: + - rocketmq-producer-perf-test + networks: + - e2e + + rocketmq-exporter: + image: apache/rocketmq-exporter:0.0.2 + hostname: rocketmq-exporter + ports: + - "5557:5557" + environment: + TZ: Asia/Shanghai + rocketmq.config.namesrvAddr: "rocketmq-namesrv-1:9876" + depends_on: + - rocketmq-consumer-perf-test + networks: + - e2e + + otel-collector: + image: otel/opentelemetry-collector:${OTEL_COLLECTOR_VERSION} + networks: + - e2e + command: [ "--config=/etc/otel-collector-config.yaml" ] + volumes: + - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml + expose: + - 55678 + depends_on: + oap: + condition: service_healthy + +networks: + e2e: \ No newline at end of file diff --git a/test/e2e-v2/cases/rocketmq/e2e.yaml b/test/e2e-v2/cases/rocketmq/e2e.yaml new file mode 100644 index 0000000000..0b7d36be33 --- /dev/null +++ b/test/e2e-v2/cases/rocketmq/e2e.yaml @@ -0,0 +1,37 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + +setup: + env: compose + file: docker-compose.yml + timeout: 20m + init-system-environment: ../../script/env + steps: + - name: set PATH + command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH + - name: install yq + command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq + - name: install swctl + command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl + +verify: + retry: + count: 60 + interval: 3s + cases: + - includes: + - ./rocketmq-cases.yaml \ No newline at end of file diff --git a/test/e2e-v2/cases/rocketmq/expected/endpoint.yml b/test/e2e-v2/cases/rocketmq/expected/endpoint.yml new file mode 100644 index 0000000000..a85d8e989b --- /dev/null +++ b/test/e2e-v2/cases/rocketmq/expected/endpoint.yml @@ -0,0 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{- contains .}} +- id: {{ b64enc "rocketmq::rocketmq-cluster" }}.1_{{ b64enc "TopicTest" }} + name: TopicTest +{{- end}} diff --git a/test/e2e-v2/cases/rocketmq/expected/instance.yml b/test/e2e-v2/cases/rocketmq/expected/instance.yml new file mode 100644 index 0000000000..5736faf1d5 --- /dev/null +++ b/test/e2e-v2/cases/rocketmq/expected/instance.yml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{- contains . }} +- id: {{ notEmpty .id }} + name: rocketmq-broker-1 + instanceuuid: {{ notEmpty .instanceuuid }} + attributes: [] + language: UNKNOWN +{{- end }} diff --git a/test/e2e-v2/cases/rocketmq/expected/metrics-has-value-empty-label.yml b/test/e2e-v2/cases/rocketmq/expected/metrics-has-value-empty-label.yml new file mode 100644 index 0000000000..64d711a38e --- /dev/null +++ b/test/e2e-v2/cases/rocketmq/expected/metrics-has-value-empty-label.yml @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + {{- contains .metric.labels }} + - key: _ + value: "" + {{- end}} + values: + { { - contains .values } } + - id: { { notEmpty .id } } + value: null + traceid: null + - id: { { notEmpty .id } } + value: { { .value } } + traceid: null + { { - end } } + {{- end}} +error: null diff --git a/test/e2e-v2/cases/rocketmq/expected/metrics-has-value-group-label.yml b/test/e2e-v2/cases/rocketmq/expected/metrics-has-value-group-label.yml new file mode 100644 index 0000000000..1b49846d57 --- /dev/null +++ b/test/e2e-v2/cases/rocketmq/expected/metrics-has-value-group-label.yml @@ -0,0 +1,33 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + - key: _ + value: please_rename_unique_group_name_4 + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: {{ .value }} + traceid: null + - id: {{ notEmpty .id }} + value: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/rocketmq/expected/metrics-has-value-label.yml b/test/e2e-v2/cases/rocketmq/expected/metrics-has-value-label.yml new file mode 100644 index 0000000000..a94c3c4ba0 --- /dev/null +++ b/test/e2e-v2/cases/rocketmq/expected/metrics-has-value-label.yml @@ -0,0 +1,35 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: + {{- contains .metric.labels }} + - key: _ + value: {{ .value }} + {{- end}} + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: {{ .value }} + traceid: null + - id: {{ notEmpty .id }} + value: null + traceid: null + {{- end}} + {{- end}} +error: null diff --git a/test/e2e-v2/cases/rocketmq/expected/metrics-has-value.yml b/test/e2e-v2/cases/rocketmq/expected/metrics-has-value.yml new file mode 100644 index 0000000000..c4bbbafa3b --- /dev/null +++ b/test/e2e-v2/cases/rocketmq/expected/metrics-has-value.yml @@ -0,0 +1,31 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +type: TIME_SERIES_VALUES +results: + {{- contains .results }} + - metric: + labels: [] + values: + {{- contains .values }} + - id: {{ notEmpty .id }} + value: {{ .value }} + traceid: null + - id: {{ notEmpty .id }} + value: null + traceid: null + {{- end}} + {{- end}} +error: null \ No newline at end of file diff --git a/test/e2e-v2/cases/rocketmq/expected/service.yml b/test/e2e-v2/cases/rocketmq/expected/service.yml new file mode 100644 index 0000000000..34a6e84aee --- /dev/null +++ b/test/e2e-v2/cases/rocketmq/expected/service.yml @@ -0,0 +1,24 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +{{- contains . }} +- id: {{ b64enc "rocketmq::rocketmq-cluster" }}.1 + name: rocketmq::rocketmq-cluster + group: rocketmq + shortname: rocketmq-cluster + layers: + - ROCKETMQ + normal: true +{{- end }} \ No newline at end of file diff --git a/test/e2e-v2/cases/rocketmq/otel-collector-config.yaml b/test/e2e-v2/cases/rocketmq/otel-collector-config.yaml new file mode 100644 index 0000000000..dc7db20441 --- /dev/null +++ b/test/e2e-v2/cases/rocketmq/otel-collector-config.yaml @@ -0,0 +1,45 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +receivers: + prometheus: + config: + scrape_configs: + - job_name: "rocketmq-monitoring" + scrape_interval: 30s + static_configs: + - targets: ['rocketmq-exporter:5557'] + relabel_configs: + - source_labels: [ ] + target_label: cluster + replacement: rocketmq-cluster + +exporters: + otlp: + endpoint: oap:11800 + tls: + insecure: true + +processors: + batch: +service: + pipelines: + metrics: + receivers: + - prometheus + processors: + - batch + exporters: + - otlp \ No newline at end of file diff --git a/test/e2e-v2/cases/rocketmq/rocketmq-cases.yaml b/test/e2e-v2/cases/rocketmq/rocketmq-cases.yaml new file mode 100644 index 0000000000..47a51bf195 --- /dev/null +++ b/test/e2e-v2/cases/rocketmq/rocketmq-cases.yaml @@ -0,0 +1,89 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file is used to show how to write configuration files and can be used to test. + +cases: + # service cases + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql service ls + expected: expected/service.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_messages_produced_today --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_messages_consumed_today --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_total_producer_tps --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_total_consumer_tps --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_producer_message_size --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_consumer_message_size --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_messages_produced_until_yesterday --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_messages_consumed_until_yesterday --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_max_consumer_latency --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value-label.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_max_commitLog_disk_ratio --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value-label.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_commitLog_disk_ratio --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value-label.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_pull_threadPool_queue_head_wait_time --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value-label.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_send_threadPool_queue_head_wait_time --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value-label.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_topic_count --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value-empty-label.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_cluster_broker_count --service-name=rocketmq::rocketmq-cluster + expected: expected/metrics-has-value-empty-label.yml + + # instance cases + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql instance ls --service-name=rocketmq::rocketmq-cluster + expected: expected/instance.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_broker_produce_tps --service-name=rocketmq::rocketmq-cluster --instance-name=rocketmq-broker-1 + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_broker_consume_qps --service-name=rocketmq::rocketmq-cluster --instance-name=rocketmq-broker-1 + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_broker_producer_message_size --service-name=rocketmq::rocketmq-cluster --instance-name=rocketmq-broker-1 + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_broker_consumer_message_size --service-name=rocketmq::rocketmq-cluster --instance-name=rocketmq-broker-1 + expected: expected/metrics-has-value.yml + + # endpoint cases + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql endpoint ls --service-name=rocketmq::rocketmq-cluster + expected: expected/endpoint.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_topic_max_producer_message_size --service-name=rocketmq::rocketmq-cluster --endpoint-name=TopicTest + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_topic_max_consumer_message_size --service-name=rocketmq::rocketmq-cluster --endpoint-name=TopicTest + expected: expected/metrics-has-value-group-label.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_topic_consumer_latency --service-name=rocketmq::rocketmq-cluster --endpoint-name=TopicTest + expected: expected/metrics-has-value-group-label.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_topic_producer_tps --service-name=rocketmq::rocketmq-cluster --endpoint-name=TopicTest + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_topic_consumer_group_tps --service-name=rocketmq::rocketmq-cluster --endpoint-name=TopicTest + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_topic_producer_offset --service-name=rocketmq::rocketmq-cluster --endpoint-name=TopicTest + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_topic_consumer_group_offset --service-name=rocketmq::rocketmq-cluster --endpoint-name=TopicTest + expected: expected/metrics-has-value-group-label.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_topic_producer_message_size --service-name=rocketmq::rocketmq-cluster --endpoint-name=TopicTest + expected: expected/metrics-has-value.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_topic_consumer_message_size --service-name=rocketmq::rocketmq-cluster --endpoint-name=TopicTest + expected: expected/metrics-has-value-group-label.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_topic_consumer_group_count --service-name=rocketmq::rocketmq-cluster --endpoint-name=TopicTest + expected: expected/metrics-has-value-empty-label.yml + - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_rocketmq_topic_broker_count --service-name=rocketmq::rocketmq-cluster --endpoint-name=TopicTest + expected: expected/metrics-has-value-empty-label.yml \ No newline at end of file