[
https://issues.apache.org/jira/browse/AMBARI-26638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jialiang Cai updated AMBARI-26638:
----------------------------------
Description:
h2. Problem
Ambari still depends on the legacy Ambari Metrics System, Ganglia integrations,
widget metadata, and timeline-metrics APIs. That architecture carries a large
HBase-based collector stack, is difficult to deploy and operate on current
platforms, and does not provide the discovery and query contracts required by
the React frontend.
The React frontend also assumes that known services have specialized summary
components. A newly introduced service such as VictoriaMetrics can therefore
be running correctly while its summary is blank and its sidebar health marker
remains unknown.
h2. Proposed Changes
Replace the legacy monitoring path with a Prometheus-compatible architecture
backed by VictoriaMetrics:
# Add an Ambari Agent telemetry subsystem that collects native Linux metrics,
reads component JMX endpoints from stack-defined telemetry profiles, exposes
Prometheus text format, and receives assignment updates through the existing
Ambari realtime channel.
# Add server-side target discovery, telemetry assignment compilation, HTTP
service discovery, Prometheus query proxying, datasource management,
dashboard persistence, chart sharing, and built-in dashboard provisioning.
# Add the VictoriaMetrics stack service with VictoriaMetrics Server, VMAGENT,
and VMAUTH components, verified multi-architecture artifacts, deployment
topology recommendations, runtime directory preparation, and package
metadata.
# Provision a managed Ambari identity for VMAGENT HTTP service discovery before
component startup. Persist only the required managed configuration and make
provisioning idempotent across cluster creation and service installation.
# Replace legacy Metrics, Ganglia, timeline, widget, and SCOM implementation
paths with the new Metrics APIs and schema upgrade.
# Add React monitoring routes and screens for dashboards, dashboard editing,
panel rendering, Prometheus exploration, targets, datasources, and shared
charts. Integrate monitoring links into service navigation and service
dashboards.
# Provide built-in Linux fleet and host dashboards plus service dashboards for
HDFS, YARN, HBase, and Hive.
# Add a generic React service summary for services without a specialized
summary. It must render master, worker, and client component health, counts,
alerts, maintenance state, host links, and an explicit empty state.
# Drive sidebar service health from the authoritative service state stream so
dynamically added services do not remain gray after they are started.
# Correct deployment integration by excluding the Metrics RPM from
stack-select, allowing the RPM version to be overridden, preserving encoded
PromQL parameters, tolerating missing realtime update payloads, and
enforcing a valid VictoriaMetrics component topology.
h2. Compatibility And Migration
The legacy Ambari Metrics System is intentionally removed rather than operated
in parallel. Upgrade catalog changes migrate the database model from legacy
widgets and layouts to dashboards, datasources, and chart shares. Existing
non-Metrics service management behavior remains unchanged.
h2. Acceptance Criteria
# Ambari Agent exposes assigned Linux and supported component telemetry in
Prometheus format and updates assignments without an agent restart.
# Ambari Server returns valid HTTP service-discovery targets and proxies
Prometheus queries without corrupting encoded query parameters.
# A Blueprint deployment can install and start VictoriaMetrics Server, VMAGENT,
and VMAUTH without manual credential creation or stack-select failures.
# The managed discovery identity is created before VMAGENT starts and repeated
provisioning remains idempotent.
# React users can manage datasources, inspect targets, explore metrics, and
view or edit provisioned dashboards.
# VictoriaMetrics and other services without specialized React summaries show
component status and navigation instead of a blank page.
# The service sidebar reflects the current service state and alert health.
# Focused server, agent, stack advisor, React tests, and a production frontend
build pass.
# A three-node Rocky Linux 8 ARM64 Blueprint deployment completes with all
VictoriaMetrics components in STARTED state and the React summary displaying
healthy service status.
h2. Runtime Evidence
The implementation was exercised on a local three-node Rocky Linux 8 ARM64
container cluster. The Blueprint request completed at 100 percent; the initial
managed {{victoriametrics-scrape}} configuration contained the generated service
discovery identity; VMAGENT completed its first INSTALL and START commands; and
VictoriaMetrics Server, VMAGENT, and VMAUTH all reached STARTED state. The
updated React assets were then deployed into the running Ambari Server and the
VictoriaMetrics summary and green sidebar health state were manually verified.
h2. Architecture Overview And Reviewer Guide
The monitoring architecture separates topology and assignment distribution from
metric values. Ambari remains the control plane and never transports
time-series samples in heartbeat or STOMP payloads.
{code:text}
Control plane
Stack telemetry.json + JMX profiles
|
v
Ambari Server assignment compiler
|
STOMP telemetry-v1
|
v
Agent last-known-good assignment cache
Data plane
Ambari HTTP SD -> VMAGENT -> Agent /metrics -> Linux collectors
-> /metrics/components/{routeId} -> /prom or /jmx
VMAGENT -> remote write -> VictoriaMetrics
React Monitoring -> Ambari query proxy -> VictoriaMetrics datasource
{code}
Review the design in this order:
# [End-to-end telemetry
architecture|https://github.com/JiaLiangC/ambari/blob/AMBARI-26638/docs/ambari-agent-prometheus-telemetry-architecture.md]:
control plane, data plane, Agent exporter, component routes, Stack contracts,
assignment lifecycle, security, HTTP service discovery, VictoriaMetrics
topology, query proxy, and dashboards.
# [Native React monitoring migration
plan|https://github.com/JiaLiangC/ambari/blob/AMBARI-26638/docs/frontend-refactor/prometheus-migration-plan.md]:
goals, non-goals, frontend/backend ownership, API and persistence boundaries,
upgrade behavior, and implementation phases.
# [Prometheus source behavior
baseline|https://github.com/JiaLiangC/ambari/blob/AMBARI-26638/docs/frontend-refactor/prometheus-source-baseline.md]:
source routes, datasource/dashboard contracts, permissions, failure handling,
and recovery requirements.
# [Prometheus source
inventory|https://github.com/JiaLiangC/ambari/blob/AMBARI-26638/docs/frontend-refactor/prometheus-source-inventory.md]:
source revisions, selected subsystems, and explicit exclusions.
# [Agent telemetry implementation
plan|https://github.com/JiaLiangC/ambari/blob/AMBARI-26638/docs/ambari-agent-prometheus-telemetry-plan.md]:
phased Agent/Server integration and verification scope.
# [Three-node runtime
evidence|https://github.com/JiaLiangC/ambari/blob/AMBARI-26638/docs/frontend-refactor/runtime-evidence/AMBARI-26638/README.md]:
deployed service state, scrape targets, and dashboard screenshots.
was:
h2. Problem
Ambari still depends on the legacy Ambari Metrics System, Ganglia integrations,
widget metadata, and timeline-metrics APIs. That architecture carries a large
HBase-based collector stack, is difficult to deploy and operate on current
platforms, and does not provide the discovery and query contracts required by
the React frontend.
The React frontend also assumes that known services have specialized summary
components. A newly introduced service such as VictoriaMetrics can therefore
be running correctly while its summary is blank and its sidebar health marker
remains unknown.
h2. Proposed Changes
Replace the legacy monitoring path with a Prometheus-compatible architecture
backed by VictoriaMetrics:
# Add an Ambari Agent telemetry subsystem that collects native Linux metrics,
reads component JMX endpoints from stack-defined telemetry profiles, exposes
Prometheus text format, and receives assignment updates through the existing
Ambari realtime channel.
# Add server-side target discovery, telemetry assignment compilation, HTTP
service discovery, Prometheus query proxying, datasource management,
dashboard persistence, chart sharing, and built-in dashboard provisioning.
# Add the VictoriaMetrics stack service with VictoriaMetrics Server, VMAGENT,
and VMAUTH components, verified multi-architecture artifacts, deployment
topology recommendations, runtime directory preparation, and package
metadata.
# Provision a managed Ambari identity for VMAGENT HTTP service discovery before
component startup. Persist only the required managed configuration and make
provisioning idempotent across cluster creation and service installation.
# Replace legacy Metrics, Ganglia, timeline, widget, and SCOM implementation
paths with the new Metrics APIs and schema upgrade.
# Add React monitoring routes and screens for dashboards, dashboard editing,
panel rendering, Prometheus exploration, targets, datasources, and shared
charts. Integrate monitoring links into service navigation and service
dashboards.
# Provide built-in Linux fleet and host dashboards plus service dashboards for
HDFS, YARN, HBase, and Hive.
# Add a generic React service summary for services without a specialized
summary. It must render master, worker, and client component health, counts,
alerts, maintenance state, host links, and an explicit empty state.
# Drive sidebar service health from the authoritative service state stream so
dynamically added services do not remain gray after they are started.
# Correct deployment integration by excluding the Metrics RPM from
stack-select, allowing the RPM version to be overridden, preserving encoded
PromQL parameters, tolerating missing realtime update payloads, and
enforcing a valid VictoriaMetrics component topology.
h2. Compatibility And Migration
The legacy Ambari Metrics System is intentionally removed rather than operated
in parallel. Upgrade catalog changes migrate the database model from legacy
widgets and layouts to dashboards, datasources, and chart shares. Existing
non-Metrics service management behavior remains unchanged.
h2. Acceptance Criteria
# Ambari Agent exposes assigned Linux and supported component telemetry in
Prometheus format and updates assignments without an agent restart.
# Ambari Server returns valid HTTP service-discovery targets and proxies
Prometheus queries without corrupting encoded query parameters.
# A Blueprint deployment can install and start VictoriaMetrics Server, VMAGENT,
and VMAUTH without manual credential creation or stack-select failures.
# The managed discovery identity is created before VMAGENT starts and repeated
provisioning remains idempotent.
# React users can manage datasources, inspect targets, explore metrics, and
view or edit provisioned dashboards.
# VictoriaMetrics and other services without specialized React summaries show
component status and navigation instead of a blank page.
# The service sidebar reflects the current service state and alert health.
# Focused server, agent, stack advisor, React tests, and a production frontend
build pass.
# A three-node Rocky Linux 8 ARM64 Blueprint deployment completes with all
VictoriaMetrics components in STARTED state and the React summary displaying
healthy service status.
h2. Runtime Evidence
The implementation was exercised on a local three-node Rocky Linux 8 ARM64
container cluster. The Blueprint request completed at 100 percent; the initial
managed {{victoriametrics-scrape}} configuration contained the generated service
discovery identity; VMAGENT completed its first INSTALL and START commands; and
VictoriaMetrics Server, VMAGENT, and VMAUTH all reached STARTED state. The
updated React assets were then deployed into the running Ambari Server and the
VictoriaMetrics summary and green sidebar health state were manually verified.
> Replace legacy Ambari Metrics with Prometheus and VictoriaMetrics
> -----------------------------------------------------------------
>
> Key: AMBARI-26638
> URL: https://issues.apache.org/jira/browse/AMBARI-26638
> Project: Ambari
> Issue Type: New Feature
> Reporter: Jialiang Cai
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h2. Problem
> Ambari still depends on the legacy Ambari Metrics System, Ganglia
> integrations,
> widget metadata, and timeline-metrics APIs. That architecture carries a large
> HBase-based collector stack, is difficult to deploy and operate on current
> platforms, and does not provide the discovery and query contracts required by
> the React frontend.
> The React frontend also assumes that known services have specialized summary
> components. A newly introduced service such as VictoriaMetrics can therefore
> be running correctly while its summary is blank and its sidebar health marker
> remains unknown.
> h2. Proposed Changes
> Replace the legacy monitoring path with a Prometheus-compatible architecture
> backed by VictoriaMetrics:
> # Add an Ambari Agent telemetry subsystem that collects native Linux metrics,
> reads component JMX endpoints from stack-defined telemetry profiles,
> exposes
> Prometheus text format, and receives assignment updates through the
> existing
> Ambari realtime channel.
> # Add server-side target discovery, telemetry assignment compilation, HTTP
> service discovery, Prometheus query proxying, datasource management,
> dashboard persistence, chart sharing, and built-in dashboard provisioning.
> # Add the VictoriaMetrics stack service with VictoriaMetrics Server, VMAGENT,
> and VMAUTH components, verified multi-architecture artifacts, deployment
> topology recommendations, runtime directory preparation, and package
> metadata.
> # Provision a managed Ambari identity for VMAGENT HTTP service discovery
> before
> component startup. Persist only the required managed configuration and make
> provisioning idempotent across cluster creation and service installation.
> # Replace legacy Metrics, Ganglia, timeline, widget, and SCOM implementation
> paths with the new Metrics APIs and schema upgrade.
> # Add React monitoring routes and screens for dashboards, dashboard editing,
> panel rendering, Prometheus exploration, targets, datasources, and shared
> charts. Integrate monitoring links into service navigation and service
> dashboards.
> # Provide built-in Linux fleet and host dashboards plus service dashboards for
> HDFS, YARN, HBase, and Hive.
> # Add a generic React service summary for services without a specialized
> summary. It must render master, worker, and client component health,
> counts,
> alerts, maintenance state, host links, and an explicit empty state.
> # Drive sidebar service health from the authoritative service state stream so
> dynamically added services do not remain gray after they are started.
> # Correct deployment integration by excluding the Metrics RPM from
> stack-select, allowing the RPM version to be overridden, preserving
> encoded
> PromQL parameters, tolerating missing realtime update payloads, and
> enforcing a valid VictoriaMetrics component topology.
> h2. Compatibility And Migration
> The legacy Ambari Metrics System is intentionally removed rather than operated
> in parallel. Upgrade catalog changes migrate the database model from legacy
> widgets and layouts to dashboards, datasources, and chart shares. Existing
> non-Metrics service management behavior remains unchanged.
> h2. Acceptance Criteria
> # Ambari Agent exposes assigned Linux and supported component telemetry in
> Prometheus format and updates assignments without an agent restart.
> # Ambari Server returns valid HTTP service-discovery targets and proxies
> Prometheus queries without corrupting encoded query parameters.
> # A Blueprint deployment can install and start VictoriaMetrics Server,
> VMAGENT,
> and VMAUTH without manual credential creation or stack-select failures.
> # The managed discovery identity is created before VMAGENT starts and repeated
> provisioning remains idempotent.
> # React users can manage datasources, inspect targets, explore metrics, and
> view or edit provisioned dashboards.
> # VictoriaMetrics and other services without specialized React summaries show
> component status and navigation instead of a blank page.
> # The service sidebar reflects the current service state and alert health.
> # Focused server, agent, stack advisor, React tests, and a production frontend
> build pass.
> # A three-node Rocky Linux 8 ARM64 Blueprint deployment completes with all
> VictoriaMetrics components in STARTED state and the React summary
> displaying
> healthy service status.
> h2. Runtime Evidence
> The implementation was exercised on a local three-node Rocky Linux 8 ARM64
> container cluster. The Blueprint request completed at 100 percent; the initial
> managed {{victoriametrics-scrape}} configuration contained the generated
> service
> discovery identity; VMAGENT completed its first INSTALL and START commands;
> and
> VictoriaMetrics Server, VMAGENT, and VMAUTH all reached STARTED state. The
> updated React assets were then deployed into the running Ambari Server and the
> VictoriaMetrics summary and green sidebar health state were manually verified.
> h2. Architecture Overview And Reviewer Guide
> The monitoring architecture separates topology and assignment distribution
> from metric values. Ambari remains the control plane and never transports
> time-series samples in heartbeat or STOMP payloads.
> {code:text}
> Control plane
> Stack telemetry.json + JMX profiles
> |
> v
> Ambari Server assignment compiler
> |
> STOMP telemetry-v1
> |
> v
> Agent last-known-good assignment cache
> Data plane
> Ambari HTTP SD -> VMAGENT -> Agent /metrics -> Linux collectors
> -> /metrics/components/{routeId} -> /prom or
> /jmx
> VMAGENT -> remote write -> VictoriaMetrics
> React Monitoring -> Ambari query proxy -> VictoriaMetrics datasource
> {code}
> Review the design in this order:
> # [End-to-end telemetry
> architecture|https://github.com/JiaLiangC/ambari/blob/AMBARI-26638/docs/ambari-agent-prometheus-telemetry-architecture.md]:
> control plane, data plane, Agent exporter, component routes, Stack
> contracts, assignment lifecycle, security, HTTP service discovery,
> VictoriaMetrics topology, query proxy, and dashboards.
> # [Native React monitoring migration
> plan|https://github.com/JiaLiangC/ambari/blob/AMBARI-26638/docs/frontend-refactor/prometheus-migration-plan.md]:
> goals, non-goals, frontend/backend ownership, API and persistence
> boundaries, upgrade behavior, and implementation phases.
> # [Prometheus source behavior
> baseline|https://github.com/JiaLiangC/ambari/blob/AMBARI-26638/docs/frontend-refactor/prometheus-source-baseline.md]:
> source routes, datasource/dashboard contracts, permissions, failure
> handling, and recovery requirements.
> # [Prometheus source
> inventory|https://github.com/JiaLiangC/ambari/blob/AMBARI-26638/docs/frontend-refactor/prometheus-source-inventory.md]:
> source revisions, selected subsystems, and explicit exclusions.
> # [Agent telemetry implementation
> plan|https://github.com/JiaLiangC/ambari/blob/AMBARI-26638/docs/ambari-agent-prometheus-telemetry-plan.md]:
> phased Agent/Server integration and verification scope.
> # [Three-node runtime
> evidence|https://github.com/JiaLiangC/ambari/blob/AMBARI-26638/docs/frontend-refactor/runtime-evidence/AMBARI-26638/README.md]:
> deployed service state, scrape targets, and dashboard screenshots.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
