This is an automated email from the ASF dual-hosted git repository. wu-sheng pushed a commit to branch feat/ai-history-and-enhancement in repository https://gitbox.apache.org/repos/asf/skywalking-horizon-ui.git
commit c0b4009348915a1348c59d380eb980c282a8a6d7 Author: Wu Sheng <[email protected]> AuthorDate: Wed Jul 8 18:58:06 2026 +0800 refactor(ai): de-hardcode the RCA playbooks — read the template, don't bake it in The situational RCA playbooks named per-layer facts that drift from the layer template: bundled-widget names (Pod Restarts / Pods Waiting / CPU-Memory Resources / xDS pushes / Envoy health), raw metric-id prefixes (k8s_node_), per-layer entity vocabulary (K8S_SERVICE Service=Pod, K8S Service=Cluster), and native-vs-zipkin claims tied to named layers (Envoy/ALS, "mesh + k8s = zipkin"). Each per-layer noun is replaced with a runtime read — kb_layer_capabilities for vocabulary / components / trace source, kb_browse_catalog for the metric list — so a layer authored tomorrow (or a renamed widget) Just Works with no prompt edit. The diagnostic METHOD, ordering, decision forks, and the generic golden-signal vocabulary (SLA / cpm / p95 / apdex) + OAP entity model (Virtual_*, scope drill, hierarchy) stay verbatim. Layer identifiers used as tool args (K8S_SERVICE / K8S) are kept; the control-plane layer key is resolved at runtime. skills.md drops the "(Envoy ALS / rover mesh + k8s layers)" zipkin hardcode and the now-removed windowMinutes arg from the list-view tools. Validated live: a K8S_SERVICE root-cause run retrieved the playbook then read kb_layer_capabilities + kb_browse_catalog and rendered catalog widgets by id — no hardcoded widget names in the loop. Residual-hardcode grep is clean. --- apps/bff/src/ai/resources/prompts/skills.md | 10 +++++----- apps/bff/src/ai/resources/skills/rca/errors-sla.md | 2 +- apps/bff/src/ai/resources/skills/rca/k8s.md | 22 +++++++++++----------- apps/bff/src/ai/resources/skills/rca/latency.md | 2 +- apps/bff/src/ai/resources/skills/rca/mesh.md | 6 +++--- .../ai/resources/skills/rca/middleware-remote.md | 2 +- apps/bff/src/ai/resources/skills/rca/root-cause.md | 4 ++-- apps/bff/src/ai/resources/skills/rca/saturation.md | 2 +- 8 files changed, 25 insertions(+), 25 deletions(-) diff --git a/apps/bff/src/ai/resources/prompts/skills.md b/apps/bff/src/ai/resources/prompts/skills.md index 4c7debb..f524713 100644 --- a/apps/bff/src/ai/resources/prompts/skills.md +++ b/apps/bff/src/ai/resources/prompts/skills.md @@ -56,12 +56,12 @@ INLINE GRAPH TOOLS — these DRAW a compact figure directly in the chat (the BFF - show_hierarchy(layer, service) — the service's CROSS-LAYER hierarchy (the topology page's Smartscape overlay): the focus service + the same logical service projected UP (its GENERAL / MESH / K8S_SERVICE mirrors) and DOWN (its backing infra layer), grouped by layer. Use this to SHOW how one service maps across layers — the visual companion to kb_resolve_hierarchy. NOT for same-layer dependencies (that is show_topology). INLINE VIEW TOOLS — mount a REAL feature view inline (read-only, focused on the service) for the operator to browse. You do NOT get the underlying rows back — the view is for the human; return a one-line note. These are per-LAYER capabilities: a view only renders if the service's layer TEMPLATE carries that component (traces / logs / browser). First place the service in its layer (list_services), then use the tool for that layer; a layer without the component just shows a short "not avai [...] -- TRACES have TWO modes, set by the layer template's trace source. Pick the right tool: a NATIVE-tracing layer (SkyWalking segments) → show_traces. A ZIPKIN-tracing layer (Envoy ALS / rover mesh + k8s layers) → do NOT use show_traces; use list_zipkin_services then show_zipkin_traces. A 'both' layer can use either; default to native (show_traces) unless the operator asks for Zipkin. Read the layer's trace source from kb_layer_capabilities (native | zipkin | both) to pick the tool rather t [...] -- show_traces(layer, service, [windowMinutes]) — the NATIVE distributed-tracing view: the trace LIST + the span WATERFALL (the operator clicks a trace to open its spans). Use it to surface slow / erroring traces for a service so a human can inspect the span tree. There is no tool for you to read individual span data — this hands the traces to the operator. +- TRACES have TWO modes, set by the layer template's trace source. Pick the right tool: a NATIVE-tracing layer (SkyWalking segments) → show_traces. A ZIPKIN-tracing layer → do NOT use show_traces; use list_zipkin_services then show_zipkin_traces. A 'both' layer can use either; default to native (show_traces) unless the operator asks for Zipkin. Read the layer's trace source from kb_layer_capabilities (native | zipkin | both) to pick the tool rather than guessing from the layer name; if s [...] +- show_traces(layer, service) — the NATIVE distributed-tracing view: the trace LIST + the span WATERFALL (the operator clicks a trace to open its spans). Use it to surface slow / erroring traces for a service so a human can inspect the span tree. There is no tool for you to read individual span data — this hands the traces to the operator. - list_zipkin_services([keyword]) — the ZIPKIN service names (Zipkin's own service universe — GLOBAL, and DIFFERENT from the SkyWalking service names). Zipkin keys traces on span localEndpoint.serviceName, so you cannot query it with the SkyWalking name. Call this FIRST for a Zipkin layer, then MATCH the SkyWalking/user service to a Zipkin service name (they are often close but not identical — e.g. a namespace/pod-suffix difference), and pass the matched name to show_zipkin_traces. -- show_zipkin_traces(layer, service, [windowMinutes]) — the ZIPKIN trace view inline (trace LIST + span WATERFALL), where `service` is a ZIPKIN service name from list_zipkin_services (NOT the SkyWalking name). Use for a Zipkin-tracing layer after you've matched the service. If the block comes back empty, the name probably wasn't a real Zipkin service — re-check list_zipkin_services. -- show_logs(layer, service, [windowMinutes]) — the layer LOGS view: the stored log stream the operator browses (click a row for its detail). Use it to surface a service's logs for a human to read. This is the layer Logs tab — NOT fetch_pod_logs (which is the Kubernetes on-demand LIVE tail you analyse yourself). -- show_browser_logs(layer, service, [windowMinutes]) — the browser-monitoring ERROR list for a BROWSER-family app: the client-side JS error stream the operator browses (click a row for its stack trace). Only for a browser layer's app. +- show_zipkin_traces(layer, service) — the ZIPKIN trace view inline (trace LIST + span WATERFALL), where `service` is a ZIPKIN service name from list_zipkin_services (NOT the SkyWalking name). Use for a Zipkin-tracing layer after you've matched the service. If the block comes back empty, the name probably wasn't a real Zipkin service — re-check list_zipkin_services. +- show_logs(layer, service) — the layer LOGS view: the stored log stream the operator browses (click a row for its detail). Use it to surface a service's logs for a human to read. This is the layer Logs tab — NOT fetch_pod_logs (which is the Kubernetes on-demand LIVE tail you analyse yourself). +- show_browser_logs(layer, service) — the browser-monitoring ERROR list for a BROWSER-family app: the client-side JS error stream the operator browses (click a row for its stack trace). Only for a browser layer's app. - show_deployment(layer, service) — the per-service DEPLOYMENT graph inline: the instance-to-instance call graph WITHIN one service (its instances/pods + the intra-service relations), the real Deployment-tab view with pan/zoom + node/edge detail. This is the service's OWN instances talking to each other — NOT the cross-service map (that is show_topology). Deployment and show_instance_topology share ONE OAP backend (the instance-relation API), but are DIFFERENT concepts: deployment puts t [...] - show_instance_topology(layer, sourceService, destService) — the per-PAIR INSTANCE map inline: the instances of a SOURCE (client) service and a DEST (server) service as two columns with the instance-to-instance calls BETWEEN them (the topology tab's edge drill-down). Needs BOTH services and they must have a call relationship (source calls dest) — resolve two service names. Use it for "how do service A's instances talk to service B's"; contrast show_deployment (one service's own instance [...] - show_endpoint_dependency(layer, service) — the per-ENDPOINT API-dependency chain inline: the service's PRIMARY endpoint (auto-picked) and its upstream callers + downstream callees across services, the real API-dependency-tab view (expand any node). This is API-DEPENDENCY detection (endpoint→endpoint). It reads EndpointRelation, which is SERVER-SIDE ONLY (no client edge) and comes from NATIVE SkyWalking tracing — so it only has data on a native-trace layer; on a Zipkin/mesh layer (check [...] diff --git a/apps/bff/src/ai/resources/skills/rca/errors-sla.md b/apps/bff/src/ai/resources/skills/rca/errors-sla.md index a67427c..eab51a0 100644 --- a/apps/bff/src/ai/resources/skills/rca/errors-sla.md +++ b/apps/bff/src/ai/resources/skills/rca/errors-sla.md @@ -9,6 +9,6 @@ Refines the root-cause master for error-rate / SLA drops. Ordering: root service 2. Confirm the symptom at Service scope. kb_browse_catalog the service's page; kb_describe_metric the success-rate (SLA) and cpm entries and render with show_line. The signature is SLA dropping while cpm climbs. Use catalog MQE verbatim. 3. show_topology -> walk UPSTREAM. For each upstream dependency, re-run the SLA/cpm check. If an upstream is also unhealthy, it is likely the real cause — fix the upstream first. Keep climbing to the root service. If an upstream is an unmanaged Virtual_* remote, only the client-side edge metric exists — check it and stop. If the root service is backed by an infra/database layer, kb_resolve_hierarchy into it and check its error/connection metrics. 4. At the root service, drill scope — OAP does not roll up. kb_resolve_scope_drill to Endpoint scope and re-query the same SLA/cpm; show_top / show_table the top failing endpoints (outliers stand out). -5. kb_resolve_scope_drill to ServiceInstance scope -> re-query SLA plus runtime metrics (JVM / Node VM / host) with show_line; pick a few abnormal instances. For a k8s workload, read their pod logs now — list_pod_containers + fetch_pod_logs (instance attributes are not an assistant tool yet). +5. kb_resolve_scope_drill to ServiceInstance scope -> re-query SLA plus the instance runtime metrics kb_browse_catalog(<layer>, instance) exposes with show_line; pick a few abnormal instances. For a k8s workload, read their pod logs now — list_pod_containers + fetch_pod_logs (instance attributes are not an assistant tool yet). 6. Error stack. On a k8s pod, fetch_pod_logs names the failure (filter status/errors with keywordsOfContent). Native trace queries (status=ERROR) + span logs are NOT an assistant tool yet — surface the failing-endpoint show_table and name trace inspection as the next step. 7. If the cause stays hidden. AVAILABLE now: show_deployment for a manifest / resource-limit fault; propose_profiling (JVM / Go / eBPF). NOT wired yet: the events feed (recent reboot?), git / code-line — name the relevant one rather than fabricating a call. diff --git a/apps/bff/src/ai/resources/skills/rca/k8s.md b/apps/bff/src/ai/resources/skills/rca/k8s.md index 056363a..ccc5985 100644 --- a/apps/bff/src/ai/resources/skills/rca/k8s.md +++ b/apps/bff/src/ai/resources/skills/rca/k8s.md @@ -3,20 +3,20 @@ id: k8s title: Kubernetes workload root-cause whenToUse: The symptom is on a Kubernetes workload — a K8S_SERVICE service/pod, or a service you traced down to its k8s pods: pods restarting or stuck, OOMKilled / CrashLoopBackOff, container CPU/memory saturation, or node pressure. Also the endgame when another playbook follows the hierarchy down into k8s. --- -Refines the root-cause master for Kubernetes workloads. k8s pod-lifecycle signal is METRICS-FIRST — OAP does NOT natively collect k8s Event objects, so pod restarts / OOMKilled / CrashLoopBackOff surface as METRICS (a status metric carrying a reason tag), not an events feed. Order: root service -> calling chain -> error stack; upstream-first; catalog MQE verbatim, never invented. +Refines the root-cause master for Kubernetes workloads. k8s pod-lifecycle signal is METRICS-FIRST — OAP does NOT natively collect k8s Event objects, so pod-lifecycle faults surface as METRICS (a status metric carrying a reason tag), not an events feed. Order: root service -> calling chain -> error stack; upstream-first; catalog MQE verbatim, never invented. -WHICH LAYER to read: -- K8S_SERVICE (richest, start here): a k8s Service at Service scope, a POD at ServiceInstance scope, an API at Endpoint scope — HTTP/TCP golden metrics + pod resources + pod status. -- K8S: the CLUSTER at Service scope, a NODE at ServiceInstance scope — node/cluster CPU/memory/storage/pod counts. Go here for node pressure. -- CILIUM_SERVICE: eBPF L4 / HTTP / DNS network signal (drops, non-2xx) when the fault looks network-side. -- AWS_EKS: the same on EKS via CloudWatch. +WHICH LAYER to read — resolve each one's vocabulary + metrics from kb_layer_capabilities + kb_browse_catalog, never from a hardcoded name: +- The workload/pod layer (K8S_SERVICE — richest, start here): kb_layer_capabilities for what Service / Instance / Endpoint MEAN here, then kb_browse_catalog for its golden-signal + pod-resource + pod-status metrics. +- The cluster/node layer (K8S): go here for node / cluster pressure — kb_layer_capabilities for its vocabulary, kb_browse_catalog for the node + cluster metrics it returns. +- The network layer, where an eBPF / network layer is present: the on-the-wire signal when the fault looks network-side. +- The EKS variant, where the workload runs on EKS: the same, read from its own catalog. 1. list_alarms — a firing alarm on the k8s Service names the entity. None? Near-miss; continue. -2. Confirm the symptom at K8S_SERVICE Service scope. kb_browse_catalog(K8S_SERVICE, service); show_line the HTTP success / RPM / response-time SERIES, and show_table the Pod Restarts / Pods Waiting counts (they are latest()-wrapped labeled tables, not time series). Restarts climbing or pods stuck Waiting is the k8s tell. -3. Pod status REASON. show_table the catalog's Pods Waiting and Pod Restarts tables (Pods Waiting carries a waiting-reason tag — CrashLoopBackOff, ImagePullBackOff, ContainerCreating). A climbing restart count plus a Waiting reason is the first real clue and points at the next move: a restart storm or OOM -> logs (step 5) and memory-limit headroom (step 6). Only Pods Waiting + Pod Restarts are bundled widgets on this layer — do not kb_search_metrics for a Terminated table, it is not in t [...] +2. Confirm the symptom at K8S_SERVICE Service scope. kb_browse_catalog(K8S_SERVICE, service); render its golden-signal series (the success-rate / throughput / response-time entries the catalog returns) with show_line, and render its pod-status entries (the catalog's latest()-wrapped labeled tables, not time series) with show_table. Restarts climbing or pods stuck in a waiting state is the k8s tell. +3. Pod status REASON. show_table the catalog's pod-status tables — the waiting table carries a waiting-reason tag whose value comes over the wire (why the pod won't start). A climbing restart count plus a waiting reason is the first real clue and points at the next move: a restart storm or OOM -> logs (step 5) and memory-limit headroom (step 6). The catalog IS the closed world here: if a status table isn't what kb_browse_catalog returns for this layer, it doesn't exist — don't kb_search_ [...] 4. Drill to the outlier pod (no auto-rollup). kb_resolve_scope_drill(serviceId=<id>, toScope='instance') for the pod {id,name}; re-query the same metric per pod with show_top / show_table to find the bad pod. 5. Pod logs = the ERROR STACK. list_pod_containers(podId) then fetch_pod_logs(layer=K8S_SERVICE, serviceInstanceId=podId, container) — narrow with keywordsOfContent like ERROR / Exception / OOM. This is where a CrashLoop or an app exception is finally named. On-demand logs are OAP-gated; if disabled the tool says so — fall back to metrics + node pressure. -6. Limits vs usage HEADROOM. Overlay the pod's CPU/Memory Resources (requests + limits) against Pod CPU/Memory Usage with show_line: usage pinned at the limit = CPU throttle or the OOM cause; usage climbing monotonically = a leak. This is the saturation fork for a container. -7. NODE pressure. If several pods on one node misbehave, the node is suspect: kb_browse_catalog(K8S, instance) for that node — k8s_node_ CPU / memory / storage; a full-disk or memory-pressured node evicts and restarts pods. Cluster-wide: kb_browse_catalog(K8S, service) for pods-not-running / allocatable headroom. +6. Limits vs usage HEADROOM. Overlay the pod's resource entries the catalog returns — its requests/limits against its actual usage — with show_line: usage pinned at the limit = CPU throttle or the OOM cause; usage climbing monotonically = a leak. This is the saturation fork for a container. +7. NODE pressure. If several pods on one node misbehave, the node is suspect: kb_browse_catalog(K8S, instance) for that node's resource metrics (the CPU / memory / storage entries the catalog returns); a full-disk or memory-pressured node evicts and restarts pods. Cluster-wide: kb_browse_catalog(K8S, service) for the cluster capacity / headroom entries it returns. 8. Cross-layer hierarchy — BOTH directions. UP: kb_resolve_hierarchy from the K8S_SERVICE to the app service (GENERAL / MESH) that owns the request, to tie the pod fault to the user-facing symptom. DOWN: if the workload IS a backing store (a PostgreSQL / MongoDB / Kafka pod), continue root-cause on that infra layer's own metrics (slow query, connections, disk). -9. Network-side (optional). If HTTP/TCP errors look network-driven not app-driven, CILIUM_SERVICE (or the K8S_SERVICE TCP metrics) show drops / non-2xx / retransmits on the wire. +9. Network-side (optional). If HTTP/TCP errors look network-driven not app-driven, and an eBPF / network layer is present (or the K8S_SERVICE layer itself carries wire metrics), read the wire entries kb_browse_catalog returns there — an app that is up but unreachable shows on the network, not in its own metrics. diff --git a/apps/bff/src/ai/resources/skills/rca/latency.md b/apps/bff/src/ai/resources/skills/rca/latency.md index 70f4e90..2d567ff 100644 --- a/apps/bff/src/ai/resources/skills/rca/latency.md +++ b/apps/bff/src/ai/resources/skills/rca/latency.md @@ -9,5 +9,5 @@ Refines the root-cause master for latency. Ordering: root service -> calling cha 2. Frame the symptom at Service scope. kb_browse_catalog / kb_search_metrics the service's (layer, Service) page -> pull the verbatim percentile MQE (p50/p75/p90/p95/p99) and the avg resp-time MQE; show_line. Read the shape: a p95/p99 spike over a flat p50 = tail latency (GC, locks, a few slow calls); the whole curve shifting up = systemic. Overlay cpm and SLA: cpm climbing in lockstep => load-driven; latency up with flat or falling cpm => code / GC / dependency-driven. 3. show_topology -> walk UPSTREAM. Check each dependency's resp-time. If an upstream is slow, fix it first — your spike is inherited, not owned. If the slow dependency is a Virtual_* remote absent from list_services, you only hold the client-side edge metric — check that edge resp-time and stop there. If the root service is backed by an infra/database layer, kb_resolve_hierarchy into it and check that layer's latency/connection metrics. 4. At the root service, drill (no auto-rollup). kb_resolve_scope_drill -> Endpoint scope, re-query the same percentile MQE, show_top / show_table for the worst endpoints. Then drill -> ServiceInstance scope, re-query resp-time, show_top — one hot instance = node-local cause. -5. Correlate runtime at the hot instance. kb_browse_catalog the instance runtime page (JVM GC / heap / CPU, Node VM...) -> show_line; line a p99 spike up against GC pauses or CPU saturation. Cross-check instance attributes (JVM version, host, jar+version) — attributes tool coming. +5. Correlate runtime at the hot instance. kb_browse_catalog(<layer>, instance) for the instance runtime metrics it exposes -> show_line; line a p99 spike up against GC pauses or CPU saturation. Cross-check instance attributes (JVM version, host, jar+version) — attributes tool coming. 6. Deeper. AVAILABLE now: fetch_pod_logs on the worst pod's container for the error stack (k8s workloads); propose_profiling (JVM / Go / eBPF network-packet) for hot methods, lock contention, or network latency; show_deployment for a manifest / resource-limit fault. NOT an assistant tool yet: trace query + span logs, the events feed — name them as the operator's next step. diff --git a/apps/bff/src/ai/resources/skills/rca/mesh.md b/apps/bff/src/ai/resources/skills/rca/mesh.md index d49a8de..2a42eca 100644 --- a/apps/bff/src/ai/resources/skills/rca/mesh.md +++ b/apps/bff/src/ai/resources/skills/rca/mesh.md @@ -3,12 +3,12 @@ id: mesh title: Service mesh (Istio) root-cause whenToUse: The symptom is on a MESH / Istio service (sidecar-reported telemetry): SLA dropping or cpm climbing on a MESH-layer service, or you must decide whether a mesh fault sits in the app, the sidecar/network, or the Istio control plane. --- -Refines the root-cause master for MESH. Mesh telemetry is sidecar-reported (Envoy/ALS), not in-process agent — so every call has a client side and a server side, and there is a separate control plane (Istiod/Pilot) whose health gates the whole data plane. Order: root service -> calling chain -> error stack. Do NOT type metric ids yourself — kb_browse_catalog / kb_search_metrics the (layer, scope) page and render ONLY the expressions it returns. +Refines the root-cause master for MESH. Mesh telemetry is sidecar-reported, not in-process agent — so every call has a client side and a server side, and there is a separate control plane whose health gates the whole data plane. Don't assume the reporter or the trace source by name (Envoy / ALS / Istiod) — confirm them from kb_layer_capabilities: who reports, and tracesSource (native | zipkin | both) to pick the trace tool. If tracesSource is zipkin, use list_zipkin_services -> show_zipk [...] 1. list_alarms — look for a firing MESH alarm (SLA drop / cpm spike). If none fired, treat as a near-miss and continue. 2. Confirm the MESH service. list_layers / list_services to find it, then kb_browse_catalog(MESH, service) and render the success-rate, cpm, apdex and response-time entries it returns with show_card / show_line (use the catalog's expressions, don't retype ids). This is the symptom, not yet the cause. 3. show_topology -> walk toward the DEPENDENCIES this service calls (the callee / "server" side of each edge — see the master playbook's Direction note). The topology view surfaces each edge's health — client-vs-server relation/edge metrics are NOT renderable as standalone figures, so read them ON the topology. If the CLIENT side shows errors/latency the SERVER side never recorded, the fault is in the sidecar / network (mTLS, connection pool), not app code. If an upstream service is itse [...] 4. Unmanaged remote: if the bad upstream is a Virtual_* (Virtual Database / MQ / Cache), it appears as its OWN service — kb_browse_catalog that Virtual service and render its service-scope client metrics, or read its edge on show_topology, and stop there (no server side exists). -5. Control plane: if many services degrade at once, kb_browse_catalog(MESH_CP) and render the control-plane entries it returns — xDS config-rejects, xDS pushes, push latency, sidecar-injection success. A reject or push spike is a whole-mesh root cause. -6. Root service -> scope drill (no auto-rollup): kb_resolve_scope_drill(serviceId=<id>, toScope='instance'), then re-query the same catalog metric per sidecar with show_top to find the outlier; kb_browse_catalog the ServiceInstance scope and render the Envoy health entries it returns (cluster membership, pending requests, failures). Then kb_resolve_scope_drill(serviceId=<id>, toScope='endpoint') for the failing API. +5. Control plane: if many services degrade at once, find the control-plane layer (resolve it via kb_layer_capabilities / kb_resolve_hierarchy — don't hardcode its key) and render the control-plane health entries its catalog returns. When those go bad it's a whole-mesh root cause — the control plane gates every sidecar. +6. Root service -> scope drill (no auto-rollup): kb_resolve_scope_drill(serviceId=<id>, toScope='instance'), then re-query the same catalog metric per sidecar with show_top to find the outlier; kb_browse_catalog the ServiceInstance scope and render the sidecar / proxy health entries it returns. Then kb_resolve_scope_drill(serviceId=<id>, toScope='endpoint') for the failing API. 7. Error stack + deeper. AVAILABLE now: show_deployment for a sidecar-injection / config manifest fault; fetch_pod_logs on the sidecar or app container (mesh pods run in k8s); propose_profiling (eBPF network) to inspect the sidecar path. NOT an assistant tool yet: trace query + span logs, the events feed, instance attributes (Envoy/Istio version) — name these as the operator's next step. diff --git a/apps/bff/src/ai/resources/skills/rca/middleware-remote.md b/apps/bff/src/ai/resources/skills/rca/middleware-remote.md index 7dc3d8f..facb01f 100644 --- a/apps/bff/src/ai/resources/skills/rca/middleware-remote.md +++ b/apps/bff/src/ai/resources/skills/rca/middleware-remote.md @@ -12,6 +12,6 @@ Refines the root-cause master toward a middleware / remote dependency (a databas 3. THE LEAF PIVOT — three moves, because there is nothing downstream to walk: a. LOGS. If the middleware runs in Kubernetes, its own log is usually the fastest name for the fault (slow query, disk-full, connection-refused, election/rebalance, auth error). kb_resolve_hierarchy from the middleware service to its K8S_SERVICE, kb_resolve_scope_drill to the pod, then list_pod_containers + fetch_pod_logs (keywordsOfContent ERROR / slow / timeout). On-demand logs are OAP-gated; if off the tool says so. b. k8s HIERARCHY / infra. kb_resolve_hierarchy from the middleware down to its backing K8S_SERVICE / host layer and read THAT layer's pod/node metrics — memory pressure, DISK / volume full, CPU throttle, connection limits. This is how you cross from "the app is slow" to "its database node is out of disk". Infra causes live on the infra layer, not the middleware's golden metrics. - c. NETWORK. Read the client→remote edge on show_topology (client/server relation metrics are not renderable as standalone figures); where eBPF is present, the TCP / network metrics ARE service-scope and renderable — kb_browse_catalog + show_line the K8S_SERVICE TCP or CILIUM_SERVICE drops / retransmits. A broker that is up but unreachable shows as network errors on the wire, not broker errors. + c. NETWORK. Read the client→remote edge on show_topology (client/server relation metrics are not renderable as standalone figures); where an eBPF / network layer is present its wire metrics ARE service-scope and renderable — kb_browse_catalog + show_line the wire entries it returns. A broker that is up but unreachable shows as network errors on the wire, not broker errors. 4. Broker outlier (managed only). If the managed middleware IS unhealthy on its own metrics, kb_resolve_scope_drill to ServiceInstance scope (brokers/nodes) and re-query the SAME metric — OAP does NOT auto-roll-up — show_top / show_table for the outlier broker/partition; drill to Endpoint for per-topic / per-command hotspots. 5. Deeper (name as next moves): events feed for a recent broker reboot/rebalance; instance attributes for client-driver + broker version; show_deployment for a connection-limit / resource misconfig. Trace + span logs on the slow endpoint remain the caller-side error stack. diff --git a/apps/bff/src/ai/resources/skills/rca/root-cause.md b/apps/bff/src/ai/resources/skills/rca/root-cause.md index fb9674d..009f393 100644 --- a/apps/bff/src/ai/resources/skills/rca/root-cause.md +++ b/apps/bff/src/ai/resources/skills/rca/root-cause.md @@ -6,11 +6,11 @@ whenToUse: The default entry point for any "why is X wrong / what is the root ca This is the spine. Ordering: root service -> calling chain -> error stack. Upstream-first — never diagnose a downstream service before proving its upstream is healthy; the symptom is usually downstream of the real cause. Always render catalog MQE verbatim via kb_browse_catalog / kb_describe_metric; never invent a metric name. 1. list_alarms — read active alarms; the firing entity is the first suspect. If the symptom is real but nothing fired, treat it as a near-miss (a threshold that just missed) and continue on the symptom — do not assume healthy. (Formal alarm-rule inspection is not yet a tool.) -2. Confirm the symptom at Service scope. Use list_layers / list_services to orient, then kb_browse_catalog the (layer, Service) page and show_line the catalog SLA (success rate %) and cpm (calls/min) — falling SLA + climbing cpm is the classic signature. Add response-time p95/p99 if latency is the complaint. +2. Confirm the symptom at Service scope. Use list_layers / list_services to orient, then kb_browse_catalog the (layer, Service) page and render the success-rate + throughput entries it returns — falling SLA + climbing cpm is the classic signature. Add the response-time percentile entries if latency is the complaint. 3. show_topology -> walk toward the DEPENDENCIES. Direction (used throughout these playbooks): from the symptomatic service, follow its OUTGOING topology edges to the services it CALLS — the callee / "server" side of each edge. A slow or failing dependency is the usual inherited cause, so the symptom sits downstream of it. (That is what "walk upstream to the cause" means here — NOT the callers sending traffic IN; those matter only for a load surge, see the saturation playbook.) For each [...] - If an upstream is a Virtual_* remote (Virtual MQ / Database / Cache — unmanaged, no server-side metric), read only the client-side edge metric from its catalog page and stop the walk there. That edge is all you have. 4. Cross-layer hierarchy. If the root service is backed by an INFRA / database layer (a K8S_SERVICE down to its PostgreSQL / MongoDB, a service to its host/VM layer), kb_resolve_hierarchy to follow it INTO that layer and continue root-cause THERE — infra causes (memory pressure, disk/volume, connection limits) surface on the infra layer's own metrics, not the app service's. Re-query that layer via kb_browse_catalog. (kb_resolve_hierarchy crosses layers; kb_resolve_scope_drill stays withi [...] 5. At the root service, find outliers. kb_resolve_scope_drill to ServiceInstance scope and re-query the same catalog metric — OAP does NOT roll up between scopes — then show_top / show_table to spot abnormal instances. Repeat the drill to Endpoint scope for the worst API. -6. Correlate runtime + logs. show_line the outlier instance's runtime metrics (JVM / Node VM / host). For a Kubernetes workload, pod logs are AVAILABLE now — list_pod_containers + fetch_pod_logs on the outlier pod for the error stack. (Instance attributes / dependency versions are not an assistant tool yet.) +6. Correlate runtime + logs. show_line the instance runtime metrics kb_browse_catalog(<layer>, instance) exposes. For a Kubernetes workload, pod logs are AVAILABLE now — list_pod_containers + fetch_pod_logs on the outlier pod for the error stack. (Instance attributes / dependency versions are not an assistant tool yet.) 7. Error stack for the specific endpoint. Native trace query + span logs are NOT an assistant tool yet — surface the endpoint + time window and name trace inspection as the operator's next step. 8. Deeper, if hidden. AVAILABLE now: show_deployment for a k8s/helm manifest / resource-limit fault; propose_profiling (JVM / Go / eBPF network-packet) when a profile would confirm a specific hypothesis. NOT an assistant tool yet: the events feed (recent reboot/deploy), git / code-line inspection — name them as the operator's next move. diff --git a/apps/bff/src/ai/resources/skills/rca/saturation.md b/apps/bff/src/ai/resources/skills/rca/saturation.md index fa367d6..d4fa8e8 100644 --- a/apps/bff/src/ai/resources/skills/rca/saturation.md +++ b/apps/bff/src/ai/resources/skills/rca/saturation.md @@ -9,7 +9,7 @@ Refines the root-cause master for load/saturation. Ordering: root service -> cal 2. list_services / list_layers -> confirm the service and its layer. Pull Service-scope MQE verbatim from kb_browse_catalog: render SLA and cpm with show_line. Confirm SLA falling while cpm climbs — the saturation signature. 3. show_topology -> walk UPSTREAM. If an upstream caller is unhealthy, it is the real root — fix it first. If an upstream dependency is a Virtual_* remote, only the client-side edge metric exists — check it and stop. Land on the service that actually owns the load = root service. If it is backed by an infra/database layer, kb_resolve_hierarchy into it (memory/disk/connection saturation often lives there). 4. kb_resolve_scope_drill -> ServiceInstance scope. OAP does not roll up between scopes — re-query the same cpm/latency metric per instance. show_top / show_table to rank instances and surface the saturated outlier(s). -5. For each outlier, kb_browse_catalog the ServiceInstance runtime scope — JVM heap / GC time / thread count, or host / Node VM CPU & memory — then kb_describe_metric and show_line against the cpm curve. Decision fork: resources tracking cpm linearly across all instances = genuine load surge (scale out); one instance's heap climbing monotonically with rising GC while cpm is flat = leak; resources pinned at a ceiling = capacity limit. +5. For each outlier, kb_browse_catalog(<layer>, instance) for the instance runtime metrics it exposes, then kb_describe_metric and show_line against the cpm curve. Decision fork: resources tracking cpm linearly across all instances = genuine load surge (scale out); one instance's heap climbing monotonically with rising GC while cpm is flat = leak; resources pinned at a ceiling = capacity limit. 6. kb_resolve_scope_drill -> Endpoint scope; show_top endpoints by cpm to see if one API drives the surge. 7. show_deployment (available) — inspect for under-provisioned limits / missing HPA that turns a surge into saturation. 8. Error stack + deeper. AVAILABLE now: fetch_pod_logs on the outlier pod — the error stack (OOM, thread-pool or connection-pool exhaustion) usually names the cause; propose_profiling (JVM async / pprof / eBPF network) if a profile would confirm it. NOT an assistant tool yet (do not fabricate): the events feed (recent reboot/rescale), instance attributes, trace + span logs, git code-line — name them as the operator's next move.
