[
https://issues.apache.org/jira/browse/CAMEL-25499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luigi De Masi reassigned CAMEL-25499:
-------------------------------------
Assignee: Luigi De Masi
> camel-semantic: Design audit SPI, TUI history and observability hooks
> ---------------------------------------------------------------------
>
> Key: CAMEL-25499
> URL: https://issues.apache.org/jira/browse/CAMEL-25499
> Project: Camel
> Issue Type: Improvement
> Components: camel-ai, camel-jbang, camel-telemetry
> Reporter: Luigi De Masi
> Assignee: Luigi De Masi
> Priority: Major
> Attachments: camel-semantic-audit-tui.png,
> camel-semantic-audit-tui.svg
>
>
> h2. Purpose and status
> Design discussion for pluggable semantic auditing and an Audit view in Camel
> TUI, building on CAMEL-25491 / [PR
> #27602|https://github.com/apache/camel/pull/27602]. This issue captures a
> brainstorm and its open questions; it does not claim an implemented audit
> API, accepted upstream design or scheduled release.
> The implementation baseline inspected on 9 October 2026 is PR #27602 at
> {{0dcb5c2f15078331c0df2aad860d71c32e276c75}} (open at the time of
> inspection). It provides contract-driven Definitions, Experts and
> Relationships views, runtime metadata, explicit named-definition sample
> evaluation and direct expert calls, and bounded asynchronous
> connector/console evaluation.
> The starting observation is [Claus's CAMEL-25311
> comment|https://issues.apache.org/jira/browse/CAMEL-25311?focusedCommentId=18123470&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-18123470]:
> the expert's result, the policy applied and the action enforced by the
> caller must remain distinct. The same runtime evidence could support
> monitoring and an audit trail, with different consumers and delivery
> requirements.
> h2. Relationship to existing work
> * CAMEL-25491 / PR #27602 supplies the concrete runtime, connector and TUI
> baseline. Extend its Semantic tooling with a fourth Audit view.
> * CAMEL-25357 already proposes a recent-evaluation recorder, bounded buffer
> and console-backed view. Coordinate with and reuse/refine that recorder; do
> not introduce a competing recent-evaluation mechanism. This issue develops
> the pluggable backend contract, linked route-decision records, enablement
> semantics and shared observation lifecycle beyond that proposal.
> * CAMEL-23369 tracks broader GenAI observability. The common semantic hooks
> proposed here should feed that effort through existing Camel telemetry
> infrastructure, rather than create an independent OpenTelemetry SDK/exporter
> configuration.
> * CAMEL-25382 supplies expert-owned contracts, shared validation and typed
> result semantics.
> * CAMEL-25311 supplies the Wolf Defender example and the original
> observability discussion. The proposed audit API must remain
> provider-independent and also support ordinary business
> classification/scoring.
> h2. Working requirements from the brainstorm
> * Audit capture runs in the integration and continues without a connected TUI.
> * Audit backends are pluggable. Writing and querying are separate
> capabilities so a write-only destination does not have to implement browsing.
> * Auditing and OpenTelemetry can be enabled independently. Audit enabled with
> OpenTelemetry disabled is a supported combination.
> * A master audit setting supplies the default. An explicit per-expert value
> overrides it in either direction. Experts without an override inherit the
> master value.
> * Overrides identify configured expert instances by their registry bean name,
> not by provider class or static annotation identity.
> * Disabling audit capture does not disable inference, erase retained records
> or prevent reading existing history.
> * The UI is a screen inside the existing Camel TUI. No webpage is proposed.
> * Configuration placement remains open: application properties, semantic DSL,
> or both. The examples below describe alternatives, not existing supported
> syntax.
> h2. Record model: evaluation evidence and route decisions
> Use immutable, append-only records with two principal categories:
> * *Evaluation*: which expert operation ran, its typed result and documented
> meaning, optional probability/confidence, duration and execution status.
> Camel captures this at the semantic runtime boundary.
> * *Decision*: the route/policy's explicit action (for example allow, block,
> review, quarantine or continue), policy identity and reason, referencing the
> evaluation record(s) that supported it.
> An expert returning {{true}} for an injection operation does not prove that
> the route blocked a request. A permitted tool call does not establish
> successful tool execution. A policy can use several evaluations or make a
> decision without one; keep those relationships explicit. A later decision
> references earlier evaluation records instead of mutating them.
> Provide a route-facing API usable through normal Camel Java and YAML/XML bean
> invocation to record application decisions. Do not infer business policy
> meaning from an arbitrary Choice/Switch branch. The API shape and how a route
> obtains the relevant invocation references remain design questions.
> h3. Table fields and their ownership
> ||Field||Meaning / source||
> |Timestamp|Event occurrence time, stored in UTC; display timezone is a
> presentation choice.|
> |Decision / action|Explicit route/policy action; absent for an evaluation
> alone.|
> |Category|Evaluation or decision. Lifecycle/request observations may require
> an additional discriminator.|
> |Operation|Application operation such as {{tools/call}}; keep the expert
> operation, such as {{injection}}, separately.|
> |Target|Logical tool, resource or business target supplied by the
> application. An evaluation row can display its definition name as its target.|
> |Namespace|Optional application/tenant namespace from trusted application
> context or configuration; do not infer it from arbitrary inbound headers.|
> |Reason code|Stable runtime or application-policy code; keep explanatory text
> separate. A code such as {{governance_no_match}} cannot determine allow/block
> by itself.|
> |Correlation ID|Application correlation identifier. Keep Camel exchange
> identifiers and tracing identifiers as distinct fields.|
> |Details|Structured evidence and linked records, shown in the TUI inspector.|
> Additional detail fields:
> * Schema version, unique event ID, invocation ID, optional batch/attempt
> identifiers, and links to supporting evaluation events.
> * Application/Camel context, route and processor IDs where available.
> * Named definition when present, expert bean reference and static
> provider/operation identity.
> * Origin: route execution, definition sample, or direct expert call. Do not
> claim a particular client identity unless it was propagated and is known.
> * Typed result and its meaning, optional scalar/label probabilities and
> confidence, with expert-defined semantics. Missing values stay absent.
> * Allowlisted provider/model/revision metadata and supplied policy
> ID/version/rule.
> * Duration, execution status and sanitized failure code. Failures, invalid
> results and uncertain outcomes retain their declared semantics; they do not
> become benign answers.
> Capture the permitted result semantics needed to interpret historical
> records. In particular, PR #27602's {{scoreLevelsParameter}} and submitted
> level descriptions must not be replaced by today's edited definition when
> displaying an older score. Sensitive descriptions can be omitted/redacted.
> h2. Proposed runtime and backend boundary
> Illustrative names and signatures, subject to discussion:
> {code:java}
> public interface SemanticAuditSink extends Service {
> void append(SemanticAuditRecord record) throws Exception;
> }
> public interface SemanticAuditReader {
> SemanticAuditPage query(SemanticAuditQuery query);
> Optional<SemanticAuditRecord> get(String eventId);
> }
> {code}
> Register configured backends as ordinary Camel beans. A context-managed audit
> service handles lifecycle and dispatch. Multiple sinks can receive the same
> record; configure the reader used by the TUI explicitly. The sink
> acknowledgment/durability contract, threading requirements and pagination
> contract need definition before implementation.
> Candidate initial backends:
> * A bounded in-memory recorder/reader for recent TUI history, coordinated
> with CAMEL-25357. Report capacity and evictions; this is not durable storage.
> * Structured JSON logging to integrate with existing collection
> infrastructure.
> * The SPI permits database, messaging and external audit-service integrations
> without provider-specific changes. Which persistent backend belongs in the
> first implementation remains open.
> h3. Capture points in PR #27602
> * {{SemanticLanguage}} has named-expression/batch execution and a separate
> {{evaluate(SemanticEvaluation, Object)}} direct path. Cover both, including
> common validation failures; instrumenting only {{SemanticEvaluateConsole}}
> would miss normal routes.
> * Preserve input validation before batch provider calls. Distinguish
> completed members, failed groups and members never executed if a later group
> fails. Do not split a provider's optimized batch into individual calls merely
> to audit it.
> * Give repeated expressions, nested evaluations and redelivery attempts
> distinct invocation identities. Export retries retain the same event ID.
> * {{SemanticEvaluateConsole}} supplies sample/direct origin and request
> identity. Completion must not be recorded twice merely because both the
> console and language observe it.
> * Startup validation and metadata discovery do not perform inference or
> produce fabricated evaluation records.
> * Console timeout/disconnect requests cancellation but may not stop
> native/remote inference. Record the request outcome separately from observed
> provider completion, linked by invocation ID. Define late-completion handling
> explicitly.
> h3. Data handling and delivery
> * Create safe immutable snapshots before asynchronous dispatch. Do not retain
> the live {{Exchange}}, mutable provider objects, message body, input text,
> headers, credentials or unfiltered exception messages.
> * Apply allowlisting/redaction and size bounds before records reach any sink,
> query endpoint or telemetry consumer. Provider metadata is not a blanket
> permission to persist arbitrary data.
> * Proposed initial delivery is bounded and asynchronous, with visible backend
> errors, dropped-record counters and lifecycle-aware shutdown. Observer
> failures must not rewrite expert results or route policy actions.
> * Document whether a successful append means queued, written or durably
> committed. A memory record does not prove another sink persisted it. Retries
> and deduplication must use stable event IDs.
> * Guaranteed recording before a protected side effect would require a
> separate explicit delivery mode. It is not promised by this brainstorm's
> asynchronous baseline. Crash behavior, partial delivery, backpressure and
> retention are open design details.
> h2. Audit enablement semantics
> The master setting is a default that explicit expert settings can override,
> not an unconditional disable-all control:
> ||Master||Expert override||Effective evaluation audit||
> |false|omitted|disabled|
> |false|true|enabled|
> |false|false|disabled|
> |true|omitted|enabled|
> |true|true|enabled|
> |true|false|disabled|
> For example, master enabled + {{security=true}} + {{decisions=false}} audits
> the security expert but not the decision expert. Setting the master to false
> still audits {{security}} because its explicit true wins. Removing that
> override restores inheritance.
> Use cases include retaining guardrail evidence without all routine
> classifications, reducing volume from frequently invoked experts, and
> enabling targeted investigation for one configured instance. Per-expert
> filtering concerns evaluation records. The treatment of explicitly recorded
> route decisions, particularly decisions based on multiple experts, must be
> specified independently; do not arbitrarily inherit one supporting expert's
> filter.
> h3. Open point: configuration placement
> Both properties and the semantic DSL are valid candidates. No choice was made
> in the brainstorm.
> * Properties support deployment-specific settings and centralized operator
> overrides, but separate audit choices from the route definitions they affect.
> * DSL settings make the relationship visible and versioned with declarations,
> but couple operational capture choices to route resources and their reload
> behavior.
> * Supporting both needs a documented precedence and reload model. Avoid
> multiple conflicting authorities.
> Properties illustration (*proposed audit keys; not available in PR #27602*):
> {code:properties}
> camel.semantic.audit.enabled=true
> camel.semantic.audit.experts.security.enabled=true
> camel.semantic.audit.experts.decisions.enabled=false
> # Independent existing Camel OpenTelemetry2 setting
> camel.opentelemetry2.enabled=false
> {code}
> Alternative DSL illustration (*proposed syntax; not available in PR #27602*):
> {code:yaml}
> - semantic:
> audit:
> enabled: true
> experts:
> security:
> enabled: true
> decisions:
> enabled: false
> evaluation:
> screenPrompt:
> expert: security
> operation: injection
> state: "${body}"
> {code}
> The audit settings belong to the semantic framework, not provider bean
> properties or the static {{@SemanticExpert}} contract. The default value when
> nothing is configured, whether changes are live, and validation of
> unknown/late-registered expert references remain open.
> h2. Shared foundation for observability
> Use provider-independent semantic lifecycle hooks and a shared safe metadata
> model. Audit recording and OpenTelemetry instrumentation consume those hooks
> independently:
> {noformat}
> Semantic evaluation start / completion + explicit route decision
> |-- audit consumer --> configured sinks --> reader --> TUI
> `-- telemetry consumer --> Camel telemetry / configured SDK exporters
> {noformat}
> Completed records can supply structured events and counters. Full tracing
> also needs start/completion hooks and parent context propagation, including
> across the bounded executor boundaries introduced by #27602. Hook names and
> interfaces remain illustrative; keep OpenTelemetry types/dependencies outside
> the generic semantic API.
> * Support audit on / telemetry off, audit off / telemetry on, and both on.
> * The agreed audit master and expert overrides govern audit capture.
> Telemetry uses its own configuration; trace sampling must not suppress audit
> records.
> * Reuse Camel telemetry / {{camel-opentelemetry2}} integration and coordinate
> with CAMEL-23369. Do not introduce another independently configured SDK or
> OTLP client.
> * Preserve expert results and caller actions separately in telemetry.
> Successful detection of injection is not an inference failure.
> * Keep high-cardinality correlation IDs out of metric labels. Preserve them
> in records/traces where appropriate.
> * Apply guardrail conventions only to relevant operations, not all business
> classifications/scores.
> [OpenTelemetry guardrail PR
> #427|https://github.com/open-telemetry/semantic-conventions-genai/pull/427]
> is still open at inspected head {{7b8e842f84db31d0c8784ff26390d0eba9414593}}.
> Its [current event
> definition|https://github.com/open-telemetry/semantic-conventions-genai/blob/7b8e842f84db31d0c8784ff26390d0eba9414593/model/gen-ai/events.yaml]
> uses {{gen_ai.guardrail.result}}; names have evolved since the original
> comment. Keep the mapping versioned and isolated from the audit schema
> instead of treating draft names as a stable Camel API.
> h2. TUI integration and mockup
> Add *Audit* alongside *Definitions*, *Experts* and *Relationships* in the
> Semantic screen from #27602. The attached mockup follows those screens'
> orange focus accents, blue selection, terminal frame and keyboard hints. It
> uses illustrative data and is not an implemented screen.
> !camel-semantic-audit-tui.png|width=1200!
> Attachments: [PNG preview|^camel-semantic-audit-tui.png] and [editable
> SVG|^camel-semantic-audit-tui.svg].
> * A chronological table shows the fields above, with a selected-record
> inspector separating route decision from linked evaluation evidence.
> * Filter by time, category, action, expert, route, namespace and correlation
> ID. Use bounded cursor-based queries and preserve selected event identity
> during refresh.
> * Show effective audit master/expert settings, telemetry state, selected
> backend, retention/evictions and delivery problems. Disabled capture and
> empty history must be distinguishable.
> * Use text labels as well as colors. Adapt columns to terminal width, moving
> secondary fields into details when needed.
> * Query through {{SemanticTab}} / a focused {{SemanticAuditView}},
> {{MonitorContext.executeIndependentAction}}, a new {{semantic-audit}}
> connector action and a read-only {{SemanticAuditConsole}} backed by the
> configured reader.
> * Keep audit query state separate from the definition metadata snapshot.
> Browsing history must not require that an old definition still exists or run
> an expert.
> * Extend {{LocalCliConnector}}'s request-specific asynchronous dispatch for
> potentially slow audit queries, with bounds independent of inference workers.
> Preserve its stop/disconnect behavior and avoid blocking normal
> polling/control actions.
> * Reuse {{SemanticResultView}} formatting helpers using captured semantics;
> remove playground-specific draft/retry instructions from historical records.
> * Extend the existing TUI MCP inspection surface with visible audit rows,
> filters and selected details. This is inspection, not inference replay.
> h2. Proposed validation and completion criteria
> Before implementation, settle API/data ownership, configuration placement,
> delivery/acknowledgment semantics and coordination with CAMEL-25357 /
> CAMEL-23369.
> An implementation should then demonstrate:
> # Named, direct, nested and batch evaluations produce correctly correlated
> records without changing provider invocation count, policy interpretation or
> normal Camel error handling.
> # Input/result failures, provider errors, redelivery, timeouts, cancellation
> requests and late completion are represented accurately without stale or
> duplicate outcomes.
> # All six master/override combinations above work by expert instance, and
> audit-only / telemetry-only operation works independently.
> # Explicit decisions can reference multiple evaluations; their recording
> policy is documented separately from expert filtering.
> # No submitted content, secrets, mutable exchange state or unfiltered
> provider metadata leaks through storage, querying, TUI or telemetry.
> # Bounded capacity, concurrency, sink failure, partial delivery, retry
> identities and shutdown/restart behavior match the documented contract. No
> durability claim exceeds backend guarantees.
> # The TUI remains responsive during slow inference/backend queries, handles
> paging/eviction/disconnection and shows historical result semantics correctly.
> # Relevant documentation, console descriptors/catalog mirrors and tests are
> updated without expanding the static expert contract into an audit-backend
> contract.
> _Generated by OpenAI Codex on behalf of @luigidemasi._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
