[
https://issues.apache.org/jira/browse/CAMEL-25499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luigi De Masi updated CAMEL-25499:
----------------------------------
Description:
h2. Purpose and status
Design discussion for pluggable semantic auditing and an Audit view in Camel
TUI, building on CAMEL-25491 / [PR
#27602|https://github.com/apache/camel/pull/27602]. This issue captures a
brainstorm and its open questions; it does not claim an implemented audit API,
accepted upstream design or scheduled release.
The implementation baseline inspected on 9 October 2026 is PR #27602 at
{{0dcb5c2f15078331c0df2aad860d71c32e276c75}} (open at the time of inspection).
It provides contract-driven Definitions, Experts and Relationships views,
runtime metadata, explicit named-definition sample evaluation and direct expert
calls, and bounded asynchronous connector/console evaluation.
The starting observation is [Claus's CAMEL-25311
comment|https://issues.apache.org/jira/browse/CAMEL-25311?focusedCommentId=18123470&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-18123470]:
the expert's result, the policy applied and the action enforced by the caller
must remain distinct. The same runtime evidence could support monitoring and an
audit trail, with different consumers and delivery requirements.
h2. Relationship to existing work
* CAMEL-25491 / PR #27602 supplies the concrete runtime, connector and TUI
baseline. Extend its Semantic tooling with a fourth Audit view.
* CAMEL-25357 already proposes a recent-evaluation recorder, bounded buffer and
console-backed view. Coordinate with and reuse/refine that recorder; do not
introduce a competing recent-evaluation mechanism. This issue develops the
pluggable backend contract, linked route-decision records, enablement semantics
and shared observation lifecycle beyond that proposal.
* CAMEL-23369 tracks broader GenAI observability. The common semantic hooks
proposed here should feed that effort through existing Camel telemetry
infrastructure, rather than create an independent OpenTelemetry SDK/exporter
configuration.
* CAMEL-25382 supplies expert-owned contracts, shared validation and typed
result semantics.
* CAMEL-25311 supplies the Wolf Defender example and the original observability
discussion. The proposed audit API must remain provider-independent and also
support ordinary business classification/scoring.
h2. Working requirements from the brainstorm
* Audit capture runs in the integration and continues without a connected TUI.
* Audit backends are pluggable. Writing and querying are separate capabilities
so a write-only destination does not have to implement browsing.
* Auditing and OpenTelemetry can be enabled independently. Audit enabled with
OpenTelemetry disabled is a supported combination.
* A master audit setting supplies the default. An explicit per-expert value
overrides it in either direction. Experts without an override inherit the
master value.
* Overrides identify configured expert instances by their registry bean name,
not by provider class or static annotation identity.
* Disabling audit capture does not disable inference, erase retained records or
prevent reading existing history.
* The UI is a screen inside the existing Camel TUI. No webpage is proposed.
* Configuration placement remains open: application properties, semantic DSL,
or both. The examples below describe alternatives, not existing supported
syntax.
h2. Record model: evaluation evidence and route decisions
Use immutable, append-only records with two principal categories:
* *Evaluation*: which expert operation ran, its typed result and documented
meaning, optional probability/confidence, duration and execution status. Camel
captures this at the semantic runtime boundary.
* *Decision*: the route/policy's explicit action (for example allow, block,
review, quarantine or continue), policy identity and reason, referencing the
evaluation record(s) that supported it.
An expert returning {{true}} for an injection operation does not prove that the
route blocked a request. A permitted tool call does not establish successful
tool execution. A policy can use several evaluations or make a decision without
one; keep those relationships explicit. A later decision references earlier
evaluation records instead of mutating them.
Provide a route-facing API usable through normal Camel Java and YAML/XML bean
invocation to record application decisions. Do not infer business policy
meaning from an arbitrary Choice/Switch branch. The API shape and how a route
obtains the relevant invocation references remain design questions.
h3. Table fields and their ownership
||Field||Meaning / source||
|Timestamp|Event occurrence time, stored in UTC; display timezone is a
presentation choice.|
|Decision / action|Explicit route/policy action; absent for an evaluation
alone.|
|Category|Evaluation or decision. Lifecycle/request observations may require an
additional discriminator.|
|Operation|Application operation such as {{tools/call}}; keep the expert
operation, such as {{injection}}, separately.|
|Target|Logical tool, resource or business target supplied by the application.
An evaluation row can display its definition name as its target.|
|Namespace|Optional application/tenant namespace from trusted application
context or configuration; do not infer it from arbitrary inbound headers.|
|Reason code|Stable runtime or application-policy code; keep explanatory text
separate. A code such as {{governance_no_match}} cannot determine allow/block
by itself.|
|Correlation ID|Application correlation identifier. Keep Camel exchange
identifiers and tracing identifiers as distinct fields.|
|Details|Structured evidence and linked records, shown in the TUI inspector.|
Additional detail fields:
* Schema version, unique event ID, invocation ID, optional batch/attempt
identifiers, and links to supporting evaluation events.
* Application/Camel context, route and processor IDs where available.
* Named definition when present, expert bean reference and static
provider/operation identity.
* Origin: route execution, definition sample, or direct expert call. Do not
claim a particular client identity unless it was propagated and is known.
* Typed result and its meaning, optional scalar/label probabilities and
confidence, with expert-defined semantics. Missing values stay absent.
* Allowlisted provider/model/revision metadata and supplied policy
ID/version/rule.
* Duration, execution status and sanitized failure code. Failures, invalid
results and uncertain outcomes retain their declared semantics; they do not
become benign answers.
Capture the permitted result semantics needed to interpret historical records.
In particular, PR #27602's {{scoreLevelsParameter}} and submitted level
descriptions must not be replaced by today's edited definition when displaying
an older score. Sensitive descriptions can be omitted/redacted.
h2. Proposed runtime and backend boundary
Illustrative names and signatures, subject to discussion:
{code:java}
public interface SemanticAuditSink extends Service {
void append(SemanticAuditRecord record) throws Exception;
}
public interface SemanticAuditReader {
SemanticAuditPage query(SemanticAuditQuery query);
Optional<SemanticAuditRecord> get(String eventId);
}
{code}
Register configured backends as ordinary Camel beans. A context-managed audit
service handles lifecycle and dispatch. Multiple sinks can receive the same
record; configure the reader used by the TUI explicitly. The sink
acknowledgment/durability contract, threading requirements and pagination
contract need definition before implementation.
Candidate initial backends:
* A bounded in-memory recorder/reader for recent TUI history, coordinated with
CAMEL-25357. Report capacity and evictions; this is not durable storage.
* Structured JSON logging to integrate with existing collection infrastructure.
* The SPI permits database, messaging and external audit-service integrations
without provider-specific changes. Which persistent backend belongs in the
first implementation remains open.
h3. Capture points in PR #27602
* {{SemanticLanguage}} has named-expression/batch execution and a separate
{{evaluate(SemanticEvaluation, Object)}} direct path. Cover both, including
common validation failures; instrumenting only {{SemanticEvaluateConsole}}
would miss normal routes.
* Preserve input validation before batch provider calls. Distinguish completed
members, failed groups and members never executed if a later group fails. Do
not split a provider's optimized batch into individual calls merely to audit it.
* Give repeated expressions, nested evaluations and redelivery attempts
distinct invocation identities. Export retries retain the same event ID.
* {{SemanticEvaluateConsole}} supplies sample/direct origin and request
identity. Completion must not be recorded twice merely because both the console
and language observe it.
* Startup validation and metadata discovery do not perform inference or produce
fabricated evaluation records.
* Console timeout/disconnect requests cancellation but may not stop
native/remote inference. Record the request outcome separately from observed
provider completion, linked by invocation ID. Define late-completion handling
explicitly.
h3. Data handling and delivery
* Create safe immutable snapshots before asynchronous dispatch. Do not retain
the live {{Exchange}}, mutable provider objects, message body, input text,
headers, credentials or unfiltered exception messages.
* Apply allowlisting/redaction and size bounds before records reach any sink,
query endpoint or telemetry consumer. Provider metadata is not a blanket
permission to persist arbitrary data.
* Proposed initial delivery is bounded and asynchronous, with visible backend
errors, dropped-record counters and lifecycle-aware shutdown. Observer failures
must not rewrite expert results or route policy actions.
* Document whether a successful append means queued, written or durably
committed. A memory record does not prove another sink persisted it. Retries
and deduplication must use stable event IDs.
* Guaranteed recording before a protected side effect would require a separate
explicit delivery mode. It is not promised by this brainstorm's asynchronous
baseline. Crash behavior, partial delivery, backpressure and retention are open
design details.
h2. Audit enablement semantics
The master setting is a default that explicit expert settings can override, not
an unconditional disable-all control:
||Master||Expert override||Effective evaluation audit||
|false|omitted|disabled|
|false|true|enabled|
|false|false|disabled|
|true|omitted|enabled|
|true|true|enabled|
|true|false|disabled|
For example, master enabled + {{security=true}} + {{decisions=false}} audits
the security expert but not the decision expert. Setting the master to false
still audits {{security}} because its explicit true wins. Removing that
override restores inheritance.
Use cases include retaining guardrail evidence without all routine
classifications, reducing volume from frequently invoked experts, and enabling
targeted investigation for one configured instance. Per-expert filtering
concerns evaluation records. The treatment of explicitly recorded route
decisions, particularly decisions based on multiple experts, must be specified
independently; do not arbitrarily inherit one supporting expert's filter.
h3. Open point: configuration placement
Both properties and the semantic DSL are valid candidates. No choice was made
in the brainstorm.
* Properties support deployment-specific settings and centralized operator
overrides, but separate audit choices from the route definitions they affect.
* DSL settings make the relationship visible and versioned with declarations,
but couple operational capture choices to route resources and their reload
behavior.
* Supporting both needs a documented precedence and reload model. Avoid
multiple conflicting authorities.
Properties illustration (*proposed audit keys; not available in PR #27602*):
{code:java}
camel.semantic.audit.enabled=true
camel.semantic.audit.experts.security.enabled=true
camel.semantic.audit.experts.decisions.enabled=false
# Independent existing Camel OpenTelemetry2 setting
camel.opentelemetry2.enabled=false
{code}
Alternative DSL illustration (*proposed syntax; not available in PR #27602*):
{noformat}
- semantic:
audit:
enabled: true
experts:
security:
enabled: true
decisions:
enabled: false
evaluation:
screenPrompt:
expert: security
operation: injection
state: "${body}"
{noformat}
The audit settings belong to the semantic framework, not provider bean
properties or the static {{@SemanticExpert}} contract. The default value when
nothing is configured, whether changes are live, and validation of
unknown/late-registered expert references remain open.
h2. Shared foundation for observability
Use provider-independent semantic lifecycle hooks and a shared safe metadata
model. Audit recording and OpenTelemetry instrumentation consume those hooks
independently:
{noformat}
Semantic evaluation start / completion + explicit route decision
|-- audit consumer --> configured sinks --> reader --> TUI
`-- telemetry consumer --> Camel telemetry / configured SDK exporters
{noformat}
Completed records can supply structured events and counters. Full tracing also
needs start/completion hooks and parent context propagation, including across
the bounded executor boundaries introduced by #27602. Hook names and interfaces
remain illustrative; keep OpenTelemetry types/dependencies outside the generic
semantic API.
* Support audit on / telemetry off, audit off / telemetry on, and both on.
* The agreed audit master and expert overrides govern audit capture. Telemetry
uses its own configuration; trace sampling must not suppress audit records.
* Reuse Camel telemetry / {{camel-opentelemetry2}} integration and coordinate
with CAMEL-23369. Do not introduce another independently configured SDK or OTLP
client.
* Preserve expert results and caller actions separately in telemetry.
Successful detection of injection is not an inference failure.
* Keep high-cardinality correlation IDs out of metric labels. Preserve them in
records/traces where appropriate.
* Apply guardrail conventions only to relevant operations, not all business
classifications/scores.
[OpenTelemetry guardrail PR
#427|https://github.com/open-telemetry/semantic-conventions-genai/pull/427] is
still open at inspected head {{7b8e842f84db31d0c8784ff26390d0eba9414593}}. Its
[current event
definition|https://github.com/open-telemetry/semantic-conventions-genai/blob/7b8e842f84db31d0c8784ff26390d0eba9414593/model/gen-ai/events.yaml]
uses {{gen_ai.guardrail.result}}; names have evolved since the original
comment. Keep the mapping versioned and isolated from the audit schema instead
of treating draft names as a stable Camel API.
h2. TUI integration and mockup
Add *Audit* alongside *Definitions*, *Experts* and *Relationships* in the
Semantic screen from #27602. The attached mockup follows those screens' orange
focus accents, blue selection, terminal frame and keyboard hints. It uses
illustrative data and is not an implemented screen.
!camel-semantic-audit-tui.png|width=800!
Attachments: [PNG preview|^camel-semantic-audit-tui.png] and [editable
SVG|^camel-semantic-audit-tui.svg].
* A chronological table shows the fields above, with a selected-record
inspector separating route decision from linked evaluation evidence.
* Filter by time, category, action, expert, route, namespace and correlation
ID. Use bounded cursor-based queries and preserve selected event identity
during refresh.
* Show effective audit master/expert settings, telemetry state, selected
backend, retention/evictions and delivery problems. Disabled capture and empty
history must be distinguishable.
* Use text labels as well as colors. Adapt columns to terminal width, moving
secondary fields into details when needed.
* Query through {{SemanticTab}} / a focused {{SemanticAuditView}},
{{MonitorContext.executeIndependentAction}}, a new {{semantic-audit}} connector
action and a read-only {{SemanticAuditConsole}} backed by the configured reader.
* Keep audit query state separate from the definition metadata snapshot.
Browsing history must not require that an old definition still exists or run an
expert.
* Extend {{LocalCliConnector}}'s request-specific asynchronous dispatch for
potentially slow audit queries, with bounds independent of inference workers.
Preserve its stop/disconnect behavior and avoid blocking normal polling/control
actions.
* Reuse {{SemanticResultView}} formatting helpers using captured semantics;
remove playground-specific draft/retry instructions from historical records.
* Extend the existing TUI MCP inspection surface with visible audit rows,
filters and selected details. This is inspection, not inference replay.
h2. Proposed validation and completion criteria
Before implementation, settle API/data ownership, configuration placement,
delivery/acknowledgment semantics and coordination with CAMEL-25357 /
CAMEL-23369.
An implementation should then demonstrate:
# Named, direct, nested and batch evaluations produce correctly correlated
records without changing provider invocation count, policy interpretation or
normal Camel error handling.
# Input/result failures, provider errors, redelivery, timeouts, cancellation
requests and late completion are represented accurately without stale or
duplicate outcomes.
# All six master/override combinations above work by expert instance, and
audit-only / telemetry-only operation works independently.
# Explicit decisions can reference multiple evaluations; their recording policy
is documented separately from expert filtering.
# No submitted content, secrets, mutable exchange state or unfiltered provider
metadata leaks through storage, querying, TUI or telemetry.
# Bounded capacity, concurrency, sink failure, partial delivery, retry
identities and shutdown/restart behavior match the documented contract. No
durability claim exceeds backend guarantees.
# The TUI remains responsive during slow inference/backend queries, handles
paging/eviction/disconnection and shows historical result semantics correctly.
# Relevant documentation, console descriptors/catalog mirrors and tests are
updated without expanding the static expert contract into an audit-backend
contract.
_Generated by OpenAI Codex on behalf of @luigidemasi._
was:
h2. Purpose and status
Design discussion for pluggable semantic auditing and an Audit view in Camel
TUI, building on CAMEL-25491 / [PR
#27602|https://github.com/apache/camel/pull/27602]. This issue captures a
brainstorm and its open questions; it does not claim an implemented audit API,
accepted upstream design or scheduled release.
The implementation baseline inspected on 9 October 2026 is PR #27602 at
{{0dcb5c2f15078331c0df2aad860d71c32e276c75}} (open at the time of inspection).
It provides contract-driven Definitions, Experts and Relationships views,
runtime metadata, explicit named-definition sample evaluation and direct expert
calls, and bounded asynchronous connector/console evaluation.
The starting observation is [Claus's CAMEL-25311
comment|https://issues.apache.org/jira/browse/CAMEL-25311?focusedCommentId=18123470&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-18123470]:
the expert's result, the policy applied and the action enforced by the caller
must remain distinct. The same runtime evidence could support monitoring and an
audit trail, with different consumers and delivery requirements.
h2. Relationship to existing work
* CAMEL-25491 / PR #27602 supplies the concrete runtime, connector and TUI
baseline. Extend its Semantic tooling with a fourth Audit view.
* CAMEL-25357 already proposes a recent-evaluation recorder, bounded buffer and
console-backed view. Coordinate with and reuse/refine that recorder; do not
introduce a competing recent-evaluation mechanism. This issue develops the
pluggable backend contract, linked route-decision records, enablement semantics
and shared observation lifecycle beyond that proposal.
* CAMEL-23369 tracks broader GenAI observability. The common semantic hooks
proposed here should feed that effort through existing Camel telemetry
infrastructure, rather than create an independent OpenTelemetry SDK/exporter
configuration.
* CAMEL-25382 supplies expert-owned contracts, shared validation and typed
result semantics.
* CAMEL-25311 supplies the Wolf Defender example and the original observability
discussion. The proposed audit API must remain provider-independent and also
support ordinary business classification/scoring.
h2. Working requirements from the brainstorm
* Audit capture runs in the integration and continues without a connected TUI.
* Audit backends are pluggable. Writing and querying are separate capabilities
so a write-only destination does not have to implement browsing.
* Auditing and OpenTelemetry can be enabled independently. Audit enabled with
OpenTelemetry disabled is a supported combination.
* A master audit setting supplies the default. An explicit per-expert value
overrides it in either direction. Experts without an override inherit the
master value.
* Overrides identify configured expert instances by their registry bean name,
not by provider class or static annotation identity.
* Disabling audit capture does not disable inference, erase retained records or
prevent reading existing history.
* The UI is a screen inside the existing Camel TUI. No webpage is proposed.
* Configuration placement remains open: application properties, semantic DSL,
or both. The examples below describe alternatives, not existing supported
syntax.
h2. Record model: evaluation evidence and route decisions
Use immutable, append-only records with two principal categories:
* *Evaluation*: which expert operation ran, its typed result and documented
meaning, optional probability/confidence, duration and execution status. Camel
captures this at the semantic runtime boundary.
* *Decision*: the route/policy's explicit action (for example allow, block,
review, quarantine or continue), policy identity and reason, referencing the
evaluation record(s) that supported it.
An expert returning {{true}} for an injection operation does not prove that the
route blocked a request. A permitted tool call does not establish successful
tool execution. A policy can use several evaluations or make a decision without
one; keep those relationships explicit. A later decision references earlier
evaluation records instead of mutating them.
Provide a route-facing API usable through normal Camel Java and YAML/XML bean
invocation to record application decisions. Do not infer business policy
meaning from an arbitrary Choice/Switch branch. The API shape and how a route
obtains the relevant invocation references remain design questions.
h3. Table fields and their ownership
||Field||Meaning / source||
|Timestamp|Event occurrence time, stored in UTC; display timezone is a
presentation choice.|
|Decision / action|Explicit route/policy action; absent for an evaluation
alone.|
|Category|Evaluation or decision. Lifecycle/request observations may require an
additional discriminator.|
|Operation|Application operation such as {{tools/call}}; keep the expert
operation, such as {{injection}}, separately.|
|Target|Logical tool, resource or business target supplied by the application.
An evaluation row can display its definition name as its target.|
|Namespace|Optional application/tenant namespace from trusted application
context or configuration; do not infer it from arbitrary inbound headers.|
|Reason code|Stable runtime or application-policy code; keep explanatory text
separate. A code such as {{governance_no_match}} cannot determine allow/block
by itself.|
|Correlation ID|Application correlation identifier. Keep Camel exchange
identifiers and tracing identifiers as distinct fields.|
|Details|Structured evidence and linked records, shown in the TUI inspector.|
Additional detail fields:
* Schema version, unique event ID, invocation ID, optional batch/attempt
identifiers, and links to supporting evaluation events.
* Application/Camel context, route and processor IDs where available.
* Named definition when present, expert bean reference and static
provider/operation identity.
* Origin: route execution, definition sample, or direct expert call. Do not
claim a particular client identity unless it was propagated and is known.
* Typed result and its meaning, optional scalar/label probabilities and
confidence, with expert-defined semantics. Missing values stay absent.
* Allowlisted provider/model/revision metadata and supplied policy
ID/version/rule.
* Duration, execution status and sanitized failure code. Failures, invalid
results and uncertain outcomes retain their declared semantics; they do not
become benign answers.
Capture the permitted result semantics needed to interpret historical records.
In particular, PR #27602's {{scoreLevelsParameter}} and submitted level
descriptions must not be replaced by today's edited definition when displaying
an older score. Sensitive descriptions can be omitted/redacted.
h2. Proposed runtime and backend boundary
Illustrative names and signatures, subject to discussion:
{code:java}
public interface SemanticAuditSink extends Service {
void append(SemanticAuditRecord record) throws Exception;
}
public interface SemanticAuditReader {
SemanticAuditPage query(SemanticAuditQuery query);
Optional<SemanticAuditRecord> get(String eventId);
}
{code}
Register configured backends as ordinary Camel beans. A context-managed audit
service handles lifecycle and dispatch. Multiple sinks can receive the same
record; configure the reader used by the TUI explicitly. The sink
acknowledgment/durability contract, threading requirements and pagination
contract need definition before implementation.
Candidate initial backends:
* A bounded in-memory recorder/reader for recent TUI history, coordinated with
CAMEL-25357. Report capacity and evictions; this is not durable storage.
* Structured JSON logging to integrate with existing collection infrastructure.
* The SPI permits database, messaging and external audit-service integrations
without provider-specific changes. Which persistent backend belongs in the
first implementation remains open.
h3. Capture points in PR #27602
* {{SemanticLanguage}} has named-expression/batch execution and a separate
{{evaluate(SemanticEvaluation, Object)}} direct path. Cover both, including
common validation failures; instrumenting only {{SemanticEvaluateConsole}}
would miss normal routes.
* Preserve input validation before batch provider calls. Distinguish completed
members, failed groups and members never executed if a later group fails. Do
not split a provider's optimized batch into individual calls merely to audit it.
* Give repeated expressions, nested evaluations and redelivery attempts
distinct invocation identities. Export retries retain the same event ID.
* {{SemanticEvaluateConsole}} supplies sample/direct origin and request
identity. Completion must not be recorded twice merely because both the console
and language observe it.
* Startup validation and metadata discovery do not perform inference or produce
fabricated evaluation records.
* Console timeout/disconnect requests cancellation but may not stop
native/remote inference. Record the request outcome separately from observed
provider completion, linked by invocation ID. Define late-completion handling
explicitly.
h3. Data handling and delivery
* Create safe immutable snapshots before asynchronous dispatch. Do not retain
the live {{Exchange}}, mutable provider objects, message body, input text,
headers, credentials or unfiltered exception messages.
* Apply allowlisting/redaction and size bounds before records reach any sink,
query endpoint or telemetry consumer. Provider metadata is not a blanket
permission to persist arbitrary data.
* Proposed initial delivery is bounded and asynchronous, with visible backend
errors, dropped-record counters and lifecycle-aware shutdown. Observer failures
must not rewrite expert results or route policy actions.
* Document whether a successful append means queued, written or durably
committed. A memory record does not prove another sink persisted it. Retries
and deduplication must use stable event IDs.
* Guaranteed recording before a protected side effect would require a separate
explicit delivery mode. It is not promised by this brainstorm's asynchronous
baseline. Crash behavior, partial delivery, backpressure and retention are open
design details.
h2. Audit enablement semantics
The master setting is a default that explicit expert settings can override, not
an unconditional disable-all control:
||Master||Expert override||Effective evaluation audit||
|false|omitted|disabled|
|false|true|enabled|
|false|false|disabled|
|true|omitted|enabled|
|true|true|enabled|
|true|false|disabled|
For example, master enabled + {{security=true}} + {{decisions=false}} audits
the security expert but not the decision expert. Setting the master to false
still audits {{security}} because its explicit true wins. Removing that
override restores inheritance.
Use cases include retaining guardrail evidence without all routine
classifications, reducing volume from frequently invoked experts, and enabling
targeted investigation for one configured instance. Per-expert filtering
concerns evaluation records. The treatment of explicitly recorded route
decisions, particularly decisions based on multiple experts, must be specified
independently; do not arbitrarily inherit one supporting expert's filter.
h3. Open point: configuration placement
Both properties and the semantic DSL are valid candidates. No choice was made
in the brainstorm.
* Properties support deployment-specific settings and centralized operator
overrides, but separate audit choices from the route definitions they affect.
* DSL settings make the relationship visible and versioned with declarations,
but couple operational capture choices to route resources and their reload
behavior.
* Supporting both needs a documented precedence and reload model. Avoid
multiple conflicting authorities.
Properties illustration (*proposed audit keys; not available in PR #27602*):
{code:java}
camel.semantic.audit.enabled=true
camel.semantic.audit.experts.security.enabled=true
camel.semantic.audit.experts.decisions.enabled=false
# Independent existing Camel OpenTelemetry2 setting
camel.opentelemetry2.enabled=false
{code}
Alternative DSL illustration (*proposed syntax; not available in PR #27602*):
{noformat}
- semantic:
audit:
enabled: true
experts:
security:
enabled: true
decisions:
enabled: false
evaluation:
screenPrompt:
expert: security
operation: injection
state: "${body}"
{noformat}
The audit settings belong to the semantic framework, not provider bean
properties or the static {{@SemanticExpert}} contract. The default value when
nothing is configured, whether changes are live, and validation of
unknown/late-registered expert references remain open.
h2. Shared foundation for observability
Use provider-independent semantic lifecycle hooks and a shared safe metadata
model. Audit recording and OpenTelemetry instrumentation consume those hooks
independently:
{noformat}
Semantic evaluation start / completion + explicit route decision
|-- audit consumer --> configured sinks --> reader --> TUI
`-- telemetry consumer --> Camel telemetry / configured SDK exporters
{noformat}
Completed records can supply structured events and counters. Full tracing also
needs start/completion hooks and parent context propagation, including across
the bounded executor boundaries introduced by #27602. Hook names and interfaces
remain illustrative; keep OpenTelemetry types/dependencies outside the generic
semantic API.
* Support audit on / telemetry off, audit off / telemetry on, and both on.
* The agreed audit master and expert overrides govern audit capture. Telemetry
uses its own configuration; trace sampling must not suppress audit records.
* Reuse Camel telemetry / {{camel-opentelemetry2}} integration and coordinate
with CAMEL-23369. Do not introduce another independently configured SDK or OTLP
client.
* Preserve expert results and caller actions separately in telemetry.
Successful detection of injection is not an inference failure.
* Keep high-cardinality correlation IDs out of metric labels. Preserve them in
records/traces where appropriate.
* Apply guardrail conventions only to relevant operations, not all business
classifications/scores.
[OpenTelemetry guardrail PR
#427|https://github.com/open-telemetry/semantic-conventions-genai/pull/427] is
still open at inspected head {{7b8e842f84db31d0c8784ff26390d0eba9414593}}. Its
[current event
definition|https://github.com/open-telemetry/semantic-conventions-genai/blob/7b8e842f84db31d0c8784ff26390d0eba9414593/model/gen-ai/events.yaml]
uses {{gen_ai.guardrail.result}}; names have evolved since the original
comment. Keep the mapping versioned and isolated from the audit schema instead
of treating draft names as a stable Camel API.
h2. TUI integration and mockup
Add *Audit* alongside *Definitions*, *Experts* and *Relationships* in the
Semantic screen from #27602. The attached mockup follows those screens' orange
focus accents, blue selection, terminal frame and keyboard hints. It uses
illustrative data and is not an implemented screen.
!camel-semantic-audit-tui.png|width=1200!
Attachments: [PNG preview|^camel-semantic-audit-tui.png] and [editable
SVG|^camel-semantic-audit-tui.svg].
* A chronological table shows the fields above, with a selected-record
inspector separating route decision from linked evaluation evidence.
* Filter by time, category, action, expert, route, namespace and correlation
ID. Use bounded cursor-based queries and preserve selected event identity
during refresh.
* Show effective audit master/expert settings, telemetry state, selected
backend, retention/evictions and delivery problems. Disabled capture and empty
history must be distinguishable.
* Use text labels as well as colors. Adapt columns to terminal width, moving
secondary fields into details when needed.
* Query through {{SemanticTab}} / a focused {{SemanticAuditView}},
{{MonitorContext.executeIndependentAction}}, a new {{semantic-audit}} connector
action and a read-only {{SemanticAuditConsole}} backed by the configured reader.
* Keep audit query state separate from the definition metadata snapshot.
Browsing history must not require that an old definition still exists or run an
expert.
* Extend {{LocalCliConnector}}'s request-specific asynchronous dispatch for
potentially slow audit queries, with bounds independent of inference workers.
Preserve its stop/disconnect behavior and avoid blocking normal polling/control
actions.
* Reuse {{SemanticResultView}} formatting helpers using captured semantics;
remove playground-specific draft/retry instructions from historical records.
* Extend the existing TUI MCP inspection surface with visible audit rows,
filters and selected details. This is inspection, not inference replay.
h2. Proposed validation and completion criteria
Before implementation, settle API/data ownership, configuration placement,
delivery/acknowledgment semantics and coordination with CAMEL-25357 /
CAMEL-23369.
An implementation should then demonstrate:
# Named, direct, nested and batch evaluations produce correctly correlated
records without changing provider invocation count, policy interpretation or
normal Camel error handling.
# Input/result failures, provider errors, redelivery, timeouts, cancellation
requests and late completion are represented accurately without stale or
duplicate outcomes.
# All six master/override combinations above work by expert instance, and
audit-only / telemetry-only operation works independently.
# Explicit decisions can reference multiple evaluations; their recording policy
is documented separately from expert filtering.
# No submitted content, secrets, mutable exchange state or unfiltered provider
metadata leaks through storage, querying, TUI or telemetry.
# Bounded capacity, concurrency, sink failure, partial delivery, retry
identities and shutdown/restart behavior match the documented contract. No
durability claim exceeds backend guarantees.
# The TUI remains responsive during slow inference/backend queries, handles
paging/eviction/disconnection and shows historical result semantics correctly.
# Relevant documentation, console descriptors/catalog mirrors and tests are
updated without expanding the static expert contract into an audit-backend
contract.
_Generated by OpenAI Codex on behalf of @luigidemasi._
> camel-semantic: Design audit SPI, TUI history and observability hooks
> ---------------------------------------------------------------------
>
> Key: CAMEL-25499
> URL: https://issues.apache.org/jira/browse/CAMEL-25499
> Project: Camel
> Issue Type: Improvement
> Components: camel-ai, camel-jbang, camel-telemetry
> Reporter: Luigi De Masi
> Assignee: Luigi De Masi
> Priority: Major
> Attachments: camel-semantic-audit-tui.png,
> camel-semantic-audit-tui.svg
>
>
> h2. Purpose and status
> Design discussion for pluggable semantic auditing and an Audit view in Camel
> TUI, building on CAMEL-25491 / [PR
> #27602|https://github.com/apache/camel/pull/27602]. This issue captures a
> brainstorm and its open questions; it does not claim an implemented audit
> API, accepted upstream design or scheduled release.
> The implementation baseline inspected on 9 October 2026 is PR #27602 at
> {{0dcb5c2f15078331c0df2aad860d71c32e276c75}} (open at the time of
> inspection). It provides contract-driven Definitions, Experts and
> Relationships views, runtime metadata, explicit named-definition sample
> evaluation and direct expert calls, and bounded asynchronous
> connector/console evaluation.
> The starting observation is [Claus's CAMEL-25311
> comment|https://issues.apache.org/jira/browse/CAMEL-25311?focusedCommentId=18123470&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-18123470]:
> the expert's result, the policy applied and the action enforced by the
> caller must remain distinct. The same runtime evidence could support
> monitoring and an audit trail, with different consumers and delivery
> requirements.
> h2. Relationship to existing work
> * CAMEL-25491 / PR #27602 supplies the concrete runtime, connector and TUI
> baseline. Extend its Semantic tooling with a fourth Audit view.
> * CAMEL-25357 already proposes a recent-evaluation recorder, bounded buffer
> and console-backed view. Coordinate with and reuse/refine that recorder; do
> not introduce a competing recent-evaluation mechanism. This issue develops
> the pluggable backend contract, linked route-decision records, enablement
> semantics and shared observation lifecycle beyond that proposal.
> * CAMEL-23369 tracks broader GenAI observability. The common semantic hooks
> proposed here should feed that effort through existing Camel telemetry
> infrastructure, rather than create an independent OpenTelemetry SDK/exporter
> configuration.
> * CAMEL-25382 supplies expert-owned contracts, shared validation and typed
> result semantics.
> * CAMEL-25311 supplies the Wolf Defender example and the original
> observability discussion. The proposed audit API must remain
> provider-independent and also support ordinary business
> classification/scoring.
> h2. Working requirements from the brainstorm
> * Audit capture runs in the integration and continues without a connected TUI.
> * Audit backends are pluggable. Writing and querying are separate
> capabilities so a write-only destination does not have to implement browsing.
> * Auditing and OpenTelemetry can be enabled independently. Audit enabled with
> OpenTelemetry disabled is a supported combination.
> * A master audit setting supplies the default. An explicit per-expert value
> overrides it in either direction. Experts without an override inherit the
> master value.
> * Overrides identify configured expert instances by their registry bean name,
> not by provider class or static annotation identity.
> * Disabling audit capture does not disable inference, erase retained records
> or prevent reading existing history.
> * The UI is a screen inside the existing Camel TUI. No webpage is proposed.
> * Configuration placement remains open: application properties, semantic DSL,
> or both. The examples below describe alternatives, not existing supported
> syntax.
> h2. Record model: evaluation evidence and route decisions
> Use immutable, append-only records with two principal categories:
> * *Evaluation*: which expert operation ran, its typed result and documented
> meaning, optional probability/confidence, duration and execution status.
> Camel captures this at the semantic runtime boundary.
> * *Decision*: the route/policy's explicit action (for example allow, block,
> review, quarantine or continue), policy identity and reason, referencing the
> evaluation record(s) that supported it.
> An expert returning {{true}} for an injection operation does not prove that
> the route blocked a request. A permitted tool call does not establish
> successful tool execution. A policy can use several evaluations or make a
> decision without one; keep those relationships explicit. A later decision
> references earlier evaluation records instead of mutating them.
> Provide a route-facing API usable through normal Camel Java and YAML/XML bean
> invocation to record application decisions. Do not infer business policy
> meaning from an arbitrary Choice/Switch branch. The API shape and how a route
> obtains the relevant invocation references remain design questions.
> h3. Table fields and their ownership
> ||Field||Meaning / source||
> |Timestamp|Event occurrence time, stored in UTC; display timezone is a
> presentation choice.|
> |Decision / action|Explicit route/policy action; absent for an evaluation
> alone.|
> |Category|Evaluation or decision. Lifecycle/request observations may require
> an additional discriminator.|
> |Operation|Application operation such as {{tools/call}}; keep the expert
> operation, such as {{injection}}, separately.|
> |Target|Logical tool, resource or business target supplied by the
> application. An evaluation row can display its definition name as its target.|
> |Namespace|Optional application/tenant namespace from trusted application
> context or configuration; do not infer it from arbitrary inbound headers.|
> |Reason code|Stable runtime or application-policy code; keep explanatory text
> separate. A code such as {{governance_no_match}} cannot determine allow/block
> by itself.|
> |Correlation ID|Application correlation identifier. Keep Camel exchange
> identifiers and tracing identifiers as distinct fields.|
> |Details|Structured evidence and linked records, shown in the TUI inspector.|
> Additional detail fields:
> * Schema version, unique event ID, invocation ID, optional batch/attempt
> identifiers, and links to supporting evaluation events.
> * Application/Camel context, route and processor IDs where available.
> * Named definition when present, expert bean reference and static
> provider/operation identity.
> * Origin: route execution, definition sample, or direct expert call. Do not
> claim a particular client identity unless it was propagated and is known.
> * Typed result and its meaning, optional scalar/label probabilities and
> confidence, with expert-defined semantics. Missing values stay absent.
> * Allowlisted provider/model/revision metadata and supplied policy
> ID/version/rule.
> * Duration, execution status and sanitized failure code. Failures, invalid
> results and uncertain outcomes retain their declared semantics; they do not
> become benign answers.
> Capture the permitted result semantics needed to interpret historical
> records. In particular, PR #27602's {{scoreLevelsParameter}} and submitted
> level descriptions must not be replaced by today's edited definition when
> displaying an older score. Sensitive descriptions can be omitted/redacted.
> h2. Proposed runtime and backend boundary
> Illustrative names and signatures, subject to discussion:
> {code:java}
> public interface SemanticAuditSink extends Service {
> void append(SemanticAuditRecord record) throws Exception;
> }
> public interface SemanticAuditReader {
> SemanticAuditPage query(SemanticAuditQuery query);
> Optional<SemanticAuditRecord> get(String eventId);
> }
> {code}
> Register configured backends as ordinary Camel beans. A context-managed audit
> service handles lifecycle and dispatch. Multiple sinks can receive the same
> record; configure the reader used by the TUI explicitly. The sink
> acknowledgment/durability contract, threading requirements and pagination
> contract need definition before implementation.
> Candidate initial backends:
> * A bounded in-memory recorder/reader for recent TUI history, coordinated
> with CAMEL-25357. Report capacity and evictions; this is not durable storage.
> * Structured JSON logging to integrate with existing collection
> infrastructure.
> * The SPI permits database, messaging and external audit-service integrations
> without provider-specific changes. Which persistent backend belongs in the
> first implementation remains open.
> h3. Capture points in PR #27602
> * {{SemanticLanguage}} has named-expression/batch execution and a separate
> {{evaluate(SemanticEvaluation, Object)}} direct path. Cover both, including
> common validation failures; instrumenting only {{SemanticEvaluateConsole}}
> would miss normal routes.
> * Preserve input validation before batch provider calls. Distinguish
> completed members, failed groups and members never executed if a later group
> fails. Do not split a provider's optimized batch into individual calls merely
> to audit it.
> * Give repeated expressions, nested evaluations and redelivery attempts
> distinct invocation identities. Export retries retain the same event ID.
> * {{SemanticEvaluateConsole}} supplies sample/direct origin and request
> identity. Completion must not be recorded twice merely because both the
> console and language observe it.
> * Startup validation and metadata discovery do not perform inference or
> produce fabricated evaluation records.
> * Console timeout/disconnect requests cancellation but may not stop
> native/remote inference. Record the request outcome separately from observed
> provider completion, linked by invocation ID. Define late-completion handling
> explicitly.
> h3. Data handling and delivery
> * Create safe immutable snapshots before asynchronous dispatch. Do not retain
> the live {{Exchange}}, mutable provider objects, message body, input text,
> headers, credentials or unfiltered exception messages.
> * Apply allowlisting/redaction and size bounds before records reach any sink,
> query endpoint or telemetry consumer. Provider metadata is not a blanket
> permission to persist arbitrary data.
> * Proposed initial delivery is bounded and asynchronous, with visible backend
> errors, dropped-record counters and lifecycle-aware shutdown. Observer
> failures must not rewrite expert results or route policy actions.
> * Document whether a successful append means queued, written or durably
> committed. A memory record does not prove another sink persisted it. Retries
> and deduplication must use stable event IDs.
> * Guaranteed recording before a protected side effect would require a
> separate explicit delivery mode. It is not promised by this brainstorm's
> asynchronous baseline. Crash behavior, partial delivery, backpressure and
> retention are open design details.
> h2. Audit enablement semantics
> The master setting is a default that explicit expert settings can override,
> not an unconditional disable-all control:
> ||Master||Expert override||Effective evaluation audit||
> |false|omitted|disabled|
> |false|true|enabled|
> |false|false|disabled|
> |true|omitted|enabled|
> |true|true|enabled|
> |true|false|disabled|
> For example, master enabled + {{security=true}} + {{decisions=false}} audits
> the security expert but not the decision expert. Setting the master to false
> still audits {{security}} because its explicit true wins. Removing that
> override restores inheritance.
> Use cases include retaining guardrail evidence without all routine
> classifications, reducing volume from frequently invoked experts, and
> enabling targeted investigation for one configured instance. Per-expert
> filtering concerns evaluation records. The treatment of explicitly recorded
> route decisions, particularly decisions based on multiple experts, must be
> specified independently; do not arbitrarily inherit one supporting expert's
> filter.
> h3. Open point: configuration placement
> Both properties and the semantic DSL are valid candidates. No choice was made
> in the brainstorm.
> * Properties support deployment-specific settings and centralized operator
> overrides, but separate audit choices from the route definitions they affect.
> * DSL settings make the relationship visible and versioned with declarations,
> but couple operational capture choices to route resources and their reload
> behavior.
> * Supporting both needs a documented precedence and reload model. Avoid
> multiple conflicting authorities.
> Properties illustration (*proposed audit keys; not available in PR #27602*):
> {code:java}
> camel.semantic.audit.enabled=true
> camel.semantic.audit.experts.security.enabled=true
> camel.semantic.audit.experts.decisions.enabled=false
> # Independent existing Camel OpenTelemetry2 setting
> camel.opentelemetry2.enabled=false
> {code}
> Alternative DSL illustration (*proposed syntax; not available in PR #27602*):
> {noformat}
> - semantic:
> audit:
> enabled: true
> experts:
> security:
> enabled: true
> decisions:
> enabled: false
> evaluation:
> screenPrompt:
> expert: security
> operation: injection
> state: "${body}"
> {noformat}
> The audit settings belong to the semantic framework, not provider bean
> properties or the static {{@SemanticExpert}} contract. The default value when
> nothing is configured, whether changes are live, and validation of
> unknown/late-registered expert references remain open.
> h2. Shared foundation for observability
> Use provider-independent semantic lifecycle hooks and a shared safe metadata
> model. Audit recording and OpenTelemetry instrumentation consume those hooks
> independently:
> {noformat}
> Semantic evaluation start / completion + explicit route decision
> |-- audit consumer --> configured sinks --> reader --> TUI
> `-- telemetry consumer --> Camel telemetry / configured SDK exporters
> {noformat}
> Completed records can supply structured events and counters. Full tracing
> also needs start/completion hooks and parent context propagation, including
> across the bounded executor boundaries introduced by #27602. Hook names and
> interfaces remain illustrative; keep OpenTelemetry types/dependencies outside
> the generic semantic API.
> * Support audit on / telemetry off, audit off / telemetry on, and both on.
> * The agreed audit master and expert overrides govern audit capture.
> Telemetry uses its own configuration; trace sampling must not suppress audit
> records.
> * Reuse Camel telemetry / {{camel-opentelemetry2}} integration and coordinate
> with CAMEL-23369. Do not introduce another independently configured SDK or
> OTLP client.
> * Preserve expert results and caller actions separately in telemetry.
> Successful detection of injection is not an inference failure.
> * Keep high-cardinality correlation IDs out of metric labels. Preserve them
> in records/traces where appropriate.
> * Apply guardrail conventions only to relevant operations, not all business
> classifications/scores.
> [OpenTelemetry guardrail PR
> #427|https://github.com/open-telemetry/semantic-conventions-genai/pull/427]
> is still open at inspected head {{7b8e842f84db31d0c8784ff26390d0eba9414593}}.
> Its [current event
> definition|https://github.com/open-telemetry/semantic-conventions-genai/blob/7b8e842f84db31d0c8784ff26390d0eba9414593/model/gen-ai/events.yaml]
> uses {{gen_ai.guardrail.result}}; names have evolved since the original
> comment. Keep the mapping versioned and isolated from the audit schema
> instead of treating draft names as a stable Camel API.
> h2. TUI integration and mockup
> Add *Audit* alongside *Definitions*, *Experts* and *Relationships* in the
> Semantic screen from #27602. The attached mockup follows those screens'
> orange focus accents, blue selection, terminal frame and keyboard hints. It
> uses illustrative data and is not an implemented screen.
> !camel-semantic-audit-tui.png|width=800!
> Attachments: [PNG preview|^camel-semantic-audit-tui.png] and [editable
> SVG|^camel-semantic-audit-tui.svg].
> * A chronological table shows the fields above, with a selected-record
> inspector separating route decision from linked evaluation evidence.
> * Filter by time, category, action, expert, route, namespace and correlation
> ID. Use bounded cursor-based queries and preserve selected event identity
> during refresh.
> * Show effective audit master/expert settings, telemetry state, selected
> backend, retention/evictions and delivery problems. Disabled capture and
> empty history must be distinguishable.
> * Use text labels as well as colors. Adapt columns to terminal width, moving
> secondary fields into details when needed.
> * Query through {{SemanticTab}} / a focused {{SemanticAuditView}},
> {{MonitorContext.executeIndependentAction}}, a new {{semantic-audit}}
> connector action and a read-only {{SemanticAuditConsole}} backed by the
> configured reader.
> * Keep audit query state separate from the definition metadata snapshot.
> Browsing history must not require that an old definition still exists or run
> an expert.
> * Extend {{LocalCliConnector}}'s request-specific asynchronous dispatch for
> potentially slow audit queries, with bounds independent of inference workers.
> Preserve its stop/disconnect behavior and avoid blocking normal
> polling/control actions.
> * Reuse {{SemanticResultView}} formatting helpers using captured semantics;
> remove playground-specific draft/retry instructions from historical records.
> * Extend the existing TUI MCP inspection surface with visible audit rows,
> filters and selected details. This is inspection, not inference replay.
> h2. Proposed validation and completion criteria
> Before implementation, settle API/data ownership, configuration placement,
> delivery/acknowledgment semantics and coordination with CAMEL-25357 /
> CAMEL-23369.
> An implementation should then demonstrate:
> # Named, direct, nested and batch evaluations produce correctly correlated
> records without changing provider invocation count, policy interpretation or
> normal Camel error handling.
> # Input/result failures, provider errors, redelivery, timeouts, cancellation
> requests and late completion are represented accurately without stale or
> duplicate outcomes.
> # All six master/override combinations above work by expert instance, and
> audit-only / telemetry-only operation works independently.
> # Explicit decisions can reference multiple evaluations; their recording
> policy is documented separately from expert filtering.
> # No submitted content, secrets, mutable exchange state or unfiltered
> provider metadata leaks through storage, querying, TUI or telemetry.
> # Bounded capacity, concurrency, sink failure, partial delivery, retry
> identities and shutdown/restart behavior match the documented contract. No
> durability claim exceeds backend guarantees.
> # The TUI remains responsive during slow inference/backend queries, handles
> paging/eviction/disconnection and shows historical result semantics correctly.
> # Relevant documentation, console descriptors/catalog mirrors and tests are
> updated without expanding the static expert contract into an audit-backend
> contract.
> _Generated by OpenAI Codex on behalf of @luigidemasi._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
