[
https://issues.apache.org/jira/browse/CAMEL-25310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18123469#comment-18123469
]
Claus Ibsen commented on CAMEL-25310:
-------------------------------------
A follow-up idea for observability, not a request to change the current PR.
Jev, and fixed classifiers like Wolf-Defender, return typed answers and
probabilities and do not produce a rationale. So there is no "why" to
standardise from the model itself. What is emerging in OpenTelemetry describes
*which* evaluation ran and *what* it returned:
* {{gen_ai.evaluation.result}} event (OTel GenAI semconv, since 1.38, still in
Development status): {{gen_ai.evaluation.name}},
{{gen_ai.evaluation.score.value}}, {{gen_ai.evaluation.score.label}}, and an
optional {{gen_ai.evaluation.explanation}}. A named semantic evaluation maps
onto it directly: name = the declaration name (e.g. {{injection}}), value = the
probability, label = the answer. Experts without a rationale leave
{{explanation}} empty.
* Camel already has a shared GenAI observability module (used by camel-openai,
camel-langchain4j and camel-spring-ai-chat). camel-semantic could emit this
event per evaluation through it, carrying the selected expert's identity from
{{SemanticCapabilities}}.
Reference:
https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-events.md
_Claude Code on behalf of davsclaus_
> camel-semantic: Add experts and runtime capability discovery
> ------------------------------------------------------------
>
> Key: CAMEL-25310
> URL: https://issues.apache.org/jira/browse/CAMEL-25310
> Project: Camel
> Issue Type: Improvement
> Components: camel-ai, camel-catalog, tooling
> Reporter: Luigi De Masi
> Assignee: Luigi De Masi
> Priority: Major
>
> h2. Problem and motivation
> Extend camel-semantic so that general-purpose semantic evaluators and
> specialised classifiers can be used through the same named-evaluation
> contract, with explicit expert selection and discoverable runtime
> capabilities.
> Today, SemanticQuestion requires nonblank instructions, SemanticLanguage
> selects one adapter for the language, and SemanticAdapter offers
> validate(...) and evaluate(...) but no machine-readable capability
> descriptor. The generated semantic-adapter service descriptor identifies the
> implementation class without describing its supported inputs or results.
> These assumptions work for an instruction-driven provider such as Jev. They
> are awkward for a fixed-purpose classifier such as Wolf-Defender, which
> receives text and classifies it as BENIGN or INJECTION without accepting a
> question or configurable instructions. Its injection probability can already
> fit SemanticResult's boolean probability field. It does not provide arbitrary
> Choice categories or an application-defined Score rubric.
> The gap is in declaring, selecting, and describing evaluations, rather than
> in requiring a different routing language or inventing unsupported model
> behaviour. Applications should be able to use both kinds of expert in the
> same CamelContext while keeping provider-specific formats and inference
> libraries outside the route logic.
> h2. Goals and terminology
> * Use *expert* as the user-facing term and the proposed declaration key. An
> expert is a configured implementation of the semantic evaluation SPI; it is
> not an autonomous agent.
> * Retain the semantic language, named declarations, and ref:name /
> refs:name1,name2 expressions.
> * Let each expert support a subset of the common result contracts: BOOLEAN,
> CHOICE, and SCORE.
> * Support both instruction-driven evaluations and fixed evaluations whose
> meaning is supplied by the configured expert.
> * Make capabilities discoverable at runtime, with one authoritative
> definition of static provider capabilities.
> * Keep camel-semantic independent of LangChain4j and concrete inference
> runtimes. Provider dependencies, models, tokenizers, and transport resources
> belong to optional provider modules.
> The syntax and API names below are proposals for review. They are not claims
> about existing supported options. Using expert in the DSL does not require an
> incompatible rename of the existing SemanticAdapter SPI.
> h2. Expert selection
> Add an optional expert reference to each named evaluation. Resolve it
> deterministically:
> # If the declaration explicitly names an expert, use that configured instance.
> # Otherwise, use an explicitly configured default expert.
> # Otherwise, if exactly one eligible semantic expert is
> registered/discovered, select it automatically.
> # If there are no eligible experts, fail initialization with an actionable
> error.
> # If there are multiple eligible experts and no default, fail initialization
> and list the available names.
> An unknown explicit expert or invalid default is an error; it must not
> trigger fallback. Do not select the first discovered implementation or infer
> selection from supported result type. Two BOOLEAN experts may evaluate
> entirely different properties of the input.
> Expert references identify configured application instances, not merely
> catalog entries. The same implementation may have multiple differently
> configured instances. Catalog availability must not cause a provider to be
> selected or instantiated automatically.
> Example diagnostic:
> {noformat}
> Semantic evaluation 'injection' has no expert configured.
> Available experts: security, general.
> Specify 'expert' or configure a default expert.
> {noformat}
> h2. Capabilities and validation
> Introduce a machine-readable capability description, for example
> SemanticCapabilities exposed through capabilities(). Keep
> validate(SemanticQuestion) for checking an individual declaration against the
> selected expert.
> The descriptor should express the supported contract, including:
> * Accepted input kinds, such as text and structured state.
> * Supported result types.
> * Whether instructions are required, optional, or unsupported.
> * Whether the expert accepts caller-defined criteria/rubrics, or has fixed
> output semantics.
> * Availability and meaning of optional probability/confidence fields, per
> supported result type where necessary.
> * Descriptions of fixed semantics, especially what a positive BOOLEAN result
> means.
> * Provider identity, owning artifact, and any statically known limits needed
> for tooling.
> Capabilities describe what is possible; validation checks the actual
> declaration, including category/scale restrictions, instruction requirements,
> and decision-policy compatibility. Static metadata must not imply that every
> possible question of a supported type is meaningful for that expert.
> Validate declarations before the associated route handles traffic, and
> validate changed declarations when they are reloaded. Validation must not
> perform inference or contact a model service. Dynamic input-shape and
> provider-response validation remains necessary during evaluation.
> For example, binding a CHOICE evaluation to a Wolf boolean expert must
> produce an initialization error:
> {noformat}
> Semantic evaluation 'department' requires CHOICE.
> Expert 'security' supports BOOLEAN for prompt-injection detection.
> {noformat}
> Do not silently convert CHOICE or SCORE into BOOLEAN, ignore unsupported
> instructions, or reroute to another expert.
> h2. Optional instructions and result semantics
> Allow instructions to be absent in the common declaration model. Move the
> requirement to the relevant expert validation: the TypeSafe/Jev expert still
> requires instructions, while a fixed-purpose Wolf expert accepts a
> declaration without them and rejects instruction/criteria combinations it
> cannot honour.
> The Wolf-specific integration owns model/tokenizer selection, input
> preparation, document windowing and aggregation, inference lifecycle, and the
> mapping of class scores to the common result. A new task keyword is not
> required for a dedicated fixed-purpose expert.
> For the injection evaluation:
> * The selected text is passed as data to the classifier; no synthetic
> question is prepended.
> * SemanticResult.probability represents the score assigned to INJECTION, not
> the confidence of whichever class happened to win.
> * The existing threshold and uncertainty policy determine the boolean
> decision.
> * BENIGN means no injection detected by this evaluation; it is not a general
> safety or authorization guarantee.
> * A classification probability must not be presented as a SCORE severity
> rating or as confidence metadata with a different meaning.
> * Missing optional metadata remains absent. Invalid outputs and operational
> failures remain errors rather than false decisions.
> Actions such as continue, quarantine, or review remain application policy
> expressed through Camel EIPs. A binary model can drive several policy
> branches without pretending to support arbitrary Choice classification.
> Preserve model/provider identity in diagnostic metadata where available.
> h2. Illustrative declarations
> The following assumes two configured expert instances: security (a fixed
> injection detector) and general (an instruction-driven provider). The expert
> field and omission of instructions for injection are proposed extensions.
> {noformat}
> - semantic:
> question:
> injection:
> expert: security
> type: boolean
> state: "${body}"
> threshold: "{{security.injection.threshold}}"
> uncertainty: "{{security.injection.uncertainty}}"
> uncertaintyPolicy: fail
> department:
> expert: general
> type: choice
> state: "${body}"
> instructions: Which department should handle this message?
> criteria:
> billing: Invoices, payments and refunds
> technical: Bugs, outages and technical problems
> other: Other requests
> {noformat}
> Routes continue to reference ref:injection or ref:department through the
> semantic language. Keep threshold configuration explicit and preserve
> placeholders through declaration parsing and tooling until normal runtime
> resolution.
> h2. Batching and lifecycle
> Expert selection must work consistently for both single references and refs:
> batches. For a batch spanning experts, group evaluations by the resolved
> expert instance, delegate each group through the existing batch contract, and
> return one result map keyed by the original evaluation names.
> Preserve existing requirements for compatible state selectors, a consistent
> declaration snapshot, and complete result validation before publishing
> decisions/diagnostics. Do not expose partial results if any group fails.
> Grouping by implementation class alone is insufficient because two instances
> may have different models or configuration. A mixed-expert batch must not be
> described as one inference request.
> Preserve managed lifecycle ownership, bounded resources, interruption, and
> cleanup for discovered and explicitly registered providers. Keep
> resource/model loading separate from capability reporting and declaration
> validation.
> h2. Agreed catalog and metadata scope
> Expert capabilities are runtime metadata in camel-semantic and optional
> provider modules such as camel-typesafe-ai. Define static capabilities once
> with the component-local SemanticExpert annotation and reuse that definition
> from SemanticAdapter.capabilities(). Configured instances may override the
> descriptor; runtime capabilities and validate(...) remain authoritative.
> Generated provider capability descriptors, catalog capability entries, expert
> listing/lookup APIs, a new catalog kind/model, and changes to shared SPI
> annotations or metadata generators are outside this issue's agreed scope.
> They are not acceptance requirements for CAMEL-25310.
> Regenerate the existing YAML declaration schemas and their catalog mirror so
> expert and optional instructions are represented correctly. Keep the existing
> semantic documentation mirror synchronized. These generated-content updates
> do not change catalog structure, models, or APIs and do not provide catalog
> discovery of expert capabilities.
> Catalog/tooling support for provider capabilities may be considered
> separately. Catalog availability must not select or instantiate an expert,
> and unresolved application configuration must not be represented as a known
> capability.
> h2. Provider packaging and scope
> Wolf-Defender is the concrete motivating use case for a narrow expert. Its
> runtime integration should live in an optional provider module, provisionally
> camel-wolf-defender, alongside camel-typesafe-ai. The common semantic
> contracts must not depend on Wolf-specific libraries or model assets.
> This issue tracks the reusable expert and runtime capability support and the
> contract needed by that provider. The concrete Wolf inference backend and
> full provider implementation can be delivered as a linked follow-up if
> needed. Validate the common design with both the existing instruction-driven
> provider and a deterministic fixed BOOLEAN expert, without requiring network
> inference in ordinary tests.
> h2. Compatibility and implementation areas
> * Preserve existing declarations, default expert selection for applications
> with one provider, ref: expressions, batch result keys, and the current
> global adapter configuration or a documented compatible alias.
> * Introduce capability reporting without forcing existing third-party
> adapters to implement a new abstract method immediately. Unknown legacy
> capabilities must not be represented as support for every operation; existing
> validation remains usable.
> * Keep Java, YAML and XML declarations aligned, including expert references,
> optional instructions, placeholders, validation errors, resource reload, and
> generated schemas/documentation. Document the existing YAML reload ordering
> constraint: experts referenced by previously initialized declarations must
> already be registered before pre-parsing; newly declared beans in the same
> reload are prepared too late for that validation.
> * Reuse the existing SemanticResult representation where possible; retain the
> distinction between the normalized decision, probabilities, provider
> confidence, and operational errors.
> Relevant code areas:
> * components/camel-ai/camel-semantic: SemanticAdapter, SemanticQuestion,
> SemanticQuestionBuilder, SemanticResult, SemanticLanguage, and declaration
> loaders.
> * components/camel-ai/camel-typesafe-ai: TypeSafeAiSemanticAdapter and
> provider discovery metadata.
> * Existing generated YAML schemas and their catalog mirror, plus the existing
> semantic documentation mirror. No shared SPI annotation, catalog model/API,
> or expert-metadata generator changes are included.
> h2. Acceptance criteria
> # Existing semantic applications and the TypeSafe provider retain their
> behaviour and compatibility.
> # A fixed BOOLEAN expert works without instructions and supplies an injection
> probability through the common result contract.
> # Explicit expert, configured default, sole-provider discovery, missing
> provider, unknown reference, and ambiguous selection are covered.
> # Incompatible result types, unsupported criteria/instructions, and invalid
> decision policies fail before inference with errors identifying the
> evaluation and expert.
> # BOOLEAN threshold boundaries, uncertainty handling, positive-class mapping,
> and provider failures remain distinct and are tested.
> # Single-expert and mixed-expert batches preserve names, instance identity,
> state selection, and all-or-error result publication.
> # Java, YAML and XML support the same declaration fields and validation
> rules, including reload. Document the YAML bean-preparation ordering
> constraint and its workaround without changing the shared YAML loader
> contract.
> # Static capabilities are declared once with the component-local expert
> annotation and reused by the default runtime capabilities implementation;
> configured providers may override their effective capabilities.
> # Capability reporting and declaration validation do not perform inference or
> load model resources. Unknown legacy capabilities remain explicitly unknown.
> Existing YAML schemas and documentation mirrors are regenerated without
> changing catalog models or APIs.
> # Documentation explains expert selection, fixed versus instruction-driven
> evaluations, probability versus rubric score, annotation defaults versus
> configured runtime capabilities, and the absence of expert capability
> discovery in the catalog.
> h2. Related work and references
> * CAMEL-24977: provider-independent semantic evaluation.
> * CAMEL-25049: batching named questions.
> * CAMEL-25138: Java and XML semantic declarations.
> * CAMEL-25259 and CAMEL-25257: related DSL-extension/model/discovery work;
> coordinate declaration and generation changes rather than introduce parallel
> mechanisms. Provider capabilities are distinct from the schema of the
> semantic declaration DSL. Catalog capability publication is outside this
> issue's agreed scope.
> * [Semantic language
> documentation|https://camel.apache.org/components/next/languages/semantic-language.html]
> * [Camel Catalog
> documentation|https://camel.apache.org/manual/camel-catalog.html]
> * [Wolf-Defender model
> card|https://huggingface.co/patronus-studio/wolf-defender-prompt-injection-small]
> * [Wolf-Defender v2
> article|https://patronus.studio/en/posts/wolf-defender-v2-prompt-injection-detection-on-device]
> _Scope clarification prepared by Codex on behalf of luigidemasi._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
