[
https://issues.apache.org/jira/browse/CAMEL-25310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luigi De Masi updated CAMEL-25310:
----------------------------------
Description:
h2. Problem and motivation
Extend camel-semantic so that general-purpose semantic evaluators and
specialised classifiers can be used through the same named-evaluation contract,
with explicit expert selection and discoverable runtime capabilities.
Today, SemanticQuestion requires nonblank instructions, SemanticLanguage
selects one adapter for the language, and SemanticAdapter offers validate(...)
and evaluate(...) but no machine-readable capability descriptor. The generated
semantic-adapter service descriptor identifies the implementation class without
describing its supported inputs or results.
These assumptions work for an instruction-driven provider such as Jev. They are
awkward for a fixed-purpose classifier such as Wolf-Defender, which receives
text and classifies it as BENIGN or INJECTION without accepting a question or
configurable instructions. Its injection probability can already fit
SemanticResult's boolean probability field. It does not provide arbitrary
Choice categories or an application-defined Score rubric.
The gap is in declaring, selecting, and describing evaluations, rather than in
requiring a different routing language or inventing unsupported model
behaviour. Applications should be able to use both kinds of expert in the same
CamelContext while keeping provider-specific formats and inference libraries
outside the route logic.
h2. Goals and terminology
* Use *expert* as the user-facing term and the proposed declaration key. An
expert is a configured implementation of the semantic evaluation SPI; it is not
an autonomous agent.
* Retain the semantic language, named declarations, and ref:name /
refs:name1,name2 expressions.
* Let each expert support a subset of the common result contracts: BOOLEAN,
CHOICE, and SCORE.
* Support both instruction-driven evaluations and fixed evaluations whose
meaning is supplied by the configured expert.
* Make capabilities discoverable at runtime, with one authoritative definition
of static provider capabilities.
* Keep camel-semantic independent of LangChain4j and concrete inference
runtimes. Provider dependencies, models, tokenizers, and transport resources
belong to optional provider modules.
The syntax and API names below are proposals for review. They are not claims
about existing supported options. Using expert in the DSL does not require an
incompatible rename of the existing SemanticAdapter SPI.
h2. Expert selection
Add an optional expert reference to each named evaluation. Resolve it
deterministically:
# If the declaration explicitly names an expert, use that configured instance.
# Otherwise, use an explicitly configured default expert.
# Otherwise, if exactly one eligible semantic expert is registered/discovered,
select it automatically.
# If there are no eligible experts, fail initialization with an actionable
error.
# If there are multiple eligible experts and no default, fail initialization
and list the available names.
An unknown explicit expert or invalid default is an error; it must not trigger
fallback. Do not select the first discovered implementation or infer selection
from supported result type. Two BOOLEAN experts may evaluate entirely different
properties of the input.
Expert references identify configured application instances, not merely catalog
entries. The same implementation may have multiple differently configured
instances. Catalog availability must not cause a provider to be selected or
instantiated automatically.
Example diagnostic:
{noformat}
Semantic evaluation 'injection' has no expert configured.
Available experts: security, general.
Specify 'expert' or configure a default expert.
{noformat}
h2. Capabilities and validation
Introduce a machine-readable capability description, for example
SemanticCapabilities exposed through capabilities(). Keep
validate(SemanticQuestion) for checking an individual declaration against the
selected expert.
The descriptor should express the supported contract, including:
* Accepted input kinds, such as text and structured state.
* Supported result types.
* Whether instructions are required, optional, or unsupported.
* Whether the expert accepts caller-defined criteria/rubrics, or has fixed
output semantics.
* Availability and meaning of optional probability/confidence fields, per
supported result type where necessary.
* Descriptions of fixed semantics, especially what a positive BOOLEAN result
means.
* Provider identity, owning artifact, and any statically known limits needed
for tooling.
Capabilities describe what is possible; validation checks the actual
declaration, including category/scale restrictions, instruction requirements,
and decision-policy compatibility. Static metadata must not imply that every
possible question of a supported type is meaningful for that expert.
Validate declarations before the associated route handles traffic, and validate
changed declarations when they are reloaded. Validation must not perform
inference or contact a model service. Dynamic input-shape and provider-response
validation remains necessary during evaluation.
For example, binding a CHOICE evaluation to a Wolf boolean expert must produce
an initialization error:
{noformat}
Semantic evaluation 'department' requires CHOICE.
Expert 'security' supports BOOLEAN for prompt-injection detection.
{noformat}
Do not silently convert CHOICE or SCORE into BOOLEAN, ignore unsupported
instructions, or reroute to another expert.
h2. Optional instructions and result semantics
Allow instructions to be absent in the common declaration model. Move the
requirement to the relevant expert validation: the TypeSafe/Jev expert still
requires instructions, while a fixed-purpose Wolf expert accepts a declaration
without them and rejects instruction/criteria combinations it cannot honour.
The Wolf-specific integration owns model/tokenizer selection, input
preparation, document windowing and aggregation, inference lifecycle, and the
mapping of class scores to the common result. A new task keyword is not
required for a dedicated fixed-purpose expert.
For the injection evaluation:
* The selected text is passed as data to the classifier; no synthetic question
is prepended.
* SemanticResult.probability represents the score assigned to INJECTION, not
the confidence of whichever class happened to win.
* The existing threshold and uncertainty policy determine the boolean decision.
* BENIGN means no injection detected by this evaluation; it is not a general
safety or authorization guarantee.
* A classification probability must not be presented as a SCORE severity rating
or as confidence metadata with a different meaning.
* Missing optional metadata remains absent. Invalid outputs and operational
failures remain errors rather than false decisions.
Actions such as continue, quarantine, or review remain application policy
expressed through Camel EIPs. A binary model can drive several policy branches
without pretending to support arbitrary Choice classification. Preserve
model/provider identity in diagnostic metadata where available.
h2. Illustrative declarations
The following assumes two configured expert instances: security (a fixed
injection detector) and general (an instruction-driven provider). The expert
field and omission of instructions for injection are proposed extensions.
{noformat}
- semantic:
question:
injection:
expert: security
type: boolean
state: "${body}"
threshold: "{{security.injection.threshold}}"
uncertainty: "{{security.injection.uncertainty}}"
uncertaintyPolicy: fail
department:
expert: general
type: choice
state: "${body}"
instructions: Which department should handle this message?
criteria:
billing: Invoices, payments and refunds
technical: Bugs, outages and technical problems
other: Other requests
{noformat}
Routes continue to reference ref:injection or ref:department through the
semantic language. Keep threshold configuration explicit and preserve
placeholders through declaration parsing and tooling until normal runtime
resolution.
h2. Batching and lifecycle
Expert selection must work consistently for both single references and refs:
batches. For a batch spanning experts, group evaluations by the resolved expert
instance, delegate each group through the existing batch contract, and return
one result map keyed by the original evaluation names.
Preserve existing requirements for compatible state selectors, a consistent
declaration snapshot, and complete result validation before publishing
decisions/diagnostics. Do not expose partial results if any group fails.
Grouping by implementation class alone is insufficient because two instances
may have different models or configuration. A mixed-expert batch must not be
described as one inference request.
Preserve managed lifecycle ownership, bounded resources, interruption, and
cleanup for discovered and explicitly registered providers. Keep resource/model
loading separate from capability reporting and declaration validation.
h2. Agreed catalog and metadata scope
Expert capabilities are runtime metadata in camel-semantic and optional
provider modules such as camel-typesafe-ai. Define static capabilities once
with the component-local SemanticExpert annotation and reuse that definition
from SemanticAdapter.capabilities(). Configured instances may override the
descriptor; runtime capabilities and validate(...) remain authoritative.
Generated provider capability descriptors, catalog capability entries, expert
listing/lookup APIs, a new catalog kind/model, and changes to shared SPI
annotations or metadata generators are outside this issue's agreed scope. They
are not acceptance requirements for CAMEL-25310.
Regenerate the existing YAML declaration schemas and their catalog mirror so
expert and optional instructions are represented correctly. Keep the existing
semantic documentation mirror synchronized. These generated-content updates do
not change catalog structure, models, or APIs and do not provide catalog
discovery of expert capabilities.
Catalog/tooling support for provider capabilities may be considered separately.
Catalog availability must not select or instantiate an expert, and unresolved
application configuration must not be represented as a known capability.
h2. Provider packaging and scope
Wolf-Defender is the concrete motivating use case for a narrow expert. Its
runtime integration should live in an optional provider module, provisionally
camel-wolf-defender, alongside camel-typesafe-ai. The common semantic contracts
must not depend on Wolf-specific libraries or model assets.
This issue tracks the reusable expert and runtime capability support and the
contract needed by that provider. The concrete Wolf inference backend and full
provider implementation can be delivered as a linked follow-up if needed.
Validate the common design with both the existing instruction-driven provider
and a deterministic fixed BOOLEAN expert, without requiring network inference
in ordinary tests.
h2. Compatibility and implementation areas
* Preserve existing declarations, default expert selection for applications
with one provider, ref: expressions, batch result keys, and the current global
adapter configuration or a documented compatible alias.
* Introduce capability reporting without forcing existing third-party adapters
to implement a new abstract method immediately. Unknown legacy capabilities
must not be represented as support for every operation; existing validation
remains usable.
* Keep Java, YAML and XML declarations aligned, including expert references,
optional instructions, placeholders, validation errors, resource reload, and
generated schemas/documentation. Document the existing YAML reload ordering
constraint: experts referenced by previously initialized declarations must
already be registered before pre-parsing; newly declared beans in the same
reload are prepared too late for that validation.
* Reuse the existing SemanticResult representation where possible; retain the
distinction between the normalized decision, probabilities, provider
confidence, and operational errors.
Relevant code areas:
* components/camel-ai/camel-semantic: SemanticAdapter, SemanticQuestion,
SemanticQuestionBuilder, SemanticResult, SemanticLanguage, and declaration
loaders.
* components/camel-ai/camel-typesafe-ai: TypeSafeAiSemanticAdapter and provider
discovery metadata.
* Existing generated YAML schemas and their catalog mirror, plus the existing
semantic documentation mirror. No shared SPI annotation, catalog model/API, or
expert-metadata generator changes are included.
h2. Acceptance criteria
# Existing semantic applications and the TypeSafe provider retain their
behaviour and compatibility.
# A fixed BOOLEAN expert works without instructions and supplies an injection
probability through the common result contract.
# Explicit expert, configured default, sole-provider discovery, missing
provider, unknown reference, and ambiguous selection are covered.
# Incompatible result types, unsupported criteria/instructions, and invalid
decision policies fail before inference with errors identifying the evaluation
and expert.
# BOOLEAN threshold boundaries, uncertainty handling, positive-class mapping,
and provider failures remain distinct and are tested.
# Single-expert and mixed-expert batches preserve names, instance identity,
state selection, and all-or-error result publication.
# Java, YAML and XML support the same declaration fields and validation rules,
including reload. Document the YAML bean-preparation ordering constraint and
its workaround without changing the shared YAML loader contract.
# Static capabilities are declared once with the component-local expert
annotation and reused by the default runtime capabilities implementation;
configured providers may override their effective capabilities.
# Capability reporting and declaration validation do not perform inference or
load model resources. Unknown legacy capabilities remain explicitly unknown.
Existing YAML schemas and documentation mirrors are regenerated without
changing catalog models or APIs.
# Documentation explains expert selection, fixed versus instruction-driven
evaluations, probability versus rubric score, annotation defaults versus
configured runtime capabilities, and the absence of expert capability discovery
in the catalog.
h2. Related work and references
* CAMEL-24977: provider-independent semantic evaluation.
* CAMEL-25049: batching named questions.
* CAMEL-25138: Java and XML semantic declarations.
* CAMEL-25259 and CAMEL-25257: related DSL-extension/model/discovery work;
coordinate declaration and generation changes rather than introduce parallel
mechanisms. Provider capabilities are distinct from the schema of the semantic
declaration DSL. Catalog capability publication is outside this issue's agreed
scope.
* [Semantic language
documentation|https://camel.apache.org/components/next/languages/semantic-language.html]
* [Camel Catalog
documentation|https://camel.apache.org/manual/camel-catalog.html]
* [Wolf-Defender model
card|https://huggingface.co/patronus-studio/wolf-defender-prompt-injection-small]
* [Wolf-Defender v2
article|https://patronus.studio/en/posts/wolf-defender-v2-prompt-injection-detection-on-device]
_Scope clarification prepared by Codex on behalf of luigidemasi._
was:
h2. Problem and motivation
Extend camel-semantic so that general-purpose semantic evaluators and
specialised classifiers can be used through the same named-evaluation contract,
with explicit expert selection, discoverable capabilities, and Camel Catalog
metadata.
Today, SemanticQuestion requires nonblank instructions, SemanticLanguage
selects one adapter for the language, and SemanticAdapter offers validate(...)
and evaluate(...) but no machine-readable capability descriptor. The generated
semantic-adapter service descriptor identifies the implementation class without
describing its supported inputs or results.
These assumptions work for an instruction-driven provider such as Jev. They are
awkward for a fixed-purpose classifier such as Wolf-Defender, which receives
text and classifies it as BENIGN or INJECTION without accepting a question or
configurable instructions. Its injection probability can already fit
SemanticResult's boolean probability field. It does not provide arbitrary
Choice categories or an application-defined Score rubric.
The gap is in declaring, selecting, and describing evaluations, rather than in
requiring a different routing language or inventing unsupported model
behaviour. Applications should be able to use both kinds of expert in the same
CamelContext while keeping provider-specific formats and inference libraries
outside the route logic.
h2. Goals and terminology
* Use *expert* as the user-facing term and the proposed declaration key. An
expert is a configured implementation of the semantic evaluation SPI; it is not
an autonomous agent.
* Retain the semantic language, named declarations, and ref:name /
refs:name1,name2 expressions.
* Let each expert support a subset of the common result contracts: BOOLEAN,
CHOICE, and SCORE.
* Support both instruction-driven evaluations and fixed evaluations whose
meaning is supplied by the configured expert.
* Make capabilities discoverable at runtime and through the Camel Catalog, with
one authoritative definition of static metadata.
* Keep camel-semantic independent of LangChain4j and concrete inference
runtimes. Provider dependencies, models, tokenizers, and transport resources
belong to optional provider modules.
The syntax and API names below are proposals for review. They are not claims
about existing supported options. Using expert in the DSL does not require an
incompatible rename of the existing SemanticAdapter SPI.
h2. Expert selection
Add an optional expert reference to each named evaluation. Resolve it
deterministically:
# If the declaration explicitly names an expert, use that configured instance.
# Otherwise, use an explicitly configured default expert.
# Otherwise, if exactly one eligible semantic expert is registered/discovered,
select it automatically.
# If there are no eligible experts, fail initialization with an actionable
error.
# If there are multiple eligible experts and no default, fail initialization
and list the available names.
An unknown explicit expert or invalid default is an error; it must not trigger
fallback. Do not select the first discovered implementation or infer selection
from supported result type. Two BOOLEAN experts may evaluate entirely different
properties of the input.
Expert references identify configured application instances, not merely catalog
entries. The same implementation may have multiple differently configured
instances. Catalog availability must not cause a provider to be selected or
instantiated automatically.
Example diagnostic:
{noformat}
Semantic evaluation 'injection' has no expert configured.
Available experts: security, general.
Specify 'expert' or configure a default expert.
{noformat}
h2. Capabilities and validation
Introduce a machine-readable capability description, for example
SemanticCapabilities exposed through capabilities(). Keep
validate(SemanticQuestion) for checking an individual declaration against the
selected expert.
The descriptor should express the supported contract, including:
* Accepted input kinds, such as text and structured state.
* Supported result types.
* Whether instructions are required, optional, or unsupported.
* Whether the expert accepts caller-defined criteria/rubrics, or has fixed
output semantics.
* Availability and meaning of optional probability/confidence fields, per
supported result type where necessary.
* Descriptions of fixed semantics, especially what a positive BOOLEAN result
means.
* Provider identity, owning artifact, and any statically known limits needed
for tooling.
Capabilities describe what is possible; validation checks the actual
declaration, including category/scale restrictions, instruction requirements,
and decision-policy compatibility. Static metadata must not imply that every
possible question of a supported type is meaningful for that expert.
Validate declarations before the associated route handles traffic, and validate
changed declarations when they are reloaded. Validation must not perform
inference or contact a model service. Dynamic input-shape and provider-response
validation remains necessary during evaluation.
For example, binding a CHOICE evaluation to a Wolf boolean expert must produce
an initialization error:
{noformat}
Semantic evaluation 'department' requires CHOICE.
Expert 'security' supports BOOLEAN for prompt-injection detection.
{noformat}
Do not silently convert CHOICE or SCORE into BOOLEAN, ignore unsupported
instructions, or reroute to another expert.
h2. Optional instructions and result semantics
Allow instructions to be absent in the common declaration model. Move the
requirement to the relevant expert validation: the TypeSafe/Jev expert still
requires instructions, while a fixed-purpose Wolf expert accepts a declaration
without them and rejects instruction/criteria combinations it cannot honour.
The Wolf-specific integration owns model/tokenizer selection, input
preparation, document windowing and aggregation, inference lifecycle, and the
mapping of class scores to the common result. A new task keyword is not
required for a dedicated fixed-purpose expert.
For the injection evaluation:
* The selected text is passed as data to the classifier; no synthetic question
is prepended.
* SemanticResult.probability represents the score assigned to INJECTION, not
the confidence of whichever class happened to win.
* The existing threshold and uncertainty policy determine the boolean decision.
* BENIGN means no injection detected by this evaluation; it is not a general
safety or authorization guarantee.
* A classification probability must not be presented as a SCORE severity rating
or as confidence metadata with a different meaning.
* Missing optional metadata remains absent. Invalid outputs and operational
failures remain errors rather than false decisions.
Actions such as continue, quarantine, or review remain application policy
expressed through Camel EIPs. A binary model can drive several policy branches
without pretending to support arbitrary Choice classification. Preserve
model/provider identity in diagnostic metadata where available.
h2. Illustrative declarations
The following assumes two configured expert instances: security (a fixed
injection detector) and general (an instruction-driven provider). The expert
field and omission of instructions for injection are proposed extensions.
{noformat}
- semantic:
question:
injection:
expert: security
type: boolean
state: "${body}"
threshold: "{{security.injection.threshold}}"
uncertainty: "{{security.injection.uncertainty}}"
uncertaintyPolicy: fail
department:
expert: general
type: choice
state: "${body}"
instructions: Which department should handle this message?
criteria:
billing: Invoices, payments and refunds
technical: Bugs, outages and technical problems
other: Other requests
{noformat}
Routes continue to reference ref:injection or ref:department through the
semantic language. Keep threshold configuration explicit and preserve
placeholders through declaration parsing and tooling until normal runtime
resolution.
h2. Batching and lifecycle
Expert selection must work consistently for both single references and refs:
batches. For a batch spanning experts, group evaluations by the resolved expert
instance, delegate each group through the existing batch contract, and return
one result map keyed by the original evaluation names.
Preserve existing requirements for compatible state selectors, a consistent
declaration snapshot, and complete result validation before publishing
decisions/diagnostics. Do not expose partial results if any group fails.
Grouping by implementation class alone is insufficient because two instances
may have different models or configuration. A mixed-expert batch must not be
described as one inference request.
Preserve managed lifecycle ownership, bounded resources, interruption, and
cleanup for discovered and explicitly registered providers. Keep resource/model
loading separate from catalog inspection and declaration validation.
h2. Camel Catalog integration
Publish static expert capabilities as generated catalog metadata. A dedicated
semantic-experts collection is one possible design; agree the catalog kind and
schema during implementation rather than storing provider capability data in
the generic semantic language expression options.
Define the static contract once, for example with an expert annotation or
another declarative descriptor. Generate the provider-packaged metadata and
catalog entry from that source. The runtime capabilities implementation should
reuse the same static data so runtime declarations, catalog metadata, and
documentation do not drift.
An illustrative entry could contain:
{code:json}
{
"kind": "semantic-expert",
"name": "wolf-defender",
"artifactId": "camel-wolf-defender",
"inputTypes": ["string"],
"resultTypes": ["boolean"],
"instructions": "unsupported",
"booleanProbability": true,
"trueMeaning": "Prompt injection or jailbreak detected"
}
{code}
Include normal artifact/version identity and implementation linkage in the
final schema. A provider artifact may advertise more than one expert. Catalog
retrieval should support listing experts and retrieving their capability
models, with API names such as findSemanticExpertNames() and
semanticExpertModel(name) subject to review.
The catalog describes expert implementations or documented configurations. It
does not enumerate application bean names, prove that a provider is installed,
or establish the effective capabilities of an arbitrary configured model.
Runtime capabilities and validate(...) remain authoritative. Tooling should
distinguish a definite incompatibility from configuration it cannot resolve
offline.
This metadata should enable IDEs, visual designers, CLI/TUI and MCP tooling to
discover provider artifacts, explain supported operations, and check
declarations once those consumers adopt the new catalog contract. Do not
require model downloads, provider instantiation, credentials, or native
inference libraries to read catalog entries.
h2. Provider packaging and scope
Wolf-Defender is the concrete motivating use case for a narrow expert. Its
runtime integration should live in an optional provider module, provisionally
camel-wolf-defender, alongside camel-typesafe-ai. The common semantic and
catalog contracts must not depend on Wolf-specific libraries or model assets.
This issue tracks the reusable expert/capability/catalog support and the
contract needed by that provider. The concrete Wolf inference backend and full
provider implementation can be delivered as a linked follow-up if needed.
Validate the common design with both the existing instruction-driven provider
and a deterministic fixed BOOLEAN expert, without requiring network inference
in ordinary tests.
h2. Compatibility and implementation areas
* Preserve existing declarations, default expert selection for applications
with one provider, ref: expressions, batch result keys, and the current global
adapter configuration or a documented compatible alias.
* Introduce capability reporting without forcing existing third-party adapters
to implement a new abstract method immediately. Unknown legacy capabilities
must not be represented as support for every operation; existing validation
remains usable.
* Keep Java, YAML and XML declarations aligned, including expert references,
optional instructions, placeholders, validation errors, resource reload, and
generated schemas/documentation.
* Reuse the existing SemanticResult representation where possible; retain the
distinction between the normalized decision, probabilities, provider
confidence, and operational errors.
Relevant code areas:
* components/camel-ai/camel-semantic: SemanticAdapter, SemanticQuestion,
SemanticQuestionBuilder, SemanticResult, SemanticLanguage, and declaration
loaders.
* components/camel-ai/camel-typesafe-ai: TypeSafeAiSemanticAdapter and provider
discovery metadata.
* tooling/spi-annotations, camel-tooling-model, and camel-package-maven-plugin
for generated expert metadata.
* catalog/camel-catalog and its aggregation, lookup, and documentation support.
h2. Acceptance criteria
# Existing semantic applications and the TypeSafe provider retain their
behaviour and compatibility.
# A fixed BOOLEAN expert works without instructions and supplies an injection
probability through the common result contract.
# Explicit expert, configured default, sole-provider discovery, missing
provider, unknown reference, and ambiguous selection are covered.
# Incompatible result types, unsupported criteria/instructions, and invalid
decision policies fail before inference with errors identifying the evaluation
and expert.
# BOOLEAN threshold boundaries, uncertainty handling, positive-class mapping,
and provider failures remain distinct and are tested.
# Single-expert and mixed-expert batches preserve names, instance identity,
state selection, and all-or-error result publication.
# Java, YAML and XML support the same declarations and validation behaviour,
including reload.
# Static capability metadata is generated from one definition, packaged with
the provider, discoverable through the catalog, and consistent with runtime
reporting.
# Catalog access works offline without constructing providers or loading
model/runtime dependencies; unresolved application configuration is not
misreported as a known capability.
# Documentation explains expert selection, fixed versus instruction-driven
evaluations, probability versus rubric score, and static catalog metadata
versus configured runtime instances.
h2. Related work and references
* CAMEL-24977: provider-independent semantic evaluation.
* CAMEL-25049: batching named questions.
* CAMEL-25138: Java and XML semantic declarations.
* CAMEL-25259 and CAMEL-25257: related DSL-extension/model/discovery work;
coordinate declaration and generation changes rather than introduce parallel
mechanisms. Expert capability metadata describes providers and is distinct from
the schema of the semantic declaration DSL.
* [Semantic language
documentation|https://camel.apache.org/components/next/languages/semantic-language.html]
* [Camel Catalog
documentation|https://camel.apache.org/manual/camel-catalog.html]
* [Wolf-Defender model
card|https://huggingface.co/patronus-studio/wolf-defender-prompt-injection-small]
* [Wolf-Defender v2
article|https://patronus.studio/en/posts/wolf-defender-v2-prompt-injection-detection-on-device]
Summary: camel-semantic: Add experts and runtime capability discovery
(was: camel-semantic: Add experts, capability discovery and catalog metadata)
> camel-semantic: Add experts and runtime capability discovery
> ------------------------------------------------------------
>
> Key: CAMEL-25310
> URL: https://issues.apache.org/jira/browse/CAMEL-25310
> Project: Camel
> Issue Type: Improvement
> Components: camel-ai, camel-catalog, tooling
> Reporter: Luigi De Masi
> Assignee: Luigi De Masi
> Priority: Major
>
> h2. Problem and motivation
> Extend camel-semantic so that general-purpose semantic evaluators and
> specialised classifiers can be used through the same named-evaluation
> contract, with explicit expert selection and discoverable runtime
> capabilities.
> Today, SemanticQuestion requires nonblank instructions, SemanticLanguage
> selects one adapter for the language, and SemanticAdapter offers
> validate(...) and evaluate(...) but no machine-readable capability
> descriptor. The generated semantic-adapter service descriptor identifies the
> implementation class without describing its supported inputs or results.
> These assumptions work for an instruction-driven provider such as Jev. They
> are awkward for a fixed-purpose classifier such as Wolf-Defender, which
> receives text and classifies it as BENIGN or INJECTION without accepting a
> question or configurable instructions. Its injection probability can already
> fit SemanticResult's boolean probability field. It does not provide arbitrary
> Choice categories or an application-defined Score rubric.
> The gap is in declaring, selecting, and describing evaluations, rather than
> in requiring a different routing language or inventing unsupported model
> behaviour. Applications should be able to use both kinds of expert in the
> same CamelContext while keeping provider-specific formats and inference
> libraries outside the route logic.
> h2. Goals and terminology
> * Use *expert* as the user-facing term and the proposed declaration key. An
> expert is a configured implementation of the semantic evaluation SPI; it is
> not an autonomous agent.
> * Retain the semantic language, named declarations, and ref:name /
> refs:name1,name2 expressions.
> * Let each expert support a subset of the common result contracts: BOOLEAN,
> CHOICE, and SCORE.
> * Support both instruction-driven evaluations and fixed evaluations whose
> meaning is supplied by the configured expert.
> * Make capabilities discoverable at runtime, with one authoritative
> definition of static provider capabilities.
> * Keep camel-semantic independent of LangChain4j and concrete inference
> runtimes. Provider dependencies, models, tokenizers, and transport resources
> belong to optional provider modules.
> The syntax and API names below are proposals for review. They are not claims
> about existing supported options. Using expert in the DSL does not require an
> incompatible rename of the existing SemanticAdapter SPI.
> h2. Expert selection
> Add an optional expert reference to each named evaluation. Resolve it
> deterministically:
> # If the declaration explicitly names an expert, use that configured instance.
> # Otherwise, use an explicitly configured default expert.
> # Otherwise, if exactly one eligible semantic expert is
> registered/discovered, select it automatically.
> # If there are no eligible experts, fail initialization with an actionable
> error.
> # If there are multiple eligible experts and no default, fail initialization
> and list the available names.
> An unknown explicit expert or invalid default is an error; it must not
> trigger fallback. Do not select the first discovered implementation or infer
> selection from supported result type. Two BOOLEAN experts may evaluate
> entirely different properties of the input.
> Expert references identify configured application instances, not merely
> catalog entries. The same implementation may have multiple differently
> configured instances. Catalog availability must not cause a provider to be
> selected or instantiated automatically.
> Example diagnostic:
> {noformat}
> Semantic evaluation 'injection' has no expert configured.
> Available experts: security, general.
> Specify 'expert' or configure a default expert.
> {noformat}
> h2. Capabilities and validation
> Introduce a machine-readable capability description, for example
> SemanticCapabilities exposed through capabilities(). Keep
> validate(SemanticQuestion) for checking an individual declaration against the
> selected expert.
> The descriptor should express the supported contract, including:
> * Accepted input kinds, such as text and structured state.
> * Supported result types.
> * Whether instructions are required, optional, or unsupported.
> * Whether the expert accepts caller-defined criteria/rubrics, or has fixed
> output semantics.
> * Availability and meaning of optional probability/confidence fields, per
> supported result type where necessary.
> * Descriptions of fixed semantics, especially what a positive BOOLEAN result
> means.
> * Provider identity, owning artifact, and any statically known limits needed
> for tooling.
> Capabilities describe what is possible; validation checks the actual
> declaration, including category/scale restrictions, instruction requirements,
> and decision-policy compatibility. Static metadata must not imply that every
> possible question of a supported type is meaningful for that expert.
> Validate declarations before the associated route handles traffic, and
> validate changed declarations when they are reloaded. Validation must not
> perform inference or contact a model service. Dynamic input-shape and
> provider-response validation remains necessary during evaluation.
> For example, binding a CHOICE evaluation to a Wolf boolean expert must
> produce an initialization error:
> {noformat}
> Semantic evaluation 'department' requires CHOICE.
> Expert 'security' supports BOOLEAN for prompt-injection detection.
> {noformat}
> Do not silently convert CHOICE or SCORE into BOOLEAN, ignore unsupported
> instructions, or reroute to another expert.
> h2. Optional instructions and result semantics
> Allow instructions to be absent in the common declaration model. Move the
> requirement to the relevant expert validation: the TypeSafe/Jev expert still
> requires instructions, while a fixed-purpose Wolf expert accepts a
> declaration without them and rejects instruction/criteria combinations it
> cannot honour.
> The Wolf-specific integration owns model/tokenizer selection, input
> preparation, document windowing and aggregation, inference lifecycle, and the
> mapping of class scores to the common result. A new task keyword is not
> required for a dedicated fixed-purpose expert.
> For the injection evaluation:
> * The selected text is passed as data to the classifier; no synthetic
> question is prepended.
> * SemanticResult.probability represents the score assigned to INJECTION, not
> the confidence of whichever class happened to win.
> * The existing threshold and uncertainty policy determine the boolean
> decision.
> * BENIGN means no injection detected by this evaluation; it is not a general
> safety or authorization guarantee.
> * A classification probability must not be presented as a SCORE severity
> rating or as confidence metadata with a different meaning.
> * Missing optional metadata remains absent. Invalid outputs and operational
> failures remain errors rather than false decisions.
> Actions such as continue, quarantine, or review remain application policy
> expressed through Camel EIPs. A binary model can drive several policy
> branches without pretending to support arbitrary Choice classification.
> Preserve model/provider identity in diagnostic metadata where available.
> h2. Illustrative declarations
> The following assumes two configured expert instances: security (a fixed
> injection detector) and general (an instruction-driven provider). The expert
> field and omission of instructions for injection are proposed extensions.
> {noformat}
> - semantic:
> question:
> injection:
> expert: security
> type: boolean
> state: "${body}"
> threshold: "{{security.injection.threshold}}"
> uncertainty: "{{security.injection.uncertainty}}"
> uncertaintyPolicy: fail
> department:
> expert: general
> type: choice
> state: "${body}"
> instructions: Which department should handle this message?
> criteria:
> billing: Invoices, payments and refunds
> technical: Bugs, outages and technical problems
> other: Other requests
> {noformat}
> Routes continue to reference ref:injection or ref:department through the
> semantic language. Keep threshold configuration explicit and preserve
> placeholders through declaration parsing and tooling until normal runtime
> resolution.
> h2. Batching and lifecycle
> Expert selection must work consistently for both single references and refs:
> batches. For a batch spanning experts, group evaluations by the resolved
> expert instance, delegate each group through the existing batch contract, and
> return one result map keyed by the original evaluation names.
> Preserve existing requirements for compatible state selectors, a consistent
> declaration snapshot, and complete result validation before publishing
> decisions/diagnostics. Do not expose partial results if any group fails.
> Grouping by implementation class alone is insufficient because two instances
> may have different models or configuration. A mixed-expert batch must not be
> described as one inference request.
> Preserve managed lifecycle ownership, bounded resources, interruption, and
> cleanup for discovered and explicitly registered providers. Keep
> resource/model loading separate from capability reporting and declaration
> validation.
> h2. Agreed catalog and metadata scope
> Expert capabilities are runtime metadata in camel-semantic and optional
> provider modules such as camel-typesafe-ai. Define static capabilities once
> with the component-local SemanticExpert annotation and reuse that definition
> from SemanticAdapter.capabilities(). Configured instances may override the
> descriptor; runtime capabilities and validate(...) remain authoritative.
> Generated provider capability descriptors, catalog capability entries, expert
> listing/lookup APIs, a new catalog kind/model, and changes to shared SPI
> annotations or metadata generators are outside this issue's agreed scope.
> They are not acceptance requirements for CAMEL-25310.
> Regenerate the existing YAML declaration schemas and their catalog mirror so
> expert and optional instructions are represented correctly. Keep the existing
> semantic documentation mirror synchronized. These generated-content updates
> do not change catalog structure, models, or APIs and do not provide catalog
> discovery of expert capabilities.
> Catalog/tooling support for provider capabilities may be considered
> separately. Catalog availability must not select or instantiate an expert,
> and unresolved application configuration must not be represented as a known
> capability.
> h2. Provider packaging and scope
> Wolf-Defender is the concrete motivating use case for a narrow expert. Its
> runtime integration should live in an optional provider module, provisionally
> camel-wolf-defender, alongside camel-typesafe-ai. The common semantic
> contracts must not depend on Wolf-specific libraries or model assets.
> This issue tracks the reusable expert and runtime capability support and the
> contract needed by that provider. The concrete Wolf inference backend and
> full provider implementation can be delivered as a linked follow-up if
> needed. Validate the common design with both the existing instruction-driven
> provider and a deterministic fixed BOOLEAN expert, without requiring network
> inference in ordinary tests.
> h2. Compatibility and implementation areas
> * Preserve existing declarations, default expert selection for applications
> with one provider, ref: expressions, batch result keys, and the current
> global adapter configuration or a documented compatible alias.
> * Introduce capability reporting without forcing existing third-party
> adapters to implement a new abstract method immediately. Unknown legacy
> capabilities must not be represented as support for every operation; existing
> validation remains usable.
> * Keep Java, YAML and XML declarations aligned, including expert references,
> optional instructions, placeholders, validation errors, resource reload, and
> generated schemas/documentation. Document the existing YAML reload ordering
> constraint: experts referenced by previously initialized declarations must
> already be registered before pre-parsing; newly declared beans in the same
> reload are prepared too late for that validation.
> * Reuse the existing SemanticResult representation where possible; retain the
> distinction between the normalized decision, probabilities, provider
> confidence, and operational errors.
> Relevant code areas:
> * components/camel-ai/camel-semantic: SemanticAdapter, SemanticQuestion,
> SemanticQuestionBuilder, SemanticResult, SemanticLanguage, and declaration
> loaders.
> * components/camel-ai/camel-typesafe-ai: TypeSafeAiSemanticAdapter and
> provider discovery metadata.
> * Existing generated YAML schemas and their catalog mirror, plus the existing
> semantic documentation mirror. No shared SPI annotation, catalog model/API,
> or expert-metadata generator changes are included.
> h2. Acceptance criteria
> # Existing semantic applications and the TypeSafe provider retain their
> behaviour and compatibility.
> # A fixed BOOLEAN expert works without instructions and supplies an injection
> probability through the common result contract.
> # Explicit expert, configured default, sole-provider discovery, missing
> provider, unknown reference, and ambiguous selection are covered.
> # Incompatible result types, unsupported criteria/instructions, and invalid
> decision policies fail before inference with errors identifying the
> evaluation and expert.
> # BOOLEAN threshold boundaries, uncertainty handling, positive-class mapping,
> and provider failures remain distinct and are tested.
> # Single-expert and mixed-expert batches preserve names, instance identity,
> state selection, and all-or-error result publication.
> # Java, YAML and XML support the same declaration fields and validation
> rules, including reload. Document the YAML bean-preparation ordering
> constraint and its workaround without changing the shared YAML loader
> contract.
> # Static capabilities are declared once with the component-local expert
> annotation and reused by the default runtime capabilities implementation;
> configured providers may override their effective capabilities.
> # Capability reporting and declaration validation do not perform inference or
> load model resources. Unknown legacy capabilities remain explicitly unknown.
> Existing YAML schemas and documentation mirrors are regenerated without
> changing catalog models or APIs.
> # Documentation explains expert selection, fixed versus instruction-driven
> evaluations, probability versus rubric score, annotation defaults versus
> configured runtime capabilities, and the absence of expert capability
> discovery in the catalog.
> h2. Related work and references
> * CAMEL-24977: provider-independent semantic evaluation.
> * CAMEL-25049: batching named questions.
> * CAMEL-25138: Java and XML semantic declarations.
> * CAMEL-25259 and CAMEL-25257: related DSL-extension/model/discovery work;
> coordinate declaration and generation changes rather than introduce parallel
> mechanisms. Provider capabilities are distinct from the schema of the
> semantic declaration DSL. Catalog capability publication is outside this
> issue's agreed scope.
> * [Semantic language
> documentation|https://camel.apache.org/components/next/languages/semantic-language.html]
> * [Camel Catalog
> documentation|https://camel.apache.org/manual/camel-catalog.html]
> * [Wolf-Defender model
> card|https://huggingface.co/patronus-studio/wolf-defender-prompt-injection-small]
> * [Wolf-Defender v2
> article|https://patronus.studio/en/posts/wolf-defender-v2-prompt-injection-detection-on-device]
> _Scope clarification prepared by Codex on behalf of luigidemasi._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
