This is an automated email from the ASF dual-hosted git repository.
kaxil pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new fc12cb2ca8f Document progressive tool disclosure for Common AI
toolsets (#74736)
fc12cb2ca8f is described below
commit fc12cb2ca8fcdafcdfd83a10bd3a384ca97f0cb3
Author: Kaxil Naik <[email protected]>
AuthorDate: Thu Oct 15 16:36:32 2026 +0100
Document progressive tool disclosure for Common AI toolsets (#74736)
The partial-deferral example no longer refers to a list defined in the
first snippet, and the page no longer names which models Pydantic AI
gives native tool search, which goes stale with each Pydantic AI release.
The counts drift with every hook docstring or Pydantic AI change and say
no more than the sentence that the saving grows with the number of tools
hidden. The measurements stay in the PR description as evidence.
---
providers/common/ai/docs/features.rst | 4 +
providers/common/ai/docs/operators/agent.rst | 7 +-
providers/common/ai/docs/stability.rst | 5 +-
providers/common/ai/docs/toolsets/index.rst | 6 +-
providers/common/ai/docs/toolsets/logging.rst | 4 +-
.../ai/docs/toolsets/progressive_disclosure.rst | 146 +++++++++++++++++++++
providers/common/ai/docs/toolsets/skills.rst | 10 +-
providers/common/ai/pyproject.toml | 5 +-
.../example_progressive_tool_disclosure.py | 107 +++++++++++++++
9 files changed, 279 insertions(+), 15 deletions(-)
diff --git a/providers/common/ai/docs/features.rst
b/providers/common/ai/docs/features.rst
index 50b3dd6e5eb..58acaa0efeb 100644
--- a/providers/common/ai/docs/features.rst
+++ b/providers/common/ai/docs/features.rst
@@ -135,6 +135,10 @@ checklist.
* - :doc:`ManagedAgentToolset <toolsets/managed_agent>`
- Sends a prompt to an agent on Bedrock AgentCore or Vertex AI Agent
Engine.
- The Amazon or Google provider's ``common.ai`` extra
+ * - :doc:`Progressive tool disclosure <toolsets/progressive_disclosure>`
+ - Hides a toolset's tools from the model until it searches for them, with
+ ``.defer_loading()`` on any toolset.
+ - None
Control output and spend
------------------------
diff --git a/providers/common/ai/docs/operators/agent.rst
b/providers/common/ai/docs/operators/agent.rst
index 7e9b41d463f..fbb7b542989 100644
--- a/providers/common/ai/docs/operators/agent.rst
+++ b/providers/common/ai/docs/operators/agent.rst
@@ -378,9 +378,10 @@ Parameters
- ``enable_tool_logging``: Wrap the assembled function toolset in
:class:`~airflow.providers.common.ai.toolsets.logging.LoggingToolset` so that
tools supplied through ``toolsets=``, ``agent_params={"tools": [...]}``, and
- capabilities are logged in real time. Output tools, provider-native tools,
and
- tools another capability adds through its own wrapper (code mode's
``run_code``,
- ToolSearch's ``search_tools``) are not covered. Default ``True``.
+ capabilities are logged in real time, including tools the model found through
+ :ref:`tool search <howto/toolset:progressive-disclosure>`. Output tools,
provider-native
+ tools, and tools another capability adds through its own wrapper (code mode's
+ ``run_code``, ToolSearch's ``search_tools``) are not covered. Default
``True``.
- ``agent_params``: Additional keyword arguments passed to the pydantic-ai
``Agent`` constructor (e.g. ``retries``, ``model_settings``).
- .. _agent-usage-budget:
diff --git a/providers/common/ai/docs/stability.rst
b/providers/common/ai/docs/stability.rst
index 032f5f2ca6f..0abf63a93e6 100644
--- a/providers/common/ai/docs/stability.rst
+++ b/providers/common/ai/docs/stability.rst
@@ -223,8 +223,9 @@ Everything this provider ships that is not in the table
above is experimental.
- Thin integrations of ``pydantic-ai-harness`` and
``pydantic-ai-shields``, which are
before their 1.0 releases.
* - The Agent Skills toolset (:doc:`toolsets/skills`)
- - It may move from ``pydantic-ai-skills`` to the on-demand loading
Pydantic AI now has
- itself (`pydantic/pydantic-ai#5230
<https://github.com/pydantic/pydantic-ai/pull/5230>`__).
+ - It is built on ``pydantic-ai-skills``, a community package. Whether it
moves to
+ ``pydantic-ai-harness``'s ``Skills`` once that loads skill resources
and scripts is
+ still open.
* - :class:`~airflow.providers.common.ai.toolsets.sandbox.SandboxToolset`
and its
backends (:doc:`sandbox/index`)
- ``durable=True`` cannot be used with it yet, the ``sbx`` backend leaves
its sandbox
diff --git a/providers/common/ai/docs/toolsets/index.rst
b/providers/common/ai/docs/toolsets/index.rst
index c3716ac82c0..1e91337b879 100644
--- a/providers/common/ai/docs/toolsets/index.rst
+++ b/providers/common/ai/docs/toolsets/index.rst
@@ -67,13 +67,14 @@ with is resolved in the worker and is not part of any of
that.
- ``timeout`` for each call; what the remote agent may touch is set at the
vendor
-Two controls work on every toolset. ``.approval_required()`` pauses the task
before a
+Three controls work on every toolset. ``.approval_required()`` pauses the task
before a
matching call runs, until a person approves or rejects it on the **Required
Actions**
page. It needs Airflow 3.3 or later, pauses at every step that calls a
matching tool on
Airflow 3.4 and later (once per Dag run on 3.3), and does not combine with
``durable=True`` or a ``SandboxToolset`` that provisions its
own sandbox (:doc:`../tool_approval` lists every limit). ``.filtered()`` drops
tools
from what the model is offered, as the :ref:`MCP guide
<howto/toolset:mcp-filtered>`
-shows.
+shows. ``.defer_loading()`` hides a toolset's tools from the model until it
searches for
+them, which pays off on a toolset with many tools (see
:doc:`progressive_disclosure`).
:doc:`logging` and :doc:`langchain` do not reach a system of their own: one
wraps a
toolset to log its calls, the other converts a toolset for a LangChain agent.
@@ -92,6 +93,7 @@ toolset to log its calls, the other converts a toolset for a
LangChain agent.
Vendor-managed agents <managed_agent>
LangChain tools <langchain>
Tool call logging <logging>
+ Progressive tool disclosure <progressive_disclosure>
Importing toolsets
------------------
diff --git a/providers/common/ai/docs/toolsets/logging.rst
b/providers/common/ai/docs/toolsets/logging.rst
index 6cba50b98b5..a83d3b0984f 100644
--- a/providers/common/ai/docs/toolsets/logging.rst
+++ b/providers/common/ai/docs/toolsets/logging.rst
@@ -35,7 +35,9 @@ toolsets, nested capabilities, and MCP toolsets. Output tools
such as
``final_result`` and provider-native tools, including native MCP, are not
covered by Airflow's real-time tool-call logging. Tools added by another
capability through its own wrapper toolset, such as ToolSearch's
-``search_tools`` and CodeMode's ``run_code``, are also not covered.
+``search_tools`` and CodeMode's ``run_code``, are also not covered. A tool
hidden with
+``.defer_loading()`` is logged once the model finds it, and its result is
masked like
+any other (see :doc:`progressive_disclosure`).
``AgentOperator`` adds ``ToolLoggingCapability`` automatically when
``enable_tool_logging=True``. If ``capabilities=`` already contains one,
diff --git a/providers/common/ai/docs/toolsets/progressive_disclosure.rst
b/providers/common/ai/docs/toolsets/progressive_disclosure.rst
new file mode 100644
index 00000000000..c27c77ddac0
--- /dev/null
+++ b/providers/common/ai/docs/toolsets/progressive_disclosure.rst
@@ -0,0 +1,146 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/toolset:progressive-disclosure:
+
+Progressive tool disclosure
+===========================
+
+Every tool an agent can call costs input tokens on every model request: its
name,
+description and argument schema are sent each time, whether the model uses the
tool or
+not. ``.defer_loading()`` hides a toolset's tools until the model searches for
them, so
+an agent can carry a wide toolset and pay mostly for the tools a run looks
for. It is a
+Pydantic AI method that every toolset has, including the ones this provider
ships. Use it
+on toolsets with many tools, and leave a tool no run needs out of the toolset
altogether,
+with ``allowed_methods`` or ``.filtered()``.
+
+Hide a toolset's tools until the model needs them
+-------------------------------------------------
+
+This agent can call nine read-only ``S3Hook`` methods on one bucket, and the
model sees
+none of them until it searches:
+
+.. exampleinclude::
/../../ai/src/airflow/providers/common/ai/example_dags/example_progressive_tool_disclosure.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_toolset_defer_loading]
+ :end-before: [END howto_toolset_defer_loading]
+
+How the model finds a hidden tool depends on the model:
+
+- On a model with native tool search, the deferred tools are sent to the
provider with
+ ``defer_loading`` set, and the provider keeps their definitions out of the
model's
+ context and searches them itself.
+- On every other model, the hidden tools are left out of the request. The
model is
+ offered one ``search_tools`` tool that matches keywords against tool names
and
+ descriptions, and the tools it finds are sent with the next request. Until
then, each
+ request carries that one definition in place of the hidden ones, so the
saving grows
+ with the number of tools hidden.
+
+Pydantic AI decides which models have native tool search from a list of model
name
+prefixes in the installed version, so a model that list does not match gets
the local
+search. Its
+`tool search guide
<https://pydantic.dev/docs/ai/tools-toolsets/tools-advanced/#tool-search>`__
+has the details.
+
+Either way, a tool the model has found stays available while the search that
found it
+is in the run's message history.
+
+Keep the common tools visible
+-----------------------------
+
+A search costs a model request. When most runs use the same few tools, pass
the names of
+the tools to hide, and leave the rest in the request from the start:
+
+.. exampleinclude::
/../../ai/src/airflow/providers/common/ai/example_dags/example_progressive_tool_disclosure.py
+ :language: python
+ :dedent: 4
+ :start-after: [START howto_toolset_defer_loading_some]
+ :end-before: [END howto_toolset_defer_loading_some]
+
+The names are the tool names the model sees, so they include ``tool_prefix``.
A name that
+matches no tool is ignored without an error, so a misspelled name leaves that
tool visible.
+
+What a run looks like
+---------------------
+
+The first example, run on ``claude-haiku-5-5`` (which took the local search),
searched
+once, then called ``s3_list_keys`` once and ``s3_read_key`` twice. This is the
+``AgentOperator`` part of its task log, with timestamps and logger names
trimmed:
+
+.. code-block:: text
+
+ ::group::Tool call: s3_list_keys
+ Tool s3_list_keys returned in 0.20s
+ ::endgroup::
+ ::group::Tool call: s3_read_key
+ Tool s3_read_key returned in 0.08s
+ ::endgroup::
+ ::group::Tool call: s3_read_key
+ Tool s3_read_key returned in 0.01s
+ ::endgroup::
+ ::group::LLM run complete: model=claude-haiku-5-5, requests=4,
tool_calls=4, input_tokens=6683, output_tokens=474, total_tokens=7157
+ LLM prompt cache: cache_read_tokens=3894, cache_write_tokens=2779
+ Tool call sequence: search_tools -> s3_list_keys -> s3_read_key ->
s3_read_key
+ ::endgroup::
+
+``search_tools`` has no ``Tool call`` group of its own, because tool call
logging does not
+cover tools another capability adds. It shows only in the sequence at the end
of the run,
+and it is one of the four tool calls counted.
+
+The second example, with ``s3_list_keys`` and ``s3_read_key`` visible from the
start, made
+the same three S3 calls without a search, in one model request fewer:
+
+.. code-block:: text
+
+ ::group::LLM run complete: model=claude-haiku-5-5, requests=3,
tool_calls=3, input_tokens=4633, output_tokens=394, total_tokens=5027
+ LLM prompt cache: cache_read_tokens=2880, cache_write_tokens=1745
+ Tool call sequence: s3_list_keys -> s3_read_key -> s3_read_key
+ ::endgroup::
+
+The local search matches each keyword on its own, so one search can reveal
more tools than
+the model goes on to use. In a run where the model searched for ``list keys
read key``,
+the search revealed six of the nine S3 tools, and every request after it
carried all six.
+
+What stays the same
+-------------------
+
+A tool the model found is the same tool, called the same way:
+
+- Its result passes through Airflow's secret masker before the model sees it,
as every
+ tool result does (see :doc:`../agent_security`).
+- Its calls are logged like any other tool's (see :doc:`logging`).
+- With ``durable=True``, a retry replays the model requests and tool calls
that came
+ after a search; only the step that failed runs again (see
:doc:`../durable_execution`).
+- A templated connection ID on the toolset is rendered for each task instance,
as it is
+ without ``.defer_loading()``.
+
+.. _progressive-disclosure-when-not:
+
+When it does not help
+---------------------
+
+``SQLToolset`` has four tools whatever the database holds, and the model needs
them to
+answer almost anything, so hiding them saves a few definitions and adds a
search request
+to nearly every run. On a warehouse with hundreds of tables, ``list_tables``
+output grows with the table count; limit it with ``allowed_tables`` (see
+:ref:`sql-toolset-restricted`).
+
+``AgentSkillsToolset`` has four tools, and its list of skills goes into the
agent's
+instructions, which ``.defer_loading()`` does not touch, so the part of its
cost that grows
+with the number of skills stays. The toolset already loads each skill's body
only when
+the model asks for it.
diff --git a/providers/common/ai/docs/toolsets/skills.rst
b/providers/common/ai/docs/toolsets/skills.rst
index 52398c8a064..ea7b33c3245 100644
--- a/providers/common/ai/docs/toolsets/skills.rst
+++ b/providers/common/ai/docs/toolsets/skills.rst
@@ -33,11 +33,11 @@ prompt until the model decides it needs one, so a large
skill library costs few
tokens until used (progressive disclosure).
It is backed by the community `pydantic-ai-skills
-<https://github.com/DougTrajano/pydantic-ai-skills>`__ package (MIT). Pydantic
AI now
-has on-demand loading of its own (`pydantic/pydantic-ai#5230
-<https://github.com/pydantic/pydantic-ai/pull/5230>`__), and this toolset may
move to it.
-Install the optional
-extra to use it:
+<https://github.com/DougTrajano/pydantic-ai-skills>`__ package (MIT).
``pydantic-ai-harness``'s
+`Skills
<https://github.com/pydantic/pydantic-ai-harness/tree/main/pydantic_ai_harness/skills>`__
+capability also loads Agent Skills, as on-demand capabilities, but only their
``SKILL.md``
+instructions: no resources and no ``run_skill_script``.
+Install the optional extra to use this toolset:
.. code-block:: bash
diff --git a/providers/common/ai/pyproject.toml
b/providers/common/ai/pyproject.toml
index fb873ed778f..1e65a7fd32b 100644
--- a/providers/common/ai/pyproject.toml
+++ b/providers/common/ai/pyproject.toml
@@ -134,8 +134,9 @@ dependencies = [
"shields" = ["pydantic-ai-shields>=0.3.4"]
# Agent Skills (agentskills.io) support. pydantic-ai-skills provides the
toolset;
# the git provider supplies GitHook + GitPython for cloning GitSkills with
-# credentials from a `git` connection. Native progressive disclosure is tracked
-# upstream in pydantic/pydantic-ai#5230; revisit this extra once that lands.
+# credentials from a `git` connection. pydantic-ai-harness's Skills capability
is not
+# a replacement: it loads only SKILL.md instructions, with no skill resources
and no
+# run_skill_script.
"skills" = [
"apache-airflow-providers-git>=0.4.0",
"pydantic-ai-skills>=1.2.0",
diff --git
a/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_progressive_tool_disclosure.py
b/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_progressive_tool_disclosure.py
new file mode 100644
index 00000000000..b74379ce85f
--- /dev/null
+++
b/providers/common/ai/src/airflow/providers/common/ai/example_dags/example_progressive_tool_disclosure.py
@@ -0,0 +1,107 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Example Dags: an agent whose S3 tools stay hidden until the model searches
for them."""
+
+from __future__ import annotations
+
+from airflow.providers.common.ai.operators.agent import AgentOperator
+from airflow.providers.common.ai.toolsets import HookToolset
+from airflow.providers.common.compat.sdk import dag
+
+try:
+ from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+except ImportError:
+ S3Hook = None # type: ignore[assignment,misc]
+
+
+if S3Hook is not None:
+ # [START howto_toolset_defer_loading]
+ # Nine read-only S3Hook methods, one tool each.
+ REPORT_METHODS = [
+ "list_keys",
+ "list_prefixes",
+ "check_for_key",
+ "check_for_prefix",
+ "check_for_wildcard_key",
+ "check_for_bucket",
+ "head_object",
+ "read_key",
+ "get_bucket_tagging",
+ ]
+
+ @dag(tags=["example"])
+ def example_defer_loading():
+ AgentOperator(
+ task_id="answer_from_reports",
+ prompt="Summarize the reports under 2026/09/.",
+ llm_conn_id="pydanticai_default",
+ toolsets=[
+ # Hide every tool until the model searches for it.
+ HookToolset(
+ S3Hook(aws_conn_id="aws_reports_reader"),
+ allowed_methods=REPORT_METHODS,
+ pinned_arguments={"bucket_name": "acme-reports"},
+ tool_prefix="s3",
+ ).defer_loading()
+ ],
+ )
+
+ # [END howto_toolset_defer_loading]
+
+ example_defer_loading()
+
+ # [START howto_toolset_defer_loading_some]
+ @dag(tags=["example"])
+ def example_defer_loading_some():
+ AgentOperator(
+ task_id="answer_from_reports",
+ prompt="Summarize the reports under 2026/09/.",
+ llm_conn_id="pydanticai_default",
+ toolsets=[
+ HookToolset(
+ S3Hook(aws_conn_id="aws_reports_reader"),
+ allowed_methods=[
+ "list_keys",
+ "list_prefixes",
+ "check_for_key",
+ "check_for_prefix",
+ "check_for_wildcard_key",
+ "check_for_bucket",
+ "head_object",
+ "read_key",
+ "get_bucket_tagging",
+ ],
+ pinned_arguments={"bucket_name": "acme-reports"},
+ tool_prefix="s3",
+ ).defer_loading(
+ # Hide these seven; s3_list_keys and s3_read_key, which
most runs use, stay visible.
+ [
+ "s3_list_prefixes",
+ "s3_check_for_key",
+ "s3_check_for_prefix",
+ "s3_check_for_wildcard_key",
+ "s3_check_for_bucket",
+ "s3_head_object",
+ "s3_get_bucket_tagging",
+ ]
+ )
+ ],
+ )
+
+ # [END howto_toolset_defer_loading_some]
+
+ example_defer_loading_some()
