This is an automated email from the ASF dual-hosted git repository.
kaxil pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 002e4d0866d Nest the `common.ai` docs sidebar and lead with features
and providers (#73548)
002e4d0866d is described below
commit 002e4d0866d1d4f2dac40f0c662f5e2e1703840d
Author: Kaxil Naik <[email protected]>
AuthorDate: Tue Sep 22 14:46:47 2026 +0100
Nest the `common.ai` docs sidebar and lead with features and providers
(#73548)
The sidebar listed every page flat under nine captions, so a reader saw
about fifty entries and had to know class names to pick a page. Every
toctree also carried maxdepth 1, which stopped the Airflow theme from
rendering the nested pages it already supports.
The sidebar is now Basics, Getting started (installation, quick start,
core concepts) and seven expandable guides: What you can build, Models and
providers, Operators, Toolsets, LLM and agent features, Document and RAG
pipelines, and Reliability and operations. Each guide is a page that lists
its children, and the theme expands the branch the reader is in.
Sidebar labels lead with the feature rather than the class name, and page
titles carry both. A new supported-providers page maps each vendor to its
model prefix, extra, connection type and credentials, including Snowflake
Cortex, and links to the anchored connection examples. The classifier
models page links TypeSafe's site and documentation.
---
providers/common/ai/docs/classifier_models.rst | 9 +-
providers/common/ai/docs/concepts.rst | 3 +-
.../common/ai/docs/connections/pydantic_ai.rst | 25 +++-
providers/common/ai/docs/examples.rst | 4 +-
providers/common/ai/docs/features.rst | 54 ++++++++
providers/common/ai/docs/hooks/langchain.rst | 5 +-
providers/common/ai/docs/hooks/llamaindex.rst | 4 +-
providers/common/ai/docs/hooks/pydantic_ai.rst | 4 +-
providers/common/ai/docs/index.rst | 109 ++-------------
providers/common/ai/docs/installation.rst | 3 +-
providers/common/ai/docs/model_providers.rst | 148 +++++++++++++++++++++
providers/common/ai/docs/operations.rst | 45 +++++++
providers/common/ai/docs/operators/agent.rst | 4 +-
.../common/ai/docs/operators/document_loader.rst | 4 +-
providers/common/ai/docs/operators/index.rst | 14 ++
.../ai/docs/operators/llamaindex_embedding.rst | 4 +-
.../ai/docs/operators/llamaindex_retrieval.rst | 4 +-
providers/common/ai/docs/operators/llm.rst | 4 +-
providers/common/ai/docs/operators/llm_batch.rst | 4 +-
providers/common/ai/docs/operators/llm_branch.rst | 4 +-
.../common/ai/docs/operators/llm_file_analysis.rst | 4 +-
.../ai/docs/operators/llm_schema_compare.rst | 4 +-
providers/common/ai/docs/operators/llm_sql.rst | 4 +-
providers/common/ai/docs/rag_pipelines.rst | 45 +++++++
providers/common/ai/docs/sandbox/index.rst | 1 +
providers/common/ai/docs/toolsets/datafusion.rst | 4 +-
providers/common/ai/docs/toolsets/hook.rst | 4 +-
providers/common/ai/docs/toolsets/index.rst | 16 +++
providers/common/ai/docs/toolsets/langchain.rst | 4 +-
providers/common/ai/docs/toolsets/logging.rst | 4 +-
.../common/ai/docs/toolsets/managed_agent.rst | 4 +-
providers/common/ai/docs/toolsets/mcp.rst | 5 +-
providers/common/ai/docs/toolsets/skills.rst | 4 +-
providers/common/ai/docs/toolsets/sql.rst | 4 +-
providers/common/ai/docs/use_cases/index.rst | 15 +++
35 files changed, 426 insertions(+), 151 deletions(-)
diff --git a/providers/common/ai/docs/classifier_models.rst
b/providers/common/ai/docs/classifier_models.rst
index 4d82f08d559..64a835d4f91 100644
--- a/providers/common/ai/docs/classifier_models.rst
+++ b/providers/common/ai/docs/classifier_models.rst
@@ -23,8 +23,8 @@ some text and a typed question, and it answers with a value
from a set you named
advance, plus a confidence. Ask it for a string and the request is refused
before it
leaves your process.
-TypeSafe's Jev is the one pydantic-ai supports, as the ``typesafe:`` provider.
Nothing in
-this provider is specific to it -- it arrives through the same
+`TypeSafe <https://typesafe.ai>`__'s Jev is the one pydantic-ai supports, as
the
+``typesafe:`` provider. Nothing in this provider is specific to it -- it
arrives through the same
:class:`~airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIHook` as
every other
model, so a model id is the whole integration.
@@ -46,7 +46,7 @@ Setup
- **Connection Id**: ``jev_default``
- **Connection Type**: ``Pydantic AI``
- - **Password**: your TypeSafe API key
+ - **Password**: your TypeSafe API key, from your `TypeSafe account
<https://typesafe.ai>`__
- **Extra**: ``{"model": "typesafe:jev-1.13.0"}``
Leave **Host** empty unless you are pointing at a proxy; the provider defaults
to
@@ -182,7 +182,8 @@ classification that escalates when the confidence is low.
What it answers badly
---------------------
-Read `pydantic-ai's model page
<https://pydantic.dev/docs/ai/models/typesafe/>`__ before you
+Read `pydantic-ai's model page
<https://pydantic.dev/docs/ai/models/typesafe/>`__ and
+`TypeSafe's own documentation <https://docs.typesafe.ai/>`__ before you
trust a number from one of these models. Two of its failure modes matter more
than the
rest in a Dag:
diff --git a/providers/common/ai/docs/concepts.rst
b/providers/common/ai/docs/concepts.rst
index 7b6f99b9f23..25c0812ed6e 100644
--- a/providers/common/ai/docs/concepts.rst
+++ b/providers/common/ai/docs/concepts.rst
@@ -20,7 +20,8 @@
Core concepts
=============
-Five ideas carry the whole provider. Each has a page of its own; this page is
the map.
+The provider connects Airflow to a model through a connection, runs the call
or the agent
+loop inside a task, gives agents tools through toolsets, and returns results
through XCom.
Connections choose the model
----------------------------
diff --git a/providers/common/ai/docs/connections/pydantic_ai.rst
b/providers/common/ai/docs/connections/pydantic_ai.rst
index 41d866262d2..cc6c4b4b5e1 100644
--- a/providers/common/ai/docs/connections/pydantic_ai.rst
+++ b/providers/common/ai/docs/connections/pydantic_ai.rst
@@ -90,7 +90,10 @@ Fallback Connections
Examples
--------
-**OpenAI**
+.. _conn-example-openai:
+
+OpenAI
+^^^^^^
.. code-block:: json
@@ -100,7 +103,10 @@ Examples
"extra": "{\"model\": \"openai:gpt-5.6-sol\"}"
}
-**Anthropic**
+.. _conn-example-anthropic:
+
+Anthropic
+^^^^^^^^^
.. code-block:: json
@@ -110,7 +116,10 @@ Examples
"extra": "{\"model\": \"anthropic:claude-opus-4-6\"}"
}
-**Ollama (local)**
+.. _conn-example-ollama:
+
+Ollama (local)
+^^^^^^^^^^^^^^
.. code-block:: json
@@ -120,7 +129,10 @@ Examples
"extra": "{\"model\": \"openai:llama3\"}"
}
-**AWS Bedrock**
+.. _conn-example-bedrock:
+
+AWS Bedrock
+^^^^^^^^^^^
Leave password empty and configure ``AWS_PROFILE`` or IAM role in the
environment:
@@ -136,7 +148,10 @@ credential chain are unchanged. For AWS-specific fields
with dedicated UI
inputs (region, IAM keys, profile, bearer token, timeouts) instead of raw
``extra`` JSON, use the :doc:`pydantic_ai_bedrock` connection type.
-**Google Vertex AI / Gemini API**
+.. _conn-example-google:
+
+Google Vertex AI / Gemini API
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Leave password empty and configure ``GOOGLE_API_KEY`` (or ``GEMINI_API_KEY``)
in the environment:
diff --git a/providers/common/ai/docs/examples.rst
b/providers/common/ai/docs/examples.rst
index 89c86a2d854..c9a86288dd0 100644
--- a/providers/common/ai/docs/examples.rst
+++ b/providers/common/ai/docs/examples.rst
@@ -17,8 +17,8 @@
.. _howto/examples:
-All example Dags
-================
+Example Dags
+============
Every operator, decorator and integration has a runnable Dag under
`example_dags
<https://github.com/apache/airflow/tree/providers-common-ai/|version|/providers/common/ai/src/airflow/providers/common/ai/example_dags>`__,
diff --git a/providers/common/ai/docs/features.rst
b/providers/common/ai/docs/features.rst
new file mode 100644
index 00000000000..c9dee770cd7
--- /dev/null
+++ b/providers/common/ai/docs/features.rst
@@ -0,0 +1,54 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/features:
+
+LLM and agent features
+======================
+
+These settings change how a model call or an agent run behaves, independent of
which operator
+you use. Each is a parameter on the operator or decorator.
+
+- :doc:`structured_output` — ``output_type`` returns a typed Pydantic object
through XCom
+ instead of a string.
+- :doc:`message_history` — ``message_history`` carries a conversation across
agent runs.
+- :doc:`guardrails` — pydantic-ai capabilities and ``pydantic-ai-shields``
guards pass through
+ ``agent_params``.
+- :doc:`code_mode` — ``code_mode=True`` lets the model call several tools from
one Python
+ snippet instead of one round trip per call.
+- :doc:`approval_gates` — ``require_approval=True`` pauses an LLM operator
until a person
+ approves, edits or rejects the output.
+- :doc:`hitl_review` — ``enable_hitl_review=True`` opens an iterative review
loop on an agent,
+ with a chat UI and REST API for the reviewer.
+
+The last two are different tools for different jobs: an approval gate is a
one-shot decision on
+one output, a HITL review is a conversation with a running agent. Each page
opens with the
+other in a *see also* note.
+
+Making retries cheap with ``durable=True`` is a reliability feature and lives
under
+:doc:`operations`.
+
+.. toctree::
+ :hidden:
+ :titlesonly:
+
+ Structured output <structured_output>
+ Message history <message_history>
+ Guardrails <guardrails>
+ Code mode <code_mode>
+ Approve outputs <approval_gates>
+ Review agent sessions <hitl_review>
diff --git a/providers/common/ai/docs/hooks/langchain.rst
b/providers/common/ai/docs/hooks/langchain.rst
index 21134c8b3d3..3b988d00c73 100644
--- a/providers/common/ai/docs/hooks/langchain.rst
+++ b/providers/common/ai/docs/hooks/langchain.rst
@@ -17,10 +17,11 @@
.. _howto/hook:langchain:
-``LangChainHook``
-=================
+LangChain models: ``LangChainHook``
+===================================
.. toctree::
+ :titlesonly:
:hidden:
:maxdepth: 1
diff --git a/providers/common/ai/docs/hooks/llamaindex.rst
b/providers/common/ai/docs/hooks/llamaindex.rst
index cad520c014d..a3e0fc30f49 100644
--- a/providers/common/ai/docs/hooks/llamaindex.rst
+++ b/providers/common/ai/docs/hooks/llamaindex.rst
@@ -17,8 +17,8 @@
.. _howto/hook:llamaindex:
-``LlamaIndexHook``
-==================
+Using LlamaIndex directly: ``LlamaIndexHook``
+=============================================
Use :class:`~airflow.providers.common.ai.hooks.llamaindex.LlamaIndexHook` to
bridge an Airflow connection to `LlamaIndex <https://docs.llamaindex.ai/>`__
diff --git a/providers/common/ai/docs/hooks/pydantic_ai.rst
b/providers/common/ai/docs/hooks/pydantic_ai.rst
index 79c9548ad71..bcc9f334aaa 100644
--- a/providers/common/ai/docs/hooks/pydantic_ai.rst
+++ b/providers/common/ai/docs/hooks/pydantic_ai.rst
@@ -17,8 +17,8 @@
.. _howto/hook:pydantic_ai:
-PydanticAIHook
-==============
+Using the hook directly: ``PydanticAIHook``
+===========================================
Use :class:`~airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIHook` to
interact
with LLM providers via `pydantic-ai <https://ai.pydantic.dev/>`__.
diff --git a/providers/common/ai/docs/index.rst
b/providers/common/ai/docs/index.rst
index da65b8e9fa3..d2c0a4fee2f 100644
--- a/providers/common/ai/docs/index.rst
+++ b/providers/common/ai/docs/index.rst
@@ -74,8 +74,8 @@ Getting started
* :doc:`use_cases/index` — jobs a data team already has, each with the Dag
that does it.
.. toctree::
+ :titlesonly:
:hidden:
- :maxdepth: 1
:caption: Basics
Home <self>
@@ -83,116 +83,33 @@ Getting started
Security <security>
.. toctree::
+ :titlesonly:
:hidden:
- :maxdepth: 1
:caption: Getting started
Installation <installation>
Quick start <quickstart>
Core concepts <concepts>
- Structured output <structured_output>
-
-.. toctree::
- :hidden:
- :maxdepth: 1
- :caption: What you can build
-
- Overview <use_cases/index>
- Triage support tickets <use_cases/triage_support_tickets>
- Route pipeline failures <use_cases/route_pipeline_failures>
- Block a load on schema drift <use_cases/gate_loads_on_schema_drift>
- Explain a revenue anomaly <use_cases/explain_revenue_anomaly>
- Monthly report from a CSV <use_cases/monthly_report_from_a_csv>
- Compare 10-K filings <use_cases/compare_10k_filings>
- Ask questions over PDFs <use_cases/ask_questions_over_pdfs>
- Weekly status report <use_cases/weekly_status_report>
- Classify reviews in bulk <use_cases/classify_reviews_in_bulk>
- Research agent with review <use_cases/research_agent_with_review>
-
-.. toctree::
- :hidden:
- :maxdepth: 1
- :caption: Models and providers
-
- Pydantic AI connection <connections/pydantic_ai>
- Azure OpenAI <connections/pydantic_ai_azure>
- AWS Bedrock <connections/pydantic_ai_bedrock>
- Google Vertex AI <connections/pydantic_ai_vertex>
- Self-hosted models <self_hosted_models>
- Classifier models <classifier_models>
- Provider fallback <provider_fallback>
- PydanticAIHook <hooks/pydantic_ai>
- LangChainHook <hooks/langchain>
-
-.. toctree::
- :hidden:
- :maxdepth: 1
- :caption: Operators
-
- Choosing an operator <operators/index>
- LLMOperator <operators/llm>
- LLMBranchOperator <operators/llm_branch>
- LLMFileAnalysisOperator <operators/llm_file_analysis>
- LLMSQLQueryOperator <operators/llm_sql>
- LLMSchemaCompareOperator <operators/llm_schema_compare>
- LLMBatchOperator <operators/llm_batch>
- AgentOperator <operators/agent>
- Approval gates <approval_gates>
-
-.. toctree::
- :hidden:
- :maxdepth: 1
- :caption: Toolsets
-
- Choosing a toolset <toolsets/index>
- HookToolset <toolsets/hook>
- SQLToolset <toolsets/sql>
- DataFusionToolset <toolsets/datafusion>
- LoggingToolset <toolsets/logging>
- MCPToolset <toolsets/mcp>
- AgentSkillsToolset <toolsets/skills>
- Managed agent toolsets <toolsets/managed_agent>
- LangChain bridge <toolsets/langchain>
.. toctree::
+ :titlesonly:
:hidden:
- :maxdepth: 1
- :caption: Running agents
+ :caption: Guides
- Message history <message_history>
- Guardrails <guardrails>
- Code mode <code_mode>
- Sandboxed execution <sandbox/index>
- HITL review <hitl_review>
+ What you can build <use_cases/index>
+ Models and providers <model_providers>
+ Operators <operators/index>
+ Toolsets <toolsets/index>
+ LLM and agent features <features>
+ Document and RAG pipelines <rag_pipelines>
+ Reliability and operations <operations>
.. toctree::
+ :titlesonly:
:hidden:
- :maxdepth: 1
- :caption: Document and RAG pipelines
-
- DocumentLoaderOperator <operators/document_loader>
- LlamaIndexEmbeddingOperator <operators/llamaindex_embedding>
- LlamaIndexRetrievalOperator <operators/llamaindex_retrieval>
- LlamaIndex connection <connections/llamaindex>
- LlamaIndexHook <hooks/llamaindex>
-
-.. toctree::
- :hidden:
- :maxdepth: 1
- :caption: Reliability and operations
-
- Durable execution <durable_execution>
- Retry policies <retry_policies>
- Observability <observability>
- Securing agent tools <agent_security>
- Troubleshooting <troubleshooting>
-
-.. toctree::
- :hidden:
- :maxdepth: 1
:caption: References
- All example Dags <examples>
+ Example Dags <examples>
Configuration <configurations-ref>
Python API <_api/airflow/providers/common/ai/index>
diff --git a/providers/common/ai/docs/installation.rst
b/providers/common/ai/docs/installation.rst
index 6e0508753fd..5b45b1d9b21 100644
--- a/providers/common/ai/docs/installation.rst
+++ b/providers/common/ai/docs/installation.rst
@@ -38,7 +38,8 @@ Choosing extras
The provider's extras split into a few groups:
* **Model providers** — ``openai``, ``anthropic``, ``google``, ``bedrock``,
``typesafe``:
- pick the one matching your ``llm_conn_id`` connection. ``typesafe`` differs
from the rest
+ pick the one matching your ``llm_conn_id`` connection
(:doc:`model_providers` maps
+ vendors to extras, prefixes and connection types). ``typesafe`` differs from
the rest
in kind: it installs a classifier model that answers typed questions and
cannot write
text (see :doc:`classifier_models`). The first four mirror the identically
named
``pydantic-ai-slim`` optional dependency groups, and ``typesafe`` adds the
``typesafe-sdk``
diff --git a/providers/common/ai/docs/model_providers.rst
b/providers/common/ai/docs/model_providers.rst
new file mode 100644
index 00000000000..7e31a70b648
--- /dev/null
+++ b/providers/common/ai/docs/model_providers.rst
@@ -0,0 +1,148 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/model-providers:
+
+Supported model providers
+=========================
+
+Every pydantic-ai model call in this provider, from an operator, a decorator
or the hook,
+reaches its model through one Airflow connection, and the ``provider:`` prefix
on the model
+name picks the vendor. Switching vendors is a connection
+change, not a Dag change. Find your vendor below, install the extra, create
the connection
+type shown, and set the model name with that prefix.
+
+.. list-table::
+ :header-rows: 1
+ :widths: 18 16 18 20 28
+
+ * - Vendor
+ - Model prefix
+ - Install
+ - Connection type
+ - Credentials
+ * - :ref:`OpenAI <conn-example-openai>`
+ - ``openai:``
+ - ``[openai]``
+ - ``pydanticai``
+ - API key in **Password**
+ * - :ref:`Anthropic <conn-example-anthropic>`
+ - ``anthropic:``
+ - ``[anthropic]``
+ - ``pydanticai``
+ - API key in **Password**
+ * - :ref:`Google Gemini API <conn-example-google>`
+ - ``google:``
+ - ``[google]``
+ - ``pydanticai``
+ - API key in **Password**, or ``GOOGLE_API_KEY`` in the environment
+ * - Google Vertex AI
+ - ``google-cloud:``
+ - ``[google]``
+ - ``pydanticai_vertex`` (:doc:`connections/pydantic_ai_vertex`)
+ - Service account or Application Default Credentials
+ * - AWS Bedrock
+ - ``bedrock:``
+ - ``[bedrock]``
+ - ``pydanticai_bedrock`` (:doc:`connections/pydantic_ai_bedrock`), or
``pydanticai``
+ with AWS credentials in the environment
+ - IAM keys, profile or role
+ * - Azure OpenAI
+ - ``azure:``
+ - ``[openai]``
+ - ``pydanticai_azure`` (:doc:`connections/pydantic_ai_azure`)
+ - API key in **Password**, resource endpoint in **Host**
+ * - Groq
+ - ``groq:``
+ - ``pydantic-ai-slim[groq]``
+ - ``pydanticai``
+ - API key in **Password**
+ * - Mistral AI
+ - ``mistral:``
+ - ``pydantic-ai-slim[mistral]``
+ - ``pydanticai``
+ - API key in **Password**
+ * - DeepSeek
+ - ``deepseek:``
+ - ``[openai]``
+ - ``pydanticai``
+ - API key in **Password**
+ * - :ref:`Ollama <conn-example-ollama>`, vLLM and other OpenAI-compatible
servers
+ - ``openai:``
+ - ``[openai]``
+ - ``pydanticai`` with the server URL in **Host**
(:doc:`self_hosted_models`)
+ - Usually none
+ * - Snowflake Cortex
+ - ``snowflake:``
+ - ``pydantic-ai-slim[snowflake]``
+ - ``pydanticai``
+ - ``SNOWFLAKE_ACCOUNT`` and ``SNOWFLAKE_TOKEN`` in the worker
environment; leave
+ **Password** empty
+ * - TypeSafe Jev (classifier, does not write text)
+ - ``typesafe:``
+ - ``[typesafe]``
+ - ``pydanticai`` (:doc:`classifier_models`)
+ - API key in **Password**
+
+``[name]`` in the Install column is an extra of this provider, installed as
+``pip install "apache-airflow-providers-common-ai[name]"``. The Groq, Mistral
and Snowflake entries
+name the matching ``pydantic-ai-slim`` extra instead, because this provider
does not ship one
+for them.
+
+Any other vendor that `pydantic-ai supports
<https://ai.pydantic.dev/models/overview/>`__
+(Cohere, OpenRouter, Hugging Face and more) works the same
+way: install the ``pydantic-ai-slim`` extra named on that vendor's pydantic-ai
page, create a
+``pydanticai`` connection, and use the prefix from that page.
+
+Where the model name goes
+-------------------------
+
+Set the model on the connection's **Model** field in ``provider:model`` form,
for example
+``anthropic:claude-sonnet-5``. An operator's ``model_id`` overrides it for
that task, so one
+connection can serve several models from the same vendor. The generic
``pydanticai``
+connection has no vendor of its own, so a bare model name without a prefix is
rejected; the
+Azure, Bedrock and Vertex connection types each supply their own prefix and
accept a bare
+name. :doc:`connections/pydantic_ai` has every field and the full resolution
order.
+
+Reliability across vendors
+--------------------------
+
+A connection can name other connections to fail over to when its vendor is
unavailable, so
+one task can span OpenAI and Anthropic without a code change. See
:doc:`provider_fallback`.
+
+Using a model outside an operator
+---------------------------------
+
+:doc:`hooks/pydantic_ai` returns the pydantic-ai ``Agent`` or ``Model`` behind
a connection
+for use in a plain ``@task``. :doc:`hooks/langchain` does the same for
LangChain chat and
+embedding models, which have their own ``langchain`` connection type.
+
+Pages in this section
+---------------------
+
+.. toctree::
+ :titlesonly:
+
+ Azure OpenAI <connections/pydantic_ai_azure>
+ AWS Bedrock <connections/pydantic_ai_bedrock>
+ Google Vertex AI <connections/pydantic_ai_vertex>
+ Self-hosted models <self_hosted_models>
+ Classifier models <classifier_models>
+ Provider fallback <provider_fallback>
+ Connection reference <connections/pydantic_ai>
+ Using the hook directly <hooks/pydantic_ai>
+ LangChain models <hooks/langchain>
diff --git a/providers/common/ai/docs/operations.rst
b/providers/common/ai/docs/operations.rst
new file mode 100644
index 00000000000..77226593e30
--- /dev/null
+++ b/providers/common/ai/docs/operations.rst
@@ -0,0 +1,45 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/operations:
+
+Reliability and operations
+==========================
+
+An AI task fails in more ways than a SQL task, and it costs money each time it
runs.
+
+- :doc:`durable_execution` replays the model and tool calls an agent already
completed, so a
+ retry pays only for the steps that did not finish.
+- :doc:`retry_policies` lets a model classify a failure and decide whether a
retry is worth
+ it at all, with a plain rule table as the floor.
+- :doc:`observability` exports model and tool calls as OpenTelemetry traces
and metrics.
+- :doc:`agent_security` is the defense-layer guide for agents that hold tools:
what an agent
+ can reach, how ``allowed_tables`` is enforced, and the production checklist.
+- :doc:`provider_fallback` fails over to another vendor inside one task
attempt when a
+ model provider is down or rate limiting. It is configured on the connection,
so it also
+ appears under :doc:`model_providers`.
+- :doc:`troubleshooting` lists the errors a first Dag most often hits, with
the fix for each.
+
+.. toctree::
+ :hidden:
+ :titlesonly:
+
+ Durable execution <durable_execution>
+ Retry policies <retry_policies>
+ Observability <observability>
+ Securing agent tools <agent_security>
+ Troubleshooting <troubleshooting>
diff --git a/providers/common/ai/docs/operators/agent.rst
b/providers/common/ai/docs/operators/agent.rst
index acce8184a95..d3e24f9305f 100644
--- a/providers/common/ai/docs/operators/agent.rst
+++ b/providers/common/ai/docs/operators/agent.rst
@@ -17,8 +17,8 @@
.. _howto/operator:agent:
-``AgentOperator`` & ``@task.agent``
-===================================
+Agents with tools: ``AgentOperator`` and ``@task.agent``
+========================================================
Use :class:`~airflow.providers.common.ai.operators.agent.AgentOperator` or
the ``@task.agent`` decorator to run an LLM agent with **tools** — the agent
diff --git a/providers/common/ai/docs/operators/document_loader.rst
b/providers/common/ai/docs/operators/document_loader.rst
index 843d059e282..a699304ecc3 100644
--- a/providers/common/ai/docs/operators/document_loader.rst
+++ b/providers/common/ai/docs/operators/document_loader.rst
@@ -17,8 +17,8 @@
.. _howto/operator:document_loader:
-``DocumentLoaderOperator``
-==========================
+Load documents: ``DocumentLoaderOperator``
+==========================================
Use
:class:`~airflow.providers.common.ai.operators.document_loader.DocumentLoaderOperator`
to parse files into ``list[dict(text, metadata)]`` for downstream embedding
diff --git a/providers/common/ai/docs/operators/index.rst
b/providers/common/ai/docs/operators/index.rst
index fa411a54713..19a129d25d6 100644
--- a/providers/common/ai/docs/operators/index.rst
+++ b/providers/common/ai/docs/operators/index.rst
@@ -121,3 +121,17 @@ Use this provider when a Dag needs:
and :doc:`LlamaIndexRetrievalOperator <llamaindex_retrieval>` retrieves the
closest
chunks for an :doc:`LLMOperator <llm>` prompt (see the table above for the
full set).
+
+Operator guides
+---------------
+
+.. toctree::
+ :titlesonly:
+
+ Single prompts <llm>
+ Branch on an answer <llm_branch>
+ Analyze files and images <llm_file_analysis>
+ Natural language to SQL <llm_sql>
+ Detect schema drift <llm_schema_compare>
+ Batch processing <llm_batch>
+ Agents with tools <agent>
diff --git a/providers/common/ai/docs/operators/llamaindex_embedding.rst
b/providers/common/ai/docs/operators/llamaindex_embedding.rst
index 894045b684c..fff4ea461b0 100644
--- a/providers/common/ai/docs/operators/llamaindex_embedding.rst
+++ b/providers/common/ai/docs/operators/llamaindex_embedding.rst
@@ -17,8 +17,8 @@
.. _howto/operator:llamaindex_embedding:
-LlamaIndex ``LlamaIndexEmbeddingOperator``
-==========================================
+Embed documents: ``LlamaIndexEmbeddingOperator``
+================================================
Chunk a ``list[dict]`` of documents and produce embedding vectors using
LlamaIndex. Designed to feed the output of
diff --git a/providers/common/ai/docs/operators/llamaindex_retrieval.rst
b/providers/common/ai/docs/operators/llamaindex_retrieval.rst
index 3f064a2b825..c346cd9dd28 100644
--- a/providers/common/ai/docs/operators/llamaindex_retrieval.rst
+++ b/providers/common/ai/docs/operators/llamaindex_retrieval.rst
@@ -17,8 +17,8 @@
.. _howto/operator:llamaindex_retrieval:
-LlamaIndex ``LlamaIndexRetrievalOperator``
-==========================================
+Retrieve context: ``LlamaIndexRetrievalOperator``
+=================================================
Load a persisted LlamaIndex index and run similarity search. Designed to
sit between
diff --git a/providers/common/ai/docs/operators/llm.rst
b/providers/common/ai/docs/operators/llm.rst
index 12225e2a271..3989017ad79 100644
--- a/providers/common/ai/docs/operators/llm.rst
+++ b/providers/common/ai/docs/operators/llm.rst
@@ -17,8 +17,8 @@
.. _howto/operator:llm:
-``LLMOperator``
-===============
+Single prompts: ``LLMOperator`` and ``@task.llm``
+=================================================
Use :class:`~airflow.providers.common.ai.operators.llm.LLMOperator` for
general-purpose LLM calls — summarization, extraction, classification,
diff --git a/providers/common/ai/docs/operators/llm_batch.rst
b/providers/common/ai/docs/operators/llm_batch.rst
index b31a72c082c..f0050f2b708 100644
--- a/providers/common/ai/docs/operators/llm_batch.rst
+++ b/providers/common/ai/docs/operators/llm_batch.rst
@@ -17,8 +17,8 @@
.. _howto/operator:llm_batch:
-``LLMBatchOperator``
-=====================
+Batch processing: ``LLMBatchOperator``
+======================================
Use :class:`~airflow.providers.common.ai.operators.llm_batch.LLMBatchOperator`
to run many
prompts through a provider's **batch API** instead of one synchronous call per
prompt:
diff --git a/providers/common/ai/docs/operators/llm_branch.rst
b/providers/common/ai/docs/operators/llm_branch.rst
index 5d0ca54f9bd..30944eccbaf 100644
--- a/providers/common/ai/docs/operators/llm_branch.rst
+++ b/providers/common/ai/docs/operators/llm_branch.rst
@@ -17,8 +17,8 @@
.. _howto/operator:llm_branch:
-``LLMBranchOperator``
-=====================
+Branch on an answer: ``LLMBranchOperator``
+==========================================
Use
:class:`~airflow.providers.common.ai.operators.llm_branch.LLMBranchOperator`
for LLM-driven branching — where the LLM decides which downstream task(s) to
diff --git a/providers/common/ai/docs/operators/llm_file_analysis.rst
b/providers/common/ai/docs/operators/llm_file_analysis.rst
index cf2330ca905..75a54c805e1 100644
--- a/providers/common/ai/docs/operators/llm_file_analysis.rst
+++ b/providers/common/ai/docs/operators/llm_file_analysis.rst
@@ -17,8 +17,8 @@
.. _howto/operator:llm_file_analysis:
-``LLMFileAnalysisOperator`` & ``@task.llm_file_analysis``
-=========================================================
+Analyze files and images: ``LLMFileAnalysisOperator``
+=====================================================
Use
:class:`~airflow.providers.common.ai.operators.llm_file_analysis.LLMFileAnalysisOperator`
or the ``@task.llm_file_analysis`` decorator to analyze files from object
storage
diff --git a/providers/common/ai/docs/operators/llm_schema_compare.rst
b/providers/common/ai/docs/operators/llm_schema_compare.rst
index affe597fb85..b069fa30b27 100644
--- a/providers/common/ai/docs/operators/llm_schema_compare.rst
+++ b/providers/common/ai/docs/operators/llm_schema_compare.rst
@@ -17,8 +17,8 @@
.. _howto/operator:llm_schema_compare:
-``LLMSchemaCompareOperator``
-============================
+Detect schema drift: ``LLMSchemaCompareOperator``
+=================================================
Use
:class:`~airflow.providers.common.ai.operators.llm_schema_compare.LLMSchemaCompareOperator`
to compare schemas across different database systems and detect drift using
LLM reasoning.
diff --git a/providers/common/ai/docs/operators/llm_sql.rst
b/providers/common/ai/docs/operators/llm_sql.rst
index 50f53e62894..015845b45d8 100644
--- a/providers/common/ai/docs/operators/llm_sql.rst
+++ b/providers/common/ai/docs/operators/llm_sql.rst
@@ -17,8 +17,8 @@
.. _howto/operator:llm_sql_query:
-``LLMSQLQueryOperator``
-========================
+Natural language to SQL: ``LLMSQLQueryOperator``
+================================================
Use
:class:`~airflow.providers.common.ai.operators.llm_sql.LLMSQLQueryOperator` to
generate
SQL queries from natural language using an LLM.
diff --git a/providers/common/ai/docs/rag_pipelines.rst
b/providers/common/ai/docs/rag_pipelines.rst
new file mode 100644
index 00000000000..0825eafd650
--- /dev/null
+++ b/providers/common/ai/docs/rag_pipelines.rst
@@ -0,0 +1,45 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/rag-pipelines:
+
+Document and RAG pipelines
+==========================
+
+A retrieval pipeline in this provider is three ordinary tasks.
:doc:`operators/document_loader`
+parses files (text, CSV, JSON, PDF, DOCX) into a list of ``{"text",
"metadata"}`` dicts with
+no AI framework involved. :doc:`operators/llamaindex_embedding` chunks those
documents and
+produces embedding vectors. :doc:`operators/llamaindex_retrieval` pulls the
closest chunks back
+out for a question, ready to drop into an :doc:`LLMOperator <operators/llm>`
prompt. Each step
+is a task, so indexing can run on a schedule while querying runs on demand.
+
+The LlamaIndex operators read their embedding and language models from a
``llamaindex``
+connection. :doc:`hooks/llamaindex` returns those LlamaIndex objects for use
in a plain
+``@task`` when the operators do not fit.
+
+:doc:`use_cases/ask_questions_over_pdfs` and
:doc:`use_cases/compare_10k_filings` show the
+whole shape end to end.
+
+.. toctree::
+ :hidden:
+ :titlesonly:
+
+ Load documents <operators/document_loader>
+ Embed documents <operators/llamaindex_embedding>
+ Retrieve context <operators/llamaindex_retrieval>
+ LlamaIndex connection <connections/llamaindex>
+ Using LlamaIndex directly <hooks/llamaindex>
diff --git a/providers/common/ai/docs/sandbox/index.rst
b/providers/common/ai/docs/sandbox/index.rst
index 68de7370423..1fbaccb4ba1 100644
--- a/providers/common/ai/docs/sandbox/index.rst
+++ b/providers/common/ai/docs/sandbox/index.rst
@@ -68,6 +68,7 @@ Pages in this section
---------------------
.. toctree::
+ :titlesonly:
:maxdepth: 1
Configuration and lifecycle <configuration>
diff --git a/providers/common/ai/docs/toolsets/datafusion.rst
b/providers/common/ai/docs/toolsets/datafusion.rst
index 380689a3783..da467527d93 100644
--- a/providers/common/ai/docs/toolsets/datafusion.rst
+++ b/providers/common/ai/docs/toolsets/datafusion.rst
@@ -15,8 +15,8 @@
specific language governing permissions and limitations
under the License.
-``DataFusionToolset``
-=====================
+Files with DataFusion: ``DataFusionToolset``
+============================================
Curated toolset wrapping
:class:`~airflow.providers.common.sql.datafusion.engine.DataFusionEngine`
diff --git a/providers/common/ai/docs/toolsets/hook.rst
b/providers/common/ai/docs/toolsets/hook.rst
index a1a65c98128..fbe987af612 100644
--- a/providers/common/ai/docs/toolsets/hook.rst
+++ b/providers/common/ai/docs/toolsets/hook.rst
@@ -15,8 +15,8 @@
specific language governing permissions and limitations
under the License.
-``HookToolset``
-===============
+Airflow hooks as tools: ``HookToolset``
+=======================================
Generic adapter that exposes selected methods of any Airflow Hook as
pydantic-ai tools via introspection. Requires an explicit ``allowed_methods``
diff --git a/providers/common/ai/docs/toolsets/index.rst
b/providers/common/ai/docs/toolsets/index.rst
index e40653608fe..b9dfbe84450 100644
--- a/providers/common/ai/docs/toolsets/index.rst
+++ b/providers/common/ai/docs/toolsets/index.rst
@@ -83,6 +83,22 @@ The hook, SQL, DataFusion, MCP, Agent Skills and
managed-agent guides each have
example that exists in this repository, and where its credentials and its work
come
from. :doc:`../sandbox/index` carries the same section for ``SandboxToolset``.
+Toolset guides
+--------------
+
+.. toctree::
+ :titlesonly:
+
+ Airflow hooks as tools <hook>
+ SQL databases <sql>
+ Files with DataFusion <datafusion>
+ MCP servers <mcp>
+ Agent Skills <skills>
+ Sandboxed execution <../sandbox/index>
+ Vendor-managed agents <managed_agent>
+ LangChain tools <langchain>
+ Tool call logging <logging>
+
The toolsets
------------
diff --git a/providers/common/ai/docs/toolsets/langchain.rst
b/providers/common/ai/docs/toolsets/langchain.rst
index e83dcecd109..2901d4a836d 100644
--- a/providers/common/ai/docs/toolsets/langchain.rst
+++ b/providers/common/ai/docs/toolsets/langchain.rst
@@ -15,8 +15,8 @@
specific language governing permissions and limitations
under the License.
-LangChain bridge
-================
+LangChain tools in both directions
+==================================
Tools bridge in both directions between common.ai's toolsets and LangChain.
diff --git a/providers/common/ai/docs/toolsets/logging.rst
b/providers/common/ai/docs/toolsets/logging.rst
index f91d3e5b0e1..4e40a75329c 100644
--- a/providers/common/ai/docs/toolsets/logging.rst
+++ b/providers/common/ai/docs/toolsets/logging.rst
@@ -15,8 +15,8 @@
specific language governing permissions and limitations
under the License.
-``LoggingToolset``
-==================
+Tool call logging: ``LoggingToolset``
+=====================================
:class:`~airflow.providers.common.ai.toolsets.logging.LoggingToolset` is a
``WrapperToolset`` that intercepts ``call_tool()`` to log each tool invocation
diff --git a/providers/common/ai/docs/toolsets/managed_agent.rst
b/providers/common/ai/docs/toolsets/managed_agent.rst
index e2993a8553b..b943012ec26 100644
--- a/providers/common/ai/docs/toolsets/managed_agent.rst
+++ b/providers/common/ai/docs/toolsets/managed_agent.rst
@@ -17,8 +17,8 @@
.. _managed-agent-toolsets:
-Managed agent toolsets
-======================
+Vendor-managed agents: ``BaseManagedAgentToolset``
+==================================================
Cloud vendors now run agents on your behalf — Snowflake Cortex Agents, Amazon
Bedrock AgentCore runtimes, Azure AI Foundry hosted agents, Vertex AI Agent
diff --git a/providers/common/ai/docs/toolsets/mcp.rst
b/providers/common/ai/docs/toolsets/mcp.rst
index 2c7f378cc5a..adf1e493b6a 100644
--- a/providers/common/ai/docs/toolsets/mcp.rst
+++ b/providers/common/ai/docs/toolsets/mcp.rst
@@ -15,10 +15,11 @@
specific language governing permissions and limitations
under the License.
-``MCPToolset``
-==============
+MCP servers: ``MCPToolset``
+===========================
.. toctree::
+ :titlesonly:
:hidden:
:maxdepth: 1
diff --git a/providers/common/ai/docs/toolsets/skills.rst
b/providers/common/ai/docs/toolsets/skills.rst
index 1ce37c1391b..345becb72f3 100644
--- a/providers/common/ai/docs/toolsets/skills.rst
+++ b/providers/common/ai/docs/toolsets/skills.rst
@@ -17,8 +17,8 @@
.. _agent-skills:
-``AgentSkillsToolset``
-======================
+Agent Skills: ``AgentSkillsToolset``
+====================================
:class:`~airflow.providers.common.ai.toolsets.skills.AgentSkillsToolset` loads
`Agent Skills <https://agentskills.io>`__ -- ``SKILL.md`` bundles
(instructions,
diff --git a/providers/common/ai/docs/toolsets/sql.rst
b/providers/common/ai/docs/toolsets/sql.rst
index eacaedbd7d5..45d6f688fc7 100644
--- a/providers/common/ai/docs/toolsets/sql.rst
+++ b/providers/common/ai/docs/toolsets/sql.rst
@@ -15,8 +15,8 @@
specific language governing permissions and limitations
under the License.
-``SQLToolset``
-==============
+SQL databases: ``SQLToolset``
+=============================
Curated toolset wrapping
:class:`~airflow.providers.common.sql.hooks.sql.DbApiHook` with four tools:
diff --git a/providers/common/ai/docs/use_cases/index.rst
b/providers/common/ai/docs/use_cases/index.rst
index f6417a4581d..027d79c33e4 100644
--- a/providers/common/ai/docs/use_cases/index.rst
+++ b/providers/common/ai/docs/use_cases/index.rst
@@ -106,3 +106,18 @@ can be built from the patterns shown on the pages above.
Dags with ``HITLEntryOperator`` or ``ApprovalOperator`` pause under
``airflow dags test``
until someone answers from Required Actions in the UI of an api-server on
the same
metadata database. ``airflow standalone`` gives you one.
+
+.. toctree::
+ :titlesonly:
+ :hidden:
+
+ Triage support tickets <triage_support_tickets>
+ Route pipeline failures <route_pipeline_failures>
+ Block a load on schema drift <gate_loads_on_schema_drift>
+ Explain a revenue anomaly <explain_revenue_anomaly>
+ Monthly report from a CSV <monthly_report_from_a_csv>
+ Compare 10-K filings <compare_10k_filings>
+ Ask questions over PDFs <ask_questions_over_pdfs>
+ Weekly status report <weekly_status_report>
+ Classify reviews in bulk <classify_reviews_in_bulk>
+ Research agent with review <research_agent_with_review>
