This is an automated email from the ASF dual-hosted git repository.
Lee-W pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new b15c4852bdb Document when to use common.ai vs vendor-specific AI
providers (#69551)
b15c4852bdb is described below
commit b15c4852bdbeab70b8f9066dafc97b04de609866
Author: Wei Lee <[email protected]>
AuthorDate: Wed Jul 8 17:54:21 2026 +0800
Document when to use common.ai vs vendor-specific AI providers (#69551)
---
providers/anthropic/docs/index.rst | 35 +++++++++++++++++++++++++++++++++
providers/common/ai/docs/index.rst | 40 ++++++++++++++++++++++++++++++++++++++
providers/openai/docs/index.rst | 35 +++++++++++++++++++++++++++++++++
3 files changed, 110 insertions(+)
diff --git a/providers/anthropic/docs/index.rst
b/providers/anthropic/docs/index.rst
index 4fb3da029fd..3794d1ae58b 100644
--- a/providers/anthropic/docs/index.rst
+++ b/providers/anthropic/docs/index.rst
@@ -19,6 +19,41 @@
``apache-airflow-providers-anthropic``
======================================
+When to use this provider
+--------------------------
+
+Use ``anthropic`` when a Dag needs Anthropic's native API surface — services
Anthropic runs
+for you, which no vendor-neutral operator wraps:
+
+* ``AnthropicBatchOperator`` and ``AnthropicBatchSensor`` — submit a Claude
+ `Message Batches
<https://docs.claude.com/en/docs/build-with-claude/batch-processing>`__
+ job for asynchronous bulk processing and wait for it to complete.
+* ``AnthropicAgentSessionOperator`` — start a Managed Agents session in which
the agent loop
+ runs server-side on Anthropic's infrastructure; the Airflow task only kicks
off the session
+ and waits for its outcome.
+
+Use :doc:`apache-airflow-providers-common-ai:index` instead when the AI step
should be run by
+Airflow itself and stay vendor-neutral:
+
+* Generation, classification, or structured extraction with ``LLMOperator`` —
it works with
+ Claude via a connection, and switching to another model provider later is a
connection
+ change, not a Dag rewrite.
+* Agents whose loop runs **in the Airflow worker** with ``AgentOperator`` —
Airflow-defined
+ toolsets (SQL, hooks, MCP servers), human-in-the-loop review, and durable
step replay.
+
+Both ``AgentOperator`` and ``AnthropicAgentSessionOperator`` run agents with
Claude models;
+the difference is where the loop executes and who controls the tools.
``AgentOperator``
+orchestrates the loop in the worker and calls tools you define in the Dag.
+``AnthropicAgentSessionOperator`` hands the whole session to Anthropic's
managed service —
+the agent runs autonomously against a pre-created agent and environment,
without
+Airflow-side tool execution.
+
+For example, running the Message Batches API directly:
+
+.. exampleinclude:: /../tests/system/anthropic/example_anthropic_batch.py
+ :language: python
+ :start-after: [START howto_operator_anthropic_batch]
+ :end-before: [END howto_operator_anthropic_batch]
.. toctree::
:hidden:
diff --git a/providers/common/ai/docs/index.rst
b/providers/common/ai/docs/index.rst
index db89a9a87e6..eb1f768de60 100644
--- a/providers/common/ai/docs/index.rst
+++ b/providers/common/ai/docs/index.rst
@@ -19,6 +19,46 @@
``apache-airflow-providers-common-ai``
##################################################
+When to use this provider
+--------------------------
+
+``common.ai`` is the vendor-neutral way to put LLM and agent steps in a Dag.
It is built on
+`pydantic-ai <https://ai.pydantic.dev/>`__, so the model vendor (OpenAI,
Anthropic, Google,
+Bedrock, …) is picked by the connection ``llm_conn_id`` points at — switching
providers later
+is a connection change, not a Dag rewrite. The AI step is orchestrated by
Airflow: the model
+calls, the agent loop, and any tools all run in the Airflow worker, where they
get retries,
+logging, and observability like any other task.
+
+Use it when a Dag needs:
+
+* **Generation, classification, summarization, or structured extraction** —
+ :doc:`LLMOperator and @task.llm <operators/llm>`, with Pydantic-typed output
pushed to XCom.
+* **Branching on a model's decision** — :doc:`LLMBranchOperator
<operators/llm_branch>`.
+* **Agents with tools** — :doc:`AgentOperator <operators/agent>` runs a
multi-turn agent loop
+ in the worker, calling Airflow-defined toolsets (SQL, hooks, MCP servers),
with optional
+ human-in-the-loop review and durable step replay.
+* **Document pipelines** — loading, file analysis, embeddings, and retrieval
for RAG
+ (see :doc:`operators/index`).
+
+Use a vendor's own provider instead when the Dag needs that vendor's **native
API surface** —
+a service the vendor runs for you, which no vendor-neutral operator wraps:
+
+* :doc:`apache-airflow-providers-openai:index` — the Embeddings, Responses,
and Batch APIs.
+* :doc:`apache-airflow-providers-anthropic:index` — the Claude Message Batches
API, and
+ Managed Agents sessions where the agent loop runs on Anthropic's
infrastructure rather
+ than in the Airflow worker.
+
+As a rule of thumb: if Airflow should *run* the AI step (and the model should
stay
+swappable), use ``common.ai``; if the Dag *submits work to* a vendor-managed
service and
+waits for the result, use that vendor's provider.
+
+For example, this ``LLMOperator`` call is unchanged whether ``llm_conn_id``
points at an
+OpenAI, Anthropic, or other pydantic-ai-supported connection:
+
+.. exampleinclude::
/../../ai/src/airflow/providers/common/ai/example_dags/example_llm.py
+ :language: python
+ :start-after: [START howto_operator_llm_basic]
+ :end-before: [END howto_operator_llm_basic]
.. toctree::
:hidden:
diff --git a/providers/openai/docs/index.rst b/providers/openai/docs/index.rst
index 7b2d212ea53..3f8dbfe12dc 100644
--- a/providers/openai/docs/index.rst
+++ b/providers/openai/docs/index.rst
@@ -19,6 +19,41 @@
``apache-airflow-providers-openai``
======================================
+When to use this provider
+--------------------------
+
+Use ``openai`` when a Dag needs OpenAI's native API surface — thin wrappers
over
+OpenAI-specific endpoints and options:
+
+* ``OpenAIEmbeddingOperator`` — call the Embeddings API directly, e.g. to feed
a vector
+ store.
+* ``OpenAIResponseOperator`` — call the
+ `Responses API <https://platform.openai.com/docs/api-reference/responses>`__
with
+ OpenAI-specific parameters.
+* ``OpenAITriggerBatchOperator`` and ``OpenAIHook`` — submit a
+ `Batch API <https://platform.openai.com/docs/guides/batch>`__ job for
asynchronous bulk
+ processing and wait for it to complete.
+
+Use :doc:`apache-airflow-providers-common-ai:index` instead when the AI step
should be run by
+Airflow itself and stay vendor-neutral:
+
+* Generation, classification, or structured extraction with ``LLMOperator`` —
it works with
+ OpenAI models via a connection, and switching to another model provider
later is a
+ connection change, not a Dag rewrite.
+* Agents whose loop runs in the Airflow worker with ``AgentOperator`` —
Airflow-defined
+ toolsets (SQL, hooks, MCP servers), human-in-the-loop review, and durable
step replay.
+* Document-to-vector-store pipelines with its document loader, embedding, and
retrieval
+ operators, which are not tied to OpenAI's embedding models.
+
+In short: pick ``openai`` to reach an OpenAI-only endpoint; pick ``common.ai``
to keep the
+Dag portable across model providers.
+
+For example, calling the Responses API directly:
+
+.. exampleinclude:: /../../openai/tests/system/openai/example_openai.py
+ :language: python
+ :start-after: [START howto_operator_openai_response]
+ :end-before: [END howto_operator_openai_response]
.. toctree::
:hidden:
