aarmens702-hub commented on code in PR #77073:
URL: https://github.com/apache/airflow/pull/77073#discussion_r4638755188
##########
providers/common/ai/src/airflow/providers/common/ai/operators/agent.py:
##########
@@ -824,8 +824,18 @@ def llm_hook(self) -> PydanticAIHook:
}
return PydanticAIHook.get_hook(self.llm_conn_id,
hook_params=hook_params)
- def _build_agent(self) -> Agent[object, Any]:
- """Build and return a pydantic-ai Agent from the operator's config."""
+ def build_agent(self) -> Agent[object, Any]:
Review Comment:
Done. Main's `_build_agent(self, *, approval_context=None)` is unchanged.
`execute()` still calls it with the task context. `build_agent()` takes no
arguments and returns `self._build_agent()`, so a subclass that overrides
`_build_agent` gets the same agent from both. The docstring says a durable
task's agent from `build_agent()` has no tool-approval step, and a new test
checks that `build_agent()` passes no approval context.
Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before
posting
##########
providers/common/ai/src/airflow/providers/common/ai/operators/llm.py:
##########
@@ -389,14 +389,28 @@ def _create_agent(self, *, output_type: Any,
instructions: str) -> Agent[object,
retries=self.output_retries,
)
+ def build_agent(self) -> Agent[object, Any]:
Review Comment:
Added: "A subclass that builds its agent differently should override this
and call it from `execute`, so both return the same agent."
Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before
posting
##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
The test above scripts the model's replies, so it can't tell you whether a
changed
prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model
and
-a library such as `pydantic-evals
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
Review Comment:
Added a sentence saying mapped tasks aren't covered: `get_task()` returns a
`MappedOperator` for them, which has no `build_agent()`.
Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before
posting
##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
The test above scripts the model's replies, so it can't tell you whether a
changed
prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model
and
-a library such as `pydantic-evals
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
+connection, system prompt, toolsets and capabilities, so the cases check the
task
+as written rather than a copy of it:
+
+.. code-block:: python
+
+ # evals/eval_orders_report.py
+ from pydantic_evals import Case, Dataset
+ from pydantic_evals.evaluators import Contains
+
+ from orders_report import orders_report # dags/orders_report.py from above
Review Comment:
Added the command: `PYTHONPATH=dags python evals/eval_orders_report.py`, run
from the project root.
Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before
posting
##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
The test above scripts the model's replies, so it can't tell you whether a
changed
prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model
and
-a library such as `pydantic-evals
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
+connection, system prompt, toolsets and capabilities, so the cases check the
task
+as written rather than a copy of it:
+
+.. code-block:: python
+
+ # evals/eval_orders_report.py
+ from pydantic_evals import Case, Dataset
+ from pydantic_evals.evaluators import Contains
+
+ from orders_report import orders_report # dags/orders_report.py from above
+
+ agent = orders_report().get_task("orders_question").build_agent()
+
+ dataset = Dataset(
+ name="orders_question",
+ cases=[Case(inputs="How many orders are there?",
evaluators=[Contains("3")])],
+ )
+ report = dataset.evaluate_sync(lambda question:
agent.run_sync(question).output)
+ report.print()
+
+``build_agent()`` runs outside a task, so templates aren't rendered and a
templated
Review Comment:
The docs and both docstrings now say every template field is used as written
outside a task run, naming `system_prompt`, `llm_conn_id` and `model_id`, and
that a templated `llm_conn_id` fails at connection lookup.
Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before
posting
##########
providers/common/ai/docs/local_development.rst:
##########
@@ -149,6 +149,31 @@ Check the answers themselves
The test above scripts the model's replies, so it can't tell you whether a
changed
prompt still gives good answers. For that, run a set of cases with expected
-outcomes against the notebook agent from the first section, using a real model
and
-a library such as `pydantic-evals
<https://pydantic.dev/docs/ai/evals/evals/>`__,
-before you ship the change.
+outcomes against a real model, using a library such as
+`pydantic-evals <https://pydantic.dev/docs/ai/evals/evals/>`__, before you
ship the
+change. A task's ``build_agent()`` returns the agent the task runs, with its
+connection, system prompt, toolsets and capabilities, so the cases check the
task
+as written rather than a copy of it:
+
+.. code-block:: python
+
+ # evals/eval_orders_report.py
+ from pydantic_evals import Case, Dataset
+ from pydantic_evals.evaluators import Contains
+
+ from orders_report import orders_report # dags/orders_report.py from above
+
+ agent = orders_report().get_task("orders_question").build_agent()
+
+ dataset = Dataset(
+ name="orders_question",
+ cases=[Case(inputs="How many orders are there?",
evaluators=[Contains("3")])],
+ )
+ report = dataset.evaluate_sync(lambda question:
agent.run_sync(question).output)
+ report.print()
+
+``build_agent()`` runs outside a task, so templates aren't rendered and a
templated
+``system_prompt`` reaches the agent as written. ``@task.llm``, ``LLMOperator``
and
+the other LLM operators have it too; for ``LLMSQLQueryOperator`` and
Review Comment:
The docs now list the six operators that have it, each with its decorator:
`AgentOperator`, `LLMOperator`, `LLMBranchOperator`, `LLMFileAnalysisOperator`,
`LLMSQLQueryOperator` and `LLMSchemaCompareOperator`. `LLMBatchOperator` sends
requests through the provider batch APIs and never builds a pydantic-ai agent,
so it has nothing to return.
Drafted-by: Claude Code (Opus 5.5); reviewed by @aarmens702-hub before
posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]