This is an automated email from the ASF dual-hosted git repository.
kaxil pushed a change to branch durable-default-on
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard 5b96a82804b Turn on durable execution by default for agent tasks that
can retry
discard 1e984e45d4e Fix durable agent runs replaying stale steps after a
first-attempt pause
discard 452231895c5 Let durable agent tasks use HITL review
discard dd1ad5a45ec Merge branch 'hitl-review-park' into durable-hitl-review
discard 3846c9e7bc8 Let durable agent tasks pause for tool approval
discard 67227e29bd0 Rename the durable callable toolset id helper and tidy
step naming
discard 08e26d8c7b2 Let durable agent tasks use toolsets without an id
discard 7dbe824be54 Name the pydantic-ai-slim release the system-one decision
model prefix needs
discard c03541da74b Let durable agent tasks run chosen tools on every attempt
instead of replaying them
discard a654a0d895f Clarify which durable journal store calls apply before
Airflow 3.3
discard 87fbfb41b4a Clarify that a durable journal is deleted when the agent
run succeeds
discard 7d9637d1361 Reduce task state store calls made by durable agent tasks
discard 2db175294f2 Fix durable agent retries replaying the same failing tool
call on every attempt
add ba57629ded2 fix the command for contributors to verify python client
and airflowctl rc (#76870)
add f5a994353e4 Copy the stack trace from the Dag import error modals
(#76924)
add 17e7eed7cf4 Bump @chakra-ui/react (#76754)
add 961da0f3b10 UI: Complete the missing Hebrew (he) translations (#76562)
add fc65819a7b2 docs(CODEOWNERS): add Indonesian UI translation owner
(#76929)
add c1ad31f66c3 UI: Move most lint rules from ESLint to oxlint (#76491)
add 646b92b8b9b Re-anchor custom queued deadline references when a Dag run
is cleared (#71850)
add 1f407b62dcb Restore clearing `ExternalTaskMarker` targets in other
Dags (#75874)
add 4a059ea539b Stop ObjectStorage journal write failures from failing
durable tasks (#75916)
add 7b163446c17 Link asset events emitted in a task loop to their
iteration (#76840)
add 29096e46255 Speed up PROD image stashes and restores with zstd (#76279)
add 83b140fbcc6 Apply Gantt date filters to mapped tasks and task groups
(#75725)
add 37a7713bde4 Fix outdated Python and Kubernetes versions in README
table (#76936)
add 34c96509ebc Move agent-assisted contribution docs out of the main
README (#76951)
add 3647e3c7e17 i18n(ko): add missing translaction in common.json (Nov 11)
(#76920)
add b4d1f1c22be Support durable execution in SageMakerTrainingOperator
(#76036)
add 54970db82f1 Reject reserved loop keys when deleting an XCom (#76892)
add b2d58334144 Regenerate skill-eval hash after AGENTS.md link update
(#76960)
add 426357062fb Prevent Databricks task failure after successful repair
(#76904)
add 9a02819b4e4 Fix AwsEksExecutor dying with a 401 after its credentials
rotate (#76332)
add 2448be40082 Make common.ai tool_prefix and output_type signatures
consistent (#74695)
add c43ff0fc1dd Make durable flag reach default_args in
PowerBIDatasetRefreshOperator (#76793)
add 509124275f4 Clamp negative offset in mapped XCom slice queries (#75015)
add 020428773e7 Remove duplicate GroupCommandParser type from airflowctl
CLI (#76345)
add 30f51ff2d4b Apply the configured XCom backend to language SDK task
XComs (#75184)
add e024d4c15a5 Fix TimeSensor test failing when host time zone is ahead
of UTC (#76901)
add b52246805e8 Fix durable agent retries replaying the same failing tool
call on every attempt
add 77aa7697e56 Split durable run failure handling into named helpers
add 24e4aa44f10 Fix durable agent runs asking the model again after a
retry already succeeded
add c7243994708 Reduce task state store calls made by durable agent tasks
add 7a5694cc355 Clarify that a durable journal is deleted when the agent
run succeeds
add 19e83835d4f Clarify which durable journal store calls apply before
Airflow 3.3
add c4eb1198200 Let durable agent tasks run chosen tools on every attempt
instead of replaying them
add f2d1cabf0dc Name the pydantic-ai-slim release the system-one decision
model prefix needs
add be8340156d3 Fail durable agent tasks once on invalid tool opt-out
metadata
add 5417708f87c Let durable agent tasks use toolsets without an id
add 7c4fd0ee02a Rename the durable callable toolset id helper and tidy
step naming
add 79212d5d22e Keep generated durable toolset ids out of pydantic-ai's
reserved format
add d46a7aa624e Let durable agent tasks pause for tool approval
add 997746bc0ca Merge branch 'hitl-review-park' into durable-hitl-review
add 2ca7b68d7ac Let durable agent tasks use HITL review
add 3c64bdd4024 Fix durable agent runs replaying stale steps after a
first-attempt pause
add 589ed14109e Turn on durable execution by default for agent tasks that
can retry
add 80f8e53baaf Delete a stale tool approval transcript on every fresh
agent run
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (5b96a82804b)
\
N -- N -- N refs/heads/durable-default-on (80f8e53baaf)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/CODEOWNERS | 1 +
.github/dependabot.yml | 1 +
.github/workflows/additional-prod-image-tests.yml | 19 +
.github/workflows/airflow-e2e-tests.yml | 28 +-
.github/workflows/ci-amd.yml | 2 +
.github/workflows/ci-arm.yml | 2 +
.github/workflows/prod-image-build.yml | 10 +-
.../scheduled-legacy-prod-image-canary.yml | 2 +
.rat-excludes | 2 +
AGENTS.md | 4 +-
README.md | 60 +-
airflow-core/.pre-commit-config.yaml | 2 +-
.../language-sdks/index.rst | 8 +-
airflow-core/docs/core-concepts/xcoms.rst | 2 +
airflow-core/newsfragments/71850.bugfix.rst | 1 +
airflow-core/newsfragments/75874.significant.rst | 17 +
.../src/airflow/api_fastapi/common/db/dag_runs.py | 9 +
.../src/airflow/api_fastapi/common}/xcom.py | 27 +-
.../api_fastapi/core_api/datamodels/assets.py | 8 +
.../core_api/datamodels/task_instances.py | 6 +
.../api_fastapi/core_api/openapi/_private_ui.yaml | 72 +
.../core_api/openapi/v2-rest-api-generated.yaml | 23 +-
.../api_fastapi/core_api/routes/public/assets.py | 4 +-
.../api_fastapi/core_api/routes/public/dag_run.py | 4 +-
.../core_api/routes/public/task_instances.py | 46 +-
.../api_fastapi/core_api/routes/public/xcom.py | 29 +-
.../api_fastapi/core_api/routes/ui/gantt.py | 14 +-
.../airflow/api_fastapi/core_api/routes/ui/grid.py | 81 +-
.../api_fastapi/core_api/services/public/assets.py | 12 +-
.../core_api/services/public/task_instances.py | 341 ++++-
.../execution_api/datamodels/taskinstance.py | 12 +
.../execution_api/routes/task_instances.py | 30 +-
.../api_fastapi/execution_api/routes/xcoms.py | 13 +-
.../api_fastapi/execution_api/versions/__init__.py | 2 +
.../execution_api/versions/v2027_02_28.py | 12 +-
airflow-core/src/airflow/models/deadline.py | 9 +
airflow-core/src/airflow/models/serialized_dag.py | 44 +-
airflow-core/src/airflow/models/taskinstance.py | 38 +-
airflow-core/src/airflow/models/xcom.py | 7 +
.../serialization/definitions/baseoperator.py | 6 +
.../src/airflow/serialization/definitions/dag.py | 25 +-
.../airflow/serialization/definitions/deadline.py | 12 +-
.../serialization/definitions/mappedoperator.py | 8 +
airflow-core/src/airflow/serialization/encoders.py | 5 +
.../airflow/serialization/serialized_objects.py | 12 +-
airflow-core/src/airflow/ui/.oxlintrc.json | 611 ++++++++
airflow-core/src/airflow/ui/eslint.config.js | 18 +-
.../src/airflow/ui/openapi-gen/queries/common.ts | 12 +-
.../ui/openapi-gen/queries/ensureQueryData.ts | 22 +-
.../src/airflow/ui/openapi-gen/queries/prefetch.ts | 22 +-
.../src/airflow/ui/openapi-gen/queries/queries.ts | 22 +-
.../src/airflow/ui/openapi-gen/queries/suspense.ts | 22 +-
.../airflow/ui/openapi-gen/requests/schemas.gen.ts | 35 +
.../ui/openapi-gen/requests/services.gen.ts | 20 +-
.../airflow/ui/openapi-gen/requests/types.gen.ts | 16 +-
airflow-core/src/airflow/ui/package.json | 8 +-
airflow-core/src/airflow/ui/pnpm-lock.yaml | 1618 +++++++++++---------
.../airflow/ui/public/i18n/locales/en/dags.json | 1 +
.../airflow/ui/public/i18n/locales/he/admin.json | 2 +
.../airflow/ui/public/i18n/locales/he/browse.json | 2 +
.../airflow/ui/public/i18n/locales/he/common.json | 15 +-
.../ui/public/i18n/locales/he/components.json | 11 +
.../src/airflow/ui/public/i18n/locales/he/dag.json | 36 +-
.../airflow/ui/public/i18n/locales/he/dags.json | 2 +
.../ui/public/i18n/locales/he/dashboard.json | 1 +
.../airflow/ui/public/i18n/locales/he/hitl.json | 1 +
.../airflow/ui/public/i18n/locales/ko/common.json | 2 +
airflow-core/src/airflow/ui/rules/core.js | 19 -
airflow-core/src/airflow/ui/rules/i18next.js | 68 -
airflow-core/src/airflow/ui/rules/perfectionist.js | 143 --
.../src/airflow/ui/rules/plugin-patch.d.ts | 4 -
airflow-core/src/airflow/ui/rules/stylistic.js | 90 --
.../components/ActionAccordion/ActionAccordion.tsx | 9 +-
.../ui/src/components/ActionAccordion/columns.tsx | 18 +-
.../ui/src/components/Assets/AssetEvent.test.tsx | 33 +
.../ui/src/components/Assets/AssetEvent.tsx | 11 +-
.../TaskInstance/ClearGroupTaskInstanceDialog.tsx | 11 +
.../ClearTaskInstanceDialog.dependentDags.test.tsx | 137 ++
.../Clear/TaskInstance/ClearTaskInstanceDialog.tsx | 11 +
.../ui/src/components/DagImportErrorDetails.tsx | 2 +-
.../ui/src/components/DagImportErrorModal.test.tsx | 15 +-
.../airflow/ui/src/components/GraphTaskFilters.tsx | 2 +-
.../ui/src/components/ReactMarkdownBlocks.tsx | 4 +-
.../ui/src/layouts/Details/Gantt/Gantt.test.tsx | 19 +-
.../airflow/ui/src/layouts/Details/Gantt/Gantt.tsx | 4 +
.../src/airflow/ui/src/layouts/Nav/Nav.tsx | 3 +-
.../airflow/ui/src/pages/Dag/Overview/Overview.tsx | 1 +
.../Dashboard/Stats/DagImportErrorsModal.test.tsx | 59 +
.../pages/Dashboard/Stats/DagImportErrorsModal.tsx | 2 +-
.../ui/src/pages/Task/Overview/Overview.tsx | 6 +-
.../ui/src/queries/useGridTISummaries.test.tsx | 72 +
.../airflow/ui/src/queries/useGridTISummaries.ts | 25 +-
airflow-core/src/airflow/ui/src/utils/links.ts | 16 +-
.../core_api/routes/public/test_assets.py | 61 +-
.../core_api/routes/public/test_dag_run.py | 2 +
.../core_api/routes/public/test_task_instances.py | 169 +-
.../core_api/routes/public/test_xcom.py | 21 +-
.../api_fastapi/core_api/routes/ui/test_grid.py | 245 +++
.../services/public/test_task_instances.py | 361 ++++-
.../versions/head/test_task_instances.py | 57 +-
.../execution_api/versions/head/test_xcoms.py | 45 +-
airflow-core/tests/unit/models/test_deadline.py | 13 +-
.../tests/unit/models/test_deadline_alert.py | 57 +-
.../tests/unit/models/test_serialized_dag.py | 64 +-
.../tests/unit/models/test_taskinstance.py | 167 +-
.../src/airflowctl/api/datamodels/generated.py | 9 +
airflow-ctl/src/airflowctl/ctl/cli_config.py | 25 +-
airflow-ctl/src/airflowctl/ctl/cli_parser.py | 18 +-
...ml-dag.yml => lang-sdk-xcom-object-storage.yml} | 24 +-
.../tests/airflow_e2e_tests/conftest.py | 178 ++-
.../tests/airflow_e2e_tests/constants.py | 5 +
.../__init__.py | 0
.../test_lang_sdk_xcom_backend.py | 291 ++++
contributing-docs/15_node_environment_setup.rst | 5 +-
contributing-docs/25_maintainer_pr_triage.md | 2 +-
contributing-docs/30_new_language_sdk.rst | 5 +
.../33_agent_assisted_contribution.md | 59 +
contributing-docs/README.rst | 3 +
dev/README_RELEASE_AIRFLOWCTL.md | 14 +-
dev/README_RELEASE_PYTHON_CLIENT.md | 14 +-
dev/breeze/doc/ci/04_selective_checks.md | 11 +
.../images/output_testing_airflow-e2e-tests.svg | 32 +-
.../images/output_testing_airflow-e2e-tests.txt | 2 +-
.../airflow_breeze/commands/testing_commands.py | 1 +
.../src/airflow_breeze/utils/selective_checks.py | 17 +
.../src/airflow_breeze/utils/verification_plan.py | 1 +
dev/breeze/tests/test_selective_checks.py | 64 +
dev/skill-evals/last-eval-hash.txt | 2 +-
dev/verify_airflow_ctl_rc.sh | 16 +-
dev/verify_python_client_rc.sh | 17 +-
generated/PYPI_README.md | 18 +-
go-sdk/README.md | 1 +
go-sdk/capabilities.yaml | 5 +
go-sdk/pkg/execution/genmodels/models.gen.go | 83 +-
go-sdk/schema/supervisor-schema.json | 48 +-
java-sdk/README.md | 3 +-
java-sdk/capabilities.yaml | 9 +-
java-sdk/gradle.properties | 2 +-
java-sdk/sdk/module.md | 3 +-
java-sdk/sdk/schema/schema.json | 168 +-
providers/amazon/docs/changelog.rst | 9 +
providers/amazon/docs/executors/eks-executor.rst | 5 +
.../amazon/aws/executors/eks/_client_factory.py | 94 +-
.../providers/amazon/aws/hooks/sagemaker.py | 36 +-
.../providers/amazon/aws/operators/sagemaker.py | 188 ++-
.../aws/executors/eks/test__client_factory.py | 232 ++-
.../aws/operators/test_sagemaker_training.py | 228 ++-
providers/common/ai/docs/changelog.rst | 36 +
providers/common/ai/docs/concepts.rst | 2 +-
providers/common/ai/docs/durable_execution.rst | 10 +-
providers/common/ai/docs/operators/agent.rst | 13 +-
providers/common/ai/docs/operators/llm.rst | 1 +
.../common/ai/docs/operators/llm_file_analysis.rst | 13 +-
providers/common/ai/docs/stability.rst | 11 +-
providers/common/ai/docs/structured_output.rst | 48 +-
providers/common/ai/docs/toolsets/datafusion.rst | 3 +
providers/common/ai/docs/toolsets/mcp.rst | 4 +-
providers/common/ai/docs/toolsets/sql.rst | 10 +-
.../airflow/providers/common/ai/decorators/llm.py | 3 +-
.../providers/common/ai/durable/capability.py | 48 +-
.../airflow/providers/common/ai/durable/journal.py | 97 +-
.../airflow/providers/common/ai/durable/storage.py | 28 +-
.../src/airflow/providers/common/ai/exceptions.py | 9 +
.../src/airflow/providers/common/ai/hooks/mcp.py | 4 +-
.../airflow/providers/common/ai/operators/agent.py | 108 +-
.../airflow/providers/common/ai/operators/llm.py | 37 +-
.../common/ai/operators/llm_file_analysis.py | 10 +-
.../providers/common/ai/toolsets/datafusion.py | 33 +-
.../airflow/providers/common/ai/toolsets/hook.py | 6 +-
.../airflow/providers/common/ai/toolsets/mcp.py | 6 +-
.../providers/common/ai/toolsets/object_storage.py | 17 +-
.../providers/common/ai/toolsets/sandbox.py | 24 +-
.../airflow/providers/common/ai/toolsets/sql.py | 35 +-
.../providers/common/ai/utils/output_type.py | 81 +-
.../providers/common/ai/utils/query_results.py | 20 +-
.../providers/common/ai/utils/tool_prefix.py | 56 +
.../unit/common/ai/durable/test_capability.py | 138 +-
.../tests/unit/common/ai/durable/test_journal.py | 51 +-
.../tests/unit/common/ai/durable/test_storage.py | 73 +
.../ai/tests/unit/common/ai/hooks/test_mcp.py | 18 +
.../tests/unit/common/ai/operators/test_agent.py | 82 +-
.../ai/operators/test_agent_tool_approval.py | 27 +-
.../ai/tests/unit/common/ai/operators/test_llm.py | 78 +-
.../common/ai/operators/test_llm_file_analysis.py | 28 +
.../unit/common/ai/toolsets/test_datafusion.py | 90 ++
.../ai/tests/unit/common/ai/toolsets/test_mcp.py | 9 +-
.../unit/common/ai/toolsets/test_object_storage.py | 13 +-
.../tests/unit/common/ai/toolsets/test_sandbox.py | 3 +
.../ai/tests/unit/common/ai/toolsets/test_sql.py | 114 ++
.../tests/unit/common/ai/utils/test_output_type.py | 343 ++++-
.../unit/common/ai/utils/test_query_results.py | 27 +
.../tests/unit/common/ai/utils/test_tool_prefix.py | 78 +
.../providers/databricks/operators/databricks.py | 1 +
.../unit/databricks/operators/test_databricks.py | 16 +
.../providers/microsoft/azure/operators/powerbi.py | 5 +
.../unit/microsoft/azure/operators/test_powerbi.py | 34 +-
.../providers/standard/sensors/external_task.py | 90 +-
.../unit/standard/sensors/test_external_task.py | 169 +-
.../tests/unit/standard/sensors/test_time.py | 2 +-
scripts/ci/prek/lang_sdk_compat_matrix.py | 1 +
scripts/ci/prek/ts_compile_lint_ui.py | 1 +
task-sdk/docs/api.rst | 10 +
task-sdk/src/airflow/sdk/__init__.py | 3 +
task-sdk/src/airflow/sdk/__init__.pyi | 2 +
task-sdk/src/airflow/sdk/api/client.py | 3 +
.../src/airflow/sdk/api/datamodels/_generated.py | 15 +
task-sdk/src/airflow/sdk/bases/xcom.py | 50 +-
.../src/airflow/sdk/coordinators/_subprocess.py | 351 ++++-
.../src/airflow/sdk/coordinators/_xcom_backend.py | 357 +++++
task-sdk/src/airflow/sdk/definitions/deadline.py | 15 +
.../src/airflow/sdk/definitions/external_clear.py | 96 ++
.../src/airflow/sdk/definitions/mappedoperator.py | 9 +-
task-sdk/src/airflow/sdk/exceptions.py | 1 +
task-sdk/src/airflow/sdk/execution_time/comms.py | 19 +
.../airflow/sdk/execution_time/schema/schema.json | 48 +-
.../sdk/execution_time/schema/versions/__init__.py | 4 +
.../execution_time/schema/versions/v2027_02_28.py | 44 +-
.../src/airflow/sdk/execution_time/supervisor.py | 27 +-
.../src/airflow/sdk/execution_time/task_runner.py | 47 +-
task-sdk/tests/task_sdk/api/test_client.py | 27 +
task-sdk/tests/task_sdk/bases/test_xcom.py | 17 +-
.../tests/task_sdk/coordinators/test_subprocess.py | 710 ++++++++-
.../task_sdk/coordinators/test_xcom_backend.py | 464 ++++++
.../task_sdk/definitions/test_mappedoperator.py | 37 +
.../execution_time/schema/test_migrator.py | 26 +
.../tests/task_sdk/execution_time/test_comms.py | 13 +
.../task_sdk/execution_time/test_supervisor.py | 21 +-
.../task_sdk/execution_time/test_task_runner.py | 94 ++
ts-sdk/README.md | 1 +
ts-sdk/capabilities.yaml | 5 +
ts-sdk/schema/supervisor-schema.json | 48 +-
ts-sdk/src/generated/supervisor.ts | 131 +-
232 files changed, 10447 insertions(+), 2076 deletions(-)
create mode 100644 airflow-core/newsfragments/71850.bugfix.rst
create mode 100644 airflow-core/newsfragments/75874.significant.rst
copy {task-sdk/src/airflow/sdk/execution_time =>
airflow-core/src/airflow/api_fastapi/common}/xcom.py (59%)
create mode 100644 airflow-core/src/airflow/ui/.oxlintrc.json
delete mode 100644 airflow-core/src/airflow/ui/rules/i18next.js
delete mode 100644 airflow-core/src/airflow/ui/rules/perfectionist.js
delete mode 100644 airflow-core/src/airflow/ui/rules/stylistic.js
create mode 100644
airflow-core/src/airflow/ui/src/components/Clear/TaskInstance/ClearTaskInstanceDialog.dependentDags.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/pages/Dashboard/Stats/DagImportErrorsModal.test.tsx
copy airflow-e2e-tests/docker/{yaml-dag.yml =>
lang-sdk-xcom-object-storage.yml} (52%)
copy {airflow-core/src/airflow/_shared =>
airflow-e2e-tests/tests/airflow_e2e_tests/lang_sdk_xcom_object_storage_tests}/__init__.py
(100%)
create mode 100644
airflow-e2e-tests/tests/airflow_e2e_tests/lang_sdk_xcom_object_storage_tests/test_lang_sdk_xcom_backend.py
create mode 100644 contributing-docs/33_agent_assisted_contribution.md
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/utils/tool_prefix.py
create mode 100644
providers/common/ai/tests/unit/common/ai/utils/test_tool_prefix.py
create mode 100644 task-sdk/src/airflow/sdk/coordinators/_xcom_backend.py
create mode 100644 task-sdk/src/airflow/sdk/definitions/external_clear.py
create mode 100644 task-sdk/tests/task_sdk/coordinators/test_xcom_backend.py
