This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a change to branch
jason/lang-sdk-e2e/03c-coordinator-dag-parsing
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard b8301553fcd Explain dag_bundle_name by the coordinator that parses a
Dag file
discard 9945516128c Hand out a coordinator's Dag importer through
get_dag_importer alone
discard a6e64d49dd1 Say dag_bundle_name is only needed for two coordinators of
one class
discard c2a7345653a Drop serves_bundle so get_parsed_bundles is the only answer
discard a626c0b3a29 Check Dag file claims per bundle on the parse path
discard c1839dffaa8 Import the coordinator manager at the top of the importer
base
discard ef85b6fdbe8 Let a coordinator parse the Dag files of the bundles it
serves
discard a07d7ada8cc Sync the Go SDK copy of the Dag schema
discard ed14e4b7ff7 Sync the Java SDK copy of the Dag schema
discard 74701a99d52 Serialize the SDK Dag in tests that write a dag_maker Dag
discard d4b3980ef6c Note in the parsing ADR that runtimes may omit
config-backed fields
discard 18d47d0b083 Check SDK Dags with the Dag processor's validation in
conformance
discard f7094b86f30 Reject malformed task entries in a serialized Dag
discard 7b102103078 Fill a serialized Dag's unset settings from the Airflow
config
discard d20b1d13503 Check that a serialized Dag can be stored and loaded
add e3dd28607b9 Document team-scoped plugins in multi-team mode (#73995)
add b62299ced8a Cap microsoft-kiota-http below 1.13 in the Microsoft Azure
provider (#74090)
add d7b672f44e0 Update docs and diagrams for the Airflow 3 Dag processor
(#74073)
add ec5b621a9cc Prepare ad-hoc providers release 2026-10-02 (#74097)
add 7800fc44c07 Fix scheduler crash on restart with queued or running Edge
tasks (#73832)
add 49a52987956 Stop version-check hooks from clobbering the committing
worktree's index (#74115)
add d7b88f36197 UI: Fix external plugin nav links not navigating on click
(#73913)
add 66d751bfd22 Skip already-released providers in
update-providers-next-version (#74098)
add 04194b1de1f Fix flaky Kafka trigger tests by awaiting the event
instead of a fixed sleep (#74124)
add 4c0ba9e41da Make Edge executor adoption test immune to executor_loader
reloads (#74122)
add 7a8ce83f385 Go SDK: reject a task function whose last result type is
not error (#74077)
add 9c14681a4a0 Add create and update support to Snowflake Cortex Agent
hook (#70703)
add e029fcef2bb Bump the registry-security-updates group across 1
directory with 3 updates (#74099)
add da28f1fcac4 Bump the auth-ui-package-updates group across 1 directory
with 5 updates (#74101)
add 971e8db54ea Bump the edge-ui-package-updates group across 1 directory
with 12 updates (#74107)
add 536f535d9dd Bump the core-ui-package-updates group across 1 directory
with 12 updates (#74105)
add 95826e38547 Bump the uv-dependency-updates group in /dev/breeze with 3
updates (#74100)
add df76e2e0d5e Bump the ui-plugin-template-package-updates group across 1
directory with 7 updates (#74102)
add 954d21df92d Bump vitest from 4.1.7 to 5.0.1 in /ts-sdk (#73483)
add 0adf1378017 Bump swagger-ui-dist from 5.32.15 to 5.33.0 in /registry
(#73480)
add e37a39aa34e Bump pyjwt from 2.14.0 to 2.15.0 in /dev/breeze (#73969)
add da53196d812 Time-box and retry pre-extras SDK downloads (#71544)
add 232c264968f Deprecate Dataproc ClusterGenerator helper class (#71427)
add b9c52951378 Create GCS folder markers as directories in
GCSToSFTPOperator (#73880)
add 91353550246 Honor download_num_attempts and upload_num_attempts in
GCSTimeSpanFileTransformOperator (#72953)
add 02f2c852c80 Shrink the CI image by skipping unused Rust docs and the
Go archive (#74076)
add 5523443d98c Set end date and duration on tasks skipped by a Dag run
timeout (#74128)
add 0f3a356e768 Add a user setting for the default search match mode
(#74021)
add 12ee0862774 Carry task attempt UUIDs through remote executor providers
(#73917)
add 867327c48dd Run only the tests that import a changed test helper
(#74055)
add da5af33fd02 Create the Athena Spark work group and results bucket in
the system test (#73929)
add c011b5960ac Install ibmmq from a wheel in IBM MQ lowest-dependency
tests (#74132)
add 855a2ce92f9 Isolate Edge job state by task attempt UUID (#73918)
add 79de1d48039 Fix `refresh_from_task` not applied on retries (#65932)
add bf558f6f29a Go SDK: add dag.If with Then and Else so native Dags can
branch (#74075)
add d21496f7581 Fetch existing task instances in instead of rebuilding
them in tests (#74146)
add 11d25a587ac Run leaking provider tests with providers changed after
them in CI (#74123)
add a89208e5405 Bump the github-actions-updates group across 1 directory
with 5 updates (#74110)
add 4ae133cd807 Bump @chakra-ui/react from 3.36.1 to 3.37.0 in
/airflow-core/src/airflow/ui (#72524)
add 484a87b54bc Add get_async_extra_dejson to the common.compat connection
helpers (#74147)
add e4736b46968 Pass ClientTimeout to aiohttp in Cloud Composer async hook
(#74164)
add f070997ba63 Go SDK: order tasks with Before, After and Label (#74119)
add b5e58c96bb4 Check that a serialized Dag can be stored and loaded
add 5bd6db26da1 Fill a serialized Dag's unset settings from the Airflow
config
add 9ee09fe33c2 Reject malformed task entries in a serialized Dag
add b44ff24cdb1 Check SDK Dags with the Dag processor's validation in
conformance
add 6866e98d3b0 Note in the parsing ADR that runtimes may omit
config-backed fields
add 8659361cd63 Serialize the SDK Dag in tests that write a dag_maker Dag
add 340063d8a97 Sync the Java SDK copy of the Dag schema
add c3113a96e4f Sync the Go SDK copy of the Dag schema
add 962acf2c16b Check a serialized Dag as the SDK Dag is checked and
return it
add 25b751285b7 Let a coordinator parse the Dag files of the bundles it
serves
add 1627decb9c5 Import the coordinator manager at the top of the importer
base
add 580781113f1 Check Dag file claims per bundle on the parse path
add 0f9651cd1d0 Drop serves_bundle so get_parsed_bundles is the only answer
add 5c266f4ed45 Say dag_bundle_name is only needed for two coordinators of
one class
add 1116deb6088 Hand out a coordinator's Dag importer through
get_dag_importer alone
add e781bd36a9d Explain dag_bundle_name by the coordinator that parses a
Dag file
add 52ded1289ac Find a Dag importer's coordinator from its class
add def439999a4 Say dag_bundle_name only names the stub task artifact
bundle
add cf492837534 Describe how a Dag importer finds its coordinator in
ADR-0010
add 003bfdaf617 Add a newsfragment for the removed coordinator artifact
roots
add f611b22c5d2 Say which failures fall back to a Python file in ADR-0010
add b9fcd4587b5 Break the new dag_bundle_name docs only at sentence ends
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (b8301553fcd)
\
N -- N -- N
refs/heads/jason/lang-sdk-e2e/03c-coordinator-dag-parsing (b9fcd4587b5)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../prepare-providers-documentation/SKILL.md | 6 +-
.github/workflows/ci-notification.yml | 2 +-
.github/workflows/codeql-analysis.yml | 6 +-
.github/workflows/java-sdk-dependency-security.yml | 6 +-
.github/workflows/notify-uv-lock-conflicts.yml | 2 +-
.github/workflows/registry-tests.yml | 2 +-
.../scheduled-verify-release-calendar.yml | 2 +-
Dockerfile | 5 +-
Dockerfile.ci | 7 +-
.../adr/lang-sdk/0010-native-dag-processing.md | 107 +-
.../cluster-policies.rst | 11 +-
.../modules_management.rst | 2 +-
.../docs/administration-and-deployment/plugins.rst | 172 +-
.../authoring-and-scheduling/language-sdks/go.rst | 20 +-
.../language-sdks/java.rst | 18 +-
.../language-sdks/typescript.rst | 20 +-
airflow-core/docs/best-practices.rst | 10 +-
airflow-core/docs/core-concepts/dags.rst | 6 +-
airflow-core/docs/core-concepts/multi-team.rst | 16 +-
airflow-core/docs/core-concepts/overview.rst | 4 +-
.../img/diagram_basic_airflow_architecture.md5sum | 2 +-
.../img/diagram_basic_airflow_architecture.png | Bin 116194 -> 127853
bytes
.../docs/img/diagram_basic_airflow_architecture.py | 9 +-
...agram_dag_processor_airflow_architecture.md5sum | 2 +-
.../diagram_dag_processor_airflow_architecture.png | Bin 237657 -> 237572
bytes
.../diagram_dag_processor_airflow_architecture.py | 23 +-
...diagram_distributed_airflow_architecture.md5sum | 2 +-
.../diagram_distributed_airflow_architecture.png | Bin 192497 -> 215541
bytes
.../diagram_distributed_airflow_architecture.py | 20 +-
airflow-core/newsfragments/74042.significant.rst | 8 +
.../auth/managers/simple/ui/package.json | 10 +-
.../auth/managers/simple/ui/pnpm-lock.yaml | 253 +--
.../src/airflow/config_templates/config.yml | 22 +-
.../src/airflow/jobs/scheduler_job_runner.py | 3 +-
airflow-core/src/airflow/models/dagrun.py | 4 +
.../airflow/serialization/serialized_objects.py | 30 +-
airflow-core/src/airflow/ui/package.json | 26 +-
airflow-core/src/airflow/ui/pnpm-lock.yaml | 1891 +++++++++++---------
.../airflow/ui/public/i18n/locales/en/common.json | 7 +
.../src/airflow/ui/src/constants/localStorage.ts | 1 +
.../ui/src/hooks/useAdvancedSearch.test.tsx | 29 +-
.../src/airflow/ui/src/hooks/useAdvancedSearch.ts | 13 +-
.../airflow/ui/src/hooks/useUserSettings.test.tsx | 21 +
.../src/airflow/ui/src/hooks/useUserSettings.ts | 7 +
.../airflow/ui/src/layouts/Nav/NavButton.test.tsx | 14 +
.../src/airflow/ui/src/layouts/Nav/NavButton.tsx | 8 +-
.../ui/src/pages/Settings/Settings.test.tsx | 18 +
.../src/airflow/ui/src/pages/Settings/Settings.tsx | 16 +
.../tests/unit/always/test_project_structure.py | 1 -
airflow-core/tests/unit/jobs/test_scheduler_job.py | 74 +-
airflow-core/tests/unit/models/test_dagrun.py | 73 +-
.../tests/unit/models/test_taskinstance.py | 25 +-
.../unit/serialization/test_dag_serialization.py | 85 +-
contributing-docs/12_provider_distributions.rst | 14 +-
dev/README_RELEASE_PROVIDERS.md | 5 +-
dev/breeze/doc/ci/04_selective_checks.md | 33 +-
dev/breeze/src/airflow_breeze/utils/packages.py | 26 +
.../src/airflow_breeze/utils/selective_checks.py | 235 ++-
dev/breeze/tests/test_packages.py | 36 +
dev/breeze/tests/test_selective_checks.py | 343 +++-
dev/breeze/uv.lock | 44 +-
.../react_plugin_template/package.json | 14 +-
.../react_plugin_template/pnpm-lock.yaml | 369 ++--
go-sdk/README.md | 7 +-
go-sdk/airflow/bundle.go | 15 +-
go-sdk/airflow/bundle_test.go | 54 +
go-sdk/airflow/dag.go | 106 +-
go-sdk/airflow/dag_test.go | 5 +
go-sdk/airflow/if.go | 228 +++
go-sdk/airflow/if_test.go | 693 +++++++
go-sdk/airflow/inputs.go | 51 +-
go-sdk/airflow/node.go | 383 ++++
go-sdk/airflow/node_test.go | 538 ++++++
go-sdk/airflow/task_handler_test.go | 9 +
go-sdk/airflow/task_option.go | 12 +-
.../{foreigntaskoption => foreignnode}/main.go | 19 +-
go-sdk/internal/bundle/task.go | 127 +-
go-sdk/internal/bundle/task_test.go | 242 +++
go-sdk/pkg/execution/client.go | 8 +
go-sdk/pkg/execution/client_test.go | 47 +
go-sdk/pkg/execution/integration_test.go | 124 ++
go-sdk/pkg/execution/task_runner.go | 1 +
java-sdk/README.md | 6 +-
.../kubernetes_tests/test_kubernetes_executor.py | 89 +-
providers/.last_release_date.txt | 2 +-
providers/amazon/docs/executors/batch-executor.rst | 4 +
providers/amazon/docs/executors/ecs-executor.rst | 4 +
providers/amazon/docs/executors/general.rst | 13 +
.../amazon/docs/executors/lambda-executor.rst | 5 +
.../aws/executors/aws_lambda/lambda_executor.py | 67 +-
.../amazon/aws/executors/batch/batch_executor.py | 15 +-
.../amazon/aws/executors/ecs/ecs_executor.py | 5 +-
.../system/amazon/aws/example_athena_spark.py | 52 +-
.../executors/aws_lambda/test_lambda_executor.py | 150 +-
.../aws/executors/batch/test_batch_executor.py | 105 +-
.../tests/unit/amazon/aws/executors/conftest.py} | 41 +-
.../amazon/aws/executors/ecs/test_ecs_executor.py | 135 +-
.../apache/kafka/triggers/test_await_message.py | 49 +-
.../unit/apache/kafka/triggers/test_msg_queue.py | 5 +-
providers/celery/docs/celery_executor.rst | 8 +
.../providers/celery/executors/celery_executor.py | 30 +-
.../integration/celery/test_celery_executor.py | 30 +-
.../unit/celery/executors/test_celery_executor.py | 258 ++-
providers/cncf/kubernetes/docs/changelog.rst | 5 +
.../cncf/kubernetes/docs/kubernetes_executor.rst | 9 +
providers/cncf/kubernetes/provider.yaml | 2 +-
.../kubernetes/executors/kubernetes_executor.py | 123 +-
.../executors/kubernetes_executor_types.py | 43 +-
.../executors/kubernetes_executor_utils.py | 37 +-
.../executors/test_kubernetes_executor.py | 818 ++++++++-
.../executors/test_kubernetes_executor_types.py | 89 +
.../providers/common/compat/connection/__init__.py | 30 +-
.../common/compat/connection/test_connection.py | 58 +-
providers/duckdb/provider.yaml | 2 +-
providers/edge3/docs/changelog.rst | 8 +
providers/edge3/docs/migrations-ref.rst | 4 +-
providers/edge3/provider.yaml | 14 +-
.../src/airflow/providers/edge3/cli/api_client.py | 14 +-
.../src/airflow/providers/edge3/cli/worker.py | 22 +-
.../providers/edge3/executors/edge_executor.py | 214 ++-
.../airflow/providers/edge3/get_provider_info.py | 12 +-
.../0006_5_0_0_add_task_instance_id_to_edge_job.py | 71 +
.../edge3/src/airflow/providers/edge3/models/db.py | 1 +
.../src/airflow/providers/edge3/models/edge_job.py | 10 +
.../www/openapi-gen/requests/schemas.gen.ts | 12 +
.../plugins/www/openapi-gen/requests/types.gen.ts | 10 +
.../providers/edge3/plugins/www/package.json | 24 +-
.../providers/edge3/plugins/www/pnpm-lock.yaml | 817 ++++-----
.../edge3/plugins/www/src/pages/JobsPage.test.tsx | 105 ++
.../edge3/plugins/www/src/pages/JobsPage.tsx | 17 +-
.../providers/edge3/plugins/www/vite.config.ts | 4 +-
.../providers/edge3/worker_api/datamodels.py | 7 +
.../providers/edge3/worker_api/datamodels_ui.py | 2 +
.../providers/edge3/worker_api/routes/jobs.py | 84 +-
.../providers/edge3/worker_api/routes/ui.py | 1 +
.../edge3/worker_api/v2-edge-generated.yaml | 39 +
.../edge3/tests/unit/edge3/cli/test_api_client.py | 76 +-
.../edge3/tests/unit/edge3/cli/test_worker.py | 46 +-
.../unit/edge3/executors/test_edge_executor.py | 456 ++++-
.../migrations/test_task_instance_identity.py | 170 ++
providers/edge3/tests/unit/edge3/models/test_db.py | 30 +-
.../unit/edge3/worker_api/routes/test_jobs.py | 144 +-
.../tests/unit/edge3/worker_api/routes/test_ui.py | 31 +
providers/google/docs/changelog.rst | 9 +
providers/google/docs/operators/cloud/dataproc.rst | 14 +-
.../providers/google/cloud/hooks/cloud_composer.py | 4 +-
.../providers/google/cloud/operators/dataproc.py | 19 +-
.../providers/google/cloud/operators/gcs.py | 24 +-
.../google/cloud/transfers/gcs_to_sftp.py | 55 +-
.../dataproc/example_dataproc_batch_persistent.py | 32 +-
.../dataproc/example_dataproc_cluster_generator.py | 156 --
.../unit/google/cloud/hooks/test_cloud_composer.py | 24 +
.../unit/google/cloud/operators/test_dataproc.py | 7 +-
.../tests/unit/google/cloud/operators/test_gcs.py | 135 ++
.../google/cloud/transfers/test_gcs_to_sftp.py | 72 +
providers/ibm/mq/docs/index.rst | 2 +-
providers/ibm/mq/pre_extras_install.yaml | 33 -
providers/ibm/mq/pyproject.toml | 4 +-
providers/ibm/mq/tests/conftest.py | 8 +-
providers/microsoft/azure/README.rst | 2 +-
providers/microsoft/azure/docs/index.rst | 2 +-
providers/microsoft/azure/pyproject.toml | 7 +-
.../snowflake/hooks/snowflake_cortex_agent.py | 176 ++
.../snowflake/hooks/test_snowflake_cortex_agent.py | 211 +++
registry/package.json | 2 +-
registry/pnpm-lock.yaml | 44 +-
scripts/ci/prek/check_execution_api_versions.py | 15 +-
.../ci/prek/check_supervisor_schemas_versions.py | 11 +-
scripts/ci/prek/common_prek_utils.py | 21 +
scripts/docker/entrypoint_ci.sh | 2 +-
scripts/docker/install_os_dependencies.sh | 5 +-
scripts/in_container/run_pre_extras_install.py | 64 +-
.../in_container/test_run_pre_extras_install.py | 142 ++
.../src/airflow/sdk/coordinators/_dag_importer.py | 142 ++
.../src/airflow/sdk/execution_time/coordinator.py | 173 +-
task-sdk/src/airflow/sdk/importers/base.py | 33 +-
.../task_sdk/coordinators/test_dag_importer.py | 204 +++
.../task_sdk/execution_time/test_coordinator.py | 384 ++--
task-sdk/tests/task_sdk/importers/test_registry.py | 116 +-
ts-sdk/README.md | 8 +-
ts-sdk/example/README.md | 6 +-
ts-sdk/package.json | 2 +-
ts-sdk/pnpm-lock.yaml | 413 +----
uv.lock | 10 +-
184 files changed, 11161 insertions(+), 3275 deletions(-)
create mode 100644 airflow-core/newsfragments/74042.significant.rst
create mode 100644 go-sdk/airflow/if.go
create mode 100644 go-sdk/airflow/if_test.go
create mode 100644 go-sdk/airflow/node.go
create mode 100644 go-sdk/airflow/node_test.go
copy go-sdk/airflow/testdata/{foreigntaskoption => foreignnode}/main.go (64%)
copy providers/{keycloak/tests/unit/keycloak/auth_manager/test_user.py =>
amazon/tests/unit/amazon/aws/executors/conftest.py} (56%)
create mode 100644
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor_types.py
create mode 100644
providers/edge3/src/airflow/providers/edge3/migrations/versions/0006_5_0_0_add_task_instance_id_to_edge_job.py
create mode 100644
providers/edge3/src/airflow/providers/edge3/plugins/www/src/pages/JobsPage.test.tsx
create mode 100644
providers/edge3/tests/unit/edge3/migrations/test_task_instance_identity.py
delete mode 100644
providers/google/tests/system/google/cloud/dataproc/example_dataproc_cluster_generator.py
delete mode 100644 providers/ibm/mq/pre_extras_install.yaml
create mode 100644 scripts/tests/in_container/test_run_pre_extras_install.py
create mode 100644 task-sdk/src/airflow/sdk/coordinators/_dag_importer.py
create mode 100644 task-sdk/tests/task_sdk/coordinators/test_dag_importer.py
