This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a change to branch ci-upgrade-main
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit a36c605e2eb [main] CI: Upgrade important CI environment
add f5234c744a1 Harden OpenLineage task outcomes system test (#76595)
add 82b33126d3a Chart: Default airflow version to 3.4.0 (#76601)
add c2ff56eff6b UI: Complete the Portuguese translation (#76412)
add a0924b53f86 Mark Airflow 3.4.0 as the latest released version (#76602)
add f396a3b2271 Add Cloud SQL Auth Proxy v2 support to CloudSqlProxyRunner
(#73646)
add a7b8dc96928 Give the SMTP emailer its own defaults for the [smtp]
options (#76346)
add 498c1cd470c Fix CeleryExecutor failing to load with Celery 5.7 (#76591)
add c8e01650d56 Add a regression test for WITH in the asset orphanage
DELETE (#75127)
add 887f6d2b559 Keep Dag parse results tied to their source snapshot
(#74789)
add dac96bcf61c Publish Dag results and inventories through the Execution
API (#74790)
add f2edf564af4 Recover Dag processor work and expire inactive processors
(#74791)
add 2c69a21eb65 Add HTTP publication to the Dag processor manager (#74793)
add f01f7ef65d6 Make the Variable value non-nullable in the Execution API
(#76007)
add 335fd71f58f Stop AWS Glue job run when a deferred task is cleared
(#73702)
add abc3a20f080 Escape server and user text in airflowctl command messages
(#76265)
add 2cc7bb0c702 Fix scheduler crash loop from large Edge deadline
callbacks (#74445)
add 6ae6994206a Stop deferred AWS tasks failing on a botocore Config
object (#74843)
add ea5cd982f6b Bump the registry-package-updates group across 1 directory
with 2 updates (#76608)
add 0b9bf0cd855 Stream remote task logs from S3 instead of reading whole
objects (#73517)
add 3e076e8a64e Edge: Return 400 instead of 500 on invalid worker version
(#72925)
add 92b4e57df13 Bump oauthlib from 4.0.0 to 4.0.1 in /dev/breeze (#76569)
add 4d54940ceef Fix PgBouncer NetworkPolicy using namespace labels of
disabled triggerer KEDA (#74565)
add 81b487227dc Use airflowHome for logs mount in wait-for-migrations init
container (#75060)
add a694dc8281f Honor API server worker timeout configuration (#72859)
add 696d09de527 Add annotations to helm chart priorityclasses (#74457)
add a2d88223bc8 Fix providers release guide steps that use wrong dates or
labels (#76625)
add 2a1961ac68c Allow extra init containers for PgBouncer (#74612)
add 553c41633ed Fix Deadline Alert SmtpNotifier example in Dag docs
(#75597)
add 46321fb722f UI: Reduce duplicate Grid polling (#75979)
add e5ce464abab Fix S3 remote log test for streaming responses (#76632)
add 05d86b21c28 Post the first fork CI comment without waiting for the
next run (#76510)
add aee5b913b5e [main] CI: Upgrade important CI environment
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a36c605e2eb)
\
N -- N -- N refs/heads/ci-upgrade-main (aee5b913b5e)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.agents/skills/airflow-translations/locales/pt.md | 26 +
.../prepare-providers-documentation/SKILL.md | 5 +-
.pre-commit-config.yaml | 4 +-
Dockerfile | 2 +-
README.md | 12 +-
RELEASE_NOTES.rst | 1099 ++++++-
.../administration-and-deployment/dag-bundles.rst | 43 +
.../administration-and-deployment/listeners.rst | 13 +
airflow-core/docs/core-concepts/dags.rst | 23 +-
.../docs/installation/supported-versions.rst | 2 +-
airflow-core/docs/migrations-ref.rst | 6 +-
airflow-core/newsfragments/74789.significant.rst | 12 +
airflow-core/newsfragments/76007.significant.rst | 22 +
.../core_api/routes/public/dag_parsing.py | 18 +
.../execution_api/datamodels/dag_parsing.py | 201 ++
.../api_fastapi/execution_api/datamodels/job.py | 4 +-
.../execution_api/datamodels/variable.py | 4 +-
.../api_fastapi/execution_api/routes/__init__.py | 2 +
.../execution_api/routes/dag_parsing.py | 630 ++++
.../api_fastapi/execution_api/routes/jobs.py | 22 +-
.../api_fastapi/execution_api/versions/__init__.py | 8 +-
.../execution_api/versions/v2027_02_28.py | 40 +-
airflow-core/src/airflow/cli/cli_config.py | 2 +-
.../src/airflow/config_templates/config.yml | 21 +-
.../src/airflow/dag_processing/api_client.py | 115 +-
.../src/airflow/dag_processing/bundles/base.py | 21 +-
.../src/airflow/dag_processing/bundles/local.py | 1 +
.../src/airflow/dag_processing/bundles/manager.py | 30 +-
.../src/airflow/dag_processing/claims.py | 24 +-
.../src/airflow/dag_processing/collection.py | 282 +-
.../src/airflow/dag_processing/maintenance.py | 155 +
airflow-core/src/airflow/dag_processing/manager.py | 1310 ++++++--
.../src/airflow/dag_processing/processor.py | 77 +-
.../src/airflow/dag_processing/requested_work.py | 139 +
.../src/airflow/dag_processing/timeouts.py | 22 +-
.../src/airflow/jobs/scheduler_job_runner.py | 4 +
.../src/airflow/listeners/spec/importerrors.py | 20 +-
...55_3_5_0_add_dag_processor_publication_state.py | 78 +
.../0156_3_5_0_add_processor_work_claims.py | 74 +
airflow-core/src/airflow/models/__init__.py | 2 +
airflow-core/src/airflow/models/callback.py | 4 +
airflow-core/src/airflow/models/dag.py | 1 +
...log.py => dag_processor_publication_receipt.py} | 31 +-
...dag_favorite.py => dag_processor_work_claim.py} | 23 +-
airflow-core/src/airflow/models/dagbag.py | 16 +-
airflow-core/src/airflow/models/dagbundle.py | 3 +
airflow-core/src/airflow/models/dagcode.py | 2 +-
airflow-core/src/airflow/models/dagwarning.py | 30 +-
airflow-core/src/airflow/models/errors.py | 4 +-
.../src/airflow/serialization/definitions/dag.py | 2 +
.../airflow/serialization/serialized_objects.py | 18 +-
.../airflow/ui/public/i18n/locales/pt/admin.json | 33 +-
.../airflow/ui/public/i18n/locales/pt/assets.json | 39 +-
.../airflow/ui/public/i18n/locales/pt/browse.json | 60 +
.../airflow/ui/public/i18n/locales/pt/common.json | 314 +-
.../ui/public/i18n/locales/pt/components.json | 92 +-
.../src/airflow/ui/public/i18n/locales/pt/dag.json | 236 +-
.../airflow/ui/public/i18n/locales/pt/dags.json | 65 +-
.../ui/public/i18n/locales/pt/dashboard.json | 44 +-
.../airflow/ui/public/i18n/locales/pt/hitl.json | 15 +
.../ui/src/layouts/Details/DetailsLayout.test.tsx | 108 +
.../ui/src/layouts/Details/DetailsLayout.tsx | 26 +-
.../ui/src/layouts/Details/Gantt/Gantt.test.tsx | 41 +-
.../airflow/ui/src/layouts/Details/Gantt/Gantt.tsx | 6 +-
.../layouts/Details/Grid/useGridPagination.test.ts | 36 +-
.../src/layouts/Details/Grid/useGridPagination.ts | 14 +-
.../airflow/ui/src/queries/useGridRuns.test.tsx | 48 +
.../src/airflow/ui/src/queries/useGridRuns.ts | 50 +-
airflow-core/src/airflow/utils/db.py | 2 +-
.../core_api/routes/public/test_dag_parsing.py | 118 +
.../execution_api/test_dag_processor_client.py | 2 +
.../execution_api/test_token_scope_boundaries.py | 10 +
.../versions/head/test_dag_parsing.py | 2137 ++++++++++++
.../execution_api/versions/head/test_jobs.py | 56 +
.../versions/v2027_02_28/test_jobs.py | 10 +
.../versions/v2027_02_28/test_variables.py | 65 +
.../unit/cli/commands/test_api_server_command.py | 18 +
.../tests/unit/dag_processing/bundles/test_base.py | 6 +-
.../bundles/test_dag_bundle_manager.py | 48 +-
.../unit/dag_processing/bundles/test_local.py | 3 +
airflow-core/tests/unit/dag_processing/conftest.py | 67 +
.../tests/unit/dag_processing/test_api_client.py | 422 ++-
.../tests/unit/dag_processing/test_claims.py | 36 +-
.../tests/unit/dag_processing/test_collection.py | 59 +-
.../tests/unit/dag_processing/test_maintenance.py | 408 +++
.../tests/unit/dag_processing/test_manager.py | 3462 ++++++++++++++++++--
.../dag_processing/test_manager_api_failures.py | 351 ++
.../dag_processing/test_manager_api_inventory.py | 453 +++
.../dag_processing/test_manager_api_isolation.py | 206 ++
.../dag_processing/test_manager_api_results.py | 72 +
.../tests/unit/dag_processing/test_processor.py | 153 +-
.../unit/dag_processing/test_requested_work.py | 141 +
airflow-core/tests/unit/jobs/test_scheduler_job.py | 50 +-
.../tests/unit/listeners/test_importerrors.py | 85 +
.../test_0156_add_processor_work_claims.py | 165 +
airflow-core/tests/unit/models/test_dagwarning.py | 54 -
.../unit/serialization/test_dag_serialization.py | 39 +
airflow-core/tests/unit/utils/test_db_cleanup.py | 2 +
airflow-core/tests/unit/utils/test_log_handlers.py | 2 +-
.../airflowctl/api/datamodels/auth_generated.py | 2 +-
.../src/airflowctl/api/datamodels/generated.py | 2 +-
.../src/airflowctl/ctl/commands/auth_command.py | 5 +-
.../src/airflowctl/ctl/commands/config_command.py | 3 +-
.../airflowctl/ctl/commands/connection_command.py | 5 +-
.../src/airflowctl/ctl/commands/dag_command.py | 10 +-
.../src/airflowctl/ctl/commands/task_command.py | 11 +-
airflow-ctl/src/airflowctl/ctl/utils/dag_run.py | 9 +-
.../airflow_ctl/ctl/commands/test_auth_command.py | 52 +-
.../ctl/commands/test_config_command.py | 14 +-
.../ctl/commands/test_connections_command.py | 25 +
.../airflow_ctl/ctl/commands/test_dag_command.py | 69 +-
.../airflow_ctl/ctl/commands/test_task_command.py | 48 +
.../tests/airflow_ctl/ctl/utils/test_dag_run.py | 29 +
chart/Chart.yaml | 20 +-
chart/docs/using-additional-containers.rst | 11 +-
chart/newsfragments/76601.significant.rst | 3 +
.../templates/pgbouncer/pgbouncer-deployment.yaml | 4 +
.../pgbouncer/pgbouncer-networkpolicy.yaml | 2 +-
.../priorityclasses/priority-classes.yaml | 3 +
.../templates/scheduler/scheduler-deployment.yaml | 2 +-
.../templates/triggerer/triggerer-deployment.yaml | 2 +-
chart/templates/workers/worker-deployment.yaml | 2 +-
.../airflow_aux/test_basic_helm_chart.py | 8 +-
.../helm_tests/airflow_core/test_scheduler.py | 27 +-
.../helm_tests/airflow_core/test_triggerer.py | 31 +-
chart/tests/helm_tests/airflow_core/test_worker.py | 31 +-
chart/tests/helm_tests/other/test_pgbouncer.py | 75 +
chart/values.schema.json | 18 +-
chart/values.yaml | 9 +-
dev/README_RELEASE_PROVIDERS.md | 28 +-
.../doc/ci/08_fork_ci_for_external_contributors.md | 11 +-
.../files/hardened_os_deps_image_digests.json | 24 +-
dev/breeze/src/airflow_breeze/global_constants.py | 6 +
.../src/airflow_breeze/utils/fork_ci_reconcile.py | 77 +-
dev/breeze/tests/test_fork_ci_reconcile.py | 104 +
dev/breeze/uv.lock | 18 +-
docs/spelling_wordlist.txt | 26 +
generated/PYPI_README.md | 10 +-
generated/known_sdk_imports_in_core.txt | 2 +-
go-sdk/pkg/execution/client.go | 20 +-
go-sdk/pkg/execution/genmodels/models.gen.go | 64 +-
go-sdk/schema/supervisor-schema.json | 33 +-
providers/.pre-commit-config.yaml | 2 +-
.../src/airflow/providers/amazon/aws/exceptions.py | 4 +
.../providers/amazon/aws/log/s3_task_handler.py | 86 +-
.../airflow/providers/amazon/aws/operators/glue.py | 51 +-
.../airflow/providers/amazon/aws/triggers/glue.py | 30 +-
.../airflow/providers/amazon/aws/utils/mixins.py | 26 +-
.../unit/amazon/aws/log/test_s3_task_handler.py | 84 +
.../tests/unit/amazon/aws/operators/test_glue.py | 141 +-
.../tests/unit/amazon/aws/triggers/test_glue.py | 41 +
.../tests/unit/amazon/aws/utils/test_mixins.py | 62 +-
.../celery/executors/celery_executor_utils.py | 12 +-
.../integration/celery/test_celery_executor.py | 7 +-
providers/edge3/docs/migrations-ref.rst | 4 +-
.../versions/0008_5_2_0_widen_edge_job_command.py | 41 +-
.../edge3/src/airflow/providers/edge3/models/db.py | 1 +
.../src/airflow/providers/edge3/models/edge_job.py | 3 +-
.../providers/edge3/worker_api/routes/worker.py | 33 +-
.../unit/edge3/executors/test_edge_executor.py | 13 +
.../unit/edge3/worker_api/routes/test_worker.py | 68 +-
.../git/src/airflow/providers/git/bundles/git.py | 51 +-
providers/git/tests/unit/git/bundles/test_git.py | 147 +-
providers/google/docs/connections/gcp_sql.rst | 55 +-
.../providers/google/cloud/hooks/cloud_sql.py | 116 +-
.../unit/google/cloud/hooks/test_cloud_sql.py | 234 ++
.../example_openlineage_task_outcomes_dag.py | 4 +-
.../openlineage_task_outcomes_dag.json | 6 +-
.../tests/system/openlineage/manual_state.py | 28 +-
providers/smtp/docs/connections/smtp.rst | 9 +-
.../src/airflow/providers/smtp/utils/emailer.py | 24 +-
.../smtp/tests/unit/smtp/utils/test_emailer.py | 108 +-
registry/package.json | 4 +-
registry/pnpm-lock.yaml | 76 +-
reproducible_build.yaml | 4 +-
scripts/ci/prek/supported_versions.py | 2 +-
.../observability/metrics/metrics_template.yaml | 24 +
task-sdk/pyproject.toml | 1 +
task-sdk/src/airflow/sdk/api/client.py | 2 +-
.../src/airflow/sdk/api/datamodels/_generated.py | 312 +-
task-sdk/src/airflow/sdk/exceptions.py | 11 +
task-sdk/src/airflow/sdk/execution_time/cache.py | 2 +-
task-sdk/src/airflow/sdk/execution_time/comms.py | 2 +-
task-sdk/src/airflow/sdk/execution_time/context.py | 17 +
.../airflow/sdk/execution_time/schema/schema.json | 33 +-
.../sdk/execution_time/schema/versions/__init__.py | 7 +-
.../execution_time/schema/versions/v2027_02_28.py | 29 +-
.../sdk/execution_time/secrets/execution_api.py | 4 +-
.../tests/task_sdk/definitions/test_variables.py | 39 +-
.../execution_time/schema/test_migrator.py | 65 +-
ts-sdk/schema/supervisor-schema.json | 33 +-
ts-sdk/src/coordinator/client.ts | 8 +-
ts-sdk/src/generated/supervisor.ts | 6 +-
ts-sdk/src/sdk/client.ts | 2 +-
ts-sdk/tests/coordinator/client.test.ts | 16 +-
uv.lock | 183 +-
196 files changed, 16425 insertions(+), 1541 deletions(-)
create mode 100644 airflow-core/newsfragments/74789.significant.rst
create mode 100644 airflow-core/newsfragments/76007.significant.rst
create mode 100644
airflow-core/src/airflow/api_fastapi/execution_api/datamodels/dag_parsing.py
create mode 100644
airflow-core/src/airflow/api_fastapi/execution_api/routes/dag_parsing.py
copy providers/standard/src/airflow/providers/standard/operators/smooth.py =>
airflow-core/src/airflow/dag_processing/claims.py (61%)
create mode 100644 airflow-core/src/airflow/dag_processing/maintenance.py
create mode 100644 airflow-core/src/airflow/dag_processing/requested_work.py
copy providers/http/src/airflow/providers/http/exceptions.py =>
airflow-core/src/airflow/dag_processing/timeouts.py (63%)
create mode 100644
airflow-core/src/airflow/migrations/versions/0155_3_5_0_add_dag_processor_publication_state.py
create mode 100644
airflow-core/src/airflow/migrations/versions/0156_3_5_0_add_processor_work_claims.py
copy airflow-core/src/airflow/models/{tasklog.py =>
dag_processor_publication_receipt.py} (51%)
copy airflow-core/src/airflow/models/{dag_favorite.py =>
dag_processor_work_claim.py} (56%)
create mode 100644
airflow-core/src/airflow/ui/src/layouts/Details/DetailsLayout.test.tsx
create mode 100644 airflow-core/src/airflow/ui/src/queries/useGridRuns.test.tsx
create mode 100644
airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_dag_parsing.py
create mode 100644
airflow-core/tests/unit/api_fastapi/execution_api/versions/v2027_02_28/test_variables.py
create mode 100644 airflow-core/tests/unit/dag_processing/conftest.py
copy providers/google/tests/unit/google/cloud/operators/test_managed_spark.py
=> airflow-core/tests/unit/dag_processing/test_claims.py (50%)
create mode 100644 airflow-core/tests/unit/dag_processing/test_maintenance.py
create mode 100644
airflow-core/tests/unit/dag_processing/test_manager_api_failures.py
create mode 100644
airflow-core/tests/unit/dag_processing/test_manager_api_inventory.py
create mode 100644
airflow-core/tests/unit/dag_processing/test_manager_api_isolation.py
create mode 100644
airflow-core/tests/unit/dag_processing/test_manager_api_results.py
create mode 100644
airflow-core/tests/unit/dag_processing/test_requested_work.py
create mode 100644 airflow-core/tests/unit/listeners/test_importerrors.py
create mode 100644
airflow-core/tests/unit/migrations/test_0156_add_processor_work_claims.py
create mode 100644 chart/newsfragments/76601.significant.rst
copy
airflow-core/src/airflow/migrations/versions/0090_3_2_0_add_length_dag_bundle_team_bundle_name.py
=>
providers/edge3/src/airflow/providers/edge3/migrations/versions/0008_5_2_0_widen_edge_job_command.py
(59%)
