This is an automated email from the ASF dual-hosted git repository. pankaj pushed a change to branch state_check_bq in repository https://gitbox.apache.org/repos/asf/airflow.git
omit 48d5f47e22f Remove loop omit 425ecfcd02c Add tests omit 44ba70aef2a Add tests omit 7a221b80eab Fix existing tests omit 8380b367bd9 BigQueryInsertJobOperator: log transient error and reload job property before marking task as failed or success add cf9edd1bc17 Update documentation to add rebase is optional for Github commit (#42577) add 9fb3d0773b2 docs(newsfragment): these deprecated things are functions instead of arguments (#44242) add a9242844706 make paginated_select kw-only (#44239) add 82e0157c7c2 Remove `/webapp` prefix from new UI (#44227) add d0f433b0245 Fix TriggerDagRunOperator extra_link when trigger_dag_id is templated (#42810) add a1307113103 Revert "Remove `/webapp` prefix from new UI (#44227)" (#44251) add 1307e37bc63 AIP-72: Add support for fetching variables and connections in Supervisor (#44229) add 970cb274466 Use airflow config in new UI (#44252) add 5a0272c272e Create dag graph with nested groups and join_ids (#44199) add 9bc28403811 Update newsfragment text for clarification (#44266) add 22d1406af24 Make filters param optional and fix typing (#44226) add fc52d7d1279 Restrict pydantic 2.10.0 (#44249) add f33166a6122 AIP-81 Add Insert Multiple Pools API (#44121) add f6bbc633cb4 fix google datacatalog operator tests (#44281) add 6c3b6b10aa5 Ensure that Task SDK supervisor closes all its handles correctly. (#44263) add 03d6c2b9ec6 AIP-84: Migrate get_log endpoint (#44238) add 8f567ec4b9a Lower-bind pydantic to 2.10.1 (#44284) add cf2656179a8 Handle XCom value errors in FAB's XCom list view (#44275) add 00fd540fde3 AIP-84 Migrate public endpoint Clear Task Instances to FastAPI (#44220) add 4f8cb6e328a AIP84: UI endpoint for config (#44265) add 208f07b9e29 Add base model for rest api; set from_attributes=True (#44267) add fe166ec0391 Specify workspaceFolder so devcontainer will start with local docker from vscode (#44273) add 5b2a96ee72a Don't exit doc preparation even if changelog is empty for any provider (#44207) add d79c6c21f2d [edge] Clean up of dead tasks in edge_jobs table (#44280) add 66407e8d226 Update backfill `list` endpoint to be async (#44208) add fedfaa703e5 Use `JSONB` type for `XCom.value` column in PostgreSQL (#44290) add ab38d01c5c0 Bump Ruff to 0.8.0 (#44287) add 91bd1eafb03 Introduce gcp advance API (V3) translate native datasets operators (#44271) add d6364992b8a Allow dropping `_xcom_archive` table via CLI (#44291) add b22e3c1fcd5 Fix Dataplex Data Quality partial update (#44262) add 5f6b233906e feat: add OpenLineage support for BigQueryToBigQueryOperator (#44214) add da946865205 Fix incorrect query in `BigQueryAsyncHook.create_job_for_partition_get`. (#44225) add 32f064f73c9 Trigger openlineage test when asset files changes (#44172) add e5de5506d54 Fix the Show Down text (#44292) add 3c58e01266f Bump `google-cloud-translate` to `3.16` (#44297) add 2179b1778a3 Remove unused code from `airflow/api/common/mark_tasks.py` (#44296) add 00a3099b352 rename run_id to dag_run_id in DagRun response (#44305) add 21238d1d613 Bump `termcolor` to `2.5.0` (#44295) add d2c4c120984 fix mypy failure due to termcolor upgrade (#44313) add b1e02120e50 use __file__ for fileloc (#44312) add 29483384be5 Only install eval-type-backport for Python < 3.10 (#44294) add ab72e944dba Replacee `breeze testing tests` command in a few places remaining (#44299) add 49c60677dbd Add copy-pasteable cherry-picker command if it fails to cherry-pick (#44316) add 5e52bd29abd Avoid grouping task instance stats by try_number for dynamic mapped tasks (#44300) add a46e9412823 Update README_AIRFLOW3_DEV.md (#44304) add 2ed30210c9a refactor TestPostDagRun.test_should_respond_200 (#44308) add 1275fec92fd Use Python 3.9 as target version for Ruff & Black rules (#44298) add f0da8eeab3e Prepare docs for Nov 2nd wave of ad hoc providers (#44321) add 33afa8f51b6 Workaround Pydantic 2.10.0/2.10.1 for compatibility tests (#44317) add 18d1b97b79a Refactor package checks to be test and remove duplication (#44318) add 2154e89967b Support multiple executors in chart (#43606) add 53814271df0 Remove py38 as breeze's target (#44328) add cf6cdca2257 setup default logging in test dag serialization tests (#44329) add 3545cde50c2 Bugfix/caplog flaky test in extra operator link test (#44331) add 5a68bca9b03 Add edge provider to boring-cyborg.yml (#44338) add 074c822c6a1 Fix 401 and 403 handling new UI (#44342) add 05f935d2a6c utilize more information to deterministically generate OpenLineage run_id (#43936) add acf106b5a17 AIP-84 Add Lists Jobs with Filters API (#43859) add b4c48062788 Add `wait_policy` option to `EmrCreateJobFlowOperator` (#44055) add bee7f0cdeac AIP-82 Save references between assets and triggers (#43826) add d6887637d56 fix ruff errors in routes public job (#44363) add db260b053fe bump uv version to 0.5.4 (#44349) add 1351b08d86f AIP-84: Migrate Extra Links endpoint to fastapi (#44277) add 19e97dac673 AIP-84 Get Task Instance Tries (#44301) add 8dc5fcdc6af Use UI config (#44364) add 2d9e7a51bb3 Independently update source code changes in DagCode (#44189) add 222dbdc022e Clean up the pre-commit config (#44347) add 4404e64247d Move triggers to standard provider (#43608) add 28ce656772c Replace var names `select`, `base_select`, `base_query` with `statement` (#44270) add c9484fc741e Reset signals on exiting SchedulerJob loop (#44370) add 6748b2a5f71 Fix edge in rare conditions that task state can not be reported as cleaned in parallel (#44314) add e46453966ef Support connection extra parameters in MsSqlHook (#44310) add f0af1b3d23b Move tests `bootstrap_dagbag` into test utils (#44371) add 6d075cb7b8b Update the system test `lib/system-tests/tests/example_comprehend_document_classifier.ts` to use example files from S3 (#44368) add d8c91aa6195 AIP-84 Migrate POST list Dag Runs(batch) endpoint to FastAPI (#44170) add a832c418f35 Migrate public endpoint Patch Task Instance to FastAPI (#44223) add f8a61cb6af1 Simplify asset decorator implementation (#44344) add dc8680163a2 Update dag reserialize command (#43949) add 6b4aff89343 Remove caching inside ExecutorLoader in favour of `job.executors` property (#44367) add b3f57c51af9 AIP-84 Improve testing (#44381) add 624a9421b29 Don't set headers on the `OTLPSpanExporter` (#44346) add a238d06b8a1 AIP-84 Get Mapped Task Instance Tries (#44303) add 2c28dfad57a refactor order by tests (#44380) add e1ff64fc307 AIP-84 Fix default ordering when directly using SortParam (#44393) add cd713ab7cb9 Use FOR KEY SHARE UPDATE instead of FOR UPDATE (#42082) add 0470bd95411 Bump minimum version of databricks SDK to 3.0.0 (#43272) add a18bcd7a1b5 AIP-84 Migrate XCom get entries endpoint to Fastapi (#44366) add 4ec35aaadcf Fix failing ruff check on main (#44399) add 348b61ec465 Fix formatting for backport failure message. (#44392) add b134ed089b7 Remove Pydantic 2.10.0/2.10.1 workaround (#44400) add 9ee501df011 Remove reference to hatch and add uv usage (#44401) add a1fbdb3c1eb AIP-84 Migrate Trigger Dag Run endpoint to FastAPI (#43875) add 890b3eca8d7 Also check NEW_SESSION imports in pre-commit (#44388) add 7cc36104ab9 add basic system tests for OpenLineage (#43643) add 80e938adc33 Add DAG arguments to asset decorator (#44384) add 08289c273fe add openblock labs to INTHEWILD (#44365) add e4f75819627 Add 'airflow assets list' (#44387) add 9fedfd5e390 Delete `dag_version` table when downgrading the DB from the versioned Airflow (#44397) add faa319cc75c Allow 'dag_version' table to be deleted by db clean command (#44389) add 93bcfc2bfee docs(newsfragment): add template for significant newsfragments (#44378) add 5719d195c4a AIP-84 Refactor SortParm (#44345) add 0c30c4d9af2 Move external task sensor to standard provider (#44288) add 8d9672801e3 Add note about why we still have `black` config settings (#44415) add 9bae4a5b482 chore: unify handling of gcs paths (#44410) add 21933a7543b AIP-72: Add support for fetching XComs in Supervisor (#44408) add b1a44b4e3db AIP-72: Improve Supervisor and Task Instance State Validation (#44405) add 25432dc80d8 Move OpenAPI tests to breeze container (#44326) add c75b845d098 Update providers metadata 2024-11-27 (#44418) add 1de5a965921 feat: add OpenLineage support for transfer operators between gcs and local (#44417) add 55c25458c14 AIP-84 Refactor test cases with `datetime_zulu_format` (#44323) add 2fbd3540bb6 Fixing rendering of code block for newsfragment command (#44419) add 6f0d7313506 Add CodeQL security scanning on pull requests. (#44404) add 761cedd0610 AIP 72: Handling "deferrable" tasks in execution_api and task SDK (#44241) add 90442e85775 [Edge] Edge worker supports capacity handling instead of concurrency (#43737) add 43adccf57b2 Fix deferrable mode of BeamRunPythonPipelineOperator (#44386) add cba9f7a22ec minor FastAPI tests fixes (#44429) add e81c3bab80f Add fail_on_file_not_exist option to SFTPToS3Operator (#44320) add c7992727625 Bump uv to 0.5.5 (#44432) add 60a38e1e2ee upgrade github/codeql-action/ to v3 (#44435) add b32ca97bfc9 Suppress JException when get_autocommit and set_autocommit methods aren't supported on JDBC driver (#43786) add c5832d9fb8c Added test-case for callable values in path and query parameters of MSGraphAsyncOperator (#43799) add 77912f82a6c Remove AIP-44 related CI configuration and workflows (#44437) add 933e3cee7d2 Add do_xcom_push documentation in EcsRunTaskOperator (#44440) add 14bfe39298a Fix problem with inability to remove fields from Connection form (#40421) add 0a136f96cc3 Update Dependency Detector to handle dependencies set via partial (#42578) add 335f64c6830 support grouping of log lines for KubernetesPodOperator (#44428) add 518d394119a Allow `json_result_force_utf8_encoding` specification in `providers.snowflake.hooks.SnowflakeHook` extra dict (#44264) add 5474e56f5cb Upgrading tomli to 2.2.1 as suggsested by CI (#44444) add d41bc70577c Passing down PR labels to codeql analysis too (#44448) add ff52d1737ce Get workflow information in codeql workflow (#44450) add 830a2a53f8d Dont add extra deps for providers with same cross provider deps (#44341) add 14919fae914 Convert the LocalExecutor to run tasks using new Task SDK supervisor code (#44427) add 3862e172489 Get rid of database isolation option (#44441) add 497566ff5de Edge worker connected state is sent to DB based on worker sate (#44447) add 69cd23722d7 AIP-84 Migrate private graph_data endpoint (#44394) add 3e427c92488 AIP-84: Migrate Dag Parsing endpoint to FastApi (#44416) add e9f544cc3fb Remove AIP-44 configuration from the code (#44454) add ab2bd2d4a9d Add import compatibility check (#44458) add 1fdaebd9047 feat: add OpenLineage support for S3ToGCSOperator (#44426) add 5e38319f750 Fix issues reported by ruff team before new ruff rules are out (#44460) add b64859f9630 BigQueryInsertJobOperator: log transient error and reload job property before marking task as failed or success add 8962aef8dd0 Fix existing tests add 3c0094244b6 Add tests add 842876a1ef2 Add tests add de9bbfb2af8 Remove loop This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (48d5f47e22f) \ N -- N -- N refs/heads/state_check_bq (de9bbfb2af8) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .devcontainer/mysql/devcontainer.json | 7 +- .devcontainer/postgres/devcontainer.json | 7 +- .github/actions/install-pre-commit/action.yml | 2 +- .github/boring-cyborg.yml | 9 +- .github/workflows/basic-tests.yml | 108 -- ...ic-checks-mypy-docs.yml => ci-image-checks.yml} | 62 +- .github/workflows/ci.yml | 23 +- .github/workflows/codeql-analysis.yml | 14 +- .github/workflows/news-fragment.yml | 25 +- .github/workflows/run-unit-tests.yml | 6 - .github/workflows/special-tests.yml | 26 +- ...ck-providers.yml => test-provider-packages.yml} | 111 +- .pre-commit-config.yaml | 71 +- Dockerfile | 2 +- Dockerfile.ci | 95 +- INTHEWILD.md | 1 + airflow/__main__.py | 30 +- airflow/api/common/mark_tasks.py | 77 +- airflow/api_connexion/endpoints/dag_endpoint.py | 3 +- airflow/api_connexion/endpoints/dag_parsing.py | 5 +- .../api_connexion/endpoints/dag_run_endpoint.py | 5 +- .../api_connexion/endpoints/dag_source_endpoint.py | 3 +- .../api_connexion/endpoints/extra_link_endpoint.py | 2 + .../endpoints/import_error_endpoint.py | 3 +- airflow/api_connexion/endpoints/pool_endpoint.py | 4 +- airflow/api_connexion/endpoints/request_dict.py | 3 +- .../endpoints/task_instance_endpoint.py | 7 +- airflow/api_connexion/endpoints/update_mask.py | 3 +- airflow/api_connexion/endpoints/xcom_endpoint.py | 1 + airflow/api_connexion/parameters.py | 3 +- airflow/api_connexion/types.py | 5 +- airflow/api_fastapi/app.py | 9 +- airflow/api_fastapi/common/db/common.py | 148 +- airflow/api_fastapi/common/exceptions.py | 64 + airflow/api_fastapi/common/headers.py | 3 +- airflow/api_fastapi/common/parameters.py | 101 +- airflow/api_fastapi/common/router.py | 3 +- airflow/api_fastapi/common/types.py | 11 +- airflow/api_fastapi/core_api/app.py | 8 + .../datamodels/variable.py => core_api/base.py} | 14 +- airflow/api_fastapi/core_api/datamodels/assets.py | 5 +- .../api_fastapi/core_api/datamodels/backfills.py | 3 +- airflow/api_fastapi/core_api/datamodels/config.py | 2 +- .../api_fastapi/core_api/datamodels/connections.py | 3 +- airflow/api_fastapi/core_api/datamodels/dag_run.py | 54 +- .../api_fastapi/core_api/datamodels/dag_sources.py | 2 +- .../api_fastapi/core_api/datamodels/dag_stats.py | 3 +- .../api_fastapi/core_api/datamodels/dag_warning.py | 3 +- airflow/api_fastapi/core_api/datamodels/dags.py | 28 +- .../api_fastapi/core_api/datamodels/event_logs.py | 8 +- .../api_fastapi/core_api/datamodels/extra_links.py | 9 + .../core_api/datamodels/import_error.py | 8 +- airflow/api_fastapi/core_api/datamodels/job.py | 11 +- .../core_api/datamodels/{version.py => log.py} | 8 +- airflow/api_fastapi/core_api/datamodels/monitor.py | 2 +- airflow/api_fastapi/core_api/datamodels/plugins.py | 3 +- airflow/api_fastapi/core_api/datamodels/pools.py | 14 +- .../api_fastapi/core_api/datamodels/providers.py | 2 +- .../core_api/datamodels/task_instances.py | 93 +- airflow/api_fastapi/core_api/datamodels/tasks.py | 3 +- airflow/api_fastapi/core_api/datamodels/trigger.py | 4 +- .../datamodels/{dag_stats.py => ui/config.py} | 45 +- airflow/api_fastapi/core_api/datamodels/ui/dags.py | 3 +- .../core_api/datamodels/ui/dashboard.py | 2 +- .../api_fastapi/core_api/datamodels/ui/graph.py | 60 + .../api_fastapi/core_api/datamodels/variables.py | 5 +- airflow/api_fastapi/core_api/datamodels/version.py | 2 +- airflow/api_fastapi/core_api/datamodels/xcom.py | 11 +- .../api_fastapi/core_api/openapi/v1-generated.yaml | 1937 ++++++++++++++++++-- .../api_fastapi/core_api/routes/public/__init__.py | 11 +- .../api_fastapi/core_api/routes/public/assets.py | 31 +- .../core_api/routes/public/backfills.py | 27 +- .../core_api/routes/public/connections.py | 21 +- .../core_api/routes/public}/dag_parsing.py | 49 +- .../api_fastapi/core_api/routes/public/dag_run.py | 174 +- .../core_api/routes/public/dag_stats.py | 2 +- .../core_api/routes/public/dag_warning.py | 14 +- airflow/api_fastapi/core_api/routes/public/dags.py | 56 +- .../core_api/routes/public/event_logs.py | 48 +- .../core_api/routes/public/extra_links.py} | 52 +- .../core_api/routes/public/import_error.py | 22 +- airflow/api_fastapi/core_api/routes/public/job.py | 128 ++ airflow/api_fastapi/core_api/routes/public/log.py | 146 ++ .../api_fastapi/core_api/routes/public/plugins.py | 7 +- .../api_fastapi/core_api/routes/public/pools.py | 43 +- .../core_api/routes/public/task_instances.py | 345 +++- .../api_fastapi/core_api/routes/public/tasks.py | 7 +- .../core_api/routes/public/variables.py | 11 +- .../api_fastapi/core_api/routes/public/version.py | 2 +- airflow/api_fastapi/core_api/routes/public/xcom.py | 58 +- airflow/api_fastapi/core_api/routes/ui/__init__.py | 6 +- airflow/api_fastapi/core_api/routes/ui/config.py | 71 + airflow/api_fastapi/core_api/routes/ui/dags.py | 22 +- .../api_fastapi/core_api/routes/ui/dashboard.py | 6 +- airflow/api_fastapi/core_api/routes/ui/graph.py | 66 + airflow/api_fastapi/core_api/security.py | 3 +- .../execution_api/datamodels/connection.py | 4 +- .../execution_api/datamodels/taskinstance.py | 31 +- .../api_fastapi/execution_api/datamodels/token.py | 2 +- .../execution_api/datamodels/variable.py | 6 +- .../api_fastapi/execution_api/datamodels/xcom.py | 2 +- .../execution_api/routes/connections.py | 2 +- .../execution_api/routes/task_instances.py | 25 + airflow/api_internal/internal_api_call.py | 150 +- airflow/auth/managers/base_auth_manager.py | 3 +- airflow/cli/cli_config.py | 89 +- airflow/cli/cli_parser.py | 7 +- airflow/cli/commands/asset_command.py | 53 + airflow/cli/commands/cheat_sheet_command.py | 3 +- airflow/cli/commands/dag_command.py | 9 +- airflow/cli/commands/dag_processor_command.py | 4 - airflow/cli/commands/db_command.py | 3 - airflow/cli/commands/internal_api_command.py | 8 - airflow/cli/commands/standalone_command.py | 8 +- airflow/cli/commands/task_command.py | 48 +- airflow/cli/commands/triggerer_command.py | 2 +- airflow/cli/simple_table.py | 3 +- airflow/cli/utils.py | 3 +- airflow/config_templates/config.yml | 2 +- airflow/configuration.py | 12 +- airflow/dag_processing/collection.py | 96 + airflow/dag_processing/manager.py | 3 +- airflow/dag_processing/processor.py | 11 +- airflow/decorators/__init__.pyi | 3 +- airflow/decorators/base.py | 5 +- airflow/decorators/bash.py | 3 +- airflow/decorators/python.py | 3 +- airflow/decorators/sensor.py | 3 +- airflow/decorators/task_group.py | 3 +- .../example_external_task_marker_dag.py | 2 +- airflow/example_dags/example_params_ui_tutorial.py | 422 ++--- airflow/executors/base_executor.py | 18 +- airflow/executors/executor_loader.py | 31 +- airflow/executors/local_executor.py | 140 +- airflow/executors/workloads.py | 98 + airflow/io/__init__.py | 2 +- airflow/io/path.py | 3 +- airflow/io/typedef.py | 4 +- airflow/jobs/job.py | 5 +- airflow/jobs/scheduler_job_runner.py | 74 +- airflow/metrics/otel_logger.py | 8 +- airflow/metrics/validators.py | 4 +- .../versions/0047_3_0_0_add_dag_versioning.py | 13 +- airflow/models/abstractoperator.py | 3 +- airflow/models/asset.py | 2 +- airflow/models/baseoperator.py | 6 +- airflow/models/dag.py | 10 +- airflow/models/dagbag.py | 4 + airflow/models/dagcode.py | 114 +- airflow/models/dagrun.py | 8 +- airflow/models/expandinput.py | 6 +- airflow/models/mappedoperator.py | 6 +- airflow/models/param.py | 3 +- airflow/models/skipmixin.py | 3 +- airflow/models/taskinstance.py | 51 +- airflow/models/taskmap.py | 3 +- airflow/models/trigger.py | 13 +- airflow/models/xcom.py | 6 +- airflow/models/xcom_arg.py | 3 +- airflow/notifications/basenotifier.py | 3 +- airflow/operators/branch.py | 3 +- airflow/operators/email.py | 3 +- airflow/operators/latest_only.py | 3 +- airflow/plugins_manager.py | 5 +- airflow/providers_manager.py | 3 +- airflow/reproducible_build.yaml | 2 +- airflow/sensors/base.py | 9 +- airflow/serialization/json_schema.py | 3 +- airflow/serialization/pydantic/asset.py | 6 +- airflow/serialization/pydantic/dag.py | 6 +- airflow/serialization/pydantic/dag_run.py | 5 +- airflow/serialization/pydantic/taskinstance.py | 3 +- airflow/serialization/pydantic/trigger.py | 10 +- airflow/serialization/serde.py | 11 +- airflow/serialization/serialized_objects.py | 58 +- airflow/settings.py | 174 +- airflow/task/standard_task_runner.py | 13 - airflow/template/templater.py | 3 +- airflow/ti_deps/deps/base_ti_dep.py | 3 +- airflow/ti_deps/deps/trigger_rule_dep.py | 3 +- airflow/timetables/base.py | 3 +- airflow/timetables/events.py | 3 +- airflow/timetables/simple.py | 3 +- airflow/traces/otel_tracer.py | 5 +- airflow/triggers/base.py | 11 +- airflow/ui/openapi-gen/queries/common.ts | 704 ++++--- airflow/ui/openapi-gen/queries/prefetch.ts | 917 ++++++--- airflow/ui/openapi-gen/queries/queries.ts | 1510 +++++++++++---- airflow/ui/openapi-gen/queries/suspense.ts | 1005 +++++++--- airflow/ui/openapi-gen/requests/schemas.gen.ts | 944 +++++++++- airflow/ui/openapi-gen/requests/services.gen.ts | 1374 ++++++++++---- airflow/ui/openapi-gen/requests/types.gen.ts | 1124 ++++++++++-- airflow/ui/package.json | 8 +- airflow/ui/pnpm-lock.yaml | 455 +++++ airflow/ui/src/components/ConfirmationModal.tsx | 66 + .../src/components/DataTable/useTableUrlState.ts | 24 +- airflow/ui/src/components/TogglePause.tsx | 47 +- .../src/components/TriggerDag/TriggerDAGModal.tsx | 6 +- .../ui/src/components/ui/Dialog/CloseTrigger.tsx | 37 +- airflow/ui/src/constants/sortParams.ts | 2 +- airflow/ui/src/context/colorMode/useColorMode.tsx | 2 +- .../src/context/openGroups/OpenGroupsProvider.tsx | 69 + .../useColorMode.tsx => openGroups/index.ts} | 15 +- .../useOpenGroups.ts} | 24 +- .../ui/src/context/timezone/TimezoneProvider.tsx | 6 +- airflow/ui/src/layouts/BaseLayout.tsx | 30 +- airflow/ui/src/layouts/Nav/DocsButton.tsx | 51 +- airflow/ui/src/main.tsx | 6 +- airflow/ui/src/pages/DagsList/Dag/Code/Code.tsx | 6 +- airflow/ui/src/pages/DagsList/Dag/Dag.tsx | 38 +- .../DagsList/Dag/DagVizModal.tsx} | 53 +- airflow/ui/src/pages/DagsList/Dag/Graph/Edge.tsx | 81 + airflow/ui/src/pages/DagsList/Dag/Graph/Graph.tsx | 69 + .../DagsList/Dag/Graph/JoinNode.tsx} | 26 +- .../DagsList/Dag/Graph/NodeWrapper.tsx} | 22 +- .../ui/src/pages/DagsList/Dag/Graph/TaskName.tsx | 63 + .../ui/src/pages/DagsList/Dag/Graph/TaskNode.tsx | 120 ++ airflow/ui/src/pages/DagsList/Dag/Graph/data.ts | 216 +++ .../DagsList/Dag/Graph/index.ts} | 14 +- .../src/pages/DagsList/Dag/Graph/reactflowUtils.ts | 140 ++ .../src/pages/DagsList/Dag/Graph/useGraphLayout.ts | 288 +++ airflow/ui/src/pages/DagsList/Dag/Header.tsx | 6 +- .../src/pages/DagsList/Dag/{Dag.tsx => Tabs.tsx} | 88 +- airflow/ui/src/pages/DagsList/DagCard.tsx | 6 +- airflow/ui/src/pages/DagsList/DagsFilters.tsx | 12 +- airflow/ui/src/pages/DagsList/DagsList.tsx | 52 +- airflow/ui/src/pages/DagsList/RecentRuns.tsx | 2 +- airflow/ui/src/pages/DagsList/SortSelect.tsx | 50 + airflow/ui/src/pages/Dashboard/Health/Health.tsx | 16 +- .../useColorMode.tsx => queries/useConfig.tsx} | 16 +- airflow/ui/vite.config.ts | 2 + airflow/utils/cli.py | 3 +- airflow/utils/context.py | 8 +- airflow/utils/context.pyi | 3 +- airflow/utils/db.py | 48 +- airflow/utils/db_cleanup.py | 17 +- airflow/utils/email.py | 3 +- airflow/utils/entry_points.py | 6 +- airflow/utils/file.py | 4 +- airflow/utils/helpers.py | 28 +- airflow/utils/log/file_task_handler.py | 3 +- airflow/utils/log/log_reader.py | 3 +- airflow/utils/log/secrets_masker.py | 11 +- airflow/utils/net.py | 4 +- airflow/utils/operator_helpers.py | 3 +- airflow/utils/process_utils.py | 2 +- airflow/utils/session.py | 39 +- airflow/utils/sqlalchemy.py | 7 +- airflow/utils/task_group.py | 3 +- airflow/utils/task_instance_session.py | 8 - airflow/utils/timeout.py | 4 +- airflow/utils/weekday.py | 2 +- airflow/www/app.py | 7 - airflow/www/auth.py | 3 +- airflow/www/extensions/init_wsgi_middlewares.py | 3 +- airflow/www/forms.py | 2 +- airflow/www/utils.py | 20 +- airflow/www/views.py | 22 +- chart/templates/NOTES.txt | 6 +- chart/templates/_helpers.yaml | 8 +- chart/templates/check-values.yaml | 2 +- chart/templates/configmaps/configmap.yaml | 2 +- chart/templates/flower/flower-deployment.yaml | 2 +- chart/templates/flower/flower-ingress.yaml | 2 +- chart/templates/flower/flower-service.yaml | 2 +- chart/templates/flower/flower-serviceaccount.yaml | 2 +- .../security-context-constraint-rolebinding.yaml | 2 +- chart/templates/redis/redis-networkpolicy.yaml | 2 +- chart/templates/redis/redis-service.yaml | 2 +- chart/templates/redis/redis-serviceaccount.yaml | 2 +- chart/templates/redis/redis-statefulset.yaml | 2 +- .../scheduler/scheduler-networkpolicy.yaml | 2 +- chart/templates/scheduler/scheduler-service.yaml | 2 +- .../secrets/result-backend-connection-secret.yaml | 2 +- chart/templates/workers/worker-deployment.yaml | 2 +- chart/templates/workers/worker-kedaautoscaler.yaml | 2 +- chart/templates/workers/worker-networkpolicy.yaml | 2 +- chart/templates/workers/worker-service.yaml | 2 +- chart/templates/workers/worker-serviceaccount.yaml | 2 +- chart/values.schema.json | 12 +- chart/values.yaml | 4 +- clients/python/test_python_client.py | 137 +- contributing-docs/05_pull_requests.rst | 3 +- contributing-docs/08_static_code_checks.rst | 2 +- contributing-docs/09_testing.rst | 3 + contributing-docs/10_working_with_git.rst | 3 + contributing-docs/11_provider_packages.rst | 14 +- contributing-docs/16_contribution_workflow.rst | 8 +- contributing-docs/testing/python_client_tests.rst | 47 + contributing-docs/testing/unit_tests.rst | 34 - dev/README_AIRFLOW3_DEV.md | 63 +- dev/backport/update_backport_status.py | 41 +- dev/breeze/README.md | 2 +- dev/breeze/doc/03_developer_tasks.rst | 2 +- dev/breeze/doc/05_test_commands.rst | 31 +- dev/breeze/doc/ci/02_images.md | 2 +- dev/breeze/doc/ci/04_selective_checks.md | 2 +- dev/breeze/doc/images/output-commands.svg | 148 +- .../output_setup_check-all-params-in-groups.svg | 4 +- .../output_setup_check-all-params-in-groups.txt | 2 +- .../output_setup_regenerate-command-images.svg | 28 +- .../output_setup_regenerate-command-images.txt | 2 +- dev/breeze/doc/images/output_shell.svg | 312 ++-- dev/breeze/doc/images/output_shell.txt | 2 +- dev/breeze/doc/images/output_start-airflow.svg | 206 +-- dev/breeze/doc/images/output_start-airflow.txt | 2 +- dev/breeze/doc/images/output_testing.svg | 18 +- dev/breeze/doc/images/output_testing.txt | 2 +- .../doc/images/output_testing_providers-tests.svg | 118 +- .../doc/images/output_testing_providers-tests.txt | 2 +- .../output_testing_python-api-client-tests.svg | 236 +++ .../output_testing_python-api-client-tests.txt | 1 + dev/breeze/pyproject.toml | 2 +- .../src/airflow_breeze/commands/common_options.py | 6 - .../airflow_breeze/commands/developer_commands.py | 28 +- .../commands/developer_commands_config.py | 5 +- .../src/airflow_breeze/commands/main_command.py | 2 - .../commands/release_management_commands.py | 6 +- .../airflow_breeze/commands/testing_commands.py | 88 +- .../commands/testing_commands_config.py | 25 +- dev/breeze/src/airflow_breeze/global_constants.py | 5 +- .../src/airflow_breeze/params/shell_params.py | 11 +- dev/breeze/src/airflow_breeze/utils/console.py | 2 +- dev/breeze/src/airflow_breeze/utils/packages.py | 22 + dev/breeze/src/airflow_breeze/utils/run_tests.py | 24 +- .../src/airflow_breeze/utils/selective_checks.py | 33 +- dev/breeze/tests/test_packages.py | 12 +- .../tests/test_pytest_args_for_test_types.py | 6 +- dev/breeze/tests/test_selective_checks.py | 38 +- .../test_examples_of_prod_image_building.py | 4 +- .../operators/emr/emr.rst | 9 + docs/apache-airflow-providers-celery/commits.rst | 15 +- docs/apache-airflow-providers-celery/index.rst | 6 +- .../commits.rst | 15 + .../index.rst | 2 +- .../edge_executor.rst | 46 + docs/apache-airflow-providers-google/commits.rst | 23 + docs/apache-airflow-providers-google/index.rst | 10 +- .../operators/cloud/translate.rst | 85 + .../apache-airflow-providers-openlineage/index.rst | 8 + docs/apache-airflow-providers-standard/commits.rst | 17 + docs/apache-airflow-providers-standard/index.rst | 2 +- .../sensors}/external_task_sensor.rst | 4 +- .../authoring-and-scheduling/deferring.rst | 12 +- docs/apache-airflow/core-concepts/dags.rst | 2 +- docs/apache-airflow/howto/operator/index.rst | 2 - docs/apache-airflow/img/airflow_erd.sha256 | 2 +- docs/apache-airflow/img/airflow_erd.svg | 1769 +++++++++--------- docs/apache-airflow/operators-and-hooks-ref.rst | 5 +- docs/build_docs.py | 3 +- docs/exts/docs_build/fetch_inventories.py | 2 +- docs/exts/docs_build/lint_checks.py | 2 +- docs/exts/operators_and_hooks_ref.py | 7 +- docs/exts/providers_extensions.py | 3 +- docs/exts/sphinx_script_update.py | 4 +- docs/spelling_wordlist.txt | 1 + generated/provider_dependencies.json | 9 +- generated/provider_metadata.json | 12 + hatch_build.py | 14 +- helm_tests/airflow_aux/test_basic_helm_chart.py | 10 +- helm_tests/other/test_keda.py | 4 +- newsfragments/41391.significant.rst | 4 +- newsfragments/41533.significant.rst | 8 +- newsfragments/41808.significant.rst | 2 +- newsfragments/43949.significant.rst | 5 + newsfragments/44300.bugfix.rst | 1 + newsfragments/aip-72.significant.rst | 6 +- newsfragments/template.significant.rst | 19 + .../performance_dag/performance_dag_utils.py | 4 +- providers/pyproject.toml | 2 +- .../airflow/providers/airbyte/operators/airbyte.py | 3 +- .../airflow/providers/airbyte/sensors/airbyte.py | 3 +- .../airflow/providers/airbyte/triggers/airbyte.py | 3 +- .../alibaba/cloud/hooks/analyticdb_spark.py | 3 +- .../alibaba/cloud/operators/analyticdb_spark.py | 3 +- .../alibaba/cloud/sensors/analyticdb_spark.py | 3 +- .../providers/alibaba/cloud/sensors/oss_key.py | 3 +- .../amazon/aws/auth_manager/avp/facade.py | 3 +- .../amazon/aws/auth_manager/aws_auth_manager.py | 3 +- .../amazon/aws/executors/batch/batch_executor.py | 7 +- .../providers/amazon/aws/executors/batch/utils.py | 6 +- .../amazon/aws/executors/ecs/ecs_executor.py | 3 +- .../providers/amazon/aws/executors/ecs/utils.py | 6 +- .../src/airflow/providers/amazon/aws/fs/s3.py | 4 +- .../airflow/providers/amazon/aws/hooks/appflow.py | 3 +- .../airflow/providers/amazon/aws/hooks/athena.py | 3 +- .../airflow/providers/amazon/aws/hooks/dynamodb.py | 3 +- .../src/airflow/providers/amazon/aws/hooks/eks.py | 3 +- .../airflow/providers/amazon/aws/hooks/kinesis.py | 2 +- .../src/airflow/providers/amazon/aws/hooks/logs.py | 3 +- .../providers/amazon/aws/hooks/redshift_cluster.py | 3 +- .../providers/amazon/aws/hooks/redshift_data.py | 3 +- .../src/airflow/providers/amazon/aws/hooks/s3.py | 3 +- .../providers/amazon/aws/hooks/sagemaker.py | 3 +- .../src/airflow/providers/amazon/aws/hooks/ses.py | 3 +- .../providers/amazon/aws/notifications/sns.py | 2 +- .../providers/amazon/aws/notifications/sqs.py | 2 +- .../providers/amazon/aws/operators/athena.py | 3 +- .../providers/amazon/aws/operators/base_aws.py | 2 +- .../providers/amazon/aws/operators/batch.py | 3 +- .../providers/amazon/aws/operators/bedrock.py | 3 +- .../amazon/aws/operators/cloud_formation.py | 3 +- .../providers/amazon/aws/operators/comprehend.py | 3 +- .../providers/amazon/aws/operators/datasync.py | 3 +- .../airflow/providers/amazon/aws/operators/dms.py | 3 +- .../airflow/providers/amazon/aws/operators/ec2.py | 3 +- .../airflow/providers/amazon/aws/operators/ecs.py | 5 +- .../airflow/providers/amazon/aws/operators/eks.py | 7 +- .../airflow/providers/amazon/aws/operators/emr.py | 39 +- .../providers/amazon/aws/operators/eventbridge.py | 3 +- .../providers/amazon/aws/operators/glacier.py | 3 +- .../airflow/providers/amazon/aws/operators/glue.py | 3 +- .../providers/amazon/aws/operators/glue_crawler.py | 3 +- .../amazon/aws/operators/glue_databrew.py | 3 +- .../amazon/aws/operators/kinesis_analytics.py | 3 +- .../amazon/aws/operators/lambda_function.py | 3 +- .../providers/amazon/aws/operators/neptune.py | 3 +- .../providers/amazon/aws/operators/quicksight.py | 3 +- .../airflow/providers/amazon/aws/operators/rds.py | 3 +- .../amazon/aws/operators/redshift_cluster.py | 3 +- .../airflow/providers/amazon/aws/operators/s3.py | 3 +- .../providers/amazon/aws/operators/sagemaker.py | 3 +- .../airflow/providers/amazon/aws/operators/sns.py | 3 +- .../airflow/providers/amazon/aws/operators/sqs.py | 3 +- .../amazon/aws/operators/step_function.py | 3 +- .../airflow/providers/amazon/aws/sensors/athena.py | 3 +- .../providers/amazon/aws/sensors/base_aws.py | 2 +- .../airflow/providers/amazon/aws/sensors/batch.py | 3 +- .../providers/amazon/aws/sensors/bedrock.py | 3 +- .../amazon/aws/sensors/cloud_formation.py | 3 +- .../providers/amazon/aws/sensors/comprehend.py | 3 +- .../airflow/providers/amazon/aws/sensors/dms.py | 3 +- .../providers/amazon/aws/sensors/dynamodb.py | 3 +- .../airflow/providers/amazon/aws/sensors/ec2.py | 3 +- .../airflow/providers/amazon/aws/sensors/ecs.py | 3 +- .../airflow/providers/amazon/aws/sensors/eks.py | 3 +- .../airflow/providers/amazon/aws/sensors/emr.py | 3 +- .../providers/amazon/aws/sensors/glacier.py | 3 +- .../airflow/providers/amazon/aws/sensors/glue.py | 3 +- .../amazon/aws/sensors/glue_catalog_partition.py | 3 +- .../providers/amazon/aws/sensors/glue_crawler.py | 3 +- .../amazon/aws/sensors/kinesis_analytics.py | 3 +- .../amazon/aws/sensors/lambda_function.py | 3 +- .../amazon/aws/sensors/opensearch_serverless.py | 3 +- .../providers/amazon/aws/sensors/quicksight.py | 3 +- .../airflow/providers/amazon/aws/sensors/rds.py | 3 +- .../amazon/aws/sensors/redshift_cluster.py | 3 +- .../src/airflow/providers/amazon/aws/sensors/s3.py | 3 +- .../providers/amazon/aws/sensors/sagemaker.py | 3 +- .../airflow/providers/amazon/aws/sensors/sqs.py | 3 +- .../providers/amazon/aws/sensors/step_function.py | 3 +- .../amazon/aws/transfers/azure_blob_to_s3.py | 3 +- .../airflow/providers/amazon/aws/transfers/base.py | 2 +- .../amazon/aws/transfers/dynamodb_to_s3.py | 3 +- .../providers/amazon/aws/transfers/exasol_to_s3.py | 3 +- .../providers/amazon/aws/transfers/ftp_to_s3.py | 3 +- .../providers/amazon/aws/transfers/gcs_to_s3.py | 3 +- .../amazon/aws/transfers/glacier_to_gcs.py | 3 +- .../amazon/aws/transfers/google_api_to_s3.py | 3 +- .../amazon/aws/transfers/hive_to_dynamodb.py | 3 +- .../amazon/aws/transfers/imap_attachment_to_s3.py | 3 +- .../providers/amazon/aws/transfers/local_to_s3.py | 3 +- .../providers/amazon/aws/transfers/mongo_to_s3.py | 3 +- .../amazon/aws/transfers/redshift_to_s3.py | 3 +- .../amazon/aws/transfers/s3_to_dynamodb.py | 3 +- .../providers/amazon/aws/transfers/s3_to_ftp.py | 3 +- .../amazon/aws/transfers/s3_to_redshift.py | 3 +- .../providers/amazon/aws/transfers/s3_to_sftp.py | 3 +- .../providers/amazon/aws/transfers/s3_to_sql.py | 3 +- .../amazon/aws/transfers/salesforce_to_s3.py | 3 +- .../providers/amazon/aws/transfers/sftp_to_s3.py | 15 +- .../providers/amazon/aws/transfers/sql_to_s3.py | 3 +- .../airflow/providers/amazon/aws/triggers/base.py | 3 +- .../airflow/providers/amazon/aws/triggers/glue.py | 3 +- .../amazon/aws/triggers/redshift_cluster.py | 3 +- .../providers/amazon/aws/triggers/redshift_data.py | 3 +- .../airflow/providers/amazon/aws/triggers/s3.py | 3 +- .../providers/amazon/aws/triggers/sagemaker.py | 3 +- .../airflow/providers/amazon/aws/triggers/sqs.py | 3 +- .../providers/amazon/aws/utils/task_log_fetcher.py | 3 +- .../airflow/providers/amazon/aws/utils/waiter.py | 20 + .../providers/apache/beam/operators/beam.py | 27 +- .../airflow/providers/apache/beam/triggers/beam.py | 23 +- .../providers/apache/cassandra/sensors/record.py | 3 +- .../providers/apache/cassandra/sensors/table.py | 3 +- .../airflow/providers/apache/drill/hooks/drill.py | 3 +- .../providers/apache/drill/operators/drill.py | 3 +- .../airflow/providers/apache/druid/hooks/druid.py | 3 +- .../providers/apache/druid/operators/druid.py | 3 +- .../apache/druid/transfers/hive_to_druid.py | 3 +- .../apache/flink/operators/flink_kubernetes.py | 3 +- .../apache/flink/sensors/flink_kubernetes.py | 3 +- .../providers/apache/hdfs/sensors/web_hdfs.py | 3 +- .../providers/apache/hive/operators/hive.py | 3 +- .../providers/apache/hive/operators/hive_stats.py | 3 +- .../apache/hive/sensors/hive_partition.py | 3 +- .../apache/hive/sensors/metastore_partition.py | 3 +- .../apache/hive/sensors/named_hive_partition.py | 3 +- .../apache/hive/transfers/hive_to_mysql.py | 3 +- .../apache/hive/transfers/hive_to_samba.py | 3 +- .../apache/hive/transfers/mssql_to_hive.py | 3 +- .../apache/hive/transfers/mysql_to_hive.py | 3 +- .../providers/apache/hive/transfers/s3_to_hive.py | 7 +- .../apache/hive/transfers/vertica_to_hive.py | 3 +- .../airflow/providers/apache/kafka/hooks/client.py | 3 +- .../providers/apache/kafka/hooks/consume.py | 2 +- .../providers/apache/kafka/operators/consume.py | 3 +- .../providers/apache/kafka/operators/produce.py | 3 +- .../providers/apache/kafka/sensors/kafka.py | 3 +- .../apache/kafka/triggers/await_message.py | 3 +- .../providers/apache/kylin/operators/kylin_cube.py | 3 +- .../airflow/providers/apache/livy/hooks/livy.py | 3 +- .../providers/apache/livy/operators/livy.py | 3 +- .../airflow/providers/apache/livy/sensors/livy.py | 3 +- .../airflow/providers/apache/livy/triggers/livy.py | 3 +- .../airflow/providers/apache/pig/operators/pig.py | 3 +- .../airflow/providers/apache/pinot/hooks/pinot.py | 3 +- .../providers/apache/spark/decorators/pyspark.py | 3 +- .../providers/apache/spark/hooks/spark_submit.py | 3 +- .../providers/apache/spark/operators/spark_sql.py | 3 +- .../apache/spark/operators/spark_submit.py | 3 +- .../src/airflow/providers/apprise/hooks/apprise.py | 3 +- .../providers/apprise/notifications/apprise.py | 2 +- .../providers/arangodb/operators/arangodb.py | 3 +- .../airflow/providers/arangodb/sensors/arangodb.py | 3 +- .../providers/atlassian/jira/operators/jira.py | 3 +- .../providers/atlassian/jira/sensors/jira.py | 3 +- .../src/airflow/providers/celery/CHANGELOG.rst | 13 + providers/src/airflow/providers/celery/__init__.py | 2 +- .../providers/celery/executors/celery_executor.py | 5 +- .../celery/executors/celery_executor_utils.py | 10 +- .../celery/executors/celery_kubernetes_executor.py | 3 +- .../src/airflow/providers/celery/provider.yaml | 3 +- .../providers/cncf/kubernetes/CHANGELOG.rst | 19 + .../airflow/providers/cncf/kubernetes/__init__.py | 2 +- .../cncf/kubernetes/decorators/kubernetes.py | 3 +- .../kubernetes/executors/kubernetes_executor.py | 3 +- .../executors/kubernetes_executor_types.py | 8 +- .../executors/local_kubernetes_executor.py | 3 +- .../providers/cncf/kubernetes/hooks/kubernetes.py | 3 +- .../providers/cncf/kubernetes/operators/job.py | 3 +- .../providers/cncf/kubernetes/operators/pod.py | 4 +- .../cncf/kubernetes/operators/resource.py | 3 +- .../providers/cncf/kubernetes/provider.yaml | 3 +- .../cncf/kubernetes/sensors/spark_kubernetes.py | 3 +- .../providers/cncf/kubernetes/triggers/job.py | 3 +- .../providers/cncf/kubernetes/triggers/pod.py | 3 +- .../cncf/kubernetes/utils/k8s_resource_iterator.py | 3 +- .../providers/cncf/kubernetes/utils/pod_manager.py | 19 +- .../providers/cohere/operators/embedding.py | 3 +- .../providers/common/compat/standard/triggers.py | 22 +- .../src/airflow/providers/common/io/assets/file.py | 4 +- .../providers/common/io/operators/file_transfer.py | 3 +- .../airflow/providers/common/sql/hooks/handlers.py | 2 +- .../src/airflow/providers/common/sql/hooks/sql.py | 8 +- .../airflow/providers/common/sql/operators/sql.py | 3 +- .../airflow/providers/common/sql/sensors/sql.py | 3 +- .../providers/databricks/hooks/databricks_sql.py | 48 +- .../providers/databricks/operators/databricks.py | 3 +- .../databricks/operators/databricks_repos.py | 3 +- .../databricks/operators/databricks_sql.py | 3 +- .../src/airflow/providers/databricks/provider.yaml | 5 +- .../databricks/sensors/databricks_partition.py | 3 +- .../providers/databricks/sensors/databricks_sql.py | 3 +- .../airflow/providers/dbt/cloud/triggers/dbt.py | 3 +- .../providers/dbt/cloud/utils/openlineage.py | 1 + .../providers/dingding/operators/dingding.py | 3 +- .../providers/discord/operators/discord_webhook.py | 3 +- .../airflow/providers/docker/decorators/docker.py | 3 +- .../airflow/providers/docker/operators/docker.py | 4 +- providers/src/airflow/providers/edge/CHANGELOG.rst | 31 + providers/src/airflow/providers/edge/__init__.py | 2 +- .../src/airflow/providers/edge/cli/edge_command.py | 41 +- .../providers/edge/executors/edge_executor.py | 87 +- .../src/airflow/providers/edge/models/edge_job.py | 26 +- .../airflow/providers/edge/models/edge_worker.py | 19 +- providers/src/airflow/providers/edge/provider.yaml | 12 +- .../providers/edge/worker_api/datamodels.py | 4 +- .../providers/elasticsearch/log/es_response.py | 2 +- .../providers/elasticsearch/log/es_task_handler.py | 4 +- .../src/airflow/providers/exasol/hooks/exasol.py | 3 +- .../airflow/providers/exasol/operators/exasol.py | 3 +- .../fab/auth_manager/cli_commands/utils.py | 7 +- .../providers/fab/auth_manager/decorators/auth.py | 3 +- .../providers/fab/auth_manager/fab_auth_manager.py | 3 +- .../fab/auth_manager/security_manager/override.py | 3 +- .../src/airflow/providers/ftp/operators/ftp.py | 3 +- providers/src/airflow/providers/ftp/sensors/ftp.py | 3 +- .../src/airflow/providers/google/CHANGELOG.rst | 30 + providers/src/airflow/providers/google/__init__.py | 2 +- .../airflow/providers/google/ads/operators/ads.py | 3 +- .../providers/google/ads/transfers/ads_to_gcs.py | 3 +- .../airflow/providers/google/cloud/hooks/automl.py | 3 +- .../providers/google/cloud/hooks/bigquery.py | 5 +- .../providers/google/cloud/hooks/bigquery_dts.py | 3 +- .../providers/google/cloud/hooks/bigtable.py | 3 +- .../providers/google/cloud/hooks/cloud_batch.py | 3 +- .../providers/google/cloud/hooks/cloud_build.py | 3 +- .../providers/google/cloud/hooks/cloud_composer.py | 3 +- .../google/cloud/hooks/cloud_memorystore.py | 3 +- .../providers/google/cloud/hooks/cloud_run.py | 3 +- .../providers/google/cloud/hooks/cloud_sql.py | 3 +- .../cloud/hooks/cloud_storage_transfer_service.py | 3 +- .../providers/google/cloud/hooks/compute.py | 3 +- .../providers/google/cloud/hooks/datacatalog.py | 3 +- .../providers/google/cloud/hooks/dataflow.py | 3 +- .../providers/google/cloud/hooks/dataform.py | 3 +- .../providers/google/cloud/hooks/datafusion.py | 5 +- .../providers/google/cloud/hooks/dataplex.py | 3 +- .../providers/google/cloud/hooks/dataproc.py | 4 +- .../google/cloud/hooks/dataproc_metastore.py | 3 +- .../providers/google/cloud/hooks/datastore.py | 3 +- .../airflow/providers/google/cloud/hooks/dlp.py | 3 +- .../providers/google/cloud/hooks/functions.py | 2 +- .../airflow/providers/google/cloud/hooks/gcs.py | 3 +- .../airflow/providers/google/cloud/hooks/kms.py | 3 +- .../google/cloud/hooks/kubernetes_engine.py | 3 +- .../providers/google/cloud/hooks/life_sciences.py | 2 +- .../google/cloud/hooks/natural_language.py | 3 +- .../providers/google/cloud/hooks/os_login.py | 3 +- .../airflow/providers/google/cloud/hooks/pubsub.py | 3 +- .../providers/google/cloud/hooks/secret_manager.py | 3 +- .../providers/google/cloud/hooks/spanner.py | 3 +- .../providers/google/cloud/hooks/speech_to_text.py | 3 +- .../providers/google/cloud/hooks/stackdriver.py | 3 +- .../airflow/providers/google/cloud/hooks/tasks.py | 3 +- .../providers/google/cloud/hooks/text_to_speech.py | 3 +- .../providers/google/cloud/hooks/translate.py | 244 ++- .../google/cloud/hooks/vertex_ai/auto_ml.py | 3 +- .../cloud/hooks/vertex_ai/batch_prediction_job.py | 3 +- .../google/cloud/hooks/vertex_ai/custom_job.py | 3 +- .../google/cloud/hooks/vertex_ai/dataset.py | 3 +- .../cloud/hooks/vertex_ai/endpoint_service.py | 3 +- .../hooks/vertex_ai/hyperparameter_tuning_job.py | 3 +- .../google/cloud/hooks/vertex_ai/model_service.py | 3 +- .../google/cloud/hooks/vertex_ai/pipeline_job.py | 3 +- .../cloud/hooks/vertex_ai/prediction_service.py | 3 +- .../google/cloud/hooks/video_intelligence.py | 3 +- .../airflow/providers/google/cloud/hooks/vision.py | 3 +- .../providers/google/cloud/hooks/workflows.py | 3 +- .../providers/google/cloud/links/translate.py | 56 + .../providers/google/cloud/log/gcs_task_handler.py | 3 +- .../google/cloud/log/stackdriver_task_handler.py | 3 +- .../providers/google/cloud/openlineage/utils.py | 131 +- .../providers/google/cloud/operators/automl.py | 5 +- .../providers/google/cloud/operators/bigquery.py | 3 +- .../google/cloud/operators/bigquery_dts.py | 3 +- .../providers/google/cloud/operators/bigtable.py | 3 +- .../google/cloud/operators/cloud_batch.py | 3 +- .../google/cloud/operators/cloud_build.py | 3 +- .../google/cloud/operators/cloud_composer.py | 3 +- .../google/cloud/operators/cloud_memorystore.py | 3 +- .../providers/google/cloud/operators/cloud_run.py | 3 +- .../providers/google/cloud/operators/cloud_sql.py | 3 +- .../operators/cloud_storage_transfer_service.py | 3 +- .../providers/google/cloud/operators/compute.py | 3 +- .../google/cloud/operators/datacatalog.py | 3 +- .../providers/google/cloud/operators/dataflow.py | 3 +- .../providers/google/cloud/operators/dataform.py | 3 +- .../providers/google/cloud/operators/datafusion.py | 3 +- .../providers/google/cloud/operators/dataplex.py | 68 +- .../providers/google/cloud/operators/dataprep.py | 3 +- .../providers/google/cloud/operators/dataproc.py | 4 +- .../google/cloud/operators/dataproc_metastore.py | 3 +- .../providers/google/cloud/operators/datastore.py | 3 +- .../providers/google/cloud/operators/dlp.py | 3 +- .../providers/google/cloud/operators/functions.py | 3 +- .../providers/google/cloud/operators/gcs.py | 22 +- .../google/cloud/operators/kubernetes_engine.py | 3 +- .../google/cloud/operators/life_sciences.py | 3 +- .../providers/google/cloud/operators/mlengine.py | 3 +- .../google/cloud/operators/natural_language.py | 5 +- .../providers/google/cloud/operators/pubsub.py | 3 +- .../providers/google/cloud/operators/spanner.py | 3 +- .../google/cloud/operators/speech_to_text.py | 3 +- .../google/cloud/operators/stackdriver.py | 3 +- .../providers/google/cloud/operators/tasks.py | 5 +- .../google/cloud/operators/text_to_speech.py | 3 +- .../providers/google/cloud/operators/translate.py | 347 +++- .../google/cloud/operators/translate_speech.py | 3 +- .../google/cloud/operators/vertex_ai/auto_ml.py | 3 +- .../operators/vertex_ai/batch_prediction_job.py | 3 +- .../google/cloud/operators/vertex_ai/custom_job.py | 3 +- .../google/cloud/operators/vertex_ai/dataset.py | 3 +- .../cloud/operators/vertex_ai/endpoint_service.py | 3 +- .../cloud/operators/vertex_ai/generative_model.py | 3 +- .../vertex_ai/hyperparameter_tuning_job.py | 3 +- .../cloud/operators/vertex_ai/model_service.py | 3 +- .../cloud/operators/vertex_ai/pipeline_job.py | 3 +- .../google/cloud/operators/video_intelligence.py | 3 +- .../providers/google/cloud/operators/vision.py | 5 +- .../providers/google/cloud/operators/workflows.py | 3 +- .../google/cloud/secrets/secret_manager.py | 2 +- .../providers/google/cloud/sensors/bigquery.py | 3 +- .../providers/google/cloud/sensors/bigquery_dts.py | 3 +- .../providers/google/cloud/sensors/bigtable.py | 3 +- .../google/cloud/sensors/cloud_composer.py | 3 +- .../sensors/cloud_storage_transfer_service.py | 3 +- .../providers/google/cloud/sensors/dataflow.py | 3 +- .../providers/google/cloud/sensors/dataform.py | 3 +- .../providers/google/cloud/sensors/datafusion.py | 3 +- .../providers/google/cloud/sensors/dataplex.py | 3 +- .../providers/google/cloud/sensors/dataprep.py | 3 +- .../providers/google/cloud/sensors/dataproc.py | 3 +- .../google/cloud/sensors/dataproc_metastore.py | 3 +- .../airflow/providers/google/cloud/sensors/gcs.py | 3 +- .../providers/google/cloud/sensors/pubsub.py | 3 +- .../providers/google/cloud/sensors/tasks.py | 3 +- .../providers/google/cloud/sensors/workflows.py | 3 +- .../google/cloud/transfers/adls_to_gcs.py | 3 +- .../google/cloud/transfers/azure_blob_to_gcs.py | 3 +- .../cloud/transfers/azure_fileshare_to_gcs.py | 3 +- .../google/cloud/transfers/bigquery_to_bigquery.py | 93 +- .../google/cloud/transfers/bigquery_to_gcs.py | 33 +- .../google/cloud/transfers/bigquery_to_mssql.py | 3 +- .../google/cloud/transfers/bigquery_to_mysql.py | 2 +- .../google/cloud/transfers/bigquery_to_sql.py | 3 +- .../google/cloud/transfers/calendar_to_gcs.py | 3 +- .../google/cloud/transfers/cassandra_to_gcs.py | 3 +- .../google/cloud/transfers/facebook_ads_to_gcs.py | 3 +- .../google/cloud/transfers/gcs_to_bigquery.py | 37 +- .../providers/google/cloud/transfers/gcs_to_gcs.py | 21 +- .../google/cloud/transfers/gcs_to_local.py | 12 +- .../google/cloud/transfers/gcs_to_sftp.py | 3 +- .../google/cloud/transfers/gdrive_to_gcs.py | 3 +- .../google/cloud/transfers/gdrive_to_local.py | 3 +- .../google/cloud/transfers/local_to_gcs.py | 50 +- .../google/cloud/transfers/mssql_to_gcs.py | 2 +- .../providers/google/cloud/transfers/s3_to_gcs.py | 18 +- .../google/cloud/transfers/salesforce_to_gcs.py | 3 +- .../google/cloud/transfers/sftp_to_gcs.py | 3 +- .../google/cloud/transfers/sheets_to_gcs.py | 3 +- .../providers/google/cloud/transfers/sql_to_gcs.py | 3 +- .../providers/google/cloud/triggers/bigquery.py | 3 +- .../google/cloud/triggers/bigquery_dts.py | 3 +- .../providers/google/cloud/triggers/cloud_batch.py | 3 +- .../providers/google/cloud/triggers/cloud_build.py | 3 +- .../google/cloud/triggers/cloud_composer.py | 3 +- .../providers/google/cloud/triggers/cloud_run.py | 3 +- .../providers/google/cloud/triggers/cloud_sql.py | 2 +- .../triggers/cloud_storage_transfer_service.py | 3 +- .../providers/google/cloud/triggers/dataflow.py | 3 +- .../providers/google/cloud/triggers/datafusion.py | 3 +- .../providers/google/cloud/triggers/dataplex.py | 2 +- .../providers/google/cloud/triggers/dataproc.py | 3 +- .../airflow/providers/google/cloud/triggers/gcs.py | 3 +- .../google/cloud/triggers/kubernetes_engine.py | 3 +- .../providers/google/cloud/triggers/mlengine.py | 3 +- .../providers/google/cloud/triggers/pubsub.py | 3 +- .../providers/google/cloud/triggers/vertex_ai.py | 3 +- .../google/cloud/utils/credentials_provider.py | 2 +- .../providers/google/cloud/utils/dataform.py | 2 +- .../google/cloud/utils/field_validator.py | 3 +- .../google/cloud/utils/mlengine_operator_utils.py | 3 +- .../providers/google/common/hooks/base_google.py | 3 +- .../providers/google/common/hooks/discovery_api.py | 2 +- .../providers/google/firebase/hooks/firestore.py | 2 +- .../google/firebase/operators/firestore.py | 3 +- .../marketing_platform/hooks/analytics_admin.py | 3 +- .../marketing_platform/hooks/campaign_manager.py | 3 +- .../marketing_platform/hooks/display_video.py | 3 +- .../google/marketing_platform/hooks/search_ads.py | 3 +- .../operators/analytics_admin.py | 3 +- .../operators/campaign_manager.py | 3 +- .../marketing_platform/operators/display_video.py | 3 +- .../marketing_platform/operators/search_ads.py | 3 +- .../marketing_platform/sensors/campaign_manager.py | 3 +- .../marketing_platform/sensors/display_video.py | 3 +- .../src/airflow/providers/google/provider.yaml | 9 +- .../providers/google/suite/hooks/calendar.py | 3 +- .../airflow/providers/google/suite/hooks/drive.py | 3 +- .../airflow/providers/google/suite/hooks/sheets.py | 3 +- .../providers/google/suite/operators/sheets.py | 3 +- .../providers/google/suite/sensors/drive.py | 3 +- .../google/suite/transfers/gcs_to_gdrive.py | 3 +- .../google/suite/transfers/gcs_to_sheets.py | 3 +- .../google/suite/transfers/local_to_drive.py | 3 +- .../google/suite/transfers/sql_to_sheets.py | 3 +- providers/src/airflow/providers/grpc/hooks/grpc.py | 3 +- .../src/airflow/providers/grpc/operators/grpc.py | 3 +- .../src/airflow/providers/http/operators/http.py | 3 +- .../src/airflow/providers/http/sensors/http.py | 3 +- .../src/airflow/providers/http/triggers/http.py | 3 +- providers/src/airflow/providers/imap/hooks/imap.py | 3 +- .../providers/imap/sensors/imap_attachment.py | 3 +- .../providers/influxdb/operators/influxdb.py | 3 +- providers/src/airflow/providers/jdbc/hooks/jdbc.py | 9 +- .../src/airflow/providers/jdbc/operators/jdbc.py | 3 +- .../jenkins/operators/jenkins_job_trigger.py | 3 +- .../airflow/providers/jenkins/sensors/jenkins.py | 3 +- .../providers/microsoft/azure/hooks/cosmos.py | 4 +- .../providers/microsoft/azure/operators/adls.py | 3 +- .../providers/microsoft/azure/operators/adx.py | 3 +- .../providers/microsoft/azure/operators/asb.py | 3 +- .../providers/microsoft/azure/operators/batch.py | 3 +- .../azure/operators/container_instances.py | 3 +- .../providers/microsoft/azure/operators/cosmos.py | 3 +- .../microsoft/azure/operators/data_factory.py | 3 +- .../providers/microsoft/azure/operators/msgraph.py | 4 +- .../providers/microsoft/azure/operators/powerbi.py | 3 +- .../providers/microsoft/azure/operators/synapse.py | 3 +- .../microsoft/azure/operators/wasb_delete_blob.py | 3 +- .../providers/microsoft/azure/sensors/cosmos.py | 3 +- .../microsoft/azure/sensors/data_factory.py | 3 +- .../providers/microsoft/azure/sensors/msgraph.py | 5 +- .../providers/microsoft/azure/sensors/wasb.py | 3 +- .../microsoft/azure/transfers/local_to_adls.py | 3 +- .../microsoft/azure/transfers/local_to_wasb.py | 3 +- .../azure/transfers/oracle_to_azure_data_lake.py | 3 +- .../microsoft/azure/transfers/s3_to_wasb.py | 3 +- .../microsoft/azure/transfers/sftp_to_wasb.py | 3 +- .../microsoft/azure/triggers/data_factory.py | 3 +- .../providers/microsoft/azure/triggers/msgraph.py | 3 +- .../providers/microsoft/azure/triggers/powerbi.py | 3 +- .../providers/microsoft/azure/triggers/wasb.py | 3 +- .../providers/microsoft/mssql/hooks/mssql.py | 2 + .../providers/microsoft/mssql/operators/mssql.py | 3 +- .../airflow/providers/microsoft/psrp/hooks/psrp.py | 3 +- .../providers/microsoft/winrm/operators/winrm.py | 3 +- .../src/airflow/providers/mongo/hooks/mongo.py | 3 +- .../src/airflow/providers/mongo/sensors/mongo.py | 3 +- .../src/airflow/providers/mysql/operators/mysql.py | 3 +- .../providers/mysql/transfers/presto_to_mysql.py | 3 +- .../providers/mysql/transfers/s3_to_mysql.py | 3 +- .../providers/mysql/transfers/trino_to_mysql.py | 3 +- .../providers/mysql/transfers/vertica_to_mysql.py | 3 +- .../src/airflow/providers/neo4j/operators/neo4j.py | 3 +- providers/src/airflow/providers/odbc/hooks/odbc.py | 5 +- .../airflow/providers/openai/operators/openai.py | 3 +- .../airflow/providers/openai/triggers/openai.py | 3 +- .../providers/openlineage/extractors/manager.py | 3 +- .../providers/openlineage/plugins/adapter.py | 7 +- .../providers/openlineage/plugins/listener.py | 11 + .../providers/openlineage/plugins/macros.py | 1 + .../src/airflow/providers/openlineage/utils/sql.py | 4 +- .../airflow/providers/openlineage/utils/utils.py | 37 +- .../providers/opensearch/log/os_response.py | 2 +- .../providers/opensearch/log/os_task_handler.py | 4 +- .../providers/opensearch/operators/opensearch.py | 3 +- .../providers/opsgenie/notifications/opsgenie.py | 3 +- .../providers/opsgenie/operators/opsgenie.py | 3 +- .../airflow/providers/oracle/operators/oracle.py | 3 +- .../providers/oracle/transfers/oracle_to_oracle.py | 3 +- .../providers/papermill/operators/papermill.py | 3 +- .../airflow/providers/pinecone/hooks/pinecone.py | 2 +- .../providers/pinecone/operators/pinecone.py | 3 +- .../airflow/providers/postgres/hooks/postgres.py | 3 +- .../providers/postgres/operators/postgres.py | 3 +- .../src/airflow/providers/presto/hooks/presto.py | 3 +- .../providers/presto/transfers/gcs_to_presto.py | 3 +- .../airflow/providers/qdrant/operators/qdrant.py | 3 +- .../providers/redis/operators/redis_publish.py | 3 +- .../airflow/providers/redis/sensors/redis_key.py | 3 +- .../providers/redis/sensors/redis_pub_sub.py | 3 +- .../providers/salesforce/hooks/salesforce.py | 3 +- .../airflow/providers/salesforce/operators/bulk.py | 3 +- .../providers/samba/transfers/gcs_to_samba.py | 3 +- .../segment/operators/segment_track_event.py | 3 +- .../airflow/providers/sendgrid/utils/emailer.py | 3 +- .../providers/sftp/decorators/sensors/sftp.py | 3 +- providers/src/airflow/providers/sftp/hooks/sftp.py | 3 +- .../src/airflow/providers/sftp/operators/sftp.py | 3 +- .../src/airflow/providers/sftp/sensors/sftp.py | 3 +- .../src/airflow/providers/sftp/triggers/sftp.py | 3 +- .../providers/singularity/operators/singularity.py | 3 +- .../src/airflow/providers/slack/hooks/slack.py | 3 +- .../airflow/providers/slack/notifications/slack.py | 3 +- .../src/airflow/providers/slack/operators/slack.py | 3 +- .../providers/slack/operators/slack_webhook.py | 3 +- .../providers/slack/transfers/base_sql_to_slack.py | 3 +- .../providers/slack/transfers/sql_to_slack.py | 3 +- .../slack/transfers/sql_to_slack_webhook.py | 3 +- .../src/airflow/providers/slack/utils/__init__.py | 3 +- providers/src/airflow/providers/smtp/hooks/smtp.py | 3 +- .../airflow/providers/smtp/notifications/smtp.py | 3 +- .../src/airflow/providers/smtp/operators/smtp.py | 3 +- .../providers/snowflake/decorators/snowpark.py | 3 +- .../airflow/providers/snowflake/hooks/snowflake.py | 20 +- .../providers/snowflake/operators/snowflake.py | 5 +- .../providers/snowflake/operators/snowpark.py | 3 +- .../snowflake/transfers/copy_into_snowflake.py | 3 +- .../snowflake/triggers/snowflake_trigger.py | 3 +- .../airflow/providers/sqlite/operators/sqlite.py | 3 +- providers/src/airflow/providers/ssh/hooks/ssh.py | 3 +- .../src/airflow/providers/ssh/operators/ssh.py | 3 +- .../src/airflow/providers/standard/CHANGELOG.rst | 21 + .../src/airflow/providers/standard/__init__.py | 2 +- .../airflow/providers/standard/hooks/subprocess.py | 2 +- .../airflow/providers/standard/operators/bash.py | 3 +- .../providers/standard/operators/datetime.py | 3 +- .../standard/operators/generic_transfer.py | 3 +- .../airflow/providers/standard/operators/python.py | 9 +- .../providers/standard/operators/trigger_dagrun.py | 33 +- .../providers/standard/operators/weekday.py | 3 +- .../src/airflow/providers/standard/provider.yaml | 11 +- .../src/airflow/providers/standard/sensors/bash.py | 10 +- .../providers/standard/sensors/date_time.py | 7 +- .../providers/standard}/sensors/external_task.py | 7 +- .../providers/standard/sensors/filesystem.py | 24 +- .../airflow/providers/standard/sensors/python.py | 3 +- .../src/airflow/providers/standard/sensors/time.py | 4 +- .../providers/standard/sensors/time_delta.py | 2 +- .../airflow/providers/standard/sensors/weekday.py | 3 +- .../providers/standard/triggers}/__init__.py | 0 .../providers/standard}/triggers/external_task.py | 44 +- .../airflow/providers/standard}/triggers/file.py | 14 +- .../providers/standard}/triggers/temporal.py | 15 +- .../providers/standard}/utils/sensor_helper.py | 0 .../airflow/providers/tableau/operators/tableau.py | 3 +- .../airflow/providers/tableau/sensors/tableau.py | 3 +- .../providers/telegram/operators/telegram.py | 3 +- .../providers/teradata/operators/teradata.py | 3 +- .../teradata/operators/teradata_compute_cluster.py | 3 +- .../teradata/transfers/azure_blob_to_teradata.py | 3 +- .../providers/teradata/transfers/s3_to_teradata.py | 3 +- .../teradata/transfers/teradata_to_teradata.py | 3 +- .../teradata/triggers/teradata_compute_cluster.py | 3 +- .../src/airflow/providers/trino/hooks/trino.py | 3 +- .../src/airflow/providers/trino/operators/trino.py | 3 +- .../providers/trino/transfers/gcs_to_trino.py | 3 +- .../src/airflow/providers/vertica/hooks/vertica.py | 3 +- .../airflow/providers/vertica/operators/vertica.py | 3 +- .../airflow/providers/weaviate/hooks/weaviate.py | 13 +- .../providers/weaviate/operators/weaviate.py | 3 +- .../airflow/providers/yandex/operators/dataproc.py | 3 +- .../src/airflow/providers/yandex/operators/yq.py | 3 +- providers/src/airflow/providers/ydb/hooks/ydb.py | 3 +- .../src/airflow/providers/ydb/operators/ydb.py | 2 +- providers/tests/airbyte/hooks/test_airbyte.py | 2 +- .../aws/auth_manager/cli/test_avp_commands.py | 2 - .../test_aws_security_manager_override.py | 2 - .../aws/auth_manager/test_aws_auth_manager.py | 5 +- .../amazon/aws/auth_manager/views/test_auth.py | 5 +- providers/tests/amazon/aws/hooks/test_base_aws.py | 2 +- providers/tests/amazon/aws/hooks/test_neptune.py | 2 +- providers/tests/amazon/aws/hooks/test_rds.py | 3 +- .../tests/amazon/aws/operators/test_bedrock.py | 3 +- .../tests/amazon/aws/operators/test_comprehend.py | 3 +- .../aws/operators/test_emr_create_job_flow.py | 14 +- providers/tests/amazon/aws/operators/test_glue.py | 3 +- .../amazon/aws/operators/test_glue_crawler.py | 3 +- .../amazon/aws/operators/test_glue_databrew.py | 2 +- .../amazon/aws/operators/test_kinesis_analytics.py | 3 +- .../tests/amazon/aws/operators/test_neptune.py | 2 +- .../aws/operators/test_sagemaker_notebook.py | 2 +- .../amazon/aws/transfers/test_google_api_to_s3.py | 2 +- .../tests/amazon/aws/transfers/test_sftp_to_s3.py | 37 + .../tests/amazon/aws/utils/eks_test_constants.py | 3 +- providers/tests/amazon/aws/utils/eks_test_utils.py | 5 +- providers/tests/amazon/aws/waiters/test_emr.py | 2 +- providers/tests/apache/beam/operators/test_beam.py | 10 +- providers/tests/apache/beam/triggers/test_beam.py | 15 + providers/tests/apache/druid/hooks/test_druid.py | 3 - .../tests/apache/kafka/operators/test_produce.py | 2 +- providers/tests/apache/livy/hooks/test_livy.py | 2 - providers/tests/arangodb/sensors/test_arangodb.py | 2 +- .../tests/asana/operators/test_asana_tasks.py | 2 +- .../tests/celery/log_handlers/test_log_handlers.py | 2 +- .../executors/test_kubernetes_executor.py | 2 - .../kubernetes/log_handlers/test_log_handlers.py | 2 +- .../tests/cncf/kubernetes/operators/test_pod.py | 2 +- .../cncf/kubernetes/test_template_rendering.py | 2 +- providers/tests/common/io/assets/test_file.py | 4 +- providers/tests/common/io/xcom/test_backend.py | 121 +- providers/tests/common/sql/operators/test_sql.py | 5 +- .../tests/common/sql/operators/test_sql_execute.py | 3 +- providers/tests/common/sql/sensors/test_sql.py | 5 +- .../tests/databricks/hooks/test_databricks_sql.py | 9 +- providers/tests/edge/cli/test_edge_command.py | 9 + .../tests/edge/executors/test_edge_executor.py | 109 +- providers/tests/edge/models/test_edge_job.py | 44 +- .../tests/edge/worker_api/routes/test_rpc_api.py | 297 --- .../api_endpoints/test_asset_endpoint.py | 3 +- .../fab/auth_manager/api_endpoints/test_auth.py | 1 - .../fab/auth_manager/api_endpoints/test_cors.py | 1 - .../api_endpoints/test_dag_endpoint.py | 1 - .../api_endpoints/test_dag_run_endpoint.py | 1 - .../api_endpoints/test_dag_source_endpoint.py | 1 - .../api_endpoints/test_dag_warning_endpoint.py | 1 - .../api_endpoints/test_event_log_endpoint.py | 1 - .../api_endpoints/test_import_error_endpoint.py | 1 - .../api_endpoints/test_task_instance_endpoint.py | 1 - .../api_endpoints/test_variable_endpoint.py | 1 - .../api_endpoints/test_xcom_endpoint.py | 1 - .../schemas/test_role_and_permission_schema.py | 2 +- .../tests/google/cloud/hooks/test_bigquery.py | 2 +- .../tests/google/cloud/hooks/test_datacatalog.py | 3 +- .../tests/google/cloud/hooks/test_os_login.py | 3 +- .../tests/google/cloud/openlineage/test_utils.py | 160 +- .../google/cloud/operators/test_datacatalog.py | 19 +- .../tests/google/cloud/operators/test_dataplex.py | 22 + .../tests/google/cloud/operators/test_translate.py | 205 ++- providers/tests/google/cloud/sensors/test_gcs.py | 4 +- .../cloud/transfers/test_bigquery_to_bigquery.py | 332 +++- .../google/cloud/transfers/test_bigquery_to_gcs.py | 21 +- .../google/cloud/transfers/test_gcs_to_bigquery.py | 11 +- .../google/cloud/transfers/test_gcs_to_local.py | 17 + .../google/cloud/transfers/test_local_to_gcs.py | 87 +- .../tests/google/cloud/transfers/test_s3_to_gcs.py | 32 + .../common/auth_backend/test_google_openid.py | 1 - .../tests/google/common/hooks/test_base_google.py | 2 +- .../operators/test_campaign_manager.py | 1 - .../operators/test_display_video.py | 1 - providers/tests/microsoft/azure/base.py | 1 + .../microsoft/azure/operators/test_msgraph.py | 32 +- .../tests/microsoft/azure/sensors/test_msgraph.py | 30 + providers/tests/microsoft/conftest.py | 11 +- .../tests/microsoft/mssql/hooks/test_mssql.py | 308 ++-- .../tests/openlineage/plugins/test_adapter.py | 23 + .../tests/openlineage/plugins/test_execution.py | 2 - .../tests/openlineage/plugins/test_listener.py | 59 +- providers/tests/snowflake/hooks/test_snowflake.py | 22 + .../snowflake/operators/test_snowflake_sql.py | 54 +- providers/tests/standard/operators/test_python.py | 40 +- providers/tests/standard/sensors/test_time.py | 2 +- .../tests/standard/sensors/test_time_delta.py | 1 - providers/tests/standard/sensors/test_weekday.py | 3 - .../tests/standard/triggers}/__init__.py | 0 .../tests/standard}/triggers/test_external_task.py | 58 +- .../tests/standard}/triggers/test_file.py | 6 +- .../tests/standard}/triggers/test_temporal.py | 52 +- .../aws/example_comprehend_document_classifier.py | 127 +- .../cloud/cloud_sql/example_cloud_sql_query.py | 3 +- .../cloud/cloud_sql/example_cloud_sql_query_iam.py | 3 +- .../cloud/cloud_sql/example_cloud_sql_query_ssl.py | 3 +- .../dataflow/example_dataflow_native_python.py | 18 +- .../cloud/translate/example_translate_dataset.py | 153 ++ .../tests/system/openlineage}/__init__.py | 0 .../tests/system/openlineage}/conftest.py | 28 +- .../system/openlineage/example_openlineage.json | 38 + .../system/openlineage/example_openlineage.py | 55 + .../example_openlineage_mapped_sensor.json | 75 + .../example_openlineage_mapped_sensor.py | 81 + providers/tests/system/openlineage/operator.py | 214 +++ .../system/openlineage/transport}/__init__.py | 0 .../tests/system/openlineage/transport/variable.py | 49 + pyproject.toml | 23 +- scripts/ci/docker-compose/devcontainer.env | 1 - scripts/ci/install_breeze.sh | 2 +- .../ci/pre_commit/check_common_sql_dependency.py | 2 +- scripts/ci/pre_commit/check_deferrable_default.py | 2 +- scripts/ci/pre_commit/check_deprecations.py | 4 +- .../ci/pre_commit/common_precommit_black_utils.py | 4 +- .../decorator_operator_implements_custom_name.py | 2 +- .../ci/pre_commit/update_providers_dependencies.py | 4 +- scripts/ci/pre_commit/validate_operators_init.py | 8 +- scripts/ci/pre_commit/vendor_k8s_json_schema.py | 2 +- scripts/ci/testing/summarize_captured_warnings.py | 6 +- scripts/ci/testing/summarize_junit_failures.py | 4 +- scripts/cov/other_coverage.py | 3 - scripts/cov/restapi_coverage.py | 2 +- scripts/docker/entrypoint_ci.sh | 94 +- scripts/in_container/bin/run_tmux | 6 - scripts/in_container/check_connectivity.sh | 65 + scripts/in_container/check_environment.sh | 40 +- .../in_container/install_airflow_python_client.py | 91 + scripts/in_container/run_migration_reference.py | 3 +- .../in_container/run_provider_yaml_files_check.py | 3 +- scripts/in_container/verify_providers.py | 25 +- scripts/tools/setup_breeze | 2 +- task_sdk/pyproject.toml | 5 +- task_sdk/src/airflow/sdk/api/client.py | 55 +- .../src/airflow/sdk/api/datamodels/_generated.py | 14 +- .../src/airflow/sdk/definitions/asset/__init__.py | 88 +- .../airflow/sdk/definitions/asset/decorators.py | 85 +- task_sdk/src/airflow/sdk/definitions/dag.py | 6 +- task_sdk/src/airflow/sdk/execution_time/comms.py | 46 +- .../src/airflow/sdk/execution_time/supervisor.py | 177 +- .../src/airflow/sdk/execution_time/task_runner.py | 32 +- task_sdk/src/airflow/sdk/log.py | 81 +- task_sdk/tests/api/test_client.py | 59 + .../tests/dags/super_basic_deferred_run.py | 24 +- task_sdk/tests/defintions/test_asset.py | 1 - task_sdk/tests/defintions/test_asset_decorators.py | 2 +- task_sdk/tests/execution_time/test_supervisor.py | 197 +- task_sdk/tests/execution_time/test_task_runner.py | 41 +- tests/always/test_example_dags.py | 1 - tests/always/test_secrets_local_filesystem.py | 7 +- .../api_connexion/endpoints/test_asset_endpoint.py | 4 +- .../endpoints/test_config_endpoint.py | 2 +- .../endpoints/test_connection_endpoint.py | 2 +- tests/api_connexion/endpoints/test_dag_endpoint.py | 2 +- tests/api_connexion/endpoints/test_dag_parsing.py | 2 +- .../endpoints/test_dag_run_endpoint.py | 6 +- .../endpoints/test_dag_source_endpoint.py | 2 +- .../endpoints/test_dag_stats_endpoint.py | 2 +- .../endpoints/test_dag_warning_endpoint.py | 2 +- .../endpoints/test_event_log_endpoint.py | 2 +- .../endpoints/test_extra_link_endpoint.py | 2 +- .../endpoints/test_health_endpoint.py | 2 +- .../endpoints/test_import_error_endpoint.py | 2 +- tests/api_connexion/endpoints/test_log_endpoint.py | 2 +- .../test_mapped_task_instance_endpoint.py | 2 +- .../endpoints/test_plugin_endpoint.py | 2 +- .../api_connexion/endpoints/test_pool_endpoint.py | 2 +- .../endpoints/test_provider_endpoint.py | 2 +- .../api_connexion/endpoints/test_task_endpoint.py | 2 +- .../endpoints/test_task_instance_endpoint.py | 6 +- .../endpoints/test_variable_endpoint.py | 2 +- .../endpoints/test_version_endpoint.py | 2 +- .../api_connexion/endpoints/test_xcom_endpoint.py | 2 +- tests/api_connexion/schemas/test_asset_schema.py | 2 +- .../schemas/test_connection_schema.py | 2 +- tests/api_connexion/schemas/test_dag_run_schema.py | 2 +- tests/api_connexion/schemas/test_dag_schema.py | 2 - tests/api_connexion/schemas/test_error_schema.py | 2 +- .../api_connexion/schemas/test_event_log_schema.py | 2 +- tests/api_connexion/schemas/test_pool_schemas.py | 2 +- .../schemas/test_task_instance_schema.py | 1 - tests/api_connexion/schemas/test_xcom_schema.py | 2 +- tests/api_connexion/test_auth.py | 2 +- tests/api_connexion/test_error_handling.py | 2 +- tests/api_connexion/test_security.py | 2 +- tests/api_fastapi/conftest.py | 9 + .../core_api/routes/public/test_assets.py | 111 +- .../core_api/routes/public/test_config.py | 8 +- .../core_api/routes/public/test_connections.py | 12 +- .../core_api/routes/public/test_dag_parsing.py | 80 + .../core_api/routes/public/test_dag_run.py | 816 ++++++++- .../core_api/routes/public/test_dags.py | 4 +- .../core_api/routes/public/test_event_logs.py | 7 +- .../core_api/routes/public/test_extra_links.py | 221 +++ .../core_api/routes/public/test_import_error.py | 3 +- .../api_fastapi/core_api/routes/public/test_job.py | 165 ++ .../core_api/routes/public/test_log.py} | 192 +- .../core_api/routes/public/test_monitor.py | 2 +- .../core_api/routes/public/test_plugins.py | 2 +- .../core_api/routes/public/test_pools.py | 135 +- .../core_api/routes/public/test_providers.py | 2 +- .../core_api/routes/public/test_task_instances.py | 1408 +++++++++++++- .../core_api/routes/public/test_variables.py | 4 +- .../core_api/routes/public/test_version.py | 2 +- .../core_api/routes/public/test_xcom.py | 297 ++- .../api_fastapi/core_api/routes/ui/test_config.py | 115 ++ tests/api_fastapi/core_api/routes/ui/test_dags.py | 2 +- tests/api_fastapi/core_api/routes/ui/test_graph.py | 198 ++ .../execution_api/routes/test_task_instances.py | 59 +- tests/api_internal/endpoints/__init__.py | 16 - .../endpoints/test_rpc_api_endpoint.py | 249 --- tests/api_internal/test_internal_api_call.py | 327 ---- tests/assets/test_manager.py | 2 - tests/charts/helm_template_generator.py | 4 +- tests/cli/commands/test_asset_command.py | 65 + tests/cli/commands/test_backfill_command.py | 2 +- tests/cli/commands/test_celery_command.py | 2 +- tests/cli/commands/test_connection_command.py | 2 +- tests/cli/commands/test_dag_command.py | 47 +- tests/cli/commands/test_dag_processor_command.py | 2 +- tests/cli/commands/test_db_command.py | 2 +- tests/cli/commands/test_fastapi_api_command.py | 18 +- tests/cli/commands/test_internal_api_command.py | 221 --- tests/cli/commands/test_jobs_command.py | 1 - tests/cli/commands/test_kerberos_command.py | 2 +- tests/cli/commands/test_kubernetes_command.py | 2 +- tests/cli/commands/test_plugins_command.py | 2 +- tests/cli/commands/test_pool_command.py | 2 +- .../cli/commands/test_rotate_fernet_key_command.py | 36 +- tests/cli/commands/test_scheduler_command.py | 2 +- tests/cli/commands/test_task_command.py | 24 +- tests/cli/commands/test_triggerer_command.py | 2 +- tests/cli/commands/test_variable_command.py | 2 +- tests/cli/commands/test_webserver_command.py | 7 +- tests/cli/test_cli_parser.py | 14 +- tests/core/test_configuration.py | 11 +- tests/core/test_core.py | 2 +- tests/core/test_sentry.py | 2 - tests/core/test_settings.py | 85 +- tests/core/test_sqlalchemy_config.py | 7 +- tests/dag_processing/test_job_runner.py | 17 +- tests/dag_processing/test_processor.py | 20 - .../test_external_task_sensor_check_existense.py | 2 +- tests/dags/test_only_empty_tasks.py | 2 +- tests/dags/test_retry_handling_job.py | 38 - tests/decorators/test_bash.py | 15 - tests/decorators/test_branch_external_python.py | 1 - tests/decorators/test_branch_python.py | 1 - tests/decorators/test_branch_virtualenv.py | 1 - tests/decorators/test_condition.py | 4 - tests/decorators/test_python.py | 38 +- tests/decorators/test_sensor.py | 6 - tests/decorators/test_short_circuit.py | 1 - tests/executors/test_base_executor.py | 2 - tests/executors/test_executor_loader.py | 3 - tests/executors/test_local_executor.py | 79 +- .../cli/commands/test_celery_command.py | 10 +- .../integration/executors/test_celery_executor.py | 14 +- tests/jobs/test_base_job.py | 2 - tests/jobs/test_local_task_job.py | 17 - tests/jobs/test_scheduler_job.py | 307 ++-- tests/jobs/test_triggerer_job.py | 14 +- tests/lineage/test_lineage.py | 2 - tests/listeners/test_asset_listener.py | 1 - tests/listeners/test_dag_import_error_listener.py | 2 - tests/listeners/test_listeners.py | 5 - tests/models/test_base.py | 2 +- tests/models/test_baseoperator.py | 2 - tests/models/test_baseoperatormeta.py | 7 - tests/models/test_cleartasks.py | 2 +- tests/models/test_dag.py | 9 +- tests/models/test_dag_version.py | 2 +- tests/models/test_dagbag.py | 11 - tests/models/test_dagcode.py | 69 +- tests/models/test_dagrun.py | 5 +- tests/models/test_dagwarning.py | 2 +- tests/models/test_mappedoperator.py | 26 - tests/models/test_param.py | 3 - tests/models/test_pool.py | 2 +- tests/models/test_renderedtifields.py | 7 - tests/models/test_serialized_dag.py | 12 - tests/models/test_skipmixin.py | 2 - tests/models/test_taskinstance.py | 94 - tests/models/test_trigger.py | 8 +- tests/models/test_variable.py | 10 - tests/models/test_xcom.py | 9 +- tests/models/test_xcom_arg.py | 1 - tests/models/test_xcom_arg_map.py | 8 - tests/operators/test_trigger_dagrun.py | 42 +- tests/plugins/test_plugins_manager.py | 28 +- tests/sensors/test_base.py | 1 - tests/sensors/test_external_task_sensor.py | 55 +- tests/sensors/test_filesystem.py | 3 +- .../serialization/serializers/test_serializers.py | 22 +- tests/serialization/test_dag_serialization.py | 75 +- tests/serialization/test_pydantic_models.py | 297 --- tests/serialization/test_serialized_objects.py | 6 +- .../example_external_task_parent_deferrable.py | 2 +- tests/task/test_standard_task_runner.py | 5 - .../deps/test_dag_ti_slots_available_dep.py | 2 +- tests/ti_deps/deps/test_dag_unpaused_dep.py | 2 +- tests/ti_deps/deps/test_dagrun_exists_dep.py | 2 +- .../ti_deps/deps/test_mapped_task_upstream_dep.py | 2 +- tests/ti_deps/deps/test_not_in_retry_period_dep.py | 2 +- .../deps/test_not_previously_skipped_dep.py | 2 +- .../ti_deps/deps/test_pool_slots_available_dep.py | 2 +- tests/ti_deps/deps/test_prev_dagrun_dep.py | 2 +- tests/ti_deps/deps/test_ready_to_reschedule_dep.py | 2 +- tests/ti_deps/deps/test_runnable_exec_date_dep.py | 2 +- tests/ti_deps/deps/test_task_concurrency.py | 2 +- tests/ti_deps/deps/test_task_not_running_dep.py | 2 +- tests/ti_deps/deps/test_trigger_rule_dep.py | 5 +- tests/ti_deps/deps/test_valid_state_dep.py | 2 +- tests/triggers/__init__.py | 16 - tests/utils/log/test_colored_log.py | 2 +- tests/utils/log/test_log_reader.py | 2 +- tests/utils/test_cli_util.py | 16 +- tests/utils/test_db.py | 2 +- tests/utils/test_db_cleanup.py | 13 +- tests/utils/test_dot_renderer.py | 2 +- tests/utils/test_email.py | 2 - tests/utils/test_entry_points.py | 2 +- tests/utils/test_file.py | 1 - tests/utils/test_helpers.py | 2 - tests/utils/test_log_handlers.py | 2 +- tests/utils/test_process_utils.py | 4 - tests/utils/test_retries.py | 2 - tests/utils/test_session.py | 6 +- tests/utils/test_sqlalchemy.py | 4 +- tests/utils/test_state.py | 2 +- .../test_task_handler_with_custom_formatter.py | 1 - tests/utils/test_types.py | 2 +- tests/utils/test_usage_data_collection.py | 6 +- tests/www/test_app.py | 7 +- tests/www/test_auth.py | 2 - tests/www/test_security_manager.py | 2 - tests/www/test_utils.py | 6 - tests/www/views/conftest.py | 3 +- tests/www/views/test_views_connection.py | 26 + tests/www/views/test_views_grid.py | 44 + tests_common/_internals/capture_warnings.py | 7 +- tests_common/pytest_plugin.py | 32 +- tests_common/test_utils/compat.py | 5 - tests_common/test_utils/db.py | 21 +- .../test_utils/format_datetime.py | 17 +- tests_common/test_utils/gcp_system_helpers.py | 10 +- tests_common/test_utils/mock_operators.py | 4 +- tests_common/test_utils/reset_warning_registry.py | 2 +- 1276 files changed, 25299 insertions(+), 10564 deletions(-) rename .github/workflows/{static-checks-mypy-docs.yml => ci-image-checks.yml} (83%) rename .github/workflows/{check-providers.yml => test-provider-packages.yml} (68%) create mode 100644 airflow/api_fastapi/common/exceptions.py copy airflow/api_fastapi/{execution_api/datamodels/variable.py => core_api/base.py} (79%) rename tests/api_internal/__init__.py => airflow/api_fastapi/core_api/datamodels/extra_links.py (82%) copy airflow/api_fastapi/core_api/datamodels/{version.py => log.py} (86%) copy airflow/api_fastapi/core_api/datamodels/{dag_stats.py => ui/config.py} (56%) create mode 100644 airflow/api_fastapi/core_api/datamodels/ui/graph.py copy airflow/{api_connexion/endpoints => api_fastapi/core_api/routes/public}/dag_parsing.py (60%) copy airflow/{api_connexion/endpoints/extra_link_endpoint.py => api_fastapi/core_api/routes/public/extra_links.py} (55%) create mode 100644 airflow/api_fastapi/core_api/routes/public/job.py create mode 100644 airflow/api_fastapi/core_api/routes/public/log.py create mode 100644 airflow/api_fastapi/core_api/routes/ui/config.py create mode 100644 airflow/api_fastapi/core_api/routes/ui/graph.py create mode 100644 airflow/cli/commands/asset_command.py create mode 100644 airflow/executors/workloads.py create mode 100644 airflow/ui/src/components/ConfirmationModal.tsx create mode 100644 airflow/ui/src/context/openGroups/OpenGroupsProvider.tsx copy airflow/ui/src/context/{colorMode/useColorMode.tsx => openGroups/index.ts} (71%) copy airflow/ui/src/context/{colorMode/useColorMode.tsx => openGroups/useOpenGroups.ts} (68%) copy airflow/ui/src/{context/timezone/TimezoneProvider.tsx => pages/DagsList/Dag/DagVizModal.tsx} (50%) create mode 100644 airflow/ui/src/pages/DagsList/Dag/Graph/Edge.tsx create mode 100644 airflow/ui/src/pages/DagsList/Dag/Graph/Graph.tsx copy airflow/ui/src/{layouts/BaseLayout.tsx => pages/DagsList/Dag/Graph/JoinNode.tsx} (63%) copy airflow/ui/src/{layouts/BaseLayout.tsx => pages/DagsList/Dag/Graph/NodeWrapper.tsx} (71%) create mode 100644 airflow/ui/src/pages/DagsList/Dag/Graph/TaskName.tsx create mode 100644 airflow/ui/src/pages/DagsList/Dag/Graph/TaskNode.tsx create mode 100644 airflow/ui/src/pages/DagsList/Dag/Graph/data.ts copy airflow/ui/src/{context/colorMode/useColorMode.tsx => pages/DagsList/Dag/Graph/index.ts} (71%) create mode 100644 airflow/ui/src/pages/DagsList/Dag/Graph/reactflowUtils.ts create mode 100644 airflow/ui/src/pages/DagsList/Dag/Graph/useGraphLayout.ts copy airflow/ui/src/pages/DagsList/Dag/{Dag.tsx => Tabs.tsx} (51%) create mode 100644 airflow/ui/src/pages/DagsList/SortSelect.tsx copy airflow/ui/src/{context/colorMode/useColorMode.tsx => queries/useConfig.tsx} (71%) create mode 100644 contributing-docs/testing/python_client_tests.rst create mode 100644 dev/breeze/doc/images/output_testing_python-api-client-tests.svg create mode 100644 dev/breeze/doc/images/output_testing_python-api-client-tests.txt rename docs/{apache-airflow/howto/operator => apache-airflow-providers-standard/sensors}/external_task_sensor.rst (94%) create mode 100644 newsfragments/43949.significant.rst create mode 100644 newsfragments/44300.bugfix.rst create mode 100644 newsfragments/template.significant.rst copy airflow/api_fastapi/core_api/datamodels/providers.py => providers/src/airflow/providers/common/compat/standard/triggers.py (70%) rename {airflow => providers/src/airflow/providers/standard}/sensors/external_task.py (98%) copy {airflow/api_connexion => providers/src/airflow/providers/standard/triggers}/__init__.py (100%) rename {airflow => providers/src/airflow/providers/standard}/triggers/external_task.py (80%) rename {airflow => providers/src/airflow/providers/standard}/triggers/file.py (85%) rename {airflow => providers/src/airflow/providers/standard}/triggers/temporal.py (88%) rename {airflow => providers/src/airflow/providers/standard}/utils/sensor_helper.py (100%) delete mode 100644 providers/tests/edge/worker_api/routes/test_rpc_api.py copy {airflow/api_connexion => providers/tests/standard/triggers}/__init__.py (100%) rename {tests => providers/tests/standard}/triggers/test_external_task.py (86%) rename {tests => providers/tests/standard}/triggers/test_file.py (91%) rename {tests => providers/tests/standard}/triggers/test_temporal.py (71%) create mode 100644 providers/tests/system/google/cloud/translate/example_translate_dataset.py copy {airflow/api_connexion => providers/tests/system/openlineage}/__init__.py (100%) copy {tests/api_fastapi => providers/tests/system/openlineage}/conftest.py (59%) create mode 100644 providers/tests/system/openlineage/example_openlineage.json create mode 100644 providers/tests/system/openlineage/example_openlineage.py create mode 100644 providers/tests/system/openlineage/example_openlineage_mapped_sensor.json create mode 100644 providers/tests/system/openlineage/example_openlineage_mapped_sensor.py create mode 100644 providers/tests/system/openlineage/operator.py copy {airflow/api_connexion => providers/tests/system/openlineage/transport}/__init__.py (100%) create mode 100644 providers/tests/system/openlineage/transport/variable.py create mode 100644 scripts/in_container/check_connectivity.sh create mode 100644 scripts/in_container/install_airflow_python_client.py copy airflow/api_fastapi/core_api/datamodels/providers.py => task_sdk/tests/dags/super_basic_deferred_run.py (66%) create mode 100644 tests/api_fastapi/core_api/routes/public/test_dag_parsing.py create mode 100644 tests/api_fastapi/core_api/routes/public/test_extra_links.py create mode 100644 tests/api_fastapi/core_api/routes/public/test_job.py copy tests/{api_connexion/endpoints/test_log_endpoint.py => api_fastapi/core_api/routes/public/test_log.py} (66%) create mode 100644 tests/api_fastapi/core_api/routes/ui/test_config.py create mode 100644 tests/api_fastapi/core_api/routes/ui/test_graph.py delete mode 100644 tests/api_internal/endpoints/__init__.py delete mode 100644 tests/api_internal/endpoints/test_rpc_api_endpoint.py delete mode 100644 tests/api_internal/test_internal_api_call.py create mode 100644 tests/cli/commands/test_asset_command.py delete mode 100644 tests/cli/commands/test_internal_api_command.py delete mode 100644 tests/dags/test_retry_handling_job.py delete mode 100644 tests/serialization/test_pydantic_models.py delete mode 100644 tests/triggers/__init__.py copy airflow/api_fastapi/core_api/datamodels/trigger.py => tests_common/test_utils/format_datetime.py (70%)