This is an automated email from the ASF dual-hosted git repository.
Lee-W pushed a change to branch ci-upgrade-main
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard f5447b14b8f [main] CI: Upgrade important CI environment
add 72d9b5d8243 Add S3RemoteLogIO.from_config and register s3 remote
logging scheme (#69817)
add 532e7d0911c Allow Common AI SQL imports without DataFusion (#69990)
add c173545e8d0 Surface YARN diagnostics for spark on yarn clusters
(#70183)
add b0f8d26481d fix ruff check docs command in FAQ (#70212)
add 373ca8614d5 OpenLineage: add execute_in_thread to emit task events
without forking (#68708)
add 7c03dabdcdd Route SQLAlchemyError through global exception handler
(#69267)
add 5cbb64bf60a Add streaming task log support to KubernetesExecutor
(#69300)
add 38d94ba8188 Move Lang-SDK ADRs from java-sdk/adr to
airflow-core/adr/lang-sdk (#70156)
add e420143333a Support image digest in worker pod template image (#69776)
add ec8da5c25eb Populate dag and note before firing dag run state-change
listeners (#69873)
add 02a956f5cf9 Add Flower HTTPRoute support to Helm chart (#69945)
add 8c205ef4083 Add React plugin support to Dag and task overviews (#69992)
add 8ce7c84fa7b UI: Pass Dag, DagRun, TaskInstance, and Asset objects to
React app plugins (#70149)
add b6d81b94ed9 Call listeners for running task instance when a Dag Run
state is manually set (#69874)
add 74c76cbf02f Document `apply_function` dot notation requirement for
Kafka message queues (#68414)
add 7c272f1ea97 Make psycopg (v3) the default synchronous Postgres driver
(#69526)
add cfe1b37aa06 Fix the google provider dataform system test (#70136)
add 2060ff322b9 Fix NPM vulenaribilities in simple auth manager (#70191)
add a564b789a67 fix: Patch ti note before state so that listeners can see
it (#70247)
add f2802c459fe Preserve Dataform workflow invocation config (#53843)
(#69161)
add 0f0873630af Emit per-statement OpenLineage events for BigQuery script
jobs (#69234)
add b703da5326d Quote file paths in Teradata TPT shell commands and pass
the openssl passphrase via stdin (#69380)
add fecccf8eeb5 Fix Amazon provider log messages to show actual values
(#69967)
add 45a93a1e1fb Make psycopg (v3) the default synchronous Postgres driver
in airflow-core (#69469)
add fcccec2f2d5 Upgrade NPM dependencies in edge provider (#70253)
add cd28fbde0be Fix secrets not masked in Rendered Templates view with
KubernetesPodOperator (#68975)
add 7ef372a4638 Clarify unreleased-feature changes are not breaking in
provider-doc skill (#70255)
add 1fd77cdd428 Make RedshiftDeleteClusterOperator delete reliably during
cluster transitions (#69574)
add 79f2c3c6208 Note the allow-provider-dependency-bump label in
provider-doc release skill (#70257)
add c578603205b Fix conn-fields check crash for nested provider packages
(#70224)
add edbd947137f [main] CI: Upgrade important CI environment
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (f5447b14b8f)
\
N -- N -- N refs/heads/ci-upgrade-main (edbd947137f)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../prepare-providers-documentation/SKILL.md | 43 +-
.../lang-sdk}/0001-java-sdk-airflow-integration.md | 0
.../adr/lang-sdk}/0002-workload-execution.md | 0
.../adr/lang-sdk}/0003-pure-java-dags.md | 0
.../adr/lang-sdk}/0004-dag-parsing.md | 0
.../adr/lang-sdk}/0005-coordinator-packaging.md | 0
.../lang-sdk}/0006-no-lang-sdk-source-display.md | 0
airflow-core/adr/lang-sdk/README.md | 37 ++
.../logging-monitoring/logging-tasks.rst | 2 +-
.../modules_management.rst | 2 +-
.../docs/administration-and-deployment/plugins.rst | 21 +
airflow-core/docs/core-concepts/message-queues.rst | 5 +
airflow-core/docs/core-concepts/multi-team.rst | 8 +-
airflow-core/docs/faq.rst | 2 +-
.../docs/howto/docker-compose/docker-compose.yaml | 4 +-
airflow-core/docs/howto/set-up-database.rst | 26 +-
airflow-core/newsfragments/69469.improvement.rst | 1 +
airflow-core/newsfragments/69874.bugfix.rst | 1 +
airflow-core/src/airflow/api/common/mark_tasks.py | 44 +-
airflow-core/src/airflow/api_fastapi/app.py | 3 +-
.../auth/managers/simple/ui/package.json | 2 +-
.../auth/managers/simple/ui/pnpm-lock.yaml | 265 ++++-------
.../src/airflow/api_fastapi/common/exceptions.py | 48 +-
.../src/airflow/api_fastapi/core_api/app.py | 7 -
.../api_fastapi/core_api/datamodels/plugins.py | 2 +-
.../core_api/openapi/v2-rest-api-generated.yaml | 2 +
.../api_fastapi/core_api/routes/public/dag_run.py | 14 +-
.../core_api/routes/public/task_instances.py | 73 ++-
.../core_api/services/public/dag_run.py | 25 +-
.../core_api/services/public/task_instances.py | 32 +-
.../src/airflow/api_fastapi/execution_api/app.py | 2 +
.../execution_api/routes/task_instances.py | 14 +-
airflow-core/src/airflow/configuration.py | 12 +-
...nconsistency_between_ORM_and_migration_files.py | 2 +-
...0_replace_deadline_inline_callback_with_fkey.py | 6 +-
airflow-core/src/airflow/models/serialized_dag.py | 6 +-
airflow-core/src/airflow/settings.py | 7 +-
.../airflow/ui/openapi-gen/requests/schemas.gen.ts | 2 +-
.../airflow/ui/openapi-gen/requests/types.gen.ts | 4 +-
.../ui/src/pages/Dag/Overview/Overview.test.tsx | 68 +++
.../airflow/ui/src/pages/Dag/Overview/Overview.tsx | 37 +-
.../src/airflow/ui/src/pages/ReactPlugin.test.tsx | 144 ++++++
.../src/airflow/ui/src/pages/ReactPlugin.tsx | 78 +++-
.../ui/src/pages/Task/Overview/Overview.test.tsx | 63 +++
.../ui/src/pages/Task/Overview/Overview.tsx | 28 +-
.../tests/unit/api/common/test_mark_tasks.py | 9 +-
.../unit/api_fastapi/common/test_exceptions.py | 60 ++-
.../core_api/datamodels/test_plugins.py | 31 +-
.../core_api/routes/public/test_dag_run.py | 147 +++++-
.../core_api/routes/public/test_task_instances.py | 61 +++
.../unit/api_fastapi/execution_api/test_app.py | 18 +-
.../versions/head/test_task_instances.py | 60 ++-
airflow-core/tests/unit/core/test_configuration.py | 38 +-
.../tests/unit/core/test_sqlalchemy_config.py | 24 +
.../tests/unit/listeners/class_listener.py | 14 +
airflow-core/tests/unit/utils/test_log_handlers.py | 7 +-
.../src/airflowctl/api/datamodels/generated.py | 2 +
airflow-e2e-tests/docker/openlineage-compat-db.yml | 48 ++
.../tests/airflow_e2e_tests/conftest.py | 9 +-
.../tests/airflow_e2e_tests/constants.py | 5 +
chart/docs/production-guide.rst | 37 +-
chart/templates/_helpers.yaml | 9 +-
chart/templates/flower/flower-httproute.yaml | 70 +++
chart/templates/flower/flower-ingress.yaml | 3 +
.../airflow_aux/test_basic_helm_chart.py | 44 +-
.../airflow_aux/test_pod_template_file.py | 33 ++
.../test_httproute_flower.py} | 201 ++++++---
chart/values.schema.json | 104 +++++
chart/values.yaml | 65 ++-
dev/breeze/tests/test_selective_checks.py | 14 -
docs/spelling_wordlist.txt | 3 +
go-sdk/README.md | 3 +
.../adr/0003-coordinator-protocol-msgpack-ipc.md | 14 +-
java-sdk/README.md | 4 +-
providers/amazon/docs/changelog.rst | 7 +
providers/amazon/docs/executors/batch-executor.rst | 2 +-
providers/amazon/docs/executors/ecs-executor.rst | 2 +-
.../amazon/docs/executors/lambda-executor.rst | 2 +-
providers/amazon/provider.yaml | 2 +
.../providers/amazon/aws/hooks/redshift_sql.py | 37 +-
.../src/airflow/providers/amazon/aws/hooks/s3.py | 2 +-
.../providers/amazon/aws/log/s3_task_handler.py | 26 ++
.../amazon/aws/operators/redshift_cluster.py | 122 +++--
.../providers/amazon/aws/sensors/dynamodb.py | 2 +-
.../amazon/aws/triggers/redshift_cluster.py | 54 +++
.../providers/amazon/aws/waiters/redshift.json | 55 +++
.../airflow/providers/amazon/get_provider_info.py | 3 +-
.../unit/amazon/aws/hooks/test_redshift_sql.py | 43 +-
.../amazon/tests/unit/amazon/aws/hooks/test_s3.py | 32 +-
.../unit/amazon/aws/log/test_s3_task_handler.py | 83 +++-
.../amazon/aws/operators/test_redshift_cluster.py | 153 ++++++-
.../tests/unit/amazon/aws/sensors/test_dynamodb.py | 6 +-
.../amazon/aws/triggers/test_redshift_cluster.py | 36 +-
.../apache/kafka/docs/message-queues/index.rst | 44 ++
providers/apache/kafka/docs/sensors.rst | 9 +
.../providers/apache/spark/hooks/spark_submit.py | 24 +-
.../unit/apache/spark/hooks/test_spark_submit.py | 57 ++-
providers/atlassian/jira/provider.yaml | 12 +
.../providers/atlassian/jira/get_provider_info.py | 7 +
.../airflow/providers/atlassian/jira/hooks/jira.py | 1 -
providers/celery/docs/changelog.rst | 7 +
providers/celery/provider.yaml | 2 +-
.../celery/executors/celery_kubernetes_executor.py | 7 +
.../providers/celery/executors/default_celery.py | 21 +-
.../airflow/providers/celery/get_provider_info.py | 2 +-
.../unit/celery/executors/test_celery_executor.py | 26 ++
.../executors/test_celery_kubernetes_executor.py | 21 +
.../kubernetes/executors/kubernetes_executor.py | 37 +-
.../executors/local_kubernetes_executor.py | 7 +
.../executors/test_kubernetes_executor.py | 59 ++-
.../executors/test_local_kubernetes_executor.py | 21 +
.../kubernetes/log_handlers/test_log_handlers.py | 12 +-
providers/common/ai/README.rst | 4 +-
providers/common/ai/docs/index.rst | 4 +-
providers/common/ai/docs/operators/llm_sql.rst | 6 +
providers/common/ai/pyproject.toml | 4 +-
.../providers/common/ai/operators/llm_sql.py | 12 +-
.../tests/unit/common/ai/operators/test_llm_sql.py | 87 +++-
.../src/airflow/providers/common/sql/hooks/sql.py | 37 +-
.../sql/tests/unit/common/sql/hooks/test_sql.py | 92 ++++
.../providers/edge3/plugins/www/package.json | 9 +-
.../providers/edge3/plugins/www/pnpm-lock.yaml | 493 +++++++++++----------
.../providers/google/cloud/hooks/dataform.py | 2 +
.../providers/google/cloud/openlineage/mixins.py | 116 ++++-
.../google/cloud/transfers/bigquery_to_postgres.py | 13 +-
.../google/cloud/dataform/example_dataform.py | 56 ++-
.../tests/unit/google/cloud/hooks/test_dataform.py | 23 +-
.../unit/google/cloud/openlineage/test_mixins.py | 390 +++++++++++++++-
.../cloud/transfers/test_bigquery_to_postgres.py | 51 ++-
providers/openlineage/docs/troubleshooting.rst | 44 ++
providers/openlineage/provider.yaml | 20 +
.../src/airflow/providers/openlineage/conf.py | 6 +
.../providers/openlineage/get_provider_info.py | 7 +
.../providers/openlineage/plugins/listener.py | 53 ++-
.../unit/openlineage/plugins/test_listener.py | 98 ++++
.../tests/unit/openlineage/test_conf.py | 17 +
providers/pgvector/pyproject.toml | 1 +
.../providers/pgvector/operators/pgvector.py | 21 +-
.../tests/unit/pgvector/operators/test_pgvector.py | 57 ++-
providers/postgres/README.rst | 5 +-
providers/postgres/docs/changelog.rst | 14 +
providers/postgres/docs/index.rst | 5 +-
providers/postgres/pyproject.toml | 24 +-
.../airflow/providers/postgres/hooks/postgres.py | 42 +-
.../tests/unit/postgres/hooks/test_postgres.py | 21 +
.../unit/postgres/test_provider_dependencies.py | 46 ++
.../providers/teradata/utils/encryption_utils.py | 9 +-
.../airflow/providers/teradata/utils/tpt_util.py | 16 +-
.../unit/teradata/utils/test_encryption_utils.py | 14 +-
.../tests/unit/teradata/utils/test_tpt_util.py | 28 ++
scripts/ci/docker-compose/backend-postgres.yml | 4 +-
scripts/ci/prek/check_provider_yaml_files.py | 11 +-
scripts/ci/prek/common_prek_utils.py | 2 +-
task-sdk-integration-tests/docker-compose.yaml | 2 +-
task-sdk/src/airflow/sdk/execution_time/context.py | 28 +-
.../tests/task_sdk/execution_time/test_context.py | 93 ++++
uv.lock | 9 +-
157 files changed, 4492 insertions(+), 928 deletions(-)
rename {java-sdk/adr =>
airflow-core/adr/lang-sdk}/0001-java-sdk-airflow-integration.md (100%)
rename {java-sdk/adr => airflow-core/adr/lang-sdk}/0002-workload-execution.md
(100%)
rename {java-sdk/adr => airflow-core/adr/lang-sdk}/0003-pure-java-dags.md
(100%)
rename {java-sdk/adr => airflow-core/adr/lang-sdk}/0004-dag-parsing.md (100%)
rename {java-sdk/adr =>
airflow-core/adr/lang-sdk}/0005-coordinator-packaging.md (100%)
rename {java-sdk/adr =>
airflow-core/adr/lang-sdk}/0006-no-lang-sdk-source-display.md (100%)
create mode 100644 airflow-core/adr/lang-sdk/README.md
create mode 100644 airflow-core/newsfragments/69469.improvement.rst
create mode 100644 airflow-core/newsfragments/69874.bugfix.rst
create mode 100644
airflow-core/src/airflow/ui/src/pages/Dag/Overview/Overview.test.tsx
create mode 100644 airflow-core/src/airflow/ui/src/pages/ReactPlugin.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/pages/Task/Overview/Overview.test.tsx
copy ts-sdk/example/dags/typescript_example.py =>
airflow-core/tests/unit/api_fastapi/core_api/datamodels/test_plugins.py (66%)
create mode 100644 airflow-e2e-tests/docker/openlineage-compat-db.yml
create mode 100644 chart/templates/flower/flower-httproute.yaml
copy chart/tests/helm_tests/{apiserver/test_httproute_apiserver.py =>
other/test_httproute_flower.py} (61%)
create mode 100644
providers/postgres/tests/unit/postgres/test_provider_dependencies.py
