This is an automated email from the ASF dual-hosted git repository.
mobuchowski pushed a change to branch openlineage_dont_run_tis_executor
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 62aec10394 openlineage: don't run task instance listener in executor
add 23d5076635 openlineage: defensively check for provided datetimes in
listener (#33343)
add f1fad44650 Clarify UI user types in security model (#33021)
add 0cb256411a Fix links in security docs (#33329)
add 42638549ef replace ~ with . for marking up sections (#33369)
add 2caa186935 Remove user sessions when resetting password (#33347)
add bd11ea81e5 Add Deferrable mode to GlueCatalogPartitionSensor (#33239)
add 93c3ccbdf2 feat: remove empty log folders when cleanup (#33252)
add dfb2403ec4 Add Dataplex Data Quality operators. (#32256)
add 4460356c03 Set logging level to WARNING (#33314)
add 416cae48f5 Migrate system tests for CloudSQLExecuteQueryOperator
(non-SSL) (#33316)
add 64c2a56faf D205 Support - WWW (#33298)
add a003781420 Remove updating description in Dockerhub (#33374)
add 44a752aa96 D401 Support - Secrets to Triggers (Inclusive) (#33338)
add de780e0d08 Merge system tests for CloudSQLExportInstanceOperator into
one test (#33315)
add 50a6385c7f Refactor: Simplify code in scripts (#33295)
add 8871ef4d1b Use tmp_path in test_views_home (#33322)
add 95e9d83720 Refactor: Simplify code in settings (#33267)
add 58817e368f Add a feature to be able to modify comments in released
constraints (#33372)
add 4d3d4a1567 Allow templating flower ingress hostnames (#33363)
add 5e1e5fa637 D401 Support - Root Files (#33352)
add 38aada7b8b Introducing class constant to make worker pod log lines
configurable (#33378)
add 21656fcfe8 Refactor: Simplify code in configuration.py (#33160)
add 72c489da5e Adding typing for a KPO test (#33380)
add 2976a56f85 Allow templating webserver ingress hostnames (#33142)
add 1cf960d5d5 Improve handling of CI image timeout when backtracking
(#33364)
add 2ab0d2d683 Restore `--builder` command for cache building (#33387)
add a1b5bdb25a respect soft_fail argument when ExternalTaskSensor runs in
deferrable mode (#33196)
add 8bf53dd554 Add Service Principal OAuth for Databricks. (#33005)
add a2a0d05345 Add Source Code and Testing for Google Cloud's Data
Pipelines Create Operator (#32843)
add 04d3f4560f Order triggers by - TI priority_weight when assign
unassigned triggers (#32318)
add 2950fd7685 [Models] [Postgres] Check if the dynamically-added index is
in the table schema before adding (#32731)
add b4102ce0b5 Update Error details for Generic Error Code (#32847)
add a86313ff14 Replace common.tplvalues.render by tpl in ingress template
files (#33384)
add c17140cb37 Fix Storage Transfer system tests (#33385)
add 46ffc25995 Adding typing for KPO SCC objects (#33381)
add b8cdd284dd Add better instructions to constraint files (#33367)
new 9cf049c471 Merge branch 'main' into openlineage_dont_run_tis_executor
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/actions/build-ci-images/action.yml | 7 -
.github/actions/build-prod-images/action.yml | 4 -
.github/workflows/build-images.yml | 1 +
.github/workflows/ci.yml | 1 +
.github/workflows/release_dockerhub_image.yml | 7 -
BREEZE.rst | 23 +-
Dockerfile | 14 +-
Dockerfile.ci | 7 +-
airflow/__main__.py | 1 -
airflow/auth/managers/fab/models/__init__.py | 8 +-
.../auth/managers/fab/security_manager/override.py | 53 +-
airflow/config_templates/config.yml | 16 +-
airflow/configuration.py | 70 +-
airflow/models/trigger.py | 4 +-
airflow/policies.py | 10 +-
airflow/providers/amazon/aws/hooks/glue_catalog.py | 41 +
.../amazon/aws/sensors/glue_catalog_partition.py | 32 +-
airflow/providers/amazon/aws/triggers/glue.py | 83 +
.../kubernetes/executors/kubernetes_executor.py | 3 +-
airflow/providers/cncf/kubernetes/operators/pod.py | 4 +-
.../providers/databricks/hooks/databricks_base.py | 148 +-
airflow/providers/google/cloud/hooks/dataflow.py | 6 +-
.../providers/google/cloud/hooks/datapipeline.py | 89 +
airflow/providers/google/cloud/hooks/dataplex.py | 500 ++-
.../{life_sciences.py => datapipeline.py} | 89 +-
.../providers/google/cloud/operators/dataplex.py | 859 +++-
airflow/providers/google/cloud/sensors/dataplex.py | 126 +-
airflow/providers/google/provider.yaml | 11 +
.../providers/microsoft/azure/secrets/key_vault.py | 9 +
airflow/providers/openlineage/plugins/adapter.py | 4 +-
airflow/providers/openlineage/plugins/listener.py | 21 +-
airflow/secrets/cache.py | 18 +-
airflow/secrets/local_filesystem.py | 6 +-
airflow/security/permissions.py | 2 +-
airflow/security/utils.py | 6 +-
airflow/sensors/base.py | 10 +-
airflow/sensors/date_time.py | 4 +-
airflow/sensors/external_task.py | 50 +-
airflow/sensors/time_delta.py | 2 +-
airflow/sensors/time_sensor.py | 2 +-
airflow/sentry.py | 4 +-
airflow/serialization/serde.py | 4 +-
airflow/serialization/serialized_objects.py | 20 +-
airflow/settings.py | 43 +-
airflow/task/task_runner/base_task_runner.py | 2 +-
airflow/template/templater.py | 5 +-
airflow/ti_deps/dep_context.py | 2 +-
airflow/ti_deps/deps/base_ti_dep.py | 6 +-
airflow/ti_deps/deps/dagrun_backfill_dep.py | 2 +-
airflow/ti_deps/deps/pool_slots_available_dep.py | 2 +-
airflow/ti_deps/deps/ready_to_reschedule.py | 2 +-
airflow/timetables/interval.py | 3 +-
airflow/triggers/base.py | 4 +-
airflow/triggers/external_task.py | 18 +-
airflow/triggers/file.py | 2 +-
airflow/utils/db_cleanup.py | 4 +
airflow/www/api/experimental/endpoints.py | 13 +-
airflow/www/decorators.py | 3 +-
airflow/www/extensions/init_appbuilder.py | 12 +-
airflow/www/extensions/init_dagbag.py | 5 +-
airflow/www/extensions/init_robots.py | 6 +-
airflow/www/extensions/init_security.py | 5 +-
airflow/www/fab_security/manager.py | 24 +-
airflow/www/fab_security/sqla/manager.py | 11 +-
airflow/www/forms.py | 5 +-
airflow/www/security.py | 25 +-
airflow/www/utils.py | 25 +-
airflow/www/views.py | 48 +-
chart/templates/flower/flower-ingress.yaml | 2 +-
chart/templates/webserver/webserver-ingress.yaml | 2 +-
chart/values.yaml | 2 +
...UALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md | 197 +
.../src/airflow_breeze/commands/ci_commands.py | 10 +
.../airflow_breeze/commands/ci_commands_config.py | 1 +
.../airflow_breeze/commands/ci_image_commands.py | 187 +-
.../commands/ci_image_commands_config.py | 10 +-
.../commands/production_image_commands.py | 61 +-
.../commands/production_image_commands_config.py | 3 +-
.../commands/release_management_commands.py | 77 +-
.../commands/release_management_commands_config.py | 14 +-
dev/breeze/src/airflow_breeze/global_constants.py | 16 +-
.../src/airflow_breeze/params/build_ci_params.py | 9 +
.../airflow_breeze/params/common_build_params.py | 1 -
.../src/airflow_breeze/utils/backtracking.py | 89 +
.../src/airflow_breeze/utils/common_options.py | 28 +-
.../airflow_breeze/utils/docker_command_utils.py | 11 -
.../connections/databricks.rst | 10 +-
.../operators/cloud/cloud_sql.rst | 26 +-
.../cloud/cloud_storage_transfer_service.rst | 2 +-
.../operators/cloud/datapipeline.rst | 60 +
.../operators/cloud/dataplex.rst | 165 +-
docs/apache-airflow/security/access-control.rst | 19 +-
docs/apache-airflow/security/index.rst | 4 +-
docs/apache-airflow/security/security_model.rst | 34 +-
docs/spelling_wordlist.txt | 3 +
helm_tests/other/test_keda.py | 2 +-
helm_tests/webserver/test_ingress_flower.py | 34 +
helm_tests/webserver/test_ingress_web.py | 33 +
images/breeze/output-commands-hash.txt | 21 +-
images/breeze/output_ci-image_build.svg | 124 +-
images/breeze/output_ci.svg | 18 +-
.../output_ci_find-backtracking-candidates.svg | 91 +
images/breeze/output_prod-image_build.svg | 134 +-
...utput_release-management_update-constraints.svg | 70 +-
.../output_setup_check-all-params-in-groups.svg | 36 +-
.../output_setup_regenerate-command-images.svg | 62 +-
kubernetes_tests/test_kubernetes_pod_operator.py | 22 +-
.../pre_commit_check_pre_commit_hooks.py | 5 +-
scripts/ci/pre_commit/pre_commit_json_schema.py | 2 +-
.../pre_commit_update_common_sql_api_stubs.py | 22 +-
.../pre_commit_update_example_dags_paths.py | 5 +-
scripts/docker/clean-logs.sh | 2 +
scripts/docker/install_additional_dependencies.sh | 2 +-
scripts/docker/install_airflow.sh | 2 +-
.../docker/install_from_docker_context_files.sh | 2 +-
scripts/in_container/_in_container_utils.sh | 4 +-
scripts/in_container/run_generate_constraints.sh | 20 +
scripts/in_container/run_migration_reference.py | 4 +-
.../in_container/update_quarantined_test_status.py | 5 +-
scripts/in_container/verify_providers.py | 5 +-
setup.py | 16 +-
tests/auth/managers/fab/test_models.py | 62 +
tests/core/test_policies.py | 4 +-
tests/jobs/test_triggerer_job.py | 18 +-
tests/models/test_trigger.py | 124 +-
.../aws/sensors/test_glue_catalog_partition.py | 19 +
tests/providers/amazon/aws/triggers/test_glue.py | 40 +-
.../executors/test_kubernetes_executor.py | 13 +
.../cncf/kubernetes/operators/test_pod.py | 6 +-
.../providers/databricks/hooks/test_databricks.py | 123 +-
.../google/cloud/hooks/test_datapipeline.py | 110 +
.../providers/google/cloud/hooks/test_dataplex.py | 155 +-
.../google/cloud/operators/test_datapipeline.py | 138 +
.../google/cloud/operators/test_dataplex.py | 274 ++
.../google/cloud/sensors/test_dataplex.py | 47 +-
.../google/cloud/cloud_sql/example_cloud_sql.py | 49 +-
.../cloud_sql/example_cloud_sql_deferrable.py | 184 -
.../cloud_sql/example_cloud_sql_query_mysql.py | 280 ++
.../cloud_sql/example_cloud_sql_query_postgres.py | 288 ++
.../google/cloud/datapipelines}/__init__.py | 0
.../cloud/datapipelines/example_datapipeline.py | 124 +
.../cloud/datapipelines/resources}/__init__.py | 0
.../cloud/datapipelines/resources/kinglear.txt | 4273 ++++++++++++++++++++
.../cloud/datapipelines/resources/word-count.json | 29 +
.../google/cloud/dataplex/example_dataplex_dq.py | 343 ++
.../example_cloud_storage_transfer_service_gcp.py | 42 +-
...e_cloud_storage_transfer_service_gcs_to_gcs.py} | 20 +-
.../resources/{file => transfer_service_gcp_file} | 0
.../resources/transfer_service_gcs_to_gcs_file | 0
tests/www/views/test_views_custom_user_views.py | 112 +
tests/www/views/test_views_home.py | 52 +-
151 files changed, 10239 insertions(+), 1132 deletions(-)
create mode 100644 airflow/providers/google/cloud/hooks/datapipeline.py
copy airflow/providers/google/cloud/operators/{life_sciences.py =>
datapipeline.py} (51%)
create mode 100644 dev/breeze/src/airflow_breeze/utils/backtracking.py
create mode 100644
docs/apache-airflow-providers-google/operators/cloud/datapipeline.rst
create mode 100644 images/breeze/output_ci_find-backtracking-candidates.svg
create mode 100644 tests/auth/managers/fab/test_models.py
create mode 100644 tests/providers/google/cloud/hooks/test_datapipeline.py
create mode 100644 tests/providers/google/cloud/operators/test_datapipeline.py
delete mode 100644
tests/system/providers/google/cloud/cloud_sql/example_cloud_sql_deferrable.py
create mode 100644
tests/system/providers/google/cloud/cloud_sql/example_cloud_sql_query_mysql.py
create mode 100644
tests/system/providers/google/cloud/cloud_sql/example_cloud_sql_query_postgres.py
copy {airflow/api_connexion =>
tests/system/providers/google/cloud/datapipelines}/__init__.py (100%)
create mode 100644
tests/system/providers/google/cloud/datapipelines/example_datapipeline.py
copy {airflow/api_connexion =>
tests/system/providers/google/cloud/datapipelines/resources}/__init__.py (100%)
create mode 100644
tests/system/providers/google/cloud/datapipelines/resources/kinglear.txt
create mode 100644
tests/system/providers/google/cloud/datapipelines/resources/word-count.json
create mode 100644
tests/system/providers/google/cloud/dataplex/example_dataplex_dq.py
rename
tests/system/providers/google/cloud/storage_transfer/{example_cloud_storage_transfer_service_gcp_to_gcs.py
=> example_cloud_storage_transfer_service_gcs_to_gcs.py} (84%)
rename tests/system/providers/google/cloud/storage_transfer/resources/{file =>
transfer_service_gcp_file} (100%)
copy airflow/_vendor/__init__.py =>
tests/system/providers/google/cloud/storage_transfer/resources/transfer_service_gcs_to_gcs_file
(100%)
