This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a change to branch v2-7-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
from 11320c4a15 Add __init__ to params subpackage (#34710)
new 5712271ff7 Add missing asterisk for MsSQL on stable Airflow's
footnotes (#34824)
new 0e5511f6d7 Bump azure-mgmt-containerinstance (#34738)
new 5bc33f1edc Airflow 2.7.2 has been released (#34885)
new d620b96154 Limit WTForms to below 3.1.0 (#34943)
new 311f1f7521 Update recording of SVG output to rich-click 1.7.0 (#34941)
new edcde3e137 update moto version (#34938)
new ebdbcec604 Clean local tags in tag_providers for network issues with
Github (#34951)
new d18e34ea1e Add Open Search Provider (#34705)
new 4b7a83556f Fix docs building failing on new but not released yet
provider (#34963)
new 3e69dfdfb8 Fix step overriding github actions (#34888)
new 05ef26b307 Inline restoring target workflow scripts in the
build-images workflow (#34901)
new b8878af4bc Temporarily disable PyPI constrainst generation (#34918)
new 04efa34840 Move constraints generation to separate job in CI workflow
(#34990)
new 9614a3519b Bring back source constraints generation for PROD image
build (#35004)
new bed2b81ed6 Better parallellise Other/Operator tests (#35002)
new a4d24ff5f3 Fix Precommit hook console.print' (#35016)
new 49f17b7a3b Use constraints generated at preview time rather than
regenerate (#35013)
new f3a69cf2d7 Add opensearch to boring-cyborg.yml (#35031)
new f953f7c90c Upgrade `pip` to 23.3 (#35029)
new eeed11173f ADd missing dependency generate-constraints => build prod
image (#35040)
new c31392eb45 Fix update-constraints command example (#35042)
new 7e5c335313 Add comment explaining that commits.rst is generated
(#35065)
new cb36f28bed Use short packages names for non-providers packages (#35067)
new 638c3adc5a Use arguments inestead of --package-filter option for build
docs in CI (#35068)
new 41b63087c0 Handle "all-providers" when building documentation (#35069)
new 4a9641db6c Move installed_providers.txt to airflow/providers. (#35081)
new 67c8fee97c Run constraint branch checkout in non-canary build (#35088)
new 7bd6852de1 Refactor string splitting (#34185)
new c479887588 Fix CI's docs filter generation (#35087)
new e55c835cdd Describe workaround to packaging 23.2 breaking pipx local
detection (#35092)
new d9ff1aaead Use airflow released in PyPI for k8s test environment
(#35099)
new 3421adef5f Fix handling selective docs builds for "special" packages
(#35102)
new 70c481b17b Switch to new labelling scheme allowing for ASF self-hosted
runners (#35109)
new c4a7deb239 Add updating providers metadata during release (#35112)
new a2b6cbb56b Only run generate-constraints step when image is build
(#35116)
new 58901906b5 Add providers metadata for newly released providers (#35111)
new b7ee481fe0 Cleanup dependencies of cache image pushing and ARM image
build (#35117)
new 1ea7b0bac1 Bump min version of presto client to bring better dbapi
compatibility (#35132)
new 5116253e79 Add `--standalone-dag-processor` option to `start-airflow`
command (#35139)
new 875c88eca6 Add `--database-isolation` as option to run breeze with
(#35140)
new d1a994335e Providers requirements for every python version (#35086)
new 0614364838 Allow to install MariaDB client libraries instead of MySQL
on x86_64 (#35070)
new ec4a108e0d Bump docker cli version in Airflow Images (#35153)
new a50685ebd2 Upgrade `pip` to 23.3.1 (#35150)
new 5245a6fadb Improve breeze command image generation and consistency
check (#35169)
new fc337ab549 Limit Airflow and breeze to < 3.12 (#35123)
new 53f765f262 update README_RELEASE_PROVIDER_PACKAGES.md (#35175)
new 53791bb597 Update dev instructions to use breeze command directly with
short package ids to build docs (#35080)
new f91f6317fc Fix local k8s test venv installation to accomodate for
local changes (#35191)
new 2e4f136784 Add missing --no-deps when installing airflow without
providers (#35224)
new 2dc5688a31 Rename the "system" tests where we run Airflow in K8S
(#35238)
new e30fd42e87 Add connexion<3.0 upper bound (#35218)
new 291d4378da Add Duckdb as devel dependency (#35246)
new 6748e17965 Bump azure-mgmt-containerinstance (#34738)
The 54 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/ISSUE_TEMPLATE/airflow_bug_report.yml | 2 +-
.../airflow_providers_bug_report.yml | 1 +
.github/actions/build-ci-images/action.yml | 26 -
.github/actions/build-prod-images/action.yml | 2 +-
.../get-target-branch-build-scripts/action.yml | 43 -
.github/boring-cyborg.yml | 6 +
.github/workflows/build-images.yml | 130 +-
.github/workflows/ci.yml | 276 ++--
.github/workflows/codeql-analysis.yml | 4 +-
.github/workflows/recheck-old-bug-report.yml | 2 +-
.github/workflows/release_dockerhub_image.yml | 10 +-
.github/workflows/stale.yml | 2 +-
.pre-commit-config.yaml | 4 +-
.rat-excludes | 2 +-
BREEZE.rst | 81 +-
CONTRIBUTING.rst | 8 +-
CONTRIBUTORS_QUICK_START.rst | 17 +-
CONTRIBUTORS_QUICK_START_GITPOD.rst | 15 +
Dockerfile | 28 +-
Dockerfile.ci | 58 +-
IMAGES.rst | 2 +-
INSTALL | 8 +-
README.md | 2 +-
airflow/providers/amazon/aws/hooks/s3.py | 8 +-
.../providers}/installed_providers.txt | 0
airflow/providers/microsoft/azure/CHANGELOG.rst | 136 ++
.../microsoft/azure/hooks/container_instance.py | 77 +-
.../azure/operators/container_instances.py | 10 +-
airflow/providers/microsoft/azure/provider.yaml | 8 +-
.../{openlineage => opensearch}/CHANGELOG.rst | 9 +-
.../opensearch}/__init__.py | 0
.../opensearch/hooks}/__init__.py | 0
airflow/providers/opensearch/hooks/opensearch.py | 122 ++
.../opensearch/operators}/__init__.py | 0
.../providers/opensearch/operators/opensearch.py | 194 +++
.../providers/{neo4j => opensearch}/provider.yaml | 49 +-
airflow/providers/presto/provider.yaml | 2 +-
...UALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md | 6 +-
dev/README_RELEASE_AIRFLOW.md | 2 +
dev/README_RELEASE_PROVIDER_PACKAGES.md | 28 +-
dev/breeze/README.md | 16 +-
dev/breeze/SELECTIVE_CHECKS.md | 102 +-
dev/breeze/setup.cfg | 9 +-
.../airflow_breeze/commands/developer_commands.py | 30 +-
.../commands/developer_commands_config.py | 71 +-
.../src/airflow_breeze/commands/main_command.py | 4 +
.../commands/release_management_commands.py | 236 +++-
.../commands/release_management_commands_config.py | 24 +-
.../src/airflow_breeze/commands/sbom_commands.py | 266 +++-
.../commands/sbom_commands_config.py | 37 +-
.../src/airflow_breeze/commands/setup_commands.py | 75 +-
.../commands/setup_commands_config.py | 1 +
.../airflow_breeze/commands/testing_commands.py | 2 +-
dev/breeze/src/airflow_breeze/global_constants.py | 65 +-
.../src/airflow_breeze/params/doc_build_params.py | 13 +-
.../src/airflow_breeze/params/shell_params.py | 2 +
.../airflow_breeze/utils/add_back_references.py | 3 +
dev/breeze/src/airflow_breeze/utils/cdxgen.py | 156 ++-
.../src/airflow_breeze/utils/common_options.py | 48 +-
.../airflow_breeze/utils/docker_command_utils.py | 2 +
.../src/airflow_breeze/utils/general_utils.py | 42 +
.../src/airflow_breeze/utils/kubernetes_utils.py | 47 +-
dev/breeze/src/airflow_breeze/utils/path_utils.py | 5 +-
.../airflow_breeze/utils/publish_docs_helpers.py | 12 +-
dev/breeze/src/airflow_breeze/utils/recording.py | 36 +-
dev/breeze/src/airflow_breeze/utils/reinstall.py | 3 +
.../src/airflow_breeze/utils/selective_checks.py | 96 +-
dev/breeze/tests/test_general_utils.py | 46 +
dev/breeze/tests/test_pr_info.py | 16 +-
dev/breeze/tests/test_selective_checks.py | 511 +++----
.../PROVIDER_COMMITS_TEMPLATE.rst.jinja2 | 1 +
.../build_provider_documentation.sh | 34 -
dev/provider_packages/tag_providers.sh | 14 +-
dev/refresh_images.sh | 2 +-
docker_tests/test_prod_image.py | 2 +-
.../changelog.rst | 4 +-
.../commits.rst} | 18 +-
.../connections}/index.rst | 4 +-
.../connections/opensearch.rst} | 22 +-
.../index.rst | 39 +-
.../installing-providers-from-sources.rst | 0
.../operators}/index.rst | 3 +-
.../operators/opensearch.rst | 72 +
.../security.rst | 0
docs/apache-airflow/extra-packages-ref.rst | 2 +
docs/docker-stack/build-arg-ref.rst | 7 +-
docs/docker-stack/changelog.rst | 10 +
docs/integration-logos/opensearch/opensearch.png | Bin 0 -> 1955 bytes
docs/spelling_wordlist.txt | 2 +
generated/PYPI_README.md | 2 +-
generated/provider_dependencies.json | 17 +-
generated/provider_metadata.json | 1388 +++++++++++++++++---
images/breeze/output-commands-hash.txt | 79 +-
images/breeze/output-commands.svg | 152 ++-
images/breeze/output_build-docs.svg | 196 +--
images/breeze/output_ci-image.svg | 36 +-
images/breeze/output_ci-image_build.svg | 178 +--
images/breeze/output_ci-image_pull.svg | 74 +-
images/breeze/output_ci-image_verify.svg | 48 +-
images/breeze/output_ci.svg | 44 +-
.../output_ci_find-backtracking-candidates.svg | 22 +-
images/breeze/output_ci_fix-ownership.svg | 32 +-
images/breeze/output_ci_free-space.svg | 28 +-
images/breeze/output_ci_get-workflow-info.svg | 30 +-
images/breeze/output_ci_resource-check.svg | 26 +-
images/breeze/output_ci_selective-check.svg | 58 +-
images/breeze/output_cleanup.svg | 34 +-
images/breeze/output_compile-www-assets.svg | 34 +-
images/breeze/output_down.svg | 34 +-
images/breeze/output_exec.svg | 26 +-
images/breeze/output_k8s.svg | 73 +-
images/breeze/output_k8s_build-k8s-image.svg | 60 +-
images/breeze/output_k8s_configure-cluster.svg | 70 +-
images/breeze/output_k8s_create-cluster.svg | 72 +-
images/breeze/output_k8s_delete-cluster.svg | 42 +-
images/breeze/output_k8s_deploy-airflow.svg | 82 +-
images/breeze/output_k8s_k9s.svg | 40 +-
images/breeze/output_k8s_logs.svg | 49 +-
images/breeze/output_k8s_run-complete-tests.svg | 104 +-
images/breeze/output_k8s_setup-env.svg | 32 +-
images/breeze/output_k8s_shell.svg | 48 +-
images/breeze/output_k8s_status.svg | 44 +-
images/breeze/output_k8s_tests.svg | 76 +-
images/breeze/output_k8s_upload-k8s-image.svg | 66 +-
images/breeze/output_prod-image.svg | 32 +-
images/breeze/output_prod-image_build.svg | 200 +--
images/breeze/output_prod-image_pull.svg | 74 +-
images/breeze/output_prod-image_verify.svg | 50 +-
images/breeze/output_release-management.svg | 68 +-
...tput_release-management_add-back-references.svg | 82 +-
...tput_release-management_create-minor-branch.svg | 30 +-
...put_release-management_generate-constraints.svg | 76 +-
...management_generate-issue-content-providers.svg | 108 +-
...ease-management_generate-providers-metadata.svg | 32 +-
...elease-management_install-provider-packages.svg | 88 +-
..._release-management_prepare-airflow-package.svg | 38 +-
...e-management_prepare-provider-documentation.svg | 112 +-
...elease-management_prepare-provider-packages.svg | 94 +-
.../output_release-management_publish-docs.svg | 167 +--
...tput_release-management_release-prod-images.svg | 56 +-
.../output_release-management_start-rc-process.svg | 34 +-
.../output_release-management_start-release.svg | 32 +-
...utput_release-management_update-constraints.svg | 88 +-
...release-management_verify-provider-packages.svg | 70 +-
images/breeze/output_sbom.svg | 40 +-
.../output_sbom_build-all-airflow-images.svg | 156 +++
.../output_sbom_generate-provider-requirements.svg | 64 +-
...output_sbom_generate-providers-requirements.svg | 220 ++++
.../breeze/output_sbom_update-sbom-information.svg | 68 +-
images/breeze/output_setup.svg | 52 +-
images/breeze/output_setup_autocomplete.svg | 34 +-
.../output_setup_check-all-params-in-groups.svg | 68 +-
images/breeze/output_setup_config.svg | 52 +-
.../output_setup_regenerate-command-images.svg | 76 +-
images/breeze/output_setup_self-upgrade.svg | 28 +-
images/breeze/output_setup_version.svg | 26 +-
images/breeze/output_shell.svg | 176 +--
images/breeze/output_start-airflow.svg | 170 ++-
images/breeze/output_static-checks.svg | 158 +--
images/breeze/output_testing.svg | 34 +-
.../breeze/output_testing_docker-compose-tests.svg | 54 +-
images/breeze/output_testing_helm-tests.svg | 50 +-
images/breeze/output_testing_integration-tests.svg | 76 +-
images/breeze/output_testing_tests.svg | 134 +-
scripts/ci/constraints/ci_commit_constraints.sh | 1 -
.../constraints/ci_diff_constraints.sh} | 7 +-
scripts/ci/docker-compose/_docker.env | 4 +-
scripts/ci/docker-compose/base.yml | 2 +
scripts/ci/kubernetes/k8s_requirements.txt | 2 +-
.../ci/pre_commit/pre_commit_breeze_cmd_line.py | 41 +-
.../pre_commit_sort_installed_providers.py | 2 +-
scripts/docker/common.sh | 2 +-
scripts/docker/entrypoint_ci.sh | 26 +-
scripts/docker/install_mysql.sh | 17 +-
scripts/docker/install_os_dependencies.sh | 2 +-
scripts/in_container/_in_container_utils.sh | 11 +-
scripts/in_container/bin/run_tmux | 11 +-
scripts/in_container/run_generate_constraints.sh | 1 +
.../in_container/run_provider_yaml_files_check.py | 381 ++++--
scripts/in_container/verify_providers.py | 6 +-
setup.cfg | 17 +-
setup.py | 15 +-
tests/providers/amazon/aws/hooks/test_s3.py | 6 +-
tests/providers/amazon/aws/triggers/test_athena.py | 35 -
.../azure/hooks/test_azure_container_instance.py | 74 +-
.../providers/opensearch}/__init__.py | 0
tests/providers/opensearch/conftest.py | 60 +
.../providers/opensearch/hooks}/__init__.py | 0
.../providers/opensearch/hooks/test_opensearch.py | 40 +-
.../providers/opensearch/operators}/__init__.py | 0
.../opensearch/operators/test_opensearch.py | 125 ++
.../system/providers/opensearch}/__init__.py | 0
.../providers/opensearch/example_opensearch.py | 129 ++
tests/test_utils/get_all_tests.py | 2 +-
194 files changed, 7419 insertions(+), 3855 deletions(-)
delete mode 100644 .github/actions/get-target-branch-build-scripts/action.yml
rename {scripts/ci => airflow/providers}/installed_providers.txt (100%)
copy airflow/providers/{openlineage => opensearch}/CHANGELOG.rst (92%)
copy airflow/{api_connexion => providers/opensearch}/__init__.py (100%)
copy airflow/{api_connexion => providers/opensearch/hooks}/__init__.py (100%)
create mode 100644 airflow/providers/opensearch/hooks/opensearch.py
copy airflow/{api_connexion => providers/opensearch/operators}/__init__.py
(100%)
create mode 100644 airflow/providers/opensearch/operators/opensearch.py
copy airflow/providers/{neo4j => opensearch}/provider.yaml (59%)
create mode 100644 dev/breeze/src/airflow_breeze/utils/general_utils.py
create mode 100644 dev/breeze/tests/test_general_utils.py
delete mode 100755 dev/provider_packages/build_provider_documentation.sh
copy docs/{apache-airflow-providers-airbyte =>
apache-airflow-providers-opensearch}/changelog.rst (92%)
copy docs/{apache-airflow-providers-qubole/operators/index.rst =>
apache-airflow-providers-opensearch/commits.rst} (71%)
copy docs/{apache-airflow-providers-microsoft-azure/sensors =>
apache-airflow-providers-opensearch/connections}/index.rst (97%)
copy docs/{apache-airflow-providers-asana/connections/asana.rst =>
apache-airflow-providers-opensearch/connections/opensearch.rst} (60%)
copy docs/{apache-airflow-providers-influxdb =>
apache-airflow-providers-opensearch}/index.rst (71%)
copy docs/{apache-airflow-providers-airbyte =>
apache-airflow-providers-opensearch}/installing-providers-from-sources.rst
(100%)
copy docs/{apache-airflow-providers-amazon/connections =>
apache-airflow-providers-opensearch/operators}/index.rst (97%)
create mode 100644
docs/apache-airflow-providers-opensearch/operators/opensearch.rst
copy docs/{apache-airflow-providers-airbyte =>
apache-airflow-providers-opensearch}/security.rst (100%)
create mode 100644 docs/integration-logos/opensearch/opensearch.png
create mode 100644 images/breeze/output_sbom_build-all-airflow-images.svg
create mode 100644
images/breeze/output_sbom_generate-providers-requirements.svg
copy scripts/{in_container/run_fix_ownership.sh =>
ci/constraints/ci_diff_constraints.sh} (82%)
delete mode 100644 tests/providers/amazon/aws/triggers/test_athena.py
copy {airflow/api_connexion => tests/providers/opensearch}/__init__.py (100%)
create mode 100644 tests/providers/opensearch/conftest.py
copy {airflow/api_connexion => tests/providers/opensearch/hooks}/__init__.py
(100%)
copy airflow/migrations/versions/0008_1_6_0_task_duration.py =>
tests/providers/opensearch/hooks/test_opensearch.py (52%)
copy {airflow/api_connexion =>
tests/providers/opensearch/operators}/__init__.py (100%)
create mode 100644 tests/providers/opensearch/operators/test_opensearch.py
copy {airflow/api_connexion => tests/system/providers/opensearch}/__init__.py
(100%)
create mode 100644 tests/system/providers/opensearch/example_opensearch.py
