This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a change to branch jason/lang-sdk-e2e/03-native-dag-parse
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard cd38d3392b0 Report a native Dag's task that reaches a Python worker
discard 5bb5f71ce57 Report a Lang-SDK parse past its import timeout as an
import error
discard d2e4f162d06 Parse coordinator-claimed Dag files with their runtime in
the Dag processor
discard 107f1eb6dc0 Let a coordinator parse the Dag files of the bundles it
serves
discard 35704801174 Fill a serialized Dag's unset settings from the Airflow
config
discard d67fee105db Check that a serialized Dag can be stored and loaded
discard 0b47cba3704 Move the Dag file processor's shared plumbing into a base
class
discard 541001b9ee1 Share Dag cycle detection between the Task SDK and core
add e4d397f021e Bound get_schema results for very wide tables in the SQL
toolsets (#74017)
add 7d6244bf191 Clarify `max_db_retries` doc wording to avoid off-by-one
confusion (#73722)
add bb463bf9230 Correlate executor task events by attempt UUID (#73916)
add c19faf0c5ae Render the HITL review workflow as a Mermaid diagram
(#74024)
add 2aa31ae7bff Use airflow.sdk imports in Airflow 3 upgrade guide example
(#74015)
add f3532c60eda Use Task SDK TaskInstanceState in OpenSearch log handler
(#72444)
add 0ec26723858 Add GKEPodExecOperator for existing Pods (#72577)
add 740986adfce Add missing cycle assertion to test_semi_complex (#71973)
add fca1c01c73e Fix Windows UnicodeDecodeError during pytest collection
(#73536)
add 1787ef7cbba Run the deferred AWS hook configuration check as a prek
hook (#73681)
add 413ae46a045 Align timetable docs with Airflow 3 data interval and
catchup behaviour (#70658)
add 57e89c0a676 Share Dag cycle detection between the Task SDK and core
add e5ae872c9a1 Pin the empty task id case in the Dag cycle detection tests
add 3ff4e27860a Move the Dag file processor's shared plumbing into a base
class
add 1153e7eecc9 Check that a serialized Dag can be stored and loaded
add d996d4f88ad Fill a serialized Dag's unset settings from the Airflow
config
add aefc2d92e62 Reject malformed task entries in a serialized Dag
add 0f3006a6769 Check SDK Dags with the Dag processor's validation in
conformance
add ad14011af01 Note in the parsing ADR that runtimes may omit
config-backed fields
add 7bf33a97660 Let a coordinator parse the Dag files of the bundles it
serves
add 16e4118bb3a Import the coordinator manager at the top of the importer
base
add 90e99e3cd7f Check Dag file claims per bundle on the parse path
add e3fc4402ad7 Drop serves_bundle so get_parsed_bundles is the only answer
add 8e0e58455bd Parse coordinator-claimed Dag files with their runtime in
the Dag processor
add 97b61d1076d Report a Lang-SDK parse past its import timeout as an
import error
add c94553e7f90 Report a native Dag's task that reaches a Python worker
add 0f933aa2b05 Drop callbacks for Lang-SDK Dag files before queuing the
file
add 11346702645 Start each Dag file processor with explicit arguments
add 273be37c009 Return a Dag importer from every SubprocessCoordinator call
add 6e7fbd6958e Find the claiming coordinator in the Task SDK and wrap it
in core
add f5134f7ca3a Fail a native Dag's task on a Python worker without retries
add b70d91a5584 Build each bundle's Dag importer registry before freezing
the heap
add 6ec3b9d872a Match the schema error's JSON path in a Lang-SDK parse test
add 5e9b6c34007 Kill what a Lang-SDK runtime leaves in its process group
add 9d2b068fc24 Name the setting in a Lang-SDK parse timeout error
add 6dceaa7f7b4 Spec the mocks of a hand-built Lang-SDK processor in tests
add 5f7f31b6776 Apply the multi-team task checks to Lang-SDK Dags
add 2a3e14ba73e Clear import errors after the native-file Dag bag sync test
add 5b9cd9c3664 Note in ADR-0010 that the manager routes claimed files for
now
add b2ce95c7167 Move the client-less Lang-SDK parse out of this change
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (cd38d3392b0)
\
N -- N -- N refs/heads/jason/lang-sdk-e2e/03-native-dag-parse
(b2ce95c7167)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
airflow-core/adr/lang-sdk/0004-dag-parsing.md | 4 +-
.../adr/lang-sdk/0010-native-dag-processing.md | 22 +-
.../docs/authoring-and-scheduling/timetable.rst | 154 ++++++---
airflow-core/docs/core-concepts/executor/index.rst | 23 +-
.../docs/installation/upgrading_to_airflow3.rst | 2 +-
.../src/airflow/config_templates/config.yml | 6 +-
airflow-core/src/airflow/dag_processing/dagbag.py | 43 +--
.../src/airflow/dag_processing/importer_routing.py | 21 +-
.../airflow/dag_processing/lang_sdk_processor.py | 158 ++++-----
airflow-core/src/airflow/dag_processing/manager.py | 54 ++-
.../src/airflow/executors/base_executor.py | 123 ++++++-
.../src/airflow/executors/local_executor.py | 13 +-
.../src/airflow/executors/workloads/task.py | 3 +-
.../src/airflow/executors/workloads/types.py | 17 +-
.../src/airflow/jobs/scheduler_job_runner.py | 49 +--
airflow-core/src/airflow/models/taskinstance.py | 2 +-
airflow-core/src/airflow/serialization/schema.json | 13 +-
.../airflow/serialization/serialized_objects.py | 15 +-
.../versions/head/test_task_instances.py | 5 +-
.../tests/unit/dag_processing/test_dagbag.py | 9 +-
.../unit/dag_processing/test_importer_routing.py | 36 +-
.../unit/dag_processing/test_lang_sdk_processor.py | 255 +++++++-------
.../tests/unit/dag_processing/test_manager.py | 93 +++--
.../tests/unit/executors/test_base_executor.py | 278 +++++++++++++--
.../tests/unit/executors/test_local_executor.py | 10 +-
.../tests/unit/executors/test_workloads.py | 16 +-
airflow-core/tests/unit/jobs/test_scheduler_job.py | 181 +++++++---
.../tests/unit/models/test_taskinstance.py | 12 +
.../unit/serialization/test_dag_serialization.py | 37 +-
.../unit/serialization/test_dag_version_diff.py | 1 -
devel-common/src/tests_common/pytest_plugin.py | 9 +-
.../src/tests_common/test_utils/mock_executor.py | 5 +-
.../src/tests_common/test_utils/providers.py | 2 +-
generated/known_sdk_imports_in_core.txt | 3 +-
providers/.pre-commit-config.yaml | 7 +
.../amazon/aws/test_deferred_hook_configuration.py | 242 +------------
.../unit/celery/executors/test_celery_executor.py | 4 +
.../executors/test_kubernetes_executor.py | 2 +
providers/common/ai/docs/changelog.rst | 12 +
providers/common/ai/docs/hitl_review.rst | 63 ++--
providers/common/ai/docs/toolsets/datafusion.rst | 11 +-
providers/common/ai/docs/toolsets/sql.rst | 41 ++-
.../providers/common/ai/toolsets/datafusion.py | 35 +-
.../airflow/providers/common/ai/toolsets/sql.py | 33 +-
.../providers/common/ai/utils/query_results.py | 202 ++++++++++-
.../unit/common/ai/toolsets/test_datafusion.py | 55 ++-
.../ai/tests/unit/common/ai/toolsets/test_sql.py | 62 +++-
.../unit/common/ai/utils/test_query_results.py | 167 ++++++++-
providers/google/docs/index.rst | 8 +-
.../docs/operators/cloud/kubernetes_engine.rst | 20 ++
providers/google/pyproject.toml | 2 +-
.../google/cloud/operators/kubernetes_engine.py | 114 +++++++
.../kubernetes_engine/example_kubernetes_engine.py | 105 +++++-
.../cloud/operators/test_kubernetes_engine.py | 146 +++++++-
.../providers/opensearch/log/os_task_handler.py | 3 +-
.../ci/lang_sdk_serialization/serialize_python.py | 23 +-
.../ci/prek/check_deferred_hook_configuration.py | 294 ++++++++++++++++
scripts/ci/prek/common_prek_utils.py | 26 +-
scripts/ci/prek/update_providers_dependencies.py | 14 +-
.../prek/test_check_deferred_hook_configuration.py | 373 +++++++++++++++++++++
shared/dagnode/tests/dagnode/test_cycle.py | 10 +-
.../src/airflow/sdk/coordinators/_dag_importer.py | 17 +-
.../src/airflow/sdk/coordinators/_subprocess.py | 16 +-
.../src/airflow/sdk/execution_time/coordinator.py | 125 +++----
.../src/airflow/sdk/execution_time/task_runner.py | 52 ++-
task-sdk/src/airflow/sdk/importers/base.py | 20 +-
.../src/airflow/sdk/importers/python_importer.py | 4 +-
task-sdk/src/airflow/sdk/importers/zip_importer.py | 6 +-
.../tests/task_sdk/coordinators/test_subprocess.py | 19 +-
task-sdk/tests/task_sdk/definitions/test_dag.py | 2 +
.../task_sdk/execution_time/test_coordinator.py | 196 +++++++----
.../task_sdk/execution_time/test_task_runner.py | 119 ++++---
task-sdk/tests/task_sdk/importers/test_registry.py | 10 +-
ts-sdk/schema/dag-schema.json | 13 +-
74 files changed, 3277 insertions(+), 1070 deletions(-)
create mode 100755 scripts/ci/prek/check_deferred_hook_configuration.py
create mode 100644
scripts/tests/ci/prek/test_check_deferred_hook_configuration.py
