This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a change to branch jason/lang-sdk-e2e/02-importer-registry
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard 98bfa5be9b4 Harden claimed-file discovery errors and archive importers
discard 89e8f78cb15 Tidy the importer routing helpers and their tests
discard 12da1d38851 Keep claimed files queued when their importer cannot list
them
discard eb7b084b292 Claim Dag files by extension and keep .py, .pyc and .zip
legacy
discard edefb917b15 Read a claimed file's Dag source through its importer
discard 441fce1cdf4 Queue and track files that a Task SDK importer claims
discard 5feefffb6b7 Parse files that a Task SDK importer claims
omit 18334707dec Reject blank subprocess coordinator artifact roots
omit e8e1a8ba506 Reject empty subprocess coordinator artifact roots
omit bada0a79306 Reject concurrent subprocess coordinator execution
consistently
omit a6557be73bd Make task bundle access dependencies explicit
omit 68cca25a3e1 Preserve custom subprocess coordinator compatibility
omit e9da1565f5b Read the Dag bundle configuration through a single entry
point
omit 8c674c638d5 Resolve coordinator Dag bundles without importing the task
runner
omit e6c1deca9d9 Make JavaCoordinator main_class optional in co-located mode
omit 0e857247f41 Harden subprocess coordinator artifact bundle resolution
omit 9257c50cf16 Document dag_bundle_name option for language-SDK
coordinators
omit d8c78379e77 Add dag_bundle_name option to subprocess coordinators
add 3f60fa9e2c7 Fix Firefox and WebKit UI e2e runs failing at login
(#73872)
add ac88accf4ed Add bundle_name tag to missing dag processor metrics
(#73717)
add 7afece8efb0 Update providers metadata 2026-09-29 (#73884)
add 4b6606543fd Distinguish OpenAI batch timeout and cancellation in
deferrable tasks (#72149)
add 8b3d4d315fa Surface the retry policy decision on task instances page
(#73030)
add bc28bb0804c Go SDK: Fix the mixed language task handler binding
semantic (#73648)
add 9ebc2644741 Revert "Secrets backends order updated (#70681)" (#73891)
add 4e23910701b Go SDK: check Go naming conventions with staticcheck
(#73874)
add 0768924a0cd Automatically isolate `breeze testing` for each git
worktree in a totally isolated project. (#73836)
add 3e8318969e1 Add ADR for Lang SDK Dag and mixed-language Task
processing flow (#71929)
add e8d43932077 Add ADR for supporting Dynamic Dag Generation across
Language SDKs (#73518)
add 8d815efdbc3 Own the BaseAsyncOperator event loop through
asyncio.Runner (#73560)
add 647add72afe Mark experimental `common.ai` features ahead of 1.0
(#73896)
add 5eba7439d76 Make cron timetables hashable (#73859)
add d776f003b71 UI: Add team column and filter to the jobs list (#73295)
add d0b8476825d Set `SameSite` on the fab auth manager session cookie
(#73894)
add 6b90003fe27 Document override semantics for EcsRunTaskOperator (#73860)
add a3555c74cfc TS SDK: default a task id to the handler's name (#73437)
add a65a3ddf8be Fix Postgres data volume path in Breeze (#73862)
add 85321901f10 Add EmrServerlessStartSessionOperator to Amazon provider
(#70763)
add 9198acb15e8 UI: Add team filter to Human-in-the-loop task instances
listing (#73726)
add 836a5565c58 Scope plugin scheduling classes to their team in
multi-team mode (#73569)
add cc19675d24f Add DuckDB support to the Amazon provider (#73301)
add 029cdf8f6cc Return an approved agent answer as `output_type` after
human review (#73904)
add 3c76a47f65c Remove accidentally adopted Magpie contributor-growth
plugin (#73927)
add 73b07c5aedb Name the tools a durable agent retry will run again
(#73873)
add 5799660c4d4 TS SDK: draw order-only edges with before and after
(#73438)
add a8a62573f54 TS SDK: embed one source region per native Dag file
(#73723)
add bfa2fe02fc1 Account for AgentOperator spend on failed runs and across
retries (#73706)
add 023f859e5a6 Mask secrets in tool results before they reach the model
(#73897)
add 2b806f949b0 Rework common.ai Strands support into a plugin and add
Google ADK (#73898)
add e46d1938619 Add `ObjectStorageToolset` for reading files on object
storage (#73899)
add 35507e99a6e Let HookToolset pin arguments the model must not choose
(#73900)
add 40ed903b6d6 Count and trace tool calls made by native agent frameworks
(#73901)
add 2eb11c14b85 Document running your own Pydantic AI agent, and sandboxes
in other frameworks (#73902)
add d4ba567a390 Tell the model how many rows a Parquet or Avro file holds
(#73925)
add 21bbf2f40a2 Java SDK: Generate Dag and task configuration from the
serialization schema (#73595)
add 76bf88a4877 API: Translate domain ValueError in Task State Store route
(#73753)
add 1a67bf46b55 Support DagBundle based deployment for Coordinator (#70805)
add 53ca2a427ba TS SDK: add task groups to the native Dag surface (#73439)
add 5e785acfb3b Discard the task state values for a task instance when it
is cleared (#72100)
add 7938c0dbb65 Make `execute_tool` the public method `AirflowToolset`
subclasses implement (#73938)
add 6cd2f6f36d9 Remove migration tests for migrations that have already
landed. (#73953)
add 5136c6c41bf Use plain ifs to normalize configured artifact roots
(#73944)
add e15b3a66137 Retry S3Hook.delete_bucket when a late write leaves bucket
non-empty (#73930)
add acd42922f3d Filter jobs API is_alive in SQL so pagination stays
correct (#73684)
add 508301ab263 Add ADR for per-Dag source in the Lang-SDK bundle artifact
(#73948)
add 91423baf366 Fix breeze down crashing when a start-airflow container is
running (#73955)
add 0fdb0f9da61 Parse files that a Task SDK importer claims
add f8ac157c155 Queue and track files that a Task SDK importer claims
add a9af8a9b209 Read a claimed file's Dag source through its importer
add f48fbb795a6 Claim Dag files by extension and keep .py, .pyc and .zip
legacy
add b01e97995a2 Keep claimed files queued when their importer cannot list
them
add 53a4447965d Tidy the importer routing helpers and their tests
add 57c9a34141c Harden claimed-file discovery errors and archive importers
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (98bfa5be9b4)
\
N -- N -- N refs/heads/jason/lang-sdk-e2e/02-importer-registry
(57c9a34141c)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.../committer-onboarding-config.md | 272 ----
.apache-magpie-overrides/committer-readiness.md | 169 ---
.../contributor-nomination-config.md | 186 ---
.../contributor-sentiment-config.md | 66 -
.../good-first-issue-config.md | 93 --
.../onboarding-concierge-config.md | 70 -
.apache-magpie.lock | 43 -
.pre-commit-config.yaml | 11 +
.../0007-taskflow-across-language-boundary.md | 39 +
.../adr/lang-sdk/0010-native-dag-processing.md | 212 +++
.../lang-sdk/0011-mixed-language-dag-processing.md | 245 ++++
.../adr/lang-sdk/0012-lang-sdk-parse-protocol.md | 233 ++++
.../0013-persisted-task-handler-bindings.md | 593 +++++++++
.../0014-bundle-metadata-and-cache-digest.md | 122 ++
.../0015-per-dag-source-in-bundle-artifact.md | 127 ++
airflow-core/adr/lang-sdk/README.md | 6 +
.../task-and-asset-state-store.rst | 4 +
.../language-sdks/java.rst | 37 +-
.../language-sdks/typescript.rst | 101 +-
airflow-core/docs/core-concepts/dag-run.rst | 11 +-
.../docs/core-concepts/resumable-tasks.rst | 75 +-
.../docs/core-concepts/task-state-store.rst | 6 +-
.../security/secrets/secrets-backend/index.rst | 23 +-
airflow-core/newsfragments/70681.significant.rst | 7 -
airflow-core/newsfragments/72100.significant.rst | 59 +
.../api_fastapi/common/parameters/__init__.py | 4 +
.../airflow/api_fastapi/common/parameters/job.py | 50 +
.../airflow/api_fastapi/common/parameters/misc.py | 43 +
.../core_api/datamodels/task_instance_history.py | 19 +-
.../core_api/datamodels/task_instances.py | 23 +-
.../api_fastapi/core_api/openapi/_private_ui.yaml | 107 +-
.../core_api/openapi/v2-rest-api-generated.yaml | 57 +-
.../api_fastapi/core_api/routes/public/hitl.py | 4 +
.../api_fastapi/core_api/routes/public/job.py | 10 +-
.../core_api/routes/public/task_instances.py | 8 +
.../core_api/routes/public/task_state_store.py | 7 +-
.../api_fastapi/core_api/routes/ui/config.py | 45 +-
.../core_api/services/public/task_instances.py | 41 +
.../airflow/cli/commands/dag_processor_command.py | 26 +-
.../src/airflow/config_templates/config.yml | 34 -
airflow-core/src/airflow/configuration.py | 7 +-
.../src/airflow/dag_processing/collection.py | 104 ++
.../src/airflow/dag_processing/importer_routing.py | 6 +-
airflow-core/src/airflow/dag_processing/manager.py | 19 +-
airflow-core/src/airflow/models/dagcode.py | 15 +-
airflow-core/src/airflow/plugins_manager.py | 42 +
airflow-core/src/airflow/secrets/__init__.py | 4 +-
airflow-core/src/airflow/secrets/base_secrets.py | 5 +-
airflow-core/src/airflow/timetables/_cron.py | 2 +-
.../src/airflow/ui/openapi-gen/queries/common.ts | 16 +-
.../ui/openapi-gen/queries/ensureQueryData.ts | 24 +-
.../src/airflow/ui/openapi-gen/queries/prefetch.ts | 24 +-
.../src/airflow/ui/openapi-gen/queries/queries.ts | 24 +-
.../src/airflow/ui/openapi-gen/queries/suspense.ts | 24 +-
.../airflow/ui/openapi-gen/requests/schemas.gen.ts | 30 +
.../ui/openapi-gen/requests/services.gen.ts | 34 +-
.../airflow/ui/openapi-gen/requests/types.gen.ts | 43 +-
.../airflow/ui/public/i18n/locales/en/admin.json | 2 -
.../airflow/ui/public/i18n/locales/en/common.json | 9 +
.../airflow/ui/public/i18n/locales/en/dags.json | 1 +
.../TaskInstance/ClearGroupTaskInstanceDialog.tsx | 40 +-
.../TaskInstance/ClearTaskInstanceDialog.test.tsx | 127 ++
.../Clear/TaskInstance/ClearTaskInstanceDialog.tsx | 23 +-
.../src/airflow/ui/src/components/StatsCard.tsx | 2 +-
.../src/airflow/ui/src/constants/localStorage.ts | 1 +
.../airflow/ui/src/hooks/useUserSettings.test.tsx | 21 +
.../src/airflow/ui/src/hooks/useUserSettings.ts | 4 +
.../ui/src/pages/HITLTaskInstances/HITLFilters.tsx | 6 +
.../HITLTaskInstances/HITLTaskInstances.test.tsx | 62 +-
.../pages/HITLTaskInstances/HITLTaskInstances.tsx | 21 +
airflow-core/src/airflow/ui/src/pages/Jobs.tsx | 22 +-
.../ui/src/pages/Settings/Settings.test.tsx | 8 +
.../src/airflow/ui/src/pages/Settings/Settings.tsx | 13 +
.../ui/src/pages/TaskInstance/Details.test.tsx | 137 ++
.../airflow/ui/src/pages/TaskInstance/Details.tsx | 16 +
.../ui/src/pages/TaskInstance/Header.test.tsx | 54 +
.../airflow/ui/src/pages/TaskInstance/Header.tsx | 20 +
.../TaskInstance/stateReason.ts} | 35 +-
.../TaskInstances/BulkClearTaskInstancesButton.tsx | 34 +-
.../ui/src/pages/Variables/BackendsOrderCard.tsx | 45 -
.../ui/src/pages/Variables/BackendsOrderModal.tsx | 64 -
.../airflow/ui/src/pages/Variables/Variables.tsx | 2 -
.../ui/src/queries/useBulkClearTaskInstances.ts | 2 +
.../src/airflow/ui/tests/e2e/global-setup.ts | 14 +-
airflow-core/tests/unit/always/test_secrets.py | 121 --
.../core_api/routes/public/test_hitl.py | 49 +-
.../api_fastapi/core_api/routes/public/test_job.py | 51 +
.../core_api/routes/public/test_task_instances.py | 210 +++
.../routes/public/test_task_state_store.py | 14 +
.../api_fastapi/core_api/routes/ui/test_config.py | 62 -
.../cli/commands/test_dag_processor_command.py | 58 +
.../tests/unit/dag_processing/fake_importers.py | 15 +-
.../tests/unit/dag_processing/test_collection.py | 347 ++++-
.../tests/unit/dag_processing/test_manager.py | 92 +-
...est_0049_remove_pickled_data_from_xcom_table.py | 237 ----
...t_0055_remove_pickled_data_from_dagrun_table.py | 143 --
.../test_0080_deadline_callback_migration.py | 268 ----
.../test_0094_deadline_callback_migration.py | 178 ---
.../test_0117_deadline_interval_json_migration.py | 91 --
...9_add_indexes_on_serialized_dag_and_dag_code.py | 55 -
.../migrations/test_0131_lower_case_team_names.py | 138 --
.../test_0136_fold_task_map_into_xcom.py | 184 ---
...t_0138_allocate_pending_task_attempt_numbers.py | 448 -------
airflow-core/tests/unit/models/test_dagcode.py | 36 +-
.../tests/unit/plugins/test_plugins_manager.py | 61 +
.../unit/timetables/test_trigger_timetable.py | 7 +
.../src/airflowctl/api/datamodels/generated.py | 21 +
airflow-ctl/src/airflowctl/api/operations.py | 2 +-
.../tests/airflow_ctl/api/test_operations.py | 19 +-
.../go_sdk_tests/test_go_sdk_taskflow_binding.py | 72 +-
dev/breeze/doc/03_developer_tasks.rst | 44 +-
dev/breeze/doc/04_troubleshooting.rst | 11 +
dev/breeze/doc/05_test_commands.rst | 14 +
dev/breeze/doc/images/output-commands.svg | 165 ++-
dev/breeze/doc/images/output_cleanup.svg | 2 +-
dev/breeze/doc/images/output_cleanup.txt | 2 +-
dev/breeze/doc/images/output_down.svg | 48 +-
dev/breeze/doc/images/output_down.txt | 2 +-
dev/breeze/doc/images/output_run.svg | 45 +-
dev/breeze/doc/images/output_run.txt | 2 +-
dev/breeze/doc/images/output_shell.svg | 285 ++--
dev/breeze/doc/images/output_shell.txt | 2 +-
dev/breeze/doc/images/output_start-airflow.svg | 237 ++--
dev/breeze/doc/images/output_start-airflow.txt | 2 +-
.../doc/images/output_testing_core-tests.svg | 145 +-
.../doc/images/output_testing_core-tests.txt | 2 +-
.../doc/images/output_testing_providers-tests.svg | 161 ++-
.../doc/images/output_testing_providers-tests.txt | 2 +-
.../src/airflow_breeze/commands/common_options.py | 10 +-
.../airflow_breeze/commands/developer_commands.py | 52 +-
.../commands/developer_commands_config.py | 2 +-
.../src/airflow_breeze/commands/main_command.py | 23 +-
.../airflow_breeze/commands/testing_commands.py | 12 +-
.../commands/testing_commands_config.py | 5 +-
dev/breeze/src/airflow_breeze/global_constants.py | 15 +-
.../src/airflow_breeze/params/shell_params.py | 17 +-
.../airflow_breeze/utils/docker_command_utils.py | 248 ++--
dev/breeze/src/airflow_breeze/utils/path_utils.py | 10 +-
dev/breeze/tests/test_developer_commands.py | 38 +-
dev/breeze/tests/test_docker_command_utils.py | 487 +++++--
dev/breeze/tests/test_selective_checks.py | 10 +-
dev/breeze/tests/test_shell_params.py | 82 ++
.../src/tests_common/test_utils/mock_plugins.py | 42 +-
docs/spelling_wordlist.txt | 1 +
generated/provider_metadata.json | 496 ++++---
go-sdk/.pre-commit-config.yaml | 7 +
go-sdk/README.md | 14 +-
go-sdk/adr/0006-cross-language-argument-binding.md | 15 +-
.../adr/0007-mixed-lang-task-handler-interface.md | 6 +-
go-sdk/airflow/bundle.go | 20 +-
go-sdk/airflow/task_handler.go | 10 +-
go-sdk/airflow/task_handler_test.go | 6 +-
.../cmd/airflow-go-pack/pack_integration_test.go | 4 +-
go-sdk/dags/go_examples.py | 28 +-
.../bundle/concurrentxcom/concurrentxcom_test.go | 2 +-
go-sdk/example/bundle/main.go | 14 +-
.../bundle/taskflowbinding/taskflowbinding.go | 61 +-
.../bundle/taskflowbinding/taskflowbinding_test.go | 44 +-
go-sdk/internal/bundle/task.go | 2 +-
go-sdk/pkg/binding/binding.go | 56 +-
go-sdk/pkg/binding/binding_test.go | 106 +-
go-sdk/pkg/execution/client.go | 22 +-
go-sdk/pkg/execution/client_test.go | 6 +-
go-sdk/pkg/execution/comms.go | 13 +-
go-sdk/pkg/execution/comms_test.go | 10 +-
go-sdk/pkg/execution/integration_test.go | 14 +-
go-sdk/pkg/execution/messages.go | 6 +-
go-sdk/pkg/execution/messages_test.go | 4 +-
go-sdk/sdk/sdk.go | 4 +-
java-sdk/README.md | 19 +-
.../example/nativedag/InterfaceExample.java | 18 +-
.../org/apache/airflow/sdk/BuilderProcessor.kt | 118 +-
.../kotlin/org/apache/airflow/sdk/BuilderTest.kt | 172 ++-
java-sdk/sdk/build.gradle.kts | 443 ++++++-
{ts-sdk => java-sdk/sdk}/schema/dag-schema.json | 0
.../src/main/kotlin/org/apache/airflow/sdk/Arg.kt | 18 +
.../main/kotlin/org/apache/airflow/sdk/Builder.kt | 109 --
.../main/kotlin/org/apache/airflow/sdk/DagDef.kt | 69 +-
.../src/main/kotlin/org/apache/airflow/sdk/Deps.kt | 18 +-
.../org/apache/airflow/sdk/internal/Fields.kt | 107 ++
.../kotlin/org/apache/airflow/sdk/DagDefTest.kt | 179 +++
providers/amazon/docs/index.rst | 2 +
providers/amazon/docs/operators/duckdb.rst | 180 +++
providers/amazon/docs/operators/ecs.rst | 35 +
.../amazon/docs/operators/emr/emr_serverless.rst | 23 +
providers/amazon/provider.yaml | 31 +
providers/amazon/pyproject.toml | 9 +-
.../airflow/providers/amazon/aws/hooks/duckdb.py | 277 ++++
.../src/airflow/providers/amazon/aws/hooks/emr.py | 51 +
.../src/airflow/providers/amazon/aws/hooks/s3.py | 8 +-
.../airflow/providers/amazon/aws/operators/ecs.py | 3 +
.../airflow/providers/amazon/aws/operators/emr.py | 112 ++
.../airflow/providers/amazon/aws/triggers/emr.py | 48 +
.../amazon/aws/waiters/emr-serverless.json | 37 +
.../airflow/providers/amazon/get_provider_info.py | 23 +
.../{example_gcs_to_s3.py => example_duckdb.py} | 120 +-
.../amazon/aws/example_emr_serverless_session.py | 103 ++
.../tests/unit/amazon/aws/hooks/test_duckdb.py | 307 +++++
.../unit/amazon/aws/hooks/test_emr_serverless.py | 57 +
.../amazon/tests/unit/amazon/aws/hooks/test_s3.py | 19 +
.../aws/operators/test_emr_serverless_session.py | 137 ++
.../tests/unit/amazon/aws/triggers/test_emr.py | 60 +
providers/common/ai/docs/agent_security.rst | 32 +-
providers/common/ai/docs/approval_gates.rst | 5 +
providers/common/ai/docs/changelog.rst | 14 +
providers/common/ai/docs/classifier_models.rst | 5 +
providers/common/ai/docs/code_mode.rst | 5 +
providers/common/ai/docs/connections/langchain.rst | 5 +
.../common/ai/docs/connections/llamaindex.rst | 5 +
providers/common/ai/docs/durable_execution.rst | 19 +-
providers/common/ai/docs/frameworks/adk.rst | 109 ++
providers/common/ai/docs/frameworks/index.rst | 87 +-
.../common/ai/docs/frameworks/pydantic_ai.rst | 78 ++
providers/common/ai/docs/frameworks/strands.rst | 96 +-
providers/common/ai/docs/guardrails.rst | 6 +
providers/common/ai/docs/hooks/langchain.rst | 5 +
providers/common/ai/docs/hooks/llamaindex.rst | 5 +
providers/common/ai/docs/index.rst | 1 +
providers/common/ai/docs/observability.rst | 119 +-
providers/common/ai/docs/operators/agent.rst | 58 +-
.../common/ai/docs/operators/document_loader.rst | 5 +
.../ai/docs/operators/llamaindex_embedding.rst | 5 +
.../ai/docs/operators/llamaindex_retrieval.rst | 5 +
providers/common/ai/docs/operators/llm.rst | 35 +-
providers/common/ai/docs/operators/llm_batch.rst | 5 +
providers/common/ai/docs/operators/llm_branch.rst | 5 +
.../common/ai/docs/operators/llm_file_analysis.rst | 5 +
.../ai/docs/operators/llm_schema_compare.rst | 5 +
providers/common/ai/docs/operators/llm_sql.rst | 5 +
providers/common/ai/docs/rag_pipelines.rst | 5 +
providers/common/ai/docs/retry_policies.rst | 21 +-
providers/common/ai/docs/sandbox/backends.rst | 5 +
providers/common/ai/docs/sandbox/configuration.rst | 5 +
providers/common/ai/docs/sandbox/index.rst | 39 +
providers/common/ai/docs/stability.rst | 188 +++
providers/common/ai/docs/tool_approval.rst | 5 +
providers/common/ai/docs/toolsets/datafusion.rst | 5 +
providers/common/ai/docs/toolsets/hook.rst | 60 +-
providers/common/ai/docs/toolsets/index.rst | 13 +-
providers/common/ai/docs/toolsets/langchain.rst | 39 +-
providers/common/ai/docs/toolsets/logging.rst | 5 +
.../common/ai/docs/toolsets/managed_agent.rst | 9 +-
.../common/ai/docs/toolsets/object_storage.rst | 124 ++
providers/common/ai/docs/toolsets/skills.rst | 5 +
providers/common/ai/provider.yaml | 1 +
.../airflow/providers/common/ai/batch/__init__.py | 2 +-
.../airflow/providers/common/ai/batch/anthropic.py | 5 +
.../src/airflow/providers/common/ai/batch/base.py | 10 +
.../airflow/providers/common/ai/batch/dispatch.py | 5 +
.../airflow/providers/common/ai/batch/openai.py | 5 +
.../providers/common/ai/decorators/llm_batch.py | 5 +
.../common/ai/decorators/llm_file_analysis.py | 5 +
.../common/ai/decorators/llm_schema_compare.py | 5 +
.../providers/common/ai/decorators/llm_sql.py | 5 +
.../airflow/providers/common/ai/durable/base.py | 19 +-
.../providers/common/ai/durable/caching_model.py | 122 +-
.../providers/common/ai/durable/caching_toolset.py | 47 +-
.../providers/common/ai/durable/replay_usage.py | 240 ++++
.../providers/common/ai/durable/step_counter.py | 4 +
.../airflow/providers/common/ai/durable/storage.py | 16 +-
.../common/ai/durable/task_state_store.py | 16 +-
.../common/ai/example_dags/example_adk_agent.py | 105 ++
.../example_dags/example_object_storage_toolset.py | 48 +
.../ai/example_dags/example_pydantic_ai_agent.py | 72 +
.../ai/example_dags/example_strands_agent.py | 38 +-
.../airflow/providers/common/ai/hooks/langchain.py | 5 +
.../providers/common/ai/hooks/llamaindex.py | 5 +
.../airflow/providers/common/ai/operators/agent.py | 438 +++++--
.../common/ai/operators/document_loader.py | 5 +
.../common/ai/operators/llamaindex_embedding.py | 5 +
.../common/ai/operators/llamaindex_retrieval.py | 5 +
.../airflow/providers/common/ai/operators/llm.py | 3 +-
.../providers/common/ai/operators/llm_batch.py | 5 +
.../providers/common/ai/operators/llm_branch.py | 6 +-
.../common/ai/operators/llm_file_analysis.py | 5 +
.../common/ai/operators/llm_schema_compare.py | 5 +
.../providers/common/ai/operators/llm_sql.py | 5 +
.../providers/common/ai/policies/decision.py | 11 +
.../airflow/providers/common/ai/policies/retry.py | 17 +-
.../airflow/providers/common/ai/sandbox/base.py | 15 +
.../airflow/providers/common/ai/sandbox/modal.py | 5 +
.../providers/common/ai/sandbox/opensandbox.py | 5 +
.../airflow/providers/common/ai/sandbox/output.py | 6 +-
.../src/airflow/providers/common/ai/sandbox/sbx.py | 5 +
.../ai/src/airflow/providers/common/ai/skills.py | 5 +
.../airflow/providers/common/ai/tools/__init__.py | 121 +-
.../providers/common/ai/tools/_from_toolset.py | 209 ++-
.../src/airflow/providers/common/ai/tools/adk.py | 112 ++
.../airflow/providers/common/ai/tools/strands.py | 107 +-
.../airflow/providers/common/ai/tools/tracing.py | 192 +++
.../providers/common/ai/toolsets/__init__.py | 2 +
.../providers/common/ai/toolsets/datafusion.py | 31 +-
.../airflow/providers/common/ai/toolsets/hook.py | 84 +-
.../common/ai/toolsets/langchain_bridge.py | 163 +--
.../providers/common/ai/toolsets/logging.py | 9 +-
.../providers/common/ai/toolsets/managed_agent.py | 28 +-
.../airflow/providers/common/ai/toolsets/mcp.py | 35 +-
.../providers/common/ai/toolsets/object_storage.py | 384 ++++++
.../providers/common/ai/toolsets/sandbox.py | 124 +-
.../airflow/providers/common/ai/toolsets/skills.py | 5 +
.../airflow/providers/common/ai/toolsets/sql.py | 59 +-
.../providers/common/ai/triggers/llm_batch.py | 5 +
.../providers/common/ai/utils/file_analysis.py | 123 +-
.../airflow/providers/common/ai/utils/logging.py | 42 +-
.../airflow/providers/common/ai/utils/masking.py | 120 ++
.../providers/common/ai/utils/query_results.py | 7 +-
.../providers/common/ai/utils/tool_definition.py | 13 +-
.../providers/common/ai/utils/tool_metrics.py | 63 +
.../providers/common/ai/utils/toolset_base.py | 250 ++++
.../providers/common/ai/utils/usage_budget.py | 214 +++
.../tests/unit/common/ai/decorators/test_agent.py | 9 +-
.../unit/common/ai/durable/test_caching_model.py | 252 +++-
.../unit/common/ai/durable/test_caching_toolset.py | 36 +-
.../unit/common/ai/durable/test_replay_cost.py | 171 ++-
.../unit/common/ai/durable/test_replay_usage.py | 114 ++
.../tests/unit/common/ai/durable/test_storage.py | 9 +
.../common/ai/durable/test_task_state_store.py | 22 +-
.../tests/unit/common/ai/operators/test_agent.py | 1381 +++++++++++++++++++-
.../ai/operators/test_agent_tool_approval.py | 45 +-
.../unit/common/ai/tools/test__from_toolset.py | 229 +++-
.../ai/tests/unit/common/ai/tools/test_adk.py | 157 +++
.../ai/tests/unit/common/ai/tools/test_strands.py | 170 ++-
.../ai/tests/unit/common/ai/tools/test_tools.py | 94 +-
.../ai/tests/unit/common/ai/tools/test_tracing.py | 230 ++++
.../unit/common/ai/toolsets/test_datafusion.py | 16 +-
.../ai/tests/unit/common/ai/toolsets/test_hook.py | 167 ++-
.../common/ai/toolsets/test_langchain_bridge.py | 58 +-
.../ai/tests/unit/common/ai/toolsets/test_mcp.py | 8 +
.../unit/common/ai/toolsets/test_object_storage.py | 416 ++++++
.../tests/unit/common/ai/toolsets/test_sandbox.py | 164 ++-
.../ai/tests/unit/common/ai/toolsets/test_sql.py | 54 +
.../unit/common/ai/utils/test_file_analysis.py | 27 +-
.../ai/tests/unit/common/ai/utils/test_logging.py | 77 ++
.../ai/tests/unit/common/ai/utils/test_masking.py | 117 ++
.../unit/common/ai/utils/test_query_results.py | 9 +
.../unit/common/ai/utils/test_tool_definition.py | 21 +-
.../unit/common/ai/utils/test_tool_metrics.py | 179 +++
.../unit/common/ai/utils/test_toolset_base.py | 281 ++++
.../unit/common/ai/utils/test_usage_budget.py | 242 ++++
providers/duckdb/docs/operators.rst | 6 +
.../src/airflow/providers/duckdb/hooks/duckdb.py | 6 +-
.../airflow/providers/duckdb/operators/duckdb.py | 48 +-
.../tests/unit/duckdb/operators/test_duckdb.py | 33 +-
.../fab/src/airflow/providers/fab/www/views.py | 6 +-
providers/fab/tests/unit/fab/www/test_views.py | 21 +-
providers/openai/docs/changelog.rst | 19 +
.../src/airflow/providers/openai/exceptions.py | 11 +
.../src/airflow/providers/openai/hooks/openai.py | 50 +-
.../airflow/providers/openai/operators/openai.py | 63 +-
.../airflow/providers/openai/triggers/openai.py | 19 +-
.../openai/tests/unit/openai/hooks/test_openai.py | 36 +
.../tests/unit/openai/operators/test_openai.py | 114 +-
.../openai/tests/unit/openai/test_exceptions.py | 15 +-
.../tests/unit/openai/triggers/test_openai.py | 47 +-
scripts/ci/docker-compose/backend-mysql.yml | 6 +
scripts/ci/docker-compose/backend-postgres.yml | 10 +-
scripts/ci/docker-compose/base.yml | 9 +
.../ci/docker-compose/integration-cassandra.yml | 5 +
scripts/ci/docker-compose/integration-celery.yml | 8 +
scripts/ci/docker-compose/integration-drill.yml | 5 +
.../docker-compose/integration-elasticsearch.yml | 5 +
scripts/ci/docker-compose/integration-kafka.yml | 2 +
scripts/ci/docker-compose/integration-kerberos.yml | 7 +
scripts/ci/docker-compose/integration-keycloak.yml | 2 +
.../ci/docker-compose/integration-localstack.yml | 2 +
scripts/ci/docker-compose/integration-mongo.yml | 5 +
scripts/ci/docker-compose/integration-mssql.yml | 5 +
.../ci/docker-compose/integration-openlineage.yml | 5 +
.../ci/docker-compose/integration-opensearch.yml | 5 +
scripts/ci/docker-compose/integration-otel.yml | 11 +
scripts/ci/docker-compose/integration-pinot.yml | 2 +
scripts/ci/docker-compose/integration-qdrant.yml | 2 +
scripts/ci/docker-compose/integration-redis.yml | 7 +
scripts/ci/docker-compose/integration-statsd.yml | 8 +
.../ci/docker-compose/integration-tinkerpop.yml | 5 +
scripts/ci/docker-compose/integration-trino.yml | 5 +
scripts/ci/docker-compose/integration-ydb.yml | 2 +
scripts/ci/docker-compose/local-all-sources.yml | 6 +
scripts/ci/docker-compose/local.yml | 6 +
.../in_container/run_provider_yaml_files_check.py | 2 +
.../configuration/secrets_backends.py | 88 --
.../observability/metrics/metrics_template.yaml | 13 +-
task-sdk/src/airflow/sdk/bases/operator.py | 69 +-
task-sdk/src/airflow/sdk/configuration.py | 10 +-
.../airflow/sdk/coordinators/_bundle_metadata.py | 9 +-
.../sdk/coordinators/node/_bundle_reader.py | 210 ++-
.../airflow/sdk/execution_time/secrets/__init__.py | 9 +-
.../src/airflow/sdk/execution_time/supervisor.py | 3 +-
.../src/airflow/sdk/execution_time/task_runner.py | 8 +-
task-sdk/src/airflow/sdk/importers/base.py | 9 +-
.../src/airflow/sdk/importers/python_importer.py | 2 +-
task-sdk/src/airflow/sdk/importers/zip_importer.py | 2 +-
task-sdk/tests/task_sdk/bases/test_operator.py | 171 +++
.../coordinators/node/_bundle_test_utils.py | 55 +-
.../coordinators/node/test_bundle_reader.py | 125 +-
.../tests/task_sdk/execution_time/test_context.py | 13 +-
.../tests/task_sdk/execution_time/test_secrets.py | 3 +-
.../task_sdk/execution_time/test_task_runner.py | 24 +
ts-sdk/adr/0002-native-dag-interface.md | 58 +-
ts-sdk/src/cli/bundle-encoder.ts | 181 ++-
ts-sdk/src/cli/pack.ts | 144 +-
ts-sdk/src/coordinator/manifest.ts | 24 +-
ts-sdk/src/index.ts | 4 +-
ts-sdk/src/sdk/bundle.ts | 7 +
ts-sdk/src/sdk/dag.ts | 683 ++++++++--
ts-sdk/src/sdk/module-source.ts | 74 ++
ts-sdk/tests/cli/fixtures/bundle-v1.min.mjs | 6 +-
ts-sdk/tests/cli/pack.test.ts | 391 +++++-
ts-sdk/tests/coordinator/runtime-manifest.test.ts | 26 +
ts-sdk/tests/public-api.test.ts | 116 +-
ts-sdk/tests/sdk/dag.test.ts | 723 ++++++++--
uv.lock | 8 +-
412 files changed, 21039 insertions(+), 6292 deletions(-)
delete mode 100644 .apache-magpie-overrides/committer-onboarding-config.md
delete mode 100644 .apache-magpie-overrides/committer-readiness.md
delete mode 100644 .apache-magpie-overrides/contributor-nomination-config.md
delete mode 100644 .apache-magpie-overrides/contributor-sentiment-config.md
delete mode 100644 .apache-magpie-overrides/good-first-issue-config.md
delete mode 100644 .apache-magpie-overrides/onboarding-concierge-config.md
create mode 100644 airflow-core/adr/lang-sdk/0010-native-dag-processing.md
create mode 100644
airflow-core/adr/lang-sdk/0011-mixed-language-dag-processing.md
create mode 100644 airflow-core/adr/lang-sdk/0012-lang-sdk-parse-protocol.md
create mode 100644
airflow-core/adr/lang-sdk/0013-persisted-task-handler-bindings.md
create mode 100644
airflow-core/adr/lang-sdk/0014-bundle-metadata-and-cache-digest.md
create mode 100644
airflow-core/adr/lang-sdk/0015-per-dag-source-in-bundle-artifact.md
delete mode 100644 airflow-core/newsfragments/70681.significant.rst
create mode 100644 airflow-core/newsfragments/72100.significant.rst
create mode 100644
airflow-core/src/airflow/api_fastapi/common/parameters/job.py
create mode 100644
airflow-core/src/airflow/ui/src/components/Clear/TaskInstance/ClearTaskInstanceDialog.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/pages/TaskInstance/Details.test.tsx
copy airflow-core/src/airflow/ui/src/{components/StateBadge.tsx =>
pages/TaskInstance/stateReason.ts} (54%)
delete mode 100644
airflow-core/src/airflow/ui/src/pages/Variables/BackendsOrderCard.tsx
delete mode 100644
airflow-core/src/airflow/ui/src/pages/Variables/BackendsOrderModal.tsx
delete mode 100644
airflow-core/tests/unit/migrations/test_0049_remove_pickled_data_from_xcom_table.py
delete mode 100644
airflow-core/tests/unit/migrations/test_0055_remove_pickled_data_from_dagrun_table.py
delete mode 100644
airflow-core/tests/unit/migrations/test_0080_deadline_callback_migration.py
delete mode 100644
airflow-core/tests/unit/migrations/test_0094_deadline_callback_migration.py
delete mode 100644
airflow-core/tests/unit/migrations/test_0117_deadline_interval_json_migration.py
delete mode 100644
airflow-core/tests/unit/migrations/test_0129_add_indexes_on_serialized_dag_and_dag_code.py
delete mode 100644
airflow-core/tests/unit/migrations/test_0131_lower_case_team_names.py
delete mode 100644
airflow-core/tests/unit/migrations/test_0136_fold_task_map_into_xcom.py
delete mode 100644
airflow-core/tests/unit/migrations/test_0138_allocate_pending_task_attempt_numbers.py
copy {ts-sdk => java-sdk/sdk}/schema/dag-schema.json (100%)
delete mode 100644
java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Builder.kt
create mode 100644
java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/internal/Fields.kt
create mode 100644 providers/amazon/docs/operators/duckdb.rst
create mode 100644
providers/amazon/src/airflow/providers/amazon/aws/hooks/duckdb.py
copy providers/amazon/tests/system/amazon/aws/{example_gcs_to_s3.py =>
example_duckdb.py} (52%)
create mode 100644
providers/amazon/tests/system/amazon/aws/example_emr_serverless_session.py
create mode 100644 providers/amazon/tests/unit/amazon/aws/hooks/test_duckdb.py
create mode 100644
providers/amazon/tests/unit/amazon/aws/operators/test_emr_serverless_session.py
create mode 100644 providers/common/ai/docs/frameworks/adk.rst
create mode 100644 providers/common/ai/docs/frameworks/pydantic_ai.rst
create mode 100644 providers/common/ai/docs/stability.rst
create mode 100644 providers/common/ai/docs/toolsets/object_storage.rst
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/durable/replay_usage.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/example_dags/example_adk_agent.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/example_dags/example_object_storage_toolset.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/example_dags/example_pydantic_ai_agent.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/tools/adk.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/tools/tracing.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/toolsets/object_storage.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/utils/masking.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/utils/tool_metrics.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/utils/toolset_base.py
create mode 100644
providers/common/ai/src/airflow/providers/common/ai/utils/usage_budget.py
create mode 100644
providers/common/ai/tests/unit/common/ai/durable/test_replay_usage.py
create mode 100644 providers/common/ai/tests/unit/common/ai/tools/test_adk.py
create mode 100644
providers/common/ai/tests/unit/common/ai/tools/test_tracing.py
create mode 100644
providers/common/ai/tests/unit/common/ai/toolsets/test_object_storage.py
create mode 100644
providers/common/ai/tests/unit/common/ai/utils/test_masking.py
create mode 100644
providers/common/ai/tests/unit/common/ai/utils/test_tool_metrics.py
create mode 100644
providers/common/ai/tests/unit/common/ai/utils/test_toolset_base.py
create mode 100644
providers/common/ai/tests/unit/common/ai/utils/test_usage_budget.py
delete mode 100644
shared/configuration/src/airflow_shared/configuration/secrets_backends.py
create mode 100644 ts-sdk/src/sdk/module-source.ts
