This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a change to branch jason/lang-sdk-e2e/04-java-dag-importer
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit b46222bc6a4 Say a JRE parses native Java Dags, and keep main's bundle
test mocks
omit 4af75411281 Say tasks list runs a native Java Dag's main class
omit d424ec2c0b4 Fail a native task cleanly when its file or coordinator
class cannot load
omit 788fc10bc7d Run a native Dag's task from its own Dag file
omit b8f07eaa458 Fail a task cleanly when its runtime cannot be launched
omit 0bbdaff10d4 Let the Java Dag importer find its coordinator
omit 3f87309846c Read Java Dag sources from the bundle's sources.json index
omit fbf740522da Hand out the Java Dag importer through get_dag_importer
omit 54621f2ebb4 Keep the Java parse_dag test from changing the pytest
worker
omit 8cc382c6471 Stop promising that Java dependency JARs are not parsed
omit 4feb87177fa Reject a Java Dag JAR whose SDK cannot answer a parse
omit e357fd46a2b Run the JAR a native Java Dag was parsed from
omit e0e40664946 Report why a Java Dag JAR cannot be read
omit ad271c77be4 Pass the task's Dag file to the coordinator's command
builder
omit cf732698db5 Run a task on its run's version when dag_bundle_name names
its bundle
omit d1504d4396c Document native Java Dag parsing
omit 8f665071d5a Name the JAR manifest attributes the Java coordinator reads
omit defa8f9a7f5 Test the Java Dag source placeholder for a JAR without a
manifest
omit d79c05e4e05 Treat an unreadable JAR like one without a manifest
omit 77512255fb9 Parse native Java Dags with their coordinator
omit be449fcc58d Build the JVM command that parses a Dag JAR
omit 25c23d67781 Read JAR manifests per the JAR specification
omit 471491d2fe0 Default a Dag file processor's parse log to os.devnull
omit a806e5787ae Let a Dag file processor run without a parse log file
omit afbf901e917 Ignore the bagged Dag union in the external task sensor
test
omit 5888c5be40a List the tasks of a native Lang-SDK Dag
omit 6a1a42def5f Bag native Lang-SDK Dags in core, only when asked
omit c3669c2cf3a Let a Dag file processor run without an API client
omit 9b7542d5956 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
omit 79bef76bf7a Apply the team rules to the Dag that validation returns
omit d050e698ccc Test the import errors of a bundle with several
coordinators of a class
omit bf9c91a31c4 Find a claiming Dag importer instead of its coordinator in
Dag processing
omit d8e8e35f4fe Name the coordinator class a native Dag task must be
routed to
omit 7d894953d2e Stub Dag file discovery with DagFileInfo in the Lang-SDK
manager tests
omit 3400a3027d8 Pass a parse log handle to the Lang-SDK processor in tests
omit 57d4c6111e1 Find a native Dag's coordinator key through its importer
omit 62fff240b11 Wait for the killed leftover in the Lang-SDK processor test
omit e9677cbf7b5 Move the client-less Lang-SDK parse out of this change
omit 31c2b62bbbb Note in ADR-0010 that the manager routes claimed files for
now
omit 7e8364ff07b Clear import errors after the native-file Dag bag sync test
omit 4a4c5d426b8 Apply the multi-team task checks to Lang-SDK Dags
omit a60ad59da7f Spec the mocks of a hand-built Lang-SDK processor in tests
omit b3a1b4b19bc Name the setting in a Lang-SDK parse timeout error
omit 1cbc4dfb9a7 Kill what a Lang-SDK runtime leaves in its process group
omit 6102868b7da Match the schema error's JSON path in a Lang-SDK parse test
omit 6735f2a433d Fail a native Dag's task on a Python worker without retries
omit 1611330ed3e Find the claiming coordinator in the Task SDK and wrap it
in core
omit 29ae6e2fa6e Start each Dag file processor with explicit arguments
omit 96c32267afc Drop callbacks for Lang-SDK Dag files before queuing the
file
omit db47446a6ed Report a native Dag's task that reaches a Python worker
omit a658e511c49 Report a Lang-SDK parse past its import timeout as an
import error
omit ec74107656a Parse coordinator-claimed Dag files with their runtime in
the Dag processor
omit 6fd170439a4 Describe how a Dag importer finds its coordinator in
ADR-0010
omit b9bdf5419aa Let a Dag importer find the coordinator that parses its
files
add a44fd6d1f31 Allow explicit runs to drain a paused Dag (#73763)
add 40832901c59 Improve LocalExecutor bookkeeping: correctly add tasks to
the `running` list (#74142)
add 5487c080bea Add Azure Blob Storage support to the DataFusion object
storage layer (#73374)
add b191cc4767b Fail instead of silently falling behind when a Lang-SDK
vendored Dag schema drifts from airflow-core (#74091)
add e0202f40c78 Let a Dag importer find the coordinator that parses its
files (#74042)
add f1070b3884e Parse coordinator-claimed Dag files with their runtime in
the Dag processor
add c8a3306646d Report a Lang-SDK parse past its import timeout as an
import error
add bbea6488ded Report a native Dag's task that reaches a Python worker
add 18de8bfa32d Drop callbacks for Lang-SDK Dag files before queuing the
file
add a6359232b06 Start each Dag file processor with explicit arguments
add 17f65afef02 Find the claiming coordinator in the Task SDK and wrap it
in core
add 2d6dd214ff5 Fail a native Dag's task on a Python worker without retries
add 1169dbb91fa Match the schema error's JSON path in a Lang-SDK parse test
add 303b0be7821 Kill what a Lang-SDK runtime leaves in its process group
add 19810c730b3 Name the setting in a Lang-SDK parse timeout error
add b5ed50699b4 Spec the mocks of a hand-built Lang-SDK processor in tests
add 58c66651378 Apply the multi-team task checks to Lang-SDK Dags
add 913e53620c5 Clear import errors after the native-file Dag bag sync test
add 22cd5aac3cd Note in ADR-0010 that the manager routes claimed files for
now
add e434a396ff7 Move the client-less Lang-SDK parse out of this change
add 0e680a3b2e2 Wait for the killed leftover in the Lang-SDK processor test
add eb258495592 Find a native Dag's coordinator key through its importer
add 4ad06144a0e Pass a parse log handle to the Lang-SDK processor in tests
add 435e639a5e9 Stub Dag file discovery with DagFileInfo in the Lang-SDK
manager tests
add e62a4815b8f Name the coordinator class a native Dag task must be
routed to
add 18744e8f554 Find a claiming Dag importer instead of its coordinator in
Dag processing
add 302b4e0c7b5 Test the import errors of a bundle with several
coordinators of a class
add 9007b91ed18 Apply the team rules to the Dag that validation returns
add d747c98f701 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
add 2855834d2bf Let a Dag file processor run without an API client
add 16099c61f3c Bag native Lang-SDK Dags in core, only when asked
add e77020e65bd List the tasks of a native Lang-SDK Dag
add ff9e8d23f10 Ignore the bagged Dag union in the external task sensor
test
add 29d2fa7090a Let a Dag file processor run without a parse log file
add 77a24d4ee6d Default a Dag file processor's parse log to os.devnull
add 2ff465d6b09 Read JAR manifests per the JAR specification
add 22208dae655 Build the JVM command that parses a Dag JAR
add 9ea7124c7d8 Parse native Java Dags with their coordinator
add 9ad69e847b3 Treat an unreadable JAR like one without a manifest
add c600b788fd9 Test the Java Dag source placeholder for a JAR without a
manifest
add edb8a583f79 Name the JAR manifest attributes the Java coordinator reads
add b2674bc5886 Document native Java Dag parsing
add 9a5c3eca2bc Run a task on its run's version when dag_bundle_name names
its bundle
add 7245c001b1e Pass the task's Dag file to the coordinator's command
builder
add 6f31a7c1282 Report why a Java Dag JAR cannot be read
add 2abb68edad9 Run the JAR a native Java Dag was parsed from
add ab89ee7f01b Reject a Java Dag JAR whose SDK cannot answer a parse
add 8fe831d0c14 Stop promising that Java dependency JARs are not parsed
add bb6bb6f8052 Keep the Java parse_dag test from changing the pytest
worker
add f78c5f0965d Hand out the Java Dag importer through get_dag_importer
add a27ca4c81d3 Read Java Dag sources from the bundle's sources.json index
add d896e079969 Let the Java Dag importer find its coordinator
add 512c6885e21 Fail a task cleanly when its runtime cannot be launched
add 060109090e7 Run a native Dag's task from its own Dag file
add c31f319efa4 Fail a native task cleanly when its file or coordinator
class cannot load
add 116b5ceff10 Say tasks list runs a native Java Dag's main class
add 92aa2285207 Say a JRE parses native Java Dags, and keep main's bundle
test mocks
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (b46222bc6a4)
\
N -- N -- N refs/heads/jason/lang-sdk-e2e/04-java-dag-importer
(92aa2285207)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.pre-commit-config.yaml | 42 +-
airflow-core/docs/core-concepts/backfill.rst | 7 +-
airflow-core/docs/core-concepts/dags.rst | 13 +
airflow-core/newsfragments/72407.significant.rst | 10 +-
.../api_fastapi/core_api/datamodels/backfills.py | 8 +
.../api_fastapi/core_api/datamodels/dag_run.py | 7 +
.../core_api/openapi/v2-rest-api-generated.yaml | 28 ++
.../api_fastapi/core_api/routes/public/assets.py | 10 +
.../core_api/routes/public/backfills.py | 17 +-
.../api_fastapi/core_api/routes/public/dag_run.py | 11 +
.../src/airflow/executors/local_executor.py | 75 +++-
airflow-core/src/airflow/models/backfill.py | 3 +
airflow-core/src/airflow/models/dag.py | 16 +
.../airflow/ui/openapi-gen/requests/schemas.gen.ts | 18 +
.../airflow/ui/openapi-gen/requests/types.gen.ts | 12 +
.../ui/public/i18n/locales/ar/components.json | 4 +-
.../ui/public/i18n/locales/ca/components.json | 4 +-
.../ui/public/i18n/locales/de/components.json | 4 +-
.../ui/public/i18n/locales/el/components.json | 4 +-
.../ui/public/i18n/locales/en/components.json | 15 +-
.../ui/public/i18n/locales/es/components.json | 4 +-
.../ui/public/i18n/locales/fr/components.json | 4 +-
.../ui/public/i18n/locales/he/components.json | 4 +-
.../ui/public/i18n/locales/hi/components.json | 4 +-
.../ui/public/i18n/locales/hu/components.json | 4 +-
.../ui/public/i18n/locales/it/components.json | 4 +-
.../ui/public/i18n/locales/ja/components.json | 4 +-
.../ui/public/i18n/locales/ko/components.json | 4 +-
.../ui/public/i18n/locales/nl/components.json | 4 +-
.../ui/public/i18n/locales/pl/components.json | 4 +-
.../ui/public/i18n/locales/pt/components.json | 4 +-
.../ui/public/i18n/locales/ru/components.json | 4 +-
.../ui/public/i18n/locales/th/components.json | 4 +-
.../ui/public/i18n/locales/tr/components.json | 4 +-
.../ui/public/i18n/locales/zh-CN/components.json | 4 +-
.../ui/public/i18n/locales/zh-TW/components.json | 4 +-
.../components/DagActions/RunBackfillForm.test.tsx | 96 ++++-
.../src/components/DagActions/RunBackfillForm.tsx | 35 +-
.../TriggerDag/PausedDagOptions.test.tsx | 250 +++++++++++
.../src/components/TriggerDag/PausedDagOptions.tsx | 123 ++++++
.../src/components/TriggerDag/TriggerDAGButton.tsx | 4 -
.../components/TriggerDag/TriggerDAGForm.test.tsx | 78 +++-
.../src/components/TriggerDag/TriggerDAGForm.tsx | 32 +-
.../components/TriggerDag/TriggerDAGModal.test.tsx | 238 +++++++++++
.../src/components/TriggerDag/TriggerDAGModal.tsx | 21 +-
.../airflow/ui/src/components/TriggerDag/types.ts | 3 +
.../ui/src/layouts/Details/DetailsLayout.tsx | 1 -
.../src/pages/Asset/CreateAssetEventModal.test.tsx | 161 +++++++-
.../ui/src/pages/Asset/CreateAssetEventModal.tsx | 20 +-
.../ui/src/pages/DagsList/DagCardActions.tsx | 1 -
.../src/airflow/ui/src/pages/DagsList/DagsList.tsx | 1 -
.../ui/src/queries/useCreateBackfill.test.tsx | 99 +++++
.../airflow/ui/src/queries/useCreateBackfill.ts | 39 +-
.../src/airflow/ui/src/queries/useTrigger.test.tsx | 98 +++++
.../src/airflow/ui/src/queries/useTrigger.ts | 31 +-
airflow-core/tests/unit/api_fastapi/conftest.py | 13 +
.../core_api/routes/public/test_assets.py | 48 ++-
.../core_api/routes/public/test_backfills.py | 72 +++-
.../core_api/routes/public/test_dag_run.py | 75 +++-
.../tests/unit/executors/test_local_executor.py | 455 ++++++++++++++++++++-
.../executors/test_local_executor_check_workers.py | 5 +-
airflow-core/tests/unit/jobs/test_scheduler_job.py | 9 +-
airflow-core/tests/unit/models/test_backfill.py | 133 +++++-
airflow-core/tests/unit/models/test_dag.py | 40 ++
.../src/airflowctl/api/datamodels/generated.py | 21 +
airflow-ctl/src/airflowctl/api/operations.py | 7 +-
.../tests/airflow_ctl/api/test_operations.py | 54 ++-
java-sdk/sdk/build.gradle.kts | 15 +
.../ai/docs/operators/llm_schema_compare.rst | 2 +-
providers/common/ai/docs/operators/llm_sql.rst | 2 +-
providers/common/ai/docs/toolsets/datafusion.rst | 2 +-
providers/common/sql/docs/operators.rst | 60 ++-
.../sql/src/airflow/providers/common/sql/config.py | 3 +
.../providers/common/sql/datafusion/engine.py | 185 +++++++++
.../sql/datafusion/object_storage_provider.py | 36 +-
.../common/sql/example_dags/example_analytics.py | 13 +
.../unit/common/sql/datafusion/test_engine.py | 447 ++++++++++++++++++++
.../sql/datafusion/test_object_storage_provider.py | 48 +++
.../sql/tests/unit/common/sql/test_config.py | 1 +
scripts/ci/prek/sync_go_sdk_schemas.py | 21 +-
.../src/airflow/sdk/coordinators/_subprocess.py | 2 +-
.../src/airflow/sdk/execution_time/coordinator.py | 4 +
82 files changed, 3243 insertions(+), 233 deletions(-)
create mode 100644
airflow-core/src/airflow/ui/src/components/TriggerDag/PausedDagOptions.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/components/TriggerDag/PausedDagOptions.tsx
create mode 100644
airflow-core/src/airflow/ui/src/components/TriggerDag/TriggerDAGModal.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/queries/useCreateBackfill.test.tsx
create mode 100644 airflow-core/src/airflow/ui/src/queries/useTrigger.test.tsx
