This is an automated email from the ASF dual-hosted git repository.
jason810496 pushed a change to branch jason/lang-sdk-e2e/08-native-e2e
in repository https://gitbox.apache.org/repos/asf/airflow.git
discard 4f1231b47e3 Keep the k8s lang-SDK values comment within the YAML line
limit
discard dbc8f5ba7de Read a native Dag example's served Dags with bundleDags
discard ff95bcde4be Say which coordinator runs the native e2e Java tasks
discard 58f8fe8747f Drop the dag_bundle_name reason from the k8s native
TypeScript skip
discard 3a4501e519b Parse native Dags in the e2e by class, not by Dag bundle
name
discard 94e5d4c39eb Name the skipped-branch e2e test after what it checks
discard 735c06c5458 Document how the TypeScript example's native Dag gets
parsed
discard ad134583392 Explain why the k8s native TypeScript Dag test is skipped
discard 15b00159986 Pack the TypeScript example against the built SDK
discard d82ce8a28c1 Build the TypeScript SDK before packing the k8s example
bundle
discard ddceeec7f7e Key the Lang-SDK e2e jobs on native Dag modules only
discard d516d0fdf2e Build the k8s TypeScript artifacts only for the native Dag
test
discard de0497ce16d Share the Dag run lookup and the native Java queue name
discard 0cbc0c9235f Run the Lang-SDK e2e jobs when native Dag parsing changes
discard 43b3ecf645d Add a native Java Dag to the compose e2e
discard af394aa9052 Keep the containerized Java build's Maven repository on
the host
discard eeb92156c87 Check the native TypeScript Dag's source and triggered run
discard 57867972645 Parse the native TypeScript Dag in the compose e2e
discard 3fe4a8c17d4 TS SDK: cover native TypeScript Dags end to end
discard 37c2458be2d Java SDK: Pack each native Dag's source file into the
bundle JAR
discard 58199853a99 Java SDK: Record the class that declares each Dag and
describe it on request
discard 7f866a13dae Java SDK: Serialize native Dags to DagSerialization v3 on
parse requests
discard 2aff846b023 Java SDK: Declare a Dag's task graph with @Builder.Deps
discard 97f08114f3f TS SDK: add triggerDagRun to the native Dag surface
discard e20d7c86106 TS SDK: add multi-way branching with dag.switch
discard ef589dcc1ed TS SDK: add conditional branching with dag.if
discard 88dab60256c Say tasks list runs a native TypeScript Dag's bundle
discard 0ab4ff562f9 Document how a NodeCoordinator is picked for parsing
TypeScript Dags
discard 71f42fbe143 Run a native TypeScript Dag's own bundle through the
native task branch
discard 44f35fd0c5e Let the Node Dag importer find its coordinator
discard a62c1446d3a Hand out the Node Dag importer through get_dag_importer
discard aaab3f2eec0 Pin the Code view to the entry module in Node bundle tests
discard 99edac01510 Let NodeDagImporter source read errors reach the caller
discard 8f352f9d393 Import NodeDagImporter at the top of the Node coordinator
discard d5d83105540 Warn against defining a Dag in both TypeScript and Python
discard ddbd699b035 Align the TypeScript build docs with the Code view
discard e3f2d736b07 Say when the Dag processor needs Node.js for TypeScript
bundles
discard 2a2724da4ae Run a native TypeScript Dag's task from its own bundle
discard ceac0c6f533 Share the bundle section mutation helper between Node tests
discard b8eb39f2f8d Document parsing native TypeScript Dags
discard 8c291e7a8fb Parse native TypeScript Dags with the Node coordinator
discard b46222bc6a4 Say a JRE parses native Java Dags, and keep main's bundle
test mocks
discard 4af75411281 Say tasks list runs a native Java Dag's main class
discard d424ec2c0b4 Fail a native task cleanly when its file or coordinator
class cannot load
discard 788fc10bc7d Run a native Dag's task from its own Dag file
discard b8f07eaa458 Fail a task cleanly when its runtime cannot be launched
discard 0bbdaff10d4 Let the Java Dag importer find its coordinator
discard 3f87309846c Read Java Dag sources from the bundle's sources.json index
discard fbf740522da Hand out the Java Dag importer through get_dag_importer
discard 54621f2ebb4 Keep the Java parse_dag test from changing the pytest
worker
discard 8cc382c6471 Stop promising that Java dependency JARs are not parsed
discard 4feb87177fa Reject a Java Dag JAR whose SDK cannot answer a parse
discard e357fd46a2b Run the JAR a native Java Dag was parsed from
discard e0e40664946 Report why a Java Dag JAR cannot be read
discard ad271c77be4 Pass the task's Dag file to the coordinator's command
builder
discard cf732698db5 Run a task on its run's version when dag_bundle_name names
its bundle
discard d1504d4396c Document native Java Dag parsing
discard 8f665071d5a Name the JAR manifest attributes the Java coordinator reads
discard defa8f9a7f5 Test the Java Dag source placeholder for a JAR without a
manifest
discard d79c05e4e05 Treat an unreadable JAR like one without a manifest
discard 77512255fb9 Parse native Java Dags with their coordinator
discard be449fcc58d Build the JVM command that parses a Dag JAR
discard 25c23d67781 Read JAR manifests per the JAR specification
discard 471491d2fe0 Default a Dag file processor's parse log to os.devnull
discard a806e5787ae Let a Dag file processor run without a parse log file
discard afbf901e917 Ignore the bagged Dag union in the external task sensor
test
discard 5888c5be40a List the tasks of a native Lang-SDK Dag
discard 6a1a42def5f Bag native Lang-SDK Dags in core, only when asked
discard c3669c2cf3a Let a Dag file processor run without an API client
discard 9b7542d5956 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
discard 79bef76bf7a Apply the team rules to the Dag that validation returns
discard d050e698ccc Test the import errors of a bundle with several
coordinators of a class
discard bf9c91a31c4 Find a claiming Dag importer instead of its coordinator in
Dag processing
discard d8e8e35f4fe Name the coordinator class a native Dag task must be
routed to
discard 7d894953d2e Stub Dag file discovery with DagFileInfo in the Lang-SDK
manager tests
discard 3400a3027d8 Pass a parse log handle to the Lang-SDK processor in tests
discard 57d4c6111e1 Find a native Dag's coordinator key through its importer
discard 62fff240b11 Wait for the killed leftover in the Lang-SDK processor test
discard e9677cbf7b5 Move the client-less Lang-SDK parse out of this change
discard 31c2b62bbbb Note in ADR-0010 that the manager routes claimed files for
now
discard 7e8364ff07b Clear import errors after the native-file Dag bag sync test
discard 4a4c5d426b8 Apply the multi-team task checks to Lang-SDK Dags
discard a60ad59da7f Spec the mocks of a hand-built Lang-SDK processor in tests
discard b3a1b4b19bc Name the setting in a Lang-SDK parse timeout error
discard 1cbc4dfb9a7 Kill what a Lang-SDK runtime leaves in its process group
discard 6102868b7da Match the schema error's JSON path in a Lang-SDK parse test
discard 6735f2a433d Fail a native Dag's task on a Python worker without retries
discard 1611330ed3e Find the claiming coordinator in the Task SDK and wrap it
in core
discard 29ae6e2fa6e Start each Dag file processor with explicit arguments
discard 96c32267afc Drop callbacks for Lang-SDK Dag files before queuing the
file
discard db47446a6ed Report a native Dag's task that reaches a Python worker
discard a658e511c49 Report a Lang-SDK parse past its import timeout as an
import error
discard ec74107656a Parse coordinator-claimed Dag files with their runtime in
the Dag processor
omit 6fd170439a4 Describe how a Dag importer finds its coordinator in
ADR-0010
omit b9bdf5419aa Let a Dag importer find the coordinator that parses its
files
add a44fd6d1f31 Allow explicit runs to drain a paused Dag (#73763)
add 40832901c59 Improve LocalExecutor bookkeeping: correctly add tasks to
the `running` list (#74142)
add 5487c080bea Add Azure Blob Storage support to the DataFusion object
storage layer (#73374)
add b191cc4767b Fail instead of silently falling behind when a Lang-SDK
vendored Dag schema drifts from airflow-core (#74091)
add e0202f40c78 Let a Dag importer find the coordinator that parses its
files (#74042)
add f1070b3884e Parse coordinator-claimed Dag files with their runtime in
the Dag processor
add c8a3306646d Report a Lang-SDK parse past its import timeout as an
import error
add bbea6488ded Report a native Dag's task that reaches a Python worker
add 18de8bfa32d Drop callbacks for Lang-SDK Dag files before queuing the
file
add a6359232b06 Start each Dag file processor with explicit arguments
add 17f65afef02 Find the claiming coordinator in the Task SDK and wrap it
in core
add 2d6dd214ff5 Fail a native Dag's task on a Python worker without retries
add 1169dbb91fa Match the schema error's JSON path in a Lang-SDK parse test
add 303b0be7821 Kill what a Lang-SDK runtime leaves in its process group
add 19810c730b3 Name the setting in a Lang-SDK parse timeout error
add b5ed50699b4 Spec the mocks of a hand-built Lang-SDK processor in tests
add 58c66651378 Apply the multi-team task checks to Lang-SDK Dags
add 913e53620c5 Clear import errors after the native-file Dag bag sync test
add 22cd5aac3cd Note in ADR-0010 that the manager routes claimed files for
now
add e434a396ff7 Move the client-less Lang-SDK parse out of this change
add 0e680a3b2e2 Wait for the killed leftover in the Lang-SDK processor test
add eb258495592 Find a native Dag's coordinator key through its importer
add 4ad06144a0e Pass a parse log handle to the Lang-SDK processor in tests
add 435e639a5e9 Stub Dag file discovery with DagFileInfo in the Lang-SDK
manager tests
add e62a4815b8f Name the coordinator class a native Dag task must be
routed to
add 18744e8f554 Find a claiming Dag importer instead of its coordinator in
Dag processing
add 302b4e0c7b5 Test the import errors of a bundle with several
coordinators of a class
add 9007b91ed18 Apply the team rules to the Dag that validation returns
add d747c98f701 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
add 2855834d2bf Let a Dag file processor run without an API client
add 16099c61f3c Bag native Lang-SDK Dags in core, only when asked
add e77020e65bd List the tasks of a native Lang-SDK Dag
add ff9e8d23f10 Ignore the bagged Dag union in the external task sensor
test
add 29d2fa7090a Let a Dag file processor run without a parse log file
add 77a24d4ee6d Default a Dag file processor's parse log to os.devnull
add 2ff465d6b09 Read JAR manifests per the JAR specification
add 22208dae655 Build the JVM command that parses a Dag JAR
add 9ea7124c7d8 Parse native Java Dags with their coordinator
add 9ad69e847b3 Treat an unreadable JAR like one without a manifest
add c600b788fd9 Test the Java Dag source placeholder for a JAR without a
manifest
add edb8a583f79 Name the JAR manifest attributes the Java coordinator reads
add b2674bc5886 Document native Java Dag parsing
add 9a5c3eca2bc Run a task on its run's version when dag_bundle_name names
its bundle
add 7245c001b1e Pass the task's Dag file to the coordinator's command
builder
add 6f31a7c1282 Report why a Java Dag JAR cannot be read
add 2abb68edad9 Run the JAR a native Java Dag was parsed from
add ab89ee7f01b Reject a Java Dag JAR whose SDK cannot answer a parse
add 8fe831d0c14 Stop promising that Java dependency JARs are not parsed
add bb6bb6f8052 Keep the Java parse_dag test from changing the pytest
worker
add f78c5f0965d Hand out the Java Dag importer through get_dag_importer
add a27ca4c81d3 Read Java Dag sources from the bundle's sources.json index
add d896e079969 Let the Java Dag importer find its coordinator
add 512c6885e21 Fail a task cleanly when its runtime cannot be launched
add 060109090e7 Run a native Dag's task from its own Dag file
add c31f319efa4 Fail a native task cleanly when its file or coordinator
class cannot load
add 116b5ceff10 Say tasks list runs a native Java Dag's main class
add 92aa2285207 Say a JRE parses native Java Dags, and keep main's bundle
test mocks
add acb5fc0011f Parse native TypeScript Dags with the Node coordinator
add bf702a0fb6d Document parsing native TypeScript Dags
add a522c50c6b4 Share the bundle section mutation helper between Node tests
add 8d8141877ce Run a native TypeScript Dag's task from its own bundle
add dca36121099 Say when the Dag processor needs Node.js for TypeScript
bundles
add ec24bd1b93e Align the TypeScript build docs with the Code view
add b960c6be5be Warn against defining a Dag in both TypeScript and Python
add e2973a763ac Import NodeDagImporter at the top of the Node coordinator
add c58c6d6e3d6 Let NodeDagImporter source read errors reach the caller
add 0db68395bbc Pin the Code view to the entry module in Node bundle tests
add 68b1f5dd516 Hand out the Node Dag importer through get_dag_importer
add 4d445a87d9e Let the Node Dag importer find its coordinator
add fedfaafce74 Run a native TypeScript Dag's own bundle through the
native task branch
add a9518585425 Document how a NodeCoordinator is picked for parsing
TypeScript Dags
add 6ce49055da4 Say tasks list runs a native TypeScript Dag's bundle
add 489d03c9f45 TS SDK: add conditional branching with dag.if
add fee79f90c3e TS SDK: add multi-way branching with dag.switch
add 9ca9a2397e4 TS SDK: add triggerDagRun to the native Dag surface
add 085453fd50e Java SDK: Declare a Dag's task graph with @Builder.Deps
add 4f8a7d03396 Java SDK: Serialize native Dags to DagSerialization v3 on
parse requests
add 63e90456cc8 Java SDK: Record the class that declares each Dag and
describe it on request
add 981651c3f85 Java SDK: Pack each native Dag's source file into the
bundle JAR
add de24213b556 TS SDK: cover native TypeScript Dags end to end
add 4c9642280de Parse the native TypeScript Dag in the compose e2e
add 9919bbc4932 Check the native TypeScript Dag's source and triggered run
add 33df82e9409 Keep the containerized Java build's Maven repository on
the host
add f483e458e17 Add a native Java Dag to the compose e2e
add 7524b15a9a7 Run the Lang-SDK e2e jobs when native Dag parsing changes
add e3c778e7ae6 Share the Dag run lookup and the native Java queue name
add c426f2a6405 Build the k8s TypeScript artifacts only for the native Dag
test
add 6e7abe412a4 Key the Lang-SDK e2e jobs on native Dag modules only
add 6eb1c4d5ab9 Build the TypeScript SDK before packing the k8s example
bundle
add adabe49410f Pack the TypeScript example against the built SDK
add 1dfae623464 Explain why the k8s native TypeScript Dag test is skipped
add 914250c4ec6 Document how the TypeScript example's native Dag gets
parsed
add 4cdb0d5ae18 Name the skipped-branch e2e test after what it checks
add f9d3161a24e Parse native Dags in the e2e by class, not by Dag bundle
name
add 77bfd008b42 Drop the dag_bundle_name reason from the k8s native
TypeScript skip
add 6fd1bc66c05 Say which coordinator runs the native e2e Java tasks
add 7dd6d50fcd1 Read a native Dag example's served Dags with bundleDags
add 35b38c90a6c Keep the k8s lang-SDK values comment within the YAML line
limit
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4f1231b47e3)
\
N -- N -- N refs/heads/jason/lang-sdk-e2e/08-native-e2e
(35b38c90a6c)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.pre-commit-config.yaml | 42 +-
airflow-core/docs/core-concepts/backfill.rst | 7 +-
airflow-core/docs/core-concepts/dags.rst | 13 +
airflow-core/newsfragments/72407.significant.rst | 10 +-
.../api_fastapi/core_api/datamodels/backfills.py | 8 +
.../api_fastapi/core_api/datamodels/dag_run.py | 7 +
.../core_api/openapi/v2-rest-api-generated.yaml | 28 ++
.../api_fastapi/core_api/routes/public/assets.py | 10 +
.../core_api/routes/public/backfills.py | 17 +-
.../api_fastapi/core_api/routes/public/dag_run.py | 11 +
.../src/airflow/executors/local_executor.py | 75 +++-
airflow-core/src/airflow/models/backfill.py | 3 +
airflow-core/src/airflow/models/dag.py | 16 +
.../airflow/ui/openapi-gen/requests/schemas.gen.ts | 18 +
.../airflow/ui/openapi-gen/requests/types.gen.ts | 12 +
.../ui/public/i18n/locales/ar/components.json | 4 +-
.../ui/public/i18n/locales/ca/components.json | 4 +-
.../ui/public/i18n/locales/de/components.json | 4 +-
.../ui/public/i18n/locales/el/components.json | 4 +-
.../ui/public/i18n/locales/en/components.json | 15 +-
.../ui/public/i18n/locales/es/components.json | 4 +-
.../ui/public/i18n/locales/fr/components.json | 4 +-
.../ui/public/i18n/locales/he/components.json | 4 +-
.../ui/public/i18n/locales/hi/components.json | 4 +-
.../ui/public/i18n/locales/hu/components.json | 4 +-
.../ui/public/i18n/locales/it/components.json | 4 +-
.../ui/public/i18n/locales/ja/components.json | 4 +-
.../ui/public/i18n/locales/ko/components.json | 4 +-
.../ui/public/i18n/locales/nl/components.json | 4 +-
.../ui/public/i18n/locales/pl/components.json | 4 +-
.../ui/public/i18n/locales/pt/components.json | 4 +-
.../ui/public/i18n/locales/ru/components.json | 4 +-
.../ui/public/i18n/locales/th/components.json | 4 +-
.../ui/public/i18n/locales/tr/components.json | 4 +-
.../ui/public/i18n/locales/zh-CN/components.json | 4 +-
.../ui/public/i18n/locales/zh-TW/components.json | 4 +-
.../components/DagActions/RunBackfillForm.test.tsx | 96 ++++-
.../src/components/DagActions/RunBackfillForm.tsx | 35 +-
.../TriggerDag/PausedDagOptions.test.tsx | 250 +++++++++++
.../src/components/TriggerDag/PausedDagOptions.tsx | 123 ++++++
.../src/components/TriggerDag/TriggerDAGButton.tsx | 4 -
.../components/TriggerDag/TriggerDAGForm.test.tsx | 78 +++-
.../src/components/TriggerDag/TriggerDAGForm.tsx | 32 +-
.../components/TriggerDag/TriggerDAGModal.test.tsx | 238 +++++++++++
.../src/components/TriggerDag/TriggerDAGModal.tsx | 21 +-
.../airflow/ui/src/components/TriggerDag/types.ts | 3 +
.../ui/src/layouts/Details/DetailsLayout.tsx | 1 -
.../src/pages/Asset/CreateAssetEventModal.test.tsx | 161 +++++++-
.../ui/src/pages/Asset/CreateAssetEventModal.tsx | 20 +-
.../ui/src/pages/DagsList/DagCardActions.tsx | 1 -
.../src/airflow/ui/src/pages/DagsList/DagsList.tsx | 1 -
.../ui/src/queries/useCreateBackfill.test.tsx | 99 +++++
.../airflow/ui/src/queries/useCreateBackfill.ts | 39 +-
.../src/airflow/ui/src/queries/useTrigger.test.tsx | 98 +++++
.../src/airflow/ui/src/queries/useTrigger.ts | 31 +-
airflow-core/tests/unit/api_fastapi/conftest.py | 13 +
.../core_api/routes/public/test_assets.py | 48 ++-
.../core_api/routes/public/test_backfills.py | 72 +++-
.../core_api/routes/public/test_dag_run.py | 75 +++-
.../tests/unit/executors/test_local_executor.py | 455 ++++++++++++++++++++-
.../executors/test_local_executor_check_workers.py | 5 +-
airflow-core/tests/unit/jobs/test_scheduler_job.py | 9 +-
airflow-core/tests/unit/models/test_backfill.py | 133 +++++-
airflow-core/tests/unit/models/test_dag.py | 40 ++
.../src/airflowctl/api/datamodels/generated.py | 21 +
airflow-ctl/src/airflowctl/api/operations.py | 7 +-
.../tests/airflow_ctl/api/test_operations.py | 54 ++-
java-sdk/sdk/build.gradle.kts | 15 +
.../ai/docs/operators/llm_schema_compare.rst | 2 +-
providers/common/ai/docs/operators/llm_sql.rst | 2 +-
providers/common/ai/docs/toolsets/datafusion.rst | 2 +-
providers/common/sql/docs/operators.rst | 60 ++-
.../sql/src/airflow/providers/common/sql/config.py | 3 +
.../providers/common/sql/datafusion/engine.py | 185 +++++++++
.../sql/datafusion/object_storage_provider.py | 36 +-
.../common/sql/example_dags/example_analytics.py | 13 +
.../unit/common/sql/datafusion/test_engine.py | 447 ++++++++++++++++++++
.../sql/datafusion/test_object_storage_provider.py | 48 +++
.../sql/tests/unit/common/sql/test_config.py | 1 +
scripts/ci/prek/sync_go_sdk_schemas.py | 21 +-
.../src/airflow/sdk/coordinators/_subprocess.py | 2 +-
.../src/airflow/sdk/execution_time/coordinator.py | 4 +
82 files changed, 3243 insertions(+), 233 deletions(-)
create mode 100644
airflow-core/src/airflow/ui/src/components/TriggerDag/PausedDagOptions.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/components/TriggerDag/PausedDagOptions.tsx
create mode 100644
airflow-core/src/airflow/ui/src/components/TriggerDag/TriggerDAGModal.test.tsx
create mode 100644
airflow-core/src/airflow/ui/src/queries/useCreateBackfill.test.tsx
create mode 100644 airflow-core/src/airflow/ui/src/queries/useTrigger.test.tsx
