This is an automated email from the ASF dual-hosted git repository.
pierrejeambrun pushed a change to branch jason/lang-sdk-e2e/02-importer-registry
in repository https://gitbox.apache.org/repos/asf/airflow.git
omit 4ff4171bd7f Stub Dag file discovery with DagFileInfo in the Lang-SDK
manager tests
omit 17776f430b6 Handle Dags nested at any depth in Dag archives
omit 24eb53e370e Migrating from filepath based Dag to DagDefinition
discovery
omit d9cc6b1791a Pin the Code view to the entry module in Node bundle tests
omit 73a18f9cd30 Let NodeDagImporter source read errors reach the caller
omit b2f7c17adca Import NodeDagImporter at the top of the Node coordinator
omit 116c8a2dd82 Warn against defining a Dag in both TypeScript and Python
omit cc949b3316c Align the TypeScript build docs with the Code view
omit e2076b8e900 Say when the Dag processor needs Node.js for TypeScript
bundles
omit 7702d8420b9 Run a native TypeScript Dag's task from its own bundle
omit 01316f8d0a7 Share the bundle section mutation helper between Node tests
omit d98a2540a4b Document parsing native TypeScript Dags
omit 064db1cce39 Parse native TypeScript Dags with the Node coordinator
omit b045547d080 Keep the Java parse_dag test from changing the pytest
worker
omit 1a6e3ea86e1 Stop promising that Java dependency JARs are not parsed
omit f3f6bea154b Reject a Java Dag JAR whose SDK cannot answer a parse
omit ee9521355b1 Run the JAR a native Java Dag was parsed from
omit 1aac2d02868 Report why a Java Dag JAR cannot be read
omit ac38e820b06 Pass the task's Dag file to the coordinator's command
builder
omit c179527fedc Run a task on its run's version when dag_bundle_name names
its bundle
omit 97f60a3322e Document native Java Dag parsing
omit bac857dec40 Name the JAR manifest attributes the Java coordinator reads
omit 2ca56a5effe Test the Java Dag source placeholder for a JAR without a
manifest
omit 0b55a3dd2a6 Treat an unreadable JAR like one without a manifest
omit 2eb08b4308c Parse native Java Dags with their coordinator
omit a7be34b19ff Embed the Dag source in Java bundle JARs
omit 02bc2bc0794 Build the JVM command that parses a Dag JAR
omit bd060ceb11b Read JAR manifests per the JAR specification
omit 96f2076b96d Ignore the bagged Dag union in the external task sensor
test
omit fd0a742a2d7 List the tasks of a native Lang-SDK Dag
omit a8797750305 Bag native Lang-SDK Dags in core, only when asked
omit cab27a25c52 Let a Dag file processor run without an API client
omit c53c8c723e5 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
omit e072db64742 Wait for the killed leftover in the Lang-SDK processor test
omit 57a417b164d Move the client-less Lang-SDK parse out of this change
omit 2ee5ead4c9a Note in ADR-0010 that the manager routes claimed files for
now
omit 04706bdc0b5 Clear import errors after the native-file Dag bag sync test
omit 55311fe73c1 Apply the multi-team task checks to Lang-SDK Dags
omit 8b91affcf79 Spec the mocks of a hand-built Lang-SDK processor in tests
omit 56c6820d7c3 Name the setting in a Lang-SDK parse timeout error
omit f632602f806 Kill what a Lang-SDK runtime leaves in its process group
omit 280fb07887a Match the schema error's JSON path in a Lang-SDK parse test
omit 3c0b4ba6181 Build each bundle's Dag importer registry before freezing
the heap
omit 001b6d90f29 Fail a native Dag's task on a Python worker without retries
omit fd2a897053d Find the claiming coordinator in the Task SDK and wrap it
in core
omit 4f54283d0de Return a Dag importer from every SubprocessCoordinator call
omit ed8cbad9fa0 Start each Dag file processor with explicit arguments
omit 0cb6bf43a66 Drop callbacks for Lang-SDK Dag files before queuing the
file
omit 7c38262692d Report a native Dag's task that reaches a Python worker
omit 50981761a38 Report a Lang-SDK parse past its import timeout as an
import error
omit 9ef4d490013 Parse coordinator-claimed Dag files with their runtime in
the Dag processor
omit 30187220d54 Say dag_bundle_name is only needed for two coordinators of
one class
omit 52b88ba0c85 Drop serves_bundle so get_parsed_bundles is the only answer
omit 75c77d508be Check Dag file claims per bundle on the parse path
omit 9b98c04bd91 Import the coordinator manager at the top of the importer
base
omit 60d7af4cad9 Let a coordinator parse the Dag files of the bundles it
serves
omit f9fd8e01be2 Sync the Java SDK copy of the Dag schema
omit 464c9273c0c Serialize the SDK Dag in tests that write a dag_maker Dag
omit e5d1a71ca9c Note in the parsing ADR that runtimes may omit
config-backed fields
omit 76389a9ffa2 Check SDK Dags with the Dag processor's validation in
conformance
omit d8cf5805ec7 Reject malformed task entries in a serialized Dag
omit c1226408285 Fill a serialized Dag's unset settings from the Airflow
config
omit adf872d0ac6 Check that a serialized Dag can be stored and loaded
omit 0649a43f468 Move the Dag file processor's shared plumbing into a base
class
omit 20603491841 Pin the empty task id case in the Dag cycle detection tests
omit 737097a0dfe Share Dag cycle detection between the Task SDK and core
add 383a9a87a98 Stop wiping the uv cache before every CI image build
(#74052)
add 58e987defec Fix Non-DB test collection failure in Google Stackdriver
hook test (#74066)
add 498724fc13a Go SDK: enumerate every user-facing enum in the airflow
package (#74064)
add 5e476d7cf89 TS SDK: answer the Dag-parsing request from bundle.serve
(#73442)
add dd271599939 Go SDK: vendor the schemas it generates from (#74047)
add 77927390c17 Move Dag cycle detection from the Task SDK to shared
(#74034)
add 7c112fe041f Move the Dag file processor's shared plumbing into a base
class
add 0834b7149ac Check that a serialized Dag can be stored and loaded
add d7e9d5f5e72 Fill a serialized Dag's unset settings from the Airflow
config
add 2482920cb7e Reject malformed task entries in a serialized Dag
add d3effc9c25d Check SDK Dags with the Dag processor's validation in
conformance
add f1f5d581143 Note in the parsing ADR that runtimes may omit
config-backed fields
add ad6eae0144f Serialize the SDK Dag in tests that write a dag_maker Dag
add 6a995f27589 Sync the Java SDK copy of the Dag schema
add 43d117ddbca Let a coordinator parse the Dag files of the bundles it
serves
add f27fb2d3de8 Import the coordinator manager at the top of the importer
base
add 27a987d2641 Check Dag file claims per bundle on the parse path
add 39fcab0bc98 Drop serves_bundle so get_parsed_bundles is the only answer
add d21aea99762 Say dag_bundle_name is only needed for two coordinators of
one class
add 693bdf2b4d4 Parse coordinator-claimed Dag files with their runtime in
the Dag processor
add 0f7db884a00 Report a Lang-SDK parse past its import timeout as an
import error
add 84d307204bd Report a native Dag's task that reaches a Python worker
add 88abc9efdd2 Drop callbacks for Lang-SDK Dag files before queuing the
file
add dab3acf4bee Start each Dag file processor with explicit arguments
add 49840104918 Return a Dag importer from every SubprocessCoordinator call
add bfd0b241a5c Find the claiming coordinator in the Task SDK and wrap it
in core
add 52c2c4b4929 Fail a native Dag's task on a Python worker without retries
add e480eae1fa7 Build each bundle's Dag importer registry before freezing
the heap
add 571336b499c Match the schema error's JSON path in a Lang-SDK parse test
add 1b9e9d7e407 Kill what a Lang-SDK runtime leaves in its process group
add f0302a6a041 Name the setting in a Lang-SDK parse timeout error
add 50a13677d5c Spec the mocks of a hand-built Lang-SDK processor in tests
add 68072d7cdf2 Apply the multi-team task checks to Lang-SDK Dags
add 2b1e171bfe6 Clear import errors after the native-file Dag bag sync test
add 6b773354fa0 Note in ADR-0010 that the manager routes claimed files for
now
add d0eafc41a00 Move the client-less Lang-SDK parse out of this change
add 31c1a6b5a72 Wait for the killed leftover in the Lang-SDK processor test
add df5ba5d5d62 Parse coordinator-claimed Dag files with their runtime in
a Dag bag
add 0ea04fd6a01 Let a Dag file processor run without an API client
add e7657922dbb Bag native Lang-SDK Dags in core, only when asked
add ad8aaa2a6a9 List the tasks of a native Lang-SDK Dag
add 6fea2b43a31 Ignore the bagged Dag union in the external task sensor
test
add cd00618ba80 Read JAR manifests per the JAR specification
add 8f2cf075fd2 Build the JVM command that parses a Dag JAR
add 96234e6d17d Embed the Dag source in Java bundle JARs
add 3160ab91bf8 Parse native Java Dags with their coordinator
add 6ed348f241a Treat an unreadable JAR like one without a manifest
add a9ee37a8def Test the Java Dag source placeholder for a JAR without a
manifest
add 52e1b94dfe7 Name the JAR manifest attributes the Java coordinator reads
add fbd5b06be45 Document native Java Dag parsing
add 1cf7c2cc95e Run a task on its run's version when dag_bundle_name names
its bundle
add 8cd962566d3 Pass the task's Dag file to the coordinator's command
builder
add 5c769623a95 Report why a Java Dag JAR cannot be read
add ac66e0a3af2 Run the JAR a native Java Dag was parsed from
add 47aaa41a857 Reject a Java Dag JAR whose SDK cannot answer a parse
add 926eee46796 Stop promising that Java dependency JARs are not parsed
add 81f952731a2 Keep the Java parse_dag test from changing the pytest
worker
add d1cc2a44291 Parse native TypeScript Dags with the Node coordinator
add 7453fe7a3af Document parsing native TypeScript Dags
add ef480595276 Share the bundle section mutation helper between Node tests
add 50fa4d83f06 Run a native TypeScript Dag's task from its own bundle
add 6ff552bff78 Say when the Dag processor needs Node.js for TypeScript
bundles
add ef4a6022cea Align the TypeScript build docs with the Code view
add 4f597ad16e7 Warn against defining a Dag in both TypeScript and Python
add 170bd0501a7 Import NodeDagImporter at the top of the Node coordinator
add 83e1a0e950e Let NodeDagImporter source read errors reach the caller
add 52bf274b956 Pin the Code view to the entry module in Node bundle tests
add c4033cbd957 Migrating from filepath based Dag to DagDefinition
discovery
add f9c69e8140b Handle Dags nested at any depth in Dag archives
add d0ced785835 Stub Dag file discovery with DagFileInfo in the Lang-SDK
manager tests
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4ff4171bd7f)
\
N -- N -- N refs/heads/jason/lang-sdk-e2e/02-importer-registry
(d0ced785835)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.pre-commit-config.yaml | 25 ++-
.../airflow_breeze/commands/ci_image_commands.py | 4 +-
dev/breeze/tests/test_ci_image_commands.py | 20 ++
go-sdk/README.md | 52 +++--
go-sdk/adr/0008-native-dag-interface.md | 5 +
go-sdk/airflow/dag.go | 5 +
go-sdk/airflow/enums.go | 156 +++++++++++++++
go-sdk/airflow/enums_test.go | 221 +++++++++++++++++++++
go-sdk/airflow/spec.go | 33 +--
go-sdk/airflow/spec_test.go | 71 -------
go-sdk/airflow/trigger_dag_run.go | 36 ++--
go-sdk/airflow/trigger_dag_run_test.go | 24 +--
go-sdk/internal/genspec/authoring.go | 6 +-
go-sdk/internal/genspec/main.go | 9 +-
go-sdk/internal/genspec/normalize_test.go | 5 +-
go-sdk/pkg/execution/genmodels/gen.go | 13 +-
go-sdk/pkg/execution/messages_test.go | 23 +--
{ts-sdk => go-sdk}/schema/dag-schema.json | 13 +-
.../schema/supervisor-schema.json | 0
.../unit/google/cloud/hooks/test_stackdriver.py | 2 +-
scripts/ci/prek/check_go_sdk_generated_drift.py | 25 ++-
scripts/ci/prek/sync_go_sdk_schemas.py | 142 +++++++++++++
.../ci/prek/test_check_go_sdk_generated_drift.py | 13 +-
scripts/tests/ci/prek/test_sync_go_sdk_schemas.py | 127 ++++++++++++
ts-sdk/src/coordinator/runtime.ts | 69 ++++++-
ts-sdk/tests/coordinator/integration.test.ts | 108 +++++++++-
26 files changed, 980 insertions(+), 227 deletions(-)
create mode 100644 go-sdk/airflow/enums.go
create mode 100644 go-sdk/airflow/enums_test.go
delete mode 100644 go-sdk/airflow/spec_test.go
copy {ts-sdk => go-sdk}/schema/dag-schema.json (97%)
copy task-sdk/src/airflow/sdk/execution_time/schema/schema.json =>
go-sdk/schema/supervisor-schema.json (100%)
create mode 100755 scripts/ci/prek/sync_go_sdk_schemas.py
create mode 100644 scripts/tests/ci/prek/test_sync_go_sdk_schemas.py
