jason810496 opened a new issue, #74335:
URL: https://github.com/apache/airflow/issues/74335
### Description
Once #74036 and #74037 land, a native TypeScript Dag bundle can generate its
Dags from a YAML file in the Dag bundle. Editing only the YAML changes the live
Dags, and their tasks run without repacking the bundle:
- The Dag processor runs the packed bundle on every parse, so the YAML is
read fresh.
- A task of a native Dag runs the `.min.mjs` named by its `dag_rel_path`, at
the run's bundle version, with no check against the `dag_id`s recorded at pack
time.
```ts
import { readFileSync } from "node:fs";
import { parse } from "yaml";
import { Bundle, Dag } from "apache-airflow-ts-sdk";
const config = parse(readFileSync(new URL("./dags.yaml", import.meta.url),
"utf-8"));
const bundle = new Bundle();
for (const spec of config.dags) {
const dag = new Dag(spec.dag_id, { queue: "typescript" });
// add tasks from spec
bundle.register(dag);
}
await bundle.serve();
```
### Follow-ups
- [ ] Add an e2e case: pack the bundle, add a Dag to the YAML in the Dag
bundle, wait for a re-parse, then trigger the new Dag and check that it
succeeds.
- [ ] Document dynamic Dag generation in the TypeScript SDK native Dag guide:
- Read the YAML at runtime with `fs` and `import.meta.url`. Importing it
lets the bundler inline it, which freezes it at pack time.
- Keep the YAML in the same Dag bundle. `airflow-ts-pack` also runs the
bundle, so the YAML must exist at pack time, or the code must handle it being
missing.
- Generation must be deterministic, and every generated Dag needs the
queue that routes to the `NodeCoordinator`.
- A YAML change shows up on the next periodic re-parse (`[dag_processor]
min_file_process_interval`). Editing the YAML does not trigger an earlier
re-parse.
- With a versioned Dag bundle, runs keep the YAML of their bundle version.
With an unversioned bundle, a running Dag run sees the new YAML, and tasks
removed from it end as `removed`.
### Depends on
- #74036
- #74037
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
