xchwan commented on PR #75945: URL: https://github.com/apache/airflow/pull/75945#issuecomment-6650283561
@kaxil I ran this end to end on another machine (a14, 16 CPUs, 32 GB RAM) and recorded it. <img width="560" height="420" alt="ui-dags-list" src="https://github.com/user-attachments/assets/c2bf57da-1f1a-4fec-93a6-ac5fc739f100"; /> https://github.com/user-attachments/assets/93c64603-b868-43fd-97ef-31a3e27f50be **What I ran:** the whole chart on kind (Kubernetes v1.31.12) with an image built from this checkout, not only the Dag processors: migration job, API server, scheduler, PostgreSQL and two Dag processor sets, with the Dags visible in the UI and Dag runs executed. **Setup:** two `LocalDagBundle`s defined once in `dagProcessor.dagBundleConfigList`, one small Dag each, and two sets: ```yaml dagProcessor: sets: - {name: set-a, bundleNames: [bundle_a]} - {name: set-b, bundleNames: [bundle_b]} ``` **Deployment and UI** - Two Deployments, `set-a` and `set-b`, running `airflow dag-processor --bundle-name 'bundle_a'` and `--bundle-name 'bundle_b'`; no default Deployment when only `sets` is used. - `airflow dags list` shows `bundle_a_dag` under `bundle_a` and `bundle_b_dag` under `bundle_b`; both appear in the UI (screenshot) and the Dag Processor health check is green. - Both Dags ran to `success`, each task included. - Stability: 10 samples over 9 minutes, one per minute. The API server, scheduler, PostgreSQL and both Dag processor Pods were ready in every sample with 0 restarts. **Chart behaviour** - `enableDefault=true` adds the default Deployment next to both sets. - PDB enabled on one set renders only that set's PDB, selecting `dag-processor-set: set-a`. - Bad configs (undefined bundle in `bundleNames`, `dagBundleConfigList` inside a set) fail `helm template` with a clear message. - `replicas: 2` on set-a gives two Pods, both on `bundle_a`. **Isolation** - Recording: with a 40Mi memory limit on set-b, the new set-b Pod goes through `Init:OOMKilled` and `Init:CrashLoopBackOff` while both set-a Pods are `Running` (set-a is scaled to 2 replicas here). I then add a Dag to `bundle_a` and it shows up in the UI next to `bundle_b_dag`. The recording is cut where I waited for the Dag to be parsed and reconnected the port-forward, so the new Dag does not appear instantly. - A 200Mi limit on set-b makes the dag-processor container itself `OOMKilled` on every start (5 restarts, `CrashLoopBackOff`), with no running set-b Pod. A Dag added to `bundle_a` in that state was parsed by set-a and its run finished with `success`. - With `bundle_b` pointed at a path that does not exist, set-b keeps running and finds nothing; a Dag added to `bundle_a` was still parsed by set-a. **Tests** - `test_dag_processor_sets.py`: 19 passed, with `uv run --project chart pytest` and inside the breeze CI image. - Full chart suite, `breeze testing helm-tests --use-xdist`: 2278 passed. --- Drafted-by: Claude Code (Sonnet 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
