jscheffl opened a new pull request, #40900: URL: https://github.com/apache/airflow/pull/40900
This PR is a DRAFT to further get to a MVP release of AIP-69. DO NOT MERGE (yet) - but please feedback on it if you like. It is a continuation of PR #40224 which was a kind of PoC and step-by-step more stuff came onto this on top. This is a kind of "Mothership" PR where all contributions are collected. Small parts will be separately opened as PR to get on main and finally I'd split this PR up into smaller pieces for easier review. In this PR all parts and work is collected. Please treat this as experimental as noted in the docs, it is not expected to be rock-solid but a first usable version. It can be used to distribute the first task. Status of implementation is tracked in `airflow/providers/remote/TODO.md` in this PR. If you want to test it you can: - Checkout the branch - Run `breeze start-airflow --python 3.12 --load-example-dags --backend postgres --executor RemoteExecutor --answer y` To enable the API endpoint you need to re-start the webserver with `AIRFLOW__REMOTE__API_ENABLED=true airflow webserver` - Start the Remote executor with `airflow/providers/remote/_start_remote_worker.sh` - A task can be run, e.g. use example DAG `example_python_operator` or the Ìntegration Test` with a bit more complexity. Wohoo!  - If you open the UI you will see an additional plugin loaded in the menu "Admin" -> "Remote Worker Status":  Based on progress and time I will sometimes rebase and push changes to this branch. But it will not be something that will be review-able. I will not add any migration logic in case of updates for the additional tables in DB Scheme. If I update the draft PR and DB schema changes, DB might need to be reset. Of course migration support would need to be added before a "real" release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
