argibbs commented on issue #34339: URL: https://github.com/apache/airflow/issues/34339#issuecomment-1717655366
> Just wondering, are you execute Prefect flows/tasks inside of bash operator which run in celery worker? LOL, I _wondered_ if anyone would pick up on that. Bonus internet points to you, sir. I won't bore you with the minutiae (in this chain at least), but yes, we're using prefect in a limited fashion. The bash operator spins up a docker container, which contains code that builds a graph of tasks (which are quite fine-grained, think along the lines of dask, they're not something you'd want to try to create as an airflow dag) and then sends that to prefect. We don't use prefect's workers against pre-existing deployments, the graph is created dynamically executed locally in the docker container. We just mostly use prefect for its pretty gui, and other out-of-the-box functionality. FWIW, I think prefect and airflow are miles apart in terms of functionality and target audience. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
