argibbs commented on issue #34339:
URL: https://github.com/apache/airflow/issues/34339#issuecomment-1717655366

   > Just wondering, are you execute Prefect flows/tasks inside of bash 
operator which run in celery worker?
   
   LOL, I _wondered_ if anyone would pick up on that. Bonus internet points to 
you, sir.
   
   I won't bore you with the minutiae (in this chain at least), but yes, we're 
using prefect in a limited fashion. The bash operator spins up a docker 
container, which contains code that builds a graph of tasks (which are quite 
fine-grained, think along the lines of dask, they're not something you'd want 
to try to create as an airflow dag) and then sends that to prefect. We don't 
use prefect's workers against pre-existing deployments, the graph is created 
dynamically executed locally in the docker container. We just mostly use 
prefect for its pretty gui, and other out-of-the-box functionality.
   
   FWIW, I think prefect and airflow are miles apart in terms of functionality 
and target audience.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to