The problem is Airflow flexibility. All tasks are defined as Python code and this code must be executed to see if the tasks can be executed. Let's add that one machine cannot execute code from different users, because Python does not provide the required isolation. So we have a case that you still need to run the scheduler to be able to check if the task is to be carried out. It would be possible to do this if each task was static and its condition could not change during execution. I also pay attention to the fact that serverless services use one central database e.g. Google Spanner, but in Airflow every operator can perform any database query.
Some providers already provide workflow as a serverless service, but their practical use is very limited. https://aws.amazon.com/step-functions/ Note that successful autoscaling for workers is being undertaken to ensure more efficient use of resources at a time of limited activity. Next, attempts will be made to decouple the database e.g. by creating an API and communicating only through it. On Sat, Jan 25, 2020 at 7:01 PM Malthe <[email protected]> wrote: > > In a typical deployment, the scheduler and worker processes run as > daemons, lingering during idle periods (the scheduler literally sleeps > as part of its run loop). > > Meanwhile, deploying to a serverless platform is attractive for > various reasons (cost, security, simplicity, scaling). > > For example, on Azure where there's a free grant of 1,000,000 > executions per month, running a single iteration of the Airflow > scheduler loop on a timer trigger set to once every 3 seconds would > cost nothing. Still using Azure as an example, tasks might be > submitted using an event grid connected to an Azure Function App which > would immediately pick up one or more tasks. > > It seems like this wouldn't be too difficult to implement for a given > cloud provider, but I'd like to ask on this list whether there are > obvious issues with this execution model and whether this has been > suggested or even implemented previously. I wasn't able to find any > mention. > > --\-- > cheers
