You can watch this talk from Airflow Summit: https://airflowsummit.org/sessions/2021/provision-as-a-service/
This is the talk from CloudFlare on how they are maintaining their infrastructure using Airflow. I was surprised to see it (you are right Airflow is primary used to process data), but as you can see the MOST serious infra people are using Airflow for managing infrastructure and they are super happy with it. I bet you will be too :) J On Tue, Oct 5, 2021 at 3:23 PM Mathieu Poussin <[email protected]> wrote: > > Hello Everyone, > > We are currently evaluating a few solutions for our next stack of network > automation & orchestration platform (Airflow, StackStorm, BPMN based > solutions, Django/Celery manually, etc...) > Airflow is one of the candidates. > > The typical jobs would be for example: > - Get a few parameters on the type of device to deploy > - Generate the configuration from an extenal tools and push the configuration > on various internal systems > - Wait for the device to boot and get the configuration > - Validate the device health (SSH'ing or API call on it) > - Notify on slack (and via other API calls) that the provisioning is complete > > I started testing it but had a few questions I could not find answers : > > - I see that is it mostly used for data processing in general, is it good for > something else like our case ? > > - It's not clear how to handle dynamic dags, for example if I have a first > task getting a list of devices and I want the second task to be run (in > parallel if possible) for each device ? I have read many times that Airflow > is not made for this type of dynamic workflows, is this still the case ? > > - We will probably not use the scheduled tasks features but run the tasks on > demand (after receiving API calls) > > - Is there any equivalent of the sensor feature we can find on StackStorm > (that is also a tool we are evaluating) ( > https://docs.stackstorm.com/sensors.html ) > > - Is there a way to validate/enforce job parameters ? Or this have to be done > manually by checking the dict content ? > > Thanks > >
