Hi, My name is Darren and I'm head of data at Auto Trader UK. Auto Trader is the largest automotive marketplace, one of the largest sites in the UK (based on traffic).
We have been using Airflow for about 18 months with the LocalExecutor and now the K8 Executor is out we've been migrating towards that. FYI we are heavy users of Kubernetes: - https://cloud.google.com/customers/auto-trader-uk/ - https://cloudplatform.googleblog.com/2018/07/istio-reaches-1-0-ready-for-prod.html Whilst working on the migration, we've found there are some improvements which would be good to make to the executor: - Support longer DAG names - we've opened a PR for this and are working on it already. https://github.com/apache/airflow/pull/4636 - Improve the executor from a Kubernetes Operator perspective. - Define requests and limits for Kubernetes for the workers. We operate a multi-tenant cluster across teams and not being able to define limits means a "noisy"/unbounded worker could have service impacting results to other applications. It also prevents the worker auto-scaler from working. - Define the entry point for the worker pod - for some reason the original entry point for the worker is not the same as the Airflow Image. We use the entry point to do some dynamic initialisation on the worker. - Ideally, have the ability to supply the full manifest for the worker. This is a proposal over creating an adapter layer on top of Kubernetes which has to be maintained. We are keen to support Airflow and also add these features, we'd like to understand if you'd be open to contributors? We don't want to invest time on the executor if it wouldn't fit in with the design you'd like or it never got merged into mainline Airflow. Managing forks is a painful process and we are also huge advocates of contributing back to OSS. FY I spoke to Eamon (on Slack) and he's informed me Daniel from Bloomberg did most of the initial work on the executor; I've reached out to him directly as well. It's great to connect, hopefully, I'll hear back from you. Cheers, Darren
