Ash, Thanks! You put me on the right track. Unfortunately, there is a lot of logic in airflow/config_templates/default_celery.py that I need, and if I was to come up with my own class to replace:
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG then I would basically have to rewrite my own version of default_celery.py Also, in airflow/config_templates/default_airflow.cfg, there is this: [celery_broker_transport_options] # The visibility timeout defines the number of seconds to wait for the worker # to acknowledge the task before the message is redelivered to another worker. # Make sure to increase the visibility timeout to match the time of the longest # ETA you're planning to use. Especially important in case of using Redis or SQS visibility_timeout = 21600 # In case of using SSL ssl_active = False ssl_key = ssl_cert = ssl_cacert = None of those options work if using an sqla backend for celery broker_url. I need to think about this, but this needs to be cleaned up before Airflow 1.10 is released. In airflow/config_templates/default_airflow.cfg there is this: broker_url = sqla+mysql://airflow:airflow@localhost:3306/airflow So sqla is specified as the default broker_url for celery. A lot of people (including where I work) have used this template to set up Airflow + Celery, and even though sqla is an "experimental" broker_url, it actually works pretty well. Now in airflow 1.10, something that was very easy to set up is now really complicated and unintuitive. Would it be OK to change the airflow code so that in airflow/config_templates/default_airflow.cfg, all the options in [celery_broker_transport_options] are commented out? And if someone is running Redis, they would have to add those options in their own airflow.cfg file? Bolke, do you have any comments? -- Craig On Tue, May 22, 2018 at 1:50 AM Ash Berlin-Taylor < ash_airflowl...@firemirror.com> wrote: > To use with the SQLA backend to celery you need to override the options > Airflow passes to Celery. Those come from > https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py > > Since you don't want most/all of those options (and there is no way in the > config file to _remove_ a setting) you will have to point airflow to a > different file for the celery config: > > This line in the config is what you will need to change: > > # Import path for celery configuration options > celery_config_options = > airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG > > If you create something like config/celery_config.py containing: > > > CELERY_CONFIG = { > # Just the options you want to set > } > > > (config/ should exist along side your dags/ folder, and I think it should > be added to the python path already). You can then set this in the config: > > celery_config_options = celery_config.CELERY_CONFIG > > That should give you complete control > >
