Yes, you would need to duplicate a chunk of the default_celery in your copy right now. But you can just make it have the values you want - so it would be about 10 lines in total.
It seems that between AIRFLOW-966 and AIRFLOW-1840 things got a little bit out of sync in the default .cfg and the celery .py - the .py is looking for celery->ssl_* but the default config puts it in celery_broker_transport_options->ssl_*. Looking at the celery config option it looks like they aren't actually options for the other transports, it's just that they don't complain about the extra options. I think the fix is just to move them up to the celery section. -ash > On 24 May 2018, at 07:21, Craig Rodrigues <rodr...@crodrigues.org> wrote: > > Ash, > > Thanks! You put me on the right track. > Unfortunately, there is a lot of logic in > airflow/config_templates/default_celery.py that I need, > and if I was to come up with my own class to replace: > > celery_config_options = > airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG > > then I would basically have to rewrite my own version of default_celery.py > > Also, in airflow/config_templates/default_airflow.cfg, there is this: > > [celery_broker_transport_options] > # The visibility timeout defines the number of seconds to wait for the > worker > # to acknowledge the task before the message is redelivered to another > worker. > # Make sure to increase the visibility timeout to match the time of the > longest > # ETA you're planning to use. Especially important in case of using Redis > or SQS > visibility_timeout = 21600 > > # In case of using SSL > ssl_active = False > ssl_key = > ssl_cert = > ssl_cacert = > > > None of those options work if using an sqla backend for celery broker_url. > > I need to think about this, but this needs to be cleaned up before > Airflow 1.10 is released. In airflow/config_templates/default_airflow.cfg > there is this: > > broker_url = sqla+mysql://airflow:airflow@localhost:3306/airflow > > > So sqla is specified as the default broker_url for celery. > A lot of people (including where I work) have used this template to set up > Airflow + Celery, and even though sqla is an "experimental" broker_url, > it actually works pretty well. > > Now in airflow 1.10, something that was very easy to set up is now really > complicated and unintuitive. > > Would it be OK to change the airflow code so that > in airflow/config_templates/default_airflow.cfg, all the options in > [celery_broker_transport_options] are commented out? > And if someone is running Redis, they would have to add those > options in their own airflow.cfg file? > > Bolke, do you have any comments? > > -- > Craig > > On Tue, May 22, 2018 at 1:50 AM Ash Berlin-Taylor < > ash_airflowl...@firemirror.com> wrote: > >> To use with the SQLA backend to celery you need to override the options >> Airflow passes to Celery. Those come from >> https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py >> >> Since you don't want most/all of those options (and there is no way in the >> config file to _remove_ a setting) you will have to point airflow to a >> different file for the celery config: >> >> This line in the config is what you will need to change: >> >> # Import path for celery configuration options >> celery_config_options = >> airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG >> >> If you create something like config/celery_config.py containing: >> >> >> CELERY_CONFIG = { >> # Just the options you want to set >> } >> >> >> (config/ should exist along side your dags/ folder, and I think it should >> be added to the python path already). You can then set this in the config: >> >> celery_config_options = celery_config.CELERY_CONFIG >> >> That should give you complete control >> >>