Ash, According to this: http://docs.celeryproject.org/en/latest/userguide/configuration.html#broker-settings visibility_timeout is supported by Redis and SQS.
-- Craig On Thu, May 24, 2018 at 8:07 AM Craig Rodrigues <rodr...@crodrigues.org> wrote: > Ash, > > Thanks again. You are leading me on the right path! > > I can prepare a patch to move the ssl_ options into the celery section. > > What about visbility_timeout? The error I am getting is: > > File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", > line 160, in create > engineclass.__name__)) > TypeError: Invalid argument(s) > 'ssl_key','ssl_cert','ssl_active','visibility_timeout','ssl_cacert' sent to > create_engine(), using configuration > MySQLDialect_mysqldb/QueuePool/Engine. Please check that the keyword > arguments are appropriate for this combination of components. > > > So it looks like visibility_timeout does not work with sqla as well. > -- > Craig > > > On Thu, May 24, 2018 at 2:17 AM Ash Berlin-Taylor < > ash_airflowl...@firemirror.com> wrote: > >> Yes, you would need to duplicate a chunk of the default_celery in your >> copy right now. But you can just make it have the values you want - so it >> would be about 10 lines in total. >> >> It seems that between AIRFLOW-966 and AIRFLOW-1840 things got a little >> bit out of sync in the default .cfg and the celery .py - the .py is looking >> for celery->ssl_* but the default config puts it in >> celery_broker_transport_options->ssl_*. Looking at the celery config option >> it looks like they aren't actually options for the other transports, it's >> just that they don't complain about the extra options. I think the fix is >> just to move them up to the celery section. >> >> -ash >> >> >> > On 24 May 2018, at 07:21, Craig Rodrigues <rodr...@crodrigues.org> >> wrote: >> > >> > Ash, >> > >> > Thanks! You put me on the right track. >> > Unfortunately, there is a lot of logic in >> > airflow/config_templates/default_celery.py that I need, >> > and if I was to come up with my own class to replace: >> > >> > celery_config_options = >> > airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG >> > >> > then I would basically have to rewrite my own version of >> default_celery.py >> > >> > Also, in airflow/config_templates/default_airflow.cfg, there is this: >> > >> > [celery_broker_transport_options] >> > # The visibility timeout defines the number of seconds to wait for the >> > worker >> > # to acknowledge the task before the message is redelivered to another >> > worker. >> > # Make sure to increase the visibility timeout to match the time of the >> > longest >> > # ETA you're planning to use. Especially important in case of using >> Redis >> > or SQS >> > visibility_timeout = 21600 >> > >> > # In case of using SSL >> > ssl_active = False >> > ssl_key = >> > ssl_cert = >> > ssl_cacert = >> > >> > >> > None of those options work if using an sqla backend for celery >> broker_url. >> > >> > I need to think about this, but this needs to be cleaned up before >> > Airflow 1.10 is released. In >> airflow/config_templates/default_airflow.cfg >> > there is this: >> > >> > broker_url = sqla+mysql://airflow:airflow@localhost:3306/airflow >> > >> > >> > So sqla is specified as the default broker_url for celery. >> > A lot of people (including where I work) have used this template to set >> up >> > Airflow + Celery, and even though sqla is an "experimental" broker_url, >> > it actually works pretty well. >> > >> > Now in airflow 1.10, something that was very easy to set up is now >> really >> > complicated and unintuitive. >> > >> > Would it be OK to change the airflow code so that >> > in airflow/config_templates/default_airflow.cfg, all the options in >> > [celery_broker_transport_options] are commented out? >> > And if someone is running Redis, they would have to add those >> > options in their own airflow.cfg file? >> > >> > Bolke, do you have any comments? >> > >> > -- >> > Craig >> > >> > On Tue, May 22, 2018 at 1:50 AM Ash Berlin-Taylor < >> > ash_airflowl...@firemirror.com> wrote: >> > >> >> To use with the SQLA backend to celery you need to override the options >> >> Airflow passes to Celery. Those come from >> >> >> https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py >> >> >> >> Since you don't want most/all of those options (and there is no way in >> the >> >> config file to _remove_ a setting) you will have to point airflow to a >> >> different file for the celery config: >> >> >> >> This line in the config is what you will need to change: >> >> >> >> # Import path for celery configuration options >> >> celery_config_options = >> >> airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG >> >> >> >> If you create something like config/celery_config.py containing: >> >> >> >> >> >> CELERY_CONFIG = { >> >> # Just the options you want to set >> >> } >> >> >> >> >> >> (config/ should exist along side your dags/ folder, and I think it >> should >> >> be added to the python path already). You can then set this in the >> config: >> >> >> >> celery_config_options = celery_config.CELERY_CONFIG >> >> >> >> That should give you complete control >> >> >> >> >> >>
