Re: celery problem: cannot override celery_broker_transport_options

I have submitted this patch: https://github.com/apache/incubator-airflow/pull/3417 I have tested with a Redis celery broker, and with a SQLAlchemy/MySQL celery broker. It works on Redis, and with SQLAlchemy/MySQL. With this patch, I no longer get this exception: File

Re: celery problem: cannot override celery_broker_transport_options

I have submitted this patch: https://github.com/apache/incubator-airflow/pull/3417 I have tested with a Redis celery broker, and with a SQLAlchemy/MySQL celery broker. It works on Redis, and with SQLAlchemy/MySQL. With this patch, I no longer get this exception: File

Re: celery problem: cannot override celery_broker_transport_options

You are right!! This is an interesting puzzle to follow, but we are unravelling it! -- Craig On Thu, May 24, 2018 at 8:39 AM Ash Berlin-Taylor < ash_airflowl...@firemirror.com> wrote: > Kombu (a library Celery uses) 4.1.0 added it back in >

Re: celery problem: cannot override celery_broker_transport_options

Kombu (a library Celery uses) 4.1.0 added it back in https://github.com/celery/kombu/blob/master/Changelog#L75-L99 - I _thought_ that means it's supported in Celery again...? > On 24 May 2018, at 16:34, Craig Rodrigues

Re: celery problem: cannot override celery_broker_transport_options

Removal of sqla as a backend is mentioned in these release notes for celery 4.0: http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#features-removed-for-lack-of-funding -- Craig On Thu, May 24, 2018 at 8:32 AM Craig Rodrigues wrote: > It looks like in Celery,

Re: celery problem: cannot override celery_broker_transport_options

It looks like in Celery, the documentation for sqla broker was removed: https://github.com/celery/celery/commit/79810a26a116e9881c42a14d856fa94c40fefcd8#diff-29ccf8c96d521253467909a652e6ded2 I cannot find the pull request or release notes which document this. -- Craig On Thu, May 24, 2018 at

Re: celery problem: cannot override celery_broker_transport_options

Sounds like https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py#L31 should be guarded in some way to only do that for a redis:// and

Re: celery problem: cannot override celery_broker_transport_options

Ash, According to this: http://docs.celeryproject.org/en/latest/userguide/configuration.html#broker-settings visibility_timeout is supported by Redis and SQS. -- Craig On Thu, May 24, 2018 at 8:07 AM Craig Rodrigues wrote: > Ash, > > Thanks again. You are leading me

Re: celery problem: cannot override celery_broker_transport_options

Ash, Thanks again. You are leading me on the right path! I can prepare a patch to move the ssl_ options into the celery section. What about visbility_timeout? The error I am getting is: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 160, in create

Re: celery problem: cannot override celery_broker_transport_options

Yes, you would need to duplicate a chunk of the default_celery in your copy right now. But you can just make it have the values you want - so it would be about 10 lines in total. It seems that between AIRFLOW-966 and AIRFLOW-1840 things got a little bit out of sync in the default .cfg and the

Re: celery problem: cannot override celery_broker_transport_options

Ash, Thanks! You put me on the right track. Unfortunately, there is a lot of logic in airflow/config_templates/default_celery.py that I need, and if I was to come up with my own class to replace: celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG then I

Re: celery problem: cannot override celery_broker_transport_options

To use with the SQLA backend to celery you need to override the options Airflow passes to Celery. Those come from https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py Since you don't want most/all of those options (and there is no way in the

Re: celery problem: cannot override celery_broker_transport_options

Bolke, Can you help me with this? You have worked on this code with respect to parsing celery broker options. I cannot figure out how to override the defaults, and wrong values are being passed down into the mysql backend, causing things to fail. This is blocking me from doing further testing

celery problem: cannot override celery_broker_transport_options

Hi, I used this requirements.txt file to install airflow from the v1-10-test branch: git+https://github.com/celery/celery@master#egg=celery git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3] kombu>=4.1.0