Re: Airflow 1.10.0

Hi, To move things along even faster, instead of creating new packages, you can even tell people to put something like this in a requirements.txt file: git+ https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]

Enable Travis CI Auto Cancellation?

Can someone who has administrator access to Travis CI enable Auto-Cancellation on branch and Pull Requests builds? See: https://blog.travis-ci.com/2017-09-21-default-auto-cancellation -- Craig

Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator

I have submitted a patch: https://github.com/apache/incubator-airflow/pull/3442 -- Craig On 2018/05/30 19:45:23, Craig Rodrigues wrote: > Oh, OK, I just saw this in example_kubernetes_operator.py: > > try: > # Kubernetes is optional, so not available in vanilla Airflow

Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator

es stuff, so > the KubernetesPodOperator is ignored. We need to figure out how to have > example dags that are not compatible with the vanilla installation, or we > need to remove the kubernetes example for now, and move it to the > documentation. > > Cheers, Fokko > -- Craig Rodrigues rodr...@rodrigues.org

Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator

For this particular DeprecationWarning, this problem is not caused by dependencies on kubernetes stuff. On this line: https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/kubernetes_pod_operator.py#L137 The __init__() method for KubernetesPodOperator is calling the

KubernetesPodOperator: Invalid arguments were passed to BaseOperator

I tested master branch by putting the following in my requirements.txt: git+https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3] and did a pip install -r requirements.txt When I started the airflow webserver, I

Re: celery problem: cannot override celery_broker_transport_options

I have submitted this patch: https://github.com/apache/incubator-airflow/pull/3417 I have tested with a Redis celery broker, and with a SQLAlchemy/MySQL celery broker. It works on Redis, and with SQLAlchemy/MySQL. With this patch, I no longer get this exception: File

Re: celery problem: cannot override celery_broker_transport_options

I have submitted this patch: https://github.com/apache/incubator-airflow/pull/3417 I have tested with a Redis celery broker, and with a SQLAlchemy/MySQL celery broker. It works on Redis, and with SQLAlchemy/MySQL. With this patch, I no longer get this exception: File

Re: celery problem: cannot override celery_broker_transport_options

er/Changelog#L75-L99 < > https://github.com/celery/kombu/blob/master/Changelog#L75-L99> - I > _thought_ that means it's supported in Celery again...? > > > > On 24 May 2018, at 16:34, Craig Rodrigues <rodr...@crodrigues.org> > wrote: > > > > Removal

Re: celery problem: cannot override celery_broker_transport_options

Removal of sqla as a backend is mentioned in these release notes for celery 4.0: http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#features-removed-for-lack-of-funding -- Craig On Thu, May 24, 2018 at 8:32 AM Craig Rodrigues <rodr...@crodrigues.org> wrote: > It looks like

Re: celery problem: cannot override celery_broker_transport_options

tes/default_celery.py#L31> > should be guarded in some way to only do that for a redis:// and sqs:// > backends. > > > > On 24 May 2018, at 16:13, Craig Rodrigues <rodr...@crodrigues.org> > wrote: > > > > Ash, > > > > According to this: > &

Re: celery problem: cannot override celery_broker_transport_options

Ash, According to this: http://docs.celeryproject.org/en/latest/userguide/configuration.html#broker-settings visibility_timeout is supported by Redis and SQS. -- Craig On Thu, May 24, 2018 at 8:07 AM Craig Rodrigues <rodr...@crodrigues.org> wrote: > Ash, > > Thanks again. You

Re: celery problem: cannot override celery_broker_transport_options

> celery_broker_transport_options->ssl_*. Looking at the celery config option > it looks like they aren't actually options for the other transports, it's > just that they don't complain about the extra options. I think the fix is > just to move them up to the celery section. > >

Re: celery problem: cannot override celery_broker_transport_options

Ash, Thanks! You put me on the right track. Unfortunately, there is a lot of logic in airflow/config_templates/default_celery.py that I need, and if I was to come up with my own class to replace: celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG then I

Re: kwargs usage in BaseOperator

On Monday, May 21, 2018, Tao Feng wrote: > > I wonder whether we still have the plan to remove kwargs in airflow 2.0(do > we have the timeline for airflow 2.0?) as the warning generates too many > noises for our internal production env as well as the travis CI log. > > >

Re: celery problem: cannot override celery_broker_transport_options

of airflow 1.10 in my environment. Since I have found stability bugs in airflow 1.9.0 that have been fixed in master, I want to try to run airflow 1.10 from git. Thanks. -- Craig On Mon, May 21, 2018 at 1:50 AM Craig Rodrigues <crodr...@gmail.com> wrote: > Hi, > > I used this r

Re: celery problem, AttributeError: async

I have submitted: https://github.com/apache/incubator-airflow/pull/3388 -- Craig On Mon, May 21, 2018 at 7:00 AM Naik Kaxil <k.n...@reply.com> wrote: > Thanks. Please do that. > > On 21/05/2018, 14:59, "Craig Rodrigues" <crodr...@gmail.com> wrote: > >

Re: celery problem, AttributeError: async

celery 4.1.1 was just released last night which has all the async problems fixed: https://github.com/celery/celery/commits/v4.1.1 I'll test this out, and then submit a PR to bump airflow's celery version to 4.1.1 -- Craig On 2018/05/21 07:20:50, Craig Rodrigues <crodr...@gmail.com>

celery problem: cannot override celery_broker_transport_options

Hi, I used this requirements.txt file to install airflow from the v1-10-test branch: git+https://github.com/celery/celery@master#egg=celery git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3] kombu>=4.1.0

celery problem, AttributeError: async

Hi, I used a requiremens.txt file with these three lines: git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3] celery>=4.2.0rc3 kombu>=4.2.0 I did pip install -r requirements.txt When I started my

Re: v1-10-test branch reports wrong version

See: https://github.com/apache/incubator-airflow/pull/3389 -- Craig On 2018/05/20 17:16:45, Craig Rodrigues <crodr...@gmail.com> wrote: > I created a requirements.txt with one line: > > git+ > https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[cele

Re: setup.py extras for celery should depend on kombu >= 4.1.0

On 2018/05/20 23:15:43, crodr...@gmail.com wrote: > Hi, > > I had a problem where if I had this in my airflow.cfg: > broker_url = sqla+mysql://airflow:password@hostname:3306/database > > > and kombu 4.0.2 was installed, then the sqla URL did not work, and celery did

v1-10-test branch reports wrong version

I created a requirements.txt with one line: git+ https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3] I then did: 1. create a virtual environment 2. pip install -r requirements.txt 3. airflow webserver When