Hey Alex, Thanks for tracking it down. Can you elaborate want went wrong with celery? The lines below do not particularly relate to Celery directly, so I wonder why we are not seeing it with LocalExecutor?
Cheers Bolke > On 7 Feb 2017, at 15:51, Alex Van Boxel <a...@vanboxel.be> wrote: > > I have to give the RC1 a *-1*. I spend hours, or better days to get the RC > running with Celery on our test environment, till I finally found the > commit that killed it: > > e7f6212cae82c3a3a0bc17bbcbc70646f67d02eb > [AIRFLOW-813] Fix unterminated unit tests in SchedulerJobTest > Closes #2032 from fenglu-g/master > > I was always looking at the wrong this, because the commit only changes a > single default parameter from *None to -1* > > I do have the impression I'm the only one running with Celery. Are other > people running with it? > > *I propose* *reverting the commit*. Feng, can you elaborate on this change? > > Change the default back no *None* in cli.py got it finally working: > > 'num_runs': Arg( > ("-n", "--num_runs"), > default=None, type=int, > help="Set the number of runs to execute before exiting"), > > Thanks. > > On Tue, Feb 7, 2017 at 3:49 AM siddharth anand <san...@apache.org> wrote: > > I did get 1.8.0 installed and running at Agari. > > I did run into 2 problems. > 1. Most of our DAGs broke due the way Operators are now imported. > https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features > > According to the documentation, these deprecations would only cause an > issue in 2.0. However, I needed to fix them now. > > So, I needed to change "from airflow.operators import PythonOperator" to > from "from airflow.operators.python_operator import PythonOperator". Am I > missing something? > > 2. I ran into a migration problem that seems to have cleared itself up. I > did notice that some dags do not have data in their "DAG Runs" column on > the overview page computed. I am looking into that issue presently. > https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45.07.png?dl=0 > > -s > > On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov <dan.davy...@airbnb.com.invalid> > wrote: > >> Bolke, attached is the patch for the cgroups fix. Let me know which >> branches you would like me to merge it to. If anyone has complaints about >> the patch let me know (but it does not touch the core of airflow, only the >> new cgroups task runner). >> >> On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand <san...@apache.org> wrote: >> >>> Actually, I see the error is further down.. >>> >>> File >>> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", >>> line >>> 469, in do_execute >>> >>> cursor.execute(statement, parameters) >>> >>> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in >>> column "dag_id" violates not-null constraint >>> >>> DETAIL: Failing row contains (null, running, 1, f). >>> >>> [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s, >>> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running', >>> 'dirty': False}] >>> >>> It looks like an autoincrement is missing for this table. >>> >>> >>> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any > version >>> greater than 0.9.8 >>> >>> -s >>> >>> >>> >>> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand <san...@apache.org> >>> wrote: >>> >>>> I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ >>>> airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow >>>> upgradedb didn't quite work. First, I thought it completed > successfully, >>>> then saw errors some tables were indeed missing. I ran it again and >>>> encountered the following exception : >>>> >>>> DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:543 >>> 2/airflow >>>> >>>> [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables >>>> >>>> INFO [alembic.runtime.migration] Context impl PostgresqlImpl. >>>> >>>> INFO [alembic.runtime.migration] Will assume transactional DDL. >>>> >>>> INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> >>>> 211e584da130, add TI state index >>>> >>>> INFO [alembic.runtime.migration] Running upgrade 211e584da130 -> >>>> 64de9cddf6c9, add task fails journal table >>>> >>>> INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> >>>> f2ca10b85618, add dag_stats table >>>> >>>> INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 -> >>>> 4addfa1236f1, Add fractional seconds to mysql tables >>>> >>>> INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> >>>> 8504051e801b, xcom dag task indices >>>> >>>> INFO [alembic.runtime.migration] Running upgrade 8504051e801b -> >>>> 5e7d17757c7a, add pid field to TaskInstance >>>> >>>> INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> >>>> 127d2bf2dfa7, Add dag_id/state index on dag_run table >>>> >>>> /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692: >>>> SAWarning: Column 'dag_stats.dag_id' is marked as a member of the >>> primary >>>> key for table 'dag_stats', but has no Python-side or server-side > default >>>> generator indicated, nor does it indicate 'autoincrement=True' or >>>> 'nullable=True', and no explicit value is passed. Primary key columns >>>> typically may not store NULL. Note that as of SQLAlchemy 1.1, >>>> 'autoincrement=True' must be indicated explicitly for composite (e.g. >>>> multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior is >>>> expected for one of the columns in the primary key. CREATE TABLE >>> statements >>>> are impacted by this change as well on most backends. >>>> >>> >> >> > > -- > _/ > _/ Alex Van Boxel