Sid,
The behavior in your first point is odd -- it should print a deprecation
warning about 2.0 but still work. I just tested it on master and it works
(see output below), so unless something unusual happened in the 1.8 build
it should work there as well:
In [1]: from airflow.operators import PigOperator
[2017-02-07 15:24:17,551] {__init__.py:57} INFO - Using executor
SequentialExecutor
/Users/jlowin/git/airflow/airflow/utils/helpers.py:320: DeprecationWarning:
Importing PigOperator directly from <module 'airflow.operators' from
'/Users/jlowin/git/airflow/airflow/operators/__init__.py'> has been
deprecated. Please import from '<module 'airflow.operators' from
'/Users/jlowin/git/airflow/airflow/operators/__init__.py'>.[operator_module]'
instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
In [2]: PigOperator
Out[2]: pig_operator.PigOperator
On Mon, Feb 6, 2017 at 9:50 PM siddharth anand <san...@apache.org> wrote:
> I did get 1.8.0 installed and running at Agari.
>
> I did run into 2 problems.
> 1. Most of our DAGs broke due the way Operators are now imported.
>
> https://github.com/apache/incubator-airflow/blob/master/UPDATING.md#deprecated-features
>
> According to the documentation, these deprecations would only cause an
> issue in 2.0. However, I needed to fix them now.
>
> So, I needed to change "from airflow.operators import PythonOperator" to
> from "from airflow.operators.python_operator import PythonOperator". Am I
> missing something?
>
> 2. I ran into a migration problem that seems to have cleared itself up. I
> did notice that some dags do not have data in their "DAG Runs" column on
> the overview page computed. I am looking into that issue presently.
>
> https://www.dropbox.com/s/cn058mtu3vcv8sq/Screenshot%202017-02-06%2018.45.07.png?dl=0
>
> -s
>
> On Mon, Feb 6, 2017 at 4:30 PM, Dan Davydov <dan.davy...@airbnb.com
> .invalid>
> wrote:
>
> > Bolke, attached is the patch for the cgroups fix. Let me know which
> > branches you would like me to merge it to. If anyone has complaints about
> > the patch let me know (but it does not touch the core of airflow, only
> the
> > new cgroups task runner).
> >
> > On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand <san...@apache.org>
> wrote:
> >
> >> Actually, I see the error is further down..
> >>
> >> File
> >> "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py",
> >> line
> >> 469, in do_execute
> >>
> >> cursor.execute(statement, parameters)
> >>
> >> sqlalchemy.exc.IntegrityError: (psycopg2.IntegrityError) null value in
> >> column "dag_id" violates not-null constraint
> >>
> >> DETAIL: Failing row contains (null, running, 1, f).
> >>
> >> [SQL: 'INSERT INTO dag_stats (state, count, dirty) VALUES (%(state)s,
> >> %(count)s, %(dirty)s)'] [parameters: {'count': 1L, 'state': u'running',
> >> 'dirty': False}]
> >>
> >> It looks like an autoincrement is missing for this table.
> >>
> >>
> >> I'm running `SQLAlchemy==1.1.4` - I see our setup.py specifies any
> version
> >> greater than 0.9.8
> >>
> >> -s
> >>
> >>
> >>
> >> On Mon, Feb 6, 2017 at 4:11 PM, siddharth anand <san...@apache.org>
> >> wrote:
> >>
> >> > I tried upgrading to 1.8.0rc1 from 1.7.1.3 via pip install
> >> > https://dist.apache.org/repos/dist/dev/incubator/airflow/
> >> > airflow-1.8.0rc1+apache.incubating.tar.gz and then running airflow
> >> > upgradedb didn't quite work. First, I thought it completed
> successfully,
> >> > then saw errors some tables were indeed missing. I ran it again and
> >> > encountered the following exception :
> >> >
> >> > DB: postgresql://app_coust...@db-cousteau.ep.stage.agari.com:543
> >> 2/airflow
> >> >
> >> > [2017-02-07 00:03:20,309] {db.py:284} INFO - Creating tables
> >> >
> >> > INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
> >> >
> >> > INFO [alembic.runtime.migration] Will assume transactional DDL.
> >> >
> >> > INFO [alembic.runtime.migration] Running upgrade 2e82aab8ef20 ->
> >> > 211e584da130, add TI state index
> >> >
> >> > INFO [alembic.runtime.migration] Running upgrade 211e584da130 ->
> >> > 64de9cddf6c9, add task fails journal table
> >> >
> >> > INFO [alembic.runtime.migration] Running upgrade 64de9cddf6c9 ->
> >> > f2ca10b85618, add dag_stats table
> >> >
> >> > INFO [alembic.runtime.migration] Running upgrade f2ca10b85618 ->
> >> > 4addfa1236f1, Add fractional seconds to mysql tables
> >> >
> >> > INFO [alembic.runtime.migration] Running upgrade 4addfa1236f1 ->
> >> > 8504051e801b, xcom dag task indices
> >> >
> >> > INFO [alembic.runtime.migration] Running upgrade 8504051e801b ->
> >> > 5e7d17757c7a, add pid field to TaskInstance
> >> >
> >> > INFO [alembic.runtime.migration] Running upgrade 5e7d17757c7a ->
> >> > 127d2bf2dfa7, Add dag_id/state index on dag_run table
> >> >
> >> > /usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/crud.py:692:
> >> > SAWarning: Column 'dag_stats.dag_id' is marked as a member of the
> >> primary
> >> > key for table 'dag_stats', but has no Python-side or server-side
> default
> >> > generator indicated, nor does it indicate 'autoincrement=True' or
> >> > 'nullable=True', and no explicit value is passed. Primary key columns
> >> > typically may not store NULL. Note that as of SQLAlchemy 1.1,
> >> > 'autoincrement=True' must be indicated explicitly for composite (e.g.
> >> > multicolumn) primary keys if AUTO_INCREMENT/SERIAL/IDENTITY behavior
> is
> >> > expected for one of the columns in the primary key. CREATE TABLE
> >> statements
> >> > are impacted by this change as well on most backends.
> >> >
> >>
> >
> >
>
