I’ll have a look at it, and if needed release Beta 4. - Bolke
> On 26 Jan 2017, at 10:53, Alex Van Boxel <a...@vanboxel.be> wrote: > > About the database error: starting from scratch also gives the same error: > > Fresh install. Delete airflow.db sqllite db. And the : airflow initdb > > same error as above. > > On Thu, Jan 26, 2017 at 10:12 AM Alex Van Boxel <a...@vanboxel.be> wrote: > >> Not directly one I can share. I'll spend some time looking at it. I'll try >> to create a unittest of it. >> >> On Thu, Jan 26, 2017 at 9:47 AM Bolke de Bruin <bdbr...@gmail.com> wrote: >> >> Do you have a example dag for this? That makes it easier to work on. >> >> Bolke >> >> Sent from my iPhone >> >>> On 26 Jan 2017, at 08:36, Alex Van Boxel <a...@vanboxel.be> wrote: >>> >>> Another thing that I noticed (but observed it in beta 2 as well). Is the >>> following: >>> >>> - The following trigger should not fire. >>> --- Trigger rule is ONE_SUCCESS >>> --- upstream: UP_FOR_RETRY + SKIPPED >>> => task get's triggered >>> => resulting SKIPPED >>> => DAG marked success, with half of the Tasks never scheduled >>> >>> - UP_FOR_RETRY is propagated downstream (actually resulting in failure >>> described above) >>> --- *Does this make sense* ?! >>> >>> Both are a problem, and certainly the combination of both. I'll see if I >>> can spend some time investigating this. >>> >>> >>> >>>> On Thu, Jan 26, 2017 at 7:34 AM Bolke de Bruin <bdbr...@gmail.com> >> wrote: >>>> >>>> Mmm that is due to the reverting of one changes to the db. Need to look >>>> into that how to fix it. >>>> >>>> Sent from my iPhone >>>> >>>>> On 26 Jan 2017, at 00:51, Alex Van Boxel <a...@vanboxel.be> wrote: >>>>> >>>>> I do seem to have a problem upgrading the MySQL database with the last >>>>> commit: >>>>> >>>>> >>>>> 2017-01-25T23:41:55.662572654Z >>>>> >>>> >> /usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/messaging.py:69: >>>>> UserWarning: Revision 1a5a9e6bf2b5 referenced from 1a5a9e6bf2b5 -> >>>>> 127d2bf2dfa7 (head), Add dag_id/state index on dag_run table is not >>>> present >>>>> 2017-01-25T23:41:55.662613670Z warnings.warn(msg) >>>>> 2017-01-25T23:41:55.664560884Z Traceback (most recent call last): >>>>> 2017-01-25T23:41:55.664582238Z File "/usr/local/bin/airflow", line 4, >>>> in >>>>> <module> >>>>> 2017-01-25T23:41:55.664758457Z >>>>> >>>> >> __import__('pkg_resources').run_script('airflow==1.8.0b1+apache.incubating', >>>>> 'airflow') >>>>> 2017-01-25T23:41:55.664776553Z File >>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", >> line >>>>> 739, in run_script >>>>> 2017-01-25T23:41:55.664937644Z >>>>> self.require(requires)[0].run_script(script_name, ns) >>>>> 2017-01-25T23:41:55.664952001Z File >>>>> "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", >> line >>>>> 1494, in run_script >>>>> 2017-01-25T23:41:55.665072840Z exec(code, namespace, namespace) >>>>> 2017-01-25T23:41:55.665086884Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow", >>>>> line 28, in <module> >>>>> 2017-01-25T23:41:55.665211755Z args.func(args) >>>>> 2017-01-25T23:41:55.665225157Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/bin/cli.py", >>>>> line 931, in upgradedb >>>>> 2017-01-25T23:41:55.665326691Z db_utils.upgradedb() >>>>> 2017-01-25T23:41:55.665346979Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/utils/db.py", >>>>> line 292, in upgradedb >>>>> 2017-01-25T23:41:55.665449297Z command.upgrade(config, 'heads') >>>>> 2017-01-25T23:41:55.665462333Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py", >>>>> line 174, in upgrade >>>>> 2017-01-25T23:41:55.665521466Z script.run_env() >>>>> 2017-01-25T23:41:55.665527232Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py", >>>>> line 416, in run_env2017-01-25T23:41:55.665638879Z >>>>> util.load_python_file(self.dir, 'env.py') >>>>> 2017-01-25T23:41:55.665656437Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/pyfiles.py", >>>>> line 93, in load_python_file >>>>> 2017-01-25T23:41:55.665682760Z module = load_module_py(module_id, >>>> path) >>>>> 2017-01-25T23:41:55.665699257Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/compat.py", >>>>> line 79, in load_module_py >>>>> 2017-01-25T23:41:55.665725036Z mod = imp.load_source(module_id, >> path, >>>>> fp) >>>>> 2017-01-25T23:41:55.665751433Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py", >>>>> line 88, in <module> >>>>> 2017-01-25T23:41:55.665778219Z run_migrations_online() >>>>> 2017-01-25T23:41:55.665787820Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b1+apache.incubating-py2.7.egg/airflow/migrations/env.py", >>>>> line 83, in run_migrations_online >>>>> 2017-01-25T23:41:55.665811353Z context.run_migrations() >>>>> 2017-01-25T23:41:55.665818933Z File "<string>", line 8, in >>>> run_migrations >>>>> 2017-01-25T23:41:55.666707547Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/environment.py", >>>>> line 797, in run_migrations >>>>> 2017-01-25T23:41:55.666839945Z >>>> self.get_context().run_migrations(**kw) >>>>> 2017-01-25T23:41:55.666859336Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/runtime/migration.py", >>>>> line 305, in run_migrations >>>>> 2017-01-25T23:41:55.666970791Z for step in >> self._migrations_fn(heads, >>>>> self): >>>>> 2017-01-25T23:41:55.666992265Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/command.py", >>>>> line 163, in upgrade >>>>> 2017-01-25T23:41:55.667021804Z return >> script._upgrade_revs(revision, >>>>> rev) >>>>> 2017-01-25T23:41:55.667032496Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/base.py", >>>>> line 329, in _upgrade_revs >>>>> 2017-01-25T23:41:55.667144295Z revs = list(revs) >>>>> 2017-01-25T23:41:55.667157407Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py", >>>>> line 641, in _iterate_revisions >>>>> 2017-01-25T23:41:55.667302027Z requested_lowers = >>>>> self.get_revisions(lower) >>>>> 2017-01-25T23:41:55.667315913Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py", >>>>> line 298, in get_revisions >>>>> 2017-01-25T23:41:55.667390590Z return >>>> sum([self.get_revisions(id_elem) >>>>> for id_elem in id_], ()) >>>>> 2017-01-25T23:41:55.667411867Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py", >>>>> line 300, in get_revisions >>>>> 2017-01-25T23:41:55.667419226Z resolved_id, branch_label = >>>>> self._resolve_revision_number(id_) >>>>> 2017-01-25T23:41:55.667434110Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py", >>>>> line 433, in _resolve_revision_number >>>>> 2017-01-25T23:41:55.667576273Z self._revision_map >>>>> 2017-01-25T23:41:55.667590641Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/util/langhelpers.py", >>>>> line 240, in __get__ >>>>> 2017-01-25T23:41:55.667648561Z obj.__dict__[self.__name__] = >> result = >>>>> self.fget(obj) >>>>> 2017-01-25T23:41:55.667658598Z File >>>>> >>>> >> "/usr/local/lib/python2.7/site-packages/alembic-0.8.9-py2.7.egg/alembic/script/revision.py", >>>>> line 151, in _revision_map >>>>> 2017-01-25T23:41:55.667680311Z down_revision = map_[downrev] >>>>> 2017-01-25T23:41:55.667746364Z KeyError: '1a5a9e6bf2b5' >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> On Wed, Jan 25, 2017 at 11:15 PM Bolke de Bruin <bdbr...@gmail.com> >>>> wrote: >>>>>> >>>>>> Hi All, >>>>>> >>>>>> I have made the THIRD beta of Airflow 1.8.0 available at: >>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/ < >>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/> , public >>>> keys >>>>>> are available at >>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ < >>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/> . It >> is >>>>>> tagged with a local version “apache.incubating” so it allows upgrading >>>> from >>>>>> earlier releases. This beta is available for testing in a more >>>> production >>>>>> like setting (acceptance environment?). >>>>>> >>>>>> I would like to encourage everyone to try it out, to report back any >>>>>> issues so we get to a rock solid release of 1.8.0. When reporting >>>> issues a >>>>>> test case or even a fix is highly appreciated. >>>>>> >>>>>> Issues cleared: >>>>>> >>>>>> * Manual trigger not working >>>>>> * Performance issues with MySQL >>>>>> * Postgres auto-commit support left over to the driver >>>>>> * Poison pill taken while task has exited >>>>>> * Keep cgroups optional >>>>>> * Funcsigs pinned to 1.0.0 >>>>>> >>>>>> Issue(s) remaining (blocker for RC): >>>>>> * Cgroups not py3 compatible >>>>>> >>>>>> If all goes well we should have a Release Candidate on Feb 2. Thanks >> for >>>>>> reporting issues and keep on testing please :). Moving towards RC I >>>> tend to >>>>>> like small bug fixes only. When we mark RC (do we need to vote on >> this?) >>>>>> the procedure becomes even more strict. Please remember that the FINAL >>>>>> release is dependent on a vote on the IPMC mailinglist. >>>>>> >>>>>> Cheers >>>>>> Bolke >>>>> >>>>> -- >>>>> _/ >>>>> _/ Alex Van Boxel >>>> >>> -- >>> _/ >>> _/ Alex Van Boxel >> >> -- >> _/ >> _/ Alex Van Boxel >> > -- > _/ > _/ Alex Van Boxel