Need to run the scheduler also right? -- Carl Johan Gustavsson
On 13 July 2018 at 15:37:22, Driesprong, Fokko (fo...@driesprong.frl) wrote: I've installed Airflow, but using a clean install and the sequential executor, the tasks are not being picked up. This is easy to replicate: docker run -t -i --rm -p 8080:8080 python:3.5 bash pip install https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/apache-airflow-1.10.0rc1+incubating-bin.tar.gz airflow initdb airflow webserver I've enabled the http example dag and kicked of a dag, but it isn't being picked up by the SequentialScheduler. - Fokko 2018-07-13 11:51 GMT+02:00 Bolke de Bruin <bdbr...@gmail.com>: > Hi Sid, > > Do you have a JIRA and a PR to address it? I can then consider it for RC2 > > B. > > > On 12 Jul 2018, at 04:50, Sid Anand <san...@apache.org> wrote: > > > > FYI! > > I just installed the release candidate. The first thing I noticed is a > missing tool tip for the Null State in the Recent Tasks column on the > landing page. Since the null globe is new to this UI, users will likely > hover over it to inquire what it means... and will be left wanting. Of > course, they could click on the globe, which will take them to > http://localhost:8080/admin/taskinstance/?flt1_dag_id_ > equals=example_bash_operator&flt2_state_equals=null < > http://localhost:8080/admin/taskinstance/?flt1_dag_id_ > equals=example_bash_operator&flt2_state_equals=null>, which will always > show an empty list, leaving them a bit more confused. > > > > -s > > > > On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson < > carl.jo...@tictail.com.invalid> wrote: > > Hi Bolke, > > > > (Switching email to avoid moderation on my emails.) > > > > The normal Airflow test suite does not fail as it uses a LC_ALL set to > > utf-8. > > > > I think it is a proper test though, it is a minimal reproducible version > of > > the code that fails. And the only difference in behaviour is at 3.7 which > > we don’t support anyway so I’m fairly sure it is broken for all supported > > Python 3 versions. > > > > I now tried running the tests in docker using 3.5 with the LC_ALL/LANG > > unset and I see the same failure. > > > > I don’t think this is a big thing though and we could release it without > > the fix I made. I think most people run it with a sane LC_ALL, but > > apparently we didn’t. > > Here’s the log for the test: > > > > > docker run -t -i -v `pwd`:/airflow/ python:3.5 bash > > root@b99b297df111:/# locale > > LANG=C.UTF-8 > > LANGUAGE= > > LC_CTYPE="C.UTF-8" > > LC_NUMERIC="C.UTF-8" > > LC_TIME="C.UTF-8" > > LC_COLLATE="C.UTF-8" > > LC_MONETARY="C.UTF-8" > > LC_MESSAGES="C.UTF-8" > > LC_PAPER="C.UTF-8" > > LC_NAME="C.UTF-8" > > LC_ADDRESS="C.UTF-8" > > LC_TELEPHONE="C.UTF-8" > > LC_MEASUREMENT="C.UTF-8" > > LC_IDENTIFICATION="C.UTF-8" > > LC_ALL= > > > unset LANG > > root@b99b297df111:/# locale > > LANG= > > LANGUAGE= > > LC_CTYPE="POSIX" > > LC_NUMERIC="POSIX" > > LC_TIME="POSIX" > > LC_COLLATE="POSIX" > > LC_MONETARY="POSIX" > > LC_MESSAGES="POSIX" > > LC_PAPER="POSIX" > > LC_NAME="POSIX" > > LC_ADDRESS="POSIX" > > LC_TELEPHONE="POSIX" > > LC_MEASUREMENT="POSIX" > > LC_IDENTIFICATION="POSIX" > > LC_ALL= > > root@b99b297df111:/# pip install -e .[devel] > > root@b99b297df111:/airflow# ./run_unit_tests.sh > > + export AIRFLOW_HOME=/root/airflow > > + AIRFLOW_HOME=/root/airflow > > + export AIRFLOW__CORE__UNIT_TEST_MODE=True > > + AIRFLOW__CORE__UNIT_TEST_MODE=True > > + export AIRFLOW__TESTSECTION__TESTKEY=testvalue > > + AIRFLOW__TESTSECTION__TESTKEY=testvalue > > + export AIRFLOW_USE_NEW_IMPORTS=1 > > + AIRFLOW_USE_NEW_IMPORTS=1 > > +++ dirname ./run_unit_tests.sh > > ++ cd . > > ++ pwd > > + DIR=/airflow > > + export PYTHONPATH=:/airflow/tests/test_utils > > + PYTHONPATH=:/airflow/tests/test_utils > > + nose_args= > > + which airflow > > + echo 'Initializing the DB' > > Initializing the DB > > + airflow resetdb > > + yes > > Traceback (most recent call last): > > File "/usr/local/bin/airflow", line 6, in <module> > > exec(compile(open(__file__).read(), __file__, 'exec')) > > File "/airflow/airflow/bin/airflow", line 21, in <module> > > from airflow import configuration > > File "/airflow/airflow/__init__.py", line 35, in <module> > > from airflow import configuration as conf > > File "/airflow/airflow/configuration.py", line 106, in <module> > > DEFAULT_CONFIG = f.read() > > File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode > > return codecs.ascii_decode(input, self.errors)[0] > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position > 21082: > > ordinal not in range(128) > > + '[' '' ']' > > + '[' -z '' ']' > > + nose_args='--with-coverage --cover-erase --cover-html > > --cover-package=airflow --cover-html-dir=airflow/www/static/coverage > > --with-ignore-docstrings --rednose --with-timer -s -v > > --logging-level=DEBUG ' > > + echo 'Starting the unit tests with the following nose arguments: > > --with-coverage' --cover-erase --cover-html --cover-package=airflow > > --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings > > --rednose --with-timer -s -v --logging-level=DEBUG > > Starting the unit tests with the following nose arguments: > --with-coverage > > --cover-erase --cover-html --cover-package=airflow > > --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings > > --rednose --with-timer -s -v --logging-level=DEBUG > > + nosetests --with-coverage --cover-erase --cover-html > > --cover-package=airflow --cover-html-dir=airflow/www/static/coverage > > --with-ignore-docstrings --rednose --with-timer -s -v > --logging-level=DEBUG > > nose.plugins.cover: ERROR: Coverage not available: unable to import > > coverage module > > Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in > > position 21082: ordinal not in range(128)) ... ERROR > > Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in > > position 21082: ordinal not in range(128)) ... ERROR > > ====================================================================== > > 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte > 0xe2 > > in position 21082: ordinal not in range(128)) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in > > runTest > > raise self.exc_val.with_traceback(self.tb) > > /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in > > loadTestsFromName > > addr.filename, addr.module) > > /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in > > importFromPath > > return self.importFromDir(dir_path, fqname) > > /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in > > importFromDir > > mod = load_module(part_fqname, fh, filename, desc) > > /usr/local/lib/python3.5/imp.py line 245 in load_module > > return load_package(name, filename) > > /usr/local/lib/python3.5/imp.py line 217 in load_package > > return _load(spec) > > <frozen importlib._bootstrap> line 693 in _load > > > > <frozen importlib._bootstrap> line 673 in _load_unlocked > > > > <frozen importlib._bootstrap_external> line 697 in exec_module > > > > <frozen importlib._bootstrap> line 222 in _call_with_frames_removed > > > > airflow/__init__.py line 35 in <module> > > from airflow import configuration as conf > > airflow/configuration.py line 106 in <module> > > DEFAULT_CONFIG = f.read() > > /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode > > return codecs.ascii_decode(input, self.errors)[0] > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position > > 21082: ordinal not in range(128) > > ====================================================================== > > 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte > 0xe2 > > in position 21082: ordinal not in range(128)) > > ---------------------------------------------------------------------- > > Traceback (most recent call last): > > /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in > > runTest > > raise self.exc_val.with_traceback(self.tb) > > /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in > > loadTestsFromName > > addr.filename, addr.module) > > /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in > > importFromPath > > return self.importFromDir(dir_path, fqname) > > /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in > > importFromDir > > mod = load_module(part_fqname, fh, filename, desc) > > /usr/local/lib/python3.5/imp.py line 245 in load_module > > return load_package(name, filename) > > /usr/local/lib/python3.5/imp.py line 217 in load_package > > return _load(spec) > > <frozen importlib._bootstrap> line 693 in _load > > > > <frozen importlib._bootstrap> line 673 in _load_unlocked > > > > <frozen importlib._bootstrap_external> line 697 in exec_module > > > > <frozen importlib._bootstrap> line 222 in _call_with_frames_removed > > > > tests/__init__.py line 25 in <module> > > from .configuration import * > > tests/configuration.py line 28 in <module> > > from airflow import configuration > > airflow/__init__.py line 35 in <module> > > from airflow import configuration as conf > > airflow/configuration.py line 106 in <module> > > DEFAULT_CONFIG = f.read() > > /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode > > return codecs.ascii_decode(input, self.errors)[0] > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position > > 21082: ordinal not in range(128) > > > > [error] 100.00% nose.failure.Failure.runTest: 0.0004s > > ------------------------------------------------------------ > ----------------- > > 2 tests run in 0.060 seconds. > > 2 errors (0 tests passed) > > > > > > -- > > Carl Johan Gustavsson > > > > On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbr...@gmail.com <mailto: > bdbr...@gmail.com>) wrote: > > > > Hi Carl, > > > > That is not a real check, ie. Does Airflow have the same issue clean > > install on 3.5? Travis’ tests run on 3.5. > > > > B. > > > > Verstuurd vanaf mijn iPad > > > > > Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson < > > carl.j.gustavs...@gmail.com <mailto:carl.j.gustavs...@gmail.com>> het > volgende geschreven: > > > > > > Hi Bolke, > > > > > > > > > I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7 > > breaks > > > > > > Quick repro: > > > > > > ➜ ~ pyenv local 3.5.5 > > > ➜ ~ locale > > > LANG= > > > LC_COLLATE="C" > > > LC_CTYPE="C" > > > LC_MESSAGES="C" > > > LC_MONETARY="C" > > > LC_NUMERIC="C" > > > LC_TIME="C" > > > LC_ALL= > > > ➜ ~ cat testweird.txt > > > ’ > > > ➜ ~ python > > > Python 3.5.5 (default, Jul 7 2018, 17:00:56) > > > [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin > > > Type "help", "copyright", "credits" or "license" for more information. > > > >>> open('testweird.txt').read() > > > Traceback (most recent call last): > > > File "<stdin>", line 1, in <module> > > > File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ > ascii.py", > > line 26, in decode > > > return codecs.ascii_decode(input, self.errors)[0] > > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: > > ordinal not in range(128) > > > >>> > > > > > > > > > Maybe not a blocking change but it is a breaking change from 1.9 I > guess. > > > > > > > > > / Carl Johan > > > > > > > > >> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbr...@gmail.com > <mailto:bdbr...@gmail.com>) wrote: > > >> > > >> Hi Carl, > > >> > > >> Did you this on python 3.5 as well? 3.6 is not an officially supported > > (yet). As a workaround is available I won’t consider this blocking btw. > > >> > > >> Bolke > > >> > > >> Verstuurd vanaf mijn iPad > > >> > > >> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson < > > carl.j.gustavs...@gmail.com <mailto:carl.j.gustavs...@gmail.com>> het > volgende geschreven: > > >> > > >>> Hi, > > >>> > > >>> First of all, thank you for all the work with the release management. > > >>> > > >>> I ran in to a weird issue testing the RC1, running under Python > 3.6.0 / > > Ubuntu 14.04.5, upgrading from a master build from February. > > >>> > > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback > > (most recent call last): > > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File > > "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module> > > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from > airflow > > import configuration > > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File > > "/opt/virtualenv/tictail/pipeline/lib/python3.6/site- > packages/airflow/__init__.py", > > line 35, in <module> > > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from > airflow > > import configuration as conf > > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File > > "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/ > configuration.py", > > line 106, in <module> > > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 > > DEFAULT_CONFIG = f.read() > > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File > > "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", > line > > 26, in decode > > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return > > codecs.ascii_decode(input, self.errors)[0] > > >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position > 20770: > > ordinal not in range(128) > > >>> > > >>> Removing the ’ on > > https://github.com/apache/incubator-airflow/blob/master/ > airflow/config_templates/default_airflow.cfg#L613 < > https://github.com/apache/incubator-airflow/blob/master/ > airflow/config_templates/default_airflow.cfg#L613> > > solved the issue for me, and digging a bit deeper it seems Airflow now > > requires setting LC_ALL=en_US.UTF-8 in the environment or similar to > force > > Python to read the file as utf-8 and not ascii. (I think this was changed > > in to default to utf-8 in Python 3.7). > > >>> > > >>> I see 3 solutions for this > > >>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or > > similar. > > >>> 2. Change the default config file to not contain non-ascii > characters. > > >>> 3. Always read the file as unicode regardless of the LC_ALL > > environment, by the encoding='utf-8’ parameter to open(). > > >>> > > >>> I think 3 is the best solution, and I can prepare a PR for that if > > necessary . > > >>> > > >>> I guess this counts as -1 (non-binding) > > >>> > > >>> All the best > > >>> > > >>> Carl Johan > > >>> > > >>> > > >>> > > >>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbr...@gmail.com > <mailto:bdbr...@gmail.com>) wrote: > > >>>> > > >>>> Hey all, > > >>>> > > >>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the > > release, > > >>>> which will last for 72 hours. Consider this my (binding) +1. > > >>>> > > >>>> Airflow 1.10.0 RC 1 is available at: > > >>>> > > >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ > <https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/> < > > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ < > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>> > > >>>> > > >>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source > release > > that > > >>>> comes with INSTALL instructions. > > >>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python > > "sdist" > > >>>> release. > > >>>> > > >>>> Public keys are available at: > > >>>> > > >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ < > https://dist.apache.org/repos/dist/release/incubator/airflow/> < > > https://dist.apache.org/repos/dist/release/incubator/airflow/ < > https://dist.apache.org/repos/dist/release/incubator/airflow/>> > > >>>> > > >>>> The amount of JIRAs fixed is over 700. Please have a look at the > > changelog. > > >>>> > > >>>> Please note that the version number excludes the `rcX` string as > well > > >>>> as the "+incubating" string, so it's now simply 1.10.0. This will > > allow us > > >>>> to rename the artifact without modifying the artifact checksums when > > we > > >>>> actually release. > > >>>> > > >>>> > > >>>> Cheers, > > >>>> Bolke > >
