Example dags are not picked up. If you put a dag in the normal dag folder it works fine.
Please create a jira for this @fokko. A pr would be appreciated. B. Sent from my iPhone > On 13 Jul 2018, at 15:46, Driesprong, Fokko <fo...@driesprong.frl> wrote: > > With the SequentialExecutor the webserver also acts as the scheduler > (without parallelism) > > 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson <carl.jo...@tictail.com>: > >> Need to run the scheduler also right? >> >> -- >> Carl Johan Gustavsson >> >> On 13 July 2018 at 15:37:22, Driesprong, Fokko (fo...@driesprong.frl) >> wrote: >> >> I've installed Airflow, but using a clean install and the sequential >> executor, the tasks are not being picked up. >> >> This is easy to replicate: >> >> docker run -t -i --rm -p 8080:8080 python:3.5 bash >> pip install https://dist.apache.org/repos/dist/dev/incubator/airflow/1. >> 10.0rc1/apache-airflow-1.10.0rc1+incubating-bin.tar.gz >> airflow initdb >> airflow webserver >> >> I've enabled the http example dag and kicked of a dag, but it isn't being >> picked up by the SequentialScheduler. >> >> - Fokko >> >> >> >> >> 2018-07-13 11:51 GMT+02:00 Bolke de Bruin <bdbr...@gmail.com>: >> >>> Hi Sid, >>> >>> Do you have a JIRA and a PR to address it? I can then consider it for RC2 >>> >>> B. >>> >>>> On 12 Jul 2018, at 04:50, Sid Anand <san...@apache.org> wrote: >>>> >>>> FYI! >>>> I just installed the release candidate. The first thing I noticed is a >>> missing tool tip for the Null State in the Recent Tasks column on the >>> landing page. Since the null globe is new to this UI, users will likely >>> hover over it to inquire what it means... and will be left wanting. Of >>> course, they could click on the globe, which will take them to >>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals >>> =example_bash_operator&flt2_state_equals=null < >>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equal >>> s=example_bash_operator&flt2_state_equals=null>, which will always show >>> an empty list, leaving them a bit more confused. >>>> >>>> -s >>>> >>>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson < >>> carl.jo...@tictail.com.invalid> wrote: >>>> Hi Bolke, >>>> >>>> (Switching email to avoid moderation on my emails.) >>>> >>>> The normal Airflow test suite does not fail as it uses a LC_ALL set to >>>> utf-8. >>>> >>>> I think it is a proper test though, it is a minimal reproducible >>> version of >>>> the code that fails. And the only difference in behaviour is at 3.7 >>> which >>>> we don’t support anyway so I’m fairly sure it is broken for all >>> supported >>>> Python 3 versions. >>>> >>>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG >>>> unset and I see the same failure. >>>> >>>> I don’t think this is a big thing though and we could release it without >>>> the fix I made. I think most people run it with a sane LC_ALL, but >>>> apparently we didn’t. >>>> Here’s the log for the test: >>>> >>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash >>>> root@b99b297df111:/# locale >>>> LANG=C.UTF-8 >>>> LANGUAGE= >>>> LC_CTYPE="C.UTF-8" >>>> LC_NUMERIC="C.UTF-8" >>>> LC_TIME="C.UTF-8" >>>> LC_COLLATE="C.UTF-8" >>>> LC_MONETARY="C.UTF-8" >>>> LC_MESSAGES="C.UTF-8" >>>> LC_PAPER="C.UTF-8" >>>> LC_NAME="C.UTF-8" >>>> LC_ADDRESS="C.UTF-8" >>>> LC_TELEPHONE="C.UTF-8" >>>> LC_MEASUREMENT="C.UTF-8" >>>> LC_IDENTIFICATION="C.UTF-8" >>>> LC_ALL= >>>>> unset LANG >>>> root@b99b297df111:/# locale >>>> LANG= >>>> LANGUAGE= >>>> LC_CTYPE="POSIX" >>>> LC_NUMERIC="POSIX" >>>> LC_TIME="POSIX" >>>> LC_COLLATE="POSIX" >>>> LC_MONETARY="POSIX" >>>> LC_MESSAGES="POSIX" >>>> LC_PAPER="POSIX" >>>> LC_NAME="POSIX" >>>> LC_ADDRESS="POSIX" >>>> LC_TELEPHONE="POSIX" >>>> LC_MEASUREMENT="POSIX" >>>> LC_IDENTIFICATION="POSIX" >>>> LC_ALL= >>>> root@b99b297df111:/# pip install -e .[devel] >>>> root@b99b297df111:/airflow# ./run_unit_tests.sh >>>> + export AIRFLOW_HOME=/root/airflow >>>> + AIRFLOW_HOME=/root/airflow >>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True >>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True >>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue >>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue >>>> + export AIRFLOW_USE_NEW_IMPORTS=1 >>>> + AIRFLOW_USE_NEW_IMPORTS=1 >>>> +++ dirname ./run_unit_tests.sh >>>> ++ cd . >>>> ++ pwd >>>> + DIR=/airflow >>>> + export PYTHONPATH=:/airflow/tests/test_utils >>>> + PYTHONPATH=:/airflow/tests/test_utils >>>> + nose_args= >>>> + which airflow >>>> + echo 'Initializing the DB' >>>> Initializing the DB >>>> + airflow resetdb >>>> + yes >>>> Traceback (most recent call last): >>>> File "/usr/local/bin/airflow", line 6, in <module> >>>> exec(compile(open(__file__).read(), __file__, 'exec')) >>>> File "/airflow/airflow/bin/airflow", line 21, in <module> >>>> from airflow import configuration >>>> File "/airflow/airflow/__init__.py", line 35, in <module> >>>> from airflow import configuration as conf >>>> File "/airflow/airflow/configuration.py", line 106, in <module> >>>> DEFAULT_CONFIG = f.read() >>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in >>> decode >>>> return codecs.ascii_decode(input, self.errors)[0] >>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position >>> 21082: >>>> ordinal not in range(128) >>>> + '[' '' ']' >>>> + '[' -z '' ']' >>>> + nose_args='--with-coverage --cover-erase --cover-html >>>> --cover-package=airflow --cover-html-dir=airflow/www/ >>> static/coverage >>>> --with-ignore-docstrings --rednose --with-timer -s -v >>>> --logging-level=DEBUG ' >>>> + echo 'Starting the unit tests with the following nose arguments: >>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow >>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings >>>> --rednose --with-timer -s -v --logging-level=DEBUG >>>> Starting the unit tests with the following nose arguments: >>> --with-coverage >>>> --cover-erase --cover-html --cover-package=airflow >>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings >>>> --rednose --with-timer -s -v --logging-level=DEBUG >>>> + nosetests --with-coverage --cover-erase --cover-html >>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage >>>> --with-ignore-docstrings --rednose --with-timer -s -v >>> --logging-level=DEBUG >>>> nose.plugins.cover: ERROR: Coverage not available: unable to import >>>> coverage module >>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in >>>> position 21082: ordinal not in range(128)) ... ERROR >>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in >>>> position 21082: ordinal not in range(128)) ... ERROR >>>> ====================================================================== >>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte >>> 0xe2 >>>> in position 21082: ordinal not in range(128)) >>>> ---------------------------------------------------------------------- >>>> Traceback (most recent call last): >>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in >>>> runTest >>>> raise self.exc_val.with_traceback(self.tb) >>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in >>>> loadTestsFromName >>>> addr.filename, addr.module) >>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in >>>> importFromPath >>>> return self.importFromDir(dir_path, fqname) >>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in >>>> importFromDir >>>> mod = load_module(part_fqname, fh, filename, desc) >>>> /usr/local/lib/python3.5/imp.py line 245 in load_module >>>> return load_package(name, filename) >>>> /usr/local/lib/python3.5/imp.py line 217 in load_package >>>> return _load(spec) >>>> <frozen importlib._bootstrap> line 693 in _load >>>> >>>> <frozen importlib._bootstrap> line 673 in _load_unlocked >>>> >>>> <frozen importlib._bootstrap_external> line 697 in exec_module >>>> >>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed >>>> >>>> airflow/__init__.py line 35 in <module> >>>> from airflow import configuration as conf >>>> airflow/configuration.py line 106 in <module> >>>> DEFAULT_CONFIG = f.read() >>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode >>>> return codecs.ascii_decode(input, self.errors)[0] >>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position >>>> 21082: ordinal not in range(128) >>>> ====================================================================== >>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte >>> 0xe2 >>>> in position 21082: ordinal not in range(128)) >>>> ---------------------------------------------------------------------- >>>> Traceback (most recent call last): >>>> /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in >>>> runTest >>>> raise self.exc_val.with_traceback(self.tb) >>>> /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in >>>> loadTestsFromName >>>> addr.filename, addr.module) >>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in >>>> importFromPath >>>> return self.importFromDir(dir_path, fqname) >>>> /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in >>>> importFromDir >>>> mod = load_module(part_fqname, fh, filename, desc) >>>> /usr/local/lib/python3.5/imp.py line 245 in load_module >>>> return load_package(name, filename) >>>> /usr/local/lib/python3.5/imp.py line 217 in load_package >>>> return _load(spec) >>>> <frozen importlib._bootstrap> line 693 in _load >>>> >>>> <frozen importlib._bootstrap> line 673 in _load_unlocked >>>> >>>> <frozen importlib._bootstrap_external> line 697 in exec_module >>>> >>>> <frozen importlib._bootstrap> line 222 in _call_with_frames_removed >>>> >>>> tests/__init__.py line 25 in <module> >>>> from .configuration import * >>>> tests/configuration.py line 28 in <module> >>>> from airflow import configuration >>>> airflow/__init__.py line 35 in <module> >>>> from airflow import configuration as conf >>>> airflow/configuration.py line 106 in <module> >>>> DEFAULT_CONFIG = f.read() >>>> /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode >>>> return codecs.ascii_decode(input, self.errors)[0] >>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position >>>> 21082: ordinal not in range(128) >>>> >>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s >>>> ------------------------------------------------------------ >>> ----------------- >>>> 2 tests run in 0.060 seconds. >>>> 2 errors (0 tests passed) >>>> >>>> >>>> -- >>>> Carl Johan Gustavsson >>>> >>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbr...@gmail.com <mailto: >>> bdbr...@gmail.com>) wrote: >>>> >>>> Hi Carl, >>>> >>>> That is not a real check, ie. Does Airflow have the same issue clean >>>> install on 3.5? Travis’ tests run on 3.5. >>>> >>>> B. >>>> >>>> Verstuurd vanaf mijn iPad >>>> >>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson < >>>> carl.j.gustavs...@gmail.com <mailto:carl.j.gustavs...@gmail.com>> het >>> volgende geschreven: >>>>> >>>>> Hi Bolke, >>>>> >>>>> >>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7 >>>> breaks >>>>> >>>>> Quick repro: >>>>> >>>>> ➜ ~ pyenv local 3.5.5 >>>>> ➜ ~ locale >>>>> LANG= >>>>> LC_COLLATE="C" >>>>> LC_CTYPE="C" >>>>> LC_MESSAGES="C" >>>>> LC_MONETARY="C" >>>>> LC_NUMERIC="C" >>>>> LC_TIME="C" >>>>> LC_ALL= >>>>> ➜ ~ cat testweird.txt >>>>> ’ >>>>> ➜ ~ python >>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56) >>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin >>>>> Type "help", "copyright", "credits" or "license" for more information. >>>>>>>> open('testweird.txt').read() >>>>> Traceback (most recent call last): >>>>> File "<stdin>", line 1, in <module> >>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/as >>> cii.py", >>>> line 26, in decode >>>>> return codecs.ascii_decode(input, self.errors)[0] >>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position >>> 0: >>>> ordinal not in range(128) >>>>>>>> >>>>> >>>>> >>>>> Maybe not a blocking change but it is a breaking change from 1.9 I >>> guess. >>>>> >>>>> >>>>> / Carl Johan >>>>> >>>>> >>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbr...@gmail.com >>> <mailto:bdbr...@gmail.com>) wrote: >>>>>> >>>>>> Hi Carl, >>>>>> >>>>>> Did you this on python 3.5 as well? 3.6 is not an officially >>> supported >>>> (yet). As a workaround is available I won’t consider this blocking btw. >>>>>> >>>>>> Bolke >>>>>> >>>>>> Verstuurd vanaf mijn iPad >>>>>> >>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson < >>>> carl.j.gustavs...@gmail.com <mailto:carl.j.gustavs...@gmail.com>> het >>> volgende geschreven: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> First of all, thank you for all the work with the release >>> management. >>>>>>> >>>>>>> I ran in to a weird issue testing the RC1, running under Python >>> 3.6.0 / >>>> Ubuntu 14.04.5, upgrading from a master build from February. >>>>>>> >>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback >>>> (most recent call last): >>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File >>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module> >>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from >>> airflow >>>> import configuration >>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File >>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-package >>> s/airflow/__init__.py", >>>> line 35, in <module> >>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from >>> airflow >>>> import configuration as conf >>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File >>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-package >>> s/airflow/configuration.py", >>>> line 106, in <module> >>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 >>>> DEFAULT_CONFIG = f.read() >>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File >>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", >>> line >>>> 26, in decode >>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return >>>> codecs.ascii_decode(input, self.errors)[0] >>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 >>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position >>> 20770: >>>> ordinal not in range(128) >>>>>>> >>>>>>> Removing the ’ on >>>> https://github.com/apache/incubator-airflow/blob/master/airf >>> low/config_templates/default_airflow.cfg#L613 < >>> https://github.com/apache/incubator-airflow/blob/master/air >>> flow/config_templates/default_airflow.cfg#L613> >>>> solved the issue for me, and digging a bit deeper it seems Airflow now >>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to >>> force >>>> Python to read the file as utf-8 and not ascii. (I think this was >>> changed >>>> in to default to utf-8 in Python 3.7). >>>>>>> >>>>>>> I see 3 solutions for this >>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or >>>> similar. >>>>>>> 2. Change the default config file to not contain non-ascii >>> characters. >>>>>>> 3. Always read the file as unicode regardless of the LC_ALL >>>> environment, by the encoding='utf-8’ parameter to open(). >>>>>>> >>>>>>> I think 3 is the best solution, and I can prepare a PR for that if >>>> necessary . >>>>>>> >>>>>>> I guess this counts as -1 (non-binding) >>>>>>> >>>>>>> All the best >>>>>>> >>>>>>> Carl Johan >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbr...@gmail.com >>> <mailto:bdbr...@gmail.com>) wrote: >>>>>>>> >>>>>>>> Hey all, >>>>>>>> >>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the >>>> release, >>>>>>>> which will last for 72 hours. Consider this my (binding) +1. >>>>>>>> >>>>>>>> Airflow 1.10.0 RC 1 is available at: >>>>>>>> >>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.1 >>> 0.0rc1/ <https://dist.apache.org/repos/dist/dev/incubator/airflow/1. >>> 10.0rc1/> < >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ < >>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>> >>>>>>>> >>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source >>> release >>>> that >>>>>>>> comes with INSTALL instructions. >>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary >>> Python >>>> "sdist" >>>>>>>> release. >>>>>>>> >>>>>>>> Public keys are available at: >>>>>>>> >>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ < >>> https://dist.apache.org/repos/dist/release/incubator/airflow/> < >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ < >>> https://dist.apache.org/repos/dist/release/incubator/airflow/>> >>>>>>>> >>>>>>>> The amount of JIRAs fixed is over 700. Please have a look at the >>>> changelog. >>>>>>>> >>>>>>>> Please note that the version number excludes the `rcX` string as >>> well >>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This will >>>> allow us >>>>>>>> to rename the artifact without modifying the artifact checksums >>> when >>>> we >>>>>>>> actually release. >>>>>>>> >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Bolke >>> >>> >>
