Hi Bolke,
(Switching email to avoid moderation on my emails.)
The normal Airflow test suite does not fail as it uses a LC_ALL set to
utf-8.
I think it is a proper test though, it is a minimal reproducible version of
the code that fails. And the only difference in behaviour is at 3.7 which
we don’t support anyway so I’m fairly sure it is broken for all supported
Python 3 versions.
I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
unset and I see the same failure.
I don’t think this is a big thing though and we could release it without
the fix I made. I think most people run it with a sane LC_ALL, but
apparently we didn’t.
Here’s the log for the test:
> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
root@b99b297df111:/# locale
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_TIME="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_PAPER="C.UTF-8"
LC_NAME="C.UTF-8"
LC_ADDRESS="C.UTF-8"
LC_TELEPHONE="C.UTF-8"
LC_MEASUREMENT="C.UTF-8"
LC_IDENTIFICATION="C.UTF-8"
LC_ALL=
> unset LANG
root@b99b297df111:/# locale
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
root@b99b297df111:/# pip install -e .[devel]
root@b99b297df111:/airflow# ./run_unit_tests.sh
+ export AIRFLOW_HOME=/root/airflow
+ AIRFLOW_HOME=/root/airflow
+ export AIRFLOW__CORE__UNIT_TEST_MODE=True
+ AIRFLOW__CORE__UNIT_TEST_MODE=True
+ export AIRFLOW__TESTSECTION__TESTKEY=testvalue
+ AIRFLOW__TESTSECTION__TESTKEY=testvalue
+ export AIRFLOW_USE_NEW_IMPORTS=1
+ AIRFLOW_USE_NEW_IMPORTS=1
+++ dirname ./run_unit_tests.sh
++ cd .
++ pwd
+ DIR=/airflow
+ export PYTHONPATH=:/airflow/tests/test_utils
+ PYTHONPATH=:/airflow/tests/test_utils
+ nose_args=
+ which airflow
+ echo 'Initializing the DB'
Initializing the DB
+ airflow resetdb
+ yes
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 6, in <module>
exec(compile(open(__file__).read(), __file__, 'exec'))
File "/airflow/airflow/bin/airflow", line 21, in <module>
from airflow import configuration
File "/airflow/airflow/__init__.py", line 35, in <module>
from airflow import configuration as conf
File "/airflow/airflow/configuration.py", line 106, in <module>
DEFAULT_CONFIG = f.read()
File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
ordinal not in range(128)
+ '[' '' ']'
+ '[' -z '' ']'
+ nose_args='--with-coverage --cover-erase --cover-html
--cover-package=airflow --cover-html-dir=airflow/www/static/coverage
--with-ignore-docstrings --rednose --with-timer -s -v
--logging-level=DEBUG '
+ echo 'Starting the unit tests with the following nose arguments:
--with-coverage' --cover-erase --cover-html --cover-package=airflow
--cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
--rednose --with-timer -s -v --logging-level=DEBUG
Starting the unit tests with the following nose arguments: --with-coverage
--cover-erase --cover-html --cover-package=airflow
--cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
--rednose --with-timer -s -v --logging-level=DEBUG
+ nosetests --with-coverage --cover-erase --cover-html
--cover-package=airflow --cover-html-dir=airflow/www/static/coverage
--with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
nose.plugins.cover: ERROR: Coverage not available: unable to import
coverage module
Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
position 21082: ordinal not in range(128)) ... ERROR
Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
position 21082: ordinal not in range(128)) ... ERROR
======================================================================
1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
in position 21082: ordinal not in range(128))
----------------------------------------------------------------------
Traceback (most recent call last):
/usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
runTest
raise self.exc_val.with_traceback(self.tb)
/usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
loadTestsFromName
addr.filename, addr.module)
/usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
importFromPath
return self.importFromDir(dir_path, fqname)
/usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
importFromDir
mod = load_module(part_fqname, fh, filename, desc)
/usr/local/lib/python3.5/imp.py line 245 in load_module
return load_package(name, filename)
/usr/local/lib/python3.5/imp.py line 217 in load_package
return _load(spec)
<frozen importlib._bootstrap> line 693 in _load
<frozen importlib._bootstrap> line 673 in _load_unlocked
<frozen importlib._bootstrap_external> line 697 in exec_module
<frozen importlib._bootstrap> line 222 in _call_with_frames_removed
airflow/__init__.py line 35 in <module>
from airflow import configuration as conf
airflow/configuration.py line 106 in <module>
DEFAULT_CONFIG = f.read()
/usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
21082: ordinal not in range(128)
======================================================================
2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
in position 21082: ordinal not in range(128))
----------------------------------------------------------------------
Traceback (most recent call last):
/usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
runTest
raise self.exc_val.with_traceback(self.tb)
/usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
loadTestsFromName
addr.filename, addr.module)
/usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
importFromPath
return self.importFromDir(dir_path, fqname)
/usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
importFromDir
mod = load_module(part_fqname, fh, filename, desc)
/usr/local/lib/python3.5/imp.py line 245 in load_module
return load_package(name, filename)
/usr/local/lib/python3.5/imp.py line 217 in load_package
return _load(spec)
<frozen importlib._bootstrap> line 693 in _load
<frozen importlib._bootstrap> line 673 in _load_unlocked
<frozen importlib._bootstrap_external> line 697 in exec_module
<frozen importlib._bootstrap> line 222 in _call_with_frames_removed
tests/__init__.py line 25 in <module>
from .configuration import *
tests/configuration.py line 28 in <module>
from airflow import configuration
airflow/__init__.py line 35 in <module>
from airflow import configuration as conf
airflow/configuration.py line 106 in <module>
DEFAULT_CONFIG = f.read()
/usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
21082: ordinal not in range(128)
[error] 100.00% nose.failure.Failure.runTest: 0.0004s
-----------------------------------------------------------------------------
2 tests run in 0.060 seconds.
2 errors (0 tests passed)
--
Carl Johan Gustavsson
On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbr...@gmail.com) wrote:
Hi Carl,
That is not a real check, ie. Does Airflow have the same issue clean
install on 3.5? Travis’ tests run on 3.5.
B.
Verstuurd vanaf mijn iPad
> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
carl.j.gustavs...@gmail.com> het volgende geschreven:
>
> Hi Bolke,
>
>
> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
breaks
>
> Quick repro:
>
> ➜ ~ pyenv local 3.5.5
> ➜ ~ locale
> LANG=
> LC_COLLATE="C"
> LC_CTYPE="C"
> LC_MESSAGES="C"
> LC_MONETARY="C"
> LC_NUMERIC="C"
> LC_TIME="C"
> LC_ALL=
> ➜ ~ cat testweird.txt
> ’
> ➜ ~ python
> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> open('testweird.txt').read()
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
ordinal not in range(128)
> >>>
>
>
> Maybe not a blocking change but it is a breaking change from 1.9 I guess.
>
>
> / Carl Johan
>
>
>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbr...@gmail.com) wrote:
>>
>> Hi Carl,
>>
>> Did you this on python 3.5 as well? 3.6 is not an officially supported
(yet). As a workaround is available I won’t consider this blocking btw.
>>
>> Bolke
>>
>> Verstuurd vanaf mijn iPad
>>
>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
carl.j.gustavs...@gmail.com> het volgende geschreven:
>>
>>> Hi,
>>>
>>> First of all, thank you for all the work with the release management.
>>>
>>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
Ubuntu 14.04.5, upgrading from a master build from February.
>>>
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
(most recent call last):
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
import configuration
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
line 35, in <module>
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
import configuration as conf
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
line 106, in <module>
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
DEFAULT_CONFIG = f.read()
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
"/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
26, in decode
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
codecs.ascii_decode(input, self.errors)[0]
>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
ordinal not in range(128)
>>>
>>> Removing the ’ on
https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
solved the issue for me, and digging a bit deeper it seems Airflow now
requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
Python to read the file as utf-8 and not ascii. (I think this was changed
in to default to utf-8 in Python 3.7).
>>>
>>> I see 3 solutions for this
>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
similar.
>>> 2. Change the default config file to not contain non-ascii characters.
>>> 3. Always read the file as unicode regardless of the LC_ALL
environment, by the encoding='utf-8’ parameter to open().
>>>
>>> I think 3 is the best solution, and I can prepare a PR for that if
necessary .
>>>
>>> I guess this counts as -1 (non-binding)
>>>
>>> All the best
>>>
>>> Carl Johan
>>>
>>>
>>>
>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbr...@gmail.com) wrote:
>>>>
>>>> Hey all,
>>>>
>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
release,
>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>
>>>> Airflow 1.10.0 RC 1 is available at:
>>>>
>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>>>
>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
that
>>>> comes with INSTALL instructions.
>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
"sdist"
>>>> release.
>>>>
>>>> Public keys are available at:
>>>>
>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>
>>>> The amount of JIRAs fixed is over 700. Please have a look at the
changelog.
>>>>
>>>> Please note that the version number excludes the `rcX` string as well
>>>> as the "+incubating" string, so it's now simply 1.10.0. This will
allow us
>>>> to rename the artifact without modifying the artifact checksums when
we
>>>> actually release.
>>>>
>>>>
>>>> Cheers,
>>>> Bolke
