[
https://issues.apache.org/jira/browse/BEAM-8787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979785#comment-16979785
]
Tomo Suzuki edited comment on BEAM-8787 at 11/22/19 5:00 PM:
-------------------------------------------------------------
h1. Problem
The problem for my environment was that Python3.6 was missing required module
{{distutils.sysconfig}} and the latest python3-disutils does not support
Python3.6.
h1. Solution
Build and install the python version from the source after installing
libbz2-dev:
{noformat}
suztomo@suxtomo24:/tmp$ sudo apt-get install libbz2-dev # This is needed for
Python3.6's _bz package
suztomo@suxtomo24:/tmp$ git clone --branch v3.6.8
https://github.com/python/cpython.git
...
suztomo@suxtomo24:/tmp$ cd cpython
suztomo@suxtomo24:/tmp/cpython$ git status
Not currently on any branch.
nothing to commit, working tree clean
suztomo@suxtomo24:/tmp/cpython$ git log -1
commit 3c6b436a57893dd1fae4e072768f41a199076252 (HEAD, tag: v3.6.8)
Author: Ned Deily <n...@python.org>
Date: Sun Dec 23 16:37:14 2018 -0500
3.6.8final
suztomo@suxtomo24:/tmp/cpython$ ./configure --prefix=$HOME/local # pick up your
preference
...
suztomo@suxtomo24:/tmp/cpython$ make install
{noformat}
Add the directory to the path with "/bin" appended. In {{~/.bashrc}}:
{noformat}
export PATH=$HOME/local/bin:$PATH
{noformat}
Now disutils.sysconfig module is available for Python3.6:
{noformat}
suztomo@suxtomo24:/tmp/cpython$ python3.6
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Nov 21 2019, 21:11:37)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils import sysconfig
>>>
{noformat}
Now {{:sdks:python:test-suites:tox:py35:setupVirtualenv}} succeeds
{noformat}
suztomo@suxtomo24:~/beam4$ ./gradlew -p sdks/python/test-suites/tox/py35
setupVirtualenv
...
> Task :sdks:python:test-suites:tox:py35:setupVirtualenv
...
BUILD SUCCESSFUL in 5s
{noformat}
h2. testPy36Gcp failrue
The {{:sdks:python:test-suites:tox:py36:testPy36Gcp}} was failing:
https://gist.github.com/suztomo/ebfc110652b8ffaf7fede64276d7a053
It seemed that _bz2 library was missing for the Python3.6. Followed
[Stackoverflow: No module named '_bz2' in
python3|https://stackoverflow.com/questions/20280726/how-to-git-clone-a-specific-tag/24102558].
h2. testPy35Cython failure
{noformat}
./gradlew :sdks:python:test-suites:tox:py35:testPy35Cython
...
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -g
-fdebug-prefix-map=/build/python3.5-ta1Uke/python3.5-3.5.4=.
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m
-I/usr/local/google/home/suztomo/beam4/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/target/.tox-py35-cython/py35-cython/include/python3.5m
-c apache_beam/coders/stream.c -o
build/temp.linux-x86_64-3.5/apache_beam/coders/stream.o
apache_beam/coders/stream.c:17:10: fatal error: Python.h: No such file or
directory
#include "Python.h"
^~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
...
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:tox:py35:testPy35Cython'.
{noformat}
Tried {{sudo apt-get install python3-dev}} but it didn't work. Installing
python3.5 from source code resolved the issue.
{noformat}
suztomo@suxtomo24:/tmp/cpython$ git checkout v3.5.9
suztomo@suxtomo24:/tmp/cpython$ ./configure --prefix=$HOME/local
suztomo@suxtomo24:/tmp/cpython$ make install
suztomo@suxtomo24:/tmp/cpython$ which python3.5
/usr/local/google/home/suztomo/local/bin/python3.5
{noformat}
was (Author: suztomo):
h1. Problem
The problem for my environment was that Python3.6 was missing required module
{{distutils.sysconfig}} and the latest python3-disutils does not support
Python3.6.
h1. Solution
Build Python3.6 from the source:
{noformat}
suztomo@suxtomo24:/tmp$ sudo apt-get install libbz2-dev # This is needed for
Python3.6's _bz package
suztomo@suxtomo24:/tmp$ git clone --branch v3.6.8
https://github.com/python/cpython.git
...
suztomo@suxtomo24:/tmp$ cd cpython
suztomo@suxtomo24:/tmp/cpython$ git status
Not currently on any branch.
nothing to commit, working tree clean
suztomo@suxtomo24:/tmp/cpython$ git log -1
commit 3c6b436a57893dd1fae4e072768f41a199076252 (HEAD, tag: v3.6.8)
Author: Ned Deily <n...@python.org>
Date: Sun Dec 23 16:37:14 2018 -0500
3.6.8final
suztomo@suxtomo24:/tmp/cpython$ ./configure --prefix=$HOME/local # pick up your
preference
...
suztomo@suxtomo24:/tmp/cpython$ make install
{noformat}
Add the directory to the path with "/bin" appended. In {{~/.bashrc}}:
{noformat}
export PATH=$HOME/local/bin:$PATH
{noformat}
Now disutils.sysconfig module is available for Python3.6:
{noformat}
suztomo@suxtomo24:/tmp/cpython$ python3.6
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Nov 21 2019, 21:11:37)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from distutils import sysconfig
>>>
{noformat}
Now {{:sdks:python:test-suites:tox:py35:setupVirtualenv}} succeeds
{noformat}
suztomo@suxtomo24:~/beam4$ ./gradlew -p sdks/python/test-suites/tox/py35
setupVirtualenv
...
> Task :sdks:python:test-suites:tox:py35:setupVirtualenv
...
BUILD SUCCESSFUL in 5s
{noformat}
h2. testPy36Gcp failrue
The {{:sdks:python:test-suites:tox:py36:testPy36Gcp}} was failing:
https://gist.github.com/suztomo/ebfc110652b8ffaf7fede64276d7a053
It seemed that _bz2 library was missing for the Python3.6. Followed
[Stackoverflow: No module named '_bz2' in
python3|https://stackoverflow.com/questions/20280726/how-to-git-clone-a-specific-tag/24102558].
h2. testPy35Cython failure
{noformat}
./gradlew :sdks:python:test-suites:tox:py35:testPy35Cython
...
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -g
-fdebug-prefix-map=/build/python3.5-ta1Uke/python3.5-3.5.4=.
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m
-I/usr/local/google/home/suztomo/beam4/sdks/python/test-suites/tox/py35/build/srcs/sdks/python/target/.tox-py35-cython/py35-cython/include/python3.5m
-c apache_beam/coders/stream.c -o
build/temp.linux-x86_64-3.5/apache_beam/coders/stream.o
apache_beam/coders/stream.c:17:10: fatal error: Python.h: No such file or
directory
#include "Python.h"
^~~~~~~~~~
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
...
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':sdks:python:test-suites:tox:py35:testPy35Cython'.
{noformat}
Tried {{sudo apt-get install python3-dev}} but it didn't work. Installing
python3.5 from source code resolved the issue.
{noformat}
suztomo@suxtomo24:/tmp/cpython$ git checkout v3.5.9
suztomo@suxtomo24:/tmp/cpython$ ./configure --prefix=$HOME/local
suztomo@suxtomo24:/tmp/cpython$ make install
suztomo@suxtomo24:/tmp/cpython$ which python3.5
/usr/local/google/home/suztomo/local/bin/python3.5
{noformat}
> Python setup issues
> -------------------
>
> Key: BEAM-8787
> URL: https://issues.apache.org/jira/browse/BEAM-8787
> Project: Beam
> Issue Type: Bug
> Components: build-system
> Affects Versions: 2.16.0
> Environment: debian x86 (gLinux)
> Reporter: Elliotte Rusty Harold
> Priority: Major
>
> This could be an issue with incomplete or inaccurate contributing docs. tldr;
> `./gradlew check` fails on Debian after initial checkout.
> The docs say that one should first run:
> sudo apt-get install \
> openjdk-8-jdk \
> python-setuptools \
> python-pip \
> virtualenv
> but even after running this pieces are missing. I'm still debugging exactly
> what's missing but the symptoms look like this:
> > Task :sdks:python:test-suites:tox:py35:setupVirtualenv FAILED
> The path python3.5 (from --python=python3.5) does not exist
> > Task :sdks:python:test-suites:tox:py36:setupVirtualenv FAILED
> [ant:fmpp] Traceback (most recent call last):
> [ant:fmpp] File "/usr/lib/python3/dist-packages/virtualenv.py", line 25, in
> <module>
> [ant:fmpp] import distutils.sysconfig
> [ant:fmpp] ModuleNotFoundError: No module named 'distutils.sysconfig'
> ...
> FAILURE: Build completed with 2 failures.
> 1: Task failed with an exception.
> -----------
> * What went wrong:
> Execution failed for task ':sdks:python:test-suites:tox:py35:setupVirtualenv'.
> > Process 'command 'virtualenv'' finished with non-zero exit value 3
> Indeed there is no Python 3.5 on this system:
> gnome-user-share python2.6
> gnome-vfs-2.0 python2.7
> gnupg python3
> gnupg2 python3.6
> gold-ld python3.7
> goobuntu-config-tools python3.8
> But nowhere in the setup docs do we say that Python 3.5 is required to build
> this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
