So I think now that my problem is Spark-related after all. It looks like my bootstrap script installs SciPy just fine in a regular environment, but somehow interaction with PySpark breaks it.
On Fri, Jan 6, 2023 at 12:39 PM Bjørn Jørgensen <bjornjorgen...@gmail.com> wrote: > Create a Dockerfile > > FROM fedora > > RUN sudo yum install -y python3-devel > RUN sudo pip3 install -U Cython && \ > sudo pip3 install -U pybind11 && \ > sudo pip3 install -U pythran && \ > sudo pip3 install -U numpy && \ > sudo pip3 install -U scipy > > > > > > > docker build --pull --rm -f "Dockerfile" -t fedoratest:latest "." > > Sending build context to Docker daemon 2.048kB > Step 1/3 : FROM fedora > latest: Pulling from library/fedora > Digest: > sha256:3487c98481d1bba7e769cf7bcecd6343c2d383fdd6bed34ec541b6b23ef07664 > Status: Image is up to date for fedora:latest > ---> 95b7a2603d3a > Step 2/3 : RUN sudo yum install -y python3-devel > ---> Running in a7c648ae7014 > Fedora 37 - x86_64 7.5 MB/s | 64 MB > 00:08 > Fedora 37 openh264 (From Cisco) - x86_64 418 B/s | 2.5 kB > 00:06 > Fedora Modular 37 - x86_64 471 kB/s | 3.0 MB > 00:06 > Fedora 37 - x86_64 - Updates 3.0 MB/s | 20 MB > 00:06 > Fedora Modular 37 - x86_64 - Updates 179 kB/s | 1.1 MB > 00:06 > Last metadata expiration check: 0:00:01 ago on Fri Jan 6 17:37:59 2023. > Dependencies resolved. > > ================================================================================ > Package Architecture Version Repository > Size > > ================================================================================ > Installing: > python3-devel x86_64 3.11.1-1.fc37 updates > 269 k > Upgrading: > python3 x86_64 3.11.1-1.fc37 updates > 27 k > python3-libs x86_64 3.11.1-1.fc37 updates > 9.6 M > Installing dependencies: > libpkgconf x86_64 1.8.0-3.fc37 fedora > 36 k > pkgconf x86_64 1.8.0-3.fc37 fedora > 41 k > pkgconf-m4 noarch 1.8.0-3.fc37 fedora > 14 k > pkgconf-pkg-config x86_64 1.8.0-3.fc37 fedora > 10 k > Installing weak dependencies: > python3-pip noarch 22.2.2-3.fc37 updates > 3.1 M > python3-setuptools noarch 62.6.0-2.fc37 fedora > 1.6 M > > Transaction Summary > > ================================================================================ > Install 7 Packages > Upgrade 2 Packages > > Total download size: 15 M > Downloading Packages: > (1/9): pkgconf-m4-1.8.0-3.fc37.noarch.rpm 2.9 kB/s | 14 kB > 00:05 > (2/9): libpkgconf-1.8.0-3.fc37.x86_64.rpm 7.1 kB/s | 36 kB > 00:05 > (3/9): pkgconf-1.8.0-3.fc37.x86_64.rpm 8.2 kB/s | 41 kB > 00:05 > (4/9): pkgconf-pkg-config-1.8.0-3.fc37.x86_64.r 143 kB/s | 10 kB > 00:00 > (5/9): python3-devel-3.11.1-1.fc37.x86_64.rpm 458 kB/s | 269 kB > 00:00 > (6/9): python3-3.11.1-1.fc37.x86_64.rpm 442 kB/s | 27 kB > 00:00 > (7/9): python3-setuptools-62.6.0-2.fc37.noarch. 2.1 MB/s | 1.6 MB > 00:00 > (8/9): python3-pip-22.2.2-3.fc37.noarch.rpm 4.0 MB/s | 3.1 MB > 00:00 > (9/9): python3-libs-3.11.1-1.fc37.x86_64.rpm 7.2 MB/s | 9.6 MB > 00:01 > > -------------------------------------------------------------------------------- > Total 1.8 MB/s | 15 MB > 00:08 > Running transaction check > Transaction check succeeded. > Running transaction test > Transaction test succeeded. > Running transaction > Preparing : > 1/1 > Upgrading : python3-libs-3.11.1-1.fc37.x86_64 > 1/11 > Upgrading : python3-3.11.1-1.fc37.x86_64 > 2/11 > Installing : python3-setuptools-62.6.0-2.fc37.noarch > 3/11 > Installing : python3-pip-22.2.2-3.fc37.noarch > 4/11 > Installing : pkgconf-m4-1.8.0-3.fc37.noarch > 5/11 > Installing : libpkgconf-1.8.0-3.fc37.x86_64 > 6/11 > Installing : pkgconf-1.8.0-3.fc37.x86_64 > 7/11 > Installing : pkgconf-pkg-config-1.8.0-3.fc37.x86_64 > 8/11 > Installing : python3-devel-3.11.1-1.fc37.x86_64 > 9/11 > Cleanup : python3-3.11.0-1.fc37.x86_64 > 10/11 > Cleanup : python3-libs-3.11.0-1.fc37.x86_64 > 11/11 > Running scriptlet: python3-libs-3.11.0-1.fc37.x86_64 > 11/11 > Verifying : libpkgconf-1.8.0-3.fc37.x86_64 > 1/11 > Verifying : pkgconf-1.8.0-3.fc37.x86_64 > 2/11 > Verifying : pkgconf-m4-1.8.0-3.fc37.noarch > 3/11 > Verifying : pkgconf-pkg-config-1.8.0-3.fc37.x86_64 > 4/11 > Verifying : python3-setuptools-62.6.0-2.fc37.noarch > 5/11 > Verifying : python3-devel-3.11.1-1.fc37.x86_64 > 6/11 > Verifying : python3-pip-22.2.2-3.fc37.noarch > 7/11 > Verifying : python3-3.11.1-1.fc37.x86_64 > 8/11 > Verifying : python3-3.11.0-1.fc37.x86_64 > 9/11 > Verifying : python3-libs-3.11.1-1.fc37.x86_64 > 10/11 > Verifying : python3-libs-3.11.0-1.fc37.x86_64 > 11/11 > > Upgraded: > python3-3.11.1-1.fc37.x86_64 python3-libs-3.11.1-1.fc37.x86_64 > > Installed: > libpkgconf-1.8.0-3.fc37.x86_64 > > pkgconf-1.8.0-3.fc37.x86_64 > > pkgconf-m4-1.8.0-3.fc37.noarch > > pkgconf-pkg-config-1.8.0-3.fc37.x86_64 > > python3-devel-3.11.1-1.fc37.x86_64 > > python3-pip-22.2.2-3.fc37.noarch > > python3-setuptools-62.6.0-2.fc37.noarch > > > Complete! > Removing intermediate container a7c648ae7014 > ---> 593ffbdb0c08 > Step 3/3 : RUN sudo pip3 install -U Cython && sudo pip3 install -U > pybind11 && sudo pip3 install -U pythran && sudo pip3 install -U numpy && > sudo pip3 install -U scipy > ---> Running in 97aa856851b4 > Collecting Cython > Downloading > Cython-0.29.33-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl > (1.9 MB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 21.0 MB/s eta > 0:00:00 > Installing collected packages: Cython > Successfully installed Cython-0.29.33 > WARNING: Running pip as the 'root' user can result in broken permissions > and conflicting behaviour with the system package manager. It is > recommended to use a virtual environment instead: > https://pip.pypa.io/warnings/venv > Collecting pybind11 > Downloading pybind11-2.10.3-py3-none-any.whl (222 kB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 222.4/222.4 kB 4.4 MB/s eta > 0:00:00 > Installing collected packages: pybind11 > Successfully installed pybind11-2.10.3 > WARNING: Running pip as the 'root' user can result in broken permissions > and conflicting behaviour with the system package manager. It is > recommended to use a virtual environment instead: > https://pip.pypa.io/warnings/venv > Collecting pythran > Downloading pythran-0.12.0-py3-none-any.whl (4.2 MB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 26.8 MB/s eta > 0:00:00 > Collecting ply>=3.4 > Downloading ply-3.11-py2.py3-none-any.whl (49 kB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.6/49.6 kB 7.1 MB/s eta > 0:00:00 > Collecting gast~=0.5.0 > Downloading gast-0.5.3-py3-none-any.whl (19 kB) > Collecting numpy > Downloading > numpy-1.24.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl > (17.3 MB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 34.8 MB/s eta > 0:00:00 > Collecting beniget~=0.4.0 > Downloading beniget-0.4.1-py3-none-any.whl (9.4 kB) > Installing collected packages: ply, numpy, gast, beniget, pythran > Successfully installed beniget-0.4.1 gast-0.5.3 numpy-1.24.1 ply-3.11 > pythran-0.12.0 > WARNING: Running pip as the 'root' user can result in broken permissions > and conflicting behaviour with the system package manager. It is > recommended to use a virtual environment instead: > https://pip.pypa.io/warnings/venv > Requirement already satisfied: numpy in > /usr/local/lib64/python3.11/site-packages (1.24.1) > WARNING: Running pip as the 'root' user can result in broken permissions > and conflicting behaviour with the system package manager. It is > recommended to use a virtual environment instead: > https://pip.pypa.io/warnings/venv > Collecting scipy > Downloading > scipy-1.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl > (34.1 MB) > ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.1/34.1 MB 30.8 MB/s eta > 0:00:00 > Requirement already satisfied: numpy<1.27.0,>=1.19.5 in > /usr/local/lib64/python3.11/site-packages (from scipy) (1.24.1) > Installing collected packages: scipy > Successfully installed scipy-1.10.0 > WARNING: Running pip as the 'root' user can result in broken permissions > and conflicting behaviour with the system package manager. It is > recommended to use a virtual environment instead: > https://pip.pypa.io/warnings/venv > Removing intermediate container 97aa856851b4 > ---> 9b2a51c94efe > Successfully built 9b2a51c94efe > Successfully tagged fedoratest:latest > > > > > [root@afebdea35f7c /]# python > bash: python: command not found > [root@afebdea35f7c /]# python3 > Python 3.11.1 (main, Dec 7 2022, 00:00:00) [GCC 12.2.1 20221121 (Red Hat > 12.2.1-4)] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> from scipy.stats import norm > >>> > > > > fre. 6. jan. 2023 kl. 18:12 skrev Oliver Ruebenacker < > oliv...@broadinstitute.org>: > >> Thank you for the link. I already tried most of what was suggested there, >> but without success. >> >> On Fri, Jan 6, 2023 at 11:35 AM Bjørn Jørgensen <bjornjorgen...@gmail.com> >> wrote: >> >>> >>> >>> >>> https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp >>> >>> >>> >>> >>> fre. 6. jan. 2023, 16:01 skrev Oliver Ruebenacker < >>> oliv...@broadinstitute.org>: >>> >>>> >>>> Hello, >>>> >>>> I'm trying to install SciPy using a bootstrap script and then use it >>>> to calculate a new field in a dataframe, running on AWS EMR. >>>> >>>> Although the SciPy website states that only NumPy is needed, when I >>>> tried to install SciPy using pip, pip kept failing, complaining about >>>> missing software, until I ended up with this bootstrap script: >>>> >>>> >>>> >>>> >>>> >>>> >>>> *sudo yum install -y python3-develsudo pip3 install -U Cythonsudo pip3 >>>> install -U pybind11sudo pip3 install -U pythransudo pip3 install -U >>>> numpysudo pip3 install -U scipy* >>>> >>>> At this point, the bootstrap seems to be successful, but then at this >>>> line: >>>> >>>> *from scipy.stats import norm* >>>> >>>> I get the following error: >>>> >>>> *ValueError: numpy.ndarray size changed, may indicate binary >>>> incompatibility. Expected 88 from C header, got 80 from PyObject* >>>> >>>> Any advice on how to proceed? Thanks! >>>> >>>> Best, Oliver >>>> >>>> -- >>>> Oliver Ruebenacker, Ph.D. (he) >>>> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>, >>>> Flannick Lab <http://www.flannicklab.org/>, Broad Institute >>>> <http://www.broadinstitute.org/> >>>> >>> >> >> -- >> Oliver Ruebenacker, Ph.D. (he) >> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>, >> Flannick >> Lab <http://www.flannicklab.org/>, Broad Institute >> <http://www.broadinstitute.org/> >> > > > -- > Bjørn Jørgensen > Vestre Aspehaug 4, 6010 Ålesund > Norge > > +47 480 94 297 > -- Oliver Ruebenacker, Ph.D. (he) Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>, Flannick Lab <http://www.flannicklab.org/>, Broad Institute <http://www.broadinstitute.org/>
