So I think now that my problem is Spark-related after all. It looks like my
bootstrap script installs SciPy just fine in a regular environment, but
somehow interaction with PySpark breaks it.

On Fri, Jan 6, 2023 at 12:39 PM Bjørn Jørgensen <bjornjorgen...@gmail.com>
wrote:

> Create a Dockerfile
>
> FROM fedora
>
> RUN sudo yum install -y python3-devel
> RUN sudo pip3 install -U Cython && \
> sudo pip3 install -U pybind11 && \
> sudo pip3 install -U pythran && \
> sudo pip3 install -U numpy && \
> sudo pip3 install -U scipy
>
>
>
>
>
>
> docker build --pull --rm -f "Dockerfile" -t fedoratest:latest "."
>
> Sending build context to Docker daemon  2.048kB
> Step 1/3 : FROM fedora
> latest: Pulling from library/fedora
> Digest:
> sha256:3487c98481d1bba7e769cf7bcecd6343c2d383fdd6bed34ec541b6b23ef07664
> Status: Image is up to date for fedora:latest
>  ---> 95b7a2603d3a
> Step 2/3 : RUN sudo yum install -y python3-devel
>  ---> Running in a7c648ae7014
> Fedora 37 - x86_64                              7.5 MB/s |  64 MB
> 00:08
> Fedora 37 openh264 (From Cisco) - x86_64        418  B/s | 2.5 kB
> 00:06
> Fedora Modular 37 - x86_64                      471 kB/s | 3.0 MB
> 00:06
> Fedora 37 - x86_64 - Updates                    3.0 MB/s |  20 MB
> 00:06
> Fedora Modular 37 - x86_64 - Updates            179 kB/s | 1.1 MB
> 00:06
> Last metadata expiration check: 0:00:01 ago on Fri Jan  6 17:37:59 2023.
> Dependencies resolved.
>
> ================================================================================
>  Package                  Architecture Version              Repository
> Size
>
> ================================================================================
> Installing:
>  python3-devel            x86_64       3.11.1-1.fc37        updates
> 269 k
> Upgrading:
>  python3                  x86_64       3.11.1-1.fc37        updates
>  27 k
>  python3-libs             x86_64       3.11.1-1.fc37        updates
> 9.6 M
> Installing dependencies:
>  libpkgconf               x86_64       1.8.0-3.fc37         fedora
> 36 k
>  pkgconf                  x86_64       1.8.0-3.fc37         fedora
> 41 k
>  pkgconf-m4               noarch       1.8.0-3.fc37         fedora
> 14 k
>  pkgconf-pkg-config       x86_64       1.8.0-3.fc37         fedora
> 10 k
> Installing weak dependencies:
>  python3-pip              noarch       22.2.2-3.fc37        updates
> 3.1 M
>  python3-setuptools       noarch       62.6.0-2.fc37        fedora
>  1.6 M
>
> Transaction Summary
>
> ================================================================================
> Install  7 Packages
> Upgrade  2 Packages
>
> Total download size: 15 M
> Downloading Packages:
> (1/9): pkgconf-m4-1.8.0-3.fc37.noarch.rpm       2.9 kB/s |  14 kB
> 00:05
> (2/9): libpkgconf-1.8.0-3.fc37.x86_64.rpm       7.1 kB/s |  36 kB
> 00:05
> (3/9): pkgconf-1.8.0-3.fc37.x86_64.rpm          8.2 kB/s |  41 kB
> 00:05
> (4/9): pkgconf-pkg-config-1.8.0-3.fc37.x86_64.r 143 kB/s |  10 kB
> 00:00
> (5/9): python3-devel-3.11.1-1.fc37.x86_64.rpm   458 kB/s | 269 kB
> 00:00
> (6/9): python3-3.11.1-1.fc37.x86_64.rpm         442 kB/s |  27 kB
> 00:00
> (7/9): python3-setuptools-62.6.0-2.fc37.noarch. 2.1 MB/s | 1.6 MB
> 00:00
> (8/9): python3-pip-22.2.2-3.fc37.noarch.rpm     4.0 MB/s | 3.1 MB
> 00:00
> (9/9): python3-libs-3.11.1-1.fc37.x86_64.rpm    7.2 MB/s | 9.6 MB
> 00:01
>
> --------------------------------------------------------------------------------
> Total                                           1.8 MB/s |  15 MB
> 00:08
> Running transaction check
> Transaction check succeeded.
> Running transaction test
> Transaction test succeeded.
> Running transaction
>   Preparing        :
>  1/1
>   Upgrading        : python3-libs-3.11.1-1.fc37.x86_64
> 1/11
>   Upgrading        : python3-3.11.1-1.fc37.x86_64
>  2/11
>   Installing       : python3-setuptools-62.6.0-2.fc37.noarch
> 3/11
>   Installing       : python3-pip-22.2.2-3.fc37.noarch
>  4/11
>   Installing       : pkgconf-m4-1.8.0-3.fc37.noarch
>  5/11
>   Installing       : libpkgconf-1.8.0-3.fc37.x86_64
>  6/11
>   Installing       : pkgconf-1.8.0-3.fc37.x86_64
> 7/11
>   Installing       : pkgconf-pkg-config-1.8.0-3.fc37.x86_64
>  8/11
>   Installing       : python3-devel-3.11.1-1.fc37.x86_64
>  9/11
>   Cleanup          : python3-3.11.0-1.fc37.x86_64
> 10/11
>   Cleanup          : python3-libs-3.11.0-1.fc37.x86_64
>  11/11
>   Running scriptlet: python3-libs-3.11.0-1.fc37.x86_64
>  11/11
>   Verifying        : libpkgconf-1.8.0-3.fc37.x86_64
>  1/11
>   Verifying        : pkgconf-1.8.0-3.fc37.x86_64
> 2/11
>   Verifying        : pkgconf-m4-1.8.0-3.fc37.noarch
>  3/11
>   Verifying        : pkgconf-pkg-config-1.8.0-3.fc37.x86_64
>  4/11
>   Verifying        : python3-setuptools-62.6.0-2.fc37.noarch
> 5/11
>   Verifying        : python3-devel-3.11.1-1.fc37.x86_64
>  6/11
>   Verifying        : python3-pip-22.2.2-3.fc37.noarch
>  7/11
>   Verifying        : python3-3.11.1-1.fc37.x86_64
>  8/11
>   Verifying        : python3-3.11.0-1.fc37.x86_64
>  9/11
>   Verifying        : python3-libs-3.11.1-1.fc37.x86_64
>  10/11
>   Verifying        : python3-libs-3.11.0-1.fc37.x86_64
>  11/11
>
> Upgraded:
>   python3-3.11.1-1.fc37.x86_64         python3-libs-3.11.1-1.fc37.x86_64
>
> Installed:
>   libpkgconf-1.8.0-3.fc37.x86_64
>
>   pkgconf-1.8.0-3.fc37.x86_64
>
>   pkgconf-m4-1.8.0-3.fc37.noarch
>
>   pkgconf-pkg-config-1.8.0-3.fc37.x86_64
>
>   python3-devel-3.11.1-1.fc37.x86_64
>
>   python3-pip-22.2.2-3.fc37.noarch
>
>   python3-setuptools-62.6.0-2.fc37.noarch
>
>
> Complete!
> Removing intermediate container a7c648ae7014
>  ---> 593ffbdb0c08
> Step 3/3 : RUN sudo pip3 install -U Cython && sudo pip3 install -U
> pybind11 && sudo pip3 install -U pythran && sudo pip3 install -U numpy &&
> sudo pip3 install -U scipy
>  ---> Running in 97aa856851b4
> Collecting Cython
>   Downloading
> Cython-0.29.33-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
> (1.9 MB)
>      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 21.0 MB/s eta
> 0:00:00
> Installing collected packages: Cython
> Successfully installed Cython-0.29.33
> WARNING: Running pip as the 'root' user can result in broken permissions
> and conflicting behaviour with the system package manager. It is
> recommended to use a virtual environment instead:
> https://pip.pypa.io/warnings/venv
> Collecting pybind11
>   Downloading pybind11-2.10.3-py3-none-any.whl (222 kB)
>      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 222.4/222.4 kB 4.4 MB/s eta
> 0:00:00
> Installing collected packages: pybind11
> Successfully installed pybind11-2.10.3
> WARNING: Running pip as the 'root' user can result in broken permissions
> and conflicting behaviour with the system package manager. It is
> recommended to use a virtual environment instead:
> https://pip.pypa.io/warnings/venv
> Collecting pythran
>   Downloading pythran-0.12.0-py3-none-any.whl (4.2 MB)
>      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.2/4.2 MB 26.8 MB/s eta
> 0:00:00
> Collecting ply>=3.4
>   Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
>      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.6/49.6 kB 7.1 MB/s eta
> 0:00:00
> Collecting gast~=0.5.0
>   Downloading gast-0.5.3-py3-none-any.whl (19 kB)
> Collecting numpy
>   Downloading
> numpy-1.24.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> (17.3 MB)
>      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 34.8 MB/s eta
> 0:00:00
> Collecting beniget~=0.4.0
>   Downloading beniget-0.4.1-py3-none-any.whl (9.4 kB)
> Installing collected packages: ply, numpy, gast, beniget, pythran
> Successfully installed beniget-0.4.1 gast-0.5.3 numpy-1.24.1 ply-3.11
> pythran-0.12.0
> WARNING: Running pip as the 'root' user can result in broken permissions
> and conflicting behaviour with the system package manager. It is
> recommended to use a virtual environment instead:
> https://pip.pypa.io/warnings/venv
> Requirement already satisfied: numpy in
> /usr/local/lib64/python3.11/site-packages (1.24.1)
> WARNING: Running pip as the 'root' user can result in broken permissions
> and conflicting behaviour with the system package manager. It is
> recommended to use a virtual environment instead:
> https://pip.pypa.io/warnings/venv
> Collecting scipy
>   Downloading
> scipy-1.10.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
> (34.1 MB)
>      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 34.1/34.1 MB 30.8 MB/s eta
> 0:00:00
> Requirement already satisfied: numpy<1.27.0,>=1.19.5 in
> /usr/local/lib64/python3.11/site-packages (from scipy) (1.24.1)
> Installing collected packages: scipy
> Successfully installed scipy-1.10.0
> WARNING: Running pip as the 'root' user can result in broken permissions
> and conflicting behaviour with the system package manager. It is
> recommended to use a virtual environment instead:
> https://pip.pypa.io/warnings/venv
> Removing intermediate container 97aa856851b4
>  ---> 9b2a51c94efe
> Successfully built 9b2a51c94efe
> Successfully tagged fedoratest:latest
>
>
>
>
> [root@afebdea35f7c /]# python
> bash: python: command not found
> [root@afebdea35f7c /]# python3
> Python 3.11.1 (main, Dec  7 2022, 00:00:00) [GCC 12.2.1 20221121 (Red Hat
> 12.2.1-4)] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from scipy.stats import norm
> >>>
>
>
>
> fre. 6. jan. 2023 kl. 18:12 skrev Oliver Ruebenacker <
> oliv...@broadinstitute.org>:
>
>> Thank you for the link. I already tried most of what was suggested there,
>> but without success.
>>
>> On Fri, Jan 6, 2023 at 11:35 AM Bjørn Jørgensen <bjornjorgen...@gmail.com>
>> wrote:
>>
>>>
>>>
>>>
>>> https://stackoverflow.com/questions/66060487/valueerror-numpy-ndarray-size-changed-may-indicate-binary-incompatibility-exp
>>>
>>>
>>>
>>>
>>> fre. 6. jan. 2023, 16:01 skrev Oliver Ruebenacker <
>>> oliv...@broadinstitute.org>:
>>>
>>>>
>>>>      Hello,
>>>>
>>>>   I'm trying to install SciPy using a bootstrap script and then use it
>>>> to calculate a new field in a dataframe, running on AWS EMR.
>>>>
>>>>   Although the SciPy website states that only NumPy is needed, when I
>>>> tried to install SciPy using pip, pip kept failing, complaining about
>>>> missing software, until I ended up with this bootstrap script:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *sudo yum install -y python3-develsudo pip3 install -U Cythonsudo pip3
>>>> install -U pybind11sudo pip3 install -U pythransudo pip3 install -U
>>>> numpysudo pip3 install -U scipy*
>>>>
>>>>   At this point, the bootstrap seems to be successful, but then at this
>>>> line:
>>>>
>>>> *from scipy.stats import norm*
>>>>
>>>>   I get the following error:
>>>>
>>>> *ValueError: numpy.ndarray size changed, may indicate binary
>>>> incompatibility. Expected 88 from C header, got 80 from PyObject*
>>>>
>>>>   Any advice on how to proceed? Thanks!
>>>>
>>>>      Best, Oliver
>>>>
>>>> --
>>>> Oliver Ruebenacker, Ph.D. (he)
>>>> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>,
>>>> Flannick Lab <http://www.flannicklab.org/>, Broad Institute
>>>> <http://www.broadinstitute.org/>
>>>>
>>>
>>
>> --
>> Oliver Ruebenacker, Ph.D. (he)
>> Senior Software Engineer, Knowledge Portal Network <http://kp4cd.org/>, 
>> Flannick
>> Lab <http://www.flannicklab.org/>, Broad Institute
>> <http://www.broadinstitute.org/>
>>
>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>


-- 
Oliver Ruebenacker, Ph.D. (he)
Senior Software Engineer, Knowledge Portal Network
<http://kp4cd.org/>, Flannick
Lab <http://www.flannicklab.org/>, Broad Institute
<http://www.broadinstitute.org/>

Reply via email to