Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3749303678 Revision: ce85d064a7cdb5d56ede865f603ad56b4f4c5b7c Submitted crossbow builds: [ursacomputing/crossbow @ actions-3efa5e7447](https://github.com/ursacomputing/crossbow/branches/all?query=actions-3efa5e7447) |Task|Status| ||--| |example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/20993717702/job/60345219738)| |example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/20993719085/job/60345223965)| |python-sdist|[](https://github.com/ursacomputing/crossbow/actions/runs/20993717511/job/60345219151)| |test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/20993717493/job/60345219138)| |test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/20993718190/job/60345221159)| |test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/20993716245/job/60345215030)| |test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2|[](https://github.com/ursacomputing/crossbow/actions/runs/20993715180/job/60345211657)| |test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/20993715381/job/60345212329)| |test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/20993717853/job/60345220098)| |test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/20993716475/job/60345215666)| |test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/20993718492/job/6034506)| |test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/20993717361/job/60345218671)| |test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/20993716211/job/60345214758)| |test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/20993718648/job/60345222474)| |test-conda-python-3.12-cpython-debug|[](https://github.com/ursacomputing/crossbow/actions/runs/20993716741/job/60345216632)| |test-conda-python-3.12-pandas-latest-numpy-1.26|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-3efa5e7447-github
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
raulcd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2690463682
##
ci/scripts/python_sdist_build.sh:
##
@@ -23,5 +23,9 @@ source_dir=${1}/python
pushd "${source_dir}"
export SETUPTOOLS_SCM_PRETEND_VERSION=${PYARROW_VERSION:-}
-${PYTHON:-python} setup.py sdist
+# Meson dist must be run from a VCS, so initiate a dummy repo
+git init .
+git add --all .
+git commit -m "dummy commit for meson dist"
+${PYTHON:-python} -m build --sdist .
Review Comment:
I've tried a couple of things to try and fix sdist:
```diff
diff --git a/ci/scripts/python_sdist_build.sh
b/ci/scripts/python_sdist_build.sh
index 7bea9c3dc1..a8d849183c 100755
--- a/ci/scripts/python_sdist_build.sh
+++ b/ci/scripts/python_sdist_build.sh
@@ -24,8 +24,12 @@ source_dir=${1}/python
pushd "${source_dir}"
export SETUPTOOLS_SCM_PRETEND_VERSION=${PYARROW_VERSION:-}
# Meson dist must be run from a VCS, so initiate a dummy repo
-git init .
-git add --all .
-git commit -m "dummy commit for meson dist"
+#git init .
+#git config --global --add safe.directory "${source_dir}"
+#git config --global user.name "Your Name"
+#git config --global user.email [email protected]
+#git add --all .
+#git commit -m "dummy commit for meson dist"
+${PYTHON:-python} -m pip install build
${PYTHON:-python} -m build --sdist .
popd
```
I am unsure whether creating the git repo is necessary (we are already
cloning arrow) and `build` seems missing from this docker image.
The problem now is that it seems to want to build pyarrow (requires
build-tools), even though building only the sdist shouldn't require building
pyarrow as far as I understand:
```
+ /arrow-dev/bin/python -m build --sdist .
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
- cython >= 3.1
- meson-python
- numpy>=1.25
- setuptools_scm[toml]>=8
* Getting build dependencies for sdist...
* Installing packages in isolated environment:
- ninja >= 1.8.2
* Building sdist...
+ meson setup /arrow/python /arrow/python/.mesonpy-f5plfyxb
-Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md
--native-file=/arrow/python/.mesonpy-f5plfyxb/meson-python-native-file.ini
The Meson build system
Version: 1.10.0
Source dir: /arrow/python
Build dir: /arrow/python/.mesonpy-f5plfyxb
Build type: native build
Project name: pyarrow
Project version: 23.0.0.dev362+gce85d064a.d20260114
../meson.build:18:0: ERROR: Unknown compiler(s): [['c++'], ['g++'],
['clang++'], ['nvc++'], ['pgc++'], ['icpc'], ['icpx']]
The following exception(s) were encountered:
Running `c++ --version` gave "[Errno 2] No such file or directory: 'c++'"
Running `g++ --version` gave "[Errno 2] No such file or directory: 'g++'"
Running `clang++ --version` gave "[Errno 2] No such file or directory:
'clang++'"
Running `nvc++ --version` gave "[Errno 2] No such file or directory: 'nvc++'"
Running `pgc++ --version` gave "[Errno 2] No such file or directory: 'pgc++'"
Running `icpc --version` gave "[Errno 2] No such file or directory: 'icpc'"
Running `icpx --version` gave "[Errno 2] No such file or directory: 'icpx'"
A full log can be found at
/arrow/python/.mesonpy-f5plfyxb/meson-logs/meson-log.txt
ERROR Backend subprocess exited when trying to invoke build_sdist
```
If I install `build-essential` it fails locating Arrow CPP because it's
basically trying to build pyarrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
raulcd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3749279363 @github-actions crossbow -g python -g wheel -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
raulcd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3749288200 @github-actions crossbow submit -g python -g wheel -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2690601635
##
ci/scripts/python_sdist_build.sh:
##
@@ -23,5 +23,9 @@ source_dir=${1}/python
pushd "${source_dir}"
export SETUPTOOLS_SCM_PRETEND_VERSION=${PYARROW_VERSION:-}
-${PYTHON:-python} setup.py sdist
+# Meson dist must be run from a VCS, so initiate a dummy repo
+git init .
+git add --all .
+git commit -m "dummy commit for meson dist"
+${PYTHON:-python} -m build --sdist .
Review Comment:
> I am unsure whether creating the git repo is necessary
It is because building an sdist calls `meson dist` under the hood, which
itself requires to be run from a VCS
> The problem now is that it seems to want to build pyarrow
Try adding `-Csetup-args="-Dsdist=true"` to the invocation. This is
something we encountered previously on this PR, so I added that option (noted
in the top level python/meson.build file and upstream in
https://github.com/mesonbuild/meson-python/issues/647)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2690601635
##
ci/scripts/python_sdist_build.sh:
##
@@ -23,5 +23,9 @@ source_dir=${1}/python
pushd "${source_dir}"
export SETUPTOOLS_SCM_PRETEND_VERSION=${PYARROW_VERSION:-}
-${PYTHON:-python} setup.py sdist
+# Meson dist must be run from a VCS, so initiate a dummy repo
+git init .
+git add --all .
+git commit -m "dummy commit for meson dist"
+${PYTHON:-python} -m build --sdist .
Review Comment:
> I am unsure whether creating the git repo is necessary
It is because building an sdist calls `meson dist` under the hood, which
itself requires to be run from a VCS
> The problem now is that it seems to want to build pyarrow
Try adding `-Csetup-args="-Dsdist=enabled"` to the invocation. This is
something we encountered previously on this PR, so I added that option (noted
in the top level python/meson.build file and upstream in
https://github.com/mesonbuild/meson-python/issues/647)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3749281400 ``` No such option: -g The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/20993612885 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3749164955 Wow thanks for finding that! Feel free to push directly - thanks @raulcd ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3745805381 That's awesome thanks @raulcd ! The biggest challenge with this I think is just having maintained a long-lived PR. I am of the impression that the core foundation is in a good place, but some of the non-Meson items (particularly the conda library versions) can take a lot of time to troubleshoot as they come up Any help to troubleshoot and get another set of eyes on this is greatly appreciated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
raulcd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3745758724 @WillAyd I am going to have some time for this once I finish releasing Arrow 23.0.0. I've fixed the remaining pre-existing CI failures on main both for the PR checks and the extended archery ones. From what I read/understand this PR should be ready to be reviewable and the failures where just also replicated on main. In this case, can you rebase to see if we can get cleaner CI? Is there anything I can do to help you or is review / testing from my side the best approach to help? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3683108757 Rebased on main and all green. From the crossbow jobs, the freethreading and pandas job failures appear directly related to https://github.com/apache/arrow/issues/48314 and not this PR The emscripten failure also seems to be unrelated, as it fails on an install step prior to the PyArrow steps The cuda failures don't seem to have produced logs? If there is something to look into there please let me know -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3678141784 Revision: 62168a0083bccea2959ad158ac648b285dab688d Submitted crossbow builds: [ursacomputing/crossbow @ actions-180cf98f75](https://github.com/ursacomputing/crossbow/branches/all?query=actions-180cf98f75) |Task|Status| ||--| |example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/20400416107/job/58621925743)| |example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/20400415631/job/58621924828)| |test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/20400415562/job/58621924672)| |test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/20400416074/job/58621925701)| |test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/20400415638/job/58621924887)| |test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2|[](https://github.com/ursacomputing/crossbow/actions/runs/20400416296/job/58621926059)| |test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/20400415820/job/58621925258)| |test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/20400415509/job/58621924566)| |test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/20400415660/job/58621924900)| |test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/20400415504/job/58621924512)| |test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/20400415709/job/58621924996)| |test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/20400416066/job/58621925646)| |test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/20400415456/job/58621924523)| |test-conda-python-3.12-cpython-debug|[](https://github.com/ursacomputing/crossbow/actions/runs/20400416188/job/58621925879)| |test-conda-python-3.12-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/20400415693/job/58621924995)| |test-conda-python-3.12-pandas-latest-numpy-latest|[![GitHub Actions](https://github.com/ursacomputing/crossbow/a
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3678139856 @github-actions crossbow submit -g python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2608979210
##
.github/workflows/python.yml:
##
@@ -288,11 +288,39 @@ jobs:
env:
# We can invalidate the current cache by updating this.
CACHE_VERSION: "2025-09-16.1"
+ - name: Install conda
+shell: cmd
+run: |
+ curl -L
https://github.com/conda-forge/miniforge/releases/download/25.11.0-0/Miniforge3-25.11.0-0-Windows-x86_64.exe
-o miniforge.exe
+ start /wait "" .\miniforge.exe /S /D=C:\miniforge
+ del .\miniforge.exe
+ C:\miniforge\Scripts\conda.exe init cmd.exe
+ echo
C:\miniforge\Scripts;C:\miniforge\Library\bin;C:\miniforge\condabin>>
%GITHUB_PATH%
+ - name: Create conda environment
+shell: cmd
+run: |
+ conda env create -n pyarrow-dev -f .\ci\conda_env_cpp.txt -y
+ conda env update -n pyarrow-dev -f .\ci\conda_env_python.txt
- name: Build Arrow C++ and PyArrow
shell: cmd
+env:
+ PYTHON_CMD: "python"
run: |
- call "ci\scripts\python_build.bat" %cd% "${{
steps.path-info.outputs.usr-windows-dir }}"
+ call activate pyarrow-dev || exit /B 1
+ call "ci\scripts\python_build.bat" %cd% "%CONDA_PREFIX%"
+ - name: Upload wheel
+if: always() # ensures this runs even on failure
+uses: actions/upload-artifact@v4
+with:
+ name: built-wheel
+ path: python/dist/*.whl
- name: Test PyArrow
shell: cmd
+env:
+ PYTHON_CMD: "python"
+# TODO: Renaming lz4 library is a hack - see where the name gets mixed
up
run: |
+ call activate pyarrow-dev
+ move C:\miniforge\envs\pyarrow-dev\Library\bin\liblz4.dll
C:\miniforge\envs\pyarrow-dev\Library\bin\lz4.dll
Review Comment:
Jobs in this PR are now all green. I isolated the fix to the abseil
libraries in https://github.com/apache/arrow/pull/48418 and have included here
temporarily
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2600276684
##
python/meson.build:
##
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+project(
+'pyarrow',
+'cython',
+'cpp',
+version: run_command(
+'python',
+'-m',
+'setuptools_scm',
+'--force-write-version-files',
+check: true,
+).stdout().strip(),
+license: 'Apache-2.0',
+license_files: ['LICENSE.txt'],
+meson_version: '>=1.4.0',
+default_options: ['buildtype=release', 'cpp_std=c++17'],
+)
+
+py = import('python').find_installation(pure: false)
+
+install_data('NOTICE.txt', install_dir: py.get_install_dir() / 'pyarrow')
+# Meson-python suggests using a dist script to resolve symlinks to physical
+# files when creating the sdist. See
+#
https://github.com/mesonbuild/meson-python/discussions/823#discussioncomment-15078932
+meson.add_dist_script(
Review Comment:
As noted in the comment, meson-python advises that you symlink the
NOTICE/LICENSE files in the source tree and use a dist script to pack the
actual files into the Python sdist
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2600273450
##
.github/workflows/python.yml:
##
@@ -288,11 +288,39 @@ jobs:
env:
# We can invalidate the current cache by updating this.
CACHE_VERSION: "2025-09-16.1"
+ - name: Install conda
+shell: cmd
+run: |
+ curl -L
https://github.com/conda-forge/miniforge/releases/download/25.11.0-0/Miniforge3-25.11.0-0-Windows-x86_64.exe
-o miniforge.exe
+ start /wait "" .\miniforge.exe /S /D=C:\miniforge
+ del .\miniforge.exe
+ C:\miniforge\Scripts\conda.exe init cmd.exe
+ echo
C:\miniforge\Scripts;C:\miniforge\Library\bin;C:\miniforge\condabin>>
%GITHUB_PATH%
+ - name: Create conda environment
+shell: cmd
+run: |
+ conda env create -n pyarrow-dev -f .\ci\conda_env_cpp.txt -y
+ conda env update -n pyarrow-dev -f .\ci\conda_env_python.txt
- name: Build Arrow C++ and PyArrow
shell: cmd
+env:
+ PYTHON_CMD: "python"
run: |
- call "ci\scripts\python_build.bat" %cd% "${{
steps.path-info.outputs.usr-windows-dir }}"
+ call activate pyarrow-dev || exit /B 1
+ call "ci\scripts\python_build.bat" %cd% "%CONDA_PREFIX%"
+ - name: Upload wheel
+if: always() # ensures this runs even on failure
+uses: actions/upload-artifact@v4
+with:
+ name: built-wheel
+ path: python/dist/*.whl
- name: Test PyArrow
shell: cmd
+env:
+ PYTHON_CMD: "python"
+# TODO: Renaming lz4 library is a hack - see where the name gets mixed
up
run: |
+ call activate pyarrow-dev
+ move C:\miniforge\envs\pyarrow-dev\Library\bin\liblz4.dll
C:\miniforge\envs\pyarrow-dev\Library\bin\lz4.dll
Review Comment:
OK fixed the windows jobs by bumping lz4-c to 1.10.0. At the same time, a
regression with the Linux jobs was introduced, whereby abseil symbols that
flight needs are not being found. I noticed in CI that the lz4-c bump caused
abseil to jump from 20240116.2 to 20240722.0, which may or may not be related.
Will be offline for a few days, but hope to resolve when I get back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2596800779
##
.github/workflows/python.yml:
##
@@ -288,11 +288,39 @@ jobs:
env:
# We can invalidate the current cache by updating this.
CACHE_VERSION: "2025-09-16.1"
+ - name: Install conda
+shell: cmd
+run: |
+ curl -L
https://github.com/conda-forge/miniforge/releases/download/25.11.0-0/Miniforge3-25.11.0-0-Windows-x86_64.exe
-o miniforge.exe
+ start /wait "" .\miniforge.exe /S /D=C:\miniforge
+ del .\miniforge.exe
+ C:\miniforge\Scripts\conda.exe init cmd.exe
+ echo
C:\miniforge\Scripts;C:\miniforge\Library\bin;C:\miniforge\condabin>>
%GITHUB_PATH%
+ - name: Create conda environment
+shell: cmd
+run: |
+ conda env create -n pyarrow-dev -f .\ci\conda_env_cpp.txt -y
+ conda env update -n pyarrow-dev -f .\ci\conda_env_python.txt
- name: Build Arrow C++ and PyArrow
shell: cmd
+env:
+ PYTHON_CMD: "python"
run: |
- call "ci\scripts\python_build.bat" %cd% "${{
steps.path-info.outputs.usr-windows-dir }}"
+ call activate pyarrow-dev || exit /B 1
+ call "ci\scripts\python_build.bat" %cd% "%CONDA_PREFIX%"
+ - name: Upload wheel
+if: always() # ensures this runs even on failure
+uses: actions/upload-artifact@v4
+with:
+ name: built-wheel
+ path: python/dist/*.whl
- name: Test PyArrow
shell: cmd
+env:
+ PYTHON_CMD: "python"
+# TODO: Renaming lz4 library is a hack - see where the name gets mixed
up
run: |
+ call activate pyarrow-dev
+ move C:\miniforge\envs\pyarrow-dev\Library\bin\liblz4.dll
C:\miniforge\envs\pyarrow-dev\Library\bin\lz4.dll
Review Comment:
OK looks like a version issue. 1.9.4 uses the `lib` prefix:
https://conda-metadata-app.streamlit.app/?q=conda-forge%2Fwin-64%2Flz4-c-1.9.4-hcfcfb64_0.conda
but 1.10.0 seems to have dropped it:
https://conda-metadata-app.streamlit.app/?q=conda-forge%2Fwin-64%2Flz4-c-1.10.0-h2466b09_1.conda
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2595588048
##
.github/workflows/python.yml:
##
@@ -288,11 +288,39 @@ jobs:
env:
# We can invalidate the current cache by updating this.
CACHE_VERSION: "2025-09-16.1"
+ - name: Install conda
+shell: cmd
+run: |
+ curl -L
https://github.com/conda-forge/miniforge/releases/download/25.11.0-0/Miniforge3-25.11.0-0-Windows-x86_64.exe
-o miniforge.exe
+ start /wait "" .\miniforge.exe /S /D=C:\miniforge
+ del .\miniforge.exe
+ C:\miniforge\Scripts\conda.exe init cmd.exe
+ echo
C:\miniforge\Scripts;C:\miniforge\Library\bin;C:\miniforge\condabin>>
%GITHUB_PATH%
+ - name: Create conda environment
+shell: cmd
+run: |
+ conda env create -n pyarrow-dev -f .\ci\conda_env_cpp.txt -y
+ conda env update -n pyarrow-dev -f .\ci\conda_env_python.txt
- name: Build Arrow C++ and PyArrow
shell: cmd
+env:
+ PYTHON_CMD: "python"
run: |
- call "ci\scripts\python_build.bat" %cd% "${{
steps.path-info.outputs.usr-windows-dir }}"
+ call activate pyarrow-dev || exit /B 1
+ call "ci\scripts\python_build.bat" %cd% "%CONDA_PREFIX%"
+ - name: Upload wheel
+if: always() # ensures this runs even on failure
+uses: actions/upload-artifact@v4
+with:
+ name: built-wheel
+ path: python/dist/*.whl
- name: Test PyArrow
shell: cmd
+env:
+ PYTHON_CMD: "python"
+# TODO: Renaming lz4 library is a hack - see where the name gets mixed
up
run: |
+ call activate pyarrow-dev
+ move C:\miniforge\envs\pyarrow-dev\Library\bin\liblz4.dll
C:\miniforge\envs\pyarrow-dev\Library\bin\lz4.dll
Review Comment:
Hmm so then that's an issue with the conda recipe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2595090795 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: I'd rather we understand why it's not possible to do the same things we're currently doing with setuptools. Not everyone will want to use conda on Windows. Is meson less flexible than we'd like here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2595089698
##
.github/workflows/python.yml:
##
@@ -288,11 +288,39 @@ jobs:
env:
# We can invalidate the current cache by updating this.
CACHE_VERSION: "2025-09-16.1"
+ - name: Install conda
+shell: cmd
+run: |
+ curl -L
https://github.com/conda-forge/miniforge/releases/download/25.11.0-0/Miniforge3-25.11.0-0-Windows-x86_64.exe
-o miniforge.exe
+ start /wait "" .\miniforge.exe /S /D=C:\miniforge
+ del .\miniforge.exe
+ C:\miniforge\Scripts\conda.exe init cmd.exe
+ echo
C:\miniforge\Scripts;C:\miniforge\Library\bin;C:\miniforge\condabin>>
%GITHUB_PATH%
+ - name: Create conda environment
+shell: cmd
+run: |
+ conda env create -n pyarrow-dev -f .\ci\conda_env_cpp.txt -y
+ conda env update -n pyarrow-dev -f .\ci\conda_env_python.txt
- name: Build Arrow C++ and PyArrow
shell: cmd
+env:
+ PYTHON_CMD: "python"
run: |
- call "ci\scripts\python_build.bat" %cd% "${{
steps.path-info.outputs.usr-windows-dir }}"
+ call activate pyarrow-dev || exit /B 1
+ call "ci\scripts\python_build.bat" %cd% "%CONDA_PREFIX%"
+ - name: Upload wheel
+if: always() # ensures this runs even on failure
+uses: actions/upload-artifact@v4
+with:
+ name: built-wheel
+ path: python/dist/*.whl
- name: Test PyArrow
shell: cmd
+env:
+ PYTHON_CMD: "python"
+# TODO: Renaming lz4 library is a hack - see where the name gets mixed
up
run: |
+ call activate pyarrow-dev
+ move C:\miniforge\envs\pyarrow-dev\Library\bin\liblz4.dll
C:\miniforge\envs\pyarrow-dev\Library\bin\lz4.dll
Review Comment:
The `lib` prefix for libraries is hardly ever used on Windows AFAIK, how
does this happen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2595037651
##
.github/workflows/python.yml:
##
@@ -288,11 +288,39 @@ jobs:
env:
# We can invalidate the current cache by updating this.
CACHE_VERSION: "2025-09-16.1"
+ - name: Install conda
+shell: cmd
+run: |
+ curl -L
https://github.com/conda-forge/miniforge/releases/download/25.11.0-0/Miniforge3-25.11.0-0-Windows-x86_64.exe
-o miniforge.exe
+ start /wait "" .\miniforge.exe /S /D=C:\miniforge
+ del .\miniforge.exe
+ C:\miniforge\Scripts\conda.exe init cmd.exe
+ echo
C:\miniforge\Scripts;C:\miniforge\Library\bin;C:\miniforge\condabin>>
%GITHUB_PATH%
+ - name: Create conda environment
+shell: cmd
+run: |
+ conda env create -n pyarrow-dev -f .\ci\conda_env_cpp.txt -y
+ conda env update -n pyarrow-dev -f .\ci\conda_env_python.txt
- name: Build Arrow C++ and PyArrow
shell: cmd
+env:
+ PYTHON_CMD: "python"
run: |
- call "ci\scripts\python_build.bat" %cd% "${{
steps.path-info.outputs.usr-windows-dir }}"
+ call activate pyarrow-dev || exit /B 1
+ call "ci\scripts\python_build.bat" %cd% "%CONDA_PREFIX%"
+ - name: Upload wheel
+if: always() # ensures this runs even on failure
+uses: actions/upload-artifact@v4
+with:
+ name: built-wheel
+ path: python/dist/*.whl
- name: Test PyArrow
shell: cmd
+env:
+ PYTHON_CMD: "python"
+# TODO: Renaming lz4 library is a hack - see where the name gets mixed
up
run: |
+ call activate pyarrow-dev
+ move C:\miniforge\envs\pyarrow-dev\Library\bin\liblz4.dll
C:\miniforge\envs\pyarrow-dev\Library\bin\lz4.dll
Review Comment:
This is a hack to rename these libraries. I'm not sure how the name is
getting tripped up on the first place - is this something that would come out
of `FindLZ4Alt.cmake`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2595035711 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: OK conda is set up in the Windows CI job now. However, the tests which ensure a downstream build against pyarrow are failing because the arrow library is not found in the `pa.get_library_dirs()` directory. I think the options are: - Skip these tests on Windows - Use delvewheel again to pack the arrow libraries back into the wheel - Update pa.get_library_dirs() to detect a conda environment and expose those directories Do you have a preference out of these? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2567900179 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: Hmm, I think installing mamba or equivalent [manually](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html#automatic-install) may be the best course of action. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2567900179 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: Hmm, I think installing mamba or equivalent [manually](https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html#windows) may be the best course of action. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2566412299 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: I see that this repo limits use of non-verified marketplace actions like `conda-incubator/setup-miniconda` and `mamba-org/setup-micromamba`. At the same time, I don't see any pre-built Windows docker images that already have conda. Without knowing yet how Archery works, do you think its best to go down the marketplace action path or to set up something with docker/archery? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2565482811 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: I don't know why the comment mentions `delvewheel`, though. We already use `delvewheel` successfully in the Windows wheel builds, FTR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2565449841 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: Hmm I was mistakenly assuming that conda was being used for this Windows job, but on closer inspection that doesn't appear to be the case. It looks like the install prefix is being set to `cygpath --absolute --windows /usr`, so perhaps that is the issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
eli-schwartz commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2562204807 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: @pitrou aha, that is very nice. That means it should work consistently across windows / Linux / macOS for libraries linked as conda dependencies (though admittedly it doesn't help much for arbitrary windows users via the official python redistributable who *also* want to use editable builds, but at a certain point I suppose you just have to give up and say that people doing developer builds need to use the tooling designed to make that work well on Windows). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2560314392 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: Thanks for the link @pitrou that's great. And yes by "system" libraries I meant "conda-installed". As mentioned it works locally, so I'll just have to spend a little extra time to see what's going on in CI. My current best guess is that something is awry with the loading mechanisms between the build step that calls vcvarsall.bat and the test step which doesn't -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2560272704 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: > No -- there are no "system libraries" on Windows. Arrow-cpp isn't installed to C:\Windows\System32\ No, but conda patches Python so as to lookup shared libraries inside the conda environment, including `%CONDA_PREFIX%/bin`. See https://github.com/conda-forge/python-feedstock/blob/main/recipe/patches/0007-Add-CondaEcosystemModifyDllSearchPath.patch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2560272704 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: > No -- there are no "system libraries" on Windows. Arrow-cpp isn't installed to C:\Windows\System32\ No, but conda patches Python so as to lookup shared libraries inside the conda environment. See https://github.com/conda-forge/python-feedstock/blob/main/recipe/patches/0007-Add-CondaEcosystemModifyDllSearchPath.patch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
eli-schwartz commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2560258353 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: No -- there are no "system libraries" on Windows. Arrow-cpp isn't installed to C:\\Windows\\System32\\ At build time, pkgconfig or `*-config.cmake` files are used to find the absolute paths, which the linker / runtime loader doesn't understand. This is the same on Linux and Windows, but on Linux the tuntime loader can have this information transferred from pkgconfig into the binary extension itself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
eli-schwartz commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2560258353 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: No -- there are no "system libraries" on Windows. Arrow-cpp isn't installed to C:\\Windows\\System32\\ At build time, pkgconfig or `*-config.cmake` files are used to find the absolute paths, which the linker / runtime loader doesn't understand. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2560205196 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: While the lack of RPATH support on Windows is unfortunate, I don't think that's relevant here (?). This is installing the package on the same machine that was used to build it, so I would expect it can still resolve to the system libraries, no? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
eli-schwartz commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2560176793 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: At least on Unix platforms, editable builds will still have any relevant Meson (not meson-python) rpath entries pointing to private copies of dependency libraries. Windows doesn't have rpath because the technology is too powerful for them. :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2559856035 ## python/meson.build: ## @@ -0,0 +1,84 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +project( +'pyarrow', +'cython', +'cpp', +version: run_command( +'python', +'-m', +'setuptools_scm', +'--force-write-version-files', +check: true, +).stdout().strip(), +license: 'Apache-2.0', +#license_files: ['../LICENSE.txt'], Review Comment: Sure - meson has the meson.add_dist_script() function that should help. I'll give that a closer look here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2559850734 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: I couldn't reproduce this locally on a Windows machine, so development seemed fine. I'll see if I can better pinpoint the root cause from CI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2559095063 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: > The crux of the issue is that the CMake module on main will copy the Arrow libraries from the system into the package. meson-python is more strict about what you can copy into the package, and I don't believe it allows you to do that, pushing you instead to use tools like delvewheel That's gonna be annoying for development. What about editable builds? (`pip install -e ...`) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2559095063 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +set PYARROW_WITH_SUBSTRAIT=auto +) +if %ARROW_S3% == ON ( +set PYARROW_WITH_S3=enabled +) else if %ARROW_S3% == OFF ( +set PYARROW_WITH_S3=disabled +) else ( +set PYARROW_WITH_S3=auto +) +if %CMAKE_BUILD_TYPE% == Release ( +set MESON_BUILD_TYPE=release +) else ( +set MESON_BUILD_TYPE=debug +) pushd %SOURCE_DIR%\python @REM Install Python build dependencies %PYTHON_CMD% -m pip install --upgrade pip || exit /B 1 %PYTHON_CMD% -m pip install -r requirements-build.txt || exit /B 1 +%PYTHON_CMD% -m pip install build delvewheel || exit /B 1 + +@REM by default, CMake installs .lib import libs to lib and .dll libs to bin +@REM delvewheel requires these to be side-by-side to properly vendor +@REM https://github.com/adang1345/delvewheel/issues/66 +copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\ Review Comment: > The crux of the issue is that the CMake module on main will copy the Arrow libraries from the system into the package. meson-python is more strict about what you can copy into the package, and I don't believe it allows you to do that, pushing you instead to use tools like delvewheel That's gonna be annoying for development. What about editable builds? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2559085495 ## python/meson.build: ## @@ -0,0 +1,84 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +project( +'pyarrow', +'cython', +'cpp', +version: run_command( +'python', +'-m', +'setuptools_scm', +'--force-write-version-files', +check: true, +).stdout().strip(), +license: 'Apache-2.0', +#license_files: ['../LICENSE.txt'], Review Comment: Is it possible to add a build step that copies the license file(s) to the right place before packaging? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3565010015 Revision: 91acf11679e877d4cbee53a9b89bf77764a68abc Submitted crossbow builds: [ursacomputing/crossbow @ actions-ab25d09306](https://github.com/ursacomputing/crossbow/branches/all?query=actions-ab25d09306) |Task|Status| ||--| |example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/19586583021/job/56096622691)| |example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582357/job/56096620473)| |test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582640/job/56096621374)| |test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582755/job/56096621879)| |test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582988/job/56096622542)| |test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582463/job/56096620741)| |test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582314/job/56096620417)| |test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582783/job/56096621840)| |test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582643/job/56096621386)| |test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/19586583052/job/56096622837)| |test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582905/job/56096622287)| |test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582255/job/56096620238)| |test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582585/job/56096621145)| |test-conda-python-3.12-cpython-debug|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582952/job/56096622532)| |test-conda-python-3.12-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/19586582336/job/56096620426)| |test-conda-python-3.12-pandas-latest-numpy-latest|[![GitHub Actions](https://github.com/ursacomputing/crossbow/a
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3565004814 @github-actions crossbow submit -g python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2551137886 ## .github/workflows/python.yml: ## @@ -175,7 +175,7 @@ jobs: ARROW_BUILD_TESTS: OFF PYARROW_TEST_LARGE_MEMORY: ON # Current oldest supported version according to https://endoflife.date/macos - MACOSX_DEPLOYMENT_TARGET: 12.0 + MACOSX_DEPLOYMENT_TARGET: "12.0" Review Comment: I enclosed this in parentheses so that it gets evaluated as the string "12.0" and not the number; otherwise some functionality in Meson was failing to unpack the major/minor version Somewhat tangentially it looks like the minimum supported version right now is 14.0 - worth updating? ## python/meson.build: ## @@ -0,0 +1,84 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +project( +'pyarrow', +'cython', +'cpp', +version: run_command( +'python', +'-m', +'setuptools_scm', +'--force-write-version-files', +check: true, +).stdout().strip(), +license: 'Apache-2.0', +#license_files: ['../LICENSE.txt'], Review Comment: This is going to reflect the same limitation that is being worked around in https://github.com/apache/arrow/pull/47141 As far as I am aware, Meson supports the inclusion of files like this, but meson-python has opted for a strict interpretation of PEP-639 that requires the LICENSE.txt (and similar files) to exist within the python source. See some upstream discussion at https://github.com/mesonbuild/meson/pull/14387 ## ci/scripts/python_build.bat: ## @@ -110,30 +110,116 @@ ccache -sv echo "=== Building Python ===" set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE% -set PYARROW_BUILD_VERBOSE=1 -set PYARROW_BUNDLE_ARROW_CPP=ON -set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR% -set PYARROW_WITH_ACERO=%ARROW_ACERO% -set PYARROW_WITH_DATASET=%ARROW_DATASET% -set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_HDFS=%ARROW_HDFS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=%ARROW_PARQUET% -set PYARROW_WITH_PARQUET_ENCRYPTION=%PARQUET_REQUIRE_ENCRYPTION% -set PYARROW_WITH_SUBSTRAIT=%ARROW_SUBSTRAIT% -set PYARROW_WITH_S3=%ARROW_S3% -set ARROW_HOME=%CMAKE_INSTALL_PREFIX% -set CMAKE_PREFIX_PATH=%CMAKE_INSTALL_PREFIX% +if %ARROW_ACERO% == ON ( +set PYARROW_WITH_ACERO=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_ACERO=disabled +) else ( +set PYARROW_WITH_ACERO=auto +) +if %ARROW_DATASET% == ON ( +set PYARROW_WITH_DATASET=enabled +) else if %ARROW_DATASET% == OFF ( +set PYARROW_WITH_DATASET=disabled +) else ( +set PYARROW_WITH_DATASET=auto +) +if %ARROW_FLIGHT% == ON ( +set PYARROW_WITH_FLIGHT=enabled +) else if %ARROW_FLIGHT% == OFF ( +set PYARROW_WITH_FLIGHT=disabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) +if %ARROW_GANDIVA% == ON ( +set PYARROW_WITH_GANDIVA=enabled +) else if %ARROW_GANDIVA% == OFF ( +set PYARROW_WITH_GANDIVA=disabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) +if %ARROW_GCS% == ON ( +set PYARROW_WITH_GCS=enabled +) else if %ARROW_GCS% == OFF ( +set PYARROW_WITH_GCS=disabled +) else ( +set PYARROW_WITH_GCS=auto +) +if %ARROW_HDFS% == ON ( +set PYARROW_WITH_HDFS=enabled +) else if %ARROW_HDFS% == OFF ( +set PYARROW_WITH_HDFS=disabled +) else ( +set PYARROW_WITH_HDFS=auto +) +if %ARROW_ORC% == ON ( +set PYARROW_WITH_ORC=enabled +) else if %ARROW_ORC% == OFF ( +set PYARROW_WITH_ORC=disabled +) else ( +set PYARROW_WITH_ORC=auto +) +if %ARROW_PARQUET% == ON ( +set PYARROW_WITH_PARQUET=enabled +) else if %ARROW_PARQUET% == OFF ( +set PYARROW_WITH_PARQUET=disabled +) else ( +set PYARROW_WITH_PARQUET=auto +) +if %PARQUET_REQUIRE_ENCRYPTION% == ON ( +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled +) else if %ARROW_ACERO% == OFF ( +set PYARROW_WITH_PARQUET_ENCRYPTION=disabled +) else ( +set PYARROW_WITH_PARQUET_ENCRYPTION=auto +) +if %ARROW_SUBSTRAIT% == ON ( +set PYARROW_WITH_SUBSTRAIT=enabled +) else if %ARROW_SUBSTRAIT% == OFF ( +set PYARROW_WITH_SUBSTRAIT=disabled +) else ( +
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3564598088 Revision: 7ca8999c3b9ce2d1a17ce4d5c3575161b3d37f0d Submitted crossbow builds: [ursacomputing/crossbow @ actions-fea1cc1589](https://github.com/ursacomputing/crossbow/branches/all?query=actions-fea1cc1589) |Task|Status| ||--| |example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/19583556565/job/56087126850)| |example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/19583556755/job/56087127537)| |test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/19583556053/job/56087125054)| |test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/19583556897/job/56087128049)| |test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/19583556227/job/56087125490)| |test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2|[](https://github.com/ursacomputing/crossbow/actions/runs/19583556467/job/56087126462)| |test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/19583556186/job/56087125397)| |test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/19583557079/job/56087129441)| |test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/19583555881/job/56087124424)| |test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/19583555984/job/56087124680)| |test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/19583557021/job/56087128736)| |test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/19583556464/job/56087126555)| |test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/19583555884/job/56087124479)| |test-conda-python-3.12-cpython-debug|[](https://github.com/ursacomputing/crossbow/actions/runs/19583556646/job/56087127073)| |test-conda-python-3.12-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/19583555849/job/56087124364)| |test-conda-python-3.12-pandas-latest-numpy-latest|[![GitHub Actions](https://github.com/ursacomputing/crossbow/a
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3564591954 @github-actions crossbow submit -g python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3553230924 I removed all of the "build C++ automatically" components for now to keep this streamlined. There are currently a few Python failures: 1. Linux Python 3.10 job fails to get NumPy includes; I think this has to do with the job mixing conda and pip installs of NumPy 2. macOS jobs are segfaulting on import; I don't have access to macOS to test locally - open to any input 3. Windows job fails on import; I cannot reproduce this locally so I think it has to do with Windows CI job I vaguely remember https://github.com/adang1345/delvewheel/issues/54 causing issues with Windows jobs in the past, but I don't know that that is still applicable here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3529031482 Thanks @eli-schwartz for clarifying, and sorry @raulcd I was implicitly just talking about a complete source build of both the C++/Python components (essentially trying to solve https://github.com/apache/arrow/issues/45576 in one pass) If there's interest, we can trim the scope of this PR to _just_ switching from setuptools to meson-python, but still requiring the host machine to have a system install of Arrow C++ before building PyArrow. The end-to-end source build can be tackled in subsequent PRs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
eli-schwartz commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3528870290 I think you still want to support that anyway -- Linux distros want to configure that way, and won't be affected by challenge 5 at all. meson supports this easily -- subprojects (arrow-cpp) can lookup a system pkg-config dependency instead. After that it's a "simple" matter of using auditwheel for PyPI uploads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3528586430 > What would happen in a world were we build Arrow C++ independently via CMake but we build Pyarrow with meson? Are the challenges above still applying? That's actually where this PR started - using CMake to build the C++ components and using Meson's CMake wrapper to build on top of that. Its "workable" but likely more fragile; in the grand scheme of things to get a fully fledged build system that supports passing arguments from the pip front-end to the backend, packaging, and installing components correctly, I _think_ going all in on Meson will be less effort (though slower to start) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
raulcd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3528535260 @WillAyd sorry I haven't been close to this until now. I might have some time to spend in the Pyarrow build system in the near future and this looks promising. What would happen in a world were we build Arrow C++ independently via CMake but we build Pyarrow with meson? Are the challenges above still applying? Sorry if I'm being naive, I haven't spent much time investigating myself yet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-3528345646 Its been a few months but I am still keeping this branch alive. I think there has been some really good progress. As things currently stand, this uses a symlink in `python/subprojects` that allows you to build both the C++ and Python components together, all through the pip front-end. There's still a few major issues to address: - Windows support (in progress in https://github.com/apache/arrow/pull/47282, although currently blocked by issues with gRPC) - Orc support (in progress in https://github.com/apache/arrow/pull/46906; waiting on next release of Orc) - s3 support (blocked by an upstream issue https://github.com/aws/aws-sdk-cpp/issues/3613 - might be challenging depending on what AWS accepts for PRs) - CUDA support (in progress in https://github.com/apache/arrow/pull/46166 - might be easy now that Meson 1.9 is out) - Build Arrow C++ as a shared library to reduce file size (started with static for ease; upstream guidance in https://mesonbuild.com/meson-python/how-to-guides/shared-libraries.html) There are also some TODOs scattered throughout the current implementation that can use attention, although I don't think any are major If anyone has time/interest and wants to try out this branch locally I'd love to hear feedback -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
eli-schwartz commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2069210359 ## python/MANIFEST.in: ## @@ -1,15 +0,0 @@ -include README.md -include ../LICENSE.txt Review Comment: I patched meson (not meson-python) so that building binary artifacts and installing them with meson's SBOM generator (automatically produced and installed when passing `--licensedir=/usr/share/licenses/PyArrow/`) should correctly install the license. In principle, that's also how dynamically defined pyproject licenses get inserted into a wheel. Building a standalone `meson dist` tarball release (similar to Autotools `make distcheck`) will not try to resolve the license, at least not currently. My thoughts were that it is *obviously incorrect* to mishandle a clear directive: > license-file: ../XXX.md > install-licenses: yes and we can make this "obviously correct" so that monorepos will install correctly. But it's not immediately apparent whether or not most monorepos utilizing symlinks between different directories, expect that a single directory should also have standalone source code releases (they may only support installing from the entire monorepo workspace, or they may generate per-component binary packages from the monorepo and then tell people to install from binaries). Building source code tarballs is a totally optional thing that many people don't do (both in the C/C++ world and in the python world. When was the last time you saw a Torch sdist on PyPI? When was the last time you tried to build python-tensorflow using any kind of PyPA standards at all? If some organizational design makes it inconvenient or troublesome to do an sdist, that should never be a valid reason to unnecessarily restrict wheel creation. I'm definitely interested to hear more about solutions which monorepo users would like to see, as monorepos are a completely valid use case. But it's possible that producing an sdist from a monorepo requires designing a feature, not just fixing a bug. (Or not. I don't usually work with monorepos so I'm not an expert on designing workflows for them.) And those take longer to formulate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
eli-schwartz commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2069210359 ## python/MANIFEST.in: ## @@ -1,15 +0,0 @@ -include README.md -include ../LICENSE.txt Review Comment: I patched meson (not meson-python) so that building binary artifacts and installing them with meson's SBOM generator (automatically produced and installed when passing `--licensedir=/usr/share/licenses/PyArrow/`) should correctly install the license. Building a standalone `meson dist` tarball release (similar to Autotools `make distcheck`) will not try to resolve the license, at least not currently. My thoughts were that it is *obviously incorrect* to mishandle a clear directive: > license-file: ../XXX.md > install-licenses: yes and we can make this "obviously correct" so that monorepos will install correctly. But it's not immediately apparent whether or not most monorepos utilizing symlinks between different directories, expect that a single directory should also have standalone source code releases (they may only support installing from the entire monorepo workspace, or they may generate per-component binary packages from the monorepo and then tell people to install from binaries). Building source code tarballs is a totally optional thing that many people don't do (both in the C/C++ world and in the python world. When was the last time you saw a Torch sdist on PyPI? When was the last time you tried to build python-tensorflow using any kind of PyPA standards at all? If some organizational design makes it inconvenient or troublesome to do an sdist, that should never be a valid reason to unnecessarily restrict wheel creation. I'm definitely interested to hear more about solutions which monorepo users would like to see, as monorepos are a completely valid use case. But it's possible that producing an sdist from a monorepo requires designing a feature, not just fixing a bug. (Or not. I don't usually work with monorepos so I'm not an expert on designing workflows for them.) And those take longer to formulate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2841992929 @WillAyd You should see it above: https://github.com/apache/arrow/pull/45854#pullrequestreview-2751010946 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2068686252 ## python/MANIFEST.in: ## @@ -1,15 +0,0 @@ -include README.md -include ../LICENSE.txt Review Comment: Ah OK; I can get these added into the meson configuration then. I recall @eli-schwartz patched something to allow for this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2841986439 @pitrou I saw a notification for a comment from you on this, but I don't see one. I am assuming it was deleted, but let me know if I am overlooking something -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2068652909 ## python/MANIFEST.in: ## @@ -1,15 +0,0 @@ -include README.md -include ../LICENSE.txt Review Comment: Yeah, it's a bug in PyArrow, because we misunderstood how MANIFEST works. See https://github.com/pypa/setuptools/issues/4892 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2048874659
##
ci/scripts/python_sdist_build.sh:
##
@@ -23,5 +23,5 @@ source_dir=${1}/python
pushd ${source_dir}
export SETUPTOOLS_SCM_PRETEND_VERSION=${PYARROW_VERSION:-}
-${PYTHON:-python} setup.py sdist
+${PYTHON:-python} -m build --sdist .
Review Comment:
The failure was happening during the release verification script, which was
testing sdist creation outside of the git VCS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2813378287 Revision: 71d36133dcb2000dc7b01b5d687dd7bb1f395e72 Submitted crossbow builds: [ursacomputing/crossbow @ actions-9556f9e16c](https://github.com/ursacomputing/crossbow/branches/all?query=actions-9556f9e16c) |Task|Status| ||--| |example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/14519477743/job/40736612383)| |example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/14519478370/job/40736617197)| |test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/14519478333/job/40736617351)| |test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/14519477462/job/40736610738)| |test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/14519478058/job/40736614638)| |test-conda-python-3.10-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14519478343/job/40736617764)| |test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/14519477806/job/40736612950)| |test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14519477854/job/40736613006)| |test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/14519477875/job/40736614305)| |test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/14519477459/job/40736611007)| |test-conda-python-3.11-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/14519478200/job/40736614729)| |test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14519478335/job/40736617162)| |test-conda-python-3.11-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14519478400/job/40736618050)| |test-conda-python-3.11-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14519478082/job/40736614736)| |test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/14519478439/job/40736618118)| |test-cond
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2813371134 @github-actions crossbow submit -g python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
assignUser commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2048809749
##
ci/scripts/python_sdist_build.sh:
##
@@ -23,5 +23,5 @@ source_dir=${1}/python
pushd ${source_dir}
export SETUPTOOLS_SCM_PRETEND_VERSION=${PYARROW_VERSION:-}
-${PYTHON:-python} setup.py sdist
+${PYTHON:-python} -m build --sdist .
Review Comment:
Isn't this being run in a checkout anyway? (in ci etc.), I assume you saw
failures somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2811070577 The minimal job failures I think are from a regression introduced by https://github.com/apache/arrow/pull/46057#issuecomment-2811068182 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2810562790 Revision: 3b907a666f9f8e40bdb44f798bf0ce1a3b03581c Submitted crossbow builds: [ursacomputing/crossbow @ actions-2b1639a26c](https://github.com/ursacomputing/crossbow/branches/all?query=actions-2b1639a26c) |Task|Status| ||--| |example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/14500960940/job/40680250684)| |example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/14500961025/job/40680252099)| |test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/14500960712/job/40680249356)| |test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/14500961398/job/40680253178)| |test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/14500961173/job/40680253365)| |test-conda-python-3.10-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14500960886/job/40680250245)| |test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/14500961066/job/40680252063)| |test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14500960718/job/40680248922)| |test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/14500961340/job/40680254256)| |test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/14500960823/job/40680249395)| |test-conda-python-3.11-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/14500961536/job/40680254220)| |test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14500960668/job/40680248902)| |test-conda-python-3.11-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14500961323/job/40680253626)| |test-conda-python-3.11-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14500961062/job/40680251966)| |test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/14500961231/job/40680253292)| |test-cond
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2810557307 @github-actions crossbow submit -g python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
mgorny commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2044645547 ## python/pyproject.toml: ## @@ -17,19 +17,21 @@ [build-system] requires = [ +"cmake < 4", Review Comment: Why do you need to add a dependency on CMake? Such a dependency is particularly problematic for downstreams, given that regular CMake installations don't provide a `cmake` Python distribution, and following the `build-system.requires` here implies having a non-distribution prebuilt CMake version installed and overriding the correct CMake executable. If such a dependency is really required, then it should be added via `get_requires_for_build_wheel()` hook if system CMake executable is not provided, much like `scikit-build-core` does. ## python/pyproject.toml: ## @@ -17,19 +17,21 @@ [build-system] requires = [ +"cmake < 4", "cython >= 3", # Starting with NumPy 1.25, NumPy is (by default) as far back compatible # as oldest-support-numpy was (customizable with a NPY_TARGET_VERSION # define). For older Python versions (where NumPy 1.25 is not yet available) # continue using oldest-support-numpy. "oldest-supported-numpy>=0.14; python_version<'3.9'", "numpy>=1.25; python_version>='3.9'", +"meson>=1.3.0", Review Comment: Is this necessary to enforce a minimum meson version? Again, this is a problematic dependency for downstreams. To the best of my knowledge, `meson-python` calls `meson` as an external executable, and we do strongly prefer that package builds used our downstream version of `meson` rather than installing another one for the purpose of building pyarrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2044767032 ## python/pyproject.toml: ## @@ -17,19 +17,21 @@ [build-system] requires = [ +"cmake < 4", "cython >= 3", # Starting with NumPy 1.25, NumPy is (by default) as far back compatible # as oldest-support-numpy was (customizable with a NPY_TARGET_VERSION # define). For older Python versions (where NumPy 1.25 is not yet available) # continue using oldest-support-numpy. "oldest-supported-numpy>=0.14; python_version<'3.9'", "numpy>=1.25; python_version>='3.9'", +"meson>=1.3.0", Review Comment: Yes Meson wants you to specify a minimum version. In more recent releases, it will throw deprecation warnings if you do not do this: https://mesonbuild.com/Release-notes-for-1-6-0.html#default-to-printing-deprecations-when-no-minimum-version-is-specified As far as 1.3.0 is concerned, I just started with that as baseline from other projects like arrow-nanoarrow, numpy, etc...If there is a need for older versions, we can definitely analyze how far back we want to go and reassess what features may need to be tweaked With that said, what is preventing you downstream from just using build isolation when installing or building the package? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2044740300 ## python/pyproject.toml: ## @@ -17,19 +17,21 @@ [build-system] requires = [ +"cmake < 4", Review Comment: Ah nice catch - I think this is a relic from the original design which also let you build Arrow C++ from source and use it as a subproject. We scrapped that in favor of requiring Arrow C++ to be installed for PyArrow to build, so this can likely be removed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
mgorny commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2805643046 > > I wonder if this change could make resolving #45576 easier > > I started down this path and ended up removing it in [9f5df6e](https://github.com/apache/arrow/commit/9f5df6e94a8eddcba82115f631935d40e15bd57e) after some feedback and to keep the scope of this more refined. However, I don't think it would be that difficult to achieve with meson-python; generally the way meson forces you to handle dependencies as subprojects helps to streamline your build configuration Yeah, I can imagine people not wanting to have to maintain translation from Meson to CMake (much like my earlier PR #45580 was rejected). However, I think this will become simpler if it's just Meson-to-Meson calling, given that C++ part seems to support Meson now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
eli-schwartz commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2044904070 ## python/pyproject.toml: ## @@ -17,19 +17,21 @@ [build-system] requires = [ +"cmake < 4", "cython >= 3", # Starting with NumPy 1.25, NumPy is (by default) as far back compatible # as oldest-support-numpy was (customizable with a NPY_TARGET_VERSION # define). For older Python versions (where NumPy 1.25 is not yet available) # continue using oldest-support-numpy. "oldest-supported-numpy>=0.14; python_version<'3.9'", "numpy>=1.25; python_version>='3.9'", +"meson>=1.3.0", Review Comment: > Is this necessary to enforce a minimum meson version? Again, this is a problematic dependency for downstreams. To the best of my knowledge, `meson-python` calls `meson` as an external executable, and we do strongly prefer that package builds used our downstream version of `meson` rather than installing another one for the purpose of building pyarrow. I mused about this at https://github.com/mesonbuild/meson-python/pull/280#issuecomment-1411507093 It would be possible for meson-python to read the version requirements from meson.build and update `get_requires_for_*` if not detected as preinstalled at runtime. This would allow maintaining the minimum required version in *one* place, in meson.build, and have it enforced everywhere. (It appears my comment there got misinterpreted, which fair enough, I didn't provide as much detail as I should have.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
mgorny commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2044848777 ## python/pyproject.toml: ## @@ -17,19 +17,21 @@ [build-system] requires = [ +"cmake < 4", "cython >= 3", # Starting with NumPy 1.25, NumPy is (by default) as far back compatible # as oldest-support-numpy was (customizable with a NPY_TARGET_VERSION # define). For older Python versions (where NumPy 1.25 is not yet available) # continue using oldest-support-numpy. "oldest-supported-numpy>=0.14; python_version<'3.9'", "numpy>=1.25; python_version>='3.9'", +"meson>=1.3.0", Review Comment: It would. It would also prevent me from using a new enough meson if it happens to be installed for a different Python version than the one the package is being built for, because then the .dist-info wouldn't be detected. As I've said regarding CMake, the usual way of dealing with this is to enforce the dependency via `get_requires_for_build_wheel`. To be honest, I'm somewhat surprised `meson-python` doesn't do that already — they seem to have correct logic for `ninja` and `patchelf`, but for some reason insist on listing `meson` as a Python dependency rather than an external dependency. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2805774399 > given that C++ part seems to support Meson now. I have been working on this but it is highly experimental and not yet a first class supported build system for Arrow C++. So in any case the CMake -> Meson transition for the Python build I _think_ will be around for a while -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2044831701 ## python/pyproject.toml: ## @@ -17,19 +17,21 @@ [build-system] requires = [ +"cmake < 4", "cython >= 3", # Starting with NumPy 1.25, NumPy is (by default) as far back compatible # as oldest-support-numpy was (customizable with a NPY_TARGET_VERSION # define). For older Python versions (where NumPy 1.25 is not yet available) # continue using oldest-support-numpy. "oldest-supported-numpy>=0.14; python_version<'3.9'", "numpy>=1.25; python_version>='3.9'", +"meson>=1.3.0", Review Comment: Ah OK thanks for clarifying. But wouldn't having this in the pyproject.toml then prevent you from a situation where you have a version in your build environment that doesn't satisfy the requirement in meson.build? Not something I want to get hung up on, but it is also worth noting that meson and meson-python are not attached to the same release schedule, so you can't simply control the version of the latter to always get the functionality from meson that you may require. See also the meson-python docs: https://mesonbuild.com/meson-python/reference/meson-compatibility.html#meson-compatibility -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
mgorny commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2044804389 ## python/pyproject.toml: ## @@ -17,19 +17,21 @@ [build-system] requires = [ +"cmake < 4", "cython >= 3", # Starting with NumPy 1.25, NumPy is (by default) as far back compatible # as oldest-support-numpy was (customizable with a NPY_TARGET_VERSION # define). For older Python versions (where NumPy 1.25 is not yet available) # continue using oldest-support-numpy. "oldest-supported-numpy>=0.14; python_version<'3.9'", "numpy>=1.25; python_version>='3.9'", +"meson>=1.3.0", Review Comment: But that document talks about setting a minimum version in `meson.build` and not `pyproject.toml`. We don't use build isolation because our purpose is to use system packages that 1) have been verified with regards to supply chain security, 2) have been tested and marked stable (rather than using whichever newest version PyPI offers), 3) may include downstream patches if necessary at the point. Just to be clear, I'm not against requiring `>=1.3.0`. I'm against enforcing an artificial Python-level dependency on a package that is used as an external program. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2805534342 > I wonder if this change could make resolving #45576 easier I started down this path and ended up removing it in https://github.com/apache/arrow/pull/45854/commits/9f5df6e94a8eddcba82115f631935d40e15bd57e after some feedback and to keep the scope of this more refined. However, I don't think it would be that difficult to achieve with meson-python; generally the way meson forces you to handle dependencies as subprojects helps to streamline your build configuration -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2033303256
##
dev/release/02-source-test.rb:
##
@@ -84,7 +84,12 @@ def test_csharp_git_commit_information
def test_python_version
source
Dir.chdir("#{@tag_name_no_rc}/python") do
- sh("python3", "setup.py", "sdist")
+ # Meson dist must be run from a VCS, so initiate a dummy repo
+ sh("git", "init", ".")
+ sh("git", "add", "--all", ".")
+ sh("git", "commit", "-m", "dummy commit for meson dist")
+ sh("python3", "-m", "pip", "install", "build")
+ sh("python3", "-m", "build", "--sdist", ".")
Review Comment:
I added this to that script. I'm not sure if you also wanted me to remove it
from this ruby script so I've left it for now, but let me know if that is
mistaken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2033794212 ## python/MANIFEST.in: ## @@ -1,15 +0,0 @@ -include README.md -include ../LICENSE.txt Review Comment: FWIW it seems like the current setuptools implementation is trying to include LICENSE.txt and NOTICE.txt within this manifest, but the source distribution contained on pypi contains neither. Not sure if that is a bug or not; for now I've just had the meson-python implementation here only match what is currently provided today (i.e. no LICENSE.txt or NOTICE.txt) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2033316983 ## ci/appveyor-cpp-build.bat: ## @@ -158,6 +192,6 @@ set PYARROW_TZDATA_PATH=%USERPROFILE%\Downloads\test\tzdata set AWS_EC2_METADATA_DISABLED=true set PYTHONDEVMODE=1 -python -m pytest -r sxX --durations=15 pyarrow/tests || exit /B - popd + +python -m pytest -r sxX --durations=15 --pyargs pyarrow || exit /B Review Comment: I've moved this to a pre-cursor in https://github.com/apache/arrow/pull/46059, since I don't think there is any downside to doing it now and its not relevant strictly to meson-python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2021270630
##
ci/scripts/python_sdist_build.sh:
##
@@ -23,5 +23,5 @@ source_dir=${1}/python
pushd ${source_dir}
export SETUPTOOLS_SCM_PRETEND_VERSION=${PYARROW_VERSION:-}
-${PYTHON:-python} setup.py sdist
+${PYTHON:-python} -m build --sdist .
Review Comment:
This is the documented way to create a source distribution via meson-python.
See also https://mesonbuild.com/meson-python/how-to-guides/sdist.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
eli-schwartz commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2022986892
##
python/meson.build:
##
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+project(
+'pyarrow',
+'cython',
+'cpp',
+version: run_command(
+'python',
+'-m',
+'setuptools_scm',
Review Comment:
It is a standalone tool that can be configured and run via setuptools as
well. At least that's where it has been moving for quite some time now.
A bunch of build backends use it as a dependency for their "${backend}-vcs"
type plugins. Hatch, flit_scm, etc.
It could be used by non-python projects too, it just adds a dependency on a
python interpreter. :p
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
pitrou commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2021252825
##
ci/scripts/python_sdist_build.sh:
##
@@ -23,5 +23,5 @@ source_dir=${1}/python
pushd ${source_dir}
export SETUPTOOLS_SCM_PRETEND_VERSION=${PYARROW_VERSION:-}
-${PYTHON:-python} setup.py sdist
+${PYTHON:-python} -m build --sdist .
Review Comment:
We're not using Meson here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2769855936 Revision: 62f8a290d85985ef017db118d6384cc641d64d05 Submitted crossbow builds: [ursacomputing/crossbow @ actions-90eb21a915](https://github.com/ursacomputing/crossbow/branches/all?query=actions-90eb21a915) |Task|Status| ||--| |example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/14200339977/job/39785504068)| |example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/14200339680/job/39785501966)| |test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/14200340486/job/39785505320)| |test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/14200340560/job/39785506191)| |test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/14200341071/job/39785509335)| |test-conda-python-3.10-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14200339661/job/39785501946)| |test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/14200340868/job/39785509311)| |test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14200340473/job/39785505324)| |test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/14200340850/job/39785509345)| |test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/14200340082/job/39785504133)| |test-conda-python-3.11-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/14200340805/job/39785508857)| |test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14200340845/job/39785508930)| |test-conda-python-3.11-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14200339739/job/39785501964)| |test-conda-python-3.11-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14200340153/job/39785504135)| |test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/14200339895/job/39785502959)| |test-cond
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2770567272 @github-actions crossbow submit -g python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2019090418
##
python/pyarrow/meson.build:
##
@@ -0,0 +1,398 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+py = import('python').find_installation(pure: false)
+
+# When NumPy 2.0 becomes the minimum we can remove the
+# custom location check
+numpy_dep = dependency('numpy', required: false)
+if not numpy_dep.found()
+incdir_numpy = run_command(
+py,
+[
+'-c',
+'''
+import os
+import numpy as np
+try:
+# Check if include directory is inside the pyarrow dir
+# e.g. a venv created inside the pyarrow dir
+# If so, convert it to a relative path
+incdir = os.path.relpath(np.get_include())
+except Exception:
+incdir = np.get_include()
+print(incdir)
+''',
+],
+check: true,
+).stdout().strip()
+
+numpy_dep = declare_dependency(include_directories: incdir_numpy)
+endif
+
+cython_args = ['--include-dir', meson.current_source_dir()]
+if get_option('buildtype') in ['debug', 'debugoptimized']
+cython_args += ['--gdb']
+endif
+
+pyarrow_srcs = files(
+'src/arrow/python/arrow_to_pandas.cc',
+'src/arrow/python/benchmark.cc',
+'src/arrow/python/common.cc',
+'src/arrow/python/datetime.cc',
+'src/arrow/python/decimal.cc',
+'src/arrow/python/extension_type.cc',
+'src/arrow/python/gdb.cc',
+'src/arrow/python/helpers.cc',
+'src/arrow/python/inference.cc',
+'src/arrow/python/io.cc',
+'src/arrow/python/ipc.cc',
+'src/arrow/python/numpy_convert.cc',
+'src/arrow/python/numpy_init.cc',
+'src/arrow/python/numpy_to_arrow.cc',
+'src/arrow/python/pyarrow.cc',
+'src/arrow/python/python_test.cc',
+'src/arrow/python/python_to_arrow.cc',
+'src/arrow/python/udf.cc',
+)
+
+# TODO: these are optional components so should detect if needed
+# if needs_csv
+pyarrow_srcs += files('src/arrow/python/csv.cc')
+#endif
+# if needs_filesystem
+pyarrow_srcs += files('src/arrow/python/filesystem.cc')
+#endif
+
+subdir('src/arrow/python')
+
+arrow_python_lib = shared_library(
+'arrow_python',
+sources: pyarrow_srcs,
+include_directories: ['src'],
+dependencies: [arrow_dep, numpy_dep, cython_generated_dep,
py.dependency()],
+cpp_args: '-DARROW_PYTHON_EXPORTING',
+override_options: ['b_lundef=false'],
+install: true,
+install_dir: py.get_install_dir() / 'pyarrow',
+)
+
+cython_modules = {
+'lib': {},
+'_compute': {},
+'_csv': {},
+'_feather': {},
+'_fs': {},
+'_json': {},
+'_pyarrow_cpp_tests': {},
+}
+
+if get_option('azure').enabled()
+cython_modules += {'_azurefs': {}}
+endif
+
+if get_option('gcs').enabled()
+cython_modules += {'_gcsfs': {}}
+endif
+
+if get_option('s3').enabled()
+cython_modules += {'_s3fs': {}}
+endif
+
+if get_option('hdfs').enabled()
+cython_modules += {'_hdfs': {}}
+endif
+
+if get_option('cuda').enabled()
+cuda_dep = dependency(
+'arrow-cuda',
+'ArrowCUDA',
+modules: ['ArrowCUDA::arrow_cuda_shared'],
+)
+cython_modules += {'_cuda': {'dependencies': cuda_dep}}
+endif
+
+if get_option('acero').enabled()
+acero_dep = dependency(
+'arrow-acero',
+'ArrowAcero',
+modules: ['ArrowAcero::arrow_acero_shared'],
+)
+
+cython_modules += {'_acero': {'dependencies': acero_dep}}
+endif
+
+if get_option('dataset').enabled()
+dataset_dep = dependency(
+'arrow-dataset',
+'ArrowDataset',
+modules: ['ArrowDataset::arrow_dataset_shared'],
+)
+
+cython_modules += {'_dataset': {'dependencies': dataset_dep}}
+endif
+
+if get_option('parquet').enabled()
+parquet_dep = dependency(
+'parquet',
+'Parquet',
+modules: ['Parquet::parquet_shared'],
+)
+
+cython_modules += {'_parquet': {'dependencies': parquet_dep}}
+
+if get_option('parquet_encryption').enabled()
+arrow_encryption_lib = shared_library(
+'arrow_python_parquet_encryption',
+sources: ['src/arrow/python/parquet_encryption.cc'],
+include_directories: ['src'],
+link_with: [arrow_python_lib],
+
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2766537617 Revision: 14c4c2e053dd570ad370371ac2c3f6a1bf4a48ed Submitted crossbow builds: [ursacomputing/crossbow @ actions-2057340b5a](https://github.com/ursacomputing/crossbow/branches/all?query=actions-2057340b5a) |Task|Status| ||--| |example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/14174825868/job/39707067868)| |example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/14174826389/job/39707071256)| |test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/14174825877/job/39707068178)| |test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/14174826933/job/39707073774)| |test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/14174826405/job/39707071520)| |test-conda-python-3.10-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14174826191/job/39707070307)| |test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/14174825735/job/39707067664)| |test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14174825999/job/39707069107)| |test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/14174825776/job/39707067715)| |test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/14174825375/job/39707066148)| |test-conda-python-3.11-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/14174826487/job/39707072060)| |test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14174826144/job/39707070019)| |test-conda-python-3.11-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14174826444/job/39707071569)| |test-conda-python-3.11-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14174825685/job/39707067018)| |test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/14174825938/job/39707068621)| |test-cond
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2022918292
##
ci/docker/conda-python-pandas.dockerfile:
##
@@ -34,3 +34,5 @@ RUN mamba install -q -y --file arrow/ci/conda_env_sphinx.txt
&& \
COPY ci/scripts/install_pandas.sh /arrow/ci/scripts/
RUN mamba uninstall -q -y numpy && \
/arrow/ci/scripts/install_pandas.sh ${pandas} ${numpy}
+
+RUN apt-get update && apt-get install -y patchelf
Review Comment:
Thanks @raulcd . Just so I understand, is your concern that if we start
producing different pyarrow packages like pyarrow-core and pyarrow-all, and
they are both installed on a system with different versions, that an import may
load symbols from the wrong libraries at runtime?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
kou commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2771107586 > * C Glib & Ruby - appears to be an environment setup issue with CMake; likely unrelated Yes. It's unrelated: https://github.com/apache/arrow/issues/45994 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2738713615 @kou I have made some offline progress on this, but one of the things I am getting stuck on is how the pyarrow C++ modules are being compiled. From what I understand, the current build process will compile Cython modules first (at least lib.pyx) and from that auto-generate `lib.h` and `lib_api.h` headers that the pyarrow modules can then reference (?) Assuming that understanding is correct, where in the process are lib.h and lib_api.h being generated? I found the CMake command that copies them from the source to the build folder, but I can't figure out where they come from in the first place. Any guidance would be appreciated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
raulcd commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2022856380
##
python/examples/minimal_build/Dockerfile.fedora:
##
@@ -27,6 +27,7 @@ RUN dnf update -y && \
make \
cmake \
ninja-build \
+pkg-config \
Review Comment:
why is this necessary?
##
python/pyarrow/meson.build:
##
@@ -0,0 +1,426 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+py = import('python').find_installation(pure: false)
+
+# When NumPy 2.0 becomes the minimum we can remove the
+# custom location check
+numpy_dep = dependency('numpy', required: false)
Review Comment:
Currently numpy is a required build dependency but an optional runtime
dependency. This seems to point we can build without numpy, is that the case?
##
ci/appveyor-cpp-build.bat:
##
@@ -118,25 +118,60 @@ pushd python
@rem Build and install pyarrow
@rem
-set PYARROW_CMAKE_GENERATOR=%GENERATOR%
-set PYARROW_CXXFLAGS=%ARROW_CXXFLAGS%
-set PYARROW_PARALLEL=2
-set PYARROW_WITH_ACERO=ON
-set PYARROW_WITH_DATASET=ON
-set PYARROW_WITH_FLIGHT=%ARROW_BUILD_FLIGHT%
-set PYARROW_WITH_GANDIVA=%ARROW_BUILD_GANDIVA%
-set PYARROW_WITH_GCS=%ARROW_GCS%
-set PYARROW_WITH_ORC=%ARROW_ORC%
-set PYARROW_WITH_PARQUET=ON
-set PYARROW_WITH_PARQUET_ENCRYPTION=ON
-set PYARROW_WITH_S3=%ARROW_S3%
-set PYARROW_WITH_SUBSTRAIT=ON
+set PYARROW_WITH_ACERO=enabled
+set PYARROW_WITH_DATASET=enabled
+
+if /i "%ARROW_BUILD_FLIGHT%" == "ON" (
+set PYARROW_WITH_FLIGHT=enabled
+) else (
+set PYARROW_WITH_FLIGHT=auto
+)
+
+if /i "%ARROW_BUILD_GANDIVA%" == "ON" (
+set PYARROW_WITH_GANDIVA=enabled
+) else (
+set PYARROW_WITH_GANDIVA=auto
+)
+
+if /i "%ARROW_BUILD_GCS%" == "ON" (
+set PYARROW_WITH_GCS=enabled
+) else (
+set PYARROW_WITH_GCS=auto
+)
+
+if /i "%ARROW_BUILD_ORC%" == "ON" (
+set PYARROW_WITH_ORC=enabled
+) else (
+set PYARROW_WITH_ORC=auto
+)
+
+set PYARROW_WITH_PARQUET=enabled
+set PYARROW_WITH_PARQUET_ENCRYPTION=enabled
+
+if /i "%ARROW_BUILD_S3%" == "ON" (
+set PYARROW_WITH_S3=enabled
+) else (
+set PYARROW_WITH_S3=auto
Review Comment:
shouldn't this be `disabled` instead of `auto`? Same for the others
##
ci/docker/conda-python-pandas.dockerfile:
##
@@ -34,3 +34,5 @@ RUN mamba install -q -y --file arrow/ci/conda_env_sphinx.txt
&& \
COPY ci/scripts/install_pandas.sh /arrow/ci/scripts/
RUN mamba uninstall -q -y numpy && \
/arrow/ci/scripts/install_pandas.sh ${pandas} ${numpy}
+
+RUN apt-get update && apt-get install -y patchelf
Review Comment:
would be good to understand this in detail, how are the symbols updated for
example, if what we want to do in the future is split the wheels into different
components, similar to what we did for conda
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2738732561 Ah nevermind I think I have figured it out. So it looks like Cython generates files in the build directory when compiling lib.pyx, so the idea is to copy those header files to a directory structure in the build directory that the sources can resolve to. I'll have to think about the best way to accomplish that via Meson. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on code in PR #45854: URL: https://github.com/apache/arrow/pull/45854#discussion_r2022893121 ## ci/appveyor-cpp-build.bat: ## @@ -118,25 +118,60 @@ pushd python @rem Build and install pyarrow @rem -set PYARROW_CMAKE_GENERATOR=%GENERATOR% -set PYARROW_CXXFLAGS=%ARROW_CXXFLAGS% -set PYARROW_PARALLEL=2 -set PYARROW_WITH_ACERO=ON -set PYARROW_WITH_DATASET=ON -set PYARROW_WITH_FLIGHT=%ARROW_BUILD_FLIGHT% -set PYARROW_WITH_GANDIVA=%ARROW_BUILD_GANDIVA% -set PYARROW_WITH_GCS=%ARROW_GCS% -set PYARROW_WITH_ORC=%ARROW_ORC% -set PYARROW_WITH_PARQUET=ON -set PYARROW_WITH_PARQUET_ENCRYPTION=ON -set PYARROW_WITH_S3=%ARROW_S3% -set PYARROW_WITH_SUBSTRAIT=ON +set PYARROW_WITH_ACERO=enabled +set PYARROW_WITH_DATASET=enabled + +if /i "%ARROW_BUILD_FLIGHT%" == "ON" ( +set PYARROW_WITH_FLIGHT=enabled +) else ( +set PYARROW_WITH_FLIGHT=auto +) + +if /i "%ARROW_BUILD_GANDIVA%" == "ON" ( +set PYARROW_WITH_GANDIVA=enabled +) else ( +set PYARROW_WITH_GANDIVA=auto +) + +if /i "%ARROW_BUILD_GCS%" == "ON" ( +set PYARROW_WITH_GCS=enabled +) else ( +set PYARROW_WITH_GCS=auto +) + +if /i "%ARROW_BUILD_ORC%" == "ON" ( +set PYARROW_WITH_ORC=enabled +) else ( +set PYARROW_WITH_ORC=auto +) + +set PYARROW_WITH_PARQUET=enabled +set PYARROW_WITH_PARQUET_ENCRYPTION=enabled + +if /i "%ARROW_BUILD_S3%" == "ON" ( +set PYARROW_WITH_S3=enabled +) else ( +set PYARROW_WITH_S3=auto Review Comment: `auto` is disabled unless you set the `-Dauto_features=enabled` option with Meson; generally you get more control setting things to auto, but its not a huge deal either way here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
kou commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2024001410
##
dev/release/02-source-test.rb:
##
@@ -84,7 +84,12 @@ def test_csharp_git_commit_information
def test_python_version
source
Dir.chdir("#{@tag_name_no_rc}/python") do
- sh("python3", "setup.py", "sdist")
+ # Meson dist must be run from a VCS, so initiate a dummy repo
+ sh("git", "init", ".")
+ sh("git", "add", "--all", ".")
+ sh("git", "commit", "-m", "dummy commit for meson dist")
+ sh("python3", "-m", "pip", "install", "build")
+ sh("python3", "-m", "build", "--sdist", ".")
Review Comment:
Can we update
https://github.com/apache/arrow/blob/main/ci/scripts/python_sdist_build.sh and
use it here?
Our release process uses it:
https://github.com/apache/arrow/blob/main/dev/tasks/python-sdist/github.yml
Hmm... It doesn't use the source archive... We may not need this test...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
eli-schwartz commented on code in PR #45854:
URL: https://github.com/apache/arrow/pull/45854#discussion_r2023065270
##
python/pyarrow/meson.build:
##
@@ -0,0 +1,426 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+py = import('python').find_installation(pure: false)
+
+# When NumPy 2.0 becomes the minimum we can remove the
+# custom location check
+numpy_dep = dependency('numpy', required: false)
Review Comment:
See also https://github.com/mesonbuild/meson/pull/12891 (ping @rgommers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2770413824 Revision: 4129195903bba424017c0d17a78ad19ff782976c Submitted crossbow builds: [ursacomputing/crossbow @ actions-0371f4be52](https://github.com/ursacomputing/crossbow/branches/all?query=actions-0371f4be52) |Task|Status| ||--| |example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779283/job/39796592934)| |example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779360/job/39796593026)| |test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779544/job/39796594864)| |test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779571/job/39796595025)| |test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779915/job/39796597269)| |test-conda-python-3.10-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779652/job/39796596554)| |test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779709/job/39796595806)| |test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779583/job/39796595460)| |test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779885/job/39796597236)| |test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779564/job/39796595199)| |test-conda-python-3.11-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779789/job/39796596504)| |test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779593/job/39796595479)| |test-conda-python-3.11-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779311/job/39796593112)| |test-conda-python-3.11-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779721/job/39796595820)| |test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/14203779363/job/39796593330)| |test-cond
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2770572870 Revision: 391c563960177b5286fc482383d1aaf02db15e13 Submitted crossbow builds: [ursacomputing/crossbow @ actions-ef200897b1](https://github.com/ursacomputing/crossbow/branches/all?query=actions-ef200897b1) |Task|Status| ||--| |example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061351/job/39800774312)| |example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061627/job/39800777103)| |test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061626/job/39800777066)| |test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061405/job/39800774910)| |test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/14205060839/job/39800772165)| |test-conda-python-3.10-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061995/job/39800778552)| |test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061687/job/39800777464)| |test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061548/job/39800775986)| |test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061017/job/39800772824)| |test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061560/job/39800775963)| |test-conda-python-3.11-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061258/job/39800773825)| |test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061648/job/39800777410)| |test-conda-python-3.11-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061174/job/39800773701)| |test-conda-python-3.11-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061473/job/39800775616)| |test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/14205061106/job/39800773471)| |test-cond
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2770499411 OK I think all of the major CI / crossbow issues are done. Right now there are 3 remaining CI jobs failing: - C Glib & Ruby - appears to be an environment setup issue with CMake; likely unrelated - test-conda-python-emcscripten - appears to be an environment setup issue with a chrome driver; likely unrelated - test-debian-12-python-3-i386: 2 test failures with floating point precision on 32 bit platform; possibly related I think any fixes to those would be very minor in nature. Seems like there is a more general question surrounding the usage of Meson while CMake is already in use by the C++ code base. I'll comment back on the original issue with findings from this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2770406383 @github-actions crossbow submit -g python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2770341510 Revision: 604512a349907ab9684c7f3d27e58a7405ef8314 Submitted crossbow builds: [ursacomputing/crossbow @ actions-44a2fae50a](https://github.com/ursacomputing/crossbow/branches/all?query=actions-44a2fae50a) |Task|Status| ||--| |example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/14203214033/job/39794780547)| |example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/14203213859/job/39794779653)| |test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/14203214182/job/39794781534)| |test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/14203213632/job/39794778400)| |test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/14203214200/job/39794781940)| |test-conda-python-3.10-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14203214243/job/39794781963)| |test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/14203214722/job/39794784026)| |test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14203214088/job/39794780493)| |test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/14203213697/job/39794778997)| |test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/14203214593/job/39794784070)| |test-conda-python-3.11-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/14203213473/job/39794778054)| |test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14203214052/job/39794780569)| |test-conda-python-3.11-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14203213834/job/39794779667)| |test-conda-python-3.11-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14203213203/job/39794777243)| |test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/14203213947/job/39794780497)| |test-cond
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2770335513 @github-actions crossbow submit -g python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
github-actions[bot] commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2770255219 Revision: 9151b55c84d7b154565ae71a75f757e445e36d06 Submitted crossbow builds: [ursacomputing/crossbow @ actions-ede5ef3990](https://github.com/ursacomputing/crossbow/branches/all?query=actions-ede5ef3990) |Task|Status| ||--| |example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/14202573699/job/39792716560)| |example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/14202574093/job/39792719789)| |test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/14202573845/job/39792718031)| |test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/14202574125/job/39792719849)| |test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/14202574087/job/39792719765)| |test-conda-python-3.10-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14202574081/job/39792719438)| |test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/14202573748/job/39792716929)| |test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14202573725/job/39792716697)| |test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/14202573489/job/39792715580)| |test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/14202574346/job/39792721363)| |test-conda-python-3.11-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/14202573744/job/39792717057)| |test-conda-python-3.11-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/14202573734/job/39792716777)| |test-conda-python-3.11-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14202573969/job/39792719459)| |test-conda-python-3.11-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/14202574257/job/39792721299)| |test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/14202573869/job/39792718041)| |test-cond
Re: [PR] GH-36411: [C++][Python] Use meson-python for PyArrow build system [arrow]
WillAyd commented on PR #45854: URL: https://github.com/apache/arrow/pull/45854#issuecomment-2770248180 @github-actions crossbow submit -g python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
