This is an automated email from the ASF dual-hosted git repository.
pitrou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new 503cee9aee GH-51138: [Python] Fix ccache efficiency (#51139)
503cee9aee is described below
commit 503cee9aee21e1d1c65a5408349de932a405f3a9
Author: Antoine Pitrou <[email protected]>
AuthorDate: Thu Sep 3 11:16:19 2026 +0200
GH-51138: [Python] Fix ccache efficiency (#51139)
### Rationale for this change
Our current ccache support for PyArrow currently has two issues:
1. The custom configuration is set using environment variables, but those
are not passed to the build processes according to the [CMake
doc](https://cmake.org/cmake/help/latest/command/set.html#set-environment-variable)
(I have verified this by enabling ccache logging to see the configuration
values for each build command)
2. Cython-generated C++ sources are written in the temporary build
directory (because of build isolation), and therefore their path changes
everytime. Consequently, ccache would fail reusing previous compilation results.
### What changes are included in this PR?
1. Pass custom ccache configuration using command-line arguments, not
environment variables.
2. Set the ccache configuration variable `base_dir` to the temporary build
directory so that ccache strips away the build directory and hits previously
cached results obtained from a different build directory.
### Are these changes tested?
Yes, locally I confirmed that ccache now efficiently reuses compilation
outputs for Cython-generated C++ sources.
Before:
```
real 0m50,313s
user 4m22,991s
sys 0m24,805s
```
After:
```
real 0m18,698s
user 1m14,727s
sys 0m7,518s
```
### Are there any user-facing changes?
No.
* GitHub Issue: #51138
Authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
---
ci/scripts/python_build.sh | 2 +-
python/CMakeLists.txt | 17 ++++++++++++-----
2 files changed, 13 insertions(+), 6 deletions(-)
diff --git a/ci/scripts/python_build.sh b/ci/scripts/python_build.sh
index f8c1af3982..50fb37db1f 100755
--- a/ci/scripts/python_build.sh
+++ b/ci/scripts/python_build.sh
@@ -89,7 +89,7 @@ cp -aL "${source_dir}" "${python_build_dir}"
pushd "${python_build_dir}"
# - Cannot use build isolation as we want to use specific dependency versions
# (e.g. Numpy, Pandas) on some CI jobs.
-${PYTHON:-python} -m pip install --no-deps --no-build-isolation -vv -C
cmake.build-type="${CMAKE_BUILD_TYPE:-Debug}" .
+time ${PYTHON:-python} -m pip install --no-deps --no-build-isolation -vv -C
cmake.build-type="${CMAKE_BUILD_TYPE:-Debug}" .
popd
if [ "${BUILD_DOCS_PYTHON}" == "ON" ]; then
diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt
index 4bea6e7d8d..8eae91e0ed 100644
--- a/python/CMakeLists.txt
+++ b/python/CMakeLists.txt
@@ -130,11 +130,18 @@ if(CCACHE_FOUND
AND NOT CMAKE_C_COMPILER_LAUNCHER
AND NOT CMAKE_CXX_COMPILER_LAUNCHER)
message(STATUS "Using ccache: ${CCACHE_FOUND}")
- set(CMAKE_C_COMPILER_LAUNCHER ${CCACHE_FOUND})
- set(CMAKE_CXX_COMPILER_LAUNCHER ${CCACHE_FOUND})
- # ARROW-3985: let ccache preserve C++ comments, because some of them may be
- # meaningful to the compiler
- set(ENV{CCACHE_COMMENTS} "1")
+ # 1. Let ccache preserve C++ comments, because some of them may be
+ # meaningful to the compiler (ARROW-3985)
+ # 2. Set ccache base_dir to the build output directory as it is typically
+ # a temporary directory, and would otherwise fail caching because of
+ # using different paths everytime.
+ # Also, we use `cmake -E env` to set environment variables as the
+ # `ccache option=value ...` form of passing configuration options
+ # is not supported by ccache < 4.8.
+ set(ccache_command ${CMAKE_COMMAND} -E env CCACHE_COMMENTS=1
+ CCACHE_BASEDIR=${CMAKE_BINARY_DIR} -- ${CCACHE_FOUND})
+ set(CMAKE_C_COMPILER_LAUNCHER ${ccache_command})
+ set(CMAKE_CXX_COMPILER_LAUNCHER ${ccache_command})
endif()
#
