comphead commented on PR #7356: URL: https://github.com/apache/datafusion-comet/pull/7356#issuecomment-6858779747
Thanks @andygrove, @sunchao and @mbutrovich. I simplified the PR to the profile and its docs in 3aa932fe31. - **Allocator reported wrongly.** The test, the INFO line and the `run-jemalloc-tests` CI wiring are gone, so nothing reports an allocator from the feature flag anymore. The Makefile now stops with an error when `-Pjemalloc` is combined with `COMET_FEATURES` naming `mimalloc`. - **What jemalloc covers.** The docs now say that jemalloc replaces the allocator of the Rust code in the native library. C code linked into it, such as zstd, the JVM and other native libraries still allocate through glibc. - **64 KiB pages.** The docs add that a jemalloc built for 64 KiB pages can raise memory use on 4 KiB kernels. - **Process-level jemalloc.** The docs now describe preloading a jemalloc library on the executors, for example with `spark.executorEnv.LD_PRELOAD`, as the way to cover the whole executor process with a default build. The profile is for when installing jemalloc on every executor isn't practical. Either way, the published jars carry no jemalloc. - **`make release-nogit`.** It now passes `COMET_FEATURES`, and the description says so. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
