andygrove commented on issue #7344: URL: https://github.com/apache/datafusion-comet/issues/7344#issuecomment-6830733603
Data for question 1, from a Kubernetes cluster. On TPC-H q18 at SF1000, changing only the allocator to jemalloc halved the executor memory Comet needs. That brings Comet level with, or below, Spark without Comet. **Setup** - Two executors (one per node) x 16 cores on a two-node k3s cluster (AMD Ryzen 9 7950X, 128 GB each). Node-local Parquet. - q18, 2 iterations per run; the mean is reported. One run per size. - The size is the executor pod's memory limit, split into heap + `spark.memory.offHeap.size` + `memoryOverhead`. `memoryOverhead` is 10% of the pod. - `off` is the off-heap share of heap + off-heap. - Each sweep steps down in 2 GiB steps and stops at the first failure. A size counts as "clean" when it passed with no executor OOM-killed. - Baseline: the released Comet 1.2.0 jar on Spark 4.1.3 (glibc). - jemalloc: the same 1.2.0 tag built with `--features jemalloc` and `RUSTFLAGS=-Ctarget-cpu=x86-64-v3`, the release jar's flags. No `_RJEM_MALLOC_CONF`, so jemalloc's default decay. **Smallest clean pod size** | Config | glibc (1.2.0 release) | jemalloc | |---|---:|---:| | Comet, `off=0.75` | 20 GiB (150 s) | **10 GiB** (188 s) | | Comet, `off=0.5` | 24 GiB (160 s) | **12 GiB** (178 s) | | Spark without Comet, best split (50% off-heap) | 12 GiB (387 s) | | - With glibc, every failure was a cgroup OOM kill of an executor pod. None was a `Resources exhausted` from Comet's pool or a Java OOM, so it matches #7108. - With jemalloc, `off=0.75` passed cleanly at 20, 18, 16, 14, 12 and 10 GiB, and failed at 8 GiB with 5 OOM kills. - The same is true of `off=0.5` at 20 to 12 GiB. At 10 GiB it is OOM-killing executors (that run is still finishing). **Resident memory.** At 20 GiB with `off=0.75`, the most anonymous memory any executor held was: - 18.5 GiB with glibc; - 10.4 GiB with jemalloc. The pool size was the same (13.5 GiB off-heap). With jemalloc, the most stayed between 8 and 11.4 GiB at every size from 20 GiB down to 10 GiB. **Speed** (question 2, for this one query only): - At the same 20 GiB, `off=0.75`, jemalloc took 154.6 s against 150.1 s for glibc, which is within noise for single runs. - With jemalloc, time rises as the pod shrinks, to 188 s at 10 GiB, presumably from more spilling. - That is still about 2x faster than Spark at its 12 GiB minimum (387 s). Caveats: one query, one run per size, and only the memory split was tuned. I haven't run TPC-DS or the full TPC-H suite, and haven't measured on aarch64. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
