andygrove opened a new pull request, #6051:
URL: https://github.com/apache/datafusion-comet/pull/6051
## Which issue does this PR close?
N/A — documentation only. Related to #6050, which tracks the driver-plugin
side of the same gap.
## Rationale for this change
Enabling `spark.memory.offHeap.enabled` is not sufficient on its own, and
today the docs imply it
is. `spark.memory.offHeap.size` is a budget that the cluster manager already
folds into the executor
container size, so the memory Comet's operators explicitly reserve has room.
Everything Comet
allocates without reserving it does not: per-batch working memory in
expression kernels and Arrow
array builders, decompression buffers and Parquet reader structures, object
store request buffers
and the tokio runtime, Comet's JVM-side Arrow buffers, and allocator
overhead. Those come from the
Rust global allocator and live in the native heap, and nothing in the
container sizing accounts for
them.
`spark.executor.memoryOverhead` is the only slack the container has for
that, and the JVM's own
non-heap usage is already drawing on it. Spark's default of
`max(0.1 * spark.executor.memory, 384 MiB)` leaves little behind, and
`CometDriverPlugin` does not
adjust it in off-heap mode, so users get no help from either the default or
the plugin.
## What changes are included in this PR?
- A new "Configuring Executor Memory Overhead" section in the tuning guide
explaining why the
setting matters for Comet specifically, recommending `2g` as a starting
point, and noting that
raising it grows the container on Kubernetes and YARN.
- A pointer to it from the "Memory Tuning" introduction.
- `--conf spark.executor.memoryOverhead=2g` added to the `spark-shell`
examples that already set
off-heap memory: the README getting-started example, the docs home page,
both examples in the
installation guide, both examples in the Iceberg guide, and the PyArrow
UDF `SparkSession`
example.
## How are these changes tested?
Documentation only; no code changes. `npx prettier "**/*.md" --check` passes
on the edited files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
