comphead opened a new pull request, #6401:
URL: https://github.com/apache/datafusion-comet/pull/6401
## Which issue does this PR close?
There is no issue for this change.
## Rationale for this change
Each executor logs its native memory usage every
`spark.comet.memory.logInterval`, 10 seconds by default, and the memory tuning
guide sizes `spark.executor.memoryOverhead` from the most untracked memory any
executor logged during a representative run. Executor logs are spread over the
cluster, are collected differently on YARN and Kubernetes, and often go away
with their containers, so finding that peak means gathering every executor's
log first. The event log already keeps the rest of the application's history in
one place, next to the jobs and stages the samples ran alongside, and it
outlives the application.
## What changes are included in this PR?
- `CometExecutorMemoryUsage` is a new `SparkListenerEvent` that carries one
sample: the figures of the log line in bytes (`nativeAllocated`,
`poolsReserved`, `pools`, `plans`, `jvmArrowAllocated`, `jvmArrowImported`),
the `executorId`, and the executor's `time`. `EventLoggingListener` writes an
event it has no format of its own for with Jackson, as one JSON line with
`"Event"` set to the class name. The history server does not display it. One
without Comet on its classpath logs once that it dropped it.
- `CometExecIterator`: when `spark.eventLog.enabled` is true and the
executor runs the Comet plugin, each sample goes to the driver instead of the
executor's log, and the line is logged at DEBUG only. Executors receive the
driver's `spark.*` settings, so the executor's conf shows whether the driver
writes an event log. Samples keep the log's cadence: while native plans run,
and once after the last one finishes.
- Without the plugin, as when Comet is enabled through
`spark.sql.extensions` alone, the line stays at INFO, and the executor says why
once when the log starts.
- A failed send puts the log back on the executor's log for good, after
one warning.
- The warning about exceeding the executor's container always goes to the
executor's log.
- `CometExecutorPlugin` keeps its `PluginContext` for this and clears it at
shutdown, unless a later plugin in the same JVM has already replaced it.
`CometDriverPlugin.receive` posts the sample to the listener bus and returns
null, because the message is one-way.
- Docs: the memory tuning guide gains a section on reading the samples from
the event log, with a `jq` recipe that lists each executor's peak untracked
memory. The config description, the plugin overview and the memory management
guide mention the new destination.
Each executor adds one event per interval while it runs native plans, and
`EventLoggingListener` flushes the log for every event of this kind. That is
little at the default 10 seconds, but a 1 second interval on a large cluster
makes the event log noticeably larger. The guide says so.
## How are these changes tested?
- `CometPluginsEventLogSuite` (new) runs with the plugin and an uncompressed
event log. It sends a sample through the executor plugin, then checks that a
listener on the driver receives it and that the event log file holds it, read
back with `JsonProtocol`.
- `CometPluginsSuite` gains a test that a sample is not sent when the
executor runs the plugin but the application writes no event log.
- `CometExecIteratorLifecycleSuite` gains a test that the event's fields map
from `Native.getMemoryUsage` and the JVM Arrow figures, and that the event
round-trips through `JsonProtocol`, which the event log and the history server
both use.
The suites run in local mode, where the executor plugin reaches the driver
plugin inside one JVM. Across JVMs, Spark's plugin RPC Java-serializes the
sample, which a case class supports, but no test covers that path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
