david-mollitor-db opened a new pull request, #58894:
URL: https://github.com/apache/spark/pull/58894
### What changes were proposed in this pull request?
`CoarseGrainedClusterMessages.StatusUpdate` is sent from executor to driver
on every task state
change (`RUNNING` at start, `FINISHED`/`FAILED`/`KILLED` at end) -- roughly
two or more per task,
on the driver's RPC intake path. This makes it `Externalizable` with a
compact manual encoding
instead of relying on default Java serialization.
The manual form mirrors `UpdateBlockInfo` in the same message family, and
reuses
`TaskDescription`'s exact wire form for the fractional-CPU `BigDecimal`:
- `state` -> one byte (`.id`) / `TaskState(id)`
- `taskCpus` -> normalized decimal string via `CpuAmount.toDisplayString` /
`CpuAmount.normalize`
- `data` -> `SerializableBuffer`'s existing channel-based serialization (no
extra copy for large
task results)
- `resources` -> a size-prefixed nested map
### Why are the changes needed?
Measuring the Java-serialized size (a fresh `ObjectOutputStream` per RPC, so
no class-descriptor
caching) shows an empty-payload `StatusUpdate` is **1713 bytes**, dominated
by two fields:
| Field | Serialized bytes |
|---|---:|
| `state` (a Scala `Enumeration` value) | 642 |
| `taskCpus` (a `BigDecimal`) | 638 |
| `resources` (empty `Map`) | 150 |
| `data` (empty `SerializableBuffer`) | 68 |
| full `StatusUpdate` | 1713 |
A Scala `Enumeration.Value` serializes a reference to its enclosing
`Enumeration` object, so it
drags in the whole `TaskState` object; `scala.math.BigDecimal` drags in
`java.math.BigDecimal` +
`BigInteger` + `MathContext` + `RoundingMode` descriptors. The actual
payload is ~31 bytes.
The manual encoding shrinks the empty-payload message from **1713 bytes to
191 bytes** (~9x
smaller), cutting steady serialization/GC/bandwidth on a per-task-frequency
control message. This
reduces allocation churn on the driver's RPC intake path; it is not a
throughput change and no
benchmark claim is made.
### Does this PR introduce _any_ user-facing change?
No. `StatusUpdate` is an internal driver<->executor RPC message. All fields
round-trip identically
(fractional CPUs exactly, via the same `CpuAmount` form `TaskDescription`
already uses), and Java
serialization of these messages is already documented as not stable across
Spark versions.
### How was this patch tested?
New `CoarseGrainedClusterMessagesSuite` round-trips `StatusUpdate` through
`JavaSerializer` (the
RPC serializer): all fields with a non-empty payload and nested `resources`,
every `TaskState`,
fractional `taskCpus` exactly (value and scale), an empty payload, and a
size guard confirming the
message is now well under 512 bytes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
