LuciferYang opened a new issue, #12676:
URL: https://github.com/apache/gluten/issues/12676
### Backend
VL (Velox). The code is backend-agnostic and lives in `gluten-core` and
`gluten-substrait`.
### Bug description
Three Spark memory configurations are declared with a unit but read through
an accessor that assumes a different one, so the off-heap budgets Gluten
derives from them are wrong.
1. `spark.executor.memory` is declared `bytesConf(ByteUnit.MiB)`, so a value
without a suffix means MiB. `SparkResourceUtil.getExecutorMemorySize` read it
with `getSizeAsBytes`, which reads a suffix-less value as bytes.
`spark.executor.memory=8192` gave 8192 bytes instead of 8 GiB, and the dynamic
off-heap sizing path derived its budget from that.
2. `spark.executor.minMemoryOverhead` is a size string, MiB unless suffixed.
`getMemoryOverheadSize` read it with `conf.getLong`, which is `String.toLong`
and cannot parse `512m`. On a value Spark itself accepts, that threw
`NumberFormatException` from `VeloxListenerApi#onDriverStart` and aborted
driver startup.
3. A `ResourceProfile` records executor memory amounts in MiB, while
`spark.gluten.memory.offHeap.size.in.bytes` and
`spark.gluten.memory.task.offHeap.size.in.bytes` are declared
`bytesConf(ByteUnit.BYTE)`.
`GlutenAutoAdjustStageResourceProfile#updateResourceSetting` wrote the profile
amount verbatim, so a 20 GiB off-heap budget became 20480 bytes.
`ByteUnit.MiB.toBytes` rejects a negative input but wraps past 2^43 MiB
without raising. `getMemoryOverheadSize` ended in `toBytes`, so
`spark.executor.minMemoryOverhead=9000000t` produced a negative overhead
budget. `VeloxListenerApi#onDriverStart` writes that into
`spark.gluten.memory.memoryOverhead.size.in.bytes`, and `VeloxBackend.cc`
multiplies it by 0.75 to size the Velox global allocator.
`updateResourceSetting` computes the task slot count itself rather than
calling `SparkResourceUtil.getTaskSlots`, and the two disagree in local mode:
`getTaskSlots` resolves `local[8]` to 8 slots, while a profile reports
`spark.executor.cores`, 1 by default. While the total was also 2^20 times too
small the two errors cancelled, so the divergence had no visible effect.
Correct the total and it becomes an 8x over-provision, with each of 8
concurrent tasks sized as if it owned the whole budget. The same expression has
no positivity check and no floor, so a profile with `CORES=1` and
`spark.task.cpus=2` throws `ArithmeticException: / by zero` on every query.
### Gluten version
main (1.8.0-SNAPSHOT)
### Spark version
Version-agnostic (applies to spark-3.3 / 3.4 / 3.5 / 4.0 / 4.1).
### Spark configurations
The divide-by-zero needs
`spark.gluten.auto.adjustStageResource.enabled=true`,
`spark.sql.adaptive.enabled=true`, `spark.memory.offHeap.enabled=true`,
`spark.task.cpus=2`, and master `local[8]`.
The unit errors need `spark.memory.offHeap.enabled=true` plus one of
`spark.executor.memory=8192`, `spark.executor.minMemoryOverhead=512m`, or
`spark.memory.offHeap.size=20g`.
### System information
Not applicable. The affected code reads configuration in
`gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala` and
`gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala`,
and does not depend on OS or hardware.
### Relevant logs
```text
java.lang.NumberFormatException: Illegal value for config key
spark.executor.minMemoryOverhead: ... Failed to parse byte string: 512m
at
org.apache.gluten.backendsapi.velox.VeloxListenerApi.onDriverStart(VeloxListenerApi.scala:116)
java.lang.ArithmeticException: / by zero
at
org.apache.spark.sql.execution.GlutenAutoAdjustStageResourceProfile$.updateResourceSetting(GlutenAutoAdjustStageResourceProfile.scala:206)
```
### Fix direction
Read each config with the accessor that matches its declared unit. Put the
MiB-to-byte conversion behind one helper in `SparkResourceUtil` that guards
both the sign and the magnitude, and route all three boundaries through it. In
`updateResourceSetting`, defer to `SparkResourceUtil.getTaskSlots` in local
mode, where the profile does not reflect the thread count, and give the other
branch the positivity check and one-slot floor that `getTaskSlots` already has.
Read the exact conf bytes when the profile is the unmodified default, since
reading through the profile truncates to MiB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
