viirya opened a new pull request, #58353:
URL: https://github.com/apache/spark/pull/58353
### What changes were proposed in this pull request?
Adds the `cpuAmount` key to the mock task-context JSON that
`MockProtocolWriter` writes in `python/benchmarks/bench_eval_type.py`.
### Why are the changes needed?
Every benchmark in `bench_eval_type.py` currently fails at worker
initialization. `TaskContextInfo.from_stream` reads
`task_context_json["cpuAmount"]` without a default
(`python/pyspark/worker_message.py`), so the worker raises `KeyError:
'cpuAmount'` before any UDF runs:
```
File "python/pyspark/worker_message.py", line 63, in from_stream
cpu_amount=Decimal(task_context_json["cpuAmount"]),
KeyError: 'cpuAmount'
```
The key became required in SPARK-58192, which added it on the JVM side and
in `worker_message.py` but did not update this mock.
The failure is easy to miss because `setup()` only builds the input buffer;
only running `time_worker` / `peakmem_worker` reaches the worker. Nothing under
`python/benchmarks/` is listed in `dev/sparktestsupport/modules.py`, so no CI
job exercises these benchmarks.
### Does this PR introduce _any_ user-facing change?
No. Benchmark-only change.
### How was this patch tested?
Ran every benchmark class in the file, driving the worker end to end:
| | before | after |
|---|---|---|
| `*TimeBench` | 0 of 26 | 26 of 26 |
| `*PeakmemBench` | 0 of 26 | 26 of 26 |
Also checked out each release branch and ran the same benchmarks there:
| branch | result |
|---|---|
| `master` | 0 of 26; 26 of 26 with this fix |
| `branch-4.x` | same failure; the same one-line fix applies |
| `branch-4.3` | 0 of 26; 26 of 26 with this fix |
| `branch-4.2` | unaffected -- no `worker_message.py`, so the key is not
required |
So this is worth backporting to `branch-4.x` and `branch-4.3`.
The value is a plain decimal string, matching `CpuAmount.toDisplayString` on
the JVM side
(`core/src/main/scala/org/apache/spark/api/python/PythonWorkerUtils.scala`),
and is consistent with the existing `"cpus": 1` in the same mock.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
