Spenserrrr opened a new pull request, #58324: URL: https://github.com/apache/spark/pull/58324
### What changes were proposed in this pull request? Wrap the JVM error message in `str()` when building `messageParameters` in `ArrowCollectSerializer.load_stream`. ### Why are the changes needed? `PySparkRuntimeError` declares `messageParameters: Optional[Dict[str, str]]`, but `UTF8Deserializer.loads` is unannotated and its body can return `str`, raw bytes, or `None` (on a `SpecialLengths.NULL` length). Type checkers that infer return types from unannotated bodies (Pyright, for instance, which backs the Python language server in several editors) therefore report `dict[str, str | Unknown | None]` as not assignable to `Dict[str, str]`. `str()` makes the value match the declared type and follows the idiom already used for `messageParameters` a few classes down in the same file. `dev/lint-python` does not flag this: mypy treats a call to an unannotated function as returning `Any`, and `pyspark.serializers` is exempt from `disallow_untyped_defs` in `python/mypy.ini`. ### Does this PR introduce _any_ user-facing change? No. The error framework already stringifies parameters during template substitution, so the rendered message is identical. ### How was this patch tested? Type checkers on the changed file: mypy with `dev/lint-python`'s invocation stays clean, and Pyright goes from 1 error to 0. Behavior was checked against a simulated collect stream (Arrow batches, then `-1`, then the message), since the branch needs a JVM-side failure to reach: a normal message still renders `An error occurred while calling ArrowCollectSerializer.load_stream: java.lang.RuntimeException: boom.`, a NULL length renders the same `None` as before the change, and out-of-order indices `(2, 0, 1)` still reorder correctly. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
