soaring0616 opened a new pull request, #59376:
URL: https://github.com/apache/spark/pull/59376
### What changes were proposed in this pull request?
In `worker_run` (`python/pyspark/sql/worker/utils.py`), read
`sys.modules["__main__"].__spec__` into a local variable and check `spec is not
None` before accessing `spec.name`, instead of suppressing the mypy error with
`# type: ignore[union-attr]`.
Before:
```python
worker_module = (
sys.modules["__main__"].__spec__.name # type: ignore[union-attr]
)
```
After:
```python
spec = sys.modules["__main__"].__spec__
worker_module = spec.name if spec is not None else "__main__"
```
The surrounding `try/except Exception` fallback to `"__main__"` is kept.
### Why are the changes needed?
`__spec__` is declared as `ModuleSpec | None` on `types.ModuleType` in
typeshed (`stdlib/types.pyi`), so mypy reports `union-attr` on `.name`.
At runtime CPython initializes `__spec__` to `None` for a module
(`Objects/moduleobject.c`) and it stays `None` unless the module is loaded
through the import system, for example when run with `python -m`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- `dev/lint-python --ruff` passes.
- `dev/lint-python --mypy` reports no error for
`python/pyspark/sql/worker/utils.py`. With the ignore removed and no narrowing,
it reported `Item "None" of "ModuleSpec | None" has no attribute "name"
[union-attr]`.
- Compared the old and new expressions in four cases: `__spec__` set (run
with `python -m`), `__spec__` is `None`, `__main__` missing from `sys.modules`,
and no `__spec__` attribute. The result is identical in all four.
- - Existing `pyspark.sql.tests.test_python_datasource` passes (it does not
exercise the changed lines, since `main.__module__` is the test module there).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Sonnet 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
