bearomorphism opened a new pull request, #59392: URL: https://github.com/apache/spark/pull/59392
### What changes were proposed in this pull request? - Use Spark's generic `CachedAccessor` for `DataFrame.pandas_on_spark` and `Series.pandas_on_spark`. - Add descriptor overloads to distinguish class access from instance access. - Update batch callback signatures to accommodate both pandas return annotations and pandas-on-Spark schema annotations, with localized casts for pandas execution. ### Why are the changes needed? The untyped pandas descriptor makes these accessors `Any`, hiding method and callback type errors. The typed descriptor exposes the actual accessor classes while preserving lazy creation and caching. The callback changes accommodate the existing schema-annotation convention without adding suppressions. ### Does this PR introduce _any_ user-facing change? Static type checkers now recognize the accessor types and check their method calls. Runtime accessor and batch behavior is unchanged. ### How was this patch tested? - Mypy passed on `python/pyspark/pandas` and a local probe checking instance/class accessor types. A negative probe correctly rejected callbacks annotated to return integers. - Existing `pyspark.pandas.tests.test_extension` tests passed. - Existing batch tests passed: `FrameApplyFunctionTests.test_apply_batch`, `test_apply_batch_with_type`, `test_transform_batch`, `test_transform_batch_with_type`, and `test_transform_batch_same_anchor`; `CategoricalTests.test_series_transform_batch` and `test_series_transform_batch_without_shortcut`. - A runtime smoke check verified class access, lazy creation, cached identity, and owner binding. - Ruff lint/format checks and `git diff --check` passed. No new test files were added; validation used existing tests and local type/runtime probes. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: pi 1.1.0 with OpenAI gpt-6-astra -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
