laserninja opened a new pull request, #58787:
URL: https://github.com/apache/spark/pull/58787
### What changes were proposed in this pull request?
Copy the collation lookup before stripping provider names in
StructField.fromJson. Add regression coverage for reusing dictionaries with
collated strings, arrays, maps, and nested structs.
### Why are the changes needed?
Parsing a cached schema dictionary changes its collation metadata. A second
StructType.fromJson call on the same dictionary raises AssertionError because
the provider prefix was removed.
```python
from pyspark.sql.types import StringType, StructField, StructType
payload = StructType([StructField("s",
StringType("UTF8_LCASE"))]).jsonValue()
StructType.fromJson(payload)
StructType.fromJson(payload) # AssertionError
```
### Does this PR introduce _any_ user-facing change?
Yes. Parsing schema JSON no longer mutates the input dictionary, and
repeated parsing of the same dictionary succeeds.
### How was this patch tested?
The new regression tests failed before the fix and passed afterward on
Python 3.10.11 with NumPy 2.2.6. The focused run passed 25 tests, including
parameterized subtests:
```bash
PYTHONPATH=python TERM=dumb python -m unittest
pyspark.sql.tests.test_types.DataTypeTests
pyspark.sql.tests.test_types.DataTypeVerificationTests -q
PYTHON_EXECUTABLE=python ./dev/lint-python --compile --custom-pyspark-error
--ruff
```
Run the commands with the Python virtual environment activated; the local
run shared an environment from a sibling checkout. The tests call public Python
APIs directly and require no SparkSession. Python compilation, custom-error
checks, Ruff lint, and Ruff formatting passed. JVM-backed suites were not run
because these fixes are confined to Python-side logic.
Draft pending creation and linking of the corresponding ASF Jira issue.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
