Yicong Huang created SPARK-58699:
------------------------------------
Summary: Short-circuit identity conversion for string and binary
in LocalDataToArrowConversion
Key: SPARK-58699
URL: https://issues.apache.org/jira/browse/SPARK-58699
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.4.0
Reporter: Yicong Huang
LocalDataToArrowConversion converts local Python rows to Arrow element by
element. For string and binary leaf types the per-element converter is
effectively a no-op when the element is already the target Python type:
convert_string returns str inputs unchanged (str(s) is s), and convert_binary
copies bytes inputs even though bytes is immutable.
This adds a type-check fast path to convert_string and convert_binary, and
inlines it into convert_array for string/binary element types so
already-target-typed elements skip the per-element converter call. Non-target
elements (e.g. a bool coerced to string, or a bytearray) still fall back to the
existing converter, so the behavior is unchanged.
This is orthogonal to SPARK-52796, which optimized the outer convert() assembly
rather than the element-level conversion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
