[
https://issues.apache.org/jira/browse/SPARK-58699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58699:
-----------------------------------
Labels: pull-request-available (was: )
> Short-circuit identity conversion for string and binary in
> LocalDataToArrowConversion
> -------------------------------------------------------------------------------------
>
> Key: SPARK-58699
> URL: https://issues.apache.org/jira/browse/SPARK-58699
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 4.4.0
> Reporter: Yicong Huang
> Priority: Major
> Labels: pull-request-available
>
> LocalDataToArrowConversion converts local Python rows to Arrow element by
> element. For string and binary leaf types the per-element converter is
> effectively a no-op when the element is already the target Python type:
> convert_string returns str inputs unchanged (str(s) is s), and convert_binary
> copies bytes inputs even though bytes is immutable.
> This adds a type-check fast path to convert_string and convert_binary, and
> inlines it into convert_array for string/binary element types so
> already-target-typed elements skip the per-element converter call. Non-target
> elements (e.g. a bool coerced to string, or a bytearray) still fall back to
> the existing converter, so the behavior is unchanged.
> This is orthogonal to SPARK-52796, which optimized the outer convert()
> assembly rather than the element-level conversion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
