Zhou Yifan created SPARK-62053:
----------------------------------
Summary: Remove outdated unsupported `ArrayType of TimestampType`
note from `spark.sql.execution.arrow.pyspark.enabled` documentation
Key: SPARK-62053
URL: https://issues.apache.org/jira/browse/SPARK-62053
Project: Spark
Issue Type: Improvement
Components: Documentation, SQL
Affects Versions: 4.4.0
Reporter: Zhou Yifan
The documentation for spark.sql.execution.arrow.pyspark.enabled is outdated.
It still states that Arrow optimization does not support ArrayType of
TimestampType. However, current PySpark Arrow behavior supports nested
timestamp structures, including arrays and maps with TimestampType and
TimestampNTZType, as covered by existing Arrow test cases in
python/pyspark/sql/tests/arrow/test_arrow.py.
Because of this mismatch, users may incorrectly assume that valid
Arrow-optimized scenarios are unsupported.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]