Zhou Yifan created SPARK-62053:
----------------------------------

             Summary: Remove outdated unsupported `ArrayType of TimestampType` 
note from `spark.sql.execution.arrow.pyspark.enabled` documentation
                 Key: SPARK-62053
                 URL: https://issues.apache.org/jira/browse/SPARK-62053
             Project: Spark
          Issue Type: Improvement
          Components: Documentation, SQL
    Affects Versions: 4.4.0
            Reporter: Zhou Yifan


The documentation for spark.sql.execution.arrow.pyspark.enabled is outdated.
It still states that Arrow optimization does not support ArrayType of 
TimestampType. However, current PySpark Arrow behavior supports nested 
timestamp structures, including arrays and maps with TimestampType and 
TimestampNTZType, as covered by existing Arrow test cases in 
python/pyspark/sql/tests/arrow/test_arrow.py.


Because of this mismatch, users may incorrectly assume that valid 
Arrow-optimized scenarios are unsupported.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to