[ 
https://issues.apache.org/jira/browse/SPARK-62013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-62013.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 856
[https://github.com/apache/spark-connect-rust/pull/856]

> Make the result objects of a DataFrame PySpark's: the bytes WKB of spatial 
> values, the selfDestruct copy of toPandas, one executionInfo object and the 
> Arrow metadata of _to_table
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62013
>                 URL: https://issues.apache.org/jira/browse/SPARK-62013
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> The drop-in for pyspark-client 4.3.0 gives four results of a DataFrame that 
> differ from PySpark's, each against a Spark 4.3.0 server:
> 1. Spatial values under spark.sql.execution.pyspark.binaryAsBytes=false. 
> PySpark's ArrowTableToRowsConversion makes a Geometry or a Geography of the 
> `wkb` bytes of its Arrow struct (python/pyspark/sql/conversion.py:1387-1419), 
> whatever the setting, which makes only the binary values bytearray 
> (conversion.py:1313-1322). The drop-in's rows give the struct's binary as 
> bytearray, of which pyspark.sql.types._udt_converter made the spatial value.
>      spark.conf.set("spark.sql.execution.pyspark.binaryAsBytes", "false")
>      wkb = "x'0101000000000000000000f03f0000000000000040'"
>      row = spark.sql(f"select st_geomfromwkb({wkb}) g, st_geogfromwkb({wkb}) 
> h, array(st_geomfromwkb({wkb})) a").collect()[0]
>      print([type(v.getBytes()).__name__ for v in (row.g, row.h, row.a[0])])
>    pyspark-client 4.3.0: ['bytes', 'bytes', 'bytes']
>    drop-in: ['bytearray', 'bytearray', 'bytearray'] (also of toLocalIterator)
> 2. toPandas with spark.sql.execution.arrow.pyspark.selfDestruct.enabled=true. 
> PySpark's SparkConnectClient._execute_and_fetch copies each column of each 
> batch into its own allocation with pa.concat_arrays before the conversion 
> (python/pyspark/sql/connect/client/core.py:1994-2013). The drop-in's 
> DataFrame.toPandas did not.
>      spark.conf.set("spark.sql.execution.arrow.pyspark.selfDestruct.enabled", 
> "true")
>      spark.sql("select interval '1-2' year to month a").toPandas()
>      spark.range(3).select().toPandas().shape
>    pyspark-client 4.3.0: KeyError: 21 (the copy wraps the month_interval 
> array, of which PyArrow has no Python class; also of the empty result of `... 
> where false`), and (0, 0), as a copied batch without columns has no rows.
>    drop-in: pyarrow.lib.ArrowNotImplementedError: No known equivalent Pandas 
> block for Arrow data of type month_interval is known., and (3, 0).
> 3. DataFrame.executionInfo. PySpark's property returns the `_execution_info` 
> that the last execution of the DataFrame set 
> (python/pyspark/sql/connect/dataframe.py:2434-2436). The drop-in made a new 
> ExecutionInfo at each access.
>      df = spark.range(3); df.collect()
>      a = df.executionInfo; a.custom = 1
>      print(a is df.executionInfo, getattr(df.executionInfo, "custom", 
> "missing"), a.metrics is df.executionInfo.metrics)
>    pyspark-client 4.3.0: True 1 True (also after df.take(1), which runs a 
> derived DataFrame)
>    drop-in: False missing False
> 4. DataFrame._to_table(). PySpark's reads the Arrow IPC stream of the server 
> with PyArrow (core.py:1889, dataframe.py:1980-1986). The server writes the 
> metadata of the schema and of every field, an empty one where there is none, 
> which PyArrow reads as {}, and the keys of a field's metadata in its order. 
> The drop-in's table, of the batches that arrow-rs decodes, had None for each, 
> and the keys in any order.
>      wkb = "x'0101000000000000000000f03f0000000000000040'"
>      t, _ = spark.sql(f"select st_geomfromwkb({wkb}) g, 1 i")._to_table()
>      print(t.schema.metadata, t.schema.field("i").metadata, 
> t.schema.field("g").type.field(0).metadata, 
> t.schema.field("g").type.field(1).metadata)
>    pyspark-client 4.3.0: {} {} {} {b'geometry': b'true', b'srid': b'0'}
>    drop-in: None None None and the keys of the last in either order, from run 
> to run.
> Expected: the four as PySpark's.
> Two docs are corrected too:
> - dev/parity/known-differences.md: a timestamp whose wall clock in the 
> session time zone is outside Python's datetime range (years 1 to 9999), and 
> in the local time zone is not, raises OverflowError ("date value out of 
> range") in PySpark's collect, toLocalIterator and first, which localize it in 
> the session time zone; the drop-in returns the instant in the local time 
> zone, such as datetime(1, 1, 1, 0, 0) of timestamp_seconds(-62135596800) with 
> the session time zone America/Los_Angeles on a host in UTC. This is 
> intentional, as for the session time zones that Python does not know, and is 
> documented, not changed.
> - AGENTS.md said that crates/pyspark-rs/src/row.py embeds PySpark's Row 
> class. It is a port without PySpark's annotations and the @overloads of 
> __new__ (inspect.signature(Row.asDict) is (self, recursive=False), PySpark's 
> (self, recursive: bool = False) -> Dict[str, Any]).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to