[
https://issues.apache.org/jira/browse/SPARK-62012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-62012.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 855
[https://github.com/apache/spark-connect-rust/pull/855]
> Send local rows and pandas string columns of PySpark's large Arrow types
> under spark.sql.execution.arrow.useLargeVarTypes, and compare the cache
> threshold with pyarrow's nbytes
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62012
> URL: https://issues.apache.org/jira/browse/SPARK-62012
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Problem
> PySpark 4.3.0's createDataFrame reads
> spark.sql.execution.arrow.useLargeVarTypes and, when it is
> "true", builds its Arrow table with large types:
> - the strings and binaries of the schema, also in arrays, maps, structs and
> the SQL type of a
> user-defined type, are large_string and large_binary
> (LocalDataToArrowConversion.convert, conversion.py:994-1005; to_arrow_type,
> pandas/types.py:137-139);
> - the binaries of a variant, a geometry and a geography stay binary.
> It then compares `_table.nbytes` with
> spark.sql.session.localRelationCacheThreshold to choose
> between an inline and a cached relation (connect/session.py:782). pyarrow's
> nbytes counts `len`
> offsets of a string, binary, list or map array, not the `len + 1` of its
> buffer.
> The drop-in and the Rust API read the setting, but two paths ignored it:
> - local rows: the core's Arrow conversion always made Utf8 and Binary
> (crates/spark-connect/src/session.rs, datatype_to_arrow);
> - the binding's IPC of pandas object-string columns
> (crates/pyspark-rs/src/pandas_input.rs,
> _pandas_string_ipc), which writes Utf8.
> The size that every path compared with the threshold was arrow-rs's
> get_slice_memory_size, which
> counts `len + 1` offsets: 4 bytes more per string column than PySpark's
> measure
> (crates/spark-connect/src/arrow_input.rs, batch_data_size).
> Repro
> spark.conf.set("spark.sql.session.localRelationSizeLimit", "1000")
> spark.conf.set("spark.sql.session.localRelationCacheThreshold", "100000")
> spark.conf.set("spark.sql.execution.arrow.useLargeVarTypes", "true")
> schema = StructType([StructField("s", StringType())])
> spark.createDataFrame([("abcd",)] * 10000, schema).count()
> spark.createDataFrame(pd.DataFrame({"s": ["abcd"] * 10000}),
> schema).count()
> spark.conf.unset("spark.sql.execution.arrow.useLargeVarTypes")
> spark.conf.set("spark.sql.session.localRelationCacheThreshold", "8001")
> spark.createDataFrame([("abcd",)] * 1000, schema).count()
> Actual (drop-in):
> - the first two: 10000, sent inline, as strings of 80000 bytes;
> - the third: AnalysisException [LOCAL_RELATION_SIZE_LIMIT_EXCEEDED], as it
> measured 8004
> bytes.
> Expected (pyspark-client 4.3.0):
> - the first two: AnalysisException [LOCAL_RELATION_SIZE_LIMIT_EXCEEDED], as
> large strings of
> 120000 bytes are cached;
> - the third: 1000, inline, as its table is 8000 bytes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]