[
https://issues.apache.org/jira/browse/SPARK-61363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61363:
-----------------------------------
Labels: pull-request-available (was: )
> Match PySpark's literals, Observations, caches, nanosecond type names and
> local iterators in the Python drop-in
> ---------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61363
> URL: https://issues.apache.org/jira/browse/SPARK-61363
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Low-priority findings of the s16 review of the Python drop-in, each checked
> against official PySpark 4.3.0:
> - lit(), isin, when and sql args took an object that has only __index__ or
> only __float__ (or a Fraction) as an int or a double; PySpark raises
> UNSUPPORTED_DATA_TYPE.
> - DataFrame.transform did not assert that the function returned a DataFrame;
> PySpark raises AssertionError "Func returned an instance of type [<class
> 'int'>], should have been DataFrame."
> - Observation.get of a metric that PySpark's LiteralExpression._to_value does
> not convert (a year-month interval, a nanosecond timestamp) returned an
> internal marker; PySpark raises PySparkTypeError UNSUPPORTED_LITERAL from the
> action, and then gives {} from get.
> - _parse_datatype_json_value, _parse_datatype_json_string and the fromJson of
> StructType, StructField and ArrayType took the nanosecond names
> timestamp_ntz(9) and timestamp_ltz(9); PySpark raises CANNOT_PARSE_DATATYPE.
> TimestampNTZNanosType and TimestampLTZNanosType were attributes of
> pyspark.sql.types, which PySpark does not have.
> - lit(pd.Timestamp("...123456789")) emitted pandas' UserWarning "Discarding
> nonzero nanoseconds in conversion"; PySpark takes the microseconds silently.
> - A str that UTF-8 cannot encode (a lone surrogate) gave PySparkTypeError
> "Unsupported DataType `str`" from lit and sql args; PySpark raises protobuf's
> ValueError "... isn't a valid unicode string and can't be encoded in UTF-8."
> - toLocalIterator(prefetchPartitions=True) called the progress handlers on a
> runtime thread ahead of the consumer, and an exception of a handler that the
> consumer never reached stayed referenced for good; PySpark Connect ignores
> prefetchPartitions and calls the handlers in next().
> - isStreaming, semanticHash, isEmpty and inputFiles asked the server at each
> call; PySpark caches them per DataFrame (cached_property and
> functools.cache), also after stop().
> - toArrow of invalid UTF-8 raises, where PySpark returns a table that is
> invalid Arrow: recorded in dev/parity/known-differences.md rather than
> reproduced.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]