Hyukjin Kwon created SPARK-62066:
------------------------------------
Summary: [Connect][Python] Drop-in: take local rows apart by the
cached names of a StructType changed in place, raise PySpark's errors of the
types that proto_schema_to_pyspark_data_type does not convert, and raise the
MALFORMED conditions of no parameters
Key: SPARK-62066
URL: https://issues.apache.org/jira/browse/SPARK-62066
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
1. A StructType whose fields list was changed in place. PySpark's StructType
caches names as it is made (python/pyspark/sql/types.py:1471) and as add adds a
field (types.py:1541, 1557), and fieldNames() is list(self.names)
(types.py:1761), so st.fields.append(StructField("b", StringType())),
st.fields.pop() or st.fields[0] = ... leaves the names as they were. Its
conversion of local rows takes each row apart by schema.fieldNames()
(python/pyspark/sql/conversion.py:941-975, to_row), and the converter of a
struct below the top level its values (conversion.py:605-660), and converts the
values by position with the fields as they are (conversion.py:978-1003): the
arity of a row and the keys of a dict are of the cached names, a field past
them has no value (IndexError of row[i]), and below the top level a value past
the fields has no converter and fewer names than fields fail
_deduplicate_field_names (python/pyspark/sql/pandas/types.py:1630-1653), both
IndexError. createDataFrame of a pandas DataFrame names it with
toDF(*schema.fieldNames()) (python/pyspark/sql/connect/session.py:608, 793).
The drop-in took the rows apart by the names of the fields as they are.
st = StructType([StructField("a", LongType())])
st.fields.append(StructField("b", StringType()))
spark.createDataFrame([(1, "q")], st)
# PySpark: PySparkValueError [AXIS_LENGTH_MISMATCH] Length mismatch:
Expected axis has 1 elements, new values have 2 elements.
# drop-in: [Row(a=1, b='q')]
spark.createDataFrame([(1,)], st)
# PySpark: IndexError: tuple index out of range; drop-in:
AXIS_LENGTH_MISMATCH (2, 1)
st = StructType([StructField("a", LongType())])
st.fields[0] = StructField("z", StringType())
spark.createDataFrame([{"a": 1}], st).collect()
# PySpark: [Row(z='1')]; drop-in: [Row(z=None)]
An element that is not a StructField, such as a str appended, raises the
drop-in's TypeError ("StructType fields must be StructField objects") wherever
the native type lowers its fields, where PySpark's Python methods raise the
AttributeError of the first attribute that they read; this is inherent to the
native type and documented.
2. proto_schema_to_pyspark_data_type of a type that it does not convert.
PySpark's last branch raises PySparkValueError(UNSUPPORTED_OPERATION,
{"operation": f"data type {schema}"}) of the proto text of the type
(python/pyspark/sql/connect/types.py:218-333), for a type of no kind, such as
pb2.DataType() or the element type that an array does not set, and of a kind
that it has no branch of, such as an unparsed type, in the order of its
conversion. It makes a user-defined type of its Python class, of no conversion
of its SQL type.
proto_schema_to_pyspark_data_type(pb2.DataType())
# PySpark: PySparkValueError [UNSUPPORTED_OPERATION] data type is not
supported.
# drop-in: PySparkRuntimeError DataType kind not set
dt = pb2.DataType(); dt.unparsed.data_type_string = "int"
proto_schema_to_pyspark_data_type(dt)
# PySpark: PySparkValueError [UNSUPPORTED_OPERATION] data type unparsed {
data_type_string: "int" } is not supported.
# drop-in: UnparsedDataType('int')
3. The MALFORMED conditions of the conversion of local rows. PySpark's
converters raise PySparkValueError(errorClass="MALFORMED_VARIANT")
(conversion.py:889), MALFORMED_GEOGRAPHY (903) and MALFORMED_GEOMETRY (917)
without messageParameters, so getMessageParameters() is None. The drop-in's are
{}.
spark.createDataFrame([("x",)], "v variant") # getMessageParameters():
PySpark None, drop-in {}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]