NestDream commented on code in PR #59038:
URL: https://github.com/apache/spark/pull/59038#discussion_r4218985490
##########
python/pyspark/sql/tests/test_types.py:
##########
@@ -305,6 +306,42 @@ def test_infer_schema_upcast_boolean_to_string(self):
df = self.spark.createDataFrame([[True, 1], ["false", 1]],
schema=["a", "b"])
self.assertEqual([Row(a="true", b=1), Row(a="false", b=1)],
df.collect())
+ def test_infer_schema_row_length_mismatch(self):
+ # SPARK-59781: the schema is inferred from the first row. A later row
with more values used
+ # to be silently truncated when a field needed a converter (the string
column here), and a
+ # Row with an extra key came back with the values shifted (Row(a="3",
b=None)).
+ for data in [
+ [("a", 1), ("b", 2, 3)],
+ [Row(a="x", b=1), Row(c=3, a="y", b=2)],
+ [("a", 1), ("b",)],
+ ]:
+ with self.subTest(data=data):
+ rdd = self.sc.parallelize(data)
+ with self.assertRaisesRegex(PythonException,
"FIELD_STRUCT_LENGTH_MISMATCH"):
+ self.spark.createDataFrame(rdd).collect()
+
+ def test_create_dataframe_row_length_mismatch_without_verification(self):
+ # SPARK-59781: with verifySchema=False the type verifier is skipped
and a row longer than
Review Comment:
For this test yes, the explicit-schema path already rejects it under
`verifySchema=True`, so `verifySchema=False` stays as is here.
The problem is the inferred path. `verifySchema` is simply not read there:
`_create_dataframe` builds the verifier only for a `StructType` or `DataType`,
and for `schema=None` or a names list `prepare` is the identity
([session.py#L1692-L1718](https://github.com/apache/spark/blob/db7a6b6a8d5d7b0ba6f9e8a6a142e4f4daed2ae2/python/pyspark/sql/session.py#L1692-L1718)).
So
spark.createDataFrame(sc.parallelize([("a", 1), ("b", 2, 3)]),
verifySchema=True).collect()
gives `[Row(_1='a', _2=1), Row(_1='b', _2=2)]` on master, and `Row(c=3,
a="y", b=2)` after `Row(a="x", b=1)` gives `Row(a='3', b=None)`. A doc note
can't tell people to enable anything for that case.
This is also new in 4.1 for string/numeric rows. 4.0.4 fails both calls in
the JVM with `STRUCT_ARRAY_LENGTH_MISMATCH`. SPARK-52433 (#51140) routed
`StringType` through the converter (no such branch in [v4.0.0
types.py#L2502-L2512](https://github.com/apache/spark/blob/v4.0.0/python/pyspark/sql/types.py#L2502-L2512)),
so the `zip` in `convert_struct` now drops the extra values before the JVM
sees them. Date and nested columns were truncated on 4.0 as well.
Agreed on not breaking people. Pushed da8f73e0af5: it only warns now, once
per `createDataFrame` call (once per partition per job on an RDD), results
unchanged, checked against master cell by cell. Trade-offs of the other
variants will be put in the PR comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
