HyukjinKwon commented on code in PR #151:
URL:
https://github.com/apache/spark-connect-rust/pull/151#discussion_r4478258021
##########
crates/pyspark-rs/src/readwriter.rs:
##########
@@ -59,11 +59,35 @@ fn set_opt(
}
}
-/// Resolve a reader `schema` arg (a StructType/DataType or a DDL string) to
the
-/// reader's DDL schema.
+/// A reader `schema` as pyspark's `DataFrameReader.schema` and
`DataStreamReader.schema`
+/// send it: a StructType as its JSON, which keeps field metadata, and a
string as is, for
+/// the server to parse. Anything else raises `NOT_EXPECTED_TYPE`.
+pub(crate) fn reader_schema(schema: &Bound<'_, PyAny>) -> PyResult<String> {
+ if let Ok(schema) = schema.extract::<String>() {
+ return Ok(schema);
+ }
+ if schema
+ .extract::<PyRef<crate::types::PyStructType>>()
+ .is_ok()
+ {
+ return Ok(crate::types::py_to_data_type(schema)?.json());
Review Comment:
Thanks, good catch. Fixed in c0d52ea: `json_value` now names the SRID's CRS
as PySpark's `geo_utils` does: `geometry(OGC:CRS84)` for 4326,
`geometry(EPSG:3857)`, `geometry(SRID:0)` and `geometry(SRID:ANY)`, and
`geography(OGC:CRS84, SPHERICAL)` or `geography(SRID:ANY, SPHERICAL)`. The
parser reads these names back, so spatial types now round-trip through JSON,
and `types_roundtrip.rs` no longer excludes them. Against a local server, a
reader schema with `GeometryType(3857)`, `GeometryType(0)` or
`GeographyType(4326)` reaches the server with that SRID, as with official
PySpark. An SRID that PySpark rejects can still be constructed here; I filed
SPARK-60922 for the constructors, which also do not accept `"ANY"`.
I also took the simplification: the StructType case now takes `st.fields`
directly.
##########
python/tests/test_dropin_offline.py:
##########
@@ -3786,3 +3786,28 @@ def
test_connect_registry_sends_auto_cdc_define_flow(mock_connect):
assert 4 not in plain
assert 10 not in details and 11 not in details and 12 not in details
+
+
+# ---------------------------------------------------------------------------
reader schema
+
[email protected]("read", [
+ lambda spark, schema: spark.read.schema(schema),
+ lambda spark, schema: spark.read.csv("unused.csv", schema=schema),
+ lambda spark, schema: spark.read.load("unused", format="json",
schema=schema),
+ lambda spark, schema: spark.readStream.schema(schema),
+ lambda spark, schema: spark.readStream.csv("unused", schema=schema),
+ lambda spark, schema: spark.readStream.load("unused", format="json",
schema=schema),
+], ids=["read.schema", "read.csv", "read.load", "readStream.schema",
"readStream.csv",
+ "readStream.load"])
+def test_reader_schema_takes_a_struct_type_or_a_string_as_pyspark(read):
+ from pyspark.errors import PySparkTypeError
+
+ spark = _offline_session()
+ # pyspark sends a StructType as its JSON and a string as is, for the
server to parse.
+ read(spark, T.StructType([T.StructField("a", T.LongType(), True,
{"comment": "key"})]))
Review Comment:
Done in c0d52ea: the mock server now records AnalyzePlan requests, and the
test checks the `DataSource.schema` that each of the six readers sends, against
the JSON that official PySpark 4.3.0 writes for the StructType, and the string
as is. Against master, it fails: `read.schema()` raises `TypeError`, and the
format methods send `struct<a:bigint>`, without the field metadata.
##########
python/tests/test_dropin_offline.py:
##########
@@ -3786,3 +3786,28 @@ def
test_connect_registry_sends_auto_cdc_define_flow(mock_connect):
assert 4 not in plain
assert 10 not in details and 11 not in details and 12 not in details
+
+
+# ---------------------------------------------------------------------------
reader schema
Review Comment:
Fixed in c0d52ea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
