dongjoon-hyun opened a new pull request, #522:
URL: https://github.com/apache/spark-connect-swift/pull/522
### What changes were proposed in this pull request?
This PR aims to support the following 10 JSON and CSV functions in two new
files,
`JsonFunctions.swift` and `CsvFunctions.swift`.
**JSON (7)**: `from_json`, `to_json`, `schema_of_json`, `get_json_object`,
`json_tuple`,
`json_array_length`, `json_object_keys`
**CSV (3)**: `from_csv`, `to_csv`, `schema_of_csv`
Unlike the previous function batches, these functions take an option map.
Apache Spark
encodes it as a plain trailing argument rather than a dedicated proto field:
both
`Column.fnWithOptions` in
`sql/api/src/main/scala/org/apache/spark/sql/Column.scala` and
`_options_to_col` in `python/pyspark/sql/connect/functions/builtin.py`
flatten the map into
`k1, v1, k2, v2, ...`, wrap it in a `map(...)` function call, and append it
as the last
argument. When there is no option, no extra argument is appended at all.
This PR mirrors
that exactly with a new internal `fn(_ name:options:_ args:)` helper in
`Functions.swift`,
sorting the keys so that the generated expression is deterministic (Swift
`Dictionary` is
unordered).
For the `schema` argument, this PR follows the existing
`DataFrameReader.schema(_ schema: String)` / `schema(_ schema: StructType)`
precedent:
`from_json` accepts a DDL `String`, a `StructType` (converted via
`StructType.toDDL`), or a
`Column` such as a `schema_of_json` result. `from_csv` accepts a DDL
`String` or a `Column`,
matching the Spark Connect Python client. `ArrayType` and `MapType` schemas
are covered by
the DDL string form, e.g. `ARRAY<STRUCT<a: INT>>`.
`json_tuple` is a generator function, and is implemented like the existing
`explode` /
`posexplode` / `inline` functions.
### Why are the changes needed?
To improve the API coverage of the Swift Spark Connect client. Currently, no
JSON or CSV
function is supported, so Swift users cannot parse or produce JSON and CSV
strings inside a
`Column` expression without falling back to `selectExpr`.
All the added functions were introduced in Apache Spark 3.5.0 or earlier, so
no server
version gate is needed.
### Does this PR introduce _any_ user-facing change?
No, this is a new feature which adds 10 new functions.
```swift
let df = try await spark.sql("SELECT 1 AS id, 'a' AS name")
let parsed = from_json(to_json(struct(col("id"), col("name"))), "id INT,
name STRING")
try await df.select(parsed.alias("parsed")).selectExpr("parsed.id",
"parsed.name").show()
try await spark.range(1).select(schema_of_json("{\"a\": 1, \"b\":
\"x\"}")).show()
// STRUCT<a: BIGINT, b: STRING>
try await df.select(to_csv(struct(col("id"), col("name")), ["sep":
";"])).show()
// 1;a
```
### How was this patch tested?
Pass the CIs with the newly added test suites, `JsonFunctionsTests` and
`CsvFunctionsTests`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
