dongjoon-hyun opened a new pull request, #552:
URL: https://github.com/apache/spark-connect-swift/pull/552
### What changes were proposed in this pull request?
This PR aims to support `Codable`-based DataFrame creation and collection in
`SparkSession` and `DataFrame`, leveraging the in-tree `ArrowEncoder` and
`ArrowDecoder`.
Specifically:
1. **`SparkSession.createDataFrame` with `Encodable`**:
- `createDataFrame<T: Encodable>(_ data: [T]) async throws -> DataFrame`:
Automatically encodes an array of `Encodable` instances into an Apache Arrow
`RecordBatch` via `ArrowEncoder`, infers the Spark DDL schema from the batch
schema, and builds a `LocalRelation` or `CachedLocalRelation` (for payloads >=
1MiB).
- `createDataFrame<T: Encodable>(_ data: [T], _ schema: String)` and
`createDataFrame<T: Encodable>(_ data: [T], _ schema: StructType)`: Supports
explicit schema overrides and empty datasets with schema.
2. **`DataFrame.collect(as:)` with `Decodable`**:
- `collect<T: Decodable>(as type: T.Type = T.self) async throws -> [T]`:
Executes the plan and directly decodes each Arrow `RecordBatch` into `[T]`
using `ArrowDecoder`, eliminating intermediate untyped `Row` allocations and
dictionary boxing.
3. **`ArrowEncoder` and `ArrowDecoder` enhancements**:
- Added support for Swift native `Int` (mapped to `Int64`) and `UInt`
(mapped to `UInt64`) across keyed, unkeyed, and single-value containers.
- Added support for decoding `Date` and `TimestampNanos` from Arrow
`Timestamp` columns (seconds, milliseconds, microseconds, and nanoseconds).
4. **Schema mapping utilities**:
- Added `DataType.init(_ arrowType: ArrowType) throws` and
`StructType.init(_ arrowSchema: ArrowSchema) throws` to translate Arrow schemas
to Spark SQL data types.
```swift
struct Person: Codable, Sendable, Equatable {
let name: String
let age: Int
}
let people = [Person(name: "Alice", age: 20), Person(name: "Bob", age: 25)]
// 1. Create DataFrame directly from Swift models (auto-inferred schema)
let df = try await spark.createDataFrame(people)
// 2. Collect query results directly into Swift models
let results: [Person] = try await df.filter("age >= 21").collect(as:
Person.self)
```
### Why are the changes needed?
Currently, `SparkSession.createDataFrame` only accepts untyped rows
`[[Sendable?]]` with an explicit DDL string or `StructType` schema, and
`DataFrame.collect()` only returns untyped `[Row]`.
These new APIs provide a type-safe, Dataset-like programming experience in
Swift, allowing developers to work seamlessly with native Swift `Codable`
structs.
### Does this PR introduce _any_ user-facing change?
No behavior change because this PR adds only new public generic methods.
### How was this patch tested?
Pass the CIs with the newly added test cases.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Gemini 3.8 Flash (High)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
