dongjoon-hyun opened a new pull request, #534:
URL: https://github.com/apache/spark-connect-swift/pull/534
### What changes were proposed in this pull request?
This PR aims to support 5 partition transform functions in a new file,
`Sources/SparkConnect/PartitionTransforms.swift`, and to add a `Column`-based
`DataFrameWriterV2.partitionBy` overload so that these transforms can
actually be used.
| Function | Since | Signature |
| --- | --- | --- |
| `bucket` | 3.1.0 | `(Column, Column)`, `(Int32, Column)` |
| `days` | 3.1.0 | `(Column)` |
| `hours` | 3.1.0 | `(Column)` |
| `months` | 3.1.0 | `(Column)` |
| `years` | 3.1.0 | `(Column)` |
Scala groups these under `@group partition_transforms` in `functions.scala`,
so they are
placed in a dedicated file instead of `Functions.swift`. `bucket` takes an
`Int32` overload
because PySpark accepts `Union[Column, int]` for `numBuckets`; the literal
is wrapped with
`lit(...)`, matching `partitioning.bucket` in
`python/pyspark/sql/connect/functions/partitioning.py`.
In addition, `DataFrameWriterV2.partitionBy(_ columns: Column...)` is added.
The existing
`partitionBy(_ columns: String...)` doc comment already claimed to accept
"columns or
transforms", but only string column names could be passed, so there was no
way to build a
`years(col("ts"))`-partitioned table.
### Why are the changes needed?
To improve API coverage and to make the partition transforms usable end to
end.
### Does this PR introduce _any_ user-facing change?
No behavior change. Only `DataFrameWriterV2.partitionBy` gains a new
`Column`-variadic overload. This is a pure addition; the existing
`String`-variadic overload is unchanged, so no existing code breaks. The 5
functions themselves are new additions to the API.
### How was this patch tested?
Pass the CIs with the newly added test suite, `PartitionTransformsTests`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
