dongjoon-hyun opened a new pull request, #497: URL: https://github.com/apache/spark-connect-swift/pull/497
### What changes were proposed in this pull request? This PR proposes to send the precision of `TIME(p)` columns as `Apache Arrow` field metadata when `createDataFrame` serializes local data into an `Apache Arrow` IPC stream. The `Apache Arrow` `Time` type has no fractional-second precision field, so `Apache Spark` carries it in the field metadata under the `SPARK::time::precision` key. See `toPrecisionTaggedArrowField` / `fromArrowField` in [ArrowUtils.scala](https://github.com/apache/spark/blob/v4.3.0-rc1/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala). - `ArrowField` gains an optional `metadata`, defaulting to `nil`. - `ArrowWriter.writeField` writes it into the FlatBuffers `Field.custom_metadata` vector, sorted by key so that the same input yields a byte-identical stream. - `ConvertToArrow` attaches the key to `TIME` columns. `ArrowSchema.swift` and `ArrowWriter.swift` are vendored from [apache/arrow-swift](https://github.com/apache/arrow-swift). Their changes are deliberately Spark-agnostic and worth contributing back upstream; the `SPARK::time::precision` key lives entirely in `ConvertToArrow`. ### Why are the changes needed? Without the metadata, an `Apache Spark` 4.3.0 server reads a `time64[ns]` field as the canonical `TIME(6)` and rejects any other declared precision: ```swift try await spark.createDataFrame([[time]], "t TIME(0)") ``` ``` internalError: "[INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `col_0` is of type "TIME(6)" while it's required to be "TIME(0)". SQLSTATE: 42000" ``` ### Does this PR introduce _any_ user-facing change? Yes. `createDataFrame` with a `TIME(p)` schema now works for every precision against `Apache Spark` 4.3.0 and later, where it previously failed for any `p` other than `6`. ### How was this patch tested? Pass the CIs with the existing tests first. Apache Spark 4.3.0 integration test will be added soon. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
