viirya opened a new pull request, #146:
URL: https://github.com/apache/spark-connect-rust/pull/146
### What changes were proposed in this pull request?
This closes the Spark 4.3.0 client API gaps that remain after SPARK-59305,
in the Rust core, the PyO3 layer and the Python shim:
- **Nanosecond timestamp types.** The `TimestampNTZNanos` /
`TimestampLTZNanos` proto kinds used to decode as `TIME` and as a microsecond
`TIMESTAMP`. They now have their own `DataType` variants, `TimestampNtzNanos` /
`TimestampLtzNanos`, with proto, JSON (`timestamp_ntz(p)` /
`timestamp_ltz(p)`), DDL and Arrow (nanosecond) support. In Python they appear
as picklable `TimestampNTZNanosType` / `TimestampLTZNanosType`. These are not
added to `pyspark.sql.types.__all__`, because PySpark 4.3.0 has no Python class
for them.
- **Functions.** `instr(..., occurrence=k)` without `start` searches from
position 1, and `None` means the argument is omitted, as in 4.3.0.
`regexp_replace(..., position=None)` also treats `None` as omitted.
`variant_delete(v)` with no paths raises `CANNOT_BE_EMPTY`.
- **`SparkSession.__enter__` / `__exit__`.** Leaving the `with` block stops
the session. Like PySpark, `stop()` can now be called more than once and only
logs a warning when releasing the session fails. It also clears the active
session and the stopped session's cached `SQLContext`.
- **Spark Connect `SQLContext` / `HiveContext`.** These are ported from
v4.3.0 `pyspark/sql/connect/context.py`, and
`pyspark.sql.SQLContext.getOrCreate()` / `HiveContext.getOrCreate()` route to
them. The cache compares sessions by session id, because
`SparkSession.getActiveSession()` returns a new wrapper object on every call.
- **Pipelines `create_auto_cdc_flow`.** It now supports `stored_as_scd_type`
2, the keyword-only `track_history_column_list` /
`track_history_except_column_list`, and `spark_conf`, with the same client-side
validation as 4.3.0. `api.py` is v4.3.0 plus the existing import repointing.
The registry, PyO3 and core now send the track-history lists, `SCD_TYPE_2` and
`sql_conf`, which used to be hardcoded empty.
- **`operation_id`.**
- `DataFrame.executionInfo` now returns a
`pyspark.sql.metrics.ExecutionInfo`, including `operation_id`, instead of a
dict. `sql/metrics.py` is vendored verbatim from v4.3.0 and added to the drift
check.
- Errors from an ExecutePlan carry the operation id, read by
`SparkConnectException.operation_id`.
- In this drop-in, `SparkConnectException` is the same class as
`PySparkException`, so the property is defined on that shared base.
### Why are the changes needed?
These 4.3.0 APIs were missing or behaved differently from PySpark 4.3.0. The
nanosecond timestamp types were also decoded as the wrong types.
### Does this PR introduce _any_ user-facing change?
Yes, the APIs above now exist and behave as in PySpark 4.3.0.
`DataFrame.executionInfo` now returns an `ExecutionInfo` (or `None` before an
action) instead of a dict.
### How was this patch tested?
- New Rust unit tests: the nanos types in `types_roundtrip.rs`, `stop()`
being safe to call twice, `operation_id` capture, and
`SparkError::with_operation_id`.
- New offline drop-in tests in `test_dropin_offline.py`.
- The official v4.3.0 `pipelines/tests/test_auto_cdc_flow.py` passes (19
tests).
- `cargo fmt --all --check`, `cargo test -p apache-spark-connect --release
--lib`, and all integration tests compile with and without `--features
wasm-udf`.
- Not run locally, because no Spark 4.3.0 Connect server was available: the
e2e suites and the official `test_connect_context.py`,
`test_parity_sql_context.py` and `test_parity_zip.py`. These run on CI.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
