HyukjinKwon opened a new pull request, #311: URL: https://github.com/apache/spark-connect-rust/pull/311
### What changes were proposed in this pull request? [SPARK-61115](https://issues.apache.org/jira/browse/SPARK-61115): An audit of the modules of the `pyspark-client` package of Apache Spark v4.3.0 found module-level names and modules that the client lacks. This PR adds them, as PySpark has them: - `pyspark.util`: `fail_on_stopiteration`, `disable_gc`, `spark_connect_mode`, `default_api_mode`, `FuncT` and its `__all__`, from v4.3.0. - The `__all__` of `pyspark.sql.udf`, `pyspark.sql.udtf`, `pyspark.sql.avro` and `pyspark.sql.protobuf`. Star imports now export only the listed names. The `__init__` of `avro` and `protobuf` are now identical to v4.3.0. - `pyspark.sql.functions.builtin` and `pyspark.sql.connect.functions.builtin` now have every public function of `pyspark.sql.functions`. That includes the deprecated ones that `__all__` leaves out: `approxCountDistinct`, `bitwiseNOT`, `countDistinct`, `shiftLeft`, `shiftRight`, `shiftRightUnsigned`, `sumDistinct`, `toDegrees`, `toRadians`, `chr`, `random` and `uuid`. As in PySpark, they have no `__all__`. - The modules of the connect import path: - `functions.partitioning`, `avro.functions`, `protobuf.functions` and `table_arg` re-export the functions and the `TableArg` of the client. - `functions` imports `partitioning`, as PySpark's does. - `datasource`: `DataSourceRegistration`, which registers a data source with the session. - `utils`: from v4.3.0, except that `check_dependencies` checks the dependencies of this client. Its transport is the Rust extension, so it needs no gRPC, `grpcio-status`, `googleapis-common-protos` or `zstandard`. - `logging` and `_typing`, vendored byte for byte (and added to the drift check). `dev/parity/4.3.0-module-audit.md` records the audit: - the method: 352 modules, with `pyspark.core` and `py4j` blocked - each gap and the JIRA that closes it: SPARK-61111, SPARK-61113, this one, SPARK-61116, SPARK-61117 and SPARK-61118 - why the rest are not gaps: server-side modules, RDD and classic internals, the gRPC client, tools, and the native `pyspark.ml.connect` ### Why are the changes needed? Code that imports these names from where PySpark has them failed: - `from pyspark.sql.functions.builtin import countDistinct` raised `ImportError`. - `import pyspark.sql.connect.functions.partitioning` and the other connect modules raised `ModuleNotFoundError`. - `from pyspark.util import fail_on_stopiteration` raised `ImportError`. Star imports also exported every public name of these modules. ### Does this PR introduce _any_ user-facing change? Yes. These names and modules exist as in PySpark 4.3.0. ### How was this patch tested? - New offline tests check: - the names of `pyspark.util`, and the star imports of `pyspark.sql.udf`, `udtf`, `avro` and `protobuf`. This test passes with official PySpark 4.3.0. - the modules of the connect import path, the deprecated functions of the builtin modules, and the registration of a data source, against a mock Connect server - `pyspark.sql.connect.utils`, in a process without gRPC - The existing test of the exports of `pyspark.sql.functions` now expects a `builtin` module without `__all__`, as PySpark's. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2108 passed, 8 skipped), and the live drop-in and Arrow suites (164 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
