HyukjinKwon opened a new pull request, #149: URL: https://github.com/apache/spark-connect-rust/pull/149
### What changes were proposed in this pull request? [SPARK-60902](https://issues.apache.org/jira/browse/SPARK-60902): the Rust-backed Python client's `createDataFrame` now applies column names as PySpark Connect 4.3.0 does. - A list or tuple of column names is applied with `toDF`, and so are the columns of a pandas DataFrame or the field names of its schema. The server resolves `toDF` into a `Project`, so `isLocal()` is False for these DataFrames, as in PySpark. - Without a schema, plain rows are named `_1`, `_2`, ... instead of `col0`, `col1`, .... A list of names shorter than the plain rows or the pandas DataFrame is extended the same way, and an empty list gives `_1`, `_2`, .... - Rows that do not match the width of the schema raise `AXIS_LENGTH_MISMATCH`, with PySpark's expected and actual lengths, for every kind of schema. A pandas DataFrame with an atomic schema raises `UNSUPPORTED_DATA_TYPE_FOR_ARROW`. Empty input without a struct schema raises `CANNOT_INFER_EMPTY_SCHEMA` instead of a plain `ValueError`. Two pandas cases still differ, on purpose. With a StructType narrower than the DataFrame, PySpark silently drops the extra columns (its conversion zips the columns with the fields), and with a wider one it raises PyArrow's `ArrowInvalid`; this client raises `AXIS_LENGTH_MISMATCH` for both, as it raised an error before. NumPy ndarray input, which the client does not accept yet, is tracked by SPARK-60772. ### Why are the changes needed? `createDataFrame([(1, 2)])` had different column names from PySpark, `createDataFrame([(1, 2)], ["a"])` failed where PySpark returns columns `a` and `_2`, and `isLocal()` and the error classes differed from PySpark's. ### Does this PR introduce _any_ user-facing change? Yes, these now match PySpark Connect 4.3.0: - `createDataFrame([(1, 2)]).columns` is `['_1', '_2']`, previously `['col0', 'col1']`. - `createDataFrame([(1, 2)], ["a"])` has the columns `['a', '_2']`; it raised an error before. - `isLocal()` is False for `createDataFrame([(1, 2)], ["a", "b"])` and for `createDataFrame(pandas_df)`, with or without a schema; it was True. - The error classes above, where errors had no class before. ### How was this patch tested? - Against a local Spark 4.3.0 server, three probes over 24 inputs give output identical to official PySpark 4.3.0, including the column names, collected rows, `isLocal()`, and error messages. The inputs cover rows with no schema, a list or tuple of names, Rows, dicts, and StructType, DDL, and atomic schemas; pandas DataFrames with no schema, names, StructType, DDL, and atomic schemas, and empty ones; and width mismatches. - New offline tests check `isLocal()` for rows and pandas input, and the error classes and parameters. The existing width test now expects PySpark's `AXIS_LENGTH_MISMATCH`, still a `PySparkValueError`. - New live tests check the column names and rows for rows and pandas input. They also pass with official PySpark 4.3.0. - The offline Python suites (441 passed, 8 live cases skipped), the live drop-in and Arrow suites (32 passed), and the live cases (9 passed). Clippy reports no new diagnostics in `createDataFrame`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
