Hyukjin Kwon created SPARK-61095:
------------------------------------
Summary: Add the modules geo_utils, sql_formatter, table_arg and
tvf_argument of pyspark.sql in the Python client as the PySpark client has them
Key: SPARK-61095
URL: https://issues.apache.org/jira/browse/SPARK-61095
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The PySpark client ships the modules {{geo_utils}}, {{sql_formatter}},
{{table_arg}} and {{tvf_argument}} of {{pyspark.sql}} (the {{pyspark.sql}}
package of {{connect_packages}}): the spatial reference systems of the spatial
types, {{SQLStringFormatter}}, {{TableArg}} and
{{TableValuedFunctionArgument}}, of which a {{Column}} and a {{TableArg}} are
instances. Its {{import pyspark.sql}} imports {{pyspark.sql.conf}},
{{profiler}}, {{streaming}} and {{variant_utils}} too, so they are attributes
of {{pyspark.sql}}.
The Rust-backed Python client lacks the four modules ({{ModuleNotFoundError}}),
and {{pyspark.sql.streaming}} and the others are not attributes of
{{pyspark.sql}} until they are imported. Add them as the PySpark client has
them.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
