Hyukjin Kwon created SPARK-62061:
------------------------------------
Summary: SparkSession.addArtifacts reads a path that is not a str,
and pyspark.sql.connect.client exports its names, as PySpark Connect
Key: SPARK-62061
URL: https://issues.apache.org/jira/browse/SPARK-62061
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
A follow-up of SPARK-62023 (#863). Two differences of the client from
pyspark-client 4.3.0:
1. A path of addArtifacts that is not a str. PySpark's
SparkSession.addArtifacts (python/pyspark/sql/connect/session.py:1110-1121)
checks its flags, then SparkConnectClient.add_artifacts
(connect/client/core.py:2461-2470) reads, checks and uploads each path before
the next, and ArtifactManager._parse_artifacts
(connect/client/artifact.py:184-203) parses it with urllib.parse.urlparse. So:
- spark.addArtifacts(pathlib.Path("/tmp/a.py"), pyfile=True), and
client.add_artifacts(...) of one, raise AttributeError: 'PosixPath' object has
no attribute 'decode' (and 'int' object of 5);
- bytes, or a falsy object such as None or 0, parse into a scheme of bytes:
PySparkRuntimeError [UNSUPPORTED_OPERATION] b'' scheme is not supported
(b'file' of b"file:///tmp/a.py");
- spark.addArtifacts(path, pyfile=True, archive=True) raises
INVALID_MULTIPLE_ARGUMENT_CONDITIONS of any path, as the flags are checked
first;
- spark.addArtifacts("/tmp/a.py", pathlib.Path("/tmp/a.py"), pyfile=True)
uploads the first before the second raises.
The drop-in took the paths as a list of str, so each of these raised
TypeError: 'PosixPath' object is not an instance of 'str', before the flags and
before any upload.
2. The names of pyspark.sql.connect.client. PySpark's package is `from
pyspark.sql.connect.client.core import *` and `from pyspark.sql.connect.logging
import getLogLevel`, with no __all__. Its `from pyspark.sql.connect.client
import *` gives ChannelBuilder, DefaultChannelBuilder, RpcDeadlines,
SparkConnectClient, getLogLevel and the modules core, artifact, reattach and
retries. AnalyzeResult, ConfigResult and PlanObservedMetrics are of
pyspark.sql.connect.client.core only (core.py:647-798). The drop-in's package
defined those three classes, so `from pyspark.sql.connect.client import
AnalyzeResult` worked where PySpark raises ImportError, and its __all__ of the
four classes made its `import *` give neither getLogLevel nor core.
Also checked, the same in both clients on master 80046f2f: a name that is not a
str to registerJavaFunction, registerJavaUDAF, udtf.register and udf.register
raises protobuf's TypeError "bad argument type for built-in operation"
(SPARK-61911, SPARK-62023), and a UDF that is not Arrow-optimized raises the
UNSUPPORTED_OPERATION of an abstract return type at the first request, not at
the call (SPARK-61962 / #837).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]