[
https://issues.apache.org/jira/browse/SPARK-62061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62061:
-----------------------------------
Labels: pull-request-available (was: )
> SparkSession.addArtifacts reads a path that is not a str, and
> pyspark.sql.connect.client exports its names, as PySpark Connect
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62061
> URL: https://issues.apache.org/jira/browse/SPARK-62061
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> A follow-up of SPARK-62023 (#863). Two differences of the client from
> pyspark-client 4.3.0:
> 1. A path of addArtifacts that is not a str. PySpark's
> SparkSession.addArtifacts (python/pyspark/sql/connect/session.py:1110-1121)
> checks its flags, then SparkConnectClient.add_artifacts
> (connect/client/core.py:2461-2470) reads, checks and uploads each path before
> the next, and ArtifactManager._parse_artifacts
> (connect/client/artifact.py:184-203) parses it with urllib.parse.urlparse. So:
> - spark.addArtifacts(pathlib.Path("/tmp/a.py"), pyfile=True), and
> client.add_artifacts(...) of one, raise AttributeError: 'PosixPath' object
> has no attribute 'decode' (and 'int' object of 5);
> - bytes, or a falsy object such as None or 0, parse into a scheme of
> bytes: PySparkRuntimeError [UNSUPPORTED_OPERATION] b'' scheme is not
> supported (b'file' of b"file:///tmp/a.py");
> - spark.addArtifacts(path, pyfile=True, archive=True) raises
> INVALID_MULTIPLE_ARGUMENT_CONDITIONS of any path, as the flags are checked
> first;
> - spark.addArtifacts("/tmp/a.py", pathlib.Path("/tmp/a.py"), pyfile=True)
> uploads the first before the second raises.
> The drop-in took the paths as a list of str, so each of these raised
> TypeError: 'PosixPath' object is not an instance of 'str', before the flags
> and before any upload.
> 2. The names of pyspark.sql.connect.client. PySpark's package is `from
> pyspark.sql.connect.client.core import *` and `from
> pyspark.sql.connect.logging import getLogLevel`, with no __all__. Its `from
> pyspark.sql.connect.client import *` gives ChannelBuilder,
> DefaultChannelBuilder, RpcDeadlines, SparkConnectClient, getLogLevel and the
> modules core, artifact, reattach and retries. AnalyzeResult, ConfigResult and
> PlanObservedMetrics are of pyspark.sql.connect.client.core only
> (core.py:647-798). The drop-in's package defined those three classes, so
> `from pyspark.sql.connect.client import AnalyzeResult` worked where PySpark
> raises ImportError, and its __all__ of the four classes made its `import *`
> give neither getLogLevel nor core.
> Also checked, the same in both clients on master 80046f2f: a name that is not
> a str to registerJavaFunction, registerJavaUDAF, udtf.register and
> udf.register raises protobuf's TypeError "bad argument type for built-in
> operation" (SPARK-61911, SPARK-62023), and a UDF that is not Arrow-optimized
> raises the UNSUPPORTED_OPERATION of an abstract return type at the first
> request, not at the call (SPARK-61962 / #837).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]