[ 
https://issues.apache.org/jira/browse/SPARK-62061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-62061:
------------------------------------

    Assignee: Hyukjin Kwon

> SparkSession.addArtifacts reads a path that is not a str, and 
> pyspark.sql.connect.client exports its names, as PySpark Connect
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62061
>                 URL: https://issues.apache.org/jira/browse/SPARK-62061
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> A follow-up of SPARK-62023 (#863). Two differences of the client from 
> pyspark-client 4.3.0:
> 1. A path of addArtifacts that is not a str. PySpark's 
> SparkSession.addArtifacts (python/pyspark/sql/connect/session.py:1110-1121) 
> checks its flags, then SparkConnectClient.add_artifacts 
> (connect/client/core.py:2461-2470) reads, checks and uploads each path before 
> the next, and ArtifactManager._parse_artifacts 
> (connect/client/artifact.py:184-203) parses it with urllib.parse.urlparse. So:
>    - spark.addArtifacts(pathlib.Path("/tmp/a.py"), pyfile=True), and 
> client.add_artifacts(...) of one, raise AttributeError: 'PosixPath' object 
> has no attribute 'decode' (and 'int' object of 5);
>    - bytes, or a falsy object such as None or 0, parse into a scheme of 
> bytes: PySparkRuntimeError [UNSUPPORTED_OPERATION] b'' scheme is not 
> supported (b'file' of b"file:///tmp/a.py");
>    - spark.addArtifacts(path, pyfile=True, archive=True) raises 
> INVALID_MULTIPLE_ARGUMENT_CONDITIONS of any path, as the flags are checked 
> first;
>    - spark.addArtifacts("/tmp/a.py", pathlib.Path("/tmp/a.py"), pyfile=True) 
> uploads the first before the second raises.
>    The drop-in took the paths as a list of str, so each of these raised 
> TypeError: 'PosixPath' object is not an instance of 'str', before the flags 
> and before any upload.
> 2. The names of pyspark.sql.connect.client. PySpark's package is `from 
> pyspark.sql.connect.client.core import *` and `from 
> pyspark.sql.connect.logging import getLogLevel`, with no __all__. Its `from 
> pyspark.sql.connect.client import *` gives ChannelBuilder, 
> DefaultChannelBuilder, RpcDeadlines, SparkConnectClient, getLogLevel and the 
> modules core, artifact, reattach and retries. AnalyzeResult, ConfigResult and 
> PlanObservedMetrics are of pyspark.sql.connect.client.core only 
> (core.py:647-798). The drop-in's package defined those three classes, so 
> `from pyspark.sql.connect.client import AnalyzeResult` worked where PySpark 
> raises ImportError, and its __all__ of the four classes made its `import *` 
> give neither getLogLevel nor core.
> Also checked, the same in both clients on master 80046f2f: a name that is not 
> a str to registerJavaFunction, registerJavaUDAF, udtf.register and 
> udf.register raises protobuf's TypeError "bad argument type for built-in 
> operation" (SPARK-61911, SPARK-62023), and a UDF that is not Arrow-optimized 
> raises the UNSUPPORTED_OPERATION of an abstract return type at the first 
> request, not at the call (SPARK-61962 / #837).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to