Kaicheng Zhou created SPARK-61671:
-------------------------------------

             Summary:   Add SparkSession.listFiles and 
SparkSession.listArchives for Spark Connect
                 Key: SPARK-61671
                 URL: https://issues.apache.org/jira/browse/SPARK-61671
             Project: Spark
          Issue Type: Improvement
          Components: Connect, PySpark, SQL
    Affects Versions: 4.4.0
            Reporter: Kaicheng Zhou
             Fix For: 4.4.0


SparkContext.listFiles and SparkContext.listArchives return the files and 
archives added to resources, e.g. with SparkContext.addFile, addPyFile and 
addArchive, or SparkSession.addArtifacts. Spark Connect has no SparkContext, so 
they have no equivalent there, and code that moves to Spark Connect has to keep 
its own list of what it added with SparkSession.addArtifacts.

The LIST FILES / LIST ARCHIVES SQL commands do run over Spark Connect, but they 
read SparkContext.listFiles / listArchives, which hold the artifacts of every 
session on the server. A Spark Connect session therefore sees the files and 
archives of other sessions, including other users' (their 
spark://.../artifacts/<session-uuid>/... URIs), and the list does not match 
what its own tasks receive, which is only the session's own artifacts 
(JobArtifactSet). LIST JARS has the same problem.

Proposal:
1. Add SparkSession.listFiles() and SparkSession.listArchives() to PySpark, for 
both Spark Classic and Spark Connect, returning what SparkContext.listFiles / 
listArchives return. In Spark Classic they delegate to the SparkContext. In 
Spark Connect they run LIST FILES / LIST ARCHIVES, so no protocol change is 
needed and the client also works with existing servers.
2. Make LIST FILES, LIST JARS and LIST ARCHIVES list the session's 
JobArtifactSet: only the session's artifacts when artifacts are isolated 
(spark.sql.artifact.isolation.enabled, always on for Spark Connect), and 
everything on the SparkContext otherwise, which leaves Spark Classic unchanged.

Behavior matches SparkContext: Python files are listed as files, an archive 
keeps the #fragment it was added with, and adding the same path twice lists it 
once.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to