Amogh Desai created SPARK-62068:
-----------------------------------

             Summary: Support SparkSession.getOrCreateDataFrame in Spark Connect
                 Key: SPARK-62068
                 URL: https://issues.apache.org/jira/browse/SPARK-62068
             Project: Spark
          Issue Type: New Feature
          Components: Connect
    Affects Versions: 4.4.0
            Reporter: Amogh Desai
             Fix For: 4.5.0


https://issues.apache.org/jira/browse/SPARK-60775 adds 
SparkSession.getOrCreateDataFrame(path, create) for Spark Classic.

It returns the DataFrame stored at path if one is there; otherwise, it calls 
create,
writes the result as parquet, and returns it. On Spark Connect, it currently 
throws
UNSUPPORTED_CONNECT_FEATURE.SESSION_GET_OR_CREATE_DATAFRAME (Scala) and
NOT_IMPLEMENTED (Python).

The goal is to extend support in connect too.

The approach I have in mind is:
- Expose the server-side check (isDataFrameMaterialized) to the client, forĀ 
example through a new RPC (or a command?)
- The client evaluates the create function locally, the same way classic 
pyspark does, then writes the result and the marker on the server.

Context: https://github.com/apache/spark/pull/60000#discussion_r4590551003



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to