[ 
https://issues.apache.org/jira/browse/SPARK-61826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudhanva Huruli updated SPARK-61826:
------------------------------------
    Description: 
A spark-pipelines run of a project generated by spark-pipelines init uses the 
catalog of the Spark Connect server it runs against. The server defaults to 
spark.sql.catalogImplementation=in-memory, so when the server stops (including 
the temporary server that spark-pipelines starts when --remote is not given) 
the table metadata is gone, but the table folders stay under spark-warehouse/. 
The second run then fails with LOCATION_ALREADY_EXISTS, and the tables cannot 
be queried after the run.

The catalog implementation is a static setting of the server, so it cannot be 
set per pipeline in the spec: a spec configuration of 
spark.sql.catalogImplementation is not applied to a server that is already 
running. On the current master code the run prints 'Failed to set 
spark.sql.catalogImplementation (CANNOT_MODIFY_STATIC_CONFIG)' and continues on 
the server's in-memory catalog.

A pipeline needs a Spark Connect server with a persistent catalog, for example 
one started with sbin/start-connect-server.sh --conf 
spark.sql.catalogImplementation=hive. Without it, tables last only while the 
server runs. I suggest that spark-pipelines run print a warning in that case, 
naming the server setting to change when --remote is given and pointing to 
sbin/start-connect-server.sh otherwise. The warning would apply when the 
pipeline writes to spark_catalog.

  was:spark-pipelines init writes a spec without a catalog setting, so a run 
uses the in-memory session catalog. When the run ends the tables are forgotten, 
but their folders stay under spark-warehouse/, so the second spark-pipelines 
run fails with LOCATION_ALREADY_EXISTS and the tables cannot be queried from 
spark-sql or pyspark after the run. The generated spark-pipeline.yml should set 
spark.sql.catalogImplementation: hive so metadata persists in a local 
metastore_db/.


> Projects generated by spark-pipelines init lose their tables after the run 
> and fail the second run with LOCATION_ALREADY_EXISTS
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61826
>                 URL: https://issues.apache.org/jira/browse/SPARK-61826
>             Project: Spark
>          Issue Type: Bug
>          Components: Declarative Pipelines
>    Affects Versions: 4.3.0
>            Reporter: Sudhanva Huruli
>            Priority: Major
>              Labels: pull-request-available
>
> A spark-pipelines run of a project generated by spark-pipelines init uses the 
> catalog of the Spark Connect server it runs against. The server defaults to 
> spark.sql.catalogImplementation=in-memory, so when the server stops 
> (including the temporary server that spark-pipelines starts when --remote is 
> not given) the table metadata is gone, but the table folders stay under 
> spark-warehouse/. The second run then fails with LOCATION_ALREADY_EXISTS, and 
> the tables cannot be queried after the run.
> The catalog implementation is a static setting of the server, so it cannot be 
> set per pipeline in the spec: a spec configuration of 
> spark.sql.catalogImplementation is not applied to a server that is already 
> running. On the current master code the run prints 'Failed to set 
> spark.sql.catalogImplementation (CANNOT_MODIFY_STATIC_CONFIG)' and continues 
> on the server's in-memory catalog.
> A pipeline needs a Spark Connect server with a persistent catalog, for 
> example one started with sbin/start-connect-server.sh --conf 
> spark.sql.catalogImplementation=hive. Without it, tables last only while the 
> server runs. I suggest that spark-pipelines run print a warning in that case, 
> naming the server setting to change when --remote is given and pointing to 
> sbin/start-connect-server.sh otherwise. The warning would apply when the 
> pipeline writes to spark_catalog.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to