[
https://issues.apache.org/jira/browse/SPARK-61826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudhanva Huruli updated SPARK-61826:
------------------------------------
Description:
A spark-pipelines run of a project generated by spark-pipelines init uses the
catalog of the Spark Connect server it runs against. The server defaults to
spark.sql.catalogImplementation=in-memory, so when the server stops (including
the temporary server that spark-pipelines starts when --remote is not given)
the table metadata is gone, but the table folders stay under spark-warehouse/.
The second run then fails with LOCATION_ALREADY_EXISTS, and the tables cannot
be queried after the run.
The catalog implementation is a static setting of the server, so it cannot be
set per pipeline in the spec: a spec configuration of
spark.sql.catalogImplementation is not applied to a server that is already
running. On the current master code the run prints 'Failed to set
spark.sql.catalogImplementation (CANNOT_MODIFY_STATIC_CONFIG)' and continues on
the server's in-memory catalog.
A pipeline needs a Spark Connect server with a persistent catalog, for example
one started with sbin/start-connect-server.sh --conf
spark.sql.catalogImplementation=hive. Without it, tables last only while the
server runs. I suggest that spark-pipelines run print a warning in that case,
naming the server setting to change when --remote is given and pointing to
sbin/start-connect-server.sh otherwise. The warning would apply when the
pipeline writes to spark_catalog.
was:spark-pipelines init writes a spec without a catalog setting, so a run
uses the in-memory session catalog. When the run ends the tables are forgotten,
but their folders stay under spark-warehouse/, so the second spark-pipelines
run fails with LOCATION_ALREADY_EXISTS and the tables cannot be queried from
spark-sql or pyspark after the run. The generated spark-pipeline.yml should set
spark.sql.catalogImplementation: hive so metadata persists in a local
metastore_db/.
> Projects generated by spark-pipelines init lose their tables after the run
> and fail the second run with LOCATION_ALREADY_EXISTS
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61826
> URL: https://issues.apache.org/jira/browse/SPARK-61826
> Project: Spark
> Issue Type: Bug
> Components: Declarative Pipelines
> Affects Versions: 4.3.0
> Reporter: Sudhanva Huruli
> Priority: Major
> Labels: pull-request-available
>
> A spark-pipelines run of a project generated by spark-pipelines init uses the
> catalog of the Spark Connect server it runs against. The server defaults to
> spark.sql.catalogImplementation=in-memory, so when the server stops
> (including the temporary server that spark-pipelines starts when --remote is
> not given) the table metadata is gone, but the table folders stay under
> spark-warehouse/. The second run then fails with LOCATION_ALREADY_EXISTS, and
> the tables cannot be queried after the run.
> The catalog implementation is a static setting of the server, so it cannot be
> set per pipeline in the spec: a spec configuration of
> spark.sql.catalogImplementation is not applied to a server that is already
> running. On the current master code the run prints 'Failed to set
> spark.sql.catalogImplementation (CANNOT_MODIFY_STATIC_CONFIG)' and continues
> on the server's in-memory catalog.
> A pipeline needs a Spark Connect server with a persistent catalog, for
> example one started with sbin/start-connect-server.sh --conf
> spark.sql.catalogImplementation=hive. Without it, tables last only while the
> server runs. I suggest that spark-pipelines run print a warning in that case,
> naming the server setting to change when --remote is given and pointing to
> sbin/start-connect-server.sh otherwise. The warning would apply when the
> pipeline writes to spark_catalog.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]