LuciferYang commented on code in PR #56668:
URL: https://github.com/apache/spark/pull/56668#discussion_r3457295763
##########
python/pyspark/pipelines/cli.py:
##########
@@ -325,30 +325,31 @@ def run(
spark_builder = spark_builder.config(key, value)
spark = spark_builder.getOrCreate()
-
- log_with_curr_timestamp("Creating dataflow graph...")
- dataflow_graph_id = create_dataflow_graph(
- spark,
- default_catalog=spec.catalog,
- default_database=spec.database,
- sql_conf=spec.configuration,
- )
-
- log_with_curr_timestamp("Registering graph elements...")
- registry = SparkConnectGraphElementRegistry(spark, dataflow_graph_id)
- register_definitions(spec_path, registry, spec, spark, dataflow_graph_id)
-
- log_with_curr_timestamp("Starting run...")
- result_iter = start_run(
- spark,
- dataflow_graph_id,
- full_refresh=full_refresh,
- full_refresh_all=full_refresh_all,
- refresh=refresh,
- dry=dry,
- storage=spec.storage,
- )
+ # Stop the session even if graph creation, registration, or the run itself
fails, so a failure
Review Comment:
Thanks! `with ... as spark` works for the classic SparkSession, but this CLI
runs on Spark Connect, and the Connect SparkSession has `stop()` but no
`__enter__`/`__exit__` - so `with spark_builder.getOrCreate() as spark:` raises
`TypeError: ... object does not support the context manager protocol`. The
try/finally is the equivalent that works here. Adding the context manager
protocol to the Connect session would be a nice improvement, but I'd keep it
out of this fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
