Sanford Ryza created SPARK-54508:
------------------------------------
Summary: spark-pipelines fails when spec file is in different
directory than CWD
Key: SPARK-54508
URL: https://issues.apache.org/jira/browse/SPARK-54508
Project: Spark
Issue Type: Bug
Components: Declarative Pipelines
Affects Versions: 4.1.0
Reporter: Sanford Ryza
Assignee: Sanford Ryza
Succeeds:
```
spark-pipelines run --spec spark-pipelines.yml
```
Fails:
```
cd ..
spark-pipelines run --spec subdir/spark-pipelines.yml
```
Error:
```
pyspark.errors.exceptions.connect.AnalysisException: [RUN_EMPTY_PIPELINE]
Pipelines are expected to have at least one non-temporary dataset defined
(tables, persisted views) but no non-temporary datasets were found in your
pipeline.
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
