schizophrenicmaniac opened a new pull request, #41236: URL: https://github.com/apache/beam/pull/41236
Python and YAML pipelines submitted with `--spark_submit_uber_jar` fail on the Spark driver unless `--storage_level` is passed explicitly. The Python portable runner doesn't set a runner in the pipeline options it sends to Java, so `StorageLevelFactory` ends up calling the `PipelineOptions.DirectRunner` default. That throws an `IllegalArgumentException` because the DirectRunner isn't on the job server or uber jar classpath. `StorageLevelFactory` now catches that and falls back to `MEMORY_ONLY`, which is what any runner other than `SparkStructuredStreamingRunner` already gets. I added tests in `SparkPipelineOptionsTest` for the existing runner-based defaults and for the case where `getRunner()` throws, plus a CHANGES.md entry. Fixes #41080 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
