Mark Molinaro created SPARK-55806:
-------------------------------------
Summary: Document usage of `uv run` wheb submitting pyspark jobs
Key: SPARK-55806
URL: https://issues.apache.org/jira/browse/SPARK-55806
Project: Spark
Issue Type: Documentation
Components: Examples, PySpark
Affects Versions: 4.2.0
Reporter: Mark Molinaro
`uv run` has the capability to install the requested dependencies before
starting to run a given script. We can provide an example which uses `uv run`
in addition to the current examples using `conda`, `pex`, etc.
I have found this approach to be especially valuable in an environment with
heterogenous python dependencies across a large organization. We have a "base"
set of dependencies ambiently available to pyspark via installation in our
docker container, but this allows applying changes or additions for specific
use-cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
