[
https://issues.apache.org/jira/browse/SPARK-43289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129671#comment-18129671
]
Varun Bhandary commented on SPARK-43289:
----------------------------------------
Hi [~weichenxu123], I’m interested in helping move this feature forward. While
working on distributed anomaly detection with
[DQX|https://databrickslabs.github.io/dqx/] over Spark Connect, I encountered
the need to provision Python dependencies such as SHAP explicitly on workers.
I found your earlier prototype in PR #40954 and the newer language-agnostic UDF
work under SPARK-55278. With worker environment installation, verification and
cleanup hooks now available, would it make sense to revisit this feature using
that infrastructure?
Is Python package dependency management already planned under SPARK-59368 or
another part of that effort? If there’s room to contribute, I’d be happy to
develop a scoped proposal and prototype, building on the earlier review
feedback.
> PySpark UDF supports python package dependencies
> ------------------------------------------------
>
> Key: SPARK-43289
> URL: https://issues.apache.org/jira/browse/SPARK-43289
> Project: Spark
> Issue Type: New Feature
> Components: Connect, ML, PySpark
> Affects Versions: 3.5.0
> Reporter: Weichen Xu
> Assignee: Weichen Xu
> Priority: Major
>
> h3. Requirements
>
> Make the pyspark UDF support annotating python dependencies and when
> executing UDF, the UDF worker creates a new python environment with provided
> python dependencies.
> h3. Motivation
>
> We have two major cases:
>
> * For spark connect case, the client python environment is very likely to be
> different with pyspark server side python environment, this causes user's UDF
> function execution failure in pyspark server side.
> * Some machine learning third-party library (e.g. MLflow) requires pyspark
> UDF supporting dependencies, because in ML cases, we need to run model
> inference by pyspark UDF in the exactly the same python environment that
> trains the model. Currently MLflow supports it by creating a child python
> process in pyspark UDF worker, and redirecting all UDF input data to the
> child python process to run model inference, this way it causes significant
> overhead, if pyspark UDF support builtin python dependency management then we
> don't need such poorly performing approach.
>
> h3. Proposed API
> ```
> @pandas_udf("string", pip_requirements=...)
> ```
> `pip_requirements` argument means either an iterable of pip requirement
> strings (e.g. ``["scikit-learn", "-r /path/to/req2.txt", "-c
> /path/to/constraints.txt"]``) or the string path to a pip requirements file
> path on the local filesystem (e.g. ``"/path/to/requirements.txt"``)
> represents the pip requirements for the python UDF.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
