[
https://issues.apache.org/jira/browse/SPARK-38648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17528862#comment-17528862
]
Xiangrui Meng commented on SPARK-38648:
---------------------------------------
I think it is beneficial to both Spark and DL frameworks if Spark has
state-of-the-art DL capabilities. We did some work in the past to make Spark
work better with DL frameworks, e.g., iterator Scalar Pandas UDF, barrier mode,
and GPU scheduling. But most of them are low level APIs for developers, not end
users. Our Spark user guide contains little about DL and AI.
The dependency on DL frameworks might create issues. One idea is to develop in
the Spark repo and Spark namespace but publish to PyPI independently. For
example, in order to use DL features, users need to explicitly install
`pyspark-dl` and then use the features under `pyspark.dl` namespace.
Putting development inside Spark and publishing under the spark namespace would
help drive both development and adoption.
> SPIP: Simplified API for DL Inferencing
> ---------------------------------------
>
> Key: SPARK-38648
> URL: https://issues.apache.org/jira/browse/SPARK-38648
> Project: Spark
> Issue Type: Improvement
> Components: ML
> Affects Versions: 3.0.0
> Reporter: Lee Yang
> Priority: Minor
>
> h1. Background and Motivation
> The deployment of deep learning (DL) models to Spark clusters can be a point
> of friction today. DL practitioners often aren't well-versed with Spark, and
> Spark experts often aren't well-versed with the fast-changing DL frameworks.
> Currently, the deployment of trained DL models is done in a fairly ad-hoc
> manner, with each model integration usually requiring significant effort.
> To simplify this process, we propose adding an integration layer for each
> major DL framework that can introspect their respective saved models to
> more-easily integrate these models into Spark applications. You can find a
> detailed proposal here:
> [https://docs.google.com/document/d/1n7QPHVZfmQknvebZEXxzndHPV2T71aBsDnP4COQa_v0]
> h1. Goals
> - Simplify the deployment of pre-trained single-node DL models to Spark
> inference applications.
> - Follow pandas_udf for simple inference use-cases.
> - Follow Spark ML Pipelines APIs for transfer-learning use-cases.
> - Enable integrations with popular third-party DL frameworks like
> TensorFlow, PyTorch, and Huggingface.
> - Focus on PySpark, since most of the DL frameworks use Python.
> - Take advantage of built-in Spark features like GPU scheduling and Arrow
> integration.
> - Enable inference on both CPU and GPU.
> h1. Non-goals
> - DL model training.
> - Inference w/ distributed models, i.e. "model parallel" inference.
> h1. Target Personas
> - Data scientists who need to deploy DL models on Spark.
> - Developers who need to deploy DL models on Spark.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
