[
https://issues.apache.org/jira/browse/SPARK-26597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Clay updated SPARK-26597:
---------------------------------
External issue ID: https://github.com/jupyter/docker-stacks/issues/797
> Support using images with different entrypoints on Kubernetes
> -------------------------------------------------------------
>
> Key: SPARK-26597
> URL: https://issues.apache.org/jira/browse/SPARK-26597
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes
> Affects Versions: 2.4.0
> Reporter: Patrick Clay
> Priority: Minor
>
> I wish to use arbitrary pre-existing docker images containing Spark with
> Kubernetes.
> Specifically I wish to use
> [jupyter/all-spark-notebook|https://hub.docker.com/r/jupyter/all-spark-notebook]
> in in-cluster client mode ideally without image modification (I think using
> images maintained by others is a key advantage of Docker).
> It has the full Spark 2.4 binary tarball with Spark's entrypoint.sh in it,
> but I need to create a child image setting that as the entrypoint to it,
> because Spark does not let the user specify a k8s command. I needed separate
> images for kernel / driver and executor, because I need the kernel to have
> Jupyter's entrypoint. Building and setting the executor image works, but is
> obnoxious just to set the entrypoint.
> The crux of this FR is to add a property for executor (and driver) command to
> point to entrypoint.sh
> I personally don't see why you even have entrypoint.sh instead of making the
> command be _spark-class
> org.apache.spark.executor.CoarseGrainedExecutorBackend ..._, which seems a
> lot more portable (albeit reliant on the PATH).
> Speaking of reliance on PATH it also broke, because they didn't set
> JAVA_HOME, and install tini from Conda putting it on a different path. These
> are smaller issues. I'll file an issue on them and try to work them out
> between here and there.
> In general shouldn't Spark on k8s be less coupled to the layout of the image?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
