[ https://issues.apache.org/jira/browse/SPARK-26597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Clay updated SPARK-26597: --------------------------------- External issue ID: https://github.com/jupyter/docker-stacks/issues/797 > Support using images with different entrypoints on Kubernetes > ------------------------------------------------------------- > > Key: SPARK-26597 > URL: https://issues.apache.org/jira/browse/SPARK-26597 > Project: Spark > Issue Type: New Feature > Components: Kubernetes > Affects Versions: 2.4.0 > Reporter: Patrick Clay > Priority: Minor > > I wish to use arbitrary pre-existing docker images containing Spark with > Kubernetes. > Specifically I wish to use > [jupyter/all-spark-notebook|https://hub.docker.com/r/jupyter/all-spark-notebook] > in in-cluster client mode ideally without image modification (I think using > images maintained by others is a key advantage of Docker). > It has the full Spark 2.4 binary tarball with Spark's entrypoint.sh in it, > but I need to create a child image setting that as the entrypoint to it, > because Spark does not let the user specify a k8s command. I needed separate > images for kernel / driver and executor, because I need the kernel to have > Jupyter's entrypoint. Building and setting the executor image works, but is > obnoxious just to set the entrypoint. > The crux of this FR is to add a property for executor (and driver) command to > point to entrypoint.sh > I personally don't see why you even have entrypoint.sh instead of making the > command be _spark-class > org.apache.spark.executor.CoarseGrainedExecutorBackend ..._, which seems a > lot more portable (albeit reliant on the PATH). > Speaking of reliance on PATH it also broke, because they didn't set > JAVA_HOME, and install tini from Conda putting it on a different path. These > are smaller issues. I'll file an issue on them and try to work them out > between here and there. > In general shouldn't Spark on k8s be less coupled to the layout of the image? -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org