This might be helpful and start moving you towards using the gateway: https://elyra.readthedocs.io/en/stable/recipes/deploying-elyra-in-a-jupyterhub-environment.html
On Fri, Aug 27, 2021 at 18:02 Greg Gallagher <[email protected]> wrote: > I would like to run a spark master from my Jupyter notebook. Currently I > have Jupyter Hub running in Kubernetes successfully, and Kubespawner > creates notebooks and it is awesome. > > The problem is, in order to run spark master (driver) from my notebook to > a Yarn cluster (not spark-submit, where the driver would be running in the > cluster) the yarn executors cannot reach back to my notebook because the > ports aren't exposed as a service. > > Looking at Kubespawner code, I see there is a function it looks like to > make a service with each pod, but I'm not sure how it would be invoked. > > > https://github.com/jupyterhub/kubespawner/blob/master/kubespawner/objects.py#L916 > : > def make_service( > name, > port, > servername, > owner_references, > labels=None, > annotations=None, > ): > """ > Make a k8s service specification for using dns to communicate with the > notebook. > Parameters > ---------- > name: > Name of the service. Must be unique within the namespace the > object is > going to be created in. > env: > Dictionary of environment variables. > labels: > Labels to add to the service. > annotations: > Annotations to add to the service. > """ > > And then there's a call to make an ingress as well: > > > https://github.com/jupyterhub/kubespawner/blob/master/kubespawner/objects.py#L720 > > What are my missing? I can't see anything in the Kubespawner documentation > about creating service ports (ideally using a LoadBalancer IP). > > Side question: should I be using Enterprise Gateway? I'm slightly confused > at what the mainstream way to deploy Jupyter notebooks in a multi-user > environment should be. I hadn't heard of the EG until I started > researching this issue, and now I'm wondering if I'm deploying the right > thing at all. > > Thanks in Advance, > > |> Greg > > -- > You received this message because you are subscribed to the Google Groups > "Project Jupyter" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/jupyter/6b0d8628-293e-4118-9749-feae094b5d66n%40googlegroups.com > <https://groups.google.com/d/msgid/jupyter/6b0d8628-293e-4118-9749-feae094b5d66n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Sent from my Mobile device -- You received this message because you are subscribed to the Google Groups "Project Jupyter" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAGU5spfuvwTd4_m6FmxXEDTpQLBhGqN2_CiG763Xwp_juKcuvA%40mail.gmail.com.
