Hello, One pain point with various Jupyter extensions [1][2] that provide visual feedback about running spark processes is the lack of a public API to introspect the web URL. The notebook server needs to know the URL to find information about the current SparkContext.
Simply looking for "localhost:4040" works most of the time, but fails if multiple spark notebooks are being run on the same host -- spark increments the port for each new context, leading to confusion when the notebooks are trying to probe the web interface for information. I'd like to implement an analog to SparkContext.getOrCreate(), perhaps called "getIfExists()" that returns the current singleton if it exists, or None otherwise. The Jupyter code would then be able to use this entrypoint to query Spark for an active Spark context, which it could use to probe the web URL. It's a minor change, but this would be my first contribution to Spark, and I want to make sure my plan was kosher before I implemented it. Thanks! Andrew [1] https://krishnan-r.github.io/sparkmonitor/ [2] https://github.com/mozilla/jupyter-spark --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
