Hello As far as I can find from various searches of the old docs on spark-kernel wiki and issues on JIRA, atleast for jupyter, it seems like each notebook has its own spark-kernel which in turns wraps its own spark context. I am curious about how this isolation works considering that folks have encountered problems when trying to hold multiple spark contexts within the same jvm/process. Can someone help shed some light on this?
I also took a look at the zeppelin work-in-progress branch. This however seems to use a single spark kernel per interpreter ( the zeppelin folks with their current spark integration also use a single spark context across all notebooks ) which I believe would imply sharing of the single spark kernel/spark context across all zeppelin notebooks? Is this assumption correct? If yes, can anyone suggest how the zeppelin integration would likely need to change? Would it be possible to say have one spark-kernel client per notebook ( or re-use them as needed from a pool ) and somehow have each one talk to a different spark kernel instance? thanks — Hitesh
