On Tue, Apr 2, 2019 at 12:23 PM Vinoo Ganesh <vgan...@palantir.com> wrote: > @Sean – To the point that Ryan made, it feels wrong that stopping a session > force stops the global context. Building in the logic to only stop the > context when the last session is stopped also feels like a solution, but the > best way I can think about doing this involves storing the global list of > every available SparkSession, which may be difficult.
I tend to agree it would be more natural for the SparkSession to have its own lifecycle 'stop' method that only stops/releases its own resources. But is that the source of the problem here? if the state you're trying to free is needed by the SparkContext, it won't help. If it happens to be in the SparkContext but is state only needed by one SparkSession and that there isn't any way to clean up now, that's a compelling reason to change the API. Is that the situation? The only downside is making the user separately stop the SparkContext then. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
