Yeah there's one global default session, but it's possible to create others and set them as the thread's active session, to allow for different configurations in the SparkSession within one app. I think you're asking why closing one of them would effectively shut all of them down by stopping the SparkContext. My best guess is simply, well, that's how it works. You'd only call this, like SparkContext.stop(), when you know the whole app is done and want to clean up. SparkSession is a kind of wrapper on SparkContext and it wouldn't be great to make users stop all the sessions and go find and stop the context.
If there is some per-SparkSession state that needs a cleanup, then that's a good point, as I don't see a lifecycle method that means "just close this session". You're talking about SparkContext state though right, and there is definitely just one SparkContext though. It can/should only be stopped when the app is really done. Is the point that each session is adding some state to the context and doesn't have any mechanism for removing it? On Tue, Apr 2, 2019 at 10:23 AM Vinoo Ganesh <vgan...@palantir.com> wrote: > > Hey Sean - Cool, maybe I'm misunderstanding the intent of clearing a session > vs. stopping it. > > The cause of the leak looks to be because of this line here > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala#L131. > The ExecutionListenerBus that's added persists forever on the context's > listener bus (the SparkContext ListenerBus has an ExecutionListenerBus). I'm > trying to figure out the place that this cleanup should happen. > > With the current implementation, calling SparkSession.stop will clean up the > ExecutionListenerBus (since the context itself is stopped), but it's unclear > to me why terminating one session should terminate the JVM-global context. > Possible my mental model is off here, but I would expect stopping a session > to remove all traces of that session, while keeping the context alive, and > stopping a context would, well, stop the context. > > If stopping the session is expected to stop the context, what's the intended > usage of clearing the active / default session? > > Vinoo > > On 4/2/19, 10:57, "Sean Owen" <sro...@gmail.com> wrote: > > What are you expecting there ... that sounds correct? something else > needs to be closed? > > On Tue, Apr 2, 2019 at 9:45 AM Vinoo Ganesh <vgan...@palantir.com> wrote: > > > > Hi All - > > > > I’ve been digging into the code and looking into what appears to be > a memory leak > (https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.apache.org_jira_browse_SPARK-2D27337&d=DwIFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=TjtXLhnSM5M_aKQlD2NFU2wRnXPvtrUbRm-t84gBNlY&s=JUsN7EzGimus0jYxyj47_xHYUDC6KnxieeUBfUKTefk&e=) > and have noticed something kind of peculiar about the way closing a > SparkSession is handled. Despite being marked as Closeable, closing/stopping > a SparkSession simply stops the SparkContext. This changed happened as a > result of one of the PRs addressing > https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.apache.org_jira_browse_SPARK-2D15073&d=DwIFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=TjtXLhnSM5M_aKQlD2NFU2wRnXPvtrUbRm-t84gBNlY&s=Nd9eBDH-FDdzEn_BVt2nZaNQn6fXA8EfVq5rKGztOUo&e= > in > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_pull_12873_files-23diff-2Dd91c284798f1c98bf03a31855e26d71cR596&d=DwIFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=TjtXLhnSM5M_aKQlD2NFU2wRnXPvtrUbRm-t84gBNlY&s=RM9LrT3Yp2mf1BcbBf1o_m3bcNZdOjznrogBLzUzgeE&e=. > > > > > > > > I’m trying to understand why this is the intended behavior – anyone > have any knowledge of why this is the case? > > > > > > > > Thanks, > > > > Vinoo > > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
