I think Vinoo is right about the intended behavior. If we support multiple sessions in one context, then stopping any one session shouldn't stop the shared context. The last session to be stopped should stop the context, but not any before that. We don't typically run multiple sessions in the same context so we haven't hit this, but it sounds reasonable.
On Tue, Apr 2, 2019 at 8:23 AM Vinoo Ganesh <vgan...@palantir.com> wrote: > Hey Sean - Cool, maybe I'm misunderstanding the intent of clearing a > session vs. stopping it. > > The cause of the leak looks to be because of this line here > https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala#L131. > The ExecutionListenerBus that's added persists forever on the context's > listener bus (the SparkContext ListenerBus has an ExecutionListenerBus). > I'm trying to figure out the place that this cleanup should happen. > > With the current implementation, calling SparkSession.stop will clean up > the ExecutionListenerBus (since the context itself is stopped), but it's > unclear to me why terminating one session should terminate the JVM-global > context. Possible my mental model is off here, but I would expect stopping > a session to remove all traces of that session, while keeping the context > alive, and stopping a context would, well, stop the context. > > If stopping the session is expected to stop the context, what's the > intended usage of clearing the active / default session? > > Vinoo > > On 4/2/19, 10:57, "Sean Owen" <sro...@gmail.com> wrote: > > What are you expecting there ... that sounds correct? something else > needs to be closed? > > On Tue, Apr 2, 2019 at 9:45 AM Vinoo Ganesh <vgan...@palantir.com> > wrote: > > > > Hi All - > > > > I’ve been digging into the code and looking into what appears to > be a memory leak ( > https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.apache.org_jira_browse_SPARK-2D27337&d=DwIFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=TjtXLhnSM5M_aKQlD2NFU2wRnXPvtrUbRm-t84gBNlY&s=JUsN7EzGimus0jYxyj47_xHYUDC6KnxieeUBfUKTefk&e=) > and have noticed something kind of peculiar about the way closing a > SparkSession is handled. Despite being marked as Closeable, > closing/stopping a SparkSession simply stops the SparkContext. This changed > happened as a result of one of the PRs addressing > https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.apache.org_jira_browse_SPARK-2D15073&d=DwIFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=TjtXLhnSM5M_aKQlD2NFU2wRnXPvtrUbRm-t84gBNlY&s=Nd9eBDH-FDdzEn_BVt2nZaNQn6fXA8EfVq5rKGztOUo&e= > in > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_spark_pull_12873_files-23diff-2Dd91c284798f1c98bf03a31855e26d71cR596&d=DwIFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=7WzLIMu3WvZwd6AMPatqn1KZW39eI6c_oflAHIy1NUc&m=TjtXLhnSM5M_aKQlD2NFU2wRnXPvtrUbRm-t84gBNlY&s=RM9LrT3Yp2mf1BcbBf1o_m3bcNZdOjznrogBLzUzgeE&e= > . > > > > > > > > I’m trying to understand why this is the intended behavior – anyone > have any knowledge of why this is the case? > > > > > > > > Thanks, > > > > Vinoo > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- Ryan Blue Software Engineer Netflix
