Re: Spark worker memory not freed up after zeppelin run finishes

That is understandable, but what about if you stop execution by pressing button in notebook? If you do that after you cached some rdd or broadcasted a variable, the cleanup code won't be executed, right ? On Thu, Dec 3, 2015 at 6:25 PM, Felix Cheung wrote: > I think

RE: Spark worker memory not freed up after zeppelin run finishes

It's possible if the execution is interrupted. Perhaps a good practice is to have cleanup code in a separate paragraph? Date: Fri, 4 Dec 2015 10:59:45 +0100 Subject: Re: Spark worker memory not freed up after zeppelin run finishes From: liska.ja...@gmail.com To: users

Re: Spark worker memory not freed up after zeppelin run finishes

Hi, no, just running it manually. I think I need to unpersist cached rdds and destroy broadcast variables in the end, am I correct? Because it hasn't crashed since then, the following runs are always a little slower though. On Thu, Dec 3, 2015 at 8:08 AM, Felix Cheung

Re: Spark worker memory not freed up after zeppelin run finishes

Hey, I mentioned that I'm using broardcast variables, but I'm destroying them at the end... I'm using Spark 1.7.1 ... I'll let you know later if the problem still occurs. So far it seems it stopped after I started destroying them + cachedRdd.unpersist On Thu, Dec 3, 2015 at 5:52 PM, Felix Cheung

Re: Spark worker memory not freed up after zeppelin run finishes

Do you know what version of spark you are running with? On Thu, Dec 3, 2015 at 12:52 AM -0800, "Kevin (Sangwoo) Kim" wrote: Do you use broadcast variables? I've found many problems related to broadcast variables and not using it. (It's a Spark problem, rather than

Re: Spark worker memory not freed up after zeppelin run finishes

I think that's expected since Zeppelin is keeping the spark context alive even when the notebook is not executing (the idea is you could be running more things). That would keep broadcasted data and cached rdd in memory. You should see the same if you run the same code from spark-shell and not

RE: Spark worker memory not freed up after zeppelin run finishes

How are you running jobs? Do you schedule a notebook to run from Zeppelin? Date: Mon, 30 Nov 2015 12:42:16 +0100 Subject: Spark worker memory not freed up after zeppelin run finishes From: liska.ja...@gmail.com To: users@zeppelin.incubator.apache.org Hey, I'm connecting Zeppelin with a remote