No we have not submitted it back to spark community. I can send the patches for both spark and zeppelin. If it seems interesting enough then I can start a PR for zeppelin part at least. I will try to do this by weekend if it sounds interesting enough.
Rajat On Wed, Jul 1, 2015 at 5:10 PM, Litt, Shaun <sl...@conversantmedia.com> wrote: > Interesting and useful. I would love to see your changes for reference. > Have you submitted this feature back to the SPARK community? I think > this would be useful for a lot of interactive spark use-cases in shared > clusters. > > > > Also I have a question regarding your experience maintaining your own > SPARK fork. We have also been forced to maintain a very minor spark fork > to work around some bugs in our environment, but our internal goal is to > try to get back to the main apache branch or a vendor fork. Have you found > keeping up with SPARK (which is changing lightning fast – pun intended) > challenging? > > > > Thanks, > > *Shaun* *Litt* > > > > > > *From:* Rajat Gupta [mailto:rajat.iit...@gmail.com] > *Sent:* Wednesday, July 01, 2015 2:41 AM > *To:* users@zeppelin.incubator.apache.org > *Subject:* Re: Zeppelin Interpreter end > > > > We have implemented auto shutdown of spark application by adding an idle > timeout setting directly in spark. We set the value of this setting in the > zeppelin interpreter UI page and default it to 60 mins. For this fix we > also added an auto lazy restart of interpreter in zeppelin if interpreter > has shutdown because of inactivity. If you want I can send you patches for > this both for spark and zeppelin. I did not submit the patch in zeppelin > for this earlier because this makes sense only if spark has idle timeout in > open source. > > Dynamic scaling of spark applications works fine on zeppelin with > appropriate spark settings on interpreter page. Zeppelin just maintains > handle to a spark context and that spark context is free to do anything > afaik. We at Qubole are using it actively. > > Rajat > > > > On Tue, Jun 30, 2015 at 4:53 AM, Litt, Shaun <sl...@conversantmedia.com> > wrote: > > Hi, I am new to zeppelin and just got it configured to run in my YARN > cluster, but I was wondering if there is a configuration or even a hard > setting that shuts down interpreters after in-activity. It seems like the > interpreter (and it’s yarn consumption) hang out indefinitely, ideally > there would be a clean way (like logout or a shutdown button within the > notebook) to shutdown these interpreters, but additionally there should be > a way for an admin of zeppelin to impose and idle timeout. As a note to > the scope of this, is dynamic yarn resource allocation configured (such > that once a paragraph finishes it can release vcores)? > > > > Thanks, > > *Shaun* > > > > This email and any files included with it may contain privileged, > proprietary and/or confidential information that is for the sole use > of the intended recipient(s). Any disclosure, copying, distribution, > posting, or use of the information contained in or attached to this > email is prohibited unless permitted by the sender. If you have > received this email in error, please immediately notify the sender > via return email, telephone, or fax and destroy this original transmission > and its included files without reading or saving it in any manner. > Thank you. > > > > This email and any files included with it may contain privileged, > proprietary and/or confidential information that is for the sole use > of the intended recipient(s). Any disclosure, copying, distribution, > posting, or use of the information contained in or attached to this > email is prohibited unless permitted by the sender. If you have > received this email in error, please immediately notify the sender > via return email, telephone, or fax and destroy this original transmission > and its included files without reading or saving it in any manner. > Thank you. >
