I use version 0.7.3. I have been trying to investigate the reasons for the timeouts. I was trying to tune the number of cores available. Now that you've mentioned the driver memory issue, I'll try it out again and let you know if that solves the problem. What would be a back-of-the-envelope calculation for the resource requirements per notebook in a scoped + per note configuration?
Also, is there a provision for executing multiple spark paragraphs in a notebook simultaneously? I wasn't able to achieve that and hence the workaround by running each paragraph in its own notebook. Thanks Ajay Viswanathan Sr. Software Engineer, MiQ On Fri, 9 Nov 2018 at 13:12, Jeff Zhang <zjf...@gmail.com> wrote: > HI Ajay, > > Thanks for the reporting, which version do you use ? One know issue of > spark scoped mode is that each spark repl will occupy large memory and > won't be released, one workaround is to increase the driver memory. > > https://jira.apache.org/jira/browse/ZEPPELIN-3389 > > > Ajay Viswanathan <ajayviswanat...@miqdigital.com.invalid>于2018年11月9日周五 > 下午3:31写道： > > > This is an issue even I am facing in my project currently. By running the > > spark interpreter in Scoped + Per Note mode, I do manage to execute > > paragraphs in parallel, but it becomes very resource intensive and times > > out if I run more than 3-4 jobs in parallel on a 4-core cloud instance. > > Typically a Thrift Transport exception is thrown after a prolonged period > > of inactivity, > > > > On Fri, 9 Nov 2018 at 06:54, Jeff Zhang <zjf...@gmail.com> wrote: > > > > > Which version do you use ? This seems a bug. Each note should have its > > own > > > scheduler in scoped per note mode. > > > > > > <tecgi...@yahoo.com.invalid>于2018年11月9日周五 上午1:56写道： > > > > > > > Hi > > > > > > > > We use zeppelin in multi-user environment, the interpreter scope mode > > > > seems to allow notebook execution in serial only. If multiple users > are > > > > running their notebooks concurrently, these notebooks are queued for > > > serial > > > > execution. If one notebook takes a long time to complete, it > basically > > > > blocks other notebooks from execution. To enable parallel notebook > > > > execution, it seems we need to use the isolated mode, which creates a > > new > > > > interpreter instance (run on separate JVM) per user. But this can > > become > > > > expensive (compute resource intensive). what is the suggested > > interpreter > > > > mode for multi-user environment? > > > > > > > > Thanks > > > > Denny > > > > > > > > > > > > > -- > > Ajay Viswanathan > > *Sr. Software Engineer, CAPS - Processing* > > *A: *5th & 6th Floor | Skav 909 | 9/1 | Lavelle Road | Bangalore | 560001 > > *E: *yourem...@miqdigital.com > > *M: *+00 (0)0 0000 0000 > > *W: *wearemiq.com > > [image: MiQ] <http://wearemiq.com/> > > *Disclaimer: *This email and its attachments are confidential and are > > intended solely for the use of the individual to whom it is addressed. If > > you are not the intended recipient of this email and its attachments, you > > must take no action based upon them, nor must you copy or show them to > > anyone. No contracts or official orders shall be concluded by means of > this > > email. Please contact the sender if you believe you have received this > > email in error. > > > -- Ajay Viswanathan *Sr. Software Engineer, CAPS - Processing* *A: *5th & 6th Floor | Skav 909 | 9/1 | Lavelle Road | Bangalore | 560001 *E: *ajayviswanat...@miqdigital.com *M: *+00 (0)0 0000 0000 *W: *wearemiq.com [image: MiQ] <http://wearemiq.com/> *Disclaimer: *This email and its attachments are confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone. No contracts or official orders shall be concluded by means of this email. Please contact the sender if you believe you have received this email in error.
