There's no standard way to calculate the memory requirement for driver. It depends on your app, e.g if you want to fetch large data into driver, then you'd better to set a large value for driver memory.
Regarding running paragraphs simultaneously, for scala/python/r code, the execution in one paragraph is sequential, the reason is that there may be dependencies between paragraphs, e.g. paragraph 2 may use the variable defined in paragraph 1 Ajay Viswanathan <ajayviswanat...@miqdigital.com.invalid>于2018年11月9日周五 下午3:49写道： > I use version 0.7.3. I have been trying to investigate the reasons for the > timeouts. I was trying to tune the number of cores available. Now that > you've mentioned the driver memory issue, I'll try it out again and let you > know if that solves the problem. What would be a back-of-the-envelope > calculation for the resource requirements per notebook in a scoped + per > note configuration? > > Also, is there a provision for executing multiple spark paragraphs in a > notebook simultaneously? I wasn't able to achieve that and hence the > workaround by running each paragraph in its own notebook. > > Thanks > Ajay Viswanathan > Sr. Software Engineer, MiQ > > On Fri, 9 Nov 2018 at 13:12, Jeff Zhang <zjf...@gmail.com> wrote: > > > HI Ajay, > > > > Thanks for the reporting, which version do you use ? One know issue of > > spark scoped mode is that each spark repl will occupy large memory and > > won't be released, one workaround is to increase the driver memory. > > > > https://jira.apache.org/jira/browse/ZEPPELIN-3389 > > > > > > Ajay Viswanathan <ajayviswanat...@miqdigital.com.invalid>于2018年11月9日周五 > > 下午3:31写道： > > > > > This is an issue even I am facing in my project currently. By running > the > > > spark interpreter in Scoped + Per Note mode, I do manage to execute > > > paragraphs in parallel, but it becomes very resource intensive and > times > > > out if I run more than 3-4 jobs in parallel on a 4-core cloud instance. > > > Typically a Thrift Transport exception is thrown after a prolonged > period > > > of inactivity, > > > > > > On Fri, 9 Nov 2018 at 06:54, Jeff Zhang <zjf...@gmail.com> wrote: > > > > > > > Which version do you use ? This seems a bug. Each note should have > its > > > own > > > > scheduler in scoped per note mode. > > > > > > > > <tecgi...@yahoo.com.invalid>于2018年11月9日周五 上午1:56写道： > > > > > > > > > Hi > > > > > > > > > > We use zeppelin in multi-user environment, the interpreter scope > mode > > > > > seems to allow notebook execution in serial only. If multiple users > > are > > > > > running their notebooks concurrently, these notebooks are queued > for > > > > serial > > > > > execution. If one notebook takes a long time to complete, it > > basically > > > > > blocks other notebooks from execution. To enable parallel notebook > > > > > execution, it seems we need to use the isolated mode, which > creates a > > > new > > > > > interpreter instance (run on separate JVM) per user. But this can > > > become > > > > > expensive (compute resource intensive). what is the suggested > > > interpreter > > > > > mode for multi-user environment? > > > > > > > > > > Thanks > > > > > Denny > > > > > > > > > > > > > > > > > > -- > > > Ajay Viswanathan > > > *Sr. Software Engineer, CAPS - Processing* > > > *A: *5th & 6th Floor | Skav 909 | 9/1 | Lavelle Road | Bangalore | > 560001 > > > *E: *yourem...@miqdigital.com > > > *M: *+00 (0)0 0000 0000 > > > *W: *wearemiq.com > > > [image: MiQ] <http://wearemiq.com/> > > > *Disclaimer: *This email and its attachments are confidential and are > > > intended solely for the use of the individual to whom it is addressed. > If > > > you are not the intended recipient of this email and its attachments, > you > > > must take no action based upon them, nor must you copy or show them to > > > anyone. No contracts or official orders shall > <https://maps.google.com/?q=.+No+contracts+or+official+orders+shall&entry=gmail&source=g> > be concluded by means of > > this > > > email. Please contact the sender if you believe you have received this > > > email in error. > > > > > > > > -- > Ajay Viswanathan > *Sr. Software Engineer, CAPS - Processing* > *A: *5th & 6th Floor | Skav 909 | 9/1 | Lavelle Road | Bangalore | 560001 > *E: *ajayviswanat...@miqdigital.com > *M: *+00 (0)0 0000 0000 > *W: *wearemiq.com > [image: MiQ] <http://wearemiq.com/> > *Disclaimer: *This email and its attachments are confidential and are > intended solely for the use of the individual to whom it is addressed. If > you are not the intended recipient of this email and its attachments, you > must take no action based upon them, nor must you copy or show them to > anyone. No contracts or official orders shall be concluded by means of this > email. Please contact the sender if you believe you have received this > email in error. >
