Thanks for the insight!
_______________ *Mike Wright* Principal Architect, Software Engineering S&P Capital IQ and SNL 434-951-7816 *p* 434-244-4466 *f* 540-470-0119 *m* mwri...@snl.com On Fri, Dec 11, 2015 at 2:38 PM, Michael Armbrust <mich...@databricks.com> wrote: > The way that we do this is to have a single context with a server in front > that multiplexes jobs that use that shared context. Even if you aren't > sharing data this is going to give you the best fine grained sharing of the > resources that the context is managing. > > On Fri, Dec 11, 2015 at 10:55 AM, Mike Wright <mwri...@snl.com> wrote: > >> Somewhat related - What's the correct implementation when you have a >> single cluster to support multiple jobs that are unrelated and NOT sharing >> data? I was directed to figure out, via job server, to support "multiple >> contexts" and explained that multiple contexts per JVM is not really >> supported. So, via job server, how does one support multiple contexts in >> DIFFERENT JVM's? I specify multiple contexts in the conf file and the >> initialization of the subsequent contexts fail. >> >> >> >> On Fri, Dec 4, 2015 at 3:37 PM, Michael Armbrust <mich...@databricks.com> >> wrote: >> >>> On Fri, Dec 4, 2015 at 11:24 AM, Anfernee Xu <anfernee...@gmail.com> >>> wrote: >>> >>>> If multiple users are looking at the same data set, then it's good >>>> choice to share the SparkContext. >>>> >>>> But my usercases are different, users are looking at different data(I >>>> use custom Hadoop InputFormat to load data from my data source based on the >>>> user input), the data might not have any overlap. For now I'm taking below >>>> approach >>>> >>> >>> Still if you want fine grained sharing of compute resources as well, you >>> want to using single SparkContext. >>> >> >> >
