Thanks, Mark. Then, I guess JobServer can fundamentally solve my problem, so that jobs can be submitted at different time and still share RDDs.
Best Regards, Jia On Jan 17, 2016, at 3:44 PM, Mark Hamstra <m...@clearstorydata.com> wrote: > There is a 1-to-1 relationship between Spark Applications and SparkContexts > -- fundamentally, a Spark Applications is a program that creates and uses a > SparkContext, and that SparkContext is destroyed when then Application ends. > A jobserver generically and the Spark JobServer specifically is an > Application that keeps a SparkContext open for a long time and allows many > Jobs to be be submitted and run using that shared SparkContext. > > More than one Application/SparkContext unavoidably implies more than one JVM > process per Worker -- Applications/SparkContexts cannot share JVM processes. > > On Sun, Jan 17, 2016 at 1:15 PM, Jia <jacqueline...@gmail.com> wrote: > Hi, Mark, sorry for the confusion. > > Let me clarify, when an application is submitted, the master will tell each > Spark worker to spawn an executor JVM process. All the task sets of the > application will be executed by the executor. After the application runs to > completion. The executor process will be killed. > But I hope that all applications submitted can run in the same executor, can > JobServer do that? If so, it’s really good news! > > Best Regards, > Jia > > On Jan 17, 2016, at 3:09 PM, Mark Hamstra <m...@clearstorydata.com> wrote: > >> You've still got me confused. The SparkContext exists at the Driver, not on >> an Executor. >> >> Many Jobs can be run by a SparkContext -- it is a common pattern to use >> something like the Spark Jobserver where all Jobs are run through a shared >> SparkContext. >> >> On Sun, Jan 17, 2016 at 12:57 PM, Jia Zou <jacqueline...@gmail.com> wrote: >> Hi, Mark, sorry, I mean SparkContext. >> I mean to change Spark into running all submitted jobs (SparkContexts) in >> one executor JVM. >> >> Best Regards, >> Jia >> >> On Sun, Jan 17, 2016 at 2:21 PM, Mark Hamstra <m...@clearstorydata.com> >> wrote: >> -dev >> >> What do you mean by JobContext? That is a Hadoop mapreduce concept, not >> Spark. >> >> On Sun, Jan 17, 2016 at 7:29 AM, Jia Zou <jacqueline...@gmail.com> wrote: >> Dear all, >> >> Is there a way to reuse executor JVM across different JobContexts? Thanks. >> >> Best Regards, >> Jia >> >> >> > >
