Re: Question about yarn-cluster mode and spark.driver.allowMultipleContexts

On Tue, Dec 1, 2015 at 9:43 PM, Anfernee Xu wrote: > But I have a single server(JVM) that is creating SparkContext, are you > saying Spark supports multiple SparkContext in the same JVM? Could you > please clarify on this? I'm confused. Nothing you said so far requires

Question about yarn-cluster mode and spark.driver.allowMultipleContexts

Hi, I have a doubt regarding yarn-cluster mode and spark.driver. allowMultipleContexts for below usercases. I have a long running backend server where I will create a short-lived Spark job in response to each user request, base on the fact that by default multiple Spark Context cannot be created

Re: Question about yarn-cluster mode and spark.driver.allowMultipleContexts

For #1, looks like the config is used in test suites: .set("spark.driver.allowMultipleContexts", "true") ./sql/core/src/test/scala/org/apache/spark/sql/MultiSQLContextsSuite.scala .set("spark.driver.allowMultipleContexts", "true")

Re: Question about yarn-cluster mode and spark.driver.allowMultipleContexts

Looks like #2 is better choice. On Tue, Dec 1, 2015 at 4:51 PM, Anfernee Xu wrote: > Thanks Ted, so 1) is off from the table, can I go with 2), yarn-cluster > mode? As the driver is running as a Yarn container, it's should be OK for > my usercase, isn't it? > > Anfernee >

Re: Question about yarn-cluster mode and spark.driver.allowMultipleContexts

Yep, you shouldn't enable *spark.driver.allowMultipleContexts* since it has the potential to cause extremely difficult-to-debug task failures; it was originally introduced as an escape-hatch to allow users whose workloads happened to work "by accident" to continue using multiple active contexts,

Re: Question about yarn-cluster mode and spark.driver.allowMultipleContexts

Thanks Ted, so 1) is off from the table, can I go with 2), yarn-cluster mode? As the driver is running as a Yarn container, it's should be OK for my usercase, isn't it? Anfernee On Tue, Dec 1, 2015 at 4:48 PM, Ted Yu wrote: > For #1, looks like the config is used in test

Re: Question about yarn-cluster mode and spark.driver.allowMultipleContexts

On Tue, Dec 1, 2015 at 3:32 PM, Anfernee Xu wrote: > I have a long running backend server where I will create a short-lived Spark > job in response to each user request, base on the fact that by default > multiple Spark Context cannot be created in the same JVM, looks like

Re: Question about yarn-cluster mode and spark.driver.allowMultipleContexts

Thanks Marcelo, But I have a single server(JVM) that is creating SparkContext, are you saying Spark supports multiple SparkContext in the same JVM? Could you please clarify on this? Thanks Anfernee On Tue, Dec 1, 2015 at 8:14 PM, Marcelo Vanzin wrote: > On Tue, Dec 1,