Looks like #2 is better choice.
On Tue, Dec 1, 2015 at 4:51 PM, Anfernee Xu <anfernee...@gmail.com> wrote:
> Thanks Ted, so 1) is off from the table, can I go with 2), yarn-cluster
> mode? As the driver is running as a Yarn container, it's should be OK for
> my usercase, isn't it?
>
> Anfernee
>
> On Tue, Dec 1, 2015 at 4:48 PM, Ted Yu <yuzhih...@gmail.com> wrote:
>
>> For #1, looks like the config is used in test suites:
>>
>> .set("spark.driver.allowMultipleContexts", "true")
>> ./sql/core/src/test/scala/org/apache/spark/sql/MultiSQLContextsSuite.scala
>> .set("spark.driver.allowMultipleContexts", "true")
>>
>> ./sql/core/src/test/scala/org/apache/spark/sql/execution/ExchangeCoordinatorSuite.scala
>> .set("spark.driver.allowMultipleContexts", "false")
>> val conf = new SparkConf().set("spark.driver.allowMultipleContexts",
>> "false")
>> .set("spark.driver.allowMultipleContexts", "true")
>> .set("spark.driver.allowMultipleContexts", "true"))
>> ./core/src/test/scala/org/apache/spark/SparkContextSuite.scala
>>
>> FYI
>>
>> On Tue, Dec 1, 2015 at 3:32 PM, Anfernee Xu <anfernee...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have a doubt regarding yarn-cluster mode and spark.driver.
>>> allowMultipleContexts for below usercases.
>>>
>>> I have a long running backend server where I will create a short-lived
>>> Spark job in response to each user request, base on the fact that by
>>> default multiple Spark Context cannot be created in the same JVM, looks
>>> like I have 2 choices
>>>
>>> 1) enable spark.driver.allowMultipleContexts
>>>
>>> 2) run my jobs in yarn-cluster mode instead yarn-client
>>>
>>> For 1) I cannot find any official document, so looks like it's not
>>> encouraged, isn't it?
>>> For 2), I want to make sure yarn-cluster will NOT hit such
>>> limitation(single SparkContext per VM), apparently I have to something in
>>> driver side to push the result set back to my application.
>>>
>>> Thanks
>>>
>>> --
>>> --Anfernee
>>>
>>
>>
>
>
> --
> --Anfernee
>
