Re: multiple pyspark instances simultaneously (same time)
Did you read https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application Thanks Best Regards On Thu, Oct 15, 2015 at 11:31 PM, jeff.sadow...@gmail.com < jeff.sadow...@gmail.com> wrote: > I am having issues trying to setup spark to run jobs simultaneously. > > I thought I wanted FAIR scheduling? > > I used the templated fairscheduler.xml as is when I start pyspark I see the > 3 expected pools: > production, test, and default > > when I login as second user and run pyspark > I see the expected pools as that user as well > > when I open up a webbrowser to http://master:8080 > > I see my first user's state is running and my second user's state is > waiting > > so I try putting them both in the production pool which is fair scheduler > > When I refresh http://master:8080 > > the second user's status is still waiting. > > If I try to run something as the second user I get > > "Initial job has not accepted any resources" > > Maybe fair queuing is not what I want? > > I'm starting pyspark as follows > > pyspark --master spark://master:7077 > > I started spark as follows > > start-all.sh > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/multiple-pyspark-instances-simultaneously-same-time-tp25079.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Fwd: multiple pyspark instances simultaneously (same time)
On Thu, Oct 22, 2015 at 5:40 AM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Did you read
> https://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application
>
I did.
I had set the option
spark.scheduler.mode FAIR
in conf/spark-defaults.conf
and
created fairscheduler.xml
with the two pools production and test
and noticed when I start pyspark and run
sc.setLocalProperty("spark.scheduler.pool", null)
does not work it gives me
NameError: name 'null' is not defined
I tried getting in the production pool so I would have FAIR scheduling
sc.setLocalProperty("spark.scheduler.pool", *"production"*)
and
sc.getLocalProperty("spark.scheduler.pool")
shows
u'production'
I also noticed I could join pools that are not created and it shows that I
am in that uncreated pool as if
sc.setLocalProperty("spark.scheduler.pool", *"production"*)
wasn't really doing anything.
and as I was say
it still behaves as if it isn't doing FAIR scheduling if I am in the
production pool
when I start pyspark as a second user and do
.
sc.setLocalProperty("spark.scheduler.pool", *"production"*)
It still says waiting on the master's status page.
and still gives me
Initial job has not accepted any resources
If I try to do something as that second user.
> Thanks
> Best Regards
>
> On Thu, Oct 15, 2015 at 11:31 PM, jeff.sadow...@gmail.com <
> jeff.sadow...@gmail.com> wrote:
>
>> I am having issues trying to setup spark to run jobs simultaneously.
>>
>> I thought I wanted FAIR scheduling?
>>
>> I used the templated fairscheduler.xml as is when I start pyspark I see
>> the
>> 3 expected pools:
>> production, test, and default
>>
>> when I login as second user and run pyspark
>> I see the expected pools as that user as well
>>
>> when I open up a webbrowser to http://master:8080
>>
>> I see my first user's state is running and my second user's state is
>> waiting
>>
>> so I try putting them both in the production pool which is fair scheduler
>>
>> When I refresh http://master:8080
>>
>> the second user's status is still waiting.
>>
>> If I try to run something as the second user I get
>>
>> "Initial job has not accepted any resources"
>>
>> Maybe fair queuing is not what I want?
>>
>> I'm starting pyspark as follows
>>
>> pyspark --master spark://master:7077
>>
>> I started spark as follows
>>
>> start-all.sh
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/multiple-pyspark-instances-simultaneously-same-time-tp25079.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>
multiple pyspark instances simultaneously (same time)
I am having issues trying to setup spark to run jobs simultaneously. I thought I wanted FAIR scheduling? I used the templated fairscheduler.xml as is when I start pyspark I see the 3 expected pools: production, test, and default when I login as second user and run pyspark I see the expected pools as that user as well when I open up a webbrowser to http://master:8080 I see my first user's state is running and my second user's state is waiting so I try putting them both in the production pool which is fair scheduler When I refresh http://master:8080 the second user's status is still waiting. If I try to run something as the second user I get "Initial job has not accepted any resources" Maybe fair queuing is not what I want? I'm starting pyspark as follows pyspark --master spark://master:7077 I started spark as follows start-all.sh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/multiple-pyspark-instances-simultaneously-same-time-tp25144.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
multiple pyspark instances simultaneously (same time)
I am having issues trying to setup spark to run jobs simultaneously. I thought I wanted FAIR scheduling? I used the templated fairscheduler.xml as is when I start pyspark I see the 3 expected pools: production, test, and default when I login as second user and run pyspark I see the expected pools as that user as well when I open up a webbrowser to http://master:8080 I see my first user's state is running and my second user's state is waiting so I try putting them both in the production pool which is fair scheduler When I refresh http://master:8080 the second user's status is still waiting. If I try to run something as the second user I get "Initial job has not accepted any resources" Maybe fair queuing is not what I want? I'm starting pyspark as follows pyspark --master spark://master:7077 I started spark as follows start-all.sh
multiple pyspark instances simultaneously (same time)
I am having issues trying to setup spark to run jobs simultaneously. I thought I wanted FAIR scheduling? I used the templated fairscheduler.xml as is when I start pyspark I see the 3 expected pools: production, test, and default when I login as second user and run pyspark I see the expected pools as that user as well when I open up a webbrowser to http://master:8080 I see my first user's state is running and my second user's state is waiting so I try putting them both in the production pool which is fair scheduler When I refresh http://master:8080 the second user's status is still waiting. If I try to run something as the second user I get "Initial job has not accepted any resources" Maybe fair queuing is not what I want? I'm starting pyspark as follows pyspark --master spark://master:7077 I started spark as follows start-all.sh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/multiple-pyspark-instances-simultaneously-same-time-tp25079.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
