multiple pyspark instances simultaneously (same time)
I am having issues trying to setup spark to run jobs simultaneously. I thought I wanted FAIR scheduling? I used the templated fairscheduler.xml as is when I start pyspark I see the 3 expected pools: production, test, and default when I login as second user and run pyspark I see the expected pools as that user as well when I open up a webbrowser to http://master:8080 I see my first user's state is running and my second user's state is waiting so I try putting them both in the production pool which is fair scheduler When I refresh http://master:8080 the second user's status is still waiting. If I try to run something as the second user I get "Initial job has not accepted any resources" Maybe fair queuing is not what I want? I'm starting pyspark as follows pyspark --master spark://master:7077 I started spark as follows start-all.sh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/multiple-pyspark-instances-simultaneously-same-time-tp25144.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Can not subscript to mailing list
I am having issues subscribing to the user@spark.apache.org mailing list. I would like to be added to the mailing list so I can post some configuration questions I have to the list that I do not see asked on the list. When I tried adding myself I got an email titled "confirm subscribe to user@spark.apache.org" but after replying as it says to do I get nothing. I tried today to remove and re-add myself and I got a reply back saying I was not on the list when trying to unsubscribe. When I tried to add myself again I don't get any emails from it this time. I'm getting other email from other people and nothing is in spam. I tried with a second email account as well and the same thing is happening on it. I got the initial "confirm subscribe to user@spark.apache.org" email but after replying I get nothing. I can't even get another "confirm subscribe to user@spark.apache.org" message. Both of my emails are from google servers one is an organization email the first is a personal google email -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-not-subscript-to-mailing-list-tp25143.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
multiple pyspark instances simultaneously (same time)
I am having issues trying to setup spark to run jobs simultaneously. I thought I wanted FAIR scheduling? I used the templated fairscheduler.xml as is when I start pyspark I see the 3 expected pools: production, test, and default when I login as second user and run pyspark I see the expected pools as that user as well when I open up a webbrowser to http://master:8080 I see my first user's state is running and my second user's state is waiting so I try putting them both in the production pool which is fair scheduler When I refresh http://master:8080 the second user's status is still waiting. If I try to run something as the second user I get "Initial job has not accepted any resources" Maybe fair queuing is not what I want? I'm starting pyspark as follows pyspark --master spark://master:7077 I started spark as follows start-all.sh -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/multiple-pyspark-instances-simultaneously-same-time-tp25079.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
