The expensive query can take all executor slots, but no task occupy the executor permanently. i.e. The second job can possibly to take some resources to execute in-between tasks of the expensive queries.
Can the fair scheduler mode help in this case? Or is it possible to setup thrift such that no query is taking all resources. From: Sean Owen [mailto:so...@cloudera.com] Sent: Wednesday, April 1, 2015 12:28 AM To: Asad Khan Cc: user@spark.apache.org Subject: Re: Using 'fair' scheduler mode Does the expensive query take all executor slots? Then there is nothing for any other job to use regardless of scheduling policy. On Mar 31, 2015 9:20 PM, "asadrao" <as...@microsoft.com<mailto:as...@microsoft.com>> wrote: Hi, I am using the Spark ‘fair’ scheduler mode. I have noticed that if the first query is a very expensive query (ex: ‘select *’ on a really big data set) than any subsequent query seem to get blocked. I would have expected the second query to run in parallel since I am using the ‘fair’ scheduler mode not the ‘fifo’. I am submitting the query through thrift server. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Using-fair-scheduler-mode-tp22328.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org> For additional commands, e-mail: user-h...@spark.apache.org<mailto:user-h...@spark.apache.org>
