The expensive query can take all executor slots, but no task occupy the 
executor permanently.
i.e. The second job can possibly to take some resources to execute in-between 
tasks of the expensive queries.

Can the fair scheduler mode help in this case? Or is it possible to setup 
thrift such that no query is taking all resources.

From: Sean Owen [mailto:[email protected]]
Sent: Wednesday, April 1, 2015 12:28 AM
To: Asad Khan
Cc: [email protected]
Subject: Re: Using 'fair' scheduler mode


Does the expensive query take all executor slots? Then there is nothing for any 
other job to use regardless of scheduling policy.
On Mar 31, 2015 9:20 PM, "asadrao" 
<[email protected]<mailto:[email protected]>> wrote:
Hi, I am using the Spark ‘fair’ scheduler mode. I have noticed that if the
first query is a very expensive query (ex: ‘select *’ on a really big data
set) than any subsequent query seem to get blocked. I would have expected
the second query to run in parallel since I am using the ‘fair’ scheduler
mode not the ‘fifo’. I am submitting the query through thrift server.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Using-fair-scheduler-mode-tp22328.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: 
[email protected]<mailto:[email protected]>
For additional commands, e-mail: 
[email protected]<mailto:[email protected]>

Reply via email to