Re: Spark Thrift Server Concurrency

2016-06-27 Thread Prabhu Joseph
Spark Thrift Server is started with ./sbin/start-thriftserver.sh --master yarn-client --hiveconf hive.server2.thrift.port=10001 --num-executors 4 --executor-cores 2 --executor-memory 4G --conf spark.scheduler.mode=FAIR 20 parallel below queries are executed select distinct val2 from philips1

Re: Spark Thrift Server Concurrency

2016-06-23 Thread Michael Segel
Hi, There are a lot of moving parts and a lot of unknowns from your description. Besides the version stuff. How many executors, how many cores? How much memory? Are you persisting (memory and disk) or just caching (memory) During the execution… same tables… are you seeing a lot of

Re: Spark Thrift Server Concurrency

2016-06-23 Thread Mich Talebzadeh
which version of spark and are you using YARN in client mode or cluster mode? Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Spark Thrift Server Concurrency

2016-06-23 Thread Prabhu Joseph
Hi All, On submitting 20 parallel same SQL query to Spark Thrift Server, the query execution time for some queries are less than a second and some are more than 2seconds. The Spark Thrift Server logs shows all 20 queries are submitted at same time 16/06/23 12:12:01 but the result schema are at