What kind of Query are you performing?
You should set something like 2 partition per core that would be 400 Mb per
partition.
As you have a lot of ram I suggest to cache the whole table, performance will
increase a lot.
Paolo
Inviata dal mio Windows Phone
Da: vdiwakar.malladi<mailto:vdiwakar.mall...@gmail.com>
Inviato: 05/12/2014 18:52
A: u...@spark.incubator.apache.org<mailto:u...@spark.incubator.apache.org>
Oggetto: Optimized spark configuration
Hi
Could any one help what would be better / optimized configuration for driver
memory, worker memory, number of parallelisms etc., parameters to be
configured when we are running 1 master node (it itself acting as slave node
also) and 1 slave node. Both are of 32 GB RAM with 4 cores.
On this, I loaded approx. 17M rows of data (3.2 GB) to hive store and when I
try to execute a query on this from jdbc thrift server, it is taking about
10-12 sec to retrieve the data which I think is too much.
Or guide please guide me any tutorial which will explain about these
optimize configurations.
Thanks in advance.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Optimized-spark-configuration-tp20495.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
