I am using Spark 1.4.1 , in stand-alone mode, on a cluster of 3 nodes.
Using Spark sql and Hive Context, I am trying to run a simple scan query on
an existing Hive table (which is an external table consisting of rows in
text files stored in HDFS - it is NOT parquet, ORC or any other richer
format).
DataFrame res = hiveCtx.sql("SELECT * FROM lineitem WHERE L_LINENUMBER <
0");
What I observe is the performance of this full scan in Spark is not
comparable with Hive (it is almost 4 times slower). Checking the resource
usage, what I see is workers/executors do not do parallel scans but they
scan on a per-node basis (first executors from the worker(s) on node 1 do
reading from disk, while other two nodes are not doing I/O and just receive
data from the first node and through network, then 2nd node does the scan
and then the third one).
I also realized that if I load this data file directly from my spark
context (using textFile() ) and run count() on that (not using spark sql)
then I can get a better performance by increasing number of partitions. I
am just trying to do the same thing (increasing number of partitions in the
beginning) in Spark sql as:
var tab = sqlContext.read.table("lineitem");
tab.repartition(1000);
OR
tab.coalesce(1000);
but none of repartition() or coalesce() methods actually work - they do not
return an error, but if I check
var p = tab.rdd.partitions.size;
before and after calling any of them, it returns the same number of
partitions.
I am just wondering how I can change the number of partitions for a Hive
external table, in Spark Sql.
Any help/suggestion would be appreciated.
