[Spark 1.0.1][SparkSQL] reduce stage of shuffle is slow。

2014-07-28 Thread Earthson
I'm using SparkSQL with Hive 0.13, here is the SQL for inserting a partition with 2048 buckets. pre sqlsc.set(spark.sql.shuffle.partitions, 2048) hql(|insert %s table mz_log |PARTITION (date='%s') |select * from tmp_mzlog

Re: [Spark 1.0.1][SparkSQL] reduce stage of shuffle is slow。

2014-07-28 Thread Earthson
spark.MapOutputTrackerMasterActor: Asked to send map output locations for shuffle 0 to takes too much time, what should I do? What is the correct configuration? blockManager timeout if I using a small number of reduce partition.