Re: Spark SQL performance: version 1.6 vs version 1.5

2016-02-12 Thread Le Tien Dung
can revert to the old behavior by setting the > spark.sql.specializeSingleDistinctAggPlanning option to false. See also: > https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/SQLConf.scala#L452-L462 > > HTH > > Kind regards, > > Her

Spark SQL performance: version 1.6 vs version 1.5

2016-02-12 Thread Le Tien Dung
Hi folks, I have compared the performance of Spark SQL version 1.6.0 and version 1.5.2. In a simple case, Spark 1.6.0 is quite faster than Spark 1.5.2. However in a more complex query - in our case it is an aggregation query with grouping sets, Spark SQL version 1.6.0 is very much slower than