Yup it is a different path. It runs GeneratedAggregate. On Wed, May 20, 2015 at 11:43 PM, Pramod Biligiri <pramodbilig...@gmail.com> wrote:
> I hadn't turned on codegen. I enabled it and ran it again, it is running > 4-5 times faster now! :) > Since my log statements are no longer appearing, I presume the code path > seems quite different from the earlier hashmap related stuff in > Aggregates.scala? > > Pramod > > On Wed, May 20, 2015 at 9:18 PM, Reynold Xin <r...@databricks.com> wrote: > >> Does this turn codegen on? I think the performance is fairly different >> when codegen is turned on. >> >> For 1.5, we are investigating having codegen on by default, so users get >> much better performance out of the box. >> >> >> On Wed, May 20, 2015 at 5:24 PM, Pramod Biligiri < >> pramodbilig...@gmail.com> wrote: >> >>> Hi, >>> Somewhat similar to Daniel Mescheder's mail yesterday on SparkSql, I >>> have a data point regarding the performance of Group By, indicating there's >>> excessive GC and it's impacting the throughput. I want to know if the new >>> memory manager for aggregations ( >>> https://github.com/apache/spark/pull/5725/) is going to address this >>> kind of issue. >>> >>> I only have a small amount of data on each node (~360MB) with a large >>> heap size (18 Gig). I still see 2-3 minor collections happening whenever I >>> do a Select Sum() with a group by(). I have tried with different sizes for >>> Young Generation without much effect, though not with different GC >>> algorithms (Hm..I ought to try reducing the rdd storage fraction perhaps). >>> >>> I have made a chart of my results [1] by adding timing code to >>> Aggregates.scala. The query is actually Query 2 from Berkeley's AmpLab >>> benchmark, running over 10 million records. The chart is from one of the 4 >>> worker nodes in the cluster. >>> >>> I am trying to square this with a claim on the Project Tungsten blog >>> post [2]: "When profiling Spark user applications, we’ve found that a >>> large fraction of the CPU time is spent waiting for data to be fetched from >>> main memory. " >>> >>> Am I correct in assuming that SparkSql is yet to reach that level of >>> efficiency, at least in aggregation operations? >>> >>> Thanks. >>> >>> [1] - >>> https://docs.google.com/spreadsheets/d/1HSqYfic3n5s9i4Wsi1Qg0FKN_AWz2vV7_6RRMrtzplQ/edit#gid=481134174 >>> [2] >>> https://databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html >>> >>> Pramod >>> >> >> >
