Hi Spark Devs, I really need your help understanding the relationship between HashAggregateExec, TungstenAggregationIterator and UnsafeFixedWidthAggregationMap.
While exploring UnsafeFixedWidthAggregationMap and how it's used I've noticed that it's for HashAggregateExec and TungstenAggregationIterator exclusively. And given that TungstenAggregationIterator is used exclusively in HashAggregateExec and the use of UnsafeFixedWidthAggregationMap in both seems to be almost the same (if not the same), I've got a question I cannot seem to answer myself. Since HashAggregateExec supports Whole-Stage Codegen HashAggregateExec.doExecute won't be used at all, but doConsume and doProduce (unless codegen is disabled). Is that correct? If so, TungstenAggregationIterator is not used at all, but UnsafeFixedWidthAggregationMap is used directly instead (in the Java code that uses createHashMap or finishAggregate). Is that correct? Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski Mastering Spark SQL https://bit.ly/mastering-spark-sql Spark Structured Streaming https://bit.ly/spark-structured-streaming Mastering Kafka Streams https://bit.ly/mastering-kafka-streams Follow me at https://twitter.com/jaceklaskowski
