Hi,
The page in the URL explains the old style of physical plan output.
The current style adds "*" as a prefix of each operation that the
whole-stage codegen can be apply to.
So, in your test case, whole-stage codegen has been already enabled!!
FYI. I think that it is a good topic for d...@spark.apache.org.
Kazuaki Ishizaki
From: Koert Kuipers <ko...@tresata.com>
To: "user@spark.apache.org" <user@spark.apache.org>
Date: 2017/04/05 05:12
Subject: how do i force unit test to do whole stage codegen
i wrote my own expression with eval and doGenCode, but doGenCode never
gets called in tests.
also as a test i ran this in a unit test:
spark.range(10).select('id as 'asId).where('id === 4).explain
according to
https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-sql-whole-stage-codegen.html
this is supposed to show:
== Physical Plan ==
WholeStageCodegen
: +- Project [id#0L AS asId#3L]
: +- Filter (id#0L = 4)
: +- Range 0, 1, 8, 10, [id#0L]
but it doesn't. instead it shows:
== Physical Plan ==
*Project [id#12L AS asId#15L]
+- *Filter (id#12L = 4)
+- *Range (0, 10, step=1, splits=Some(4))
so i am again missing the WholeStageCodegen. any idea why?
i create spark session for unit tests simply as:
val session = SparkSession.builder
.master("local[*]")
.appName("test")
.config("spark.sql.shuffle.partitions", 4)
.getOrCreate()
