Hi Tanin,
running your test with option "spark.sql.planChangeLog.level" set to
"info" or "warn" (depending on your Spark log level) will show you
insights into the planning (which rules are applied, how long rules
take, how many iterations are done).
Hoping this helps,
Enrico
Am 25.10.22 um 21:54 schrieb Tanin Na Nakorn:
Hi All,
Our data job is very complex (e.g. 100+ joins), and we have switched
from RDD to Dataset recently.
We've found that the unit test takes much longer. We profiled it and
have found that it's the planning phase that is slow, not execution.
I wonder if anyone has encountered this issue before and if there's a
way to make the planning phase faster (e.g. maybe disabling certain
optimizers).
Any thoughts or input would be appreciated.
Thank you,
Tanin
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
