Hello Users,
I have been seeing some weird issues when I upgraded my
EMR setup to 6.11 (which uses spark 3.3.2) , the call stack seems to point
to a code location where there is no explicit union, also I have
unionByName everywhere in the codebase with allowMissingColumns set to
True. I suspect the reported union in the exception is probably inserted in
the plan by spark optimizer?
spark.sql.utils.AnalysisException: Union can only be performed on tables
with the same number of columns, but the first table has 7 columns and the
second table has 6 columns
The issue seems to have disappeared when I did a quick test with spark
3.4.0 in my local setup, I am just curious if this is a known issue in
spark user/dev community or if I am missing something.
Thanks
Srivatsan
