Hi guys, hundreds of spark jobs run on my company every day. We are running Spark 3.1.2 and we want enable Adaptive Query Execution (AQE) for all of them. We can't upgrade to 3.2 right now so we want enable it explicitly using appropriate conf when spark submit.
Some of them fails when enable AQE but I can't discover what is happening. In order to give your information I prepared a small snippet for spark shell that fails in Spark 3.1 when AQE enabled and works when disabled. It also work in 3.2 but I think maybe is a bug that can be fixed for 3.1.3. The code and explanation can be found here: https://issues.apache.org/jira/browse/SPARK-37898 Regards -- Gaspar Muñoz Soria
