it looks that this commit ( https://github.com/apache/spark/commit/a85490659f45410be3588c669248dc4f534d2a71) do the trick.
[image: image.png] Don't you think, this bug is enough important to incluide in 3.1 branch? Regards El jue, 20 ene 2022 a las 8:55, Gaspar Muñoz (<gmu...@datiobd.com>) escribió: > Hi guys, > > hundreds of spark jobs run on my company every day. We are running Spark > 3.1.2 and we want enable Adaptive Query Execution (AQE) for all of them. > We can't upgrade to 3.2 right now so we want enable it explicitly using > appropriate conf when spark submit. > > Some of them fails when enable AQE but I can't discover what is > happening. In order to give your information I prepared a small snippet > for spark shell that fails in Spark 3.1 when AQE enabled and works when > disabled. It also work in 3.2 but I think maybe is a bug that can be fixed > for 3.1.3. > > The code and explanation can be found here: > https://issues.apache.org/jira/browse/SPARK-37898 > > Regards > -- > Gaspar Muñoz Soria > -- Gaspar Muñoz Soria Vía de las dos Castillas, 33, Ática 4, 3ª Planta 28224 Pozuelo de Alarcón, Madrid Tel: +34 91 828 6473
