Hi Vibhatha, Have you tried pyspark.sql.DataFrame.explain — PySpark 3.4.1 documentation (apache.org) <https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.explain.html#pyspark.sql.DataFrame.explain> before? I am not sure what infra that you have, you can try this first. If it doesn't work, you may share more info such as what platform you are running your Spark jobs on, what cloud servies you are using ...
On Wednesday, August 2, 2023, Vibhatha Abeykoon <vibha...@gmail.com> wrote: > Hello, > > I recently upgraded the Spark version to 3.4.1 and I have encountered a > few issues. In my previous code, I was able to extract the logical plan > using `df.queryExecution` (df: DataFrame and in Scala), but it seems like > in the latest API it is not supported. Is there a way to extract the > logical plan or optimized plan from a dataframe or dataset in Spark 3.4.1? > > Best, > Vibhatha >
