Hi Winston, I am looking for a way to access the LogicalPlan object in Scala. Not sure if explain function would serve the purpose.
On Wed, Aug 2, 2023 at 9:14 AM Winston Lai <weiruanl...@gmail.com> wrote: > Hi Vibhatha, > > Have you tried pyspark.sql.DataFrame.explain — PySpark 3.4.1 > documentation (apache.org) > <https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.explain.html#pyspark.sql.DataFrame.explain> > before? > I am not sure what infra that you have, you can try this first. If it > doesn't work, you may share more info such as what platform you are running > your Spark jobs on, what cloud servies you are using ... > > On Wednesday, August 2, 2023, Vibhatha Abeykoon <vibha...@gmail.com> > wrote: > >> Hello, >> >> I recently upgraded the Spark version to 3.4.1 and I have encountered a >> few issues. In my previous code, I was able to extract the logical plan >> using `df.queryExecution` (df: DataFrame and in Scala), but it seems like >> in the latest API it is not supported. Is there a way to extract the >> logical plan or optimized plan from a dataframe or dataset in Spark 3.4.1? >> >> Best, >> Vibhatha >> > -- Vibhatha Abeykoon
