Hi Winston, I need to use the LogicalPlan object and process it with another function I have written. In earlier Spark versions we can access that via the dataframe object. So if it can be accessed via the UI, is there an API to access the object?
On Wed, Aug 2, 2023 at 1:24 PM Winston Lai <weiruanl...@gmail.com> wrote: > Hi Vibhatha, > > How about reading the logical plan from Spark UI, do you have access to > the Spark UI? I am not sure what infra you run your Spark jobs on. Usually > you should be able to view the logical and physical plan under Spark UI in > text version at least. It is independent from the language (e.g., > scala/Python/R) that you use to run Spark. > > > On Wednesday, August 2, 2023, Vibhatha Abeykoon <vibha...@gmail.com> > wrote: > >> Hi Winston, >> >> I am looking for a way to access the LogicalPlan object in Scala. Not >> sure if explain function would serve the purpose. >> >> On Wed, Aug 2, 2023 at 9:14 AM Winston Lai <weiruanl...@gmail.com> wrote: >> >>> Hi Vibhatha, >>> >>> Have you tried pyspark.sql.DataFrame.explain — PySpark 3.4.1 >>> documentation (apache.org) >>> <https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.explain.html#pyspark.sql.DataFrame.explain> >>> before? >>> I am not sure what infra that you have, you can try this first. If it >>> doesn't work, you may share more info such as what platform you are running >>> your Spark jobs on, what cloud servies you are using ... >>> >>> On Wednesday, August 2, 2023, Vibhatha Abeykoon <vibha...@gmail.com> >>> wrote: >>> >>>> Hello, >>>> >>>> I recently upgraded the Spark version to 3.4.1 and I have encountered a >>>> few issues. In my previous code, I was able to extract the logical plan >>>> using `df.queryExecution` (df: DataFrame and in Scala), but it seems like >>>> in the latest API it is not supported. Is there a way to extract the >>>> logical plan or optimized plan from a dataframe or dataset in Spark 3.4.1? >>>> >>>> Best, >>>> Vibhatha >>>> >>> -- >> Vibhatha Abeykoon >> > -- Vibhatha Abeykoon
