Unsubscribe

[Zoran Jeremic]

Unsubscribe

Re: Extracting Logical Plan

[Winston Lai]

Hi Vibhatha,

Have you tried pyspark.sql.DataFrame.explain — PySpark 3.4.1 documentation
(apache.org)

before?
I am not sure what infra that you have, you can try this first. If it
doesn't work, you may share more info such as what platform you are running
your Spark jobs on, what cloud servies you are using ...

On Wednesday, August 2, 2023, Vibhatha Abeykoon  wrote:

> Hello,
>
> I recently upgraded the Spark version to 3.4.1 and I have encountered a
> few issues. In my previous code, I was able to extract the logical plan
> using `df.queryExecution` (df: DataFrame and in Scala), but it seems like
> in the latest API it is not supported. Is there a way to extract the
> logical plan or optimized plan from a dataframe or dataset in Spark 3.4.1?
>
> Best,
> Vibhatha
>

Extracting Logical Plan

[Vibhatha Abeykoon]

Hello,

I recently upgraded the Spark version to 3.4.1 and I have encountered a few
issues. In my previous code, I was able to extract the logical plan using
`df.queryExecution` (df: DataFrame and in Scala), but it seems like in the
latest API it is not supported. Is there a way to extract the logical plan
or optimized plan from a dataframe or dataset in Spark 3.4.1?

Best,
Vibhatha

Unsubscribe

[Alex Landa]

Unsubscribe