Re: [SparkSQL, SparkUI, RESTAPI] How to extract the WholeStageCodeGen ids from SparkUI

[Chitral Verma]

try explain codegen on your DF and then pardee the string

On Fri, 7 Apr, 2023, 3:53 pm Chenghao Lyu,  wrote:

> Hi,
>
> The detailed stage page shows the involved WholeStageCodegen Ids in its
> DAG visualization from the Spark UI when running a SparkSQL. (e.g., under
> the link
> node:18088/history/application_1663600377480_62091/stages/stage/?id=1=0).
>
> However, I have trouble extracting the WholeStageCodegen ids from the DAG
> visualization via the RESTAPIs. Is there any other way to get the
> WholeStageCodegen Ids information for each stage automatically?
>
> Cheers,
> Chenghao
>

[SparkSQL, SparkUI, RESTAPI] How to extract the WholeStageCodeGen ids from SparkUI

[Chenghao Lyu]

Hi,

The detailed stage page shows the involved WholeStageCodegen Ids in its DAG 
visualization from the Spark UI when running a SparkSQL. (e.g., under the link 
node:18088/history/application_1663600377480_62091/stages/stage/?id=1=0).

However, I have trouble extracting the WholeStageCodegen ids from the DAG 
visualization via the RESTAPIs. Is there any other way to get the 
WholeStageCodegen Ids information for each stage automatically?

Cheers,
Chenghao