Hi, You could use QueryExecutionListener or Spark listeners to intercept query execution events and extract whatever is required. That's what web UI does (as it's simply a bunch of SparkListeners --> https://youtu.be/mVP9sZ6K__Y ;-)).
Pozdrawiam, Jacek Laskowski ---- "The Internals Of" Online Books <https://books.japila.pl/> Follow me on https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski> On Fri, Apr 7, 2023 at 12:23 PM Chenghao Lyu <cheng...@cs.umass.edu> wrote: > Hi, > > The detailed stage page shows the involved WholeStageCodegen Ids in its > DAG visualization from the Spark UI when running a SparkSQL. (e.g., under > the link > node:18088/history/application_1663600377480_62091/stages/stage/?id=1&attempt=0). > > However, I have trouble extracting the WholeStageCodegen ids from the DAG > visualization via the RESTAPIs. Is there any other way to get the > WholeStageCodegen Ids information for each stage automatically? > > Cheers, > Chenghao >
