Re: Can you view thread dumps on spark UI if job finished
http://spark.apache.org/docs/latest/monitoring.html#spark-configuration-options
says
"Note that this information is only available for the duration of the
application by default. To view the web UI after the fact, set
spark.eventLog.enabled to true before starting the application. This
configures Spark to log Spark events that encode the information displayed
in the UI to persisted storage."
Backbutton.co.uk
¯\_(ツ)_/¯
♡۶Java♡۶RMI ♡۶
Make Use Method {MUM}
makeuse.org
On Thu, 9 Apr 2020 at 00:50, Ruijing Li wrote:
> Thanks Zahid, Yes I am using history server to see previous UIs.
>
> However, my question still remains on viewing old thread dumps, as I
> cannot see them on the old completed spark UIs, only when spark context is
> running.
>
> On Wed, Apr 8, 2020 at 4:01 PM Zahid Rahman wrote:
>
>> Spark UI is only available while SparkContext is running.
>>
>> However You can get to the Spark UI after your application completes or
>> crashes.
>>
>> To do this Spark includes a tool called the Spark History Server that
>> allows you to reconstruct the Spark UI.
>>
>> You can find up to date information on how to use this tool in the spark
>> documentation https://spark.apache.org/docs/latest/monitoring.html
>>
>>
>>
>>
>>
>> On Wed, 8 Apr 2020, 23:47 Ruijing Li, wrote:
>>
>>> Hi all,
>>>
>>> As stated in title, currently when I view the spark UI of a completed
>>> spark job, I see there are thread dump links in the executor tab, but
>>> clicking on them does nothing. Is it possible to see the thread dumps
>>> somehow even if the job finishes? On spark 2.4.5.
>>>
>>> Thanks.
>>> --
>>> Cheers,
>>> Ruijing Li
>>>
>> --
> Cheers,
> Ruijing Li
>
Re: Can you view thread dumps on spark UI if job finished
Thanks Zahid, Yes I am using history server to see previous UIs. However, my question still remains on viewing old thread dumps, as I cannot see them on the old completed spark UIs, only when spark context is running. On Wed, Apr 8, 2020 at 4:01 PM Zahid Rahman wrote: > Spark UI is only available while SparkContext is running. > > However You can get to the Spark UI after your application completes or > crashes. > > To do this Spark includes a tool called the Spark History Server that > allows you to reconstruct the Spark UI. > > You can find up to date information on how to use this tool in the spark > documentation https://spark.apache.org/docs/latest/monitoring.html > > > > > > On Wed, 8 Apr 2020, 23:47 Ruijing Li, wrote: > >> Hi all, >> >> As stated in title, currently when I view the spark UI of a completed >> spark job, I see there are thread dump links in the executor tab, but >> clicking on them does nothing. Is it possible to see the thread dumps >> somehow even if the job finishes? On spark 2.4.5. >> >> Thanks. >> -- >> Cheers, >> Ruijing Li >> > -- Cheers, Ruijing Li
Re: Can you view thread dumps on spark UI if job finished
Spark UI is only available while SparkContext is running. However You can get to the Spark UI after your application completes or crashes. To do this Spark includes a tool called the Spark History Server that allows you to reconstruct the Spark UI. You can find up to date information on how to use this tool in the spark documentation https://spark.apache.org/docs/latest/monitoring.html On Wed, 8 Apr 2020, 23:47 Ruijing Li, wrote: > Hi all, > > As stated in title, currently when I view the spark UI of a completed > spark job, I see there are thread dump links in the executor tab, but > clicking on them does nothing. Is it possible to see the thread dumps > somehow even if the job finishes? On spark 2.4.5. > > Thanks. > -- > Cheers, > Ruijing Li >
Can you view thread dumps on spark UI if job finished
Hi all, As stated in title, currently when I view the spark UI of a completed spark job, I see there are thread dump links in the executor tab, but clicking on them does nothing. Is it possible to see the thread dumps somehow even if the job finishes? On spark 2.4.5. Thanks. -- Cheers, Ruijing Li
