Spark-UI stages and other tabs not accessible in standalone mode when reverse-proxy is enabled

Hi Team, We're encountering an issue with Spark UI. I've documented the details here: https://issues.apache.org/jira/browse/SPARK-47232 When enabled reverse proxy in master and worker configOptions. We're not able to access different tabs available in spark UI e.g.(stages, environment, storage etc

Spark-UI stages and other tabs not accessible in standalone mode when reverse-proxy is enabled

Hi Team, We're encountering an issue with Spark UI. When enabled reverse proxy in master and worker configOptions. We're not able to access different tabs available in spark UI e.g.(stages, environment, storage etc.) We're deploying spark through bitnami helm chart : https://github.com/bitnami

Spark UI - Bug Executors tab when using proxy port

Hello everyone, I’m really sorry to use this mailing list, but seems impossible to notify a strange behaviour that is happening with the Spark UI. I’m sending also the link to the stackoverflow question here https://stackoverflow.com/questions/76632692/spark-ui-executors-tab-its-empty I’m

CVE-2023-32007: Apache Spark: Shell command injection via Spark UI

Severity: important Affected versions: - Apache Spark 3.1.1 before 3.2.2 Description: ** UNSUPPORTED WHEN ASSIGNED ** The Apache Spark UI offers the possibility to enable ACLs via the configuration option spark.acls.enable. With an authentication filter, this checks whether a user has access

Re: RDD block has negative value in Spark UI

Already a know minor issue https://issues.apache.org/jira/plugins/servlet/mobile#issue/SPARK-10141 On Wed, 7 Dec 2022, 15:09 K B M Kaala Subhikshan, < kbmkaalasubhiks...@gmail.com> wrote: > Could you explain why the RDD block has a negative value? > >

Filtering by job group in the Spark UI / API

Hi All, Is there a way that we can filter in all the jobs from the history server UI / in Spark's API based on the Job Group to which the job belongs to? Ideally we would like to supply a particular job group, and only see the jobs associated with that job group in the UI. Thanks, Yeachan

CVE-2022-33891: Apache Spark shell command injection vulnerability via Spark UI

Severity: important Description: The Apache Spark UI offers the possibility to enable ACLs via the configuration option spark.acls.enable. With an authentication filter, this checks whether a user has access permissions to view or modify the application. If ACLs are enabled, a code path

Re: Reverse proxy for Spark UI on Kubernetes

ng wrote: > >> It is like Web Application Proxy in YARN ( >> https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html), >> to provide easy access for Spark UI when the Spark application is running. >> >> When running Spark on Kuberne

Re: Reverse proxy for Spark UI on Kubernetes

for Spark UI when the Spark application is running. > > When running Spark on Kubernetes with S3, there is no YARN. The reverse > proxy here is to behave like that Web Application Proxy. It will > simplify settings to access Spark UI on Kubernetes. > > > On Mon, May 16,

Re: Reverse proxy for Spark UI on Kubernetes

It is like Web Application Proxy in YARN ( https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html), to provide easy access for Spark UI when the Spark application is running. When running Spark on Kubernetes with S3, there is no YARN. The reverse proxy here

Re: Reverse proxy for Spark UI on Kubernetes

Thanks Holden :) On Mon, May 16, 2022 at 11:12 PM Holden Karau wrote: > Oh that’s rad  > > On Tue, May 17, 2022 at 7:47 AM bo yang wrote: > >> Hi Spark Folks, >> >> I built a web reverse proxy to access Spark UI on Kubernetes (working >> together with >&

Re: Reverse proxy for Spark UI on Kubernetes

what's the advantage of using reverse proxy for spark UI? Thanks On Tue, May 17, 2022 at 1:47 PM bo yang wrote: > Hi Spark Folks, > > I built a web reverse proxy to access Spark UI on Kubernetes (working > together with https://github.com/GoogleCloudPlatform/spark-on-k8s-opera

Re: Reverse proxy for Spark UI on Kubernetes

Oh that’s rad  On Tue, May 17, 2022 at 7:47 AM bo yang wrote: > Hi Spark Folks, > > I built a web reverse proxy to access Spark UI on Kubernetes (working > together with https://github.com/GoogleCloudPlatform/spark-on-k8s-operator). > Want to share here in case other people ha

Reverse proxy for Spark UI on Kubernetes

Hi Spark Folks, I built a web reverse proxy to access Spark UI on Kubernetes (working together with https://github.com/GoogleCloudPlatform/spark-on-k8s-operator). Want to share here in case other people have similar need. The reverse proxy code is here: https://github.com/datapunchorg/spark-ui

Display AggregateExec modes in Spark UI

Hello, Now Spark UI does not show HashAggregateExec modes, could we add the aggregate modes in SparkPlan? I think it's helpful when we analyze a very complicated SparkPlan. Am I right? For example: SELECT key2, sum(value2) as sum_value2 FROM ( SELECT id % 1 as key2, id as value2

spark UI not showing correct file and line numbers

Hi, I am experiencing performance issues in one of my pyspark applications. When I look at the spark UI, the file and line number of each entry is listed as . I would like to use the information in the Spark UI for debugging, but without knowing the correct file and line number

Re: Spark UI Storage Memory

any suggestion please. Thanks Amit On Fri, Dec 4, 2020 at 2:27 PM Amit Sharma wrote: > Is there any memory leak in spark 2.3.3 version as mentioned in below > Jira. > https://issues.apache.org/jira/browse/SPARK-29055. > > Please let me know how to solve it. > > Thanks > Amit > > On Fri, Dec 4,

RE: Spark UI Storage Memory

unsubsribe

Re: Spark UI Storage Memory

Is there any memory leak in spark 2.3.3 version as mentioned in below Jira. https://issues.apache.org/jira/browse/SPARK-29055. Please let me know how to solve it. Thanks Amit On Fri, Dec 4, 2020 at 1:55 PM Amit Sharma wrote: > Can someone help me on this please. > > > Thanks > Amit > > On

Re: Spark UI Storage Memory

Can someone help me on this please. Thanks Amit On Wed, Dec 2, 2020 at 11:52 AM Amit Sharma wrote: > Hi , I have a spark streaming job. When I am checking the Excetors tab , > there is a Storage Memory column. It displays used memory /total memory. > What is used memory. Is it memory in use

Spark UI Storage Memory

Hi , I have a spark streaming job. When I am checking the Excetors tab , there is a Storage Memory column. It displays used memory /total memory. What is used memory. Is it memory in use or memory used so far. How would I know how much memory is unused at 1 point of time. Thanks Amit

spark UI storage tab

Hi , I have few questions as below 1. In the spark ui storage tab is displayed 'storage level',' size in memory' and size on disk, i am not sure it displays RDD ID 16 with memory usage 76 MB not sure why it is not getting 0 once a request for spark streaming is completed. I am caching some RDD

Re: Spark UI

: https://spark.apache.org/docs/3.0.0/web-ui.html is the official doc for Spark UI. Xiao On Sun, Jul 19, 2020 at 1:38 PM venkatadevarapu mailto:ramesh.biexp...@gmail.com>> wrote: Hi, I'm looking for a tutorial/video/material which explains the content of various tabes in

Re: Spark UI

https://www.youtube.com/watch?v=YgQgJceojJY (Xiao's video ) On Mon, Jul 20, 2020 at 8:03 AM Xiao Li wrote: > https://spark.apache.org/docs/3.0.0/web-ui.html is the official doc > for Spark UI. > > Xiao > > On Sun, Jul 19, 2020 at 1:38 PM venkatadevarapu > wrote: >

Re: Spark UI

https://spark.apache.org/docs/3.0.0/web-ui.html is the official doc for Spark UI. Xiao On Sun, Jul 19, 2020 at 1:38 PM venkatadevarapu wrote: > Hi, > > I'm looking for a tutorial/video/material which explains the content of > various tabes in SPARK WEB UI. > Can som

Spark UI

Hi, I'm looking for a tutorial/video/material which explains the content of various tabes in SPARK WEB UI. Can some one direct me with the relevant info. Thanks -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Using Spark UI with Running Spark on Hadoop Yarn

Is there anyway to make the spark process visible via Spark UI when running Spark 3.0 on a Hadoop yarn cluster?  The spark documentation talked about replacing Spark UI with the spark history server, but didn't give much details.  Therefore I would assume it is still possible to use Spark UI

Re: Can you view thread dumps on spark UI if job finished

I am using history server to see previous UIs. > > However, my question still remains on viewing old thread dumps, as I > cannot see them on the old completed spark UIs, only when spark context is > running. > > On Wed, Apr 8, 2020 at 4:01 PM Zahid Rahman wrote: > >>

Re: Can you view thread dumps on spark UI if job finished

Thanks Zahid, Yes I am using history server to see previous UIs. However, my question still remains on viewing old thread dumps, as I cannot see them on the old completed spark UIs, only when spark context is running. On Wed, Apr 8, 2020 at 4:01 PM Zahid Rahman wrote: > Spark UI is o

Re: Can you view thread dumps on spark UI if job finished

Spark UI is only available while SparkContext is running. However You can get to the Spark UI after your application completes or crashes. To do this Spark includes a tool called the Spark History Server that allows you to reconstruct the Spark UI. You can find up to date information on how

Can you view thread dumps on spark UI if job finished

Hi all, As stated in title, currently when I view the spark UI of a completed spark job, I see there are thread dump links in the executor tab, but clicking on them does nothing. Is it possible to see the thread dumps somehow even if the job finishes? On spark 2.4.5. Thanks. -- Cheers, Ruijing

Re: Spark UI History server on Kubernetes

t; following properties while submitting the spark job > > spark.eventLog.enabled true > > spark.eventLog.dir > > > > Thanks and Regards, > > Abhishek > > > > *From:* Battini Lakshman > *Sent:* Wednesday, January 23, 2019 1:55 PM > *To:* Rao, Abhishek (Nokia - IN/Ban

RE: Spark UI History server on Kubernetes

spark.eventLog.dir Thanks and Regards, Abhishek From: Battini Lakshman Sent: Wednesday, January 23, 2019 1:55 PM To: Rao, Abhishek (Nokia - IN/Bangalore) Subject: Re: Spark UI History server on Kubernetes HI Abhishek, Thank you for your response. Could you please let me know the properties you configured

RE: Spark UI History server on Kubernetes

, 2019 6:02 PM To: user@spark.apache.org Subject: Spark UI History server on Kubernetes Hello, We are running Spark 2.4 on Kubernetes cluster, able to access the Spark UI using "kubectl port-forward". However, this spark UI contains currently running Spark application logs, we

Spark UI History server on Kubernetes

Hello, We are running Spark 2.4 on Kubernetes cluster, able to access the Spark UI using "kubectl port-forward". However, this spark UI contains currently running Spark application logs, we would like to maintain the 'completed' spark application logs as well. Could someone help u

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

Done: https://issues.apache.org/jira/browse/SPARK-25837 On Thu, Oct 25, 2018 at 10:21 AM Marcelo Vanzin wrote: > Ah that makes more sense. Could you file a bug with that information > so we don't lose track of this? > > Thanks > On Wed, Oct 24, 2018 at 6:13 PM Patrick Brown > wrote: > > > >

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

Ah that makes more sense. Could you file a bug with that information so we don't lose track of this? Thanks On Wed, Oct 24, 2018 at 6:13 PM Patrick Brown wrote: > > On my production application I am running ~200 jobs at once, but continue to > submit jobs in this manner for sometimes ~1 hour. >

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

When you say many jobs at once, what ballpark are you talking about? The code in 2.3+ does try to keep data about all running jobs and stages regardless of the limit. If you're running into issues because of that we may have to look again at whether that's the right thing to do. On Tue, Oct 23,

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

I believe I may be able to reproduce this now, it seems like it may be something to do with many jobs at once: Spark 2.3.1 > spark-shell --conf spark.ui.retainedJobs=1 scala> import scala.concurrent._ scala> import scala.concurrent.ExecutionContext.Implicits.global scala> for (i <- 0 until

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

Just tried on 2.3.2 and worked fine for me. UI had a single job and a single stage (+ the tasks related to that single stage), same thing in memory (checked with jvisualvm). On Sat, Oct 20, 2018 at 6:45 PM Marcelo Vanzin wrote: > > On Tue, Oct 16, 2018 at 9:34 AM Patrick Brown > wrote: > > I

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

On Tue, Oct 16, 2018 at 9:34 AM Patrick Brown wrote: > I recently upgraded to spark 2.3.1 I have had these same settings in my spark > submit script, which worked on 2.0.2, and according to the documentation > appear to not have changed: > > spark.ui.retainedTasks=1 > spark.ui.retainedStages=1

Re: [Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

I have the same problem when I upgrade my application from Spark 2.2.1 to Spark 2.3.2 and run in Yarn client mode. Also I noticed that in my Spark driver,  org.apache.spark.status.TaskDataWrapper could take up more than 2G of memory. Shing On Tuesday, 16 October 2018, 17:34:02 GMT+1,

[Spark UI] Spark 2.3.1 UI no longer respects spark.ui.retainedJobs

I recently upgraded to spark 2.3.1 I have had these same settings in my spark submit script, which worked on 2.0.2, and according to the documentation appear to not have changed: spark.ui.retainedTasks=1 spark.ui.retainedStages=1 spark.ui.retainedJobs=1 However in 2.3.1 the UI doesn't seem to

[Spark UI] find driver for an application

Hello, I am having some troubles in using Spark Master UI to figure out some basic information. The process is too tedious. I am using spark 2.2.1 with Spark standalone. - In cluster mode, how to figure out which driver is related to which application? - In supervise mode, how to track the

Understanding Event Timeline of Spark UI

Hi, I've a job running which shows the Event Timeline as follows, I am trying to guess the gaps between these single lines, they seem to be parallel but not immediately sequential with other stages. Any other insight from this, and what is the cluster doing during these gaps? Thanks, Aakash.

Re: Spark UI Source Code

No. >> >> > Also can we retrieve these metrics based on timestamp? >> >> Only if the REST API has that feature, don't remember off the top of my >> head. >> >> >> -- >> Marcelo >> >> ----- >> To unsubscribe e-mail: [hidden emai

Re: Spark UI Source Code

On Mon, May 7, 2018 at 1:44 AM, Anshi Shrivastava wrote: > I've found a KVStore wrapper which stores all the metrics in a LevelDb > store. This KVStore wrapper is available as a spark-dependency but we cannot > access the metrics directly from spark since they are

Re: What do I need to set to see the number of records and processing time for each batch in SPARK UI?

<kanth...@gmail.com> wrote: > Hi All, > > I am using spark 2.3.0 and I wondering what do I need to set to see the > number of records and processing time for each batch in SPARK UI? The > default UI doesn't seem to show this. > > Thanks@ >

What do I need to set to see the number of records and processing time for each batch in SPARK UI?

Hi All, I am using spark 2.3.0 and I wondering what do I need to set to see the number of records and processing time for each batch in SPARK UI? The default UI doesn't seem to show this. Thanks@

Spark UI Streaming batch time interval does not match batch interval

Hello, I am running a streaming app on Spark 2.1.2. The batch interval is set to 5000ms, and when I go to the "Streaming" tab in the Spark UI, it correctly reports a 5 second batch interval, but the list of batches below only shows one batch every two minutes (IE the batch time for

How to create security filter for Spark UI in Spark on YARN

*Environment*: AWS EMR, yarn cluster. *Description*: I am trying to use a java filter to protect the access to spark ui, this by using the property spark.ui.filters; the problem is that when spark is running on yarn mode, that property is being allways overriden by hadoop with the filter

Spark UI stdout/stderr links point to executors internal address

*Environment:* AWS EMR, yarn cluster. *Description:* On Spark ui, in Environment and Executors tabs, the links of stdout and stderr point to the internal address of the executors. This would imply to expose the executors so that links can be accessed. Shouldn't those links be pointed

Spark UI port

Hello all, I am running PySpark Job (v2.0.2) with checkpoint enabled in Mesos cluster and am using Marathon for orchestration. When the job is restarted using Marathon, Spark UI is not getting started at the port specified by Marathon. Instead, it is picking port from the checkpoint

Spark UI to use Marathon assigned port

Hello all, I am running PySpark Job (v2.0.2) with checkpoint enabled in Mesos cluster and am using Marathon for orchestration. When the job is restarted using Marathon, Spark UI is not getting started at the port specified by Marathon. Instead, it is picking port from the checkpoint

Re: Kill Spark Streaming JOB from Spark UI or Yarn

The batches should all have the same application ID, so use that one. You can also find the application in the YARN UI to terminate it from there. Matei > On Aug 27, 2017, at 10:27 AM, KhajaAsmath Mohammed > wrote: > > Hi, > > I am new to spark streaming and not

Re: Spark UI crashes on Large Workloads

Hi Riccardo, Thanks for your suggestions. The thing is that my Spark UI is the one thing that is crashing - and not the app. In fact the app does end up completing successfully. That's why I'm a bit confused by this issue? I'll still try out some of your suggestions. Thanks and Regards, Saatvik

Re: Spark UI crashes on Large Workloads

stopped-dropping-event. >>> All my code is running inside the main function, and the only slightly >>> peculiar thing I am doing in this app is using a custom PySpark ML >>> Transformer(Modified from >>> https://stacko

Re: Spark UI crashes on Large Workloads

htly >> peculiar thing I am doing in this app is using a custom PySpark ML >> Transformer(Modified from >> https://stackoverflow.com/questions/32331848/create-a-custom >> -transformer-in-pyspark-ml). >> Could this be the issue? How can I debug why this is happeni

Re: Spark UI crashes on Large Workloads

custom PySpark ML > Transformer(Modified from > https://stackoverflow.com/questions/32331848/create-a- > custom-transformer-in-pyspark-ml). > Could this be the issue? How can I debug why this is happening? > > > > -- > View this message in context: http://apache-spark-use

Spark UI crashes on Large Workloads

: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-UI-crashes-on-Large-Workloads-tp28873.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr

proxy on spark UI

Hi all, I am using Hadoop 2.6.5 and spark 2.1.0 and run a job using spark-submit and master is set to "yarn". When spark starts, I can load Spark UI page using port 4040 but no job is shown in the page. After the following logs (registering application master on yarn) spark UI is not

Re: Spark UI shows Jobs are processing, but the files are already written to S3

though there were a lot of Active jobs and tasks according to > the UI. And my output is correct. > > > > Is there a JIRA item tracking this? > > > > *From:* Kuchekar [mailto:kuchekar.nil...@gmail.com] > *Sent:* Wednesday, November 16, 2016 10:00 AM > *To:* spark

RE: Spark UI not coming up in EMR

Any clue on this. Jobs are running fine , But not able to access Spark UI in EMR -yarn. Where I can see statistics like , No of events /per sec and rows processed for streaming in log files (If UI is not working) -Saurabh From: Saurabh Malviya (samalviy) Sent: Monday, January 09, 2017 10:59

Spark UI not coming up in EMR

Spark web UI for detailed monitoring for streaming jobs stop rendering after 2 weeks. Its keep looping to fetch the page. Is there any clue I can get that page. Or logs where I can see how many events coming in spark for each internval -Saurabh

Spark UI - Puzzling “Input Size / Records” in Stage Details

Was trying something basic to understand tasks stages and shuffles a bit better in Spark. The dataset is 256 MB Tried this in zeppelin val tmpDF = spark.read .option("header", "true") .option("delimiter", ",") .option("inferSchema", "true")

RE: Spark UI shows Jobs are processing, but the files are already written to S3

, 2016 10:00 AM To: spark users <user@spark.apache.org> Subject: Spark UI shows Jobs are processing, but the files are already written to S3 Hi, I am running a spark job, which saves the computed data (massive data) to S3. On the Spark Ui I see the some jobs are active, but no ac

Spark UI shows Jobs are processing, but the files are already written to S3

Hi, I am running a spark job, which saves the computed data (massive data) to S3. On the Spark Ui I see the some jobs are active, but no activity in the logs. Also on S3 all the data has be written (verified each bucket --> it has _SUCCESS file) Am I missing something? Thanks. Kuche

How to interpret the Time Line on "Details for Stage" Spark UI page

Hi, I am using Spark 1.6.1, and I am looking at the Event Timeline on "Details for Stage" Spark UI web page in detail. I found that the "scheduler delay" on event timeline is somehow misrepresented. I want to confirm if my understanding is correct. Here is the detailed des

Re: Spark UI error spark 2.0.1 hadoop 2.6

I'm able to fix.. added servlet 3.0 to classpath -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-UI-error-spark-2-0-1-hadoop-2-6-tp27970p27971.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Spark UI error spark 2.0.1 hadoop 2.6

e know if I'm missing something. Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-UI-error-spark-2-0-1-hadoop-2-6-tp27970.html Sent from the Apache Spark User List mailing list archive at Nabb

Re: Is there anyway Spark UI is set to poll and refreshes itself

> > Thanks Nguyen for the link. > > I installed Super Refresh as ADD on to Chrome. By default the refresh is > stop until you set it to x seconds. However, the issue we have is that > Spark UI comes with 6+ tabs and you have to repeat the process for each tab. > > However, that m

Re: Is there anyway Spark UI is set to poll and refreshes itself

Thanks Nguyen for the link. I installed Super Refresh as ADD on to Chrome. By default the refresh is stop until you set it to x seconds. However, the issue we have is that Spark UI comes with 6+ tabs and you have to repeat the process for each tab. However, that messes up the things. For example

Re: Is there anyway Spark UI is set to poll and refreshes itself

a.kuma...@icloud.com> wrote: > >> Hi Mich, >> >> Unlikely that we can use Zeppelin for dynamic, real time update >> visualisation. It makes nice, static visuals. >> >> I was thinking more on the lines of http://dashingdemo.herokuap >> p.com/sample >

Re: Is there anyway Spark UI is set to poll and refreshes itself

tps://github.com/ > Shopify/dashing/wiki/Additional-Widgets > > The Spark UI is functional, but I am looking forward to some aesthetics > and high level picture of the process. Using Websockets, the dashboard can > be updated real time without the need of refreshing the page. >

Re: Is there anyway Spark UI is set to poll and refreshes itself

f some sort in lieu > (or at least complement it) of Spark UI . > > Regards, > > Sivakumaran S > > On 27 Aug 2016 9:34 a.m., Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Are we actually looking for a eal time dashboard of some sort for Spark

Re: Is there anyway Spark UI is set to poll and refreshes itself

wrote: > I would love to participate in developing a dashboard of some sort in lieu > (or at least complement it) of Spark UI . > > Regards, > > Sivakumaran S > > On 27 Aug 2016 9:34 a.m., Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Are we act

Re: Is there anyway Spark UI is set to poll and refreshes itself

Are we actually looking for a eal time dashboard of some sort for Spark UI interface? After all one can think a real time dashboard can do this! HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.

Re: Is there anyway Spark UI is set to poll and refreshes itself

.com/apache/spark/blob/master/core/src/ > main/scala/org/apache/spark/ui/UIUtils.scala#L202 > [3] http://spark.apache.org/docs/latest/monitoring.html#rest-api > [4] http://spark.apache.org/docs/latest/monitoring.html#metrics > > Pozdrawiam, > Jacek Laskowski > > https://

Re: Is there anyway Spark UI is set to poll and refreshes itself

/scala/org/apache/spark/ui/UIUtils.scala#L202 [3] http://spark.apache.org/docs/latest/monitoring.html#rest-api [4] http://spark.apache.org/docs/latest/monitoring.html#metrics Pozdrawiam, Jacek Laskowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache

Re: Is there anyway Spark UI is set to poll and refreshes itself

Hi you can take a look at: https://github.com/hammerlab/spree it's a bit outdated but maybe it's still possible to use with some more recent Spark version. M. 2016-08-25 11:55 GMT+02:00 Mich Talebzadeh : > Hi, > > This may be already there. > > A spark job opens up a

Is there anyway Spark UI is set to poll and refreshes itself

Hi, This may be already there. A spark job opens up a UI on port specified by --conf "spark.ui.port=${SP}" that defaults to 4040. However, on UI one needs to refresh the page to see the progress. Can this be polled so it is refreshed automatically Thanks Dr Mich Talebzadeh LinkedIn *

Re: How to give name to Spark jobs shown in Spark UI

>> >> -- >> If you reply to this email, your message will be added to the discussion >> below: >> >> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-give-name-to-Spark-jobs-shown-in-Spark-UI-tp27400p27406.html >> To s

Re: How to give name to Spark jobs shown in Spark UI

ng tool like Ganglia to determine were its slow in > your spark jobs. > > -- > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-give-name-to-Spark-jobs-sh

Spark Jobs not getting shown in Spark UI browser

Hi All, I have recently started using Spark 1.6.2 for running my spark jobs. But now my jobs are not getting shown in the spark browser UI, even though the job is running fine which i can see in shell output. Any suggestions. Thanks, Prashant Verma

Re: How to give name to Spark jobs shown in Spark UI

unning ones. For e. g. > xyz_saveAsTextFile(), abc_saveAsTextFile() etc please guide. Thanks in > advance. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-give-name-to-Spark-jobs-shown-in-Spark-UI-tp27400.html >

How to give name to Spark jobs shown in Spark UI

.1001560.n3.nabble.com/How-to-give-name-to-Spark-jobs-shown-in-Spark-UI-tp27400.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Understanding Spark UI DAGs

Ok, so those line numbers in our DAG don't refer to our code. Is there any way to display (or calculate) line numbers that refer to code we actually wrote, or is that only possible in Scala Spark? On Thu, Jul 21, 2016 at 12:24 PM, Jacek Laskowski wrote: > Hi, > > My little

Re: Understanding Spark UI DAGs

That -1 is coming from here: PythonRDD.writeIteratorToStream(inputIterator, dataOut) dataOut.writeInt(SpecialLengths.END_OF_DATA_SECTION) —> val END_OF_DATA_SECTION = -1 dataOut.writeInt(SpecialLengths.END_OF_STREAM) dataOut.flush() > On Jul 21, 2016, at 12:24 PM, Jacek Laskowski

Re: Understanding Spark UI DAGs

Hi, My little understanding of Python-Spark bridge is that at some point the python code communicates over the wire with Spark's backbone that includes PythonRDD [1]. When the CallSite can't be computed, it's null:-1 to denote "nothing could be referred to". [1]

Re: Understanding Spark UI DAGs

> > It's called a CallSite that shows where the line comes from. You can see > the code yourself given the python file and the line number. > But that's what I don't understand. Which python file? We spark submit one file called ctr_parsing.py, but it only has 150 lines. So what is MapPartitions

Re: Understanding Spark UI DAGs

On Thu, Jul 21, 2016 at 2:56 AM, C. Josephson wrote: > I just started looking at the DAG for a Spark Streaming job, and had a > couple of questions about it (image inline). > > 1.) What do the numbers in brackets mean, e.g. PythonRDD[805]? > Every RDD has its identifier (as id

spark UI what does storage memory x/y mean

My stream app is running into problems It seems to slow down over time. How can I interpret the storage memory column. I wonder if I have a GC problem? Any idea how I can get GC stats? Thanks Andy Executors (3) * Memory: 9.4 GB Used (1533.4 MB Total) * Disk: 0.0 B Used Executor IDAddressRDD

Re: Spark UI shows finished when job had an error

rk-submit log. > However ,the UI never showed any failed stage or job. It appeared as if > the job finished without error, which is not correct. > > We are trying to define our monitoring for our scheduled jobs, and we > intended to use the Spark UI to catch issues. Can we explain why

Re: Spark UI shows finished when job had an error

hed without error, which is not correct. > > We are trying to define our monitoring for our scheduled jobs, and we > intended to use the Spark UI to catch issues. Can we explain why the UI > would not report an exception like this? Is there a better approach we > should use for t

Re: Spark UI shows finished when job had an error

onitoring for our scheduled jobs, and we > intended to use the Spark UI to catch issues. Can we explain why the UI > would not report an exception like this? Is there a better approach we > should use for tracking failures in a Spark job? > > We are currently on 1.2 standalone, however we do intend to upgrade to 1.6 > shortly. > > Thanks! > Sumona >

Spark UI shows finished when job had an error

is not correct. We are trying to define our monitoring for our scheduled jobs, and we intended to use the Spark UI to catch issues. Can we explain why the UI would not report an exception like this? Is there a better approach we should use for tracking failures in a Spark job? We are currently

Re: Spark UI doesn't give visibility on which stage job actually failed (due to lazy eval nature)

it . That way we are focused on our application logic rather what framework is doing underneath. About soln, doesn't spark driver (spark context + event listner) have knowledge of every job, taskset, task and their current state? Spark UI can relate job to stage to task then why not stage

Re: Spark UI doesn't give visibility on which stage job actually failed (due to lazy eval nature)

e executing >>> anything. e.g. if there are map steps after shuffle then it doesn't stop at >>> shuffle to execute anything but goes to that next map steps until it finds >>> a reason(spark action) to execute. As a result stage that spark is running >>> can be inte

Re: Spark UI doesn't give visibility on which stage job actually failed (due to lazy eval nature)

> encountered. Sometime it goes further after shuffle dep before executing >> anything. e.g. if there are map steps after shuffle then it doesn't stop at >> shuffle to execute anything but goes to that next map steps until it finds >> a reason(spark action) to execute. As a result stag

Re: Spark UI doesn't give visibility on which stage job actually failed (due to lazy eval nature)

result stage that spark is running > can be internally series of (map -> shuffle -> map -> map -> collect) and > spark UI just shows its currently running 'collect' stage. SO if job fails > at that point spark UI just says Collect failed but in fact it could be any > stage in th

  1   2   3   >