[jira] [Commented] (SPARK-23236) Make it easier to find the rest API, especially in local mode
[ https://issues.apache.org/jira/browse/SPARK-23236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345827#comment-16345827 ] Alex Bozarth commented on SPARK-23236: -- For #1, a REST API endpoint shouldn't return html, but we could return a custom html response (such as 405 or another response code) that includes a short "maybe you meant..." description. For #2 I would be ok with a "maybe you meant..." response but with a link to the Spark REST API Doc to aid the user. > Make it easier to find the rest API, especially in local mode > - > > Key: SPARK-23236 > URL: https://issues.apache.org/jira/browse/SPARK-23236 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 2.3.0 >Reporter: Imran Rashid >Priority: Trivial > Labels: newbie > > This is really minor, but it always takes me a little bit to figure out how > to get from the UI to the rest api. Its especially a pain in local-mode, > where you need the app-id, though in general I don't know the app-id, so have > to either look in logs or go to another endpoint first in the ui just to find > the app-id. While it wouldn't really help anybody accessing the endpoints > programmatically, we could make it easier for someone doing exploration via > their browser. > Some things which could be improved: > * /api/v1 just provides a link to "/api/v1/applications" > * /api provides a link to "/api/v1/applications" > * /api/v1/applications/[app-id] gives a list of links for the other endpoints > * on the UI, there is a link to at least /api/v1/applications/[app-id] -- > better still if each UI page links to the corresponding endpoint, eg. the all > jobs page would link to /api/v1/applications/[app-id]/jobs -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23237) Add UI / endpoint for threaddumps for executors with active tasks
[
https://issues.apache.org/jira/browse/SPARK-23237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345818#comment-16345818
]
Alex Bozarth commented on SPARK-23237:
--
I would rather keep it to an api endpoint, but what I'm worried about is having
an end point that returns a specific threadDump decided by some unknown
algorithm. Again, I'm willing to look at a PR to see if the exact impl will
change my mind.
> Add UI / endpoint for threaddumps for executors with active tasks
> -
>
> Key: SPARK-23237
> URL: https://issues.apache.org/jira/browse/SPARK-23237
> Project: Spark
> Issue Type: New Feature
> Components: Web UI
>Affects Versions: 2.3.0
>Reporter: Imran Rashid
>Priority: Major
>
> Frequently, when there are a handful of straggler tasks, users want to know
> what is going on in those executors running the stragglers. Currently, that
> is a bit of a pain to do: you have to go to the page for your active stage,
> find the task, figure out which executor its on, then go to the executors
> page, and get the thread dump. Or maybe you just go to the executors page,
> find the executor with an active task, and then click on that, but that
> doesn't work if you've got multiple stages running.
> Users could figure this by extracting the info from the stage rest endpoint,
> but it's such a common thing to do that we should make it easy.
> I realize that figuring out a good way to do this is a little tricky. We
> don't want to make it easy to end up pulling thread dumps from 1000 executors
> back to the driver. So we've got to come up with a reasonable heuristic for
> choosing which executors to poll. And we've also got to find a suitable
> place to put this.
> My suggestion is that the stage page always has a link to the thread dumps
> for the *one* executor with the longest running task. And there would be a
> corresponding endpoint in the rest api with the same info, maybe at
> {{/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/slowestTaskThreadDump}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344339#comment-16344339 ] Alex Bozarth commented on SPARK-18085: -- [~vanzin] since this is complete and going into 2.3 I was hoping you could help write a short overview of the SPIP, like what's changed and why. Given how much this evolved during the project I'm not sure if the original pitch is the best description anymore and I would like to have a nice summary to describe the project's impact. Thanks > SPIP: Better History Server scalability for many / large applications > - > > Key: SPARK-18085 > URL: https://issues.apache.org/jira/browse/SPARK-18085 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI >Affects Versions: 2.0.0 >Reporter: Marcelo Vanzin >Priority: Major > Labels: SPIP > Fix For: 2.3.0 > > Attachments: screenshot-1.png, screenshot-2.png, spark_hs_next_gen.pdf > > > It's a known fact that the History Server currently has some annoying issues > when serving lots of applications, and when serving large applications. > I'm filing this umbrella to track work related to addressing those issues. > I'll be attaching a document shortly describing the issues and suggesting a > path to how to solve them. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23235) Add executor Threaddump to api
[
https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344168#comment-16344168
]
Alex Bozarth commented on SPARK-23235:
--
Your discussion clarified my concern for me. I think I wanted to see how he was
going to do it first, but based on your implementation comments, this looks
like a good add.
> Add executor Threaddump to api
> --
>
> Key: SPARK-23235
> URL: https://issues.apache.org/jira/browse/SPARK-23235
> Project: Spark
> Issue Type: New Feature
> Components: Web UI
>Affects Versions: 2.3.0
>Reporter: Imran Rashid
>Priority: Minor
> Labels: newbie
>
> It looks like the the thread dump {{/executors/threadDump/?executorId=[id]}}
> is only available in the UI, not in the rest api at all. This is especially
> a pain because that page in the UI has extra formatting which makes it a pain
> to send the output to somebody else (most likely you click "expand all" and
> then copy paste that, which is OK, but is formatted weirdly). We might also
> just want a "format=raw" option even on the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23237) Add UI / endpoint for threaddumps for executors with active tasks
[
https://issues.apache.org/jira/browse/SPARK-23237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341730#comment-16341730
]
Alex Bozarth commented on SPARK-23237:
--
Unlike the related task, I'm not sure about this one. I see the need as you
stated it, but also as you stated, it would be difficult to go about it. I'm
willing to look at a PR for this but I wouldn't hold out hope for convincing me.
> Add UI / endpoint for threaddumps for executors with active tasks
> -
>
> Key: SPARK-23237
> URL: https://issues.apache.org/jira/browse/SPARK-23237
> Project: Spark
> Issue Type: New Feature
> Components: Web UI
>Affects Versions: 2.3.0
>Reporter: Imran Rashid
>Priority: Major
>
> Frequently, when there are a handful of straggler tasks, users want to know
> what is going on in those executors running the stragglers. Currently, that
> is a bit of a pain to do: you have to go to the page for your active stage,
> find the task, figure out which executor its on, then go to the executors
> page, and get the thread dump. Or maybe you just go to the executors page,
> find the executor with an active task, and then click on that, but that
> doesn't work if you've got multiple stages running.
> Users could figure this by extracting the info from the stage rest endpoint,
> but it's such a common thing to do that we should make it easy.
> I realize that figuring out a good way to do this is a little tricky. We
> don't want to make it easy to end up pulling thread dumps from 1000 executors
> back to the driver. So we've got to come up with a reasonable heuristic for
> choosing which executors to poll. And we've also got to find a suitable
> place to put this.
> My suggestion is that the stage page always has a link to the thread dumps
> for the *one* executor with the longest running task. And there would be a
> corresponding endpoint in the rest api with the same info, maybe at
> {{/applications/[app-id]/stages/[stage-id]/[stage-attempt-id]/slowestTaskThreadDump}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23235) Add executor Threaddump to api
[
https://issues.apache.org/jira/browse/SPARK-23235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341727#comment-16341727
]
Alex Bozarth commented on SPARK-23235:
--
I think I would be ok with adding threadDump to the api, but I'd have to see
the PR before getting 100% on board.
> Add executor Threaddump to api
> --
>
> Key: SPARK-23235
> URL: https://issues.apache.org/jira/browse/SPARK-23235
> Project: Spark
> Issue Type: New Feature
> Components: Web UI
>Affects Versions: 2.3.0
>Reporter: Imran Rashid
>Priority: Minor
> Labels: newbie
>
> It looks like the the thread dump {{/executors/threadDump/?executorId=[id]}}
> is only available in the UI, not in the rest api at all. This is especially
> a pain because that page in the UI has extra formatting which makes it a pain
> to send the output to somebody else (most likely you click "expand all" and
> then copy paste that, which is OK, but is formatted weirdly). We might also
> just want a "format=raw" option even on the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23236) Make it easier to find the rest API, especially in local mode
[
https://issues.apache.org/jira/browse/SPARK-23236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341724#comment-16341724
]
Alex Bozarth commented on SPARK-23236:
--
So IIUC, you want
1. /api and /api/v1 to give the same results (a redirect) as
/api/v1/applications
2. You want /api/v1/applications/\{app-id} to include a list of rest api end
points
3. You want UI pages to include links to the rest api calls that they get their
info from
Based on that I'm on the fence about #1, it may be ok, but I'm not sure if it's
best. As for #2 I'm not sure how you're envisioning it, but it seems like a bad
idea in mu head. And for #3, I would be ok with this, but again I'm not sure
how its useful.
> Make it easier to find the rest API, especially in local mode
> -
>
> Key: SPARK-23236
> URL: https://issues.apache.org/jira/browse/SPARK-23236
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 2.3.0
>Reporter: Imran Rashid
>Priority: Trivial
> Labels: newbie
>
> This is really minor, but it always takes me a little bit to figure out how
> to get from the UI to the rest api. Its especially a pain in local-mode,
> where you need the app-id, though in general I don't know the app-id, so have
> to either look in logs or go to another endpoint first in the ui just to find
> the app-id. While it wouldn't really help anybody accessing the endpoints
> programmatically, we could make it easier for someone doing exploration via
> their browser.
> Some things which could be improved:
> * /api/v1 just provides a link to "/api/v1/applications"
> * /api provides a link to "/api/v1/applications"
> * /api/v1/applications/[app-id] gives a list of links for the other endpoints
> * on the UI, there is a link to at least /api/v1/applications/[app-id] --
> better still if each UI page links to the corresponding endpoint, eg. the all
> jobs page would link to /api/v1/applications/[app-id]/jobs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23189) reflect stage level blacklisting on executor tab
[ https://issues.apache.org/jira/browse/SPARK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336326#comment-16336326 ] Alex Bozarth commented on SPARK-23189: -- The tooltip feature does exist, but I agree with [~irashid] that I'm not sure about the use case here. The executors page is dynamically created using the rest api, which would entail adding the blacklisted stages list to the api. I still think this is a cool idea, I'm just not sure if its necessary. > reflect stage level blacklisting on executor tab > - > > Key: SPARK-23189 > URL: https://issues.apache.org/jira/browse/SPARK-23189 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.1 >Reporter: Attila Zsolt Piros >Priority: Major > > This issue is the came during working on SPARK-22577 where the conclusion was > not only stage tab should reflect stage and application level backlisting but > also the executor tab should be extended with stage level backlisting > information. > As [~irashid] and [~tgraves] are discussed the backlisted stages should be > listed for an executor like "*stage[ , ,...]*". One idea was to list only the > most recent 3 of the blacklisted stages another was list all the active > stages which are blacklisted. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23002) SparkUI inconsistent driver hostname compare with other executors
[ https://issues.apache.org/jira/browse/SPARK-23002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319562#comment-16319562 ] Alex Bozarth commented on SPARK-23002: -- I've noticed this in the past, if no one claims it I can take a look in my free moments > SparkUI inconsistent driver hostname compare with other executors > - > > Key: SPARK-23002 > URL: https://issues.apache.org/jira/browse/SPARK-23002 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.2.0 >Reporter: Ran Tao >Priority: Minor > > As the picture shows, driver name is ip address and other executors are > machine hostname. > !https://raw.githubusercontent.com/Lemonjing/issues-assets/master/pics/driver.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22173) Table CSS style needs to be adjusted in History Page and in Executors Page.
[ https://issues.apache.org/jira/browse/SPARK-22173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186789#comment-16186789 ] Alex Bozarth commented on SPARK-22173: -- Just as a note, https://github.com/apache/spark/pull/19270 is currently open moving the tasks page to data tables as well, so this might be better to hold off on until after that gets merged. > Table CSS style needs to be adjusted in History Page and in Executors Page. > --- > > Key: SPARK-22173 > URL: https://issues.apache.org/jira/browse/SPARK-22173 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.3.0 >Reporter: guoxiaolongzte >Priority: Trivial > > There is a problem with table CSS style. > 1. At present, table CSS style is too crowded, and the table width cannot > adapt itself. > 2. Table CSS style is different from job page, stage page, task page, master > page, worker page, etc. The Spark web UI needs to be consistent. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21838) "Completed Applications" links not always working in cluster with spark.ui.reverseProxy=true
[ https://issues.apache.org/jira/browse/SPARK-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144115#comment-16144115 ] Alex Bozarth commented on SPARK-21838: -- Given the behavior on master is as you described I would recommend either adding a 404 or redirect functionality to the master branch rather than trying to port the feature that fixed the bug back to 2.1.0 or 2.2.0 > "Completed Applications" links not always working in cluster with > spark.ui.reverseProxy=true > > > Key: SPARK-21838 > URL: https://issues.apache.org/jira/browse/SPARK-21838 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0, 2.2.0 > Environment: Spark Cluster with reverse proxy enabled: > spark.ui.reverseProxyUrl=http://127.0.1.1:8080/ > spark.ui.reverseProxy=true >Reporter: Ingo Schuster > > 1. Using a Spark Cluster with reverse Proxy enabled, web UI is at: > http://127.0.0.1:8080/ > 2, Starting an application and enter the application specific web UI (by > clicking on the application name): > http://127.0.0.1:8080/proxy/app-20170825151733-0001/jobs/ > 3. If you click on any link (e.g. on the "Executors") +after the application > has terminated+, the Spark master UI will be served again, howeverthis is > done under the URL of the executors page: > http://127.0.0.1:8080/proxy/app-20170825151733-0001/executors/ > 4. When you now click on the link to the just completed application, nothing > happens (you stay on the master web UI home page): > http://127.0.0.1:8080/proxy/app-20170825151733-0001/executors/app?appId=app-20170825151733-0001 > Problem is that in step 3, we cannot serve the applications executors page > since it already terminated. Falling back to the master web UI home page is > ok, but it should be done with an http redirect to that the relative URLs on > the master home page are build with the correct base URL. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21838) "Completed Applications" links not always working in cluster with spark.ui.reverseProxy=true
[ https://issues.apache.org/jira/browse/SPARK-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141925#comment-16141925 ] Alex Bozarth commented on SPARK-21838: -- This is definitely an issue, could you double check that this also occurs on the latest master? https://github.com/apache/spark/pull/18499 might have indirectly fixed it. > "Completed Applications" links not always working in cluster with > spark.ui.reverseProxy=true > > > Key: SPARK-21838 > URL: https://issues.apache.org/jira/browse/SPARK-21838 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0, 2.2.0 > Environment: Spark Cluster with reverse proxy enabled: > spark.ui.reverseProxyUrl=http://127.0.1.1:8080/ > spark.ui.reverseProxy=true >Reporter: Ingo Schuster > > 1. Using a Spark Cluster with reverse Proxy enabled, web UI is at: > http://127.0.0.1:8080/ > 2, Starting an application and enter the application specific web UI (by > clicking on the application name): > http://127.0.0.1:8080/proxy/app-20170825151733-0001/jobs/ > 3. If you click on any link (e.g. on the "Executors") +after the application > has terminated+, the Spark master UI will be served again, howeverthis is > done under the URL of the executors page: > http://127.0.0.1:8080/proxy/app-20170825151733-0001/executors/ > 4. When you now click on the link to the just completed application, nothing > happens (you stay on the master web UI home page): > http://127.0.0.1:8080/proxy/app-20170825151733-0001/executors/app?appId=app-20170825151733-0001 > Problem is that in step 3, we cannot serve the applications executors page > since it already terminated. Falling back to the master web UI home page is > ok, but it should be done with an http redirect to that the relative URLs on > the master home page are build with the correct base URL. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21700) How can I get the MetricsSystem information
[ https://issues.apache.org/jira/browse/SPARK-21700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122762#comment-16122762 ] Alex Bozarth commented on SPARK-21700: -- I would recommend taking a look at the Metrics REST API (https://spark.apache.org/docs/latest/monitoring.html#rest-api). Also in the future the email list would probably be a better place for questions like these. > How can I get the MetricsSystem information > --- > > Key: SPARK-21700 > URL: https://issues.apache.org/jira/browse/SPARK-21700 > Project: Spark > Issue Type: Question > Components: Web UI >Affects Versions: 1.6.0 >Reporter: LiuXiangyu > > I want to get the information that shows on the spark Web UI, but I don't > want to write a spider to get those information from the website, and I know > those information are come from MetricsSystem, is there any way that I can > use the MetricsSystem in my program to get those metrics information? -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21124) Wrong user shown in UI when using kerberos
[
https://issues.apache.org/jira/browse/SPARK-21124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052394#comment-16052394
]
Alex Bozarth commented on SPARK-21124:
--
Though a separate issue, I believe the open PR for SPARK-14483 actually fixes
this as well
> Wrong user shown in UI when using kerberos
> --
>
> Key: SPARK-21124
> URL: https://issues.apache.org/jira/browse/SPARK-21124
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.2.0
>Reporter: Marcelo Vanzin
>Priority: Minor
>
> When submitting an app to a kerberos-secured cluster, the OS user and the
> user running the application may differ. Although it may also happen in
> cluster mode depending on the cluster manager's configuration, it's more
> common in client mode.
> The UI should show enough information about user running the application to
> correctly identify the actual user. The "app user" can be easily retrieved
> via {{Utils.getCurrentUserName()}}, so it's mostly a matter of how to record
> this information (for showing in replayed applications) and how to present it
> in the UI.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-21018) "Completed Jobs" and "Completed Stages" support pagination
[ https://issues.apache.org/jira/browse/SPARK-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Bozarth resolved SPARK-21018. -- Resolution: Duplicate This was added in Spark 2.1 by SPARK-15590 > "Completed Jobs" and "Completed Stages" support pagination > -- > > Key: SPARK-21018 > URL: https://issues.apache.org/jira/browse/SPARK-21018 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 2.0.2 >Reporter: Jinhua Fu >Priority: Minor > Attachments: CompletedJobs.png, PagedTasks.png > > > When using Thriftsever, the number of jobs and Stages may be very large, and > if not paginated, the page will be very long and slow to load, especially > when spark.ui.retainedJobs is set to a large value. So I suggest "completed > Jobs" and "completed Stages" support pagination. > I'd like to change them to a paging display similar to the tasks in the > "Details for Stage" page. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-20853) spark.ui.reverseProxy=true leads to hanging communication to master
[
https://issues.apache.org/jira/browse/SPARK-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Bozarth closed SPARK-20853.
Resolution: Not A Problem
> spark.ui.reverseProxy=true leads to hanging communication to master
> ---
>
> Key: SPARK-20853
> URL: https://issues.apache.org/jira/browse/SPARK-20853
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.1.0
> Environment: ppc64le GNU/Linux, POWER8, only master node is reachable
> externally other nodes are in an internal network
>Reporter: Benno Staebler
> Labels: network, web-ui
>
> When *reverse proxy is enabled*
> {quote}
> spark.ui.reverseProxy=true
> spark.ui.reverseProxyUrl=/
> {quote}
> first of all any invocation of the spark master Web UI hangs forever locally
> (e.g. http://192.168.10.16:25001) and via external URL without any data
> received.
> One, sometimes two spark applications succeed without error and than workers
> start throwing exceptions:
> {quote}
> Caused by: java.io.IOException: Failed to connect to /192.168.10.16:25050
> {quote}
> The application dies during creation of SparkContext:
> {quote}
> 2017-05-22 16:11:23 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting
> to master spark://node0101:25000...
> 2017-05-22 16:11:23 INFO TransportClientFactory:254 - Successfully created
> connection to node0101/192.168.10.16:25000 after 169 ms (132 ms spent in
> bootstraps)
> 2017-05-22 16:11:43 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting
> to master spark://node0101:25000...
> 2017-05-22 16:12:03 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting
> to master spark://node0101:25000...
> 2017-05-22 16:12:23 ERROR StandaloneSchedulerBackend:70 - Application has
> been killed. Reason: All masters are unresponsive! Giving up.
> 2017-05-22 16:12:23 WARN StandaloneSchedulerBackend:66 - Application ID is
> not initialized yet.
> 2017-05-22 16:12:23 INFO Utils:54 - Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 25056.
> .
> Caused by: java.lang.IllegalArgumentException: requirement failed: Can only
> call getServletHandlers on a running MetricsSystem
> {quote}
> *This definitively does not happen without reverse proxy enabled!*
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20853) spark.ui.reverseProxy=true leads to hanging communication to master
[
https://issues.apache.org/jira/browse/SPARK-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021636#comment-16021636
]
Alex Bozarth commented on SPARK-20853:
--
Closing this as it is a question for the user email list, for reference though,
spark.ui.reverseProxyUrl should be set to a full url including http(s):// not a
a relative one like /
> spark.ui.reverseProxy=true leads to hanging communication to master
> ---
>
> Key: SPARK-20853
> URL: https://issues.apache.org/jira/browse/SPARK-20853
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.1.0
> Environment: ppc64le GNU/Linux, POWER8, only master node is reachable
> externally other nodes are in an internal network
>Reporter: Benno Staebler
> Labels: network, web-ui
>
> When *reverse proxy is enabled*
> {quote}
> spark.ui.reverseProxy=true
> spark.ui.reverseProxyUrl=/
> {quote}
> first of all any invocation of the spark master Web UI hangs forever locally
> (e.g. http://192.168.10.16:25001) and via external URL without any data
> received.
> One, sometimes two spark applications succeed without error and than workers
> start throwing exceptions:
> {quote}
> Caused by: java.io.IOException: Failed to connect to /192.168.10.16:25050
> {quote}
> The application dies during creation of SparkContext:
> {quote}
> 2017-05-22 16:11:23 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting
> to master spark://node0101:25000...
> 2017-05-22 16:11:23 INFO TransportClientFactory:254 - Successfully created
> connection to node0101/192.168.10.16:25000 after 169 ms (132 ms spent in
> bootstraps)
> 2017-05-22 16:11:43 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting
> to master spark://node0101:25000...
> 2017-05-22 16:12:03 INFO StandaloneAppClient$ClientEndpoint:54 - Connecting
> to master spark://node0101:25000...
> 2017-05-22 16:12:23 ERROR StandaloneSchedulerBackend:70 - Application has
> been killed. Reason: All masters are unresponsive! Giving up.
> 2017-05-22 16:12:23 WARN StandaloneSchedulerBackend:66 - Application ID is
> not initialized yet.
> 2017-05-22 16:12:23 INFO Utils:54 - Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 25056.
> .
> Caused by: java.lang.IllegalArgumentException: requirement failed: Can only
> call getServletHandlers on a running MetricsSystem
> {quote}
> *This definitively does not happen without reverse proxy enabled!*
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4836) Web UI should display separate information for all stage attempts
[ https://issues.apache.org/jira/browse/SPARK-4836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16008737#comment-16008737 ] Alex Bozarth commented on SPARK-4836: - I looked into this a bit on behalf of [~ckadner] and I believe this is an issue with how JobProgressListener stores completed stages. I'm not sure if I have time to continue to look into this so if someone wants to look into this ping me first to make sure I'm still working on it. > Web UI should display separate information for all stage attempts > - > > Key: SPARK-4836 > URL: https://issues.apache.org/jira/browse/SPARK-4836 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 1.1.1, 1.2.0 >Reporter: Josh Rosen > > I've run into some cases where the web UI job page will say that a job took > 12 minutes but the sum of that job's stage times is something like 10 > seconds. In this case, it turns out that my job ran a stage to completion > (which took, say, 5 minutes) then lost some partitions of that stage and had > to run a new stage attempt to recompute one or two tasks from that stage. As > a result, the latest attempt for that stage reports only one or two tasks. > In the web UI, it seems that we only show the latest stage attempt, not all > attempts, which can lead to confusing / misleading displays for jobs with > failed / partially-recomputed stages. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20630) Thread Dump link available in Executors tab irrespective of spark.ui.threadDumpsEnabled
[
https://issues.apache.org/jira/browse/SPARK-20630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001232#comment-16001232
]
Alex Bozarth commented on SPARK-20630:
--
If you want I can take a look at this, I'm not sure why it's no longer working.
I wrote the js code that toggles the thread dump link a while back after the
switch the jQuery DataTables
> Thread Dump link available in Executors tab irrespective of
> spark.ui.threadDumpsEnabled
> ---
>
> Key: SPARK-20630
> URL: https://issues.apache.org/jira/browse/SPARK-20630
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.3.0
>Reporter: Jacek Laskowski
>Priority: Minor
> Attachments: spark-webui-executors-threadDump.png
>
>
> Irrespective of {{spark.ui.threadDumpsEnabled}} property web UI's Executors
> page displays *Thread Dump* column with an active link (that does nothing
> though).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16333) Excessive Spark history event/json data size (5GB each)
[ https://issues.apache.org/jira/browse/SPARK-16333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987234#comment-15987234 ] Alex Bozarth commented on SPARK-16333: -- Was looking at this and wanted to know if this was fixed by SPARK-20084 or only partially fixed? > Excessive Spark history event/json data size (5GB each) > --- > > Key: SPARK-16333 > URL: https://issues.apache.org/jira/browse/SPARK-16333 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.0.0 > Environment: this is seen on both x86 (Intel(R) Xeon(R), E5-2699 ) > and ppc platform (Habanero, Model: 8348-21C), Red Hat Enterprise Linux Server > release 7.2 (Maipo)., Spark2.0.0-preview (May-24, 2016 build) >Reporter: Peter Liu > Labels: performance, spark2.0.0 > > With Spark2.0.0-preview (May-24 build), the history event data (the json > file), that is generated for each Spark application run (see below), can be > as big as 5GB (instead of 14 MB for exactly the same application run and the > same input data of 1TB under Spark1.6.1) > -rwxrwx--- 1 root root 5.3G Jun 30 09:39 app-20160630091959- > -rwxrwx--- 1 root root 5.3G Jun 30 09:56 app-20160630094213- > -rwxrwx--- 1 root root 5.3G Jun 30 10:13 app-20160630095856- > -rwxrwx--- 1 root root 5.3G Jun 30 10:30 app-20160630101556- > The test is done with Sparkbench V2, SQL RDD (see github: > https://github.com/SparkTC/spark-bench) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971633#comment-15971633 ] Alex Bozarth commented on SPARK-14245: -- Given it's been a year since I fixed that PR I honestly can't remember the exact details of that race condition, and looking back I wish I had been more through in detailing it in my comment. I can remember how the race condition manifested as though: After first starting up a stand-alone master (using user1), I would then submit two apps simultaneously (or close enough for testing) using two users (user1 and user2) in this case both applications always listed user1, but this would only happen if the two were the first applications submitted after starting up the master (subsequent simultaneous submittals worked as expected). As I mentioned I can't remember how I tracked down and identified the race condition in the code though, feel free to dig into it. I would look at it, but I don't have time at the moment. > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves >Assignee: Alex Bozarth > Fix For: 2.0.0 > > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-20293) In the page of 'jobs' or 'stages' of history server web ui,,click the 'Go' button, query paging data, the page error
[ https://issues.apache.org/jira/browse/SPARK-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Bozarth resolved SPARK-20293. -- Resolution: Duplicate Already fixed in master and branch-2.1 > In the page of 'jobs' or 'stages' of history server web ui,,click the 'Go' > button, query paging data, the page error > - > > Key: SPARK-20293 > URL: https://issues.apache.org/jira/browse/SPARK-20293 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0 >Reporter: guoxiaolongzte > Attachments: error1.png, error2.png, jobs.png, stages.png > > > In the page of 'jobs' or 'stages' of history server web ui, > Click on the 'Go' button, query paging data, the page error, function can not > be used. > The reasons are as follows: > '#' Was escaped by the browser as% 23. > & CompletedStage.desc = true% 23completed, the parameter value desc becomes = > true% 23, causing the page to report an error. The error is as follows: > HTTP ERROR 400 > Problem Access / history / app-20170411132432-0004 / stages /. Reason: > For input string: "true # completed" > Powered by Jetty: // > The amendments are as follows: > The URL of the accessed URL is escaped to ensure that the URL is not escaped > by the browser. > please see attachment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20044) Support Spark UI behind front-end reverse proxy using a path prefix
[ https://issues.apache.org/jira/browse/SPARK-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937356#comment-15937356 ] Alex Bozarth commented on SPARK-20044: -- I took a look at your link and it looks like it's on the right path, it seems you still need to go through and make sure it completely solves the problem. If you open up a PR I'll take a more detailed look. This addition actually solves an issue I had with the original pr, I was always worried that `spark.iu.reverseProxyUrl` was only used in one place and didn't have to be correct in many use-cases, this addition seems to leverage it across the UI to solve your issue. > Support Spark UI behind front-end reverse proxy using a path prefix > --- > > Key: SPARK-20044 > URL: https://issues.apache.org/jira/browse/SPARK-20044 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 2.1.0 >Reporter: Oliver Koeth >Priority: Minor > Labels: reverse-proxy, sso > > Purpose: allow to run the Spark web UI behind a reverse proxy with URLs > prefixed by a context root, like www.mydomain.com/spark. In particular, this > allows to access multiple Spark clusters through the same virtual host, only > distinguishing them by context root, like www.mydomain.com/cluster1, > www.mydomain.com/cluster2, and it allows to run the Spark UI in a common > cookie domain (for SSO) with other services. > [SPARK-15487] introduced some support for front-end reverse proxies by > allowing all Spark UI requests to be routed through the master UI as a single > endpoint and also added a spark.ui.reverseProxyUrl setting to define a > another proxy sitting in front of Spark. However, as noted in the comments on > [SPARK-15487], this mechanism does not currently work if the reverseProxyUrl > includes a context root like the examples above: Most links generated by the > Spark UI result in full path URLs (like /proxy/app-"id"/...) that do not > account for a path prefix (context root) and work only if the Spark UI "owns" > the entire virtual host. In fact, the only place in the UI where the > reverseProxyUrl seems to be used is the back-link from the worker UI to the > master UI. > The discussion on [SPARK-15487] proposes to open a new issue for the problem, > but that does not seem to have happened, so this issue aims to address the > remaining shortcomings of spark.ui.reverseProxyUrl > The problem can be partially worked around by doing content rewrite in a > front-end proxy and prefixing src="/..." or href="/..." links with a context > root. However, detecting and patching URLs in HTML output is not a robust > approach and breaks down for URLs included in custom REST responses. E.g. the > "allexecutors" REST call used from the Spark 2.1.0 application/executors page > returns links for log viewing that direct to the worker UI and do not work in > this scenario. > This issue proposes to honor spark.ui.reverseProxyUrl throughout Spark UI URL > generation. Experiments indicate that most of this can simply be achieved by > using/prepending spark.ui.reverseProxyUrl to the existing spark.ui.proxyBase > system property. Beyond that, the places that require adaption are > - worker and application links in the master web UI > - webui URLs returned by REST interfaces > Note: It seems that returned redirect location headers do not need to be > adapted, since URL rewriting for these is commonly done in front-end proxies > and has a well-defined interface -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20044) Support Spark UI behind front-end reverse proxy using a path prefix
[ https://issues.apache.org/jira/browse/SPARK-20044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935223#comment-15935223 ] Alex Bozarth commented on SPARK-20044: -- I like this idea in theory, but I worried it would take a large sweeping code change to work. If you have an implication idea already I would suggest opening a pr. For me, accepting this would hinge on how it's implemented, I'd rather not add lots of new code across the entire web ui. [~vanzin] and [~tgraves] what do you guys think, you helped review the reverse proxy pr > Support Spark UI behind front-end reverse proxy using a path prefix > --- > > Key: SPARK-20044 > URL: https://issues.apache.org/jira/browse/SPARK-20044 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 2.1.0 >Reporter: Oliver Koeth >Priority: Minor > Labels: reverse-proxy, sso > > Purpose: allow to run the Spark web UI behind a reverse proxy with URLs > prefixed by a context root, like www.mydomain.com/spark. In particular, this > allows to access multiple Spark clusters through the same virtual host, only > distinguishing them by context root, like www.mydomain.com/cluster1, > www.mydomain.com/cluster2, and it allows to run the Spark UI in a common > cookie domain (for SSO) with other services. > [SPARK-15487] introduced some support for front-end reverse proxies by > allowing all Spark UI requests to be routed through the master UI as a single > endpoint and also added a spark.ui.reverseProxyUrl setting to define a > another proxy sitting in front of Spark. However, as noted in the comments on > [SPARK-15487], this mechanism does not currently work if the reverseProxyUrl > includes a context root like the examples above: Most links generated by the > Spark UI result in full path URLs (like /proxy/app-"id"/...) that do not > account for a path prefix (context root) and work only if the Spark UI "owns" > the entire virtual host. In fact, the only place in the UI where the > reverseProxyUrl seems to be used is the back-link from the worker UI to the > master UI. > The discussion on [SPARK-15487] proposes to open a new issue for the problem, > but that does not seem to have happened, so this issue aims to address the > remaining shortcomings of spark.ui.reverseProxyUrl > The problem can be partially worked around by doing content rewrite in a > front-end proxy and prefixing src="/..." or href="/..." links with a context > root. However, detecting and patching URLs in HTML output is not a robust > approach and breaks down for URLs included in custom REST responses. E.g. the > "allexecutors" REST call used from the Spark 2.1.0 application/executors page > returns links for log viewing that direct to the worker UI and do not work in > this scenario. > This issue proposes to honor spark.ui.reverseProxyUrl throughout Spark UI URL > generation. Experiments indicate that most of this can simply be achieved by > using/prepending spark.ui.reverseProxyUrl to the existing spark.ui.proxyBase > system property. Beyond that, the places that require adaption are > - worker and application links in the master web UI > - webui URLs returned by REST interfaces > Note: It seems that returned redirect location headers do not need to be > adapted, since URL rewriting for these is commonly done in front-end proxies > and has a well-defined interface -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-19283) Application details UI not visible for completed actions
[ https://issues.apache.org/jira/browse/SPARK-19283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Bozarth resolved SPARK-19283. -- Resolution: Not A Problem > Application details UI not visible for completed actions > > > Key: SPARK-19283 > URL: https://issues.apache.org/jira/browse/SPARK-19283 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.2 > Environment: ubuntu 14.04 16GB RAM 16 cores >Reporter: Deenbandhu Agarwal >Priority: Minor > > I just upgraded from spark 1.6.1 to spark 2.0.2 and tried to enable logging > for completed jobs using 'spark.eventLog.enabled=true'. > log files are created in specified directory but no option is visible on UI > to see the details of the completed jobs > I checked the file permissions i don't think this will be the issue. > This was working in spark 1.6.1 with same config -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19283) Application details UI not visible for completed actions
[ https://issues.apache.org/jira/browse/SPARK-19283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830236#comment-15830236 ] Alex Bozarth commented on SPARK-19283: -- Application History was removed from the Master UI, but is still available via the History Server. > Application details UI not visible for completed actions > > > Key: SPARK-19283 > URL: https://issues.apache.org/jira/browse/SPARK-19283 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.2 > Environment: ubuntu 14.04 16GB RAM 16 cores >Reporter: Deenbandhu Agarwal >Priority: Minor > > I just upgraded from spark 1.6.1 to spark 2.0.2 and tried to enable logging > for completed jobs using 'spark.eventLog.enabled=true'. > log files are created in specified directory but no option is visible on UI > to see the details of the completed jobs > I checked the file permissions i don't think this will be the issue. > This was working in spark 1.6.1 with same config -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828866#comment-15828866 ] Alex Bozarth commented on SPARK-18085: -- I'm interested in helping with M4, but I am currently bogged down a bit in other work. I should be freed up to help by the start of February. > Better History Server scalability for many / large applications > --- > > Key: SPARK-18085 > URL: https://issues.apache.org/jira/browse/SPARK-18085 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI >Affects Versions: 2.0.0 >Reporter: Marcelo Vanzin > Attachments: spark_hs_next_gen.pdf > > > It's a known fact that the History Server currently has some annoying issues > when serving lots of applications, and when serving large applications. > I'm filing this umbrella to track work related to addressing those issues. > I'll be attaching a document shortly describing the issues and suggesting a > path to how to solve them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18837) Very long stage descriptions do not wrap in the UI
[ https://issues.apache.org/jira/browse/SPARK-18837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15746511#comment-15746511 ] Alex Bozarth commented on SPARK-18837: -- Did some digging and this *seems* to have been caused by https://github.com/apache/spark/commit/478b71d028107d42fbb6d1bd300b86efbe0dcf7d I can't seem to figure out why though. I leave on vacation tomorrow so I don't have time to look into it more until Jan. > Very long stage descriptions do not wrap in the UI > -- > > Key: SPARK-18837 > URL: https://issues.apache.org/jira/browse/SPARK-18837 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 2.1.0 >Reporter: Yuming Wang >Priority: Minor > Attachments: ui-2.0.0.gif, ui-2.1.0.gif > > > *previous*: > !ui-2.0.0.gif! > *current*: > !ui-2.1.0.gif! -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743419#comment-15743419 ] Alex Bozarth commented on SPARK-18816: -- I have a fix, just running a few tests then I'll open a pr > executor page fails to show log links if executors are added after an app is > launched > - > > Key: SPARK-18816 > URL: https://issues.apache.org/jira/browse/SPARK-18816 > Project: Spark > Issue Type: Bug > Components: Web UI >Reporter: Yin Huai >Priority: Blocker > Attachments: screenshot-1.png > > > How to reproduce with standalone mode: > 1. Launch a spark master > 2. Launch a spark shell. At this point, there is no executor associated with > this application. > 3. Launch a slave. Now, there is an executor assigned to the spark shell. > However, there is no link to stdout/stderr on the executor page (please see > https://issues.apache.org/jira/secure/attachment/12842649/screenshot-1.png). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743313#comment-15743313 ] Alex Bozarth commented on SPARK-18816: -- And I still don't think this is a blocker, but I respect that you as a committer know more about Spark than I do. > executor page fails to show log links if executors are added after an app is > launched > - > > Key: SPARK-18816 > URL: https://issues.apache.org/jira/browse/SPARK-18816 > Project: Spark > Issue Type: Bug > Components: Web UI >Reporter: Yin Huai >Priority: Blocker > Attachments: screenshot-1.png > > > How to reproduce with standalone mode: > 1. Launch a spark master > 2. Launch a spark shell. At this point, there is no executor associated with > this application. > 3. Launch a slave. Now, there is an executor assigned to the spark shell. > However, there is no link to stdout/stderr on the executor page (please see > https://issues.apache.org/jira/secure/attachment/12842649/screenshot-1.png). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743307#comment-15743307 ] Alex Bozarth commented on SPARK-18816: -- I'm actually looking into a bit right now and I think it's an issue with the jQuery code I used when I made the column conditional. If I find a solution I'll either open a quick pr or post the info here for you to fix. > executor page fails to show log links if executors are added after an app is > launched > - > > Key: SPARK-18816 > URL: https://issues.apache.org/jira/browse/SPARK-18816 > Project: Spark > Issue Type: Bug > Components: Web UI >Reporter: Yin Huai >Priority: Blocker > Attachments: screenshot-1.png > > > How to reproduce with standalone mode: > 1. Launch a spark master > 2. Launch a spark shell. At this point, there is no executor associated with > this application. > 3. Launch a slave. Now, there is an executor assigned to the spark shell. > However, there is no link to stdout/stderr on the executor page (please see > https://issues.apache.org/jira/secure/attachment/12842649/screenshot-1.png). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743288#comment-15743288 ] Alex Bozarth commented on SPARK-18816: -- Thanks for following up, I was able to recreate the issue, but I personally wont have time to fix it before my holiday vacation. It's not a blocker still because the log pages are still there, only the links to them are missing. You can still access the logs for each worker via the Worker UI links found on the Master UI. > executor page fails to show log links if executors are added after an app is > launched > - > > Key: SPARK-18816 > URL: https://issues.apache.org/jira/browse/SPARK-18816 > Project: Spark > Issue Type: Bug > Components: Web UI >Reporter: Yin Huai > Attachments: screenshot-1.png > > > How to reproduce with standalone mode: > 1. Launch a spark master > 2. Launch a spark shell. At this point, there is no executor associated with > this application. > 3. Launch a slave. Now, there is an executor assigned to the spark shell. > However, there is no link to stdout/stderr on the executor page (please see > https://issues.apache.org/jira/secure/attachment/12842649/screenshot-1.png). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15737454#comment-15737454 ] Alex Bozarth commented on SPARK-18816: -- That's odd, I tested my code on Safari, Chrome and FF when I made that change. I can look into it Monday if you don't have a fix. > executor page fails to show log links if executors are added after an app is > launched > - > > Key: SPARK-18816 > URL: https://issues.apache.org/jira/browse/SPARK-18816 > Project: Spark > Issue Type: Bug > Components: Web UI >Reporter: Yin Huai >Priority: Blocker > Attachments: screenshot-1.png > > > How to reproduce with standalone mode: > 1. Launch a spark master > 2. Launch a spark shell. At this point, there is no executor associated with > this application. > 3. Launch a slave. Now, there is an executor assigned to the spark shell. > However, there is no link to stdout/stderr on the executor page (please see > https://issues.apache.org/jira/secure/attachment/12842649/screenshot-1.png). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15737454#comment-15737454 ] Alex Bozarth edited comment on SPARK-18816 at 12/10/16 7:57 AM: That's odd, I tested my code on Safari, Chrome and FF when I made that change. I can look into it Monday if you don't have a fix. Also this is not a blocker. was (Author: ajbozarth): That's odd, I tested my code on Safari, Chrome and FF when I made that change. I can look into it Monday if you don't have a fix. > executor page fails to show log links if executors are added after an app is > launched > - > > Key: SPARK-18816 > URL: https://issues.apache.org/jira/browse/SPARK-18816 > Project: Spark > Issue Type: Bug > Components: Web UI >Reporter: Yin Huai >Priority: Blocker > Attachments: screenshot-1.png > > > How to reproduce with standalone mode: > 1. Launch a spark master > 2. Launch a spark shell. At this point, there is no executor associated with > this application. > 3. Launch a slave. Now, there is an executor assigned to the spark shell. > However, there is no link to stdout/stderr on the executor page (please see > https://issues.apache.org/jira/secure/attachment/12842649/screenshot-1.png). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18816) executor page fails to show log links if executors are added after an app is launched
[ https://issues.apache.org/jira/browse/SPARK-18816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Bozarth updated SPARK-18816: - Priority: Major (was: Blocker) > executor page fails to show log links if executors are added after an app is > launched > - > > Key: SPARK-18816 > URL: https://issues.apache.org/jira/browse/SPARK-18816 > Project: Spark > Issue Type: Bug > Components: Web UI >Reporter: Yin Huai > Attachments: screenshot-1.png > > > How to reproduce with standalone mode: > 1. Launch a spark master > 2. Launch a spark shell. At this point, there is no executor associated with > this application. > 3. Launch a slave. Now, there is an executor assigned to the spark shell. > However, there is no link to stdout/stderr on the executor page (please see > https://issues.apache.org/jira/secure/attachment/12842649/screenshot-1.png). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-18551) Add functionality to delete event logs from the History Server UI
[ https://issues.apache.org/jira/browse/SPARK-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Bozarth resolved SPARK-18551. -- Resolution: Won't Fix Closing this based on complexity and security concerns raised by [~vanzin] > Add functionality to delete event logs from the History Server UI > - > > Key: SPARK-18551 > URL: https://issues.apache.org/jira/browse/SPARK-18551 > Project: Spark > Issue Type: New Feature > Components: Spark Core, Web UI >Reporter: Alex Bozarth > > Sometimes a Spark user will only have access to a History Server to interact > with their (past) applications. But without access to the server they can > only delete applications through use of the FS Cleaner feature, which itself > can only clean logs older than a set date. > I propose adding the ability to delete specific applications via the History > Server UI with the default setting to off. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18551) Add functionality to delete event logs from the History Server UI
[ https://issues.apache.org/jira/browse/SPARK-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703562#comment-15703562 ] Alex Bozarth commented on SPARK-18551: -- Ok, now I see, I will close this and my pr and if anyone ask about this in the future I will pass on this answer. This is a very straightforward and smart reason to drop this, thanks. > Add functionality to delete event logs from the History Server UI > - > > Key: SPARK-18551 > URL: https://issues.apache.org/jira/browse/SPARK-18551 > Project: Spark > Issue Type: New Feature > Components: Spark Core, Web UI >Reporter: Alex Bozarth > > Sometimes a Spark user will only have access to a History Server to interact > with their (past) applications. But without access to the server they can > only delete applications through use of the FS Cleaner feature, which itself > can only clean logs older than a set date. > I propose adding the ability to delete specific applications via the History > Server UI with the default setting to off. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18551) Add functionality to delete event logs from the History Server UI
[ https://issues.apache.org/jira/browse/SPARK-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703466#comment-15703466 ] Alex Bozarth commented on SPARK-18551: -- I'll take a shot at convincing you with my code, I'll work on some updates with your comments in mind and if you shoot ithem down I'll accept it at that. Thanks for all the input, I knew this would be a bit controversial when I opened it, but I hadn't properly considered the security issues. > Add functionality to delete event logs from the History Server UI > - > > Key: SPARK-18551 > URL: https://issues.apache.org/jira/browse/SPARK-18551 > Project: Spark > Issue Type: New Feature > Components: Spark Core, Web UI >Reporter: Alex Bozarth > > Sometimes a Spark user will only have access to a History Server to interact > with their (past) applications. But without access to the server they can > only delete applications through use of the FS Cleaner feature, which itself > can only clean logs older than a set date. > I propose adding the ability to delete specific applications via the History > Server UI with the default setting to off. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18551) Add functionality to delete event logs from the History Server UI
[ https://issues.apache.org/jira/browse/SPARK-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15703052#comment-15703052 ] Alex Bozarth commented on SPARK-18551: -- Thanks for the feedback [~vanzin], I've paused work on my PR while we discuss what you've brought up. For your first point, my thinking was that the person who started the SHS and enabled the deletion config would be the liable user. This would/should be clearly documented. This means that in would be on the "sysadmin" who started the SHS to secure the SHS since anyone with access could delete apps. I'm not quite sure what you mean about the UI's auth code though, are you referring to the security manager? Since the security manager was already instantiated in the history server I assumed it was fair to use, though again it only looks at the startup user. The second point is one I hadn't thought of though, as long as it's documented as such I don't see the issue with the logs showing the SHS user for every deletion. I agree that trying to get the web user's username to the logs is lots of unnecessary work. The use case that raised this issue is when a user is given a SHS by their admin to access their app ui's and it only shows their apps and only they (and the admin) can access it. The user here has no access to the log folder without going through their admin. As for [~ste...@apache.org]'s GET comment, I plan to fix that, it was a leftover from [~srowen]'s update to the kill stage code that I kept because (at the time) I didn't understand its purpose (Its for YARN, which doesn't apply here). If you still think this is an unsalvageable idea I will close this, I respect your opinion [~vanzin] > Add functionality to delete event logs from the History Server UI > - > > Key: SPARK-18551 > URL: https://issues.apache.org/jira/browse/SPARK-18551 > Project: Spark > Issue Type: New Feature > Components: Spark Core, Web UI >Reporter: Alex Bozarth > > Sometimes a Spark user will only have access to a History Server to interact > with their (past) applications. But without access to the server they can > only delete applications through use of the FS Cleaner feature, which itself > can only clean logs older than a set date. > I propose adding the ability to delete specific applications via the History > Server UI with the default setting to off. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18551) Add functionality to delete event logs from the History Server UI
[ https://issues.apache.org/jira/browse/SPARK-18551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688393#comment-15688393 ] Alex Bozarth commented on SPARK-18551: -- I will have a wip pr open for this shortly > Add functionality to delete event logs from the History Server UI > - > > Key: SPARK-18551 > URL: https://issues.apache.org/jira/browse/SPARK-18551 > Project: Spark > Issue Type: New Feature > Components: Spark Core, Web UI >Reporter: Alex Bozarth > > Sometimes a Spark user will only have access to a History Server to interact > with their (past) applications. But without access to the server they can > only delete applications through use of the FS Cleaner feature, which itself > can only clean logs older than a set date. > I propose adding the ability to delete specific applications via the History > Server UI with the default setting to off. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18551) Add functionality to delete event logs from the History Server UI
Alex Bozarth created SPARK-18551: Summary: Add functionality to delete event logs from the History Server UI Key: SPARK-18551 URL: https://issues.apache.org/jira/browse/SPARK-18551 Project: Spark Issue Type: New Feature Components: Spark Core, Web UI Reporter: Alex Bozarth Sometimes a Spark user will only have access to a History Server to interact with their (past) applications. But without access to the server they can only delete applications through use of the FS Cleaner feature, which itself can only clean logs older than a set date. I propose adding the ability to delete specific applications via the History Server UI with the default setting to off. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18298) HistoryServer use GMT time all time
[
https://issues.apache.org/jira/browse/SPARK-18298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645220#comment-15645220
]
Alex Bozarth commented on SPARK-18298:
--
I'm not sure what problem are you seeing, because dates are just stored as
longs.
Is it?
1. Dates displayed are being translated from local time to GMT (Times are
correct, just wrong timezone)
2. Dates are showing local time but claimed as GMT (the actual stored time
would be incorrect in this case, since the longs are always GMT)
If its number two then it's a bug we should fix, if it's 1 then that would be a
feature request to support timezones in the History Server.
> HistoryServer use GMT time all time
> ---
>
> Key: SPARK-18298
> URL: https://issues.apache.org/jira/browse/SPARK-18298
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.0.0, 2.0.1
> Environment: suse 11.3 with CST time
>Reporter: Tao Wang
>
> When I started HistoryServer for reading event logs, the timestamp readed
> will be parsed using local timezone like "CST"(confirmed via debug).
> But the time related columns like "Started"/"Completed"/"Last Updated" in
> History Server UI using "GMT" time, which is 8 hours earlier than "CST".
> {quote}
> App IDApp NameStarted Completed DurationSpark
> User Last UpdatedEvent Log
> local-1478225166651 Spark shell 2016-11-04 02:06:06 2016-11-07
> 01:33:30 71.5 h root2016-11-07 01:33:30
> {quote}
> I've checked the REST api and found the result like:
> {color:red}
> [ {
> "id" : "local-1478225166651",
> "name" : "Spark shell",
> "attempts" : [ {
> "startTime" : "2016-11-04T02:06:06.020GMT",
> "endTime" : "2016-11-07T01:33:30.265GMT",
> "lastUpdated" : "2016-11-07T01:33:30.000GMT",
> "duration" : 257244245,
> "sparkUser" : "root",
> "completed" : true,
> "lastUpdatedEpoch" : 147848241,
> "endTimeEpoch" : 1478482410265,
> "startTimeEpoch" : 1478225166020
> } ]
> }, {
> "id" : "local-1478224925869",
> "name" : "Spark Pi",
> "attempts" : [ {
> "startTime" : "2016-11-04T02:02:02.133GMT",
> "endTime" : "2016-11-04T02:02:07.468GMT",
> "lastUpdated" : "2016-11-04T02:02:07.000GMT",
> "duration" : 5335,
> "sparkUser" : "root",
> "completed" : true,
> ...
> {color}
> So maybe the change happened in transferring between server and browser? I
> have no idea where to go from this point.
> Hope guys can offer some help, or just fix it if it's easy? :)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15606523#comment-15606523 ] Alex Bozarth commented on SPARK-18085: -- I am *very* interested in working with you on this project and (post-Spark Summit) would love to discuss some of the UI ideas my team has been tossing around (a few covered in your non-goals). > Scalability enhancements for the History Server > --- > > Key: SPARK-18085 > URL: https://issues.apache.org/jira/browse/SPARK-18085 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI >Affects Versions: 2.0.0 >Reporter: Marcelo Vanzin > Attachments: spark_hs_next_gen.pdf > > > It's a known fact that the History Server currently has some annoying issues > when serving lots of applications, and when serving large applications. > I'm filing this umbrella to track work related to addressing those issues. > I'll be attaching a document shortly describing the issues and suggesting a > path to how to solve them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18073) Migrate wiki to spark.apache.org web site
[ https://issues.apache.org/jira/browse/SPARK-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602747#comment-15602747 ] Alex Bozarth commented on SPARK-18073: -- I think we should keep the Internals pages but (open some JIRAs to) update them, I'd be willing to look at updating the Web UI Internals page, it helped me when I was first starting but it's pretty out of date now. > Migrate wiki to spark.apache.org web site > - > > Key: SPARK-18073 > URL: https://issues.apache.org/jira/browse/SPARK-18073 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.1 >Reporter: Sean Owen > > Per > http://apache-spark-developers-list.1001551.n3.nabble.com/Mini-Proposal-Make-it-easier-to-contribute-to-the-contributing-to-Spark-Guide-td19493.html > , let's consider migrating all wiki pages to documents at > github.com/apache/spark-webiste (i.e. spark.apache.org). > Some reasons: > * No pull request system or history for changes to the wiki > * Separate, not-so-clear system for granting write access to wiki > * Wiki doesn't change much > * One less place to maintain or look for docs > The idea would be to then update all wiki pages with a message pointing to > the new home of the information (or message saying it's obsolete). > Here are the current wikis and my general proposal for what to do with the > content: > * Additional Language Bindings -> roll this into wherever Third Party > Projects ends up > * Committers -> Migrate to a new /committers.html page, linked under > Community menu (alread exists) > * Contributing to Spark -> Make this CONTRIBUTING.md? or a new > /contributing.html page under Community menu > ** Jira Permissions Scheme -> obsolete > ** Spark Code Style Guide -> roll this into new contributing.html page > * Development Discussions -> obsolete? > * Powered By Spark -> Make into new /powered-by.html linked by the existing > Commnunity menu item > * Preparing Spark Releases -> see below; roll into where "versioning policy" > goes? > * Profiling Spark Applications -> roll into where Useful Developer Tools goes > ** Profiling Spark Applications Using YourKit -> ditto > * Spark Internals -> all of these look somewhat to very stale; remove? > ** Java API Internals > ** PySpark Internals > ** Shuffle Internals > ** Spark SQL Internals > ** Web UI Internals > * Spark QA Infrastructure -> tough one. Good info to document; does it belong > on the website? we can just migrate it > * Spark Versioning Policy -> new page living under Community (?) that > documents release policy and process (better menu?) > ** spark-ec2 AMI list and install file version mappings -> obsolete > ** Spark-Shark version mapping -> obsolete > * Third Party Projects -> new Community menu item > * Useful Developer Tools -> new page under new Developer menu? Community? > ** Jenkins -> obsolete, remove > Of course, another outcome is to just remove outdated wikis, migrate some, > leave the rest. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18010) Remove unneeded heavy work performed by FsHistoryProvider for building up the application listing UI page
[ https://issues.apache.org/jira/browse/SPARK-18010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15589887#comment-15589887 ] Alex Bozarth commented on SPARK-18010: -- [~srowen] [~tgraves] This seems to "fix" the long ongoing SPARK-6951, should we mark it as a duplicate or would you consider them separate? > Remove unneeded heavy work performed by FsHistoryProvider for building up the > application listing UI page > - > > Key: SPARK-18010 > URL: https://issues.apache.org/jira/browse/SPARK-18010 > Project: Spark > Issue Type: Improvement > Components: Spark Core, Web UI >Affects Versions: 1.6.2, 2.0.1, 2.1.0 >Reporter: Vinayak Joshi > > There are known complaints/cribs about History Server's Application List not > updating quickly enough when the event log files that need replay are huge. > Currently, the FsHistoryProvider design causes the entire event log file to > be replayed when building the initial application listing (refer the method > mergeApplicationListing(fileStatus: FileStatus) ). The process of replay > involves: > - each line in the event log being read as a string, > - parsing the string to a Json structure > - converting the Json to the corresponding Scala classes with nested > structures > Particularly the part involving parsing string to Json and then to Scala > classes is expensive. Tests show that majority of time spent in replay is in > doing this work. > When the replay is performed for building the application listing, the only > two events that the code really cares for are "SparkListenerApplicationStart" > and "SparkListenerApplicationEnd" - since the only listener attached to the > ReplayListenerBus at that point is the ApplicationEventListener. This means > that when processing an event log file with a huge number (hundreds of > thousands, can be more) of events, the work done to deserialize all of these > event, and then replay them is not needed. Only two events are what we're > interested in, and this can be used to ensure that when replay is performed > for the purpose of building the application list, we only make the effort to > replay these two events and not others. > My tests show that this drastically improves application list load time. For > a 150MB event log from a user, with over 100,000 events, the load time (local > on my mac) comes down from about 16 secs to under 1 second using this > approach. For customers that typically execute applications with large event > logs, and thus have multiple large event logs present, this can speed up how > soon the history server UI lists the apps considerably. > I will be updating a pull request with take at fixing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15708) Tasks table in Detailed Stage page shows ip instead of hostname under Executor ID/Host
[ https://issues.apache.org/jira/browse/SPARK-15708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586703#comment-15586703 ] Alex Bozarth commented on SPARK-15708: -- I have not seen this recently so it was possibly incidentally fixed. I had this on my back burner and did the above research before seeing it had been closed so I wanted to share it for future reference. > Tasks table in Detailed Stage page shows ip instead of hostname under > Executor ID/Host > -- > > Key: SPARK-15708 > URL: https://issues.apache.org/jira/browse/SPARK-15708 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.0 >Reporter: Thomas Graves >Priority: Minor > > If you go to the detailed Stages page in Spark 2.0, the Tasks table under the > Executor ID/Host columns hosts the hostname as an ip address rather then a > fully qualified hostname. > The table above it (Aggregated Metrics by Executor) shows the "Address" as > the full hostname. > I'm running spark on yarn on latest branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15708) Tasks table in Detailed Stage page shows ip instead of hostname under Executor ID/Host
[
https://issues.apache.org/jira/browse/SPARK-15708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583751#comment-15583751
]
Alex Bozarth commented on SPARK-15708:
--
I'm not sure closing this as cannot reproduce was correct, but I'm not sure how
it could be fixed either. Due to the nature of those tables they get the host
string from entirely different places in code. For the task table it's stored
in {{TaskInfo}} but for the Agg. Metrics tables it's stored in
{{BlockManagerId}}. The better question is when can these two end up with
different host strings (IP vs hostname) and why. [~tgraves] is this something
you would want fixed or was it just an behavioral oddity?
> Tasks table in Detailed Stage page shows ip instead of hostname under
> Executor ID/Host
> --
>
> Key: SPARK-15708
> URL: https://issues.apache.org/jira/browse/SPARK-15708
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.0.0
>Reporter: Thomas Graves
>Priority: Minor
>
> If you go to the detailed Stages page in Spark 2.0, the Tasks table under the
> Executor ID/Host columns hosts the hostname as an ip address rather then a
> fully qualified hostname.
> The table above it (Aggregated Metrics by Executor) shows the "Address" as
> the full hostname.
> I'm running spark on yarn on latest branch-2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4411) Add "kill" link for jobs in the UI
[ https://issues.apache.org/jira/browse/SPARK-4411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15566430#comment-15566430 ] Alex Bozarth commented on SPARK-4411: - I'm currently working on this. I'm updating the original pr to work with the latest code and to match how the kill stage code works > Add "kill" link for jobs in the UI > -- > > Key: SPARK-4411 > URL: https://issues.apache.org/jira/browse/SPARK-4411 > Project: Spark > Issue Type: New Feature > Components: Web UI >Affects Versions: 1.2.0 >Reporter: Kay Ousterhout > > SPARK-4145 changes the default landing page for the UI to show jobs. We > should have a "kill" link for each job, similar to what we have for each > stage, so it's easier for users to kill slow jobs (and the semantics of > killing a job are slightly different than killing a stage). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17795) Sorting on stage or job tables doesn’t reload page on that table
[ https://issues.apache.org/jira/browse/SPARK-17795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550417#comment-15550417 ] Alex Bozarth commented on SPARK-17795: -- I have a fix for this and will submit a pr soon > Sorting on stage or job tables doesn’t reload page on that table > > > Key: SPARK-17795 > URL: https://issues.apache.org/jira/browse/SPARK-17795 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0 >Reporter: Alex Bozarth >Priority: Minor > > When you sort on a table on the AllJobs, Job, or AllStages pages the page > will reload at the top of the page rather than on the table that was just > sorted. This can be frustrating if you have large tables to scroll past and > want to do quick sorting. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-17795) Sorting on stage or job tables doesn’t reload page on that table
Alex Bozarth created SPARK-17795: Summary: Sorting on stage or job tables doesn’t reload page on that table Key: SPARK-17795 URL: https://issues.apache.org/jira/browse/SPARK-17795 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 2.1.0 Reporter: Alex Bozarth Priority: Minor When you sort on a table on the AllJobs, Job, or AllStages pages the page will reload at the top of the page rather than on the table that was just sorted. This can be frustrating if you have large tables to scroll past and want to do quick sorting. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17793) Sorting on the description on the Job or Stage page doesn’t always work
[ https://issues.apache.org/jira/browse/SPARK-17793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550279#comment-15550279 ] Alex Bozarth commented on SPARK-17793: -- I have a fix for this and will be submitting the pr soon > Sorting on the description on the Job or Stage page doesn’t always work > --- > > Key: SPARK-17793 > URL: https://issues.apache.org/jira/browse/SPARK-17793 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0 >Reporter: Alex Bozarth >Priority: Minor > > When you sort on Description on the jobs or stages table when there's no > description, just the stage name, it will sort the rows in the same order > where it’s supposed to be descending or ascending. This behave oddly when > none of the stages have descriptions and the column doesn't sort on what's > displayed (the stage names) > Seems to be caused by a doing a getOrElse(“”) on the description, causing > them all to be equal. Since description doesn't always exist and the Stage > name usually does, the column should secondary sort on stage name for clarity. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-17793) Sorting on the description on the Job or Stage page doesn’t always work
Alex Bozarth created SPARK-17793: Summary: Sorting on the description on the Job or Stage page doesn’t always work Key: SPARK-17793 URL: https://issues.apache.org/jira/browse/SPARK-17793 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 2.1.0 Reporter: Alex Bozarth Priority: Minor When you sort on Description on the jobs or stages table when there's no description, just the stage name, it will sort the rows in the same order where it’s supposed to be descending or ascending. This behave oddly when none of the stages have descriptions and the column doesn't sort on what's displayed (the stage names) Seems to be caused by a doing a getOrElse(“”) on the description, causing them all to be equal. Since description doesn't always exist and the Stage name usually does, the column should secondary sort on stage name for clarity. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17598) User-friendly name for Spark Thrift Server in web UI
[
https://issues.apache.org/jira/browse/SPARK-17598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527110#comment-15527110
]
Alex Bozarth commented on SPARK-17598:
--
I have a quick easy fix for this, but you can already rename the thrift server
by adding {{--name "App Name"}} when running the start script.
> User-friendly name for Spark Thrift Server in web UI
>
>
> Key: SPARK-17598
> URL: https://issues.apache.org/jira/browse/SPARK-17598
> Project: Spark
> Issue Type: Improvement
> Components: SQL, Web UI
>Affects Versions: 2.0.1
>Reporter: Jacek Laskowski
>Priority: Minor
> Attachments: spark-thriftserver-webui.png
>
>
> The name of Spark Thrift JDBC/ODBC Server in web UI reflects the name of the
> class, i.e. {{org.apache.spark.sql.hive.thrift.HiveThriftServer2}}. We could
> do better and call it {{Thrift JDBC/ODBC Server}} (like {{Spark shell}} for
> spark-shell).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449790#comment-15449790 ] Alex Bozarth commented on SPARK-17243: -- [~ste...@apache.org] [~tgraves] The issues you mentioned are what I'm hoping to work on next month (what I mentioned above) when I'm given the bandwidth to do so. When that comes I'll file a JIRA and loop you two in to discuss implementation ideas. (Unless some brave soul decides to give it a try before then) > Spark 2.0 history server summary page gets stuck at "loading history summary" > with 10K+ application history > --- > > Key: SPARK-17243 > URL: https://issues.apache.org/jira/browse/SPARK-17243 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.0 > Environment: Linux >Reporter: Gang Wu > > The summary page of Spark 2.0 history server web UI keep displaying "Loading > history summary..." all the time and crashes the browser when there are more > than 10K application history event logs on HDFS. > I did some investigation, "historypage.js" file sends a REST request to > /api/v1/applications endpoint of history server REST endpoint and gets back > json response. When there are more than 10K applications inside the event log > directory it takes forever to parse them and render the page. When there are > only hundreds or thousands of application history it is running fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15447266#comment-15447266 ] Alex Bozarth commented on SPARK-17243: -- that's odd, how long did you wait before accessing the app url? because the history server still needs to propagate after starting and that can take a long time, I was testing with a limit of 50 and testing an app in the thousands and it took about 5min to propagate for me to see it > Spark 2.0 history server summary page gets stuck at "loading history summary" > with 10K+ application history > --- > > Key: SPARK-17243 > URL: https://issues.apache.org/jira/browse/SPARK-17243 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.0 > Environment: Linux >Reporter: Gang Wu > > The summary page of Spark 2.0 history server web UI keep displaying "Loading > history summary..." all the time and crashes the browser when there are more > than 10K application history event logs on HDFS. > I did some investigation, "historypage.js" file sends a REST request to > /api/v1/applications endpoint of history server REST endpoint and gets back > json response. When there are more than 10K applications inside the event log > directory it takes forever to parse them and render the page. When there are > only hundreds or thousands of application history it is running fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15447112#comment-15447112 ] Alex Bozarth commented on SPARK-17243: -- [~wgtmac] I'm not sure which version of the pr you tested, in my initial commit the issue you saw still existed but I updated it EOD Friday to switch to a version that only restricts the summary display, leaving all the applications available via their direct url as you would expect. > Spark 2.0 history server summary page gets stuck at "loading history summary" > with 10K+ application history > --- > > Key: SPARK-17243 > URL: https://issues.apache.org/jira/browse/SPARK-17243 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.0 > Environment: Linux >Reporter: Gang Wu > > The summary page of Spark 2.0 history server web UI keep displaying "Loading > history summary..." all the time and crashes the browser when there are more > than 10K application history event logs on HDFS. > I did some investigation, "historypage.js" file sends a REST request to > /api/v1/applications endpoint of history server REST endpoint and gets back > json response. When there are more than 10K applications inside the event log > directory it takes forever to parse them and render the page. When there are > only hundreds or thousands of application history it is running fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439799#comment-15439799 ] Alex Bozarth commented on SPARK-17243: -- So I decided to work on this as a short break from my current work and I have a fix that just requires some final testing before I open a pr, should be open by EOD. > Spark 2.0 history server summary page gets stuck at "loading history summary" > with 10K+ application history > --- > > Key: SPARK-17243 > URL: https://issues.apache.org/jira/browse/SPARK-17243 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.0 > Environment: Linux >Reporter: Gang Wu > > The summary page of Spark 2.0 history server web UI keep displaying "Loading > history summary..." all the time and crashes the browser when there are more > than 10K application history event logs on HDFS. > I did some investigation, "historypage.js" file sends a REST request to > /api/v1/applications endpoint of history server REST endpoint and gets back > json response. When there are more than 10K applications inside the event log > directory it takes forever to parse them and render the page. When there are > only hundreds or thousands of application history it is running fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439470#comment-15439470 ] Alex Bozarth commented on SPARK-17243: -- Thanks [~ste...@apache.org], this idea is great. [~wgtmac], based on this I might be able to get a small fix for this out next week instead of waiting to include it in my larger update next month. > Spark 2.0 history server summary page gets stuck at "loading history summary" > with 10K+ application history > --- > > Key: SPARK-17243 > URL: https://issues.apache.org/jira/browse/SPARK-17243 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.0 > Environment: Linux >Reporter: Gang Wu > > The summary page of Spark 2.0 history server web UI keep displaying "Loading > history summary..." all the time and crashes the browser when there are more > than 10K application history event logs on HDFS. > I did some investigation, "historypage.js" file sends a REST request to > /api/v1/applications endpoint of history server REST endpoint and gets back > json response. When there are more than 10K applications inside the event log > directory it takes forever to parse them and render the page. When there are > only hundreds or thousands of application history it is running fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437764#comment-15437764 ] Alex Bozarth commented on SPARK-17243: -- Thanks, that'll help when I look into it > Spark 2.0 history server summary page gets stuck at "loading history summary" > with 10K+ application history > --- > > Key: SPARK-17243 > URL: https://issues.apache.org/jira/browse/SPARK-17243 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.0 > Environment: Linux >Reporter: Gang Wu > > The summary page of Spark 2.0 history server web UI keep displaying "Loading > history summary..." all the time and crashes the browser when there are more > than 10K application history event logs on HDFS. > I did some investigation, "historypage.js" file sends a REST request to > /api/v1/applications endpoint of history server REST endpoint and gets back > json response. When there are more than 10K applications inside the event log > directory it takes forever to parse them and render the page. When there are > only hundreds or thousands of application history it is running fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437768#comment-15437768 ] Alex Bozarth commented on SPARK-17243: -- Sorry, my misunderstanding of your problem, I will make sure to keep this in mind once I start my work > Spark 2.0 history server summary page gets stuck at "loading history summary" > with 10K+ application history > --- > > Key: SPARK-17243 > URL: https://issues.apache.org/jira/browse/SPARK-17243 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.0 > Environment: Linux >Reporter: Gang Wu > > The summary page of Spark 2.0 history server web UI keep displaying "Loading > history summary..." all the time and crashes the browser when there are more > than 10K application history event logs on HDFS. > I did some investigation, "historypage.js" file sends a REST request to > /api/v1/applications endpoint of history server REST endpoint and gets back > json response. When there are more than 10K applications inside the event log > directory it takes forever to parse them and render the page. When there are > only hundreds or thousands of application history it is running fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history
[
https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437682#comment-15437682
]
Alex Bozarth edited comment on SPARK-17243 at 8/25/16 9:13 PM:
---
[~wgtmac] until this is fixed you can limit the number of applications
available by setting {{spark.history.retainedApplications}} It limits the apps
the history server loads
was (Author: ajbozarth):
[~wgtmac] until this is fixed you can limit the number of applications
available by setting {spark.history.retainedApplications} It limits the apps
the history server loads
> Spark 2.0 history server summary page gets stuck at "loading history summary"
> with 10K+ application history
> ---
>
> Key: SPARK-17243
> URL: https://issues.apache.org/jira/browse/SPARK-17243
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.0.0
> Environment: Linux
>Reporter: Gang Wu
>
> The summary page of Spark 2.0 history server web UI keep displaying "Loading
> history summary..." all the time and crashes the browser when there are more
> than 10K application history event logs on HDFS.
> I did some investigation, "historypage.js" file sends a REST request to
> /api/v1/applications endpoint of history server REST endpoint and gets back
> json response. When there are more than 10K applications inside the event log
> directory it takes forever to parse them and render the page. When there are
> only hundreds or thousands of application history it is running fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history
[
https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437682#comment-15437682
]
Alex Bozarth edited comment on SPARK-17243 at 8/25/16 9:13 PM:
---
[~wgtmac] until this is fixed you can limit the number of applications
available by setting {spark.history.retainedApplications} It limits the apps
the history server loads
was (Author: ajbozarth):
[~wgtmac] until this is fixed you can limit the number of applications
available by setting `spark.history.retainedApplications` It limits the apps
the history server loads
> Spark 2.0 history server summary page gets stuck at "loading history summary"
> with 10K+ application history
> ---
>
> Key: SPARK-17243
> URL: https://issues.apache.org/jira/browse/SPARK-17243
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 2.0.0
> Environment: Linux
>Reporter: Gang Wu
>
> The summary page of Spark 2.0 history server web UI keep displaying "Loading
> history summary..." all the time and crashes the browser when there are more
> than 10K application history event logs on HDFS.
> I did some investigation, "historypage.js" file sends a REST request to
> /api/v1/applications endpoint of history server REST endpoint and gets back
> json response. When there are more than 10K applications inside the event log
> directory it takes forever to parse them and render the page. When there are
> only hundreds or thousands of application history it is running fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437682#comment-15437682 ] Alex Bozarth commented on SPARK-17243: -- [~wgtmac] until this is fixed you can limit the number of applications available by setting `spark.history.retainedApplications` It limits the apps the history server loads > Spark 2.0 history server summary page gets stuck at "loading history summary" > with 10K+ application history > --- > > Key: SPARK-17243 > URL: https://issues.apache.org/jira/browse/SPARK-17243 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.0 > Environment: Linux >Reporter: Gang Wu > > The summary page of Spark 2.0 history server web UI keep displaying "Loading > history summary..." all the time and crashes the browser when there are more > than 10K application history event logs on HDFS. > I did some investigation, "historypage.js" file sends a REST request to > /api/v1/applications endpoint of history server REST endpoint and gets back > json response. When there are more than 10K applications inside the event log > directory it takes forever to parse them and render the page. When there are > only hundreds or thousands of application history it is running fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437668#comment-15437668 ] Alex Bozarth edited comment on SPARK-17243 at 8/25/16 9:05 PM: --- I'm not sure I agree that this should be a blocker, but I was actually planning on filing a JIRA and starting work on a pr next month (September) that will switch the history server to only load application data when an application ui is opened and only loading application metadata on the initial load of the history server. This is just one of many problems that would be fixed by such a change. I won't have the bandwidth to start working on it for another week or two though. tl;dr I plan to fix this but not until next month was (Author: ajbozarth): I'm not sure I agree that this should be a blocker, but I was actually planning on filing a JIRA and starting work on a pr next month (September) that will switch the history server to only load application data when an application ui is opened and only loading application metadata on the initial load of the history server. This is just one of many problems that would be fixed by such a change. I won't have the bandwidth to start working on it for another week or two though. > Spark 2.0 history server summary page gets stuck at "loading history summary" > with 10K+ application history > --- > > Key: SPARK-17243 > URL: https://issues.apache.org/jira/browse/SPARK-17243 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.0 > Environment: Linux >Reporter: Gang Wu >Priority: Blocker > > The summary page of Spark 2.0 history server web UI keep displaying "Loading > history summary..." all the time and crashes the browser when there are more > than 10K application history event logs on HDFS. > I did some investigation, "historypage.js" file sends a REST request to > /api/v1/applications endpoint of history server REST endpoint and gets back > json response. When there are more than 10K applications inside the event log > directory it takes forever to parse them and render the page. When there are > only hundreds or thousands of application history it is running fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437668#comment-15437668 ] Alex Bozarth commented on SPARK-17243: -- I'm not sure I agree that this should be a blocker, but I was actually planning on filing a JIRA and starting work on a pr next month (September) that will switch the history server to only load application data when an application ui is opened and only loading application metadata on the initial load of the history server. This is just one of many problems that would be fixed by such a change. I won't have the bandwidth to start working on it for another week or two though. > Spark 2.0 history server summary page gets stuck at "loading history summary" > with 10K+ application history > --- > > Key: SPARK-17243 > URL: https://issues.apache.org/jira/browse/SPARK-17243 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.0 > Environment: Linux >Reporter: Gang Wu >Priority: Blocker > > The summary page of Spark 2.0 history server web UI keep displaying "Loading > history summary..." all the time and crashes the browser when there are more > than 10K application history event logs on HDFS. > I did some investigation, "historypage.js" file sends a REST request to > /api/v1/applications endpoint of history server REST endpoint and gets back > json response. When there are more than 10K applications inside the event log > directory it takes forever to parse them and render the page. When there are > only hundreds or thousands of application history it is running fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16501) spark.mesos.secret exposed on UI and command line
[ https://issues.apache.org/jira/browse/SPARK-16501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437539#comment-15437539 ] Alex Bozarth commented on SPARK-16501: -- The first problem (web ui) was fixed by https://github.com/apache/spark/pull/14484 as a follow up to SPARK-16796 > spark.mesos.secret exposed on UI and command line > - > > Key: SPARK-16501 > URL: https://issues.apache.org/jira/browse/SPARK-16501 > Project: Spark > Issue Type: Improvement > Components: Spark Submit, Web UI >Affects Versions: 1.6.2 >Reporter: Eric Daniel > Labels: security > > There are two related problems with spark.mesos.secret: > 1) The web UI shows its value in the "environment" tab > 2) Passing it as a command-line option to spark-submit (or creating a > SparkContext from python, with the effect of launching spark-submit) exposes > it to "ps" > I'll be happy to submit a patch but I could use some advice first. > The first problem is easy enough, just don't show that value in the UI > For the second problem, I'm not sure what the best solution is. A > "spark.mesos.secret-file" parameter would let the user store the secret in a > non-world-readable file. Alternatively, the mesos secret could be obtained > from the environment, which other users don't have access to. Either > solution would work in client mode, but I don't know if they're workable in > cluster mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16654) UI Should show blacklisted executors & nodes
[ https://issues.apache.org/jira/browse/SPARK-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423240#comment-15423240 ] Alex Bozarth commented on SPARK-16654: -- I don't have time right now to tackle this so go right ahead. And the other part of my comment was a implementation suggestion. We currently have a "status" column that lists either Alive or Dead. I'm suggesting that when shown, Blacklisted nodes are listed as Blacklisted or Alive (Blacklisted) in the status column, this would make the ui change for this very minimal to the user even though it'll be a good chunk of code to make it work behind the scenes. > UI Should show blacklisted executors & nodes > > > Key: SPARK-16654 > URL: https://issues.apache.org/jira/browse/SPARK-16654 > Project: Spark > Issue Type: Improvement > Components: Scheduler, Web UI >Affects Versions: 2.0.0 >Reporter: Imran Rashid > > SPARK-8425 will add the ability to blacklist entire executors and nodes to > deal w/ faulty hardware. However, without displaying it on the UI, it can be > hard to realize which executor is bad, and why tasks aren't getting scheduled > on certain executors. > As a first step, we should just show nodes and executors that are blacklisted > for the entire application (no need to show blacklisting for tasks & stages). > This should also ensure that blacklisting events get into the event logs for > the history server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16796) Visible passwords on Spark environment page
[ https://issues.apache.org/jira/browse/SPARK-16796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399893#comment-15399893 ] Alex Bozarth commented on SPARK-16796: -- [~asukhenko] if you don't want to go through the contribution process I can quickly do this, but I'd recommend going through the process your self since you already have the patch > Visible passwords on Spark environment page > --- > > Key: SPARK-16796 > URL: https://issues.apache.org/jira/browse/SPARK-16796 > Project: Spark > Issue Type: Improvement > Components: Web UI >Reporter: Artur >Priority: Trivial > Attachments: > Mask_spark_ssl_keyPassword_spark_ssl_keyStorePassword_spark_ssl_trustStorePassword_from_We1.patch > > > Spark properties (passwords): > spark.ssl.keyPassword,spark.ssl.keyStorePassword,spark.ssl.trustStorePassword > are visible in Web UI in environment page. > Can we mask them from Web UI? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden
[ https://issues.apache.org/jira/browse/SPARK-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388536#comment-15388536 ] Alex Bozarth commented on SPARK-16673: -- Once SPARK-16520 is settled I'm willing to tackle this > New Executor Page displays columns that used to be conditionally hidden > --- > > Key: SPARK-16673 > URL: https://issues.apache.org/jira/browse/SPARK-16673 > Project: Spark > Issue Type: Bug > Components: Web UI >Reporter: Alex Bozarth > > SPARK-15951 switched the Executors page to use JQuery DataTables, but it also > removed the functionality of conditionally hiding the Logs and Thread Dump > columns. In the case of the Logs column this is not a big issue, but in the > case of Thread Dump, previously it was never shown on the History Server > since it isn't available. > We should reintroduce the functionality to hide these columns according to > the same conditions as before. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden
[ https://issues.apache.org/jira/browse/SPARK-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388435#comment-15388435 ] Alex Bozarth commented on SPARK-16673: -- SPARK-16520 currently has a pr submitted that may also need this functionality > New Executor Page displays columns that used to be conditionally hidden > --- > > Key: SPARK-16673 > URL: https://issues.apache.org/jira/browse/SPARK-16673 > Project: Spark > Issue Type: Bug > Components: Web UI >Reporter: Alex Bozarth > > SPARK-15951 switched the Executors page to use JQuery DataTables, but it also > removed the functionality of conditionally hiding the Logs and Thread Dump > columns. In the case of the Logs column this is not a big issue, but in the > case of Thread Dump, previously it was never shown on the History Server > since it isn't available. > We should reintroduce the functionality to hide these columns according to > the same conditions as before. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden
Alex Bozarth created SPARK-16673: Summary: New Executor Page displays columns that used to be conditionally hidden Key: SPARK-16673 URL: https://issues.apache.org/jira/browse/SPARK-16673 Project: Spark Issue Type: Bug Components: Web UI Reporter: Alex Bozarth SPARK-15951 switched the Executors page to use JQuery DataTables, but it also removed the functionality of conditionally hiding the Logs and Thread Dump columns. In the case of the Logs column this is not a big issue, but in the case of Thread Dump, previously it was never shown on the History Server since it isn't available. We should reintroduce the functionality to hide these columns according to the same conditions as before. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16654) UI Should show blacklisted executors & nodes
[ https://issues.apache.org/jira/browse/SPARK-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386338#comment-15386338 ] Alex Bozarth commented on SPARK-16654: -- Perhaps we can change the status column to "Blacklisted" or "Alive (Blacklisted)" instead on Alive or Dead? I'm not very familiar with how the blacklisting works but I would be willing to learn and add this once the other PR is merged. > UI Should show blacklisted executors & nodes > > > Key: SPARK-16654 > URL: https://issues.apache.org/jira/browse/SPARK-16654 > Project: Spark > Issue Type: Improvement > Components: Scheduler, Web UI >Affects Versions: 2.0.0 >Reporter: Imran Rashid > > SPARK-8425 will add the ability to blacklist entire executors and nodes to > deal w/ faulty hardware. However, without displaying it on the UI, it can be > hard to realize which executor is bad, and why tasks aren't getting scheduled > on certain executors. > As a first step, we should just show nodes and executors that are blacklisted > for the entire application (no need to show blacklisting for tasks & stages). > This should also ensure that blacklisting events get into the event logs for > the history server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-16047) Sort by status and id fields in Executors table
[ https://issues.apache.org/jira/browse/SPARK-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Bozarth updated SPARK-16047: - Attachment: sorted_alive_first.png I agree with Sean, I actually made the code change for this to look at it and it makes the table feel unsorted in my opinion. I've attached a screenshot of how it'd look with the suggested change. > Sort by status and id fields in Executors table > --- > > Key: SPARK-16047 > URL: https://issues.apache.org/jira/browse/SPARK-16047 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 2.0.0 >Reporter: Jacek Laskowski >Priority: Minor > Attachments: sorted_alive_first.png, spark-webui-executors.png > > > With multiple executors with the same ID the default sorting *seems* to be by > ID (descending) first and status (alphabetically ascending). > I'd like webUI to sort the Executors table by status first (with Active > first) followed by ID (ascending with driver being the last one). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15868) Executors table in Executors tab should sort Executor IDs in numerical order (not alphabetical order)
[ https://issues.apache.org/jira/browse/SPARK-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325103#comment-15325103 ] Alex Bozarth commented on SPARK-15868: -- I've run into this before and I thought I had fixed it in another JIRA, I'll take care of this. > Executors table in Executors tab should sort Executor IDs in numerical order > (not alphabetical order) > - > > Key: SPARK-15868 > URL: https://issues.apache.org/jira/browse/SPARK-15868 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 2.0.0 >Reporter: Jacek Laskowski >Priority: Minor > Attachments: spark-webui-executors-sorting-2.png, > spark-webui-executors-sorting.png > > > It _appears_ that Executors table in Executors tab sorts Executor IDs in > alphabetical order while it should in numerical. It does sorting in a more > "friendly" way yet driver executor appears between 0 and 1? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-11516) Spark application cannot be found from JSON API even though it exists
[
https://issues.apache.org/jira/browse/SPARK-11516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Bozarth closed SPARK-11516.
Resolution: Workaround
Closing as Workaround since turning on eventLogging fixes this
> Spark application cannot be found from JSON API even though it exists
> -
>
> Key: SPARK-11516
> URL: https://issues.apache.org/jira/browse/SPARK-11516
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 1.4.1, 1.5.1
>Reporter: Matt Cheah
>
> I'm running a Spark standalone cluster on my mac and playing with the JSON
> API. I start both the master and the worker daemons, then start a Spark shell.
> When I hit
> {code}
> http://localhost:8080/api/v1/applications
> {code}
> I get back
> {code}
> [ {
> "id" : "app-20151104181347-",
> "name" : "Spark shell",
> "attempts" : [ {
> "startTime" : "2015-11-05T02:13:47.980GMT",
> "endTime" : "1969-12-31T23:59:59.999GMT",
> "sparkUser" : "mcheah",
> "completed" : false
> } ]
> } ]
> {code}
> But when I hit
> {code}
> http://localhost:8080/api/v1/applications/app-20151104181347-/executors
> {code}
> To look for executor data for the job, I get back
> {code}
> no such app: app-20151104181347-
> {code}
> even though the application exists.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1301) Add UI elements to collapse "Aggregated Metrics by Executor" pane on stage page
[ https://issues.apache.org/jira/browse/SPARK-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279188#comment-15279188 ] Alex Bozarth commented on SPARK-1301: - Submitted a small fix similar to Ryan's above but in line with how it's done on the Jobs and Stages page > Add UI elements to collapse "Aggregated Metrics by Executor" pane on stage > page > --- > > Key: SPARK-1301 > URL: https://issues.apache.org/jira/browse/SPARK-1301 > Project: Spark > Issue Type: Improvement > Components: Web UI >Reporter: Matei Zaharia >Priority: Minor > Labels: Starter > > This table is useful but it takes up a lot of space on larger clusters, > hiding the more commonly accessed "stage" page. We could also move the table > below if collapsing it is difficult. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15220) Add hyperlink to "running application" and "completed application"
[ https://issues.apache.org/jira/browse/SPARK-15220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15276687#comment-15276687 ] Alex Bozarth commented on SPARK-15220: -- I'll take a look at this in my free moments today, seems quick > Add hyperlink to "running application" and "completed application" > -- > > Key: SPARK-15220 > URL: https://issues.apache.org/jira/browse/SPARK-15220 > Project: Spark > Issue Type: Improvement > Components: Web UI >Reporter: Mao, Wei >Priority: Minor > > Add hyperlink to "running application" and "completed application", so user > can jump to application table directly, In my environment, I set up 1000+ > works and it's painful to scroll down to skip worker list. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-10653) Remove unnecessary things from SparkEnv
[
https://issues.apache.org/jira/browse/SPARK-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15273117#comment-15273117
]
Alex Bozarth commented on SPARK-10653:
--
I'm currently running tests on a fix for this and will open a PR after. I have
removed blockTransferService and sparkFilesDir and replaced the few references
to them. ExecutorMemoryManager was already removed in SPARK-10984. I also took
a quick look at the other vals in the constructor and I didn't see any other
low hanging fruit to remove.
> Remove unnecessary things from SparkEnv
> ---
>
> Key: SPARK-10653
> URL: https://issues.apache.org/jira/browse/SPARK-10653
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.0.0
>Reporter: Andrew Or
>
> As of the writing of this message, there are at least two things that can be
> removed from it:
> {code}
> @DeveloperApi
> class SparkEnv (
> val executorId: String,
> private[spark] val rpcEnv: RpcEnv,
> val serializer: Serializer,
> val closureSerializer: Serializer,
> val cacheManager: CacheManager,
> val mapOutputTracker: MapOutputTracker,
> val shuffleManager: ShuffleManager,
> val broadcastManager: BroadcastManager,
> val blockTransferService: BlockTransferService, // this one can go
> val blockManager: BlockManager,
> val securityManager: SecurityManager,
> val httpFileServer: HttpFileServer,
> val sparkFilesDir: String, // this one maybe? It's only used in 1 place.
> val metricsSystem: MetricsSystem,
> val shuffleMemoryManager: ShuffleMemoryManager,
> val executorMemoryManager: ExecutorMemoryManager, // this can go
> val outputCommitCoordinator: OutputCommitCoordinator,
> val conf: SparkConf) extends Logging {
> ...
> }
> {code}
> We should avoid adding to this infinite list of things in SparkEnv's
> constructors if they're not needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-10653) Remove unnecessary things from SparkEnv
[
https://issues.apache.org/jira/browse/SPARK-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Bozarth updated SPARK-10653:
-
Summary: Remove unnecessary things from SparkEnv (was: head)
> Remove unnecessary things from SparkEnv
> ---
>
> Key: SPARK-10653
> URL: https://issues.apache.org/jira/browse/SPARK-10653
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.0.0
>Reporter: Andrew Or
>
> As of the writing of this message, there are at least two things that can be
> removed from it:
> {code}
> @DeveloperApi
> class SparkEnv (
> val executorId: String,
> private[spark] val rpcEnv: RpcEnv,
> val serializer: Serializer,
> val closureSerializer: Serializer,
> val cacheManager: CacheManager,
> val mapOutputTracker: MapOutputTracker,
> val shuffleManager: ShuffleManager,
> val broadcastManager: BroadcastManager,
> val blockTransferService: BlockTransferService, // this one can go
> val blockManager: BlockManager,
> val securityManager: SecurityManager,
> val httpFileServer: HttpFileServer,
> val sparkFilesDir: String, // this one maybe? It's only used in 1 place.
> val metricsSystem: MetricsSystem,
> val shuffleMemoryManager: ShuffleMemoryManager,
> val executorMemoryManager: ExecutorMemoryManager, // this can go
> val outputCommitCoordinator: OutputCommitCoordinator,
> val conf: SparkConf) extends Logging {
> ...
> }
> {code}
> We should avoid adding to this infinite list of things in SparkEnv's
> constructors if they're not needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-10653) head
[
https://issues.apache.org/jira/browse/SPARK-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Bozarth updated SPARK-10653:
-
Summary: head (was: Remove unnecessary things from SparkEnv)
> head
>
>
> Key: SPARK-10653
> URL: https://issues.apache.org/jira/browse/SPARK-10653
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.0.0
>Reporter: Andrew Or
>
> As of the writing of this message, there are at least two things that can be
> removed from it:
> {code}
> @DeveloperApi
> class SparkEnv (
> val executorId: String,
> private[spark] val rpcEnv: RpcEnv,
> val serializer: Serializer,
> val closureSerializer: Serializer,
> val cacheManager: CacheManager,
> val mapOutputTracker: MapOutputTracker,
> val shuffleManager: ShuffleManager,
> val broadcastManager: BroadcastManager,
> val blockTransferService: BlockTransferService, // this one can go
> val blockManager: BlockManager,
> val securityManager: SecurityManager,
> val httpFileServer: HttpFileServer,
> val sparkFilesDir: String, // this one maybe? It's only used in 1 place.
> val metricsSystem: MetricsSystem,
> val shuffleMemoryManager: ShuffleMemoryManager,
> val executorMemoryManager: ExecutorMemoryManager, // this can go
> val outputCommitCoordinator: OutputCommitCoordinator,
> val conf: SparkConf) extends Logging {
> ...
> }
> {code}
> We should avoid adding to this infinite list of things in SparkEnv's
> constructors if they're not needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11516) Spark application cannot be found from JSON API even though it exists
[
https://issues.apache.org/jira/browse/SPARK-11516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15272721#comment-15272721
]
Alex Bozarth commented on SPARK-11516:
--
Looking through some older issues, what Charles said was right, any JSON API
calls other than the /api/v1/applications call require eventLogging to be
enabled. To change this would require significant refactoring, if enabling
eventLogging was all you needed then I'd suggest closing this.
> Spark application cannot be found from JSON API even though it exists
> -
>
> Key: SPARK-11516
> URL: https://issues.apache.org/jira/browse/SPARK-11516
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 1.4.1, 1.5.1
>Reporter: Matt Cheah
>
> I'm running a Spark standalone cluster on my mac and playing with the JSON
> API. I start both the master and the worker daemons, then start a Spark shell.
> When I hit
> {code}
> http://localhost:8080/api/v1/applications
> {code}
> I get back
> {code}
> [ {
> "id" : "app-20151104181347-",
> "name" : "Spark shell",
> "attempts" : [ {
> "startTime" : "2015-11-05T02:13:47.980GMT",
> "endTime" : "1969-12-31T23:59:59.999GMT",
> "sparkUser" : "mcheah",
> "completed" : false
> } ]
> } ]
> {code}
> But when I hit
> {code}
> http://localhost:8080/api/v1/applications/app-20151104181347-/executors
> {code}
> To look for executor data for the job, I get back
> {code}
> no such app: app-20151104181347-
> {code}
> even though the application exists.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13269) Expose more executor stats in stable status API
[ https://issues.apache.org/jira/browse/SPARK-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269738#comment-15269738 ] Alex Bozarth commented on SPARK-13269: -- Hey [~andrewor14], I was interested in this and took a look at the two examples you gave and am a bit confused at what exactly you actually want. You can currently see the used memory, used disk space, and active task count for each executor by calling /applications/[app-id]/executors or (in code) getting the ExecutorSummary class for each executor and checking activeTask, memoryUsed, and diskUsed. Are these numbers different from what you were interested in surfacing? I was unsure of how those example related to JobProgressListener as well. > Expose more executor stats in stable status API > --- > > Key: SPARK-13269 > URL: https://issues.apache.org/jira/browse/SPARK-13269 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Andrew Or > > Currently the stable status API is quite limited; it exposes only a small > subset of the things exposed by JobProgressListener. It is useful for very > high level querying but falls short when the developer wants to build an > application on top of Spark with more integration. > In this issue I propose that we expose at least two things: > - Which executors are running tasks, and > - Which executors cached how much in memory and on disk > The goal is not to expose exactly these two things, but to expose something > that would allow the developer to learn about them. These concepts are very > much fundamental in Spark's design so there's almost no chance that they will > go away in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14750) Make historyServer refer application log in hdfs
[ https://issues.apache.org/jira/browse/SPARK-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15252907#comment-15252907 ] Alex Bozarth commented on SPARK-14750: -- I looked into this a bit as well as looked at you PR (I left PR related comments there) and am wondering what the issue exactly is, I don't use yarn much myself but in my tests I sometimes see the log files surfaced through the Hadoop UI, and sometimes I see the same error as you. As for the stand-alone history server, the logs are available in the same location in the UI as in the Web UI during a running application. It seems the solution here is to find out why the logs are not surfaced in this use case and to address that (may actually be a bug). > Make historyServer refer application log in hdfs > > > Key: SPARK-14750 > URL: https://issues.apache.org/jira/browse/SPARK-14750 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: SuYan > > Make history server refer application log, just like MR history server -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8171) Support Javascript-based infinite scrolling in Spark log viewers
[ https://issues.apache.org/jira/browse/SPARK-8171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250197#comment-15250197 ] Alex Bozarth commented on SPARK-8171: - [~joshrosen] I was able to resolve this, but I can't seem to set myself as the Assignee > Support Javascript-based infinite scrolling in Spark log viewers > > > Key: SPARK-8171 > URL: https://issues.apache.org/jira/browse/SPARK-8171 > Project: Spark > Issue Type: New Feature > Components: Web UI >Reporter: Josh Rosen > > It would be cool if the Spark Web UI's log viewers supported infinite > scrolling so that I can just click / scroll up to go back in the log. Maybe > there's an off-the-shelf Javascript component for this. > See SPARK-608, where the log viewer pagination was first introduced. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-8171) Support Javascript-based infinite scrolling in Spark log viewers
[ https://issues.apache.org/jira/browse/SPARK-8171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Bozarth resolved SPARK-8171. - Resolution: Fixed > Support Javascript-based infinite scrolling in Spark log viewers > > > Key: SPARK-8171 > URL: https://issues.apache.org/jira/browse/SPARK-8171 > Project: Spark > Issue Type: New Feature > Components: Web UI >Reporter: Josh Rosen > > It would be cool if the Spark Web UI's log viewers supported infinite > scrolling so that I can just click / scroll up to go back in the log. Maybe > there's an off-the-shelf Javascript component for this. > See SPARK-608, where the log viewer pagination was first introduced. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15224308#comment-15224308 ] Alex Bozarth commented on SPARK-14245: -- Thank you for that info, when I first started working on Spark I was originally given the impression that Spark Jira's weren't assigned until a pr was merged. > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves >Assignee: Alex Bozarth > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15222702#comment-15222702 ] Alex Bozarth commented on SPARK-14245: -- I don't know what you mean by "real" application submitter. My change shows the same user listed in the active/completed application lists on the driver Web UI, which is whichever system user the spark app is run by. > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15218742#comment-15218742 ] Alex Bozarth commented on SPARK-14245: -- I'm looking into this in my free moments today, will probably have something in the next day or two > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-12713) UI Executor page should keep links around to executors that died
[ https://issues.apache.org/jira/browse/SPARK-12713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163718#comment-15163718 ] Alex Bozarth commented on SPARK-12713: -- SPARK-7729 has fixed this > UI Executor page should keep links around to executors that died > > > Key: SPARK-12713 > URL: https://issues.apache.org/jira/browse/SPARK-12713 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.5.2 >Reporter: Thomas Graves > > When an executor dies the web ui no longer shows it in the executors page > which makes getting to the logs to see what happened very difficult. I'm > running on yarn so not sure if behavior is different in standalone mode. > We should figure out a way to keep links around to the ones that died so we > can show stats and log links. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8172) Driver UI should enable viewing of dead executors' logs
[ https://issues.apache.org/jira/browse/SPARK-8172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15163717#comment-15163717 ] Alex Bozarth commented on SPARK-8172: - I believe this has been fixed by SPARK-7729 > Driver UI should enable viewing of dead executors' logs > --- > > Key: SPARK-8172 > URL: https://issues.apache.org/jira/browse/SPARK-8172 > Project: Spark > Issue Type: New Feature > Components: Web UI >Reporter: Josh Rosen > > If possible, the Spark driver UI's executor page should include a list of > dead executors (perhaps of bounded size) and should have log viewer links for > viewing those dead executors' logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13459) Separate Alive and Dead Executors in Executor Totals Table
[ https://issues.apache.org/jira/browse/SPARK-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159657#comment-15159657 ] Alex Bozarth commented on SPARK-13459: -- That PR isn't for this Jira, I mixed up my Jira numbers while opening a PR for the other Jira I was working on > Separate Alive and Dead Executors in Executor Totals Table > -- > > Key: SPARK-13459 > URL: https://issues.apache.org/jira/browse/SPARK-13459 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 2.0.0 >Reporter: Alex Bozarth >Priority: Minor > > Now that dead executors are shown in the executors table (SPARK-7729) the > totals table added in SPARK-12716 should be updated to include the separate > totals for alive and dead executors as well as the current total. > (This improvement was originally discussed in the PR for SPARK-12716 while > SPARK-7729 was still in progress.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13459) Separate Alive and Dead Executors in Executor Totals Table
[ https://issues.apache.org/jira/browse/SPARK-13459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15159553#comment-15159553 ] Alex Bozarth commented on SPARK-13459: -- I am currently working on this > Separate Alive and Dead Executors in Executor Totals Table > -- > > Key: SPARK-13459 > URL: https://issues.apache.org/jira/browse/SPARK-13459 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 2.0.0 >Reporter: Alex Bozarth >Priority: Minor > > Now that dead executors are shown in the executors table (SPARK-7729) the > totals table added in SPARK-12716 should be updated to include the separate > totals for alive and dead executors as well as the current total. > (This improvement was originally discussed in the PR for SPARK-12716 while > SPARK-7729 was still in progress.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-13459) Separate Alive and Dead Executors in Executor Totals Table
Alex Bozarth created SPARK-13459: Summary: Separate Alive and Dead Executors in Executor Totals Table Key: SPARK-13459 URL: https://issues.apache.org/jira/browse/SPARK-13459 Project: Spark Issue Type: Improvement Components: Web UI Affects Versions: 2.0.0 Reporter: Alex Bozarth Priority: Minor Now that dead executors are shown in the executors table (SPARK-7729) the totals table added in SPARK-12716 should be updated to include the separate totals for alive and dead executors as well as the current total. (This improvement was originally discussed in the PR for SPARK-12716 while SPARK-7729 was still in progress.) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15158069#comment-15158069
]
Alex Bozarth commented on SPARK-13241:
--
I have a fix for this in the works, PR will come sometime tomorrow. We can
continue on discussion there unless someone wants to veto this now
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-13298) DAG visualization does not render correctly for jobs
[
https://issues.apache.org/jira/browse/SPARK-13298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Bozarth updated SPARK-13298:
-
Attachment: no-dag.png
Using your reproducer I saw the same js error but the dag image was blank, and
removing cache() didn't change the result for me.
!no-dag.png!
(Tested on Safari 9.0.3, Firefox ESR 38.6.1, Chrome 48.0.2564.116)
I don't have time now, but if no one picks this up I may come back to this
later.
> DAG visualization does not render correctly for jobs
>
>
> Key: SPARK-13298
> URL: https://issues.apache.org/jira/browse/SPARK-13298
> Project: Spark
> Issue Type: Bug
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Lucas Woltmann
> Attachments: dag_full.png, dag_viz.png, no-dag.png
>
>
> Whenever I try to open the DAG for a job, I get something like this:
> !dag_viz.png!
> Obviously the svg doesn't get resized, but if I resize it manually, only the
> first of four stages in the DAG is shown.
> The js console says (variable v is null in peg$c34):
> {code:javascript}
> Uncaught TypeError: Cannot read property '3' of null
> peg$c34 @ graphlib-dot.min.js:1
> peg$parseidDef @ graphlib-dot.min.js:1
> peg$parseaList @ graphlib-dot.min.js:1
> peg$parseattrListBlock @ graphlib-dot.min.js:1
> peg$parseattrList @ graphlib-dot.min.js:1
> peg$parsenodeStmt @ graphlib-dot.min.js:1
> peg$parsestmt @ graphlib-dot.min.js:1
> peg$parsestmtList @ graphlib-dot.min.js:1
> peg$parsesubgraphStmt @ graphlib-dot.min.js:1
> peg$parsenodeIdOrSubgraph @ graphlib-dot.min.js:1
> peg$parseedgeStmt @ graphlib-dot.min.js:1
> peg$parsestmt @ graphlib-dot.min.js:1
> peg$parsestmtList @ graphlib-dot.min.js:1
> peg$parsesubgraphStmt @ graphlib-dot.min.js:1
> peg$parsenodeIdOrSubgraph @ graphlib-dot.min.js:1
> peg$parseedgeStmt @ graphlib-dot.min.js:1
> peg$parsestmt @ graphlib-dot.min.js:1
> peg$parsestmtList @ graphlib-dot.min.js:1
> peg$parsegraphStmt @ graphlib-dot.min.js:1
> parse @ graphlib-dot.min.js:2
> readOne @ graphlib-dot.min.js:2
> renderDot @ spark-dag-viz.js:281
> (anonymous function) @ spark-dag-viz.js:248
> (anonymous function) @ d3.min.js:
> 3Y @ d3.min.js:1
> _a.each @ d3.min.js:3
> renderDagVizForJob @ spark-dag-viz.js:207
> renderDagViz @ spark-dag-viz.js:163
> toggleDagViz @ spark-dag-viz.js:100
> onclick @ ?id=2:153
> {code}
> (tested in FIrefox 44.0.1 and Chromium 48.0.2564.103)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153677#comment-15153677
]
Alex Bozarth commented on SPARK-13241:
--
Ok, that makes sense, I've made parameter additions to ExecutorSummary in the
API previously but I believe those only affected the web ui and not external
use cases.
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153671#comment-15153671
]
Alex Bozarth commented on SPARK-13241:
--
I'm still relatively new to this, but wouldn't we just need to update all the
internal calls to that class to include the new fields? Or does adding them
fundamentally break the usage of it in ways I haven't learned?
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153040#comment-15153040
]
Alex Bozarth edited comment on SPARK-13241 at 2/18/16 8:38 PM:
---
Looked into this and I could update the api to include this, but before I go
through the leg work, do we actually want it?
[~joshrosen] [~rxin] since you're in charge of the API, is adding the date
params as {{long}} s in {{ApplicationAttemptInfo}} something we actually want
to do?
It's definitely useful, but the duplicate data clusters up the api a bit.
was (Author: ajbozarth):
Looked into this and I could update the api to include this, but before I go
through the leg work, do we actually want it?
[~joshrosen] [~rxin] since you're in charge of the API, is adding the date
params as {{long}}s in {{ApplicationAttemptInfo}} something we actually want to
do?
It's definitely useful, but the duplicate data clusters up the api a bit.
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13241) add long--formatted timestamps to org.apache.spark.status.api.v1.ApplicationAttemptInfo
[
https://issues.apache.org/jira/browse/SPARK-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153040#comment-15153040
]
Alex Bozarth commented on SPARK-13241:
--
Looked into this and I could update the api to include this, but before I go
through the leg work, do we actually want it?
[~joshrosen] [~rxin] since you're in charge of the API, is adding the date
params as {{long}}s in {{ApplicationAttemptInfo}} something we actually want to
do?
It's definitely useful, but the duplicate data clusters up the api a bit.
> add long--formatted timestamps to
> org.apache.spark.status.api.v1.ApplicationAttemptInfo
> ---
>
> Key: SPARK-13241
> URL: https://issues.apache.org/jira/browse/SPARK-13241
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
>Affects Versions: 1.6.0
>Reporter: Steve Loughran
>
> While writing tests to parse
> {{org.apache.spark.status.api.v1.ApplicationAttemptInfo}} coming off the
> history rest server, I can see that I can't actually unmarshall the
> timestamps without parsing the strings —and I fear that may be local specific.
> The problem here is that jersey is marshalling {{Date}} classes by calling
> {{Date.toString()}}, leaving an entry like
> {code}
> {
> "id": "application__",
> "name": "spark-demo",
> "attempts": [
> {
> "attemptId": "1",
> "startTime": "2016-02-08T20:12:20.825GMT",
> "endTime": "1970-01-01T00:00:00.000GMT",
> "lastUpdated": "2016-02-08T20:12:20.848GMT",
> "duration": 0,
> "sparkUser": "hdfs",
> "completed": false
> }
> ]
> }
> {code}
> This is good for humans and the web UI, bad for any code trying to use the
> API for its own purposes.
> I propose adding, alongside the existing values, a simple {{long}} value for
> each time, representing the time in millis from the start of the epoch. This
> is trivial to marshall/unmarshall.
> This is easy to add, provided everyone is happy that adding a new field to
> the returned JSON is considered compatible with the existing REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-13163) Column width on new History Server DataTables not getting set correctly
[ https://issues.apache.org/jira/browse/SPARK-13163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Bozarth updated SPARK-13163: - Attachment: width_long_name.png page_width_fixed.png I have a fix and will open a PR > Column width on new History Server DataTables not getting set correctly > --- > > Key: SPARK-13163 > URL: https://issues.apache.org/jira/browse/SPARK-13163 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.0 >Reporter: Alex Bozarth >Priority: Minor > Attachments: page_width_fixed.png, width_long_name.png > > > The column width on the DataTable UI for the History Server is being set for > all entries in the table not just the current page. This means if there is > even one App with a long name in your history the table will look really odd > as seen below. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
