[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI
[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530562#comment-14530562 ] Apache Spark commented on SPARK-3454: - User 'squito' has created a pull request for this issue: https://github.com/apache/spark/pull/5940 Expose JSON representation of data shown in WebUI - Key: SPARK-3454 URL: https://issues.apache.org/jira/browse/SPARK-3454 Project: Spark Issue Type: Improvement Components: Web UI Affects Versions: 1.1.0 Reporter: Kousuke Saruta Assignee: Imran Rashid Fix For: 1.4.0 Attachments: sparkmonitoringjsondesign.pdf If WebUI support to JSON format extracting, it's helpful for user who want to analyse stage / task / executor information. Fortunately, WebUI has renderJson method so we can implement the method in each subclass. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI
[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14309969#comment-14309969 ] Apache Spark commented on SPARK-3454: - User 'squito' has created a pull request for this issue: https://github.com/apache/spark/pull/4435 Expose JSON representation of data shown in WebUI - Key: SPARK-3454 URL: https://issues.apache.org/jira/browse/SPARK-3454 Project: Spark Issue Type: Improvement Components: Web UI Affects Versions: 1.1.0 Reporter: Kousuke Saruta Attachments: sparkmonitoringjsondesign.pdf If WebUI support to JSON format extracting, it's helpful for user who want to analyse stage / task / executor information. Fortunately, WebUI has renderJson method so we can implement the method in each subclass. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI
[
https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14308324#comment-14308324
]
Marcelo Vanzin commented on SPARK-3454:
---
Hi [~imranr],
There are two ways I can see to solve the routing problem:
The first is the one you mention. I like it because, as you say, it keeps the
API consistent across different UIs. You always look at /json, not some
subpath that depends on which daemon you're looking at.
The second is to remove the notion of an application list from this spec.
That means the /json tree would be mounted under the application's path, not
at the root of the web server. The downside is that when you add an API to list
applications to the master / history server, things will look weird (you have
/json/v1/applications and /{applicationId}/json/v1 instead of a single
tree). Clients would have to adapt depending on whether they're talking to an
app directly, or to the master / history server.
So yeah, I like your suggestion better.
Expose JSON representation of data shown in WebUI
-
Key: SPARK-3454
URL: https://issues.apache.org/jira/browse/SPARK-3454
Project: Spark
Issue Type: Improvement
Components: Web UI
Affects Versions: 1.1.0
Reporter: Kousuke Saruta
Attachments: sparkmonitoringjsondesign.pdf
If WebUI support to JSON format extracting, it's helpful for user who want to
analyse stage / task / executor information.
Fortunately, WebUI has renderJson method so we can implement the method in
each subclass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI
[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14300834#comment-14300834 ] Ryan Williams commented on SPARK-3454: -- I think it looks great, [~imranr]! Is the logical next step to flesh out exactly what the new POJOs will be? Expose JSON representation of data shown in WebUI - Key: SPARK-3454 URL: https://issues.apache.org/jira/browse/SPARK-3454 Project: Spark Issue Type: Improvement Components: Web UI Affects Versions: 1.1.0 Reporter: Kousuke Saruta Attachments: sparkmonitoringjsondesign.pdf If WebUI support to JSON format extracting, it's helpful for user who want to analyse stage / task / executor information. Fortunately, WebUI has renderJson method so we can implement the method in each subclass. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI
[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14298747#comment-14298747 ] Imran Rashid commented on SPARK-3454: - design doc attached, would love any feedback Expose JSON representation of data shown in WebUI - Key: SPARK-3454 URL: https://issues.apache.org/jira/browse/SPARK-3454 Project: Spark Issue Type: Improvement Components: Web UI Affects Versions: 1.1.0 Reporter: Kousuke Saruta Attachments: sparkmonitoringjsondesign.pdf If WebUI support to JSON format extracting, it's helpful for user who want to analyse stage / task / executor information. Fortunately, WebUI has renderJson method so we can implement the method in each subclass. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI
[
https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294311#comment-14294311
]
Imran Rashid commented on SPARK-3454:
-
There are a few ways we could move forward on this.
1. Its pretty easy to have the UI generate some json -- we just need to
implement {{renderJson}} in a bunch of the {{Page}} classes. sarutak's PR came
pretty close to doing this. I have a branch where I merged it w/ master and
filled in a few holes. However, this approach has the downside that we may end
up creating some new semi-public, semi-supported api in the schema for whatever
json we output. Since we're building it on top of the UI layer, it might be
hard to really guarantee we can keep it as a stable api.
2. We could extend {{SparkStatusTracker}} to cover a lot more, and then build
up everything this on top of those events. We'd need to add more events,
especially task metrics storage info, but probably everything in the UI.
Since we want the json available from the {{HistoryServer}} as well, we might
need to change around the logging to be compatible with {{SparkStatusTracker}}
(and whatever we add to it).
Even the UI could be made a consumer of the json output, as a way to (a) unify
some of the logic in the logging UI (b) make the json api a true
first-class api, since we'd be consuming it ourselves.
Expose JSON representation of data shown in WebUI
-
Key: SPARK-3454
URL: https://issues.apache.org/jira/browse/SPARK-3454
Project: Spark
Issue Type: Improvement
Components: Web UI
Affects Versions: 1.1.0
Reporter: Kousuke Saruta
If WebUI support to JSON format extracting, it's helpful for user who want to
analyse stage / task / executor information.
Fortunately, WebUI has renderJson method so we can implement the method in
each subclass.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI
[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294348#comment-14294348 ] Ryan Williams commented on SPARK-3454: -- +1 to making the web UI a consumer of a JSON representation of metrics we are exposing more broadly than to just the web UI! For the reasons you've mentioned at the end. Expose JSON representation of data shown in WebUI - Key: SPARK-3454 URL: https://issues.apache.org/jira/browse/SPARK-3454 Project: Spark Issue Type: Improvement Components: Web UI Affects Versions: 1.1.0 Reporter: Kousuke Saruta If WebUI support to JSON format extracting, it's helpful for user who want to analyse stage / task / executor information. Fortunately, WebUI has renderJson method so we can implement the method in each subclass. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3454) Expose JSON representation of data shown in WebUI
[ https://issues.apache.org/jira/browse/SPARK-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126854#comment-14126854 ] Apache Spark commented on SPARK-3454: - User 'sarutak' has created a pull request for this issue: https://github.com/apache/spark/pull/2333 Expose JSON representation of data shown in WebUI - Key: SPARK-3454 URL: https://issues.apache.org/jira/browse/SPARK-3454 Project: Spark Issue Type: Improvement Components: Web UI Affects Versions: 1.1.0 Reporter: Kousuke Saruta If WebUI support to JSON format extracting, it's helpful for user who want to analyse stage / task / executor information. Fortunately, WebUI has renderJson method so we can implement the method in each subclass. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
