[
https://issues.apache.org/jira/browse/SPARK-3644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160183#comment-14160183
]
Daniel Darabos commented on SPARK-3644:
---------------------------------------
Notes from another consumer with a different use-case: We have an analytics
program with a web UI and a backend that uses Spark. For us the following
features of the REST API are important:
- Most importantly, be able to tell if Spark is idle or not. In other words,
get a list of active stages.
- Track the progress of a stage. A calculation can take any number of stages,
so we cannot make a very good progress indicator, but this would still tell the
user if there is any sort of progress being made. I think your pull request
covers this.
- Kill a stage, or all active stages. I think this feature already exists.
With this we could add a small UI element that queries the Spark status by an
XHR request and gives our users some visibility into the system. (Sure, they
could just go to port 4040, but that's too complex for them.)
Thanks for working on this!
> REST API for Spark application info (jobs / stages / tasks / storage info)
> --------------------------------------------------------------------------
>
> Key: SPARK-3644
> URL: https://issues.apache.org/jira/browse/SPARK-3644
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Web UI
> Reporter: Josh Rosen
>
> This JIRA is a forum to draft a design proposal for a REST interface for
> accessing information about Spark applications, such as job / stage / task /
> storage status.
> There have been a number of proposals to serve JSON representations of the
> information displayed in Spark's web UI. Given that we might redesign the
> pages of the web UI (and possibly re-implement the UI as a client of a REST
> API), the API endpoints and their responses should be independent of what we
> choose to display on particular web UI pages / layouts.
> Let's start a discussion of what a good REST API would look like from
> first-principles. We can discuss what urls / endpoints expose access to
> data, how our JSON responses will be formatted, how fields will be named, how
> the API will be documented and tested, etc.
> Some links for inspiration:
> https://developer.github.com/v3/
> http://developer.netflix.com/docs/REST_API_Reference
> https://helloreverb.com/developers/swagger
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
