Thank you all for the feedback. I’ve created a corresponding JIRA ticket at https://issues.apache.org/jira/browse/SPARK-10565, updated with a summary of this thread.
From: Mark Hamstra <m...@clearstorydata.com> Date: Thursday, September 17, 2015 at 8:00 AM To: Imran Rashid <iras...@cloudera.com> Cc: Kevin Chen <kc...@palantir.com>, "dev@spark.apache.org" <dev@spark.apache.org>, Matt Cheah <mch...@palantir.com>, Mingyu Kim <m...@palantir.com> Subject: Re: New Spark json endpoints While we're at it, adding endpoints that get results by jobGroup (cf. SparkContext#setJobGroup) instead of just for a single Job would also be very useful to some of us. On Thu, Sep 17, 2015 at 7:30 AM, Imran Rashid <iras...@cloudera.com> wrote: > Hi Kevin, > > I think it would be great if you added this. It never got added in the first > place b/c the original PR was already pretty bloated, and just never got back > to this. I agree with Reynold -- you shouldn't need to increase the version > for just adding new endpoints (or even adding new fields to existing > endpoints). See the guarantees we make here: > > http://spark.apache.org/docs/latest/monitoring.html#rest-api > <https://urldefense.proofpoint.com/v2/url?u=http-3A__spark.apache.org_docs_lat > est_monitoring.html-23rest-2Dapi&d=BQMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4t > Fb6oOnmz8&r=LsE5OY2DgDurlIyKnSpM6-DfP7v9PXYudc_RKz6BHxk&m=9LefIyuh9-7WnXvvb-5y > xOvm0d8y18FKNG7D834_kNA&s=Ov1ONemrf4hSALvJBLRhykzEIhhb0ZijAFQ6zdlunQk&e=> > > (Though if you think we should make different guarantees around versions, that > would be worth discussing as well.) > > Can you file a jira, and we move discussion there? Please cc me, and maybe > also Josh Rosen (I'm not sure if he has cycles now but he's been very helpful > on these issues in the past). > > thanks, > Imran > > > On Wed, Sep 16, 2015 at 9:10 PM, Kevin Chen <kc...@palantir.com> wrote: >> Just wanted to bring this email up again in case there were any thoughts. >> Having all the information from the web UI accessible through a supported >> json API is very important to us; are there any objections to us adding a v2 >> API to Spark? >> >> Thanks! >> >> From: Kevin Chen <kc...@palantir.com> >> Date: Friday, September 11, 2015 at 11:30 AM >> To: "dev@spark.apache.org" <dev@spark.apache.org> >> Cc: Matt Cheah <mch...@palantir.com>, Mingyu Kim <m...@palantir.com> >> Subject: New Spark json endpoints >> >> Hello Spark Devs, >> >> I noticed that [SPARK-3454], which introduces new json endpoints at >> /api/v1/[path] for information previously only shown on the web UI, does not >> expose several useful properties about Spark jobs that are exposed on the web >> UI and on the unofficial /json endpoint. >> >> Specific examples include the maximum number of allotted cores per >> application, amount of memory allotted to each slave, and number of cores >> used by each worker. These are provided at ‘app.cores, app.memoryperslave, >> and worker.coresused’ in the /json endpoint, and also all appear on the web >> UI page. >> >> Is there any specific reason that these fields are not exposed in the public >> API? If not, would it be reasonable to add them to the json blobs, possibly >> in a future /api/v2 API? >> >> Thank you, >> Kevin Chen >> >
smime.p7s
Description: S/MIME cryptographic signature
