One additional thing I'd like is the ability to flexup a nmnode on all agents with a certain attribute as opposed to a fixed number.
Also, when I get back from vacation I plan on scoping mesos-1739 (dynamic attributes) which would allow for tighter integration with the hdfs framework. An alternative would be to get hostnames from the namenode, though not as seemless. (Reviving this thread) We've discussed several great points in the thread (PUT vs POST, need for GET, JSON payload vs parameters in URL, declarative interface etc). Just to get us going, I think we should focus on a couple of things that will be useful for Myriad users, while leaving them flexible enough to be evolved in the future. What I heard from several folks (some of it brought up again at MesosCon) about the flex up/down APIs is this: - flexup doesn't support launching NMs on specific set of hosts. This is especially needed to launch NMs on same set of nodes that have HDFS DataNode running. - flexdown lacks an option to shut down NMs with a specific profile. Today, we bring down ANY arbitrary NM. - flexdown lacks an option to shutdown NMs running on specific hosts. I captured my thoughts in a document here: https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t @Jim: Happy to collaborate at one single place (Swagger/Google Doc) to finalize the APIs. Just let me know. Thanks, Santosh On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <klu...@gmail.com> wrote: > Seems like POST is a winner with people. > > Another thing to consider is how we want the REST interface to be vs what > we want the UI to do. The UI could support flexup/flexdown like it is while > the REST interface is just a declarative state like Adam suggested. The UI > would just be responsible for translating the request into the new state. > > Tomorrow I'll try to put together another swagger doc with some of the > suggested options. > > > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman > <yufeld...@yahoo.com.invalid > > wrote: > > > I think we are at the point to list all the options we want "flex" API to > > support. > > 1. Do we continue supporting flexup/down or just "flex" with additional > > "preposition" like up/down:https://hostname:port/flex/up(down) > > 2. I think we should switch to POST and may be maintain PUT for legacy > (if > > even needed to keep it). We are not DB after all and not storing any > > retrievable info here :) > > 3. We need to add status (GET) to see the status - though I think we have > > one > > 4. Define JSON payload to support different cases a. providing > > different profiles together: [{profile:"big", > > instances:2},{profile:"medium",instances:6}] b. provide what state we > > want Myriad to be in: "I want 10 medium instances" and then Myriad will > do > > whatever isnecessary to transition to that state, > adding/removing/resizing > > NMs" c. flex/down particular instance IDs d. flex up/down preferred > > hosts, delays, others > > 5. How all this fits into FineGrain Scaling? With it we would do > automatic > > flex up/down. And the less knobs admin will have to turn the easier it is > > for admin and the end users. > > > > From: Adam Bordelon <a...@mesosphere.io> > > To: dev@myriad.incubator.apache.org > > Sent: Sunday, June 14, 2015 2:54 PM > > Subject: Re: Flex API > > > > (In addition,) I'd also like to see a more declarative interface. Instead > > of "add two more instances", the user(s) could just specify the desired > > state of "I want 10 medium instances" and then Myriad will do whatever is > > necessary to transition to that state, adding/removing/resizing NMs as > > necessary. > > > > > > > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena < > > wochandar...@maprtech.com > > > wrote: > > > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <klu...@gmail.com> wrote: > > > > > > > What verb to use when outside of database land can be argued. I would > > > vote > > > > for POST over PUT just because I tend to default to POST. PUT was > there > > > > when I showed up, so I left it. > > > > > > > > > Last time I agonized about PUT vs POST the most logical distinction I > > found > > > was that PUT should be used for idempotent operations, while POST for > > > non-idempotent (like we have here with flex-up, since instance-ids are > > > generated). > > > > > > Since the api doesn't wait until the > > > > instances are created to return, we can't really return the instance > > IDs > > > we > > > > created. > > > > > > > > > > That seems OK to me. > > > > > > > > > > The GET would just return some status? > > > > > > > > > > Yeah, I was thinking that this would be needed for a future GUI where > we > > > list all instances with parameters and status for each (profile, > current > > > cpu/ram/disk, node, uptime). I'm picturing checkboxes next to each so > > > users can multi-select and hit 'delete' to wipe them away (like > flex-down > > > does now). > > > > > > The PATCH is interesting > > > > > > > > > > Yeah, I started to write PUT but to REST geeks PUT implies you always > > have > > > to rewrite the complete object when making changes. PATCH allows more > > > flexible modifications. > > > > > > The DELETE makes sense to me. > > > > > > > > Your use of instances vs instance is interesting. Perhaps we want to > > > > support POST [{profile:"big", instances:2},{profile:"medium", > > > instances:6}] > > > > > > > > > > Yeah, that'd be cool! > > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena < > > > > wochandar...@maprtech.com > > > > > wrote: > > > > > > > > > Any reason we need to make it action-based (flex-up/flex-down) > rather > > > > than > > > > > instance outcome-based? The way i've seen similar things modeled > in > > > > other > > > > > REST APIs is similar to below. This may also fit better into REST > > > client > > > > > frameworks. > > > > > > > > > > Create - POST /instances/ {"profile" : "zero", "instances" : 2, > > > "delay" : > > > > > 2, "preferredHosts" : ["host1", "host2"]} > > > > > Read - GET /instances/ or GET /instances/<instanceid> > > > > > Update (resize?) - PATCH /instances/<instanceid> {"profile" : > "big"} > > > > > Delete - DELETE /instances/<instanceid> > > > > > > > > > > Thoughts? > > > > > > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar <klu...@gmail.com> > > wrote: > > > > > > > > > > > That is always an issue with parameters in the URL, and there's > > > nothing > > > > > > worse than parameters in the URL and having to put JSON in the > body > > > > > because > > > > > > we want to extend it later. Currently flexup takes a JSON blob in > > the > > > > > body > > > > > > which we could add more to like you showed. I guess the question > is > > > if > > > > we > > > > > > want to support a quick and dirty flexup api via URL parameters. > I > > > > > wouldn't > > > > > > go any further than these parameters, but all this would do at > this > > > > point > > > > > > is make testing with a curl command easier. Getting rid of it for > > > > flexup > > > > > is > > > > > > fine with me. > > > > > > > > > > > > I do like the flexdown/instance/{instance-id} though. We should > > > > probably > > > > > do > > > > > > one with flexdown/instance with a json array of instance-ids too > > > > though. > > > > > > > > > > > > Also with the parameter verification patch I just submitted, if > you > > > ask > > > > > to > > > > > > flexdown 10 instances and you only have 5 running it will just > log > > a > > > > > > warning and kill all 5. Not sure if that is the desired behavior. > > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella < > > > > smare...@maprtech.com> > > > > > > wrote: > > > > > > > > > > > > > I think rather than having an API defined as > > > > > > > "/cluster/flexup/profile/{profile-name}", it's better to define > > it > > > > just > > > > > > as > > > > > > > "/cluster/flexup" and have parameters such as "profile: > > > > > > > <a-value-for-profile-name>" in the JSON payload. The reason is, > > if > > > > the > > > > > > > params are added into the API endpoint, it becomes less > flexible > > to > > > > > > evolve. > > > > > > > For e.g. /cluster/flexup currently can take just "profile" and > > > > > > "instances" > > > > > > > in it's payload, but in the future can optionally take > parameters > > > > such > > > > > as > > > > > > > "preferredHosts: [host1, host2, host9]", "delay: > > > > > > > <take_some_sweet_time_before_flexing_up>" etc. > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > Santosh > > > > > > > > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar <klu...@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > I thought that might happen. I created a gist here: > > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e > > > > > > > > > > > > > > > > And just in case.... > > > > > > > > > > > > > > > > swagger: '2.0' > > > > > > > > info: > > > > > > > > version: 0.0.1 > > > > > > > > title: Myriad API > > > > > > > > description: | > > > > > > > > Myriad API description > > > > > > > > basePath: /api > > > > > > > > schemes: > > > > > > > > - http > > > > > > > > consumes: > > > > > > > > - application/json > > > > > > > > - text/plain > > > > > > > > produces: > > > > > > > > - application/json > > > > > > > > - text/plain > > > > > > > > paths: > > > > > > > > /cluster/flexup: > > > > > > > > put: > > > > > > > > parameters: > > > > > > > > - name: instance > > > > > > > > in: body > > > > > > > > description: Instance profile and quantity to flex > up > > > > > > > > schema: > > > > > > > > $ref: '#/definitions/FlexUp' > > > > > > > > required: true > > > > > > > > description: Original Flexup API > > > > > > > > responses: > > > > > > > > 200: > > > > > > > > description: OK > > > > > > > > /cluster/flexup/profile/{profile-name}: > > > > > > > > put: > > > > > > > > parameters: > > > > > > > > - name: profile-name > > > > > > > > in: path > > > > > > > > type: string > > > > > > > > description: Instance profile name > > > > > > > > description: Flexup a single instance of a specified > > > profile > > > > > > > > responses: > > > > > > > > 200: > > > > > > > > description: OK > > > > > > > > /cluster/flexup/profile/{profile-name}/{instances}: > > > > > > > > put: > > > > > > > > parameters: > > > > > > > > - name: profile-name > > > > > > > > in: path > > > > > > > > type: string > > > > > > > > description: Instance profile name > > > > > > > > - name: instances > > > > > > > > in: path > > > > > > > > type: integer > > > > > > > > description: Number of instances > > > > > > > > description: Flexup a multiple instances of a specified > > > > profile > > > > > > > > responses: > > > > > > > > 200: > > > > > > > > description: OK > > > > > > > > /cluster/flexdown: > > > > > > > > put: > > > > > > > > parameters: > > > > > > > > - name: instance > > > > > > > > in: body > > > > > > > > description: Number of instances to flex down > > > > > > > > schema: > > > > > > > > $ref: '#/definitions/FlexDown' > > > > > > > > required: true > > > > > > > > description: Original Flexdown API > > > > > > > > responses: > > > > > > > > 200: > > > > > > > > description: OK > > > > > > > > /cluster/flexdown/profile/{profile-name}: > > > > > > > > put: > > > > > > > > parameters: > > > > > > > > - name: profile-name > > > > > > > > in: path > > > > > > > > type: string > > > > > > > > description: Instance profile name > > > > > > > > description: Flexup a single instance of a specified > > > profile > > > > > > > > responses: > > > > > > > > 200: > > > > > > > > description: OK > > > > > > > > /cluster/flexdown/profile/{profile-name}/{instances}: > > > > > > > > put: > > > > > > > > parameters: > > > > > > > > - name: profile-name > > > > > > > > in: path > > > > > > > > type: string > > > > > > > > description: Instance profile name > > > > > > > > - name: instances > > > > > > > > in: path > > > > > > > > type: integer > > > > > > > > description: Number of instances > > > > > > > > description: Flexdown a multiple instances of a > specified > > > > > profile > > > > > > > > responses: > > > > > > > > 200: > > > > > > > > description: OK > > > > > > > > /cluster/flexdown/instance/{instance-id}: > > > > > > > > put: > > > > > > > > parameters: > > > > > > > > - name: instance-id > > > > > > > > in: path > > > > > > > > type: string > > > > > > > > description: Instance profile name > > > > > > > > description: Flexup a single instance of a specified > > > profile > > > > > > > > responses: > > > > > > > > 200: > > > > > > > > description: OK > > > > > > > > definitions: > > > > > > > > FlexUp: > > > > > > > > properties: > > > > > > > > profile: > > > > > > > > type: string > > > > > > > > instances: > > > > > > > > type: integer > > > > > > > > format: int32 > > > > > > > > FlexDown: > > > > > > > > properties: > > > > > > > > instances: > > > > > > > > type: integer > > > > > > > > format: int32 > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella < > > > > > > smare...@maprtech.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Jim, > > > > > > > > > > > > > > > > > > Did you attach a file? I think the apache mailing list > > > > swallowed > > > > > > the > > > > > > > > > attachment. Can you please send us the document contents in > > > plain > > > > > > text? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Santosh > > > > > > > > > > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar < > > klu...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > This is in regards to > > > > https://github.com/mesos/myriad/issues/89 > > > > > > > > > > > > > > > > > > > > I created a swagger document (attached) of the current > flex > > > > APIs > > > > > > and > > > > > > > > what > > > > > > > > > > I propose for the new flex apis. If people like swagger, > I > > > can > > > > > > > document > > > > > > > > > the > > > > > > > > > > rest of the API, including the correct response codes. > > Either > > > > > way I > > > > > > > > think > > > > > > > > > > we should discuss what the API should be going forward. > > > > > > > > > > > > > > > > > > > > Paste the file contents into http://editor.swagger.io/ > and > > > > > you'll > > > > > > > get > > > > > > > > > > nice HTML to browse. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
