+1, but EQUALS might be an easier starting point than LIKE. Your choice. What about flexing down specific hosts? Is that out of scope for now?
On Wed, Sep 2, 2015 at 2:13 PM, Santosh Marella <smare...@maprtech.com> wrote: > Ping! Checking if there is anymore feedback on the APIs for flexup/down. > > Here is a textual summary of what's being proposed: > 1. Add a new “constraints” parameter in both flexup and flexdown APIs (just > like how Marathon does). > Initially, start with support for LIKE operator in "constraints" that > operates on “hostname” and “mesos slave attribute > <http://mesos.apache.org/documentation/attributes-resources/>” fields > using > regex. > 2. Add a new “profile” parameter in the flexdown API. > > #1 would give admins a better control over where the NMs should be launched > or which specific NMs should be shut down. > #2 helps with shutting down multiple NMs of same "profile" in one shot. > > If the above sounds reasonable to start with, I'd like to start working on > the code changes. > > Thanks, > Santosh > > On Thu, Aug 27, 2015 at 4:25 PM, Santosh Marella <smare...@maprtech.com> > wrote: > > > > One additional thing I'd like is the ability to flexup a nmnode on all > > > agents with a certain attribute as opposed to a fixed number. > > > > Certainly! Marathon places tasks on agents sharing a common attribute. > > Myriad could/should do the same for placing NMs. If each agent running > > HDFS data node shares an attribute such as "dfs", then NMs could always > > be launched on the "dfs" nodes. This improves data locality for YARN jobs > > dramatically. > > > > Santosh > > > > On Thu, Aug 27, 2015 at 2:47 PM, Darin Johnson <dbjohnson1...@gmail.com> > > wrote: > > > >> One additional thing I'd like is the ability to flexup a nmnode on all > >> agents with a certain attribute as opposed to a fixed number. > >> > > > > > >> > >> Also, when I get back from vacation I plan on scoping mesos-1739 > (dynamic > >> attributes) which would allow for tighter integration with the hdfs > >> framework. An alternative would be to get hostnames from the namenode, > >> though not as seemless. > >> (Reviving this thread) > >> > >> We've discussed several great points in the thread (PUT vs POST, need > for > >> GET, JSON payload vs parameters in URL, declarative interface etc). > >> Just to get us going, I think we should focus on a couple of things that > >> will be useful for Myriad users, while leaving them flexible enough to > be > >> evolved in the future. > >> > >> What I heard from several folks (some of it brought up again at > MesosCon) > >> about the flex up/down APIs is this: > >> - flexup doesn't support launching NMs on specific set of hosts. This is > >> especially needed to launch NMs on same set of nodes that have HDFS > >> DataNode running. > >> - flexdown lacks an option to shut down NMs with a specific profile. > >> Today, > >> we bring down ANY arbitrary NM. > >> - flexdown lacks an option to shutdown NMs running on specific hosts. > >> > >> I captured my thoughts in a document here: > >> > >> > https://docs.google.com/document/d/1PA_POY_abP6J4youM2Q0VJ48T4OCSe258-OAz_-EO6k/edit#heading=h.1atlx0ag9s8t > >> > >> @Jim: Happy to collaborate at one single place (Swagger/Google Doc) to > >> finalize the APIs. Just let me know. > >> > >> Thanks, > >> Santosh > >> > >> On Sun, Jun 14, 2015 at 5:29 PM, Jim Klucar <klu...@gmail.com> wrote: > >> > >> > Seems like POST is a winner with people. > >> > > >> > Another thing to consider is how we want the REST interface to be vs > >> what > >> > we want the UI to do. The UI could support flexup/flexdown like it is > >> while > >> > the REST interface is just a declarative state like Adam suggested. > The > >> UI > >> > would just be responsible for translating the request into the new > >> state. > >> > > >> > Tomorrow I'll try to put together another swagger doc with some of the > >> > suggested options. > >> > > >> > > >> > On Sun, Jun 14, 2015 at 6:37 PM, yuliya Feldman > >> > <yufeld...@yahoo.com.invalid > >> > > wrote: > >> > > >> > > I think we are at the point to list all the options we want "flex" > API > >> to > >> > > support. > >> > > 1. Do we continue supporting flexup/down or just "flex" with > >> additional > >> > > "preposition" like up/down:https://hostname:port/flex/up(down) > >> > > 2. I think we should switch to POST and may be maintain PUT for > legacy > >> > (if > >> > > even needed to keep it). We are not DB after all and not storing any > >> > > retrievable info here :) > >> > > 3. We need to add status (GET) to see the status - though I think we > >> have > >> > > one > >> > > 4. Define JSON payload to support different cases a. providing > >> > > different profiles together: [{profile:"big", > >> > > instances:2},{profile:"medium",instances:6}] b. provide what > state > >> we > >> > > want Myriad to be in: "I want 10 medium instances" and then Myriad > >> will > >> > do > >> > > whatever isnecessary to transition to that state, > >> > adding/removing/resizing > >> > > NMs" c. flex/down particular instance IDs d. flex up/down > >> preferred > >> > > hosts, delays, others > >> > > 5. How all this fits into FineGrain Scaling? With it we would do > >> > automatic > >> > > flex up/down. And the less knobs admin will have to turn the easier > it > >> is > >> > > for admin and the end users. > >> > > > >> > > From: Adam Bordelon <a...@mesosphere.io> > >> > > To: dev@myriad.incubator.apache.org > >> > > Sent: Sunday, June 14, 2015 2:54 PM > >> > > Subject: Re: Flex API > >> > > > >> > > (In addition,) I'd also like to see a more declarative interface. > >> Instead > >> > > of "add two more instances", the user(s) could just specify the > >> desired > >> > > state of "I want 10 medium instances" and then Myriad will do > whatever > >> is > >> > > necessary to transition to that state, adding/removing/resizing NMs > as > >> > > necessary. > >> > > > >> > > > >> > > > >> > > On Fri, Jun 12, 2015 at 5:23 PM, Will Ochandarena < > >> > > wochandar...@maprtech.com > >> > > > wrote: > >> > > > >> > > > On Fri, Jun 12, 2015 at 5:11 PM, Jim Klucar <klu...@gmail.com> > >> wrote: > >> > > > > >> > > > > What verb to use when outside of database land can be argued. I > >> would > >> > > > vote > >> > > > > for POST over PUT just because I tend to default to POST. PUT > was > >> > there > >> > > > > when I showed up, so I left it. > >> > > > > >> > > > > >> > > > Last time I agonized about PUT vs POST the most logical > distinction > >> I > >> > > found > >> > > > was that PUT should be used for idempotent operations, while POST > >> for > >> > > > non-idempotent (like we have here with flex-up, since instance-ids > >> are > >> > > > generated). > >> > > > > >> > > > Since the api doesn't wait until the > >> > > > > instances are created to return, we can't really return the > >> instance > >> > > IDs > >> > > > we > >> > > > > created. > >> > > > > > >> > > > > >> > > > That seems OK to me. > >> > > > > >> > > > > >> > > > > The GET would just return some status? > >> > > > > > >> > > > > >> > > > Yeah, I was thinking that this would be needed for a future GUI > >> where > >> > we > >> > > > list all instances with parameters and status for each (profile, > >> > current > >> > > > cpu/ram/disk, node, uptime). I'm picturing checkboxes next to > each > >> so > >> > > > users can multi-select and hit 'delete' to wipe them away (like > >> > flex-down > >> > > > does now). > >> > > > > >> > > > The PATCH is interesting > >> > > > > > >> > > > > >> > > > Yeah, I started to write PUT but to REST geeks PUT implies you > >> always > >> > > have > >> > > > to rewrite the complete object when making changes. PATCH allows > >> more > >> > > > flexible modifications. > >> > > > > >> > > > The DELETE makes sense to me. > >> > > > > > >> > > > > Your use of instances vs instance is interesting. Perhaps we > want > >> to > >> > > > > support POST [{profile:"big", instances:2},{profile:"medium", > >> > > > instances:6}] > >> > > > > > >> > > > > >> > > > Yeah, that'd be cool! > >> > > > > >> > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Fri, Jun 12, 2015 at 6:23 PM, Will Ochandarena < > >> > > > > wochandar...@maprtech.com > >> > > > > > wrote: > >> > > > > > >> > > > > > Any reason we need to make it action-based (flex-up/flex-down) > >> > rather > >> > > > > than > >> > > > > > instance outcome-based? The way i've seen similar things > >> modeled > >> > in > >> > > > > other > >> > > > > > REST APIs is similar to below. This may also fit better into > >> REST > >> > > > client > >> > > > > > frameworks. > >> > > > > > > >> > > > > > Create - POST /instances/ {"profile" : "zero", "instances" : > 2, > >> > > > "delay" : > >> > > > > > 2, "preferredHosts" : ["host1", "host2"]} > >> > > > > > Read - GET /instances/ or GET /instances/<instanceid> > >> > > > > > Update (resize?) - PATCH /instances/<instanceid> {"profile" : > >> > "big"} > >> > > > > > Delete - DELETE /instances/<instanceid> > >> > > > > > > >> > > > > > Thoughts? > >> > > > > > > >> > > > > > On Fri, Jun 12, 2015 at 3:07 PM, Jim Klucar <klu...@gmail.com > > > >> > > wrote: > >> > > > > > > >> > > > > > > That is always an issue with parameters in the URL, and > >> there's > >> > > > nothing > >> > > > > > > worse than parameters in the URL and having to put JSON in > the > >> > body > >> > > > > > because > >> > > > > > > we want to extend it later. Currently flexup takes a JSON > blob > >> in > >> > > the > >> > > > > > body > >> > > > > > > which we could add more to like you showed. I guess the > >> question > >> > is > >> > > > if > >> > > > > we > >> > > > > > > want to support a quick and dirty flexup api via URL > >> parameters. > >> > I > >> > > > > > wouldn't > >> > > > > > > go any further than these parameters, but all this would do > at > >> > this > >> > > > > point > >> > > > > > > is make testing with a curl command easier. Getting rid of > it > >> for > >> > > > > flexup > >> > > > > > is > >> > > > > > > fine with me. > >> > > > > > > > >> > > > > > > I do like the flexdown/instance/{instance-id} though. We > >> should > >> > > > > probably > >> > > > > > do > >> > > > > > > one with flexdown/instance with a json array of instance-ids > >> too > >> > > > > though. > >> > > > > > > > >> > > > > > > Also with the parameter verification patch I just submitted, > >> if > >> > you > >> > > > ask > >> > > > > > to > >> > > > > > > flexdown 10 instances and you only have 5 running it will > just > >> > log > >> > > a > >> > > > > > > warning and kill all 5. Not sure if that is the desired > >> behavior. > >> > > > > > > > >> > > > > > > > >> > > > > > > On Fri, Jun 12, 2015 at 5:12 PM, Santosh Marella < > >> > > > > smare...@maprtech.com> > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > I think rather than having an API defined as > >> > > > > > > > "/cluster/flexup/profile/{profile-name}", it's better to > >> define > >> > > it > >> > > > > just > >> > > > > > > as > >> > > > > > > > "/cluster/flexup" and have parameters such as "profile: > >> > > > > > > > <a-value-for-profile-name>" in the JSON payload. The > reason > >> is, > >> > > if > >> > > > > the > >> > > > > > > > params are added into the API endpoint, it becomes less > >> > flexible > >> > > to > >> > > > > > > evolve. > >> > > > > > > > For e.g. /cluster/flexup currently can take just "profile" > >> and > >> > > > > > > "instances" > >> > > > > > > > in it's payload, but in the future can optionally take > >> > parameters > >> > > > > such > >> > > > > > as > >> > > > > > > > "preferredHosts: [host1, host2, host9]", "delay: > >> > > > > > > > <take_some_sweet_time_before_flexing_up>" etc. > >> > > > > > > > > >> > > > > > > > Thoughts? > >> > > > > > > > > >> > > > > > > > Santosh > >> > > > > > > > > >> > > > > > > > On Fri, Jun 12, 2015 at 1:48 PM, Jim Klucar < > >> klu...@gmail.com> > >> > > > > wrote: > >> > > > > > > > > >> > > > > > > > > I thought that might happen. I created a gist here: > >> > > > > > > > > https://gist.github.com/klucar/c534d4ecb9f537f9e91e > >> > > > > > > > > > >> > > > > > > > > And just in case.... > >> > > > > > > > > > >> > > > > > > > > swagger: '2.0' > >> > > > > > > > > info: > >> > > > > > > > > version: 0.0.1 > >> > > > > > > > > title: Myriad API > >> > > > > > > > > description: | > >> > > > > > > > > Myriad API description > >> > > > > > > > > basePath: /api > >> > > > > > > > > schemes: > >> > > > > > > > > - http > >> > > > > > > > > consumes: > >> > > > > > > > > - application/json > >> > > > > > > > > - text/plain > >> > > > > > > > > produces: > >> > > > > > > > > - application/json > >> > > > > > > > > - text/plain > >> > > > > > > > > paths: > >> > > > > > > > > /cluster/flexup: > >> > > > > > > > > put: > >> > > > > > > > > parameters: > >> > > > > > > > > - name: instance > >> > > > > > > > > in: body > >> > > > > > > > > description: Instance profile and quantity to > >> flex > >> > up > >> > > > > > > > > schema: > >> > > > > > > > > $ref: '#/definitions/FlexUp' > >> > > > > > > > > required: true > >> > > > > > > > > description: Original Flexup API > >> > > > > > > > > responses: > >> > > > > > > > > 200: > >> > > > > > > > > description: OK > >> > > > > > > > > /cluster/flexup/profile/{profile-name}: > >> > > > > > > > > put: > >> > > > > > > > > parameters: > >> > > > > > > > > - name: profile-name > >> > > > > > > > > in: path > >> > > > > > > > > type: string > >> > > > > > > > > description: Instance profile name > >> > > > > > > > > description: Flexup a single instance of a > specified > >> > > > profile > >> > > > > > > > > responses: > >> > > > > > > > > 200: > >> > > > > > > > > description: OK > >> > > > > > > > > /cluster/flexup/profile/{profile-name}/{instances}: > >> > > > > > > > > put: > >> > > > > > > > > parameters: > >> > > > > > > > > - name: profile-name > >> > > > > > > > > in: path > >> > > > > > > > > type: string > >> > > > > > > > > description: Instance profile name > >> > > > > > > > > - name: instances > >> > > > > > > > > in: path > >> > > > > > > > > type: integer > >> > > > > > > > > description: Number of instances > >> > > > > > > > > description: Flexup a multiple instances of a > >> specified > >> > > > > profile > >> > > > > > > > > responses: > >> > > > > > > > > 200: > >> > > > > > > > > description: OK > >> > > > > > > > > /cluster/flexdown: > >> > > > > > > > > put: > >> > > > > > > > > parameters: > >> > > > > > > > > - name: instance > >> > > > > > > > > in: body > >> > > > > > > > > description: Number of instances to flex down > >> > > > > > > > > schema: > >> > > > > > > > > $ref: '#/definitions/FlexDown' > >> > > > > > > > > required: true > >> > > > > > > > > description: Original Flexdown API > >> > > > > > > > > responses: > >> > > > > > > > > 200: > >> > > > > > > > > description: OK > >> > > > > > > > > /cluster/flexdown/profile/{profile-name}: > >> > > > > > > > > put: > >> > > > > > > > > parameters: > >> > > > > > > > > - name: profile-name > >> > > > > > > > > in: path > >> > > > > > > > > type: string > >> > > > > > > > > description: Instance profile name > >> > > > > > > > > description: Flexup a single instance of a > specified > >> > > > profile > >> > > > > > > > > responses: > >> > > > > > > > > 200: > >> > > > > > > > > description: OK > >> > > > > > > > > /cluster/flexdown/profile/{profile-name}/{instances}: > >> > > > > > > > > put: > >> > > > > > > > > parameters: > >> > > > > > > > > - name: profile-name > >> > > > > > > > > in: path > >> > > > > > > > > type: string > >> > > > > > > > > description: Instance profile name > >> > > > > > > > > - name: instances > >> > > > > > > > > in: path > >> > > > > > > > > type: integer > >> > > > > > > > > description: Number of instances > >> > > > > > > > > description: Flexdown a multiple instances of a > >> > specified > >> > > > > > profile > >> > > > > > > > > responses: > >> > > > > > > > > 200: > >> > > > > > > > > description: OK > >> > > > > > > > > /cluster/flexdown/instance/{instance-id}: > >> > > > > > > > > put: > >> > > > > > > > > parameters: > >> > > > > > > > > - name: instance-id > >> > > > > > > > > in: path > >> > > > > > > > > type: string > >> > > > > > > > > description: Instance profile name > >> > > > > > > > > description: Flexup a single instance of a > specified > >> > > > profile > >> > > > > > > > > responses: > >> > > > > > > > > 200: > >> > > > > > > > > description: OK > >> > > > > > > > > definitions: > >> > > > > > > > > FlexUp: > >> > > > > > > > > properties: > >> > > > > > > > > profile: > >> > > > > > > > > type: string > >> > > > > > > > > instances: > >> > > > > > > > > type: integer > >> > > > > > > > > format: int32 > >> > > > > > > > > FlexDown: > >> > > > > > > > > properties: > >> > > > > > > > > instances: > >> > > > > > > > > type: integer > >> > > > > > > > > format: int32 > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > On Fri, Jun 12, 2015 at 4:31 PM, Santosh Marella < > >> > > > > > > smare...@maprtech.com> > >> > > > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > Hi Jim, > >> > > > > > > > > > > >> > > > > > > > > > Did you attach a file? I think the apache mailing > list > >> > > > > swallowed > >> > > > > > > the > >> > > > > > > > > > attachment. Can you please send us the document > contents > >> in > >> > > > plain > >> > > > > > > text? > >> > > > > > > > > > > >> > > > > > > > > > Thanks, > >> > > > > > > > > > Santosh > >> > > > > > > > > > > >> > > > > > > > > > On Fri, Jun 12, 2015 at 1:25 PM, Jim Klucar < > >> > > klu...@gmail.com> > >> > > > > > > wrote: > >> > > > > > > > > > > >> > > > > > > > > > > This is in regards to > >> > > > > https://github.com/mesos/myriad/issues/89 > >> > > > > > > > > > > > >> > > > > > > > > > > I created a swagger document (attached) of the > current > >> > flex > >> > > > > APIs > >> > > > > > > and > >> > > > > > > > > what > >> > > > > > > > > > > I propose for the new flex apis. If people like > >> swagger, > >> > I > >> > > > can > >> > > > > > > > document > >> > > > > > > > > > the > >> > > > > > > > > > > rest of the API, including the correct response > codes. > >> > > Either > >> > > > > > way I > >> > > > > > > > > think > >> > > > > > > > > > > we should discuss what the API should be going > >> forward. > >> > > > > > > > > > > > >> > > > > > > > > > > Paste the file contents into > >> http://editor.swagger.io/ > >> > and > >> > > > > > you'll > >> > > > > > > > get > >> > > > > > > > > > > nice HTML to browse. > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > > > >
