Yes, API v2.0 would be all Go, and in TC 5.0 we would drop support for API 1.x and remove the Perl implementation.
- Rawlin On Tue, Dec 3, 2019 at 11:04 AM Schmidt, Andrew (Contractor) <[email protected]> wrote: > > +1 Makes sense to me. Is 2.0 the target for dropping the PERL implementation? > > On 12/2/19, 12:19 PM, "Gray, Jonathan" <[email protected]> wrote: > > +1 > > I think the only point of contention was if the API v2.0 would be in ATC > v4.0 or not. Either way is fine with me, and more time to know this is > coming makes sense. > > Jonathan G > > > On 12/2/19, 11:32 AM, "Rawlin Peters" <[email protected]> wrote: > > Hey all, > > A bunch of us got together and discussed this issue offline, and we > got consensus to go with Option 1 - Graduate current API from 1.x to > 2.x - allowing us to leave behind routes in 1.x that are > dangerous/deprecated. Deprecated endpoints will not be rewritten from > Perl to Go. > > So, we will graduate the latest minor version of existing 1.x Go > endpoints to 2.0. For example, today we have > /api/1.2/deliveryservices, /api/1.3/deliveryservices, and > /api/1.4/deliveryservices. We will take the existing handler for > /api/1.4/deliveryservices and promote it to /api/2.0/deliveryservices. > We will follow this example for other 1.x Go endpoints with multiple > minor versions as well. > > API endpoints that are deemed broken, dangerous, or deprecated will > not be promoted to API 2.0. This will likely include the ~7 endpoints > that I've mentioned in this thread already, but we haven't yet > compiled the full list of endpoints that won't be promoted to 2.0. We > will come up with that list (along with its criteria) soon and bring > that list to the mailing list for everyone to look over and agree > upon. Deprecated endpoints will be updated to return deprecation > alerts where applicable. > > This API graduation will likely miss the boat for 4.0 but will > hopefully land in 4.1 or 4.2 depending on how long it takes. So this > would be the tentative timetable: > > TC 4.0: > - API 1.x supported, some deprecation notices > > TC 4.1: > - API 1.x still supported, deprecation notices added to endpoints not > graduated to 2.0 > - API 2.0 supported, consisting of 1.x endpoints that were graduated > - starting with this release, you need to start migrating external > clients off of 1.x over to 2.0 > > TC 4.2: > - internal clients (e.g. TM, TR, etc) will be migrated off API 1.x > over to 2.0. Doing this step after 4.1 adds confidence that 1.x is > still supported alongside 2.0 in order to provide a smooth migration > period for API clients. > > TC 5.0: > - API 1.x no longer supported, only API 2.x is supported > > Those who took part in that offline discussion, please keep me honest > here with a +1 if we did in fact come to a rough consensus on this > option. If you weren't part of that offline discussion, feel free to > voice your vote on this issue as well (I believe a bunch of people > were already +1 on this option a few weeks ago). > > - Rawlin > > On Tue, Nov 19, 2019 at 12:23 PM Robert O Butts <[email protected]> > wrote: > > > > > - /riak/stats > > > > Looking at the Perl, this looks easy to rewrite. It's just proxying > a > > request to Riak. > > > > +1 on rewriting this, for API stability, even if we don't "value" > the > > endpoint as a project. > > > > >- /cachegroup_fallbacks > > > > Likewise +1 on rewriting. Not as trivial as /riak/stats, but it > also looks > > easy to me, looks like a standard CRUD endpoint. +1 on leaving a > > "deprecated, please prefer X" message in, and +1 in removing in the > big API > > rewrite/refactor/2.0. > > > > > - /capabilities (all methods except for GET) -- capabilities are > > built-in, should not be changed at runtime > > > - /api_capabilities (all methods except for GET) -- same reasons > as > > /capabilities > > > > I actually don't agree with this - because they're useful to > plugins. IMO > > it should be possible to add and modify Capabilities and API > Capabilities, > > because they can be used by plugins which add new endpoints, to > control > > access to them. If there's disagreement there, I'd vote we take > that to a > > separate thread. > > > > But in terms of the endpoints themselves, the Perl also looks easy > to > > rewrite, looks like standard CRUD. -1 on deprecating or removing, > +1 on > > rewriting. > > > > > - /cdns/{name}/configs/routing -- this was a half-baked attempt at > > redoing CrConfig.json -- cannot possibly be of use to anyone in its > current > > state > > > > Ugh. I agree with the sentiment; and this one looks slightly less > trivial. > > But I personally don't see it as an unreasonable amount of work. > For the > > sake of API stability, I'm personally willing to volunteer to > rewrite this > > one, including API tests to make fixing easy if the schema changes > (which > > it appears to have in master). And hopefully it can just exist > until the > > fabled API Rewrite. Is that acceptable? > > > > > - the ATS config generation endpoints -- Rob actually proposed to > remove > > these from 1.x due to danger of being out of sync w/ cache-side > config > > generator > > > > I'm personally on the fence about this one. Like I said in that > thread, I'm > > the last one to want to break users, and I don't take this lightly. > If, > > after I made the case for it, I'm the only one, I might be inclined > to > > retract my proposal, and just continue to maintain them. Maybe some > more > > API tests would help with that. I think I'm the primary dev on the > ATS > > config gen at this point anyway. > > > > > - ATS Config Files Stuff Aren't these all already in Go anyway? > Whether > > or not they should be removed from the API aside, I don't think > anything > > actually needs to hit Perl anymore. > > > > Correct. > > > > > - /hwinfo/dtdata > > > > I think the 'render' call here is just rendering the JSON. But, the > > endpoint itself is just hanging and not returning for me, I'm not > sure why. > > I'd like to volunteer to look into this endpoint as well. I think > you might > > be right, this one might be unreasonable to rewrite. But if it is > > reasonable, I'd like to just do it. I'd like to at least look into > it. > > > > > > On Tue, Nov 19, 2019 at 12:05 PM ocket 8888 <[email protected]> > wrote: > > > > > I'm generally +1 for option 2 > > > > > > regarding those specific endpoints: > > > > > > - /riak/stats if this is easy and doesn't expose anything > dangerous, we > > > should do it IMO. > > > - /cachegroup_fallbacks should be pretty trivial, so I still say > do it > > > - /capabilities since these aren't actually used for anything, I > think it's > > > pretty safe (and easy) to rewrite. Plus there's already a PR open > for > > > exactly that > > > - /api_capabilities dropping this one could be problematic, just > because > > > people may depend on it to tell them what endpoints exist. To be > clear, I > > > think it ought to be deprecated, but unfortunately I just don't > think it's > > > terribly unlikely that this is built into scripts and tools. The > actual > > > code for the rewrite should be easy too, although maintaining it > is a pain. > > > > > > - ATS Config Files Stuff Aren't these all already in Go anyway? > Whether or > > > not they should be removed from the API aside, I don't think > anything > > > actually needs to hit Perl anymore. > > > > > > - /cdn/{{name}}/configs/routing CRConfigs are large and complex, > so if this > > > never worked anyway maybe we should just add a Go handler for it > that > > > returns a '410 Gone' response (or '501 Not Implemented' is the > other one > > > I've heard suggested) and leave the actual logic behind in the > Perl > > > - /hwinfo/dtdata if this calls into Mojolicious rendering, > returns a > > > non-json response, and/or contains information that in either > content or > > > encoding is meaningless to any sane client then it should also > return a > > > 410/501 and get left behind in Perl. > > > > > > On Tue, Nov 19, 2019 at 11:08 AM Rawlin Peters > <[email protected]> wrote: > > > > > > > While I appreciate and mostly agree with your essay on the > importance > > > > of not breaking users (or rather, breaking them in predictable > ways), > > > > I for one would really like to meet at some middle ground with > option > > > > 2. I'm not asking to remove 50 high-usage endpoints from 1.x. > IIRC > > > > I've only specifically proposed "leaving behind" a few specific > Perl > > > > endpoints that I believe serve no purpose, have no valid reason > to be > > > > used, and/or are dangerous to the system: > > > > > > > > - /riak/stats -- informational, was used for debugging the > initial > > > > Riak integration. This one might be pretty easy to rewrite > since it > > > > just calls a corresponding riak API and returns the result via > TO, so > > > > I could be on board with rewriting it if necessary. > > > > - /cachegroup_fallbacks -- these endpoints were released in 3.0 > before > > > > we managed to augment the existing /cachegroups API to handle > > > > fallbacks, so I could get on board with rewriting these if we > really > > > > think it's important. > > > > - /capabilities (all methods except for GET) -- capabilities are > > > > built-in, should not be changed at runtime > > > > - /api_capabilities (all methods except for GET) -- same > reasons as > > > > /capabilities > > > > - /cdns/{name}/configs/routing -- this was a half-baked attempt > at > > > > redoing CrConfig.json -- cannot possibly be of use to anyone in > its > > > > current state > > > > - the ATS config generation endpoints -- Rob actually proposed > to > > > > remove these from 1.x due to danger of being out of sync w/ > cache-side > > > > config generator > > > > - /hwinfo/dtdata -- AFAIK the "dtdata" endpoints were only > > > > used/required by the old Perl Mojolicious UI, no valid reason > to be > > > > using this as an external API client. In fact, I'm not even > sure _how_ > > > > this endpoint can be used outside of the Mojolicious > framework/UI. It > > > > calls the Mojolicious `render` function, which usually means > rendering > > > > an embedded Perl HTML template given some data. This is one I > would > > > > consider completely unreasonable to rewrite to Go. > > > > > > > > Can we get on board with leaving at least these 1.x endpoints > behind > > > > in the Perl? Doing a cursory skim over the remaining 1.x > endpoints to > > > > rewrite, it _seems_ like everything else would fall into the > "simpler > > > > to just rewrite rather than gain consensus to remove" category. > > > > > > > > - Rawlin > > > > > > > > On Tue, Nov 19, 2019 at 9:19 AM Robert O Butts > <[email protected]> wrote: > > > > > > > > > > >Option 1 - Graduate current API from 1.x to 2.x - allowing > us to leave > > > > > behind routes in 1.x that are dangerous or deprecated > > > > > >Option 2 - Rewrite all endpoints from Perl to Golang in > current > > > version > > > > as > > > > > we are doing now. > > > > > > > > > > +1 on either Option 1 or Option 2, without a strong > preference. Option > > > 1 > > > > > has advantages of letting us also safely remove the Cache > Config > > > > endpoints > > > > > and implement Layered Profiles. But Option 2 also has the > advantage of > > > > not > > > > > requiring users to upgrade. > > > > > > > > > > > Option 3 - Only rewrite endpoints from Perl to Golang in > current > > > > version > > > > > that we agree are not dangerous or should deprecate > > > > > > > > > > -1. As before, it breaks users unpredictably. > > > > > > > > > > >It will literally outright break clients (*all* of them, not > just > > > > *some*) > > > > > > > > > > This is technically correct: technically, new versions > 'break' all > > > users. > > > > > The difference is, it does it in a predictable, safe way that > users can > > > > > handle, inspect, and safely upgrade. > > > > > > > > > > >Can someone please explain to me why (A) artificially > graduating the > > > > > current 1.x API to 2.x, and removing the deprecated endpoints > from 1.x, > > > > is > > > > > better for API clients overall than (B) just removing > deprecated > > > > endpoints > > > > > from 1.x? Existing 1.x clients would gain absolutely nothing > from being > > > > > forced to update to 2.x like this. > > > > > > > > > > >for zero gain in value > > > > > > > > > > Users would gain something. There is a very specific gain. > > > > > > > > > > The gain is that the breakage is predictable. People > frequently upgrade > > > > > minor versions automatically, either running commands without > thinking, > > > > or > > > > > even automating the entire process. > > > > > > > > > > For TC, this applies to both CDN operators, and their > end-users. > > > > > > > > > > When the minor versions contain breaking changes, that > automatic > > > upgrade > > > > > then breaks. In this case, any scripts or tools they have > using the > > > > removed > > > > > endpoints will break. If those scripts are important, this > could bring > > > > down > > > > > their CDN, or break their Delivery Service, or otherwise > cause an > > > outage. > > > > > > > > > > Contrawise, users expect breaking changes in Major versions, > and don't > > > > > typically automatically upgrade them. Rather, they'll > manually inspect > > > > the > > > > > update, do testing, and check the changelog for breaking > changes. > > > > > > > > > > It's not a technical thing, it's a human thing. Humans > automatically > > > > > upgrade minor versions, and write scripts to do so, and > dependency > > > files > > > > to > > > > > allow it, because the software industry taught people that > minor > > > versions > > > > > shouldn't break anything. > > > > > > > > > > If we start breaking minor versions, initially users will > experience > > > > > breakage. Not all users, and not all catastrophic. But some > users will > > > > > experience catastrophic outages. And it will be our fault as > a project. > > > > And > > > > > after enough times, they'll fork TC, or find another product, > or write > > > > > their own; and stop participating in the community, or never > start. > > > > > > > > > > Then, once they've learned that we can't be trusted not to > break > > > things, > > > > > they'll experience the 2nd problem: where nothing can be > safely > > > upgraded, > > > > > and the smallest change requires long and expensive testing > to verify a > > > > > single minor version or even patch upgrade, and still risk > missing > > > > > something and bringing their CDN down. To be honest, we're > pretty much > > > > > already here. It already costs our dev and ops teams > unbelievable > > > amounts > > > > > of time and person-hours of testing for every minor upgrade. > That time > > > > far > > > > > exceeds the development time to just rewrite the small > endpoints we > > > don't > > > > > like anymore (and tangentially, also the time to write the > Integration > > > > and > > > > > E2E tests TC desperately needs). > > > > > > > > > > ATC has so much potential. There aren't many OSS CDNs out > there, and > > > > > there's certainly the demand. We have the potential to become > a huge > > > > > project, which for us existing users, translates into > potentially an > > > > > enormous amount of free development. But we'll never get > there if we > > > keep > > > > > breaking things. > > > > > > > > > > And for my particular organization (Comcast), we have a hard > enough > > > time > > > > > upgrading. Our upgrades, especially Traffic Ops, are > extremely painful, > > > > and > > > > > require enormous amounts of manual testing. Even then, we > frequently > > > have > > > > > issues that require rollback, sometimes even an outage; even > after > > > those > > > > > unbelievably amounts of developer and operator time testing. > We > > > > desperately > > > > > need a system that is safer and easier to upgrade. Breaking > changes are > > > > > always going to require more testing; but we can at least > make minor > > > > > versions safe to upgrade. > > > > > > > > > > > I know I am changing my initial stance but I am +1 on > rewriting all > > > the > > > > > Perl endpoints to Golang and not breaking the API contract we > have in > > > > > place. Most of the API routes to rewrite are rather simple to > port and > > > I > > > > > think would be faster than getting consensus on actually > deprecating it > > > > > from the mailing list. > > > > > > > > > > I'm inclined to agree with this. The things I've seen marked > > > "deprecated" > > > > > seem to mostly be small, and look like they'd be pretty easy > to > > > rewrite. > > > > > But I haven't seen the whole list. Is there a list somewhere, > so the > > > > > community can see it to better decide? > > > > > > > > > > > > > > > On Tue, Nov 19, 2019 at 7:24 AM Hoppal, Michael < > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > I would love to drive this to some type of consensus on > rewrite > > > > strategy > > > > > > as we have a good amount of resources (and growing) working > on the > > > > rewrite > > > > > > and I want to keep that momentum. > > > > > > > > > > > > Sounds like there is three options from the email thread > (please add > > > > any I > > > > > > missed) > > > > > > > > > > > > Option 1 - Graduate current API from 1.x to 2.x - allowing > us to > > > leave > > > > > > behind routes in 1.x that are dangerous or deprecated > > > > > > Option 2 - Rewrite all endpoints from Perl to Golang in > current > > > > version as > > > > > > we are doing now. > > > > > > Option 3 - Only rewrite endpoints from Perl to Golang in > current > > > > version > > > > > > that we agree are not dangerous or should deprecate > > > > > > > > > > > > I am +1 for option 2 (which I know is different from my > original > > > > stance). > > > > > > > > > > > > Which direction would others like to go? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Michael > > > > > > > > > > > > > > > > > > I know I am changing my initial stance but I am +1 on > rewriting all > > > the > > > > > > Perl endpoints to Golang and not breaking the API contract > we have in > > > > > > place. Most of the API routes to rewrite are rather simple > to port > > > and > > > > I > > > > > > think would be faster than getting consensus on actually > deprecating > > > it > > > > > > from the mailing list. > > > > > > > > > > > > On 11/16/19, 3:31 PM, "Rawlin Peters" <[email protected]> > wrote: > > > > > > > > > > > > > Moving to 2.x means that it's my fault as a consumer > of the API > > > > if I > > > > > > was still using the deprecated endpoint (and that change > should be > > > > part of > > > > > > a larger upgrade plan). Simply removing the deprecated > routes, > > > > > > retroactively, means I can't rely on having a transition > period to > > > get > > > > to > > > > > > 2.0 outside of a large completely down upgrade window. > > > > > > > > > > > > There would be a transition period either way. > Endpoints would be > > > > > > deprecated in ATC 4.0 and removed in 5.0. The release > notes for > > > 4.0 > > > > > > would clearly list out the deprecated endpoints. If > they have > > > > > > replacements, there would be instructions on how to > transition to > > > > the > > > > > > replacement. Deprecated endpoints would return alerts. > After > > > > upgrading > > > > > > to 4.0, operators should be well aware that certain > endpoints are > > > > > > being removed in 5.0, and it's on them to start using > new > > > > endpoints, > > > > > > change workflows, etc. > > > > > > > > > > > > By forcing operators to upgrade all of their clients to > start > > > using > > > > > > API 2.x, it just adds to their burden of upgrading when > that time > > > > > > comes. Rather than surgically fixing the clients of the > 2% of > > > > > > deprecated API endpoints before upgrading to ATC 5.0, > operators > > > > would > > > > > > have to fix every single client they have to use API > 2.x before > > > > > > upgrading to 5.0. Is that exercise really worth it? > Most people > > > > would > > > > > > probably be blissfully unaware that the "POST > > > > /api/1.1/capabilities" > > > > > > endpoint disappeared (among other deprecated > endpoints). Less > > > > footguns > > > > > > in the system, and they didn't even have to update > their clients. > > > > > > > > > > > > Anyways, circling back to Michael's original question, > it sounds > > > > like > > > > > > either option we choose, we at least have consensus on > this > > > > > > deprecation strategy: > > > > > > - The route IS NOT rewritten from Perl to Golang and a > > > deprecation > > > > > > notice is added to the alert response in the Perl > handler > > > > > > > > > > > > If we're content to force all API clients to update > their clients > > > > to > > > > > > 2.x for zero gain in value, at least we won't be > wasting work by > > > > > > rewriting deprecated Perl endpoints to Go. > > > > > > > > > > > > - Rawlin > > > > > > > > > > > > On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > Moving to 2.x means that it's my fault as a consumer > of the API > > > > if I > > > > > > was still using the deprecated endpoint (and that change > should be > > > > part of > > > > > > a larger upgrade plan). Simply removing the deprecated > routes, > > > > > > retroactively, means I can't rely on having a transition > period to > > > get > > > > to > > > > > > 2.0 outside of a large completely down upgrade window. > Depending on > > > > how > > > > > > the API is used, even in our own components such as TM, > that could > > > > prove > > > > > > disastrous. For the older minor revs like 1.1/1.2, I could > live with > > > > > > dropping those as part of a major project release > (including upgrade > > > > > > instructions outlining what releases you must be on > beforehand). > > > > Currently > > > > > > 1.3 I view as stable enough to code to or use a native > client library > > > > from, > > > > > > and 1.4 is my risk/overhead to take for new stuff as it's > still > > > > evolving > > > > > > and improving. Going to 1.5 or 2.0 moves that forward. > The path of > > > > least > > > > > > surprise for humans is to expect big shifts on those major > revs. > > > > Removal > > > > > > falls under that umbrella and gets as a consumer not to > strictly > > > focus > > > > on > > > > > > endpoint->endpoint moves, but rather workflow shifts and > maybe now I > > > > have > > > > > > to use the API a different way than before. I believe > getting this > > > > right > > > > > > is something that would greatly help us understand does X > component > > > at > > > > > > version 1 work with Y component at version 2, because now > you're > > > > asking do > > > > > > components X and Y both speak protocol A. With major > releases we can > > > > work > > > > > > through that each time, but it's error prone and requires > experts in > > > > all > > > > > > products to agree upon. If you do nothing but consume major > > > releases, > > > > and > > > > > > then conduct your own impact evaluation having read the > upgrade > > > > > > instructions thoroughly, maybe you'd be able to smoothly > transition. > > > > > > > > > > > > > > Jonathan G > > > > > > > > > > > > > > > > > > > > > On 11/15/19, 5:24 PM, "Rawlin Peters" > <[email protected]> > > > wrote: > > > > > > > > > > > > > > Can someone please explain to me why (A) > artificially > > > > graduating > > > > > > the > > > > > > > current 1.x API to 2.x, and removing the > deprecated > > > endpoints > > > > > > from > > > > > > > 1.x, is better for API clients overall than (B) > just > > > removing > > > > > > > deprecated endpoints from 1.x? Existing 1.x > clients would > > > > gain > > > > > > > absolutely nothing from being forced to update to > 2.x like > > > > this. > > > > > > > > > > > > > > The way I see the two choices: > > > > > > > A) break every single API client out there > > > > > > > B) break only clients of APIs that we've decided > upon as a > > > > > > community > > > > > > > to be deprecated > > > > > > > > > > > > > > IMO, when we have to break things, we should try > break the > > > > least > > > > > > > amount of clients possible. All I'm looking for > is a > > > valuable > > > > > > reason > > > > > > > to choose option A over option B, that isn't just > "because > > > > > > SemVer". > > > > > > > > > > > > > > - Rawlin > > > > > > > > > > > > > > On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew > > > (Contractor) > > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > > > If we decide to get rid of the API versioning > it makes > > > > sense > > > > > > to me that we would do that on a major version upgrade, > probably to > > > > the ATC > > > > > > version. But I think it would be too early to do it on the > 2.0 API > > > > version. > > > > > > I like this compromise +1. I was wondering why we were > stuck on the > > > > idea > > > > > > that 2.0 had to be an API rewrite. That seems like it’s a > long way > > > > out. 2.0 > > > > > > = deprecations and fixes. 3.0 or 5.0 = api rewrite and > maybe version > > > > > > convergence. > > > > > > > > > > > > > > > > On 11/13/19, 11:52 AM, "[email protected]" < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > Lemme lead with: making breaking changes on > a major > > > > > > version boundary > > > > > > > > gets +1 from me - no reason the API needs > to be fully > > > > > > redesigned for > > > > > > > > any particular number. > > > > > > > > > > > > > > > > But I just gotta bring this up again: if we > want to > > > be > > > > > > able to > > > > > > > > deprecate an endpoint for a major release > and remove > > > > it in > > > > > > the next > > > > > > > > without needing to deal with the messiness > of API > > > > > > versions, we could > > > > > > > > always just get rid of API versions. Major > ATC > > > > revisions > > > > > > should be > > > > > > > > easier to come by with our new release > strategy. > > > > > > > > > > > > > > > > On Wed, 2019-11-13 at 10:23 -0700, Rawlin > Peters > > > wrote: > > > > > > > > > > It is a compromise - it's an > inconvenience to > > > > users, > > > > > > forcing them > > > > > > > > > > to fix > > > > > > > > > scripts to upgrade. Which isn't good for > user > > > > experience > > > > > > either; but > > > > > > > > > it > > > > > > > > > doesn't outright break people. It's a > compromise > > > I'm > > > > > > willing to live > > > > > > > > > with. > > > > > > > > > > > > > > > > > > It will literally outright break clients > (*all* of > > > > them, > > > > > > not just > > > > > > > > > *some*). We have been down this road > before, so > > > I'll > > > > > > just repeat what > > > > > > > > > I said when this exact conversation came > up months > > > > > > ago[1]: > > > > > > > > > > > > > > > > > > "So it basically boils down to: > > > > > > > > > A) promote Go 1.x routes to 2.x and > eventually > > > break > > > > ALL > > > > > > users of the > > > > > > > > > 1.x API when the time comes to remove 1.x > > > > > > > > > B) deprecate and remove some 1.x Perl > routes and > > > > > > POSSIBLY break SOME > > > > > > > > > users" > > > > > > > > > > > > > > > > > > Either way we slice and dice it with > versioning, > > > > > > removing an endpoint > > > > > > > > > means that clients of that endpoint will > break. By > > > > > > graduating only > > > > > > > > > the > > > > > > > > > "good stuff" from 1.x to 2.x, and removing > > > endpoints > > > > > > from 1.x, we > > > > > > > > > will > > > > > > > > > unnecessarily break all 1.x clients when > the time > > > > comes > > > > > > to remove the > > > > > > > > > rest of 1.x. Instead, we can choose to > break only > > > > > > clients of 1.x APIs > > > > > > > > > that we already deemed dangerous, > unnecessary, > > > > useless, > > > > > > etc. enough > > > > > > > > > to > > > > > > > > > deprecate. > > > > > > > > > > > > > > > > > > Why break 100% of our API clients when we > can > > > choose > > > > to > > > > > > break only a > > > > > > > > > small number of clients that happen to use > > > > > > > > > dangerous/unnecessary/useless APIs? Why > would we > > > > break > > > > > > our API > > > > > > > > > clients > > > > > > > > > unnecessarily for no added value? There > is no sense > > > > in > > > > > > blindly > > > > > > > > > adhering to Semantic Versioning just for > the sake > > > of > > > > > > adhering to > > > > > > > > > Semantic Versioning. Instead, use it as a > > > guideline, > > > > and > > > > > > break the > > > > > > > > > rules where it makes the most sense to do > so. > > > > > > > > > > > > > > > > > > - Rawlin > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$ > > > > > > > > > > > > > > > > > > On Wed, Nov 13, 2019 at 9:43 AM Robert O > Butts < > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > But if it's really valuable to remove > these > > > > > > endpoints, maybe it's > > > > > > > > > > appropriate to bump the major version > and ask > > > > > > everybody to update > > > > > > > > > > their > > > > > > > > > > clients? > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > This sounds like the best compromise to > me. It > > > > > > addresses both sides > > > > > > > > > > - it > > > > > > > > > > lets us remove endpoints, and it also > prevents > > > > > > breaking users. > > > > > > > > > > > > > > > > > > > > It is a compromise - it's an > inconvenience to > > > > users, > > > > > > forcing them > > > > > > > > > > to fix > > > > > > > > > > scripts to upgrade. Which isn't good > for user > > > > > > experience either; > > > > > > > > > > but it > > > > > > > > > > doesn't outright break people. It's a > compromise > > > > I'm > > > > > > willing to > > > > > > > > > > live with. > > > > > > > > > > > > > > > > > > > > This actually lets us do at least 3 > things: > > > > > > > > > > 1. Remove "deprecated" endpoints. > > > > > > > > > > 2. Remove the unmaintained > dangerously-large > > > Cache > > > > > > Config > > > > > > > > > > endpoints. > > > > > > > > > > 3. Lets us do Layered Profiles, > changing Servers > > > > and > > > > > > DSes to have > > > > > > > > > > arrays of > > > > > > > > > > Profiles instead of a single Profile, > without > > > hacky > > > > > > backward- > > > > > > > > > > compatibility > > > > > > > > > > attempts. > > > > > > > > > > > > > > > > > > > > To clarify, `/api/2.0` would be > essentially the > > > > > > current API, just > > > > > > > > > > with the > > > > > > > > > > above breaking changes (possibly > others) which > > > > require > > > > > > a SemVer > > > > > > > > > > major > > > > > > > > > > version increase. And the big API > > > rewrite/redesign > > > > > > we've been > > > > > > > > > > calling "2.0" > > > > > > > > > > would become `/api/3.0` or some future > version. > > > > > > > > > > > > > > > > > > > > It's not perfect, but it seems to > address the > > > most > > > > > > concerns on all > > > > > > > > > > sides. > > > > > > > > > > Thoughts? Objections? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Nov 13, 2019 at 8:45 AM Chris > Lemmons < > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > For the endpoints, I agree in part > and disagree > > > > in > > > > > > part. > > > > > > > > > > > > > > > > > > > > > > Agree for these two: > > > > > > > > > > > - the endpoints are dangerous and > pose a risk > > > to > > > > the > > > > > > overall > > > > > > > > > > > Traffic > > > > > > > > > > > Control system > > > > > > > > > > > - the task of rewriting the endpoints > from Perl > > > > to > > > > > > Go represents > > > > > > > > > > > an > > > > > > > > > > > unreasonable amount of work for the > project > > > > > > > > > > > > > > > > > > > > > > And I would add: > > > > > > > > > > > - the endpoint does not serve its > intended > > > > purpose > > > > > > and cannot > > > > > > > > > > > reasonably > > > > > > > > > > > be used > > > > > > > > > > > - the endpoint has been made unsafe > or unusable > > > > by > > > > > > new features > > > > > > > > > > > added to TC > > > > > > > > > > > > > > > > > > > > > > Disagree for these two: > > > > > > > > > > > - the endpoints have been obsoleted > by other > > > > > > endpoints > > > > > > > > > > > - the endpoints don't seem to serve a > true > > > > purpose > > > > > > or don't seem > > > > > > > > > > > to be > > > > > > > > > > > valuable to the project anymore > > > > > > > > > > > > > > > > > > > > > > For the most part, obsoleted > endpoints can be > > > > > > maintained with a > > > > > > > > > > > shim > > > > > > > > > > > into the new code. If they would > require "an > > > > > > unreasonable amount > > > > > > > > > > > of > > > > > > > > > > > work", it might make sense to remove > them, > > > then. > > > > > > > > > > > > > > > > > > > > > > Things that don't seem valuable > usually fall > > > into > > > > > > another > > > > > > > > > > > category > > > > > > > > > > > like "cannot reasonably be used", but > if the > > > only > > > > > > reason for > > > > > > > > > > > deprecation is that the feature is > going away, > > > we > > > > > > should really > > > > > > > > > > > support it according to our promise. > > > > > > > > > > > > > > > > > > > > > > But if it's really valuable to remove > these > > > > > > endpoints, maybe it's > > > > > > > > > > > appropriate to bump the major version > and ask > > > > > > everybody to update > > > > > > > > > > > their clients? > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 5:27 PM > Rawlin Peters < > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > The real crux of the issue here is > whether or > > > > not > > > > > > we agree as a > > > > > > > > > > > > project that we can move forward > with this > > > plan > > > > > > that I believe > > > > > > > > > > > > to be a > > > > > > > > > > > > good compromise that meets most > concerns but > > > > > > "breaks the > > > > > > > > > > > > Semantic > > > > > > > > > > > > Versioning promise": > > > > > > > > > > > > 1) don't rewrite deprecated > endpoints from > > > > Perl to > > > > > > Go > > > > > > > > > > > > 2) in the Perl handler, add a > deprecation > > > > notice > > > > > > to the > > > > > > > > > > > > "alerts" > > > > > > > > > > > > section of the response > > > > > > > > > > > > 3) once a deprecated API has been > deprecated > > > > for > > > > > > one major > > > > > > > > > > > > release, > > > > > > > > > > > > delete it from the API in the > subsequent > > > major > > > > > > release (e.g. if > > > > > > > > > > > > deprecation notices are added and > released in > > > > ATC > > > > > > 4.x, we can > > > > > > > > > > > > remove > > > > > > > > > > > > the deprecated endpoint in ATC 5.x) > > > > > > > > > > > > > > > > > > > > > > > > Rob, from your latest proposal to > remove the > > > > ATS > > > > > > config API > > > > > > > > > > > > endpoints > > > > > > > > > > > > from 1.x, I thought we were past > this issue > > > > > > already and could > > > > > > > > > > > > finally > > > > > > > > > > > > agree to remove deprecated > endpoints from 1.x > > > > > > (following steps > > > > > > > > > > > > 1-3 > > > > > > > > > > > > above) without requiring an all-new > 2.x API. > > > > Are > > > > > > we not > > > > > > > > > > > > actually past > > > > > > > > > > > > that? You came up with valid > reasons for > > > > applying > > > > > > these steps > > > > > > > > > > > > to the > > > > > > > > > > > > ATS config API endpoints, and I > think it's > > > > fair to > > > > > > say that we > > > > > > > > > > > > can > > > > > > > > > > > > come up with valid reasons for > applying these > > > > > > steps to other > > > > > > > > > > > > deprecated API endpoints as well. > > > > > > > > > > > > > > > > > > > > > > > > It seems like the real problem is > actually > > > > > > agreeing on which > > > > > > > > > > > > _specific_ endpoints we should be > able to > > > > follow > > > > > > steps 1-3 for. > > > > > > > > > > > > IMO I > > > > > > > > > > > > think it should apply to any 1.x > endpoints > > > > that we > > > > > > have valid > > > > > > > > > > > > reasons > > > > > > > > > > > > for not carrying forward to Go, > including but > > > > not > > > > > > limited to: > > > > > > > > > > > > - the endpoints are dangerous and > pose a risk > > > > to > > > > > > the overall > > > > > > > > > > > > Traffic > > > > > > > > > > > > Control system > > > > > > > > > > > > - the endpoints have been obsoleted > by other > > > > > > endpoints > > > > > > > > > > > > - the endpoints don't seem to serve > a true > > > > purpose > > > > > > or don't > > > > > > > > > > > > seem to be > > > > > > > > > > > > valuable to the project anymore > > > > > > > > > > > > - the task of rewriting the > endpoints from > > > > Perl to > > > > > > Go > > > > > > > > > > > > represents an > > > > > > > > > > > > unreasonable amount of work for the > project > > > > > > > > > > > > > > > > > > > > > > > > Obviously most of these reasons are > > > > subjective, so > > > > > > we'd have to > > > > > > > > > > > > come > > > > > > > > > > > > to some level of consensus on > deprecating > > > > > > particular endpoints > > > > > > > > > > > > first, > > > > > > > > > > > > but we should be able to follow > steps 1-3 for > > > > > > whatever > > > > > > > > > > > > endpoints we > > > > > > > > > > > > decide. > > > > > > > > > > > > > > > > > > > > > > > > Making breaking changes shouldn't > be taken > > > > > > lightly, but we need > > > > > > > > > > > > a path > > > > > > > > > > > > forward that allows some > calculated, breaking > > > > > > changes where > > > > > > > > > > > > necessary. > > > > > > > > > > > > Otherwise, we are unnecessarily > holding the > > > > > > project back. > > > > > > > > > > > > > > > > > > > > > > > > - Rawlin > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 11:44 AM > Jeremy > > > > Mitchell < > > > > > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > Yeah, technically a "rewrite" of > 1.x means > > > > > > shifting the > > > > > > > > > > > > > implementation > > > > > > > > > > > from > > > > > > > > > > > > > Perl to Go for ALL of 1.x. If you > leave a > > > > subset > > > > > > of 1.x in > > > > > > > > > > > > > Perl, then > > > > > > > > > > > you > > > > > > > > > > > > > have to call it a "partial > rewrite". > > > > > > > > > > > > > > > > > > > > > > > > > > So, assuming we want to do a full > rewrite > > > (no > > > > > > more perl for > > > > > > > > > > > > > the api > > > > > > > > > > > > > whatsoever), I think that means > every 1.x > > > api > > > > > > endpoint (good > > > > > > > > > > > > > or bad) > > > > > > > > > > > needs > > > > > > > > > > > > > to be rewritten to Go so we can > remove the > > > > > > dependency on > > > > > > > > > > > > > perl. > > > > > > > > > > > > > > > > > > > > > > > > > > However, I still like marking > many as > > > > > > "deprecated" so we > > > > > > > > > > > > > train users > > > > > > > > > > > to use > > > > > > > > > > > > > "better" endpoints that we > envision will > > > > exist > > > > > > in 2.x and > > > > > > > > > > > > > beyond. > > > > > > > > > > > > > > > > > > > > > > > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 11:22 AM > Hoppal, > > > > Michael > > > > > > < > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > We do not have a lot of > deprecated routes > > > > as > > > > > > of now as we > > > > > > > > > > > > > > focused on > > > > > > > > > > > > > > routes that are more used first. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I know on a lot of the > remaining rewrites > > > > > > there has been > > > > > > > > > > > > > > discussion > > > > > > > > > > > on the > > > > > > > > > > > > > > matching Github issues on > potential > > > > > > deprecation (I cannot > > > > > > > > > > > > > > think of > > > > > > > > > > > the > > > > > > > > > > > > > > exact count but enough to > warrant this > > > > email > > > > > > __) > > > > > > > > > > > > > > > > > > > > > > > > > > > > It is key to note that I am not > arguing > > > for > > > > > > removal of > > > > > > > > > > > > > > endpoints > > > > > > > > > > > within > > > > > > > > > > > > > > the current API version but if > there is > > > an > > > > > > endpoint we > > > > > > > > > > > > > > agree should > > > > > > > > > > > be > > > > > > > > > > > > > > deprecated going forward to put > a message > > > > > > within the > > > > > > > > > > > > > > response and on > > > > > > > > > > > the > > > > > > > > > > > > > > docs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > And thanks for the input sounds > like you > > > > are > > > > > > +1 on > > > > > > > > > > > > > > rewriting all > > > > > > > > > > > routes to > > > > > > > > > > > > > > Golang as Brennan is. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 11/12/19, 9:06 AM, "Robert O > Butts" < > > > > > > [email protected]> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Whether we rewrite a route > in Go is > > > an > > > > > > implementation > > > > > > > > > > > > > > detail. To > > > > > > > > > > > the > > > > > > > > > > > > > > interface, to users, it > doesn't > > > matter > > > > > > whether a route > > > > > > > > > > > > > > is > > > > > > > > > > > rewritten or > > > > > > > > > > > > > > not. > > > > > > > > > > > > > > > > > > > > > > > > > > > > But our API follows Semantic > > > > Versioning, > > > > > > in order to > > > > > > > > > > > > > > not break > > > > > > > > > > > users. > > > > > > > > > > > > > > We > > > > > > > > > > > > > > can't just remove endpoints > that some > > > > of > > > > > > us don't use, > > > > > > > > > > > > > > and > > > > > > > > > > > assume other > > > > > > > > > > > > > > people don't, maybe people > who never > > > > speak > > > > > > up on the > > > > > > > > > > > > > > mailing > > > > > > > > > > > list. > > > > > > > > > > > > > > We'll > > > > > > > > > > > > > > never gain a big userbase > if we keep > > > > > > breaking users. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Per the _project_ SemVer, > once we > > > have > > > > API > > > > > > 2.0, we can > > > > > > > > > > > > > > deprecate > > > > > > > > > > > API > > > > > > > > > > > > > > 1.x, > > > > > > > > > > > > > > and in the next major > _project_ > > > > release, > > > > > > remove API > > > > > > > > > > > > > > 1.x. > > > > > > > > > > > Irrespective > > > > > > > > > > > > > > of > > > > > > > > > > > > > > Perl or Go. > > > > > > > > > > > > > > > > > > > > > > > > > > > > My big concern is, API 2.0 > is a big > > > > > > project. How long > > > > > > > > > > > > > > has the > > > > > > > > > > > rewrite > > > > > > > > > > > > > > to Go > > > > > > > > > > > > > > taken? Do we really believe > designing > > > > and > > > > > > implementing > > > > > > > > > > > > > > a > > > > > > > > > > > completely > > > > > > > > > > > > > > new API > > > > > > > > > > > > > > will be any less time? > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't want killing Perl > to have to > > > > wait > > > > > > on that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I know it feels like a > waste to > > > rewrite > > > > > > routes that you > > > > > > > > > > > > > > don't > > > > > > > > > > > use, and > > > > > > > > > > > > > > probably few people do. But > that's > > > the > > > > > > cost of a stable > > > > > > > > > > > > > > project. > > > > > > > > > > > How > > > > > > > > > > > > > > many > > > > > > > > > > > > > > "deprecated" routes are > there? If it > > > > comes > > > > > > down to > > > > > > > > > > > > > > taking the > > > > > > > > > > > > > > development > > > > > > > > > > > > > > time to rewrite them so we > can kill > > > > Perl > > > > > > faster, or > > > > > > > > > > > > > > leaving Perl > > > > > > > > > > > > > > around, I > > > > > > > > > > > > > > vote we just do the work > and kill > > > Perl. > > > > > > > > > > > > > > > > > > > > > > > > > > > > >If we don't rewrite them, > then Perl > > > > will > > > > > > last until > > > > > > > > > > > > > > API 2.0 has > > > > > > > > > > > been > > > > > > > > > > > > > > designed, released and then > *another > > > > full > > > > > > major release > > > > > > > > > > > > > > cycle*. > > > > > > > > > > > That's > > > > > > > > > > > > > > way > > > > > > > > > > > > > > too long to have two > codebases for > > > the > > > > > > same component, > > > > > > > > > > > > > > IMO, > > > > > > > > > > > especially > > > > > > > > > > > > > > since the rewrite is > already 50% > > > > complete. > > > > > > > > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at > 8:54 AM ocket > > > > 8888 > > > > > > < > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I vote that by and large > we DO > > > > rewrite > > > > > > them, with > > > > > > > > > > > > > > exceptions > > > > > > > > > > > for > > > > > > > > > > > > > > routes > > > > > > > > > > > > > > > that just plain don't > work, even in > > > > > > Perl. Those are > > > > > > > > > > > > > > few, > > > > > > > > > > > though. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > If we don't rewrite them, > then Perl > > > > will > > > > > > last until > > > > > > > > > > > > > > API 2.0 > > > > > > > > > > > has been > > > > > > > > > > > > > > > designed, released and > then > > > *another > > > > > > full major > > > > > > > > > > > > > > release cycle*. > > > > > > > > > > > > > > That's way > > > > > > > > > > > > > > > too long to have two > codebases for > > > > the > > > > > > same > > > > > > > > > > > > > > component, IMO, > > > > > > > > > > > > > > especially > > > > > > > > > > > > > > > since the rewrite is > already 50% > > > > > > complete. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at > 8:43 AM > > > > Hoppal, > > > > > > Michael < > > > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As the Traffic Ops API > is being > > > > > > rewritten from Perl > > > > > > > > > > > > > > to Golang > > > > > > > > > > > > > > there has > > > > > > > > > > > > > > > > been several routes > that have > > > been > > > > > > deprecated and > > > > > > > > > > > > > > probably > > > > > > > > > > > more to > > > > > > > > > > > > > > come. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the deprecation > efforts I have > > > > seen > > > > > > two > > > > > > > > > > > > > > strategies: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * The route IS NOT > rewritten > > > > from > > > > > > Perl to > > > > > > > > > > > > > > Golang and a > > > > > > > > > > > > > > deprecation > > > > > > > > > > > > > > > > notice is added to the > alert > > > > response > > > > > > in the Perl > > > > > > > > > > > > > > handler > > > > > > > > > > > > > > > > * The route IS > rewritten from > > > > Perl > > > > > > to Golang > > > > > > > > > > > > > > and a > > > > > > > > > > > deprecation > > > > > > > > > > > > > > notice > > > > > > > > > > > > > > > > is added to the alert > response in > > > > the > > > > > > Golang > > > > > > > > > > > > > > handler > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think we should have > > > consistency > > > > in > > > > > > our approach > > > > > > > > > > > > > > and > > > > > > > > > > > wanted to > > > > > > > > > > > > > > get > > > > > > > > > > > > > > > > people’s thoughts. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would vote that we do > not > > > > rewrite a > > > > > > deprecated > > > > > > > > > > > > > > route from > > > > > > > > > > > Perl to > > > > > > > > > > > > > > > Golang. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Michael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
