I would love to drive this to some type of consensus on rewrite strategy as we have a good amount of resources (and growing) working on the rewrite and I want to keep that momentum.
Sounds like there is three options from the email thread (please add any I missed) Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave behind routes in 1.x that are dangerous or deprecated Option 2 - Rewrite all endpoints from Perl to Golang in current version as we are doing now. Option 3 - Only rewrite endpoints from Perl to Golang in current version that we agree are not dangerous or should deprecate I am +1 for option 2 (which I know is different from my original stance). Which direction would others like to go? Thanks, Michael I know I am changing my initial stance but I am +1 on rewriting all the Perl endpoints to Golang and not breaking the API contract we have in place. Most of the API routes to rewrite are rather simple to port and I think would be faster than getting consensus on actually deprecating it from the mailing list. On 11/16/19, 3:31 PM, "Rawlin Peters" <[email protected]> wrote: > Moving to 2.x means that it's my fault as a consumer of the API if I was still using the deprecated endpoint (and that change should be part of a larger upgrade plan). Simply removing the deprecated routes, retroactively, means I can't rely on having a transition period to get to 2.0 outside of a large completely down upgrade window. There would be a transition period either way. Endpoints would be deprecated in ATC 4.0 and removed in 5.0. The release notes for 4.0 would clearly list out the deprecated endpoints. If they have replacements, there would be instructions on how to transition to the replacement. Deprecated endpoints would return alerts. After upgrading to 4.0, operators should be well aware that certain endpoints are being removed in 5.0, and it's on them to start using new endpoints, change workflows, etc. By forcing operators to upgrade all of their clients to start using API 2.x, it just adds to their burden of upgrading when that time comes. Rather than surgically fixing the clients of the 2% of deprecated API endpoints before upgrading to ATC 5.0, operators would have to fix every single client they have to use API 2.x before upgrading to 5.0. Is that exercise really worth it? Most people would probably be blissfully unaware that the "POST /api/1.1/capabilities" endpoint disappeared (among other deprecated endpoints). Less footguns in the system, and they didn't even have to update their clients. Anyways, circling back to Michael's original question, it sounds like either option we choose, we at least have consensus on this deprecation strategy: - The route IS NOT rewritten from Perl to Golang and a deprecation notice is added to the alert response in the Perl handler If we're content to force all API clients to update their clients to 2.x for zero gain in value, at least we won't be wasting work by rewriting deprecated Perl endpoints to Go. - Rawlin On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan <[email protected]> wrote: > > Moving to 2.x means that it's my fault as a consumer of the API if I was still using the deprecated endpoint (and that change should be part of a larger upgrade plan). Simply removing the deprecated routes, retroactively, means I can't rely on having a transition period to get to 2.0 outside of a large completely down upgrade window. Depending on how the API is used, even in our own components such as TM, that could prove disastrous. For the older minor revs like 1.1/1.2, I could live with dropping those as part of a major project release (including upgrade instructions outlining what releases you must be on beforehand). Currently 1.3 I view as stable enough to code to or use a native client library from, and 1.4 is my risk/overhead to take for new stuff as it's still evolving and improving. Going to 1.5 or 2.0 moves that forward. The path of least surprise for humans is to expect big shifts on those major revs. Removal falls under that umbrella and gets as a consumer not to strictly focus on endpoint->endpoint moves, but rather workflow shifts and maybe now I have to use the API a different way than before. I believe getting this right is something that would greatly help us understand does X component at version 1 work with Y component at version 2, because now you're asking do components X and Y both speak protocol A. With major releases we can work through that each time, but it's error prone and requires experts in all products to agree upon. If you do nothing but consume major releases, and then conduct your own impact evaluation having read the upgrade instructions thoroughly, maybe you'd be able to smoothly transition. > > Jonathan G > > > On 11/15/19, 5:24 PM, "Rawlin Peters" <[email protected]> wrote: > > Can someone please explain to me why (A) artificially graduating the > current 1.x API to 2.x, and removing the deprecated endpoints from > 1.x, is better for API clients overall than (B) just removing > deprecated endpoints from 1.x? Existing 1.x clients would gain > absolutely nothing from being forced to update to 2.x like this. > > The way I see the two choices: > A) break every single API client out there > B) break only clients of APIs that we've decided upon as a community > to be deprecated > > IMO, when we have to break things, we should try break the least > amount of clients possible. All I'm looking for is a valuable reason > to choose option A over option B, that isn't just "because SemVer". > > - Rawlin > > On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew (Contractor) > <[email protected]> wrote: > > > > If we decide to get rid of the API versioning it makes sense to me that we would do that on a major version upgrade, probably to the ATC version. But I think it would be too early to do it on the 2.0 API version. I like this compromise +1. I was wondering why we were stuck on the idea that 2.0 had to be an API rewrite. That seems like it’s a long way out. 2.0 = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version convergence. > > > > On 11/13/19, 11:52 AM, "[email protected]" <[email protected]> wrote: > > > > Lemme lead with: making breaking changes on a major version boundary > > gets +1 from me - no reason the API needs to be fully redesigned for > > any particular number. > > > > But I just gotta bring this up again: if we want to be able to > > deprecate an endpoint for a major release and remove it in the next > > without needing to deal with the messiness of API versions, we could > > always just get rid of API versions. Major ATC revisions should be > > easier to come by with our new release strategy. > > > > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters wrote: > > > > It is a compromise - it's an inconvenience to users, forcing them > > > > to fix > > > scripts to upgrade. Which isn't good for user experience either; but > > > it > > > doesn't outright break people. It's a compromise I'm willing to live > > > with. > > > > > > It will literally outright break clients (*all* of them, not just > > > *some*). We have been down this road before, so I'll just repeat what > > > I said when this exact conversation came up months ago[1]: > > > > > > "So it basically boils down to: > > > A) promote Go 1.x routes to 2.x and eventually break ALL users of the > > > 1.x API when the time comes to remove 1.x > > > B) deprecate and remove some 1.x Perl routes and POSSIBLY break SOME > > > users" > > > > > > Either way we slice and dice it with versioning, removing an endpoint > > > means that clients of that endpoint will break. By graduating only > > > the > > > "good stuff" from 1.x to 2.x, and removing endpoints from 1.x, we > > > will > > > unnecessarily break all 1.x clients when the time comes to remove the > > > rest of 1.x. Instead, we can choose to break only clients of 1.x APIs > > > that we already deemed dangerous, unnecessary, useless, etc. enough > > > to > > > deprecate. > > > > > > Why break 100% of our API clients when we can choose to break only a > > > small number of clients that happen to use > > > dangerous/unnecessary/useless APIs? Why would we break our API > > > clients > > > unnecessarily for no added value? There is no sense in blindly > > > adhering to Semantic Versioning just for the sake of adhering to > > > Semantic Versioning. Instead, use it as a guideline, and break the > > > rules where it makes the most sense to do so. > > > > > > - Rawlin > > > > > > [1] > > > https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$ > > > > > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <[email protected]> > > > wrote: > > > > > But if it's really valuable to remove these endpoints, maybe it's > > > > appropriate to bump the major version and ask everybody to update > > > > their > > > > clients? > > > > > > > > +1 > > > > > > > > This sounds like the best compromise to me. It addresses both sides > > > > - it > > > > lets us remove endpoints, and it also prevents breaking users. > > > > > > > > It is a compromise - it's an inconvenience to users, forcing them > > > > to fix > > > > scripts to upgrade. Which isn't good for user experience either; > > > > but it > > > > doesn't outright break people. It's a compromise I'm willing to > > > > live with. > > > > > > > > This actually lets us do at least 3 things: > > > > 1. Remove "deprecated" endpoints. > > > > 2. Remove the unmaintained dangerously-large Cache Config > > > > endpoints. > > > > 3. Lets us do Layered Profiles, changing Servers and DSes to have > > > > arrays of > > > > Profiles instead of a single Profile, without hacky backward- > > > > compatibility > > > > attempts. > > > > > > > > To clarify, `/api/2.0` would be essentially the current API, just > > > > with the > > > > above breaking changes (possibly others) which require a SemVer > > > > major > > > > version increase. And the big API rewrite/redesign we've been > > > > calling "2.0" > > > > would become `/api/3.0` or some future version. > > > > > > > > It's not perfect, but it seems to address the most concerns on all > > > > sides. > > > > Thoughts? Objections? > > > > > > > > > > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <[email protected]> > > > > wrote: > > > > > > > > > For the endpoints, I agree in part and disagree in part. > > > > > > > > > > Agree for these two: > > > > > - the endpoints are dangerous and pose a risk to the overall > > > > > Traffic > > > > > Control system > > > > > - the task of rewriting the endpoints from Perl to Go represents > > > > > an > > > > > unreasonable amount of work for the project > > > > > > > > > > And I would add: > > > > > - the endpoint does not serve its intended purpose and cannot > > > > > reasonably > > > > > be used > > > > > - the endpoint has been made unsafe or unusable by new features > > > > > added to TC > > > > > > > > > > Disagree for these two: > > > > > - the endpoints have been obsoleted by other endpoints > > > > > - the endpoints don't seem to serve a true purpose or don't seem > > > > > to be > > > > > valuable to the project anymore > > > > > > > > > > For the most part, obsoleted endpoints can be maintained with a > > > > > shim > > > > > into the new code. If they would require "an unreasonable amount > > > > > of > > > > > work", it might make sense to remove them, then. > > > > > > > > > > Things that don't seem valuable usually fall into another > > > > > category > > > > > like "cannot reasonably be used", but if the only reason for > > > > > deprecation is that the feature is going away, we should really > > > > > support it according to our promise. > > > > > > > > > > But if it's really valuable to remove these endpoints, maybe it's > > > > > appropriate to bump the major version and ask everybody to update > > > > > their clients? > > > > > > > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <[email protected]> > > > > > wrote: > > > > > > The real crux of the issue here is whether or not we agree as a > > > > > > project that we can move forward with this plan that I believe > > > > > > to be a > > > > > > good compromise that meets most concerns but "breaks the > > > > > > Semantic > > > > > > Versioning promise": > > > > > > 1) don't rewrite deprecated endpoints from Perl to Go > > > > > > 2) in the Perl handler, add a deprecation notice to the > > > > > > "alerts" > > > > > > section of the response > > > > > > 3) once a deprecated API has been deprecated for one major > > > > > > release, > > > > > > delete it from the API in the subsequent major release (e.g. if > > > > > > deprecation notices are added and released in ATC 4.x, we can > > > > > > remove > > > > > > the deprecated endpoint in ATC 5.x) > > > > > > > > > > > > Rob, from your latest proposal to remove the ATS config API > > > > > > endpoints > > > > > > from 1.x, I thought we were past this issue already and could > > > > > > finally > > > > > > agree to remove deprecated endpoints from 1.x (following steps > > > > > > 1-3 > > > > > > above) without requiring an all-new 2.x API. Are we not > > > > > > actually past > > > > > > that? You came up with valid reasons for applying these steps > > > > > > to the > > > > > > ATS config API endpoints, and I think it's fair to say that we > > > > > > can > > > > > > come up with valid reasons for applying these steps to other > > > > > > deprecated API endpoints as well. > > > > > > > > > > > > It seems like the real problem is actually agreeing on which > > > > > > _specific_ endpoints we should be able to follow steps 1-3 for. > > > > > > IMO I > > > > > > think it should apply to any 1.x endpoints that we have valid > > > > > > reasons > > > > > > for not carrying forward to Go, including but not limited to: > > > > > > - the endpoints are dangerous and pose a risk to the overall > > > > > > Traffic > > > > > > Control system > > > > > > - the endpoints have been obsoleted by other endpoints > > > > > > - the endpoints don't seem to serve a true purpose or don't > > > > > > seem to be > > > > > > valuable to the project anymore > > > > > > - the task of rewriting the endpoints from Perl to Go > > > > > > represents an > > > > > > unreasonable amount of work for the project > > > > > > > > > > > > Obviously most of these reasons are subjective, so we'd have to > > > > > > come > > > > > > to some level of consensus on deprecating particular endpoints > > > > > > first, > > > > > > but we should be able to follow steps 1-3 for whatever > > > > > > endpoints we > > > > > > decide. > > > > > > > > > > > > Making breaking changes shouldn't be taken lightly, but we need > > > > > > a path > > > > > > forward that allows some calculated, breaking changes where > > > > > > necessary. > > > > > > Otherwise, we are unnecessarily holding the project back. > > > > > > > > > > > > - Rawlin > > > > > > > > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell < > > > > > > [email protected]> > > > > > wrote: > > > > > > > Yeah, technically a "rewrite" of 1.x means shifting the > > > > > > > implementation > > > > > from > > > > > > > Perl to Go for ALL of 1.x. If you leave a subset of 1.x in > > > > > > > Perl, then > > > > > you > > > > > > > have to call it a "partial rewrite". > > > > > > > > > > > > > > So, assuming we want to do a full rewrite (no more perl for > > > > > > > the api > > > > > > > whatsoever), I think that means every 1.x api endpoint (good > > > > > > > or bad) > > > > > needs > > > > > > > to be rewritten to Go so we can remove the dependency on > > > > > > > perl. > > > > > > > > > > > > > > However, I still like marking many as "deprecated" so we > > > > > > > train users > > > > > to use > > > > > > > "better" endpoints that we envision will exist in 2.x and > > > > > > > beyond. > > > > > > > > > > > > > > Jeremy > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael < > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > We do not have a lot of deprecated routes as of now as we > > > > > > > > focused on > > > > > > > > routes that are more used first. > > > > > > > > > > > > > > > > I know on a lot of the remaining rewrites there has been > > > > > > > > discussion > > > > > on the > > > > > > > > matching Github issues on potential deprecation (I cannot > > > > > > > > think of > > > > > the > > > > > > > > exact count but enough to warrant this email __) > > > > > > > > > > > > > > > > It is key to note that I am not arguing for removal of > > > > > > > > endpoints > > > > > within > > > > > > > > the current API version but if there is an endpoint we > > > > > > > > agree should > > > > > be > > > > > > > > deprecated going forward to put a message within the > > > > > > > > response and on > > > > > the > > > > > > > > docs. > > > > > > > > > > > > > > > > And thanks for the input sounds like you are +1 on > > > > > > > > rewriting all > > > > > routes to > > > > > > > > Golang as Brennan is. > > > > > > > > > > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <[email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > Whether we rewrite a route in Go is an implementation > > > > > > > > detail. To > > > > > the > > > > > > > > interface, to users, it doesn't matter whether a route > > > > > > > > is > > > > > rewritten or > > > > > > > > not. > > > > > > > > > > > > > > > > But our API follows Semantic Versioning, in order to > > > > > > > > not break > > > > > users. > > > > > > > > We > > > > > > > > can't just remove endpoints that some of us don't use, > > > > > > > > and > > > > > assume other > > > > > > > > people don't, maybe people who never speak up on the > > > > > > > > mailing > > > > > list. > > > > > > > > We'll > > > > > > > > never gain a big userbase if we keep breaking users. > > > > > > > > > > > > > > > > Per the _project_ SemVer, once we have API 2.0, we can > > > > > > > > deprecate > > > > > API > > > > > > > > 1.x, > > > > > > > > and in the next major _project_ release, remove API > > > > > > > > 1.x. > > > > > Irrespective > > > > > > > > of > > > > > > > > Perl or Go. > > > > > > > > > > > > > > > > My big concern is, API 2.0 is a big project. How long > > > > > > > > has the > > > > > rewrite > > > > > > > > to Go > > > > > > > > taken? Do we really believe designing and implementing > > > > > > > > a > > > > > completely > > > > > > > > new API > > > > > > > > will be any less time? > > > > > > > > > > > > > > > > I don't want killing Perl to have to wait on that. > > > > > > > > > > > > > > > > I know it feels like a waste to rewrite routes that you > > > > > > > > don't > > > > > use, and > > > > > > > > probably few people do. But that's the cost of a stable > > > > > > > > project. > > > > > How > > > > > > > > many > > > > > > > > "deprecated" routes are there? If it comes down to > > > > > > > > taking the > > > > > > > > development > > > > > > > > time to rewrite them so we can kill Perl faster, or > > > > > > > > leaving Perl > > > > > > > > around, I > > > > > > > > vote we just do the work and kill Perl. > > > > > > > > > > > > > > > > >If we don't rewrite them, then Perl will last until > > > > > > > > API 2.0 has > > > > > been > > > > > > > > designed, released and then *another full major release > > > > > > > > cycle*. > > > > > That's > > > > > > > > way > > > > > > > > too long to have two codebases for the same component, > > > > > > > > IMO, > > > > > especially > > > > > > > > since the rewrite is already 50% complete. > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 < > > > > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I vote that by and large we DO rewrite them, with > > > > > > > > exceptions > > > > > for > > > > > > > > routes > > > > > > > > > that just plain don't work, even in Perl. Those are > > > > > > > > few, > > > > > though. > > > > > > > > > > > > > > > > > > If we don't rewrite them, then Perl will last until > > > > > > > > API 2.0 > > > > > has been > > > > > > > > > designed, released and then *another full major > > > > > > > > release cycle*. > > > > > > > > That's way > > > > > > > > > too long to have two codebases for the same > > > > > > > > component, IMO, > > > > > > > > especially > > > > > > > > > since the rewrite is already 50% complete. > > > > > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael < > > > > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > As the Traffic Ops API is being rewritten from Perl > > > > > > > > to Golang > > > > > > > > there has > > > > > > > > > > been several routes that have been deprecated and > > > > > > > > probably > > > > > more to > > > > > > > > come. > > > > > > > > > > > > > > > > > > > > In the deprecation efforts I have seen two > > > > > > > > strategies: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * The route IS NOT rewritten from Perl to > > > > > > > > Golang and a > > > > > > > > deprecation > > > > > > > > > > notice is added to the alert response in the Perl > > > > > > > > handler > > > > > > > > > > * The route IS rewritten from Perl to Golang > > > > > > > > and a > > > > > deprecation > > > > > > > > notice > > > > > > > > > > is added to the alert response in the Golang > > > > > > > > handler > > > > > > > > > > > > > > > > > > > > I think we should have consistency in our approach > > > > > > > > and > > > > > wanted to > > > > > > > > get > > > > > > > > > > people’s thoughts. > > > > > > > > > > > > > > > > > > > > I would vote that we do not rewrite a deprecated > > > > > > > > route from > > > > > Perl to > > > > > > > > > Golang. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Michael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
