origin virtual hosting relies on the Host header. So if the request from the client to the reverse proxy has a Host header and maintain_pristine_host_hdr is not enabled, remap will rewrite the Host header along with the request url. So in this case, remapping to a CNAME should work.
> On Jan 29, 2019, at 6:02 PM, Steve Malenfant <[email protected]> wrote: > > CNAME might not necessary work if the origin is using virtual hosting. It > would need to be configured as well. Might work with some origin? But not > all of them. > > On Tue, Jan 29, 2019 at 7:40 PM Fieck, Brennan <[email protected]> > wrote: > >> The issue is inherent with the structure of ATS configuration files. If I >> understand correctly - someone correct me if I don't - >> you can't have more than one rule for a "primary destination" in the >> parent.config. Or, you can, but only one of them will >> wind up catching all of your matches. Which means that if you have >> multiple Delivery Services that use the same origin FQDN >> (dest_domain) then you get proper behavior out of ATS if and only if the >> resulting rules would be identical. I'd highly >> recommend that you not move forward using duplicate origin FQDNs on your >> Delivery Services, because it probably won't >> do what you want. >> >> I'm not sure what you mean about the cachekeys. The Delivery Service URLs >> should still be unique, so you shouldn't run >> into a collision in the cache keys. >> >> Origins are an object. They're also a cachegroup type. And a profile type. >> And a server type. And a URL field on a Delivery Service. >> >> The solution that's been used in the past to use the same actual origin >> machine to service multiple Delivery Services safely >> is to add a "Canonical Name" DNS record to the ATC internal zone for each >> Delivery Service. The idea is to have a unique name >> for each Delivery Service, but they all resolve to the same IP address. >> ________________________________________ >> From: Steve Malenfant <[email protected]> >> Sent: Tuesday, January 29, 2019 4:32 PM >> To: [email protected] >> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe Delivery Services >> produces indeterminate parent.config >> >> We use and will use the same origin for multiple DS. It’s an inherent part >> of the design of some part of our delivery. >> >> Wouldn’t be less of a problem using cachekeys for different delivery >> services? >> >> I would rather see origin as an object and not just a simple URL. I’ve seen >> this in the past in other systems. >> >> I don’t believe we can create multiple http services endpoint for a single >> DS like we can for DNS type delivery service. This would solve it for us. >> >> Steve >> >> On Thu, Jan 24, 2019 at 11:22 AM Gray, Jonathan <[email protected] >>> >> wrote: >> >>> Even that has a strong element of confusion when in a self-service >>> multi-tenancy world. Multiple tenants can use the same origins for >> things >>> like akami, aws, internal load balancers and any communication about the >>> origin already being used will be confusing. >>> >>> Jonathan G >>> >>> >>> On 1/23/19, 7:37 PM, "Gelinas, Derek" <[email protected]> >> wrote: >>> >>> Perhaps we could solve both issues and just allow it to happen on DS >>> config but throw up a message when it is saved that says "hey we're going >>> to do this but it's really not without an element of risk and ats isn't >>> designed to work properly with multiple identical origins. We recommend >> you >>> use an alternative fqdn blah blah blah." >>> >>> Something along those lines, anyway. >>> >>> Derek >>> >>>> On Jan 23, 2019, at 7:18 PM, Rawlin Peters < >> [email protected]> >>> wrote: >>>> >>>> Simply prohibiting duplicate origin FQDNs has been vetoed now, so >> we >>>> can't really move forward with that unless the vetoers change their >>>> minds and remove their -1s. >>>> >>>> Currently that leaves us with checking at DS create/update whether >> or >>>> not it would conflict with a shared origin due to different DS >> types, >>>> qstring config, etc. I am -1 on that idea since I think it will be >>>> difficult to maintain and enumerate all the possible conflicting >>>> cases, will present confusing error messages to the user to which >> the >>>> solution would just be "create a CNAME record" anyways, and that >> the >>>> ATS parent.config should really be per-remap rather than "global". >> I >>>> know there has been some discussion in the ATS community about >> making >>>> parent.config per-remap, but I don't have much information there. >>>> >>>> That is to say, I don't think there is a great solution to this >>>> problem right now that we can all agree upon, but maybe our efforts >>>> would be better spent in the ATS community by making a per-remap >>>> parent.config. Then the duplicate origin problem in Traffic Control >>>> would go away. >>>> >>>> Another option could be to have some kind of generic rules engine >>> that >>>> would allow an organization to create some set of rules that DSes >>>> would have to abide by. For example, no duplicate origins, an >> origin >>>> can't be in a list of known malicious domains, you can't use the DS >>>> type "HTTP_NO_CACHE", maxDnsAnswers must be < 5, TTLs must be > 30 >>> and >>>> < 6000, protocol must be HTTPS, intitialDispersion must be < 10, >> DSCP >>>> must be 40, etc. It seems like for something truly self-service >> you'd >>>> probably need to set up some rules for your users to abide by that >>>> might not necessarily apply to ALL organizations. Just a thought. I >>>> don't think the duplicate origins problem alone would warrant >>>> something like that, but if it also provided a solution for the >>>> self-service effort, maybe it would be worth it. >>>> >>>> - Rawlin >>>> >>>> On Wed, Jan 23, 2019 at 4:15 AM Gelinas, Derek >>>> <[email protected]> wrote: >>>>> >>>>> That's the problem. We either need to be smart enough to recognize >>> that in our config and warn the user or just prevent it and suggest to >> the >>> user that they use a different fqdn. >>>>> >>>>> My vote is for preventing it at this point. >>>>> >>>>>> On Jan 23, 2019, at 12:50 AM, Gray, Jonathan < >>> [email protected]> wrote: >>>>>> >>>>>> It's not just differing types, it's anything that affects the >>> parent.config which also includes the, qstring configuration, MSO >>> parents/config, etc. as well. >>>>>> >>>>>> Jonathan G >>>>>> >>>>>> >>>>>> On 1/22/19, 2:33 PM, "Dave Neuman" <[email protected]> wrote: >>>>>> >>>>>> -1 on the one DS to Origin limitation. I think there are legit >>> use cases >>>>>> for it. >>>>>> I would be +1 on the idea if we include type though. Two DSs >>> can share the >>>>>> same origin as long as they are of the same type (DNS, HTTP, >>> HTTP_LIVE), >>>>>> etc. You shouldn't be able to have two DSs with the same >> origin >>> but >>>>>> different types. >>>>>> >>>>>> On Mon, Jan 14, 2019 at 7:53 AM Fieck, Brennan < >>> [email protected]> >>>>>> wrote: >>>>>> >>>>>>> I don't see this as a complicated use case or a barrier to >> entry, >>> an >>>>>>> extremely basic introduction to ATC would only make use of one >> DS >>>>>>> (shameless CDN-in-a-Box plug). >>>>>>> It seems far more complicated for a user to be using many DSes >> to >>> serve a >>>>>>> single origin - we can easily make it clear in the docs that the >>> two have a >>>>>>> 1:1 relationship, with a footnote about CNAME records for users >>> with the >>>>>>> knowledge, resources, and need for such workarounds. >>>>>>> ________________________________________ >>>>>>> From: Mark Torluemke <[email protected]> >>>>>>> Sent: Friday, January 11, 2019 1:21 PM >>>>>>> To: [email protected] >>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe Delivery >>> Services >>>>>>> produces indeterminate parent.config >>>>>>> >>>>>>>>> 1. Prohibit creating new delivery services that would share an >>>>>>> existing origin and prohibit updating a delivery service to a >>> shared >>>>>>> origin >>>>>>> In case my position has been lost, I'm still -1 on this. :) IMO, >>> we >>>>>>> shouldn't let the few complicated use cases increase the >> barrier >>> to entry >>>>>>> (to using ATC) for the masses. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 11, 2019 at 11:53 AM Rawlin Peters < >>> [email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Alright, I'm trying to sum up this discussion so far since it >>> seems >>>>>>>> like everyone went on vacation and didn't really get a chance >> to >>> wrap >>>>>>>> this one up: >>>>>>>> - duplicate origins cause undefined behavior >>>>>>>> - we need a way to migrate to a future that is free of >> duplicate >>>>>>>> origins in Traffic Control >>>>>>>> - we need a visual and easy way to determine if Traffic Ops >>> currently >>>>>>>> contains duplicate origins, so that operators are incentivized >>> to fix >>>>>>>> them rather than let it slide indefinitely >>>>>>>> - operators should have a fair amount of time to fix their >>> duplicate >>>>>>>> origins >>>>>>>> >>>>>>>> I believe this is what we've mostly agreed upon but haven't >>> clearly voted >>>>>>>> on: >>>>>>>> >>>>>>>> In release N we will: >>>>>>>> 1. Prohibit creating new delivery services that would share an >>>>>>>> existing origin and prohibit updating a delivery service to a >>> shared >>>>>>>> origin >>>>>>>> 2. Add some kind of visual indicator that duplicate origins >> are a >>>>>>>> problem that need to be fixed before release N+1; otherwise, an >>>>>>>> upgrade to N+1 will be prohibited. >>>>>>>> >>>>>>>> In release N+1 we will: >>>>>>>> 3. Include a DB migration that adds a uniqueness constraint on >>> origin >>>>>>>> FQDN, removing the API-level checks for that. >>>>>>>> 4. Prevent an upgrade to N+1 if duplicate origins are found >> (this >>>>>>>> might occur as a byproduct of step 3). >>>>>>>> >>>>>>>> I am +1 on this plan and believe this would hit on all the >>> summarized >>>>>>>> points above. Please provide a clear vote on this plan so that >>> we can >>>>>>>> dive deeper in the details (i.e. what release 'N' is, the best >>> visual >>>>>>>> indicator for step 2, and a friendly way to handle step 4). >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Rawlin >>>>>>>> >>>>>>>> On Wed, Dec 19, 2018 at 3:39 PM Jeremy Mitchell < >>> [email protected]> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Not sure TP is the right place for a warning like "clean up >> this >>>>>>>>> 'duplicate' origin or your next upgrade will fail". Most users >>> of our >>>>>>>>> system will be like "not my problem". >>>>>>>>> >>>>>>>>> On Wed, Dec 19, 2018 at 11:58 AM Fieck, Brennan < >>>>>>>> [email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Probably. It would impact load times by a bit, but the page >>> for an >>>>>>>>>> individual object is not our bottleneck. >>>>>>>>>> ________________________________________ >>>>>>>>>> From: Robert Butts <[email protected]> >>>>>>>>>> Sent: Wednesday, December 19, 2018 11:50 AM >>>>>>>>>> To: [email protected] >>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe >>> Delivery >>>>>>>> Services >>>>>>>>>> produces indeterminate parent.config >>>>>>>>>> >>>>>>>>>>> - Including a warning on startup and an API constraint >>> preventing >>>>>>>> adding >>>>>>>>>> more bad data in the next 3.0.0 Release Candidate >>>>>>>>>>> - Adding a database constraint immediately into master that >>> won't >>>>>>> be >>>>>>>>>> cherry-picked into 3.0.0 but should be included in 3.1.0 >>>>>>>>>> >>>>>>>>>> +1 >>>>>>>>>> >>>>>>>>>> I understand Jonathan's objection, but at some point, we have >>> to be >>>>>>>> able to >>>>>>>>>> move forward. This is a good compromise: deprecate, then >>> remove. That >>>>>>>> gives >>>>>>>>>> people a full major version to fix their data. >>>>>>>>>> >>>>>>>>>> I would be ideal if it were more than just a logged warning, >>> though. >>>>>>>> Can we >>>>>>>>>> add a big red banner in Traffic Portal, on the Delivery >>> Service page >>>>>>>> for >>>>>>>>>> any DS with a duplicate origin, telling users to fix it, and >>> that >>>>>>> they >>>>>>>>>> won't be able to upgrade to the next major version until it's >>> fixed? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Dec 19, 2018 at 10:57 AM Fieck, Brennan < >>>>>>>> [email protected] >>>>>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> So it seems like nobody has a problem with the "how" - >>> disallowing >>>>>>>>>>> duplicate origin FQDNs on Delivery Services - but we never >>> reached >>>>>>> a >>>>>>>>>>> consensus on "when". >>>>>>>>>>> >>>>>>>>>>> I stand by my previous proposal: >>>>>>>>>>> - Including a warning on startup and an API constraint >>> preventing >>>>>>>> adding >>>>>>>>>>> more bad data in the next 3.0.0 Release Candidate >>>>>>>>>>> - Adding a database constraint immediately into master that >>> won't >>>>>>> be >>>>>>>>>>> cherry-picked into 3.0.0 but should be included in 3.1.0 >>>>>>>>>>> ________________________________________ >>>>>>>>>>> From: Rawlin Peters <[email protected]> >>>>>>>>>>> Sent: Tuesday, December 18, 2018 4:59 PM >>>>>>>>>>> To: [email protected] >>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe >>> Delivery >>>>>>>> Services >>>>>>>>>>> produces indeterminate parent.config >>>>>>>>>>> >>>>>>>>>>> Also, building more around DS types will make it even harder >>> to get >>>>>>>>>>> away from DS types in the future too, which I know is >>> something >>>>>>> we've >>>>>>>>>>> discussed on this mailing list before. It also adds to the >>> overhead >>>>>>>> of >>>>>>>>>>> Delivery Service Topologies, since a lot of the DS types >> won't >>>>>>>>>>> carryover into that world. >>>>>>>>>>> >>>>>>>>>>> - Rawlin >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 18, 2018 at 2:42 PM Fieck, Brennan >>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>> +1. >>>>>>>>>>>> If there's a simple way to work around duplicate origins >>> being >>>>>>>>>>> prohibited, >>>>>>>>>>>> then we should rely on that instead of "enumerating all >> those >>>>>>>> possible >>>>>>>>>>> conflicting >>>>>>>>>>>> settings, which are not only highly complex and confusing, >>> but >>>>>>> also >>>>>>>>>>> further >>>>>>>>>>>> entrench us in only supporting ATS as a caching proxy >>> (hurting >>>>>>>> efforts >>>>>>>>>> to >>>>>>>>>>>> integrate e.g. Grove, nginx etc.) >>>>>>>>>>>> ________________________________________ >>>>>>>>>>>> From: Rawlin Peters <[email protected]> >>>>>>>>>>>> Sent: Tuesday, December 18, 2018 2:20 PM >>>>>>>>>>>> To: [email protected] >>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe >>> Delivery >>>>>>>>>>> Services produces indeterminate parent.config >>>>>>>>>>>> >>>>>>>>>>>> There are a number of different DS settings at play that >> can >>>>>>>>>>>> potentially cause conflicts. The question is: do we want to >>> get >>>>>>>> into >>>>>>>>>>>> the business of enumerating all those possible conflicting >>>>>>>> settings or >>>>>>>>>>>> just simply prohibit duplicate origins altogether? I think >>> we can >>>>>>>> dig >>>>>>>>>>>> in and get that "sufficiently advanced sql query" to check >>> for >>>>>>>>>>>> conflicting origins, but is that something we want to carry >>> along >>>>>>>> for >>>>>>>>>>>> the foreseeable future? Aren't CNAMEs relatively cheaper >> than >>>>>>>>>>>> developing and maintaining that code and the mental >> overhead >>>>>>>> required >>>>>>>>>>>> in understanding why you're getting an error that says your >>>>>>>> requested >>>>>>>>>>>> DS would cause an origin conflict? I think at the point >>> you've >>>>>>>>>>>> requested a DS that would create a conflict, you've chosen >>> those >>>>>>>>>>>> settings for a reason and would probably prefer to just >>>>>>> create/use >>>>>>>> a >>>>>>>>>>>> CNAME in your new DS and keep the rest of your settings the >>> same. >>>>>>>>>>>> >>>>>>>>>>>> Thinking in terms of errors, I'm imagining: >>>>>>>>>>>> "cannot create delivery service: origin fqdn ' >>> foo.example.com' >>>>>>>> already >>>>>>>>>>> in use" >>>>>>>>>>>> vs >>>>>>>>>>>> "cannot create delivery service: origin fqdn ' >>> foo.example.com' >>>>>>>> already >>>>>>>>>>>> in use as type DNS_LIVE_NATNL, which is incompatible with >>> your >>>>>>>> chosen >>>>>>>>>>>> type of HTTP_NO_CACHE" >>>>>>>>>>>> >>>>>>>>>>>> At that point you'd probably say to yourself, "uh, I need >>>>>>>>>>>> HTTP_NO_CACHE, so what am I supposed to do now?" >>>>>>>>>>>> >>>>>>>>>>>> As a lazy developer I'm +1 on prohibiting duplicate origin >>> fqdns >>>>>>>>>>>> because the resulting code will be simpler, but I think >>>>>>> eliminating >>>>>>>>>>>> the mental overhead for operators could be worthwhile too. >>> If we >>>>>>>> can >>>>>>>>>>>> agree on an end state of prohibiting duplicate origins >>>>>>> altogether, >>>>>>>> we >>>>>>>>>>>> can start working on a design to smoothly transition us to >>> that >>>>>>>> point. >>>>>>>>>>>> Are we willing to live with "just CNAME your origin fqdn" >> as >>> the >>>>>>>>>>>> standard solution to duplicates? >>>>>>>>>>>> >>>>>>>>>>>> - Rawlin >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Dec 18, 2018 at 1:27 PM Gelinas, Derek >>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> The only situation in which they can share origins is if >> a) >>> the >>>>>>>>>>> origins are shared in an MSO configuration but still have >>> different >>>>>>>>>> defined >>>>>>>>>>> origin fields in the delivery service, or if they're >> assigned >>> to >>>>>>>>>> completely >>>>>>>>>>> different cachegroups. It's when two delivery services >> share >>> the >>>>>>>> same >>>>>>>>>>> edges that there's an issue, because you end up with >>> parent.config >>>>>>>>>> issues. >>>>>>>>>>> Actually you could even get away with it in mids as long as >>> you >>>>>>>> weren't >>>>>>>>>>> doing anything like MSO to it. >>>>>>>>>>>>> >>>>>>>>>>>>> Could get messy real fast, though. Best to just create a >>>>>>> second >>>>>>>>>> FQDN. >>>>>>>>>>>>> >>>>>>>>>>>>> Derek >>>>>>>>>>>>> >>>>>>>>>>>>> On 12/18/18, 3:23 PM, "Fieck, Brennan" < >>>>>>>> [email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> So no two Delivery Services may share an origin >>> *regardless >>>>>>>> of >>>>>>>>>>> cache hierarchy* ? I've been told that DNS Delivery Services >>> can >>>>>>>> have the >>>>>>>>>>> same origin as HTTP Delivery Services because they obey the >>> same >>>>>>>> cache >>>>>>>>>>> hierarchy. You're saying that would still produce invalid >>> output >>>>>>>> and/or >>>>>>>>>> is >>>>>>>>>>> explicitly disallowed by ATS? >>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>> From: Robert Butts <[email protected]> >>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 1:09 PM >>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe >>>>>>>> Delivery >>>>>>>>>>> Services produces indeterminate parent.config >>>>>>>>>>>>> >>>>>>>>>>>>>> can you give an example of what parent.config looks like >>>>>>>> when 2 >>>>>>>>>>> ds's share >>>>>>>>>>>>> an origin and have different a different topology? >>>>>>>>>>>>> >>>>>>>>>>>>> Answering because I encountered this directly, when >>>>>>> rewriting >>>>>>>>>>> parent.config. >>>>>>>>>>>>> >>>>>>>>>>>>> For example: Suppose you have one Delivery Service: >>>>>>>>>>>>> XML_ID: foo >>>>>>>>>>>>> Type: HTPT_LIVE_NATL >>>>>>>>>>>>> Query String Handling: 1 - ignore in cache key, and pass >>> up >>>>>>>>>>>>> Origin Server Base URL: http://foo.example.net >>>>>>>>>>>>> >>>>>>>>>>>>> And another Delivery Service: >>>>>>>>>>>>> XML_ID: bar >>>>>>>>>>>>> Type: HTPT_LIVE_NATL >>>>>>>>>>>>> Query String Handling: 1 - ignore in cache key, and pass >>> up >>>>>>>>>>>>> Origin Server Base URL: http://foo.example.net >>>>>>>>>>>>> >>>>>>>>>>>>> ATS only supports unique `dest_domain` entries in >>>>>>>> parent.config. >>>>>>>>>>> Therefore, >>>>>>>>>>>>> the parent.config generated for a server assigned to >> both >>>>>>> of >>>>>>>>>> these >>>>>>>>>>> Delivery >>>>>>>>>>>>> Services with either be: >>>>>>>>>>>>> >>>>>>>>>>>>> dest_domain=foo.example.net port=80 go_direct=true >>>>>>>>>>>>> >>>>>>>>>>>>> Or >>>>>>>>>>>>> >>>>>>>>>>>>> dest_domain=foo.example.net port=80 go_direct=false >>>>>>>>>>> qstring=consider >>>>>>>>>>>>> >>>>>>>>>>>>> Right now, it's arbitrary which Perl Traffic Ops >> inserts, >>>>>>> and >>>>>>>>>> Perl >>>>>>>>>>> provides >>>>>>>>>>>>> no warning or error of any kind (the pending Go >>>>>>>> parent.config PR >>>>>>>>>>> logs an >>>>>>>>>>>>> error). >>>>>>>>>>>>> >>>>>>>>>>>>> Whichever is arbitrarily inserted, the resulting remaps >>> for >>>>>>>> the >>>>>>>>>>> other >>>>>>>>>>>>> delivery service will be wrong. Either `foo` requests >> will >>>>>>>> drop >>>>>>>>>>> the query >>>>>>>>>>>>> string when they shouldn't, and go to the mid when they >>>>>>>>>> shouldn't; >>>>>>>>>>> or `bar` >>>>>>>>>>>>> requests will use the query string and skip the mid when >>> it >>>>>>>>>>> shouldn't. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Does that make sense? The only correct solution, is to >>>>>>>> somehow >>>>>>>>>>> prevent >>>>>>>>>>>>> different DSes having the same origin, and tell tenants >>>>>>> they >>>>>>>> must >>>>>>>>>>> use >>>>>>>>>>>>> CNAMEs if they need. >>>>>>>>>>>>> >>>>>>>>>>>>> This isn't a bug in Traffic Control. ATS fundamentally >>>>>>>> doesn't >>>>>>>>>>> support >>>>>>>>>>>>> multiple remap rules with the same parent FQDN with >>>>>>> different >>>>>>>>>>>>> configurations. Hence, Traffic Control needs to prohibit >>>>>>>> that. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Dec 18, 2018 at 12:24 PM Jeremy Mitchell < >>>>>>>>>>> [email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> brennan, >>>>>>>>>>>>>> >>>>>>>>>>>>>> can you give an example of what parent.config looks like >>>>>>>> when 2 >>>>>>>>>>> ds's share >>>>>>>>>>>>>> an origin and have different a different topology? >>>>>>>>>>>>>> >>>>>>>>>>>>>> jeremy >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Dec 18, 2018 at 11:39 AM Fieck, Brennan < >>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> To be clear, the "Warning" I'm talking about would >>>>>>>> happen at >>>>>>>>>>> startup, but >>>>>>>>>>>>>>> I'd like a UI-only constraint to come with that to >>>>>>>> disallow >>>>>>>>>>> using the API >>>>>>>>>>>>>>> to bind the same origin to multiple Delivery Services >>>>>>>> with >>>>>>>>>>> varying >>>>>>>>>>>>>>> topography requirements. It wouldn't change the >>>>>>> existing >>>>>>>>>> data, >>>>>>>>>>> but >>>>>>>>>>>>>> prevent >>>>>>>>>>>>>>> users from creating more bad data. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> "warning" doesn't really sufficiently describe that, my >>>>>>>> bad. >>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>> From: Fieck, Brennan <[email protected]> >>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 11:24 AM >>>>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe >>>>>>>>>>> Delivery Services >>>>>>>>>>>>>>> produces indeterminate parent.config >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Well the cost of fixing this bug is a constraint on the >>>>>>>> data. >>>>>>>>>>> Unless we >>>>>>>>>>>>>>> make it a UI-only constraint - which I'm personally >>>>>>>> against - >>>>>>>>>>> there must >>>>>>>>>>>>>> be >>>>>>>>>>>>>>> some point in the future where ATC cannot reasonably be >>>>>>>>>>> expected to work >>>>>>>>>>>>>>> with data that violates that constraint. The question >>>>>>> is >>>>>>>> when >>>>>>>>>>> that should >>>>>>>>>>>>>>> occur, which should likely happen at a minor version >>>>>>>> release. >>>>>>>>>>> Minor not >>>>>>>>>>>>>>> major because it doesn't involve a change in data >>>>>>>> structures, >>>>>>>>>>> merely >>>>>>>>>>>>>>> relationships between them - in my opinion that's a >>>>>>> minor >>>>>>>>>>> version change >>>>>>>>>>>>>>> but that's definitely up for debate. With several >>>>>>> release >>>>>>>>>>> candidates for >>>>>>>>>>>>>>> 3.0.0 that _doesn't_ include this restriction already >>>>>>> in >>>>>>>> the >>>>>>>>>>> wild, I >>>>>>>>>>>>>>> wouldn't recommend putting it in there. That means to >>>>>>>> fix the >>>>>>>>>>> bug as soon >>>>>>>>>>>>>>> as possible it should go in 3.1.0 which should be the >>>>>>>> target >>>>>>>>>>> of "master" >>>>>>>>>>>>>>> after the 3.0.0 release is cut from it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> So I'd recommend immediately implementing the >>>>>>> constraint >>>>>>>> in >>>>>>>>>>> master with a >>>>>>>>>>>>>>> refusal to upgrade with bad data, and backport a >>>>>>> warning >>>>>>>>>> about >>>>>>>>>>> the future >>>>>>>>>>>>>>> behavior into 3.0.0 or as part of a 3.0.1 provided we >>>>>>> had >>>>>>>>>> more >>>>>>>>>>> changes >>>>>>>>>>>>>> that >>>>>>>>>>>>>>> would warrant a micro version bump. >>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>> From: Gray, Jonathan <[email protected]> >>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 9:34 AM >>>>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe >>>>>>>>>>> Delivery Services >>>>>>>>>>>>>>> produces indeterminate parent.config >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -1 Holding an ATC upgrade hostage to data cleanup seems >>>>>>>> like >>>>>>>>>> a >>>>>>>>>>> bad idea. >>>>>>>>>>>>>>> The issue isn't great, but it's also not new. We >>>>>>> should >>>>>>>>>> allow >>>>>>>>>>> teams to >>>>>>>>>>>>>> fix >>>>>>>>>>>>>>> their data at their normal paces if it doesn't create >>>>>>>>>>> significant >>>>>>>>>>>>>> overhead >>>>>>>>>>>>>>> or an inherant blocker for new functionality or >>>>>>>> correction of >>>>>>>>>>> other major >>>>>>>>>>>>>>> problems imho. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Jonathan G >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 12/18/18, 9:28 AM, "Fieck, Brennan" < >>>>>>>>>>> [email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Another option is we could detect collisions at >>>>>>>> startup >>>>>>>>>>> and simply >>>>>>>>>>>>>>> refuse to continue with the upgrade until the data is >>>>>>>> fixed. >>>>>>>>>>> That would >>>>>>>>>>>>>>> allow people using the now-unsupported data format to >>>>>>>>>> continue >>>>>>>>>>> to use >>>>>>>>>>>>>> their >>>>>>>>>>>>>>> old versions of Traffic Ops without wrecking their >>>>>>>> database, >>>>>>>>>>> but also >>>>>>>>>>>>>>> provide an incentive to clean up the data. >>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>> From: Gray, Jonathan <[email protected]> >>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 5:12 AM >>>>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to >>>>>>>> Multipe >>>>>>>>>>> Delivery >>>>>>>>>>>>>>> Services produces indeterminate parent.config >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm generally a fan of constrain your data in your >>>>>>>>>>> database, but not >>>>>>>>>>>>>>> necessarily exclusively. I see this as a one-way >>>>>>>>>>> cleanup/conversion so >>>>>>>>>>>>>> it >>>>>>>>>>>>>>> doesn't need to be configurable; otherwise you have to >>>>>>>> ask >>>>>>>>>> the >>>>>>>>>>> question >>>>>>>>>>>>>>> what happens if someone turns it off. That said, >>>>>>>> something >>>>>>>>>> in >>>>>>>>>>> the UI >>>>>>>>>>>>>> layer >>>>>>>>>>>>>>> would be nice to prevent spending significant >>>>>>> quantities >>>>>>>> of >>>>>>>>>>> time >>>>>>>>>>>>>> building a >>>>>>>>>>>>>>> complex DS only to have it fail to post for reasons >>>>>>> that >>>>>>>>>> could >>>>>>>>>>> have been >>>>>>>>>>>>>>> known earlier. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The way my brain works in this case: >>>>>>>>>>>>>>> If !unique_constraint_exists_query() >>>>>>>>>>>>>>> If has_duplicates_query() >>>>>>>>>>>>>>> show_warning() >>>>>>>>>>>>>>> else >>>>>>>>>>>>>>> add_unique_constraint() >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> to which the API and UI configuration could also >>>>>>>> make use >>>>>>>>>>> of >>>>>>>>>>>>>>> unique_constraint_exists_query() to drive additional >>>>>>>> layer >>>>>>>>>>> constraints if >>>>>>>>>>>>>>> desired. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Jonathan G >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 12/17/18, 1:11 PM, "Rawlin Peters" < >>>>>>>>>>> [email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> That is an interesting idea...detect at TO >>>>>>>> startup >>>>>>>>>>> whether or not >>>>>>>>>>>>>>> there are duplicate origins and operate in a >>>>>>>> "prevent >>>>>>>>>>> duplicate >>>>>>>>>>>>>>> origins" state if no duplicates are found or >>>>>>>> "prevent >>>>>>>>>>> conflicting >>>>>>>>>>>>>>> DS >>>>>>>>>>>>>>> topologies" state if duplicates are found? So >>>>>>>> once >>>>>>>>>>> operators have >>>>>>>>>>>>>>> replaced all the duplicate origins with CNAMEs, >>>>>>>> TO >>>>>>>>>> will >>>>>>>>>>>>>> essentially >>>>>>>>>>>>>>> operate in a "prohibit all duplicate origins" >>>>>>>> state. >>>>>>>>>>> That would >>>>>>>>>>>>>>> probably make for a simpler transition, but I'd >>>>>>>> want >>>>>>>>>>> to remove >>>>>>>>>>>>>> that >>>>>>>>>>>>>>> logic in a following release that strictly >>>>>>>> prohibits >>>>>>>>>>> duplicate >>>>>>>>>>>>>>> origins >>>>>>>>>>>>>>> (assuming that the community agrees we should >>>>>>>>>> prohibit >>>>>>>>>>> duplicate >>>>>>>>>>>>>>> origins altogether). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> As for DB constraints vs UI, I was thinking >>>>>>> those >>>>>>>>>>> DS-type >>>>>>>>>>>>>>> constraints >>>>>>>>>>>>>>> I pointed out would live in the API. It would >>>>>>>>>>> basically be added >>>>>>>>>>>>>>> validation in the deliveryservices POST/PUT >>>>>>>> endpoint >>>>>>>>>>> that checks >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> DB for existing DSes that conflict with the >>>>>>>> requested >>>>>>>>>>> DS. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Rawlin >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Dec 17, 2018 at 12:35 PM Gray, Jonathan >>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> These kinds of conditions should be >>>>>>> detectable >>>>>>>>>> with a >>>>>>>>>>>>>>> sufficiently advanced SQL query. Is it possible to add >>>>>>>> the >>>>>>>>>>> constraint if >>>>>>>>>>>>>>> it passes and emit a warning during TO startup >>>>>>> otherwise? >>>>>>>>>>> That would let >>>>>>>>>>>>>>> you know the condition exists at startup but not >>>>>>> getting >>>>>>>> in >>>>>>>>>>> your way and >>>>>>>>>>>>>>> keep you out of trouble once you've cleaned up. We >>>>>>> made >>>>>>>> a >>>>>>>>>>> mistake early >>>>>>>>>>>>>>> on, but this would acknowledge it was bad and encourage >>>>>>>> it to >>>>>>>>>>> be fixed at >>>>>>>>>>>>>>> the speed of operations teams. Also this puts the >>>>>>>> constraint >>>>>>>>>>> in the >>>>>>>>>>>>>>> database rather than the UI which is really where the >>>>>>>>>>> contention is for >>>>>>>>>>>>>>> usability. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Jonathan G >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 12/17/18, 11:38 AM, "Rawlin Peters" < >>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We occasionally discuss this issue but >>>>>>>> haven't >>>>>>>>>>> tackled it >>>>>>>>>>>>>>> yet. I think >>>>>>>>>>>>>>>> the main issue is just that duplicate >>>>>>>> origins >>>>>>>>>>> have been >>>>>>>>>>>>>>> allowed since >>>>>>>>>>>>>>>> the beginning, and now everyone's Traffic >>>>>>>> Ops >>>>>>>>>>> could be >>>>>>>>>>>>>>> littered with >>>>>>>>>>>>>>>> duplicate origins. Also, depending on the >>>>>>>>>> config >>>>>>>>>>> of the >>>>>>>>>>>>>>> duplicate >>>>>>>>>>>>>>>> delivery services, the origins might not >>>>>>>> be in >>>>>>>>>>> conflict at >>>>>>>>>>>>>>> all (if >>>>>>>>>>>>>>>> they don't have different topology >>>>>>>>>> constraints). >>>>>>>>>>> I would >>>>>>>>>>>>>>> love for us >>>>>>>>>>>>>>>> to just add a uniqueness constraint, but >>>>>>>> there >>>>>>>>>>> would need >>>>>>>>>>>>>> to >>>>>>>>>>>>>>> be a fair >>>>>>>>>>>>>>>> amount of warning to the community before >>>>>>>> doing >>>>>>>>>>> so and >>>>>>>>>>>>>> might >>>>>>>>>>>>>>>> invalidate a significant amount of valid >>>>>>>> use >>>>>>>>>>> cases. >>>>>>>>>>>>>>> Operators would >>>>>>>>>>>>>>>> need time to make DNS CNAME records for >>>>>>> the >>>>>>>>>>> duplicate >>>>>>>>>>>>>>> origins and >>>>>>>>>>>>>>>> update their DSes to use the different >>>>>>>> CNAMEs. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I think as a good first step to >>>>>>>> eliminating the >>>>>>>>>>> use of >>>>>>>>>>>>>>> duplicate >>>>>>>>>>>>>>>> origins altogether, we should identify >>>>>>>> which >>>>>>>>>>> "topology >>>>>>>>>>>>>>> constraints" >>>>>>>>>>>>>>>> actually cause conflicting config when >>>>>>> used >>>>>>>>>> with >>>>>>>>>>> duplicate >>>>>>>>>>>>>>> origins and >>>>>>>>>>>>>>>> prevent creating DSes with duplicate >>>>>>>> origins >>>>>>>>>> _if >>>>>>>>>>> it would >>>>>>>>>>>>>>> cause a >>>>>>>>>>>>>>>> conflict with an existing DS that uses >>>>>>> the >>>>>>>> same >>>>>>>>>>> origin_. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> For instance, I believe an HTTP and >>>>>>>> DNS-type DS >>>>>>>>>>> can live >>>>>>>>>>>>>>> happily >>>>>>>>>>>>>>>> side-by-side using the same origin >>>>>>>> (probably >>>>>>>>>>> need different >>>>>>>>>>>>>>>> routing_names?), but scenarios like HTTP >>>>>>>> and >>>>>>>>>>> HTTP_LIVE, or >>>>>>>>>>>>>>> DNS and >>>>>>>>>>>>>>>> HTTP_NO_CACHE sharing the same origin >>>>>>> will >>>>>>>>>> cause >>>>>>>>>>> conflicts >>>>>>>>>>>>>>> for sure. >>>>>>>>>>>>>>>> So maybe we can start by making sure the >>>>>>> DS >>>>>>>>>>> types "match" >>>>>>>>>>>>>>> when using >>>>>>>>>>>>>>>> the same origin: >>>>>>>>>>>>>>>> HTTP + DNS: possibly good, if they have >>>>>>>>>>> different routing >>>>>>>>>>>>>>> names? >>>>>>>>>>>>>>>> HTTP_LIVE + HTTP_LIVE_NATNL: bad >>>>>>>>>>>>>>>> HTTP_NO_CACHE + [any other type]: bad >>>>>>>>>>>>>>>> HTTP_LIVE + HTTP: bad >>>>>>>>>>>>>>>> etc. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> There are most likely other conflict >>>>>>>> scenarios >>>>>>>>>>> that don't >>>>>>>>>>>>>>> involve the >>>>>>>>>>>>>>>> DS types, but I think this would be a >>>>>>> good >>>>>>>>>>> start. In the >>>>>>>>>>>>>>> future with >>>>>>>>>>>>>>>> Delivery Service Topologies (aka Flexible >>>>>>>>>>> Cachegroups aka >>>>>>>>>>>>>>> Bring Your >>>>>>>>>>>>>>>> Own Topology), we might be able to >>>>>>> prohibit >>>>>>>>>>> assigning a DS >>>>>>>>>>>>>>> to a >>>>>>>>>>>>>>>> Topology if the DS's origin is already >>>>>>>> used by >>>>>>>>>>> another DS >>>>>>>>>>>>>> in >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>> different Topology. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Rawlin >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Dec 17, 2018 at 10:52 AM Fieck, >>>>>>>> Brennan >>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> As some of you may be aware, >>>>>>>> `parent.config` >>>>>>>>>>> files >>>>>>>>>>>>>>> generated by Traffic Ops can vary wildly when an origin >>>>>>>> is >>>>>>>>>>> assigned to >>>>>>>>>>>>>>> multiple Delivery Services. This results in undefined >>>>>>>>>>> behavior. I'm told >>>>>>>>>>>>>>> that the conflict only happens when two Delivery >>>>>>> Services >>>>>>>>>> with >>>>>>>>>>> different >>>>>>>>>>>>>>> "topology requirements" use the same origin, whatever >>>>>>>> that >>>>>>>>>>> means (content >>>>>>>>>>>>>>> routing type?). Regardless, the issue should be >>>>>>>> addressed. >>>>>>>>>> The >>>>>>>>>>> obvious >>>>>>>>>>>>>>> solution is to put in place a database constraint that >>>>>>>>>>> prevents an origin >>>>>>>>>>>>>>> from being assigned to more that one Delivery Service >>>>>>>> with >>>>>>>>>> API >>>>>>>>>>> checks in >>>>>>>>>>>>>>> place that would provide helpful error messages when an >>>>>>>>>>> attempt is made >>>>>>>>>>>>>> to >>>>>>>>>>>>>>> violate the constraint. However, would that mess with >>>>>>>> things >>>>>>>>>>> like >>>>>>>>>>>>>>> Multi-Site Origin? Or is it just not viable for some >>>>>>>> other >>>>>>>>>>> reason? If it >>>>>>>>>>>>>> is >>>>>>>>>>>>>>> a good solution, I'm prepared to work on a fix that >>>>>>>> utilizes >>>>>>>>>>> it. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>> >>> >>> >>
