Re: Making parent.config's go_direct directive configurable via Delivery service
To me it looks like DS types with go_direct TRUE/False will add more confusion compared to adding go_direct value as a DS configurable parameter defaulting it to false except for HTTP_LIVE DNS_LIVE and HTTP_NO_CACHE. Thanks, Vijayanand S On Thu, Jun 21, 2018 at 4:53 AM, Rawlin Peters wrote: > I think adding new DS types for this makes sense because traditionally > the DS type determines the value of go_direct as well as how content > is cached (disk/ram/not cached at all). If we make the field directly > configurable on the Delivery Service, then we now have the complexity > of prohibiting certain go_direct values in certain types of delivery > services, and that adds to the cognitive load required to create/edit > a delivery service. > > For example, it's a bad experience for the user to find out through an > API error that they are prohibited from setting go_direct to certain > values in certain scenarios. If we just create new types for that > instead, the user doesn't have to worry about conflicting settings in > specific DS types and rather just has to choose the DS type that they > want (where the proper/best settings are chosen for them under the > hood). > > Basically we'd need 4 new types (naming could be different): > HTTP_LIVE_NATNL_BYPASS > DNS_LIVE_NATNL_BYPASS > HTTP_BYPASS > DNS_BYPASS > > The new types would mimic the matching original types except for > setting go_direct=true, which would allow the edges to fetch from the > origin when all the mids are down. > > - Rawlin > > On Wed, Jun 20, 2018 at 6:33 AM, Vijay Anand > wrote: > > All, > > > > The PR given below is a perl implemention for making parent.config's > > go_direct directive configurable via Delivery service > > https://github.com/apache/trafficcontrol/pull/2407 > > > > This PR has been hosted to discuss about various approaches to make > > go_direct configurable. GoLang implementation will be added once we > > finalize on the approach. > > > > Background: > > --- > > Right now it is hard coded as false in parent.config for DS of types > other > > than HTTP_LIVE, HTTP_NO_CACHE, DNS_LIVE and hence for such delivery > > services, if there occurs a problem in the network / in the Mids and they > > become unreachable / offline, all the requests fail because of this hard > > coded GO_DIRECT setting. > > > > By making this configurable, we are giving a choice to the operators to > > fetch directly from origin under such scenarios. > > > > > > Question: > > > > Originally it was thought of as a new column in Deliveryservice table > > (Go_Direct). But then Rawlin suggested adding a new delivery serice type > to > > avoid some of the conflict scenario disccussed in the PR. Eric and Rob > > feels that adding new DS type is not a desired one. > > > > Request your views to get a consensus on the suitable approach for this. > > > > Thanks, > > Vijayanand S > > > > > > > > -- Forwarded message -- > > From: Vijay Anand > > Date: Thu, Jun 7, 2018 at 6:45 PM > > Subject: Re: Making parent.config's go_direct directive configurable via > > Delivery service > > To: dev@trafficcontrol.apache.org > > > > > > Rawlin, > > > > Yes, I am using a version which Eric referred to (i.e) Cisco's version. > And > > looks like in this code it is actually possible to create MSO groups > > (origin server) which may not contain the org_serv_fqdn. So do you think, > > MSO enabling and Go Direct = True as mutually exclusive will work? > > > > Thanks, > > Vijayanand S > > > > On Mon, Jun 4, 2018 at 8:13 PM, Rawlin Peters > > wrote: > > > >> Also, it's very possible this might be a nuance between the version of > >> TO you are running and the vanilla open source TO. We don't assign > >> delivery services to "groups"; we assign them to individual servers > >> (although that might mean every single server in a cachegroup). I > >> think I recall Eric mentioning "server groups" at Cisco. Might that be > >> the case? > >> > >> - Rawlin > >> > >> On Mon, Jun 4, 2018 at 8:36 AM, Rawlin Peters > >> wrote: > >> > It might be possible to create an MSO delivery service with an > >> > `orgServerFqdn` that doesn't match the [hostname + domain name] of a > >> > server it's assigned to, but I'm not sure it would work as expected. > >> > In a bug I recently fixed, I found a bit of code [1] that makes me > >> > think that
Fwd: Making parent.config's go_direct directive configurable via Delivery service
All, The PR given below is a perl implemention for making parent.config's go_direct directive configurable via Delivery service https://github.com/apache/trafficcontrol/pull/2407 This PR has been hosted to discuss about various approaches to make go_direct configurable. GoLang implementation will be added once we finalize on the approach. Background: --- Right now it is hard coded as false in parent.config for DS of types other than HTTP_LIVE, HTTP_NO_CACHE, DNS_LIVE and hence for such delivery services, if there occurs a problem in the network / in the Mids and they become unreachable / offline, all the requests fail because of this hard coded GO_DIRECT setting. By making this configurable, we are giving a choice to the operators to fetch directly from origin under such scenarios. Question: Originally it was thought of as a new column in Deliveryservice table (Go_Direct). But then Rawlin suggested adding a new delivery serice type to avoid some of the conflict scenario disccussed in the PR. Eric and Rob feels that adding new DS type is not a desired one. Request your views to get a consensus on the suitable approach for this. Thanks, Vijayanand S -- Forwarded message -- From: Vijay Anand Date: Thu, Jun 7, 2018 at 6:45 PM Subject: Re: Making parent.config's go_direct directive configurable via Delivery service To: dev@trafficcontrol.apache.org Rawlin, Yes, I am using a version which Eric referred to (i.e) Cisco's version. And looks like in this code it is actually possible to create MSO groups (origin server) which may not contain the org_serv_fqdn. So do you think, MSO enabling and Go Direct = True as mutually exclusive will work? Thanks, Vijayanand S On Mon, Jun 4, 2018 at 8:13 PM, Rawlin Peters wrote: > Also, it's very possible this might be a nuance between the version of > TO you are running and the vanilla open source TO. We don't assign > delivery services to "groups"; we assign them to individual servers > (although that might mean every single server in a cachegroup). I > think I recall Eric mentioning "server groups" at Cisco. Might that be > the case? > > - Rawlin > > On Mon, Jun 4, 2018 at 8:36 AM, Rawlin Peters > wrote: > > It might be possible to create an MSO delivery service with an > > `orgServerFqdn` that doesn't match the [hostname + domain name] of a > > server it's assigned to, but I'm not sure it would work as expected. > > In a bug I recently fixed, I found a bit of code [1] that makes me > > think that scenario would lead to a bad parent.config. The issue [2] > > was that having a trailing slash in `orgServerFqdn` led to an empty > > parent field in parent.config (because before the fix, TO didn't find > > an origin Server that matched the `orgServerFqdn` of the MSO delivery > > service. > > > > - Rawlin > > > > [1] https://github.com/apache/incubator-trafficcontrol/blob/mast > er/traffic_ops/app/lib/API/Configs/ApacheTrafficServer.pm#L2407 > > [2] https://github.com/apache/incubator-trafficcontrol/issues/2062 > > > > On Fri, Jun 1, 2018 at 6:57 AM, Vijay Anand > > wrote: > >> Hi Rawlin, > >> > >> I thought, we can always create an origin server group (for MSO) which > wont > >> necessarily contain the origin server which we configure while creating > a > >> DS and that is the logic behind adding this conflict. " MSO = true and > >> go_direct = true". IF that is not the case, we dont need this conflict. > >> > >> Thanks, > >> Vijayanand S > >> > >> > >> > >> On Thu, May 31, 2018 at 9:08 PM, Rawlin Peters > > >> wrote: > >> > >>> When configuring MSO currently (note this will change soon once I get > >>> the MSO implementation refactored to use Origins rather than Servers), > >>> you have to set `orgServerFqdn` to an FQDN that matches at least one > >>> Server with a combined (hostname, domain_name) matching that > >>> `orgServerFqdn` used in the Delivery Service. That Delivery Service is > >>> then assigned to that Server and any other Server acting as an origin. > >>> So the request flow for an MSO Delivery Service should still flow to > >>> both the origin Server matching the DS's `orgServerFqdn` and any other > >>> origin Server that is assigned that DS. > >>> > >>> - Rawlin > >>> > >>> On Thu, May 31, 2018 at 9:12 AM, Vijay Anand > >>> wrote: > >>> > Hi Rawlin, > >>> > > >>> > Adding CNAME Alias for sharing an origin to resolve the conflicts > looks > >>> > good and i
Re: Making parent.config's go_direct directive configurable via Delivery service
Hi Rawlin, I thought, we can always create an origin server group (for MSO) which wont necessarily contain the origin server which we configure while creating a DS and that is the logic behind adding this conflict. " MSO = true and go_direct = true". IF that is not the case, we dont need this conflict. Thanks, Vijayanand S On Thu, May 31, 2018 at 9:08 PM, Rawlin Peters wrote: > When configuring MSO currently (note this will change soon once I get > the MSO implementation refactored to use Origins rather than Servers), > you have to set `orgServerFqdn` to an FQDN that matches at least one > Server with a combined (hostname, domain_name) matching that > `orgServerFqdn` used in the Delivery Service. That Delivery Service is > then assigned to that Server and any other Server acting as an origin. > So the request flow for an MSO Delivery Service should still flow to > both the origin Server matching the DS's `orgServerFqdn` and any other > origin Server that is assigned that DS. > > - Rawlin > > On Thu, May 31, 2018 at 9:12 AM, Vijay Anand > wrote: > > Hi Rawlin, > > > > Adding CNAME Alias for sharing an origin to resolve the conflicts looks > > good and it should work. So that , if a conflict is detected, the > operator > > has to setup up CNAME alias for that particular origin. > > > > For MSO, when some one configures an MSO, he would probably meant to use > an > > origin server(s) which is different from Delivery service's orgin_fqdn. > So > > when go_direct is true and mid cache is offline, Edge will go to Delivery > > service configured orgin_fqdn which is not an intented behaviour if some > > one configured MSO. > > > > Thanks, > > Vijayanand S > > > > On Thu, May 31, 2018 at 3:17 AM, Rawlin Peters > > wrote: > > > >> Hi Vijayanand S, > >> > >> Generally we've found it's bad practice to have multiple delivery > >> services sharing the same origin due to the conflicts in configuration > >> on the caches serving those delivery services like you mentioned. But > >> this can be fixed by setting up CNAME DNS records for the shared > >> origin and using a distinct CNAME in each delivery service. In fact > >> I've discussed duplicate origins here fairly recently due to my effort > >> to refactor the Origin implementation, and the tentative plan was to > >> phase-in a uniqueness constraint on Origin FQDN so that there will be > >> no possibility of conflicts that we experience today with duplicate > >> Origin FQDNs. > >> > >> Would that fix your issue? > >> > >> The `go_direct` option isn't hardcoded per se but is determined by the > >> delivery service type in order to bypass or use the mid tier. So for > >> HTTP_NO_CACHE, HTTP_LIVE, and DNS_LIVE, we bypass the mid tier because > >> that's what those types are for. Why do you want to bypass the mid > >> tier for MSO? > >> > >> - Rawlin > >> > >> On Wed, May 30, 2018 at 8:58 AM, Vijay Anand > >> wrote: > >> > Hi All, > >> > > >> > Planning for a PR on making parent.config's go_direct directive > >> > configurable via Delivery service. Right now, go_direct is > >> > being hard coded. > >> > > >> > Given below is a brief write up on the implementing the same. > >> > > >> > Assumption: > >> > All DS-es sharing an origin server should have same value for > go_direct. > >> > > >> > Implementation: > >> > Add a new column 'go_direct' in Deliveryservice table. Its value > defaults > >> > to False. > >> > Delivery service UI (Traffic Ops) and API will be enhanced to support > >> this > >> > new column. > >> > > >> > Conflicts: > >> > 1. DS Type HTTP_NO_CACHE and go direct False > >> > 2. DS Type HTTP_LIVE and go direct False > >> > 3. DS Type DNS_LIVE and go direct False > >> > 4. MSO = true and go_direct = true > >> > > >> > > >> > When we have more than 2 DS-es sharing same origin, then updating one > >> > particular DS's go_direct value will result > >> > in conflict, since all DS-es sharing an origin should have same value > for > >> > go_direct. > >> > > >> > Such conflicts should be resolved by deleting and recreating these > DS-es > >> > with new value for go_direct. > >> > > >> > This method of deleting and recreating DS-es is preferred over > updating > >> all > >> > the affected DS-es implicitly. > >> > > >> > would like your comments on this. > >> > > >> > Thanks, > >> > Vijayanand S > >> >
Re: Making parent.config's go_direct directive configurable via Delivery service
Hi Rawlin, Adding CNAME Alias for sharing an origin to resolve the conflicts looks good and it should work. So that , if a conflict is detected, the operator has to setup up CNAME alias for that particular origin. For MSO, when some one configures an MSO, he would probably meant to use an origin server(s) which is different from Delivery service's orgin_fqdn. So when go_direct is true and mid cache is offline, Edge will go to Delivery service configured orgin_fqdn which is not an intented behaviour if some one configured MSO. Thanks, Vijayanand S On Thu, May 31, 2018 at 3:17 AM, Rawlin Peters wrote: > Hi Vijayanand S, > > Generally we've found it's bad practice to have multiple delivery > services sharing the same origin due to the conflicts in configuration > on the caches serving those delivery services like you mentioned. But > this can be fixed by setting up CNAME DNS records for the shared > origin and using a distinct CNAME in each delivery service. In fact > I've discussed duplicate origins here fairly recently due to my effort > to refactor the Origin implementation, and the tentative plan was to > phase-in a uniqueness constraint on Origin FQDN so that there will be > no possibility of conflicts that we experience today with duplicate > Origin FQDNs. > > Would that fix your issue? > > The `go_direct` option isn't hardcoded per se but is determined by the > delivery service type in order to bypass or use the mid tier. So for > HTTP_NO_CACHE, HTTP_LIVE, and DNS_LIVE, we bypass the mid tier because > that's what those types are for. Why do you want to bypass the mid > tier for MSO? > > - Rawlin > > On Wed, May 30, 2018 at 8:58 AM, Vijay Anand > wrote: > > Hi All, > > > > Planning for a PR on making parent.config's go_direct directive > > configurable via Delivery service. Right now, go_direct is > > being hard coded. > > > > Given below is a brief write up on the implementing the same. > > > > Assumption: > > All DS-es sharing an origin server should have same value for go_direct. > > > > Implementation: > > Add a new column 'go_direct' in Deliveryservice table. Its value defaults > > to False. > > Delivery service UI (Traffic Ops) and API will be enhanced to support > this > > new column. > > > > Conflicts: > > 1. DS Type HTTP_NO_CACHE and go direct False > > 2. DS Type HTTP_LIVE and go direct False > > 3. DS Type DNS_LIVE and go direct False > > 4. MSO = true and go_direct = true > > > > > > When we have more than 2 DS-es sharing same origin, then updating one > > particular DS's go_direct value will result > > in conflict, since all DS-es sharing an origin should have same value for > > go_direct. > > > > Such conflicts should be resolved by deleting and recreating these DS-es > > with new value for go_direct. > > > > This method of deleting and recreating DS-es is preferred over updating > all > > the affected DS-es implicitly. > > > > would like your comments on this. > > > > Thanks, > > Vijayanand S >
Making parent.config's go_direct directive configurable via Delivery service
Hi All, Planning for a PR on making parent.config's go_direct directive configurable via Delivery service. Right now, go_direct is being hard coded. Given below is a brief write up on the implementing the same. Assumption: All DS-es sharing an origin server should have same value for go_direct. Implementation: Add a new column 'go_direct' in Deliveryservice table. Its value defaults to False. Delivery service UI (Traffic Ops) and API will be enhanced to support this new column. Conflicts: 1. DS Type HTTP_NO_CACHE and go direct False 2. DS Type HTTP_LIVE and go direct False 3. DS Type DNS_LIVE and go direct False 4. MSO = true and go_direct = true When we have more than 2 DS-es sharing same origin, then updating one particular DS's go_direct value will result in conflict, since all DS-es sharing an origin should have same value for go_direct. Such conflicts should be resolved by deleting and recreating these DS-es with new value for go_direct. This method of deleting and recreating DS-es is preferred over updating all the affected DS-es implicitly. would like your comments on this. Thanks, Vijayanand S
