Re: TO jobs_* Table Cleanup

[Jan van Doorn]

I think they should be nuked.

Rgds,
JvD

> On May 1, 2018, at 2:04 PM, Dewayne Richardson  wrote:
> 
> I'm beginning to evaluate the Perl cleanup effort and noticed that the
> *job_agent*, *job_status*, and *job_result* are minimally used (except for
> the "job" table which has the list of Purge Jobs").  I wanted to get a feel
> for the usage outside of Comcast and what the sentiment is to those tables
> (or not).
> 
> 
> 
> -Dew

Re: [VOTE] Resolution for Traffic Control graduation to TLP

[Jan van Doorn]

+1

On 2018/04/02 20:11:55, David Neuman  wrote:
> Dear Traffic Control community members:>
>
> I would like to call a vote on the resolution for Traffic Control to>
> graduate from to an Apache TLP. We have already voted on whether or 
not we>
> should start the graduation process [1] and this is the next step. 
Please>

> see the resolution below and vote as follows:>
>
> [ ] +1 Graduate Traffic Control from the incubator>
> [ ] +0 No Opinion>
> [ ] -1 Do not graduate Traffic Control from the incubator (please 
provide a>

> reason)>
>
> The vote is open for a minimum of 72 hours and will need at least 3 
more +1>

> votes than -1 votes from PMC members to succeed.>
>
> If this VOTE succeeds, a similar VOTE will be started on the>
> gene...@incubator.apache.org mailing list. If that VOTE succeeds, a>
> resolution will be included in the next Apache Board Meeting.>
>
> Please feel free to reach out with any questions.>
>
> Thanks,>
> Dave>
>
> [1]>
> 
https://lists.apache.org/thread.html/fb1fae0785feb6568cef6deb6fa20723eba54ed63a445462d44564d3@%3Cdev.trafficcontrol.apache.org%3E> 


>
> ->
>
> Establish the Apache Traffic Control Project>
>
> WHEREAS, the Board of Directors deems it to be in the best interests of>
> the Foundation and consistent with the Foundation's purpose to 
establish>
> a Project Management Committee charged with the creation and 
maintenance>

> of open-source software, for distribution at no charge to the public,>
> related to building, monitoring, configuring, and provisioning a large>
> scale content delivery network (CDN)..>
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee>
> (PMC), to be known as the "Apache Traffic Control Project", be and>
> hereby is established pursuant to Bylaws of the Foundation; and be it>
> further>
>
> RESOLVED, that the Apache Traffic Control Project be and hereby is>
> responsible for the creation and maintenance of software related to>
> building, monitoring, configuring, and provisioning a large scale>
> content delivery network (CDN).; and be it further>
>
> RESOLVED, that the office of "Vice President, Apache Traffic Control" 
be>

> and hereby is created, the person holding such office to serve at the>
> direction of the Board of Directors as the chair of the Apache Traffic>
> Control Project, and to have primary responsibility for management of>
> the projects within the scope of responsibility of the Apache Traffic>
> Control Project; and be it further>
>
> RESOLVED, that the persons listed immediately below be and hereby are>
> appointed to serve as the initial members of the Apache Traffic Control>
> Project:>
>
> * Dan Kirkwood >
> * David Neuman >
> * Dewayne Richardson >
> * Eric Covener >
> * Eric Friedrich >
> * Hank Beatty >
> * Jan van Doorn >
> * Jeff Elsloo >
> * Jeremy Mitchell >
> * Leif Hedstrom >
> * Mark Torluemke >
> * Phil Sorber >
> * Steve Malenfant >
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that David Neuman be appointed>
> to the office of Vice President, Apache Traffic Control, to serve in>
> accordance with and subject to the direction of the Board of Directors>
> and the Bylaws of the Foundation until death, resignation, retirement,>
> removal or disqualification, or until a successor is appointed; and be>
> it further>
>
> RESOLVED, that the initial Apache Traffic Control PMC be and hereby is>
> tasked with the creation of a set of bylaws intended to encourage open>
> development and increased participation in the Apache Traffic Control>
> Project; and be it further>
>
> RESOLVED, that the Apache Traffic Control Project be and hereby is>
> tasked with the migration and rationalization of the Apache Incubator>
> Traffic Control podling; and be it further>
>
> RESOLVED, that all responsibilities pertaining to the Apache Incubator>
> Traffic Control podling encumbered upon the Apache Incubator PMC are>
> hereafter discharged.>
>

Re: [VOTE] Traffic Control graduation to TLP

[Jan van Doorn]

+1

On Thu, Mar 1, 2018 at 9:38 AM Chris Lemmons  wrote:

> The Traffic Control project has grown so much over the last year, it's
> incredible.
>
> +1
>
> On Thu, Mar 1, 2018 at 9:33 AM, Gelinas, Derek
>  wrote:
> > +1
> >
> >> On Mar 1, 2018, at 10:41 AM, Dave Neuman  wrote:
> >>
> >> Hey All,
> >>
> >> After a great discussion amongst the Apache Traffic Control PPMC,
> reviewing
> >> the graduation checklist[1], updating the podling status page[2], and
> >> updating the project website to ensure the whimsy podling website checks
> >> pass[3], I would like to call a vote for Apache Traffic Control
> graduating
> >> to a top level project.
> >>
> >> Apache Traffic Control entered the incubator on July 12, 2016.  Since
> then
> >> we have announced 4 releases, nominated 4 new committers, organized 3
> >> summits, had almost 8,000 commits from 63 different contributors, and --
> >> most importantly -- we have grown and diversified our community.  Apache
> >> Traffic Control is a healthy project that is already acting like an
> Apache
> >> top level project, so we should take the next step.
> >>
> >> If we agree that we should graduate to a top level project, the next
> steps
> >> will be to pick the initial PMC chair for the TLP and draft a Resolution
> >> for the PPMC and IPMC to vote upon.
> >>
> >> Please take a minute to vote on wheter or not Traffic Control should
> >> graduate to a Top Level Project by responding with one of the following:
> >>
> >> [ ] +1 Apache Traffic Control should graduate.
> >> [ ] +0 No opinion
> >> [ ] -1 Apache Traffic Control should not graduate (please provide the
> >> reason)
> >>
> >> The VOTE will be opened for at least the next 72 hours.  Per Apache
> >> guidelines[4] I will also be notifying the incubator mailing list that a
> >> community vote is under way.
> >>
> >> Thanks,
> >> Dave
> >>
> >>
> >> [1]
> >>
> https://incubator.apache.org/guides/graduation.html#graduation_check_list
> >> [2] http://incubator.apache.org/projects/trafficcontrol.html
> >> [3] https://whimsy.apache.org/pods/project/trafficcontrol
> >> [4]
> >>
> https://incubator.apache.org/guides/graduation.html#community_graduation_vote
> >
>

Re: Golang Proxy Validation Dependencies

[Jan van Doorn]

+1 on using libs.

> On Jan 16, 2018, at 10:52 AM, Dan Kirkwood  wrote:
> 
> +1 -- agree with Jeff -- the validation of the fields of
> deliveryservice is something that is incomplete in the Perl
> traffic_ops.
> 
> These libraries make for concise code to do the validation so it will
> be easier to extend without much extra code.   It will not be called
> on every API function,  but only once on each POST/PUT which are a
> small minority of calls.   It also need not be used in every case.
> That, to me,  makes the reflection argument much less of a concern.
> 
> I would like to see it go in sooner than Friday,  but won't argue that point..
> 
> -dan
> 
> 
> On Tue, Jan 16, 2018 at 10:10 AM, Dewayne Richardson  
> wrote:
>> So, it's been a few days on this topic and I'd like to call a vote for the
>> dependencies listed in this thread.  Please vote +1 or -1 by Noon Friday so
>> that we can move forward the Golang Proxy development.
>> 
>> Thanks,
>> 
>> -Dew
>> 
>> On Thu, Jan 11, 2018 at 10:53 AM, Jeff Elsloo  wrote:
>> 
>>> I don't think we should assume anything about the performance just
>>> because it uses reflection. Yes, traditionally reflection is
>>> computationally expensive, however, when used properly the penalty can
>>> be negligible. I don't think we have enough understanding of these
>>> libraries to know whether there is a concerning performance penalty.
>>> 
>>> As Dewayne said, create, update and delete actions represent a tiny
>>> fraction of the overall requests into TO. Given that the majority of
>>> these actions are performed by humans, I would be shocked if there was
>>> a perceptible performance difference with the reflection based
>>> validation in place. It's not like we're trying to validate enormous
>>> and complex objects here; we're talking 20 fields or so for any given
>>> post.
>>> 
>>> I'm +1 on using validation libraries such as these even if they use
>>> reflection, provided that we do not see dramatic changes in
>>> performance. I think that's highly unlikely in this case.
>>> --
>>> Thanks,
>>> Jeff
>>> 
>>> 
>>> On Thu, Jan 11, 2018 at 10:07 AM, Chris Lemmons 
>>> wrote:
 True, but how many of those out-of-the-box checks are both useful and
 relevantly complex?
 
 To me, the cool part of ozzo is the way it collects the output and
 formats it. That's unfortunately also the computationally expensive
 part.
 
 On Thu, Jan 11, 2018 at 9:49 AM, Dewayne Richardson 
>>> wrote:
> Please keep in mind that we do not Create/Update/Delete very often in
> Traffic Ops, so the performance penalty for Validation should be taken
>>> into
> consideration.  I also don't want to re-invent all of those
>>> out-of-the-box
> field level checks by hand when I can just use them from here:
> https://github.com/asaskevich/govalidator#list-of-functions
> 
> -Dew
> 
> On Thu, Jan 11, 2018 at 9:24 AM, Chris Lemmons 
>>> wrote:
> 
>> I like the output style, but I'm a bit concerned on the performance
>> front. ozzo appears to do all it's magic with heavy use of reflection,
>> which is often a slow spot in go. Most places, it wouldn't matter
>> much, but this will be called on every element of every API function,
>> so a nod toward performance may be in order. Have we done some
>> measurement to see whether this adds a relevant amount of overhead to
>> the calls? Or are the calls still dominated by the DB lookup?
>> 
>> Relatedly, is this a major advantage over something like this:
>> 
>> if ds.Active == nil { errMsgs = append(errMsgs, `"active" must be
>> provided`) }
>> 
>> On Thu, Jan 11, 2018 at 8:49 AM, Dewayne Richardson <
>>> dewr...@apache.org>
>> wrote:
>>> We've been moving along with more functionality in the Golang proxy,
>> mostly
>>> the Read's up until now, comparatively TO does much fewer
>>> Create/Updates.
>>> Our current task is to circle back and start implementing the
>>> (C)reate,
>>> (U)pdate, and (D)eletes.  One of the obvious needs for the this task
>>> are
>>> validation rules.  I've been doing research to figure out the
>>> cleanest
>> and
>>> most maintainable way to rewrite the Perl validation rules in Go.
>>> 
>>> TC Issue for tracking
>>> https://github.com/apache/incubator-trafficcontrol/issues/1756
>>> 
>>> These are the two dependencies I'd like to leverage and provide
>>> feedback:
>>> 
>>> Both are MIT Licenses
>>> Uses normal programming constructs rather than error-prone struct
>>> tags to
>>> specify how data should be validated.
>>> https://github.com/go-ozzo/ozzo-validation
>>> https://github.com/go-ozzo/ozzo-validation/blob/master/LICENSE
>>> 
>>> Core Validation library that the prior library uses that has a lot of
>>> useful convenience methods that I'd rather not re-invent
>>> https://github.com/asaskevich/govalidator
>>> https://g

Move Docs to readthedocs.io ?

[Jan van Doorn]

I propose we move our docs to readthedocs - like : 
http://incubator-trafficcontrol.readthedocs.io/en/latest/ (this is setup 
w my clone).


This will allow us to have multiple versions of docs online, do 
automatic updates of the released and latest docs and will get us the 
"Edit on Github" button that Ryan requested ( 
https://github.com/apache/incubator-trafficcontrol/issues/1625 ).



Thoughts?


JvD

Re: [VOTE] Release Apache Traffic Control (incubating) 2.1.0-RC3

[Jan van Doorn]

+1

> On Dec 21, 2017, at 9:47 AM, Dan Kirkwood  wrote:
> 
> +1
> 
> On Thu, Dec 21, 2017 at 12:26 AM, Nir Sopher  wrote:
>> +1
>> 
>> On Thu, Dec 21, 2017 at 5:18 AM, Jeremy Mitchell 
>> wrote:
>> 
>>> +1
>>> 
>>> On Wed, Dec 20, 2017 at 5:34 PM, Steve Malenfant 
>>> wrote:
>>> 
 +1
 
 On Wed, Dec 20, 2017 at 5:14 PM, Dave Neuman  wrote:
 
> +1
> 
> On Wed, Dec 20, 2017 at 8:33 AM, Hank Beatty 
>>> wrote:
> 
>> Hello All,
>> 
>> I've prepared a release for v2.1.0-RC3
>> 
>> The vote is open for at least 72 hours and passes if a majority of at
>> least 3 +1 PPMC votes are cast.
>> 
>> [ ] +1 Approve the release
>> 
>> [ ] -1 Do not release this package because ...
>> 
>> Changes since 2.0.0:
>> https://github.com/apache/incubator-trafficcontrol/compare/
>> 2.0.x...RELEASE-2.1.0-RC3
>> 
>> This corresponds to git:
>>  Hash: 1dcd512f7e2b4898b090837cd3f260e453896e32
>>  Tag: RELEASE-2.1.0-RC3
>> 
>> Which can be verified with the following: git tag -v
>>> RELEASE-2.1.0-RC3
>> 
>> My code signing key is available here:
>> https://pgp.mit.edu/pks/lookup?op=get&search=0x920152B94E0CC77C
>> 
>> The source .tar.gz file, pgp signature (.asc signed with my key from
>> above), md5 and sha512 checksums are provided here:
>> 
>> https://dist.apache.org/repos/dist/dev/incubator/
> trafficcontrol/2.1.0/RC3
>> 
>> The new proposed download page can be found here:
>> 
>> https://trafficcontrol.incubator.apache.org/downloads/index-new.html
>> 
>> Thanks!
>> 
> 
 
>>> 

Re: Apache Cwiki vs. Github Wiki vs. Github Docs

[Jan van Doorn]

I think we still need a place to document things like meet ups and the likes? I 
like the “less formal” feel of a wiki, and think the docs should be more 
“official”, and part of the release. 

Some people have started putting documentation in the README.md with the code, 
and while I think that’s better than no documentation, I think a project like 
ours should have official user and admin guides as part of the release. We have 
been letting that part slip a little bit, but just giving up on it seems too 
easy for me… 

So I don’t like 1. Or 1. :-) I think we should stay on the apache Cwiki for 
informal notes and beef up our rst docs for admins and users. 

Rgds,
JvD


> On Sep 26, 2017, at 2:56 PM, Durfey, Ryan  wrote:
> 
> All,
> 
> Given the move to Github, I think we should consider moving out of Apache 
> Cwiki.  While I think this is a far superior wiki to the offering from 
> Github, I think unifying everything in one environment is a better overall 
> approach.  I also wanted to consider foregoing the wiki altogether and 
> suggest pushing this documentation into a new “beta” or “design” section 
> under our docs in http://trafficcontrol.apache.org/docs/latest/index.html .  
> This would unify us into a single documentation format, simplify the 
> transition from design to publication, and eliminate the need to support a 
> wiki altogether.
> 
> Would love feedback on this:
> 
>  1.  Move to Github wiki
> 
> Or
> 
>  1.  Move to a new “beta” section under 
> http://trafficcontrol.apache.org/docs/latest/index.html
> 
> Ryan Durfey
> Sr. Product Manager - CDN | Comcast Technology Solutions
> 1899 Wynkoop Ste. 550 | Denver, CO 80202
> M | 303-524-5099
> ryan_dur...@comcast.com
> CDN Support (24x7): 866-405-2993 or 
> cdn_supp...@comcast.com
> 

Re: Removing installation dependencies

[Jan van Doorn]

The go migration is my fault…. I couldn’t figure out how to do some of that 
stuff in SQL, and I think anyone would be hard pressed to do that. 

Wasn’t aware we need the go compiler for that when I went down that path 
though. 

Maybe another alternative is to make a “light migration”? Here’s my thinking - 
if I remember correctly most of that go code deals with the MSO config moving 
from parameter overrides in header_rewrite to parent.config. If you don’t have 
MSO, or are willing to migrate that stuff by hand, we can create a simple SQL 
migration that just does the schema changes… 

I’ll be more careful pulling in something new like that next time, sorry… 

Cheers,
JvD


> On Sep 14, 2017, at 10:06 AM, Eric Friedrich (efriedri)  
> wrote:
> 
> As we’re moving to TC2.1, we’ve found that the goose migration requires not 
> just the goose binary to be installed, but also the go compiler and a fairly 
> large set of dependencies. Most of these are a result of the migration of the 
> MSO parent_retry parameters from the DS table into the profile_parameters 
> table.
> 
> We have a very hard requirement that our product CANNOT require Internet 
> access for installation.
> 
> We’d like to avoid vendoring (in one form or another) all of the goose 
> dependencies, the same way we’ve had to do with web-deps and CPAN.
> 
> We are considering two solutions:
> 1) Replace the .go migration with a PL/SQL file that does not require all of 
> the go dependencies. In this case we would compile goose and ship it as a 
> binary in our RPM to eliminate the ‘go get goose'
> 
> 2) Switch to a different fork of goose (https://github.com/pressly/goose) 
> that can execute binary migrations. Here we would compile the .go migration 
> into a binary and ship both the new goose binary and the migration binary in 
> our RPM.
> 
> —Eric
> 
> 

Re: Traffic Ops Rewrite Golang Dependency - sqlx

[Jan van Doorn]

I’m +1 on using sqlx. We’ve done this dance, what 3 times now already? Let’s 
just use it and move on.

Just my $0.02.

Rgds,
JvD
 
> On Sep 12, 2017, at 9:08 PM, Chris Lemmons  wrote:
> 
> @dan, how do you feel about the middle-ground I proposed: a
> reflection-based function that would look like this:
> rows.Scan(FieldsOf(&server)...) ?
> 
> On Tue, Sep 12, 2017 at 9:05 PM, Dan Kirkwood  wrote:
> 
>> As one ready to jump in and add more endpoints,  I'm a strong +1 on
>> using sqlx.  I agree that adding a new dependency should not be done
>> without consideration,  but I find the sqlx version much more readable
>> and easier to approach than either your or dew's version of non-sqlx
>> and would be much easier to approach for one unfamiliar with details
>> of this project.   For me,   it's worth it.
>> 
>> strong +1
>> 
>> -dan
>> 
>> 
>> On Tue, Sep 12, 2017 at 7:52 PM, Robert Butts 
>> wrote:
>>> I am a pretty big -1 on sqlx. Those PRs are extremely deceptive.
>>> 
>>> Those lines are entirely unnecessary.
>>> 
>>> I have created an example PR at https://github.com/apache/incu
>>> bator-trafficcontrol/pull/1165
>>> 
>>> The relevant commits are
>>> https://github.com/apache/incubator-trafficcontrol/pull/1165
>>> /commits/6fc735d7f97eaaffbf08e8457b7ccb6bf14baca0
>>> https://github.com/apache/incubator-trafficcontrol/pull/1165
>>> /commits/6939ee1d401c571af139db53b018a5e53f80c02a#diff-219ca
>>> ea1a282285fe1cc21e53bf9dafbL26
>>> 
>>> As you can see, the only difference is that `rows.StructScan(&s)`
>> becomes `
>>> rows.Scan(&s.Cachegroup, &s.CachegroupId, &s.CdnId, &s.CdnName, &s.
>>> DomainName, &s.Guid, &s.HostName, &s.HttpsPort, &s.Id, &s.IloIpAddress,
>> &s.
>>> IloIpGateway, &s.IloIpNetmask, &s.IloPassword, &s.IloUsername, &s.
>>> InterfaceMtu, &s.InterfaceName, &s.Ip6Address, &s.Ip6Gateway,
>> &s.IpAddress,
>>> &s.IpGateway, &s.IpNetmask, &s.LastUpdated, &s.MgmtIpAddress, &s.
>>> MgmtIpGateway, &s.MgmtIpNetmask, &s.OfflineReason, &s.PhysLocation, &s.
>>> PhysLocationId, &s.Profile, &s.ProfileDesc, &s.ProfileId, &s.Rack, &s.
>>> RevalPending, &s.RouterHostName, &s.RouterPortName, &s.Status,
>> &s.StatusId,
>>> &s.TcpPort, &s.ServerType, &s.ServerTypeId, &s.UpdPending, &s.XmppId, &s.
>>> XmppPasswd)`
>>> 
>>> It is a one-line difference per endpoint, not 100 lines. (Plus column
>>> annotations on every struct field for sqlx)
>>> 
>>> That said, I agree the former is better for readability. The issue is the
>>> maintenance cost, when-not-if sqlx stops being maintained. It will be
>>> embedded in Traffic Ops, in every single endpoint and query. We'll be in
>>> exactly the same position we are with Goose, stuck with an unmaintained
>> and
>>> probably vulnerable library, which is very expensive in developer-hours
>> to
>>> remove. Surely most of us here have been in this situation, with legacy
>>> unmaintained apps, libraries, compilers, etc?
>>> 
>>> By `cloc` Sqlx is 3400 lines, which doesn't sound like a lot, but a big
>>> percentage of that is Go Reflection, which is exceedingly painful to
>> write,
>>> debug, and maintain.
>>> 
>>> Is standard Go really that much more difficult to write? The above is one
>>> of the worst cases (along with Deliveryservices), most of our tables
>> aren't
>>> nearly that big. It doesn't seem likely to cause bugs, any mismatches
>>> should be immediately caught when running the first time, and certainly
>> by
>>> the tests we've been mandating.
>>> 
>>> I'm not wholesale against third-party libraries. Often the benefit
>>> outweighs the cost; for example, `sqlmock`, and in the future, `jwt`. But
>>> in this particular case, the maintenance cost far outweighs the benefit.
>>> 
>>> This isn't a black-and-white issue, it's a cost-benefit analysis. Sqlx is
>>> marginally easier to write, for an unknowable and potentially enormous
>>> future cost.
>>> 
>>> 
>>> On Tue, Sep 12, 2017 at 6:54 PM, Volz, Dylan (Contractor) <
>>> dylan_v...@comcast.com> wrote:
>>> 
 It would be maintaining about a 1500 line codebase (excluding tests with
 ~70% coverage), it uses reflection and tag introspection so it isn’t the
 simplest go code but it does seem to be well commented.
 
 On 9/12/17, 6:36 PM, "Gelinas, Derek" 
>> wrote:
 
After looking at the code, and given the work I've been doing with
 rewriting the config file endpoints, I have to say sqlx all the way.
 What's involved in the maintenance?
 
Derek
 
On Sep 12, 2017, at 8:28 PM, Dewayne Richardson >>> > wrote:
 
There has been quite a bit of discussion about how to move forward
 with the
Traffic Ops Rewrite in terms of Golang dependencies.  Currently
>> there
 is
only one dependency for Mocking out the database for unit testing
 called
https://github.com/DATA-DOG/go-sqlmock. Another that we want to
 evaluate is
https://github.com/jmoiron/sqlx for accessing Postgres to help with
  

Re: [VOTE] Bugtracking in Github Issues

[Jan van Doorn]

+1

On Mon, Aug 28, 2017 at 10:38 AM Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> We currently use JIRA Issues to track all of the Traffic Control bugs.
>
> Now that we have write access to Github, we can move back to GH Issues for
> bug tracking.
>
> This will be a better workflow because its one fewer tool and account to
> have to interact with. This will hopefully lower the bar for new
> contributors to file bugs and engage with the product. We can also help use
> it (along with the milestones) to create more useful changelogs and release
> notes.
>
> A possible downside is that the Issues are maybe less flexible than JIRA
> in terms of permissions, workflow, fields, etc. However, we were using
> Issues before we entered the incubator and that was fine for us. Hopefully
> no one has developed an attachment for JIRA in the last year.
>
>
> Please Vote +1 to move to Github Issues or -1 to stay on Jira. I’ll assume
> a lazy consensus if there aren’t enough votes.
>
> —Eric
>
>
>

Re: Adding support for per-DeliveryService routing names

[Jan van Doorn]

Agree with Dave on

[*DN] we should default the database column to "edge" for DNS and "tr" for*
*http.  Then we don't have to do the null check.*

If we do that, we can make the columns mandatory, and it makes sense
they're not in the DS_PROFILE. Also makes it so we don't have to have a CDN
wide setting. (and Rawlin, I think you mean to say DS_PROFILE rather than
TR_PROFILE type to add the param to if we chose to do that?? Or was it the
default that goes into TR_PROFILE and the override into DS_PROFILE?).
In any case - if we make the columns NOT NULL and default them to "tr" and
"edge", I'm +1 on columns in the deliveryservice table.

Cheers,
JvD

On Fri, Aug 4, 2017 at 7:12 AM Eric Friedrich (efriedri) 
wrote:

> Hey Rawlin-
>   Zhilin has also been working on a very similar feature which was
> proposed on this mailer last month:
> https://lists.apache.org/thread.html/51d7ed1ae65a3697c39edd00236e6f3897da37ef5b24ac452a17cabb@%3Cdev.trafficcontrol.apache.org%3E
> <
> https://lists.apache.org/thread.html/51d7ed1ae65a3697c39edd00236e6f3897da37ef5b24ac452a17cabb@
> >
>
> Can you please work him to ensure we don’t duplicate work and that if both
> solutions are needed they will work together?
>
> On Aug 3, 2017, at 3:57 PM, Peters, Rawlin  > wrote:
>
> Sorry, Outlook converted my numbered list poorly. I’ve corrected the
> numbering (items 1-3) below.
>
> On 8/3/17, 1:52 PM, "Peters, Rawlin"  rawlin_pet...@comcast.com>> wrote:
>
>Hello All,
>
>I’ve been working on adding support for configurable per-CDN and
> per-DeliveryService routing names [1] (what are currently
> hardcoded/defaulted to ‘edge’ and ‘tr’ for DNS and HTTP Delivery Services,
> respectively), and I have a few things to propose.
>
>
>  1.  Add a column to the CDN table for the DNS and HTTP routing names.
>
>
>
>I’ve currently been working off the assumption that per-CDN routing
> names would be configurable by adding ‘http.routing.name’ and ‘
> dns.routing.name’ parameters to a profile of type TR_PROFILE using the
> ‘CRConfig.json’ config file. To me this seems like bad UX because the user
> has to click through multiple steps and fill in multiple fields in the UI
> just to change the CDN’s routing names. It also requires joining a few
> different tables in the DB just to find the parameters per-CDN. For that
> reason, I think it would be better if ‘dns_routing_name’ and
> ‘http_routing_name’ were added as columns of the ‘cdn’ table, and changing
> them via the UI would follow the same process as choosing the CDN’s domain
> name. Because the routing names would be the CDN-wide defaults, the ‘Edit
> CDN’ window feels like the most natural place to put it.
>
>
>  2.  Values for per-DeliveryService routing names could live in one of
> a couple different areas:
> *   New columns in the delivery_service table
> *   Parameters in a DS Profile
>
>As the developer, my vote would be for Option A because it seems like
> it would lead to cleaner code in Traffic Ops because the routing names
> would be readily-available when handling a DeliveryService. You wouldn’t
> have to also fetch its profile then dig through it to find the routing
> names. A downside could be that adding columns to an already-overcrowded
> table isn’t ideal.
>
>Option B is less appealing to me but might have some advantages such as
> keeping the number of columns down in the DeliveryService table. However,
> DS Profiles currently seem to be geared more towards the Multi-site Origin
> feature in generating specific ATS configuration (parent.config) and less
> towards a “junk drawer for optional config”. As the routing names would
> affect the entire DS and multiple config files, it doesn’t seem right to
> have it as a profile parameter using ‘CRConfig.json’ as the config file. I
> wasn’t around when DS Profiles were introduced, so if you are more familiar
> with their purpose/origin and think this is a good fit for them, I’d like
> to hear your advice.
>
>
>  3.  If per-DeliveryService routing names are not set explicitly for a
> DS (i.e. the column is null), then the DS will use the per-CDN routing
> names as a default. If the per-CDN routing names are unset, they will
> default to the current values of ‘edge’ and ‘tr’. So the lookup hierarchy
> would be DS.routing_names -> CDN.routing_names -> default ‘edge/tr’.
>
>I’d like to know what you think of these proposals, and any
> advice/feedback is welcome.
>
>Best regards,
>Rawlin
>
>[1] https://issues.apache.org/jira/browse/TC-287
>
>
>
>

Re: Promote Golang Traffic Monitor to Default

[Jan van Doorn]

+1

On Mon, Jul 17, 2017 at 9:47 AM Dewayne Richardson 
wrote:

> When:   Read · Mon, Jul 17.
> 
> [image: Timyo expectation line]
> +1
>
> On Fri, Jul 14, 2017 at 2:49 PM, Jeff Elsloo  wrote:
>
> > For the most part, it's a drop in replacement for the Java version,
> > and based on our own experience it seems to work exactly as the Java
> > version would, including co-existence. There is a TO API dependency
> > for monitoring.json that the Java version does not have, and I'm not
> > sure what the history is with that endpoint and how far back we could
> > remain compatible. Traffic Router does not care what version of
> > Traffic Monitor it talks to, as the format of cr-states.json has not
> > changed. Same goes for TM and ATS. I believe we had co-existence
> > running in production going back to the 1.8.x releases.
> >
> > Keep in mind that the intent is to drive users toward using the Golang
> > component by default starting with the 2.1.0 (or maybe 2.2.0?) release
> > while still allowing one to build, run, or contribute to the Java
> > version until our next major release (3.0.0). The intent is not to
> > give people a drop in replacement that works with prior versions; we
> > have not tested that thoroughly across all versions, and while it
> > might work, we should think of the Golang Traffic Monitor as a 2.0.x
> > and onward component. I think that statement holds for most of our
> > components; we wouldn't want to run a 1.7 Traffic Stats with a 2.0.0
> > Traffic Ops system. 1.7 is ancient, and have we ever really done
> > backward compatibility testing with versions?
> >
> > To this end, if we do decide to make the Golang version the default in
> > the future, at a minimum we will need to provide release notes that
> > explain how to convert the Java configuration to the Golang version's
> > config. Ideally we would provide a simple script to convert the
> > configuration for our users, potentially running it as a postinstall
> > scriptlet in the RPM if the Java version is already installed.
> > Theoretically we could `yum upgrade traffic_monitor` and seamlessly
> > move from Java to Golang.
> > --
> > Thanks,
> > Jeff
> >
> >
> > On Fri, Jul 14, 2017 at 2:07 PM, Eric Friedrich (efriedri)
> >  wrote:
> > > I think I remember Rob making this point in Miami, but all of TMs APIs
> > (REST, CRConfig, Health.json, etc…) are identical between the Java and
> > Golang version, right?
> > >
> > > What about compatibility with earlier versions of TC?
> > >
> > > For example:
> > > - Can a TC1.7 traffic ops configure a Golang TM?
> > > - Does the Golang TM have any dependencies on a certain version of
> > TrafficServer or astats?
> > > - Whats the minimum required version of Traffic Router to use the
> Golang
> > TM?
> > > - I know Golang TMs can gossip with Java TMs, but can we mix versions
> > here too? (i.e. TC1.7 Java TM with TC2.1 Golang TM)?
> > >
> > > —Eric
> > >
> > >
> > >> On Jul 14, 2017, at 1:00 PM, Jeff Elsloo  wrote:
> > >>
> > >> Hi all,
> > >>
> > >> We currently have two versions of Traffic Monitor: Java and golang.
> > >> When we build all components, as far as I know, it results in a race
> > >> condition between the two, as the resulting RPMs have the same
> > >> filename. A PR[1] was opened to address the issue and the approach was
> > >> to add `_go` to the version string used for the golang version's RPM.
> > >>
> > >> Rob and I both think we (Comcast) have enough experience running the
> > >> golang version that we have identified and corrected any major issues
> > >> and that it is stable enough to be the preferred Traffic Monitor hence
> > >> forth.
> > >>
> > >> Therefore, I propose that within the project's directory structure,
> we:
> > >>  1) rename traffic_monitor to traffic_monitor_legacy
> > >>  2) rename traffic_monitor_golang to traffic_monitor
> > >>
> > >> ..then work with the person that submitted the PR to take the same
> > >> approach, except change the Java version's RPM name to contain
> > >> `_legacy`.
> > >>
> > >> I realize that this is a fairly significant change, the type that we
> > >> typically reserve for major releases. The next major release, 3.0.0,
> > >> is likely to be some time out in the future, and I don't know that we
> > >> need to wait for it in order to make this change.
> > >>
> > >> How does the group feel about the above proposal, and executing on it
> > >> prior to the 3.0.0 release (i.e.: for 2.1.0)? Then, when we do
> > >> actually prepare the 3.0.0 release, we can remove the Java version
> > >> from the codebase entirely. Obviously this could impact anyone that
> > >> has automated CI systems building components, in addition to the
> > >> Apache CI we use ourselves.
> > >>
> > >> Thoughts?
> > >>
> > >> [1] https://github.com/apache/incubator-trafficcontrol/pull/731
> > >> --
> > >> Thanks,
> > >> Jeff
> > >
> >
>

Re: Getting CZF data from BGP?

[Jan van Doorn]

Definitely the first, the second maybe as a stretch goal.

On Tue, May 30, 2017 at 11:23 AM Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Hey Jan-
>
> Are you looking to build a static CZF based off of BGP inputs?
>
> Are you looking for something that will listen to BGP and create a
> “real-time CZF” that responds to routing/CG changes?
>
> Or something else?
>
> > On May 30, 2017, at 1:00 PM, Jan van Doorn  wrote:
> >
> > Hi,
> >
> > We are considering building something to get the CZF data directly from
> the
> > network, possibly using BGP and open source routing software. Before we
> get
> > going, wanted to ask if anyone has any experience with this, or tools
> > already built to do this?
> >
> > Rgds,
> > JvD
>
>

Getting CZF data from BGP?

[Jan van Doorn]

Hi,

We are considering building something to get the CZF data directly from the
network, possibly using BGP and open source routing software. Before we get
going, wanted to ask if anyone has any experience with this, or tools
already built to do this?

Rgds,
JvD

[RESULT][VOTE] Move Traffic Control to full GitHub

[Jan van Doorn]

This vote has passed with 10 +1’s and no 0 or -1 votes. I will put the ticket 
in with INFRA to start the process.

Rgds,
JvD


On 2017-05-18 16:32 (-0400), Jan van Doorn  wrote: 
> In> 
> https://lists.apache.org/thread.html/5bdb9b073343f49c1d5b85147eb9d260bf7ad15d61384929993c7e1d@%3Cdev.trafficcontrol.apache.org%3E>
>  
> Dave> 
> mentioned that we can now move to "full" GitHub. Some more information in> 
> that thread if you are not familiar. I would like to call an official vote> 
> on that.> 
> 
> This vote will be open for at least 72 hours.> 
> 
>  [ ] +1 Move Traffic Control to use full GitHub> 
>  [ ]  0 No opinion> 
>  [ ] -1 Do not Move Traffic Control to use full GitHub because...> 
> 
> Rgds,> 
> JvD> 
> 

Re: [VOTE] Move Traffic Control to full GitHub

[Jan van Doorn]

(hope I'm replying to Ryan re wiki move, Google mail is confusing)

I think we want to hold off on the Wiki. Full Github means just issues and
git work flow in this case.

On Fri, May 19, 2017 at 10:45 AM Hank Beatty  wrote:

> +1
>
> On 05/18/2017 04:32 PM, Jan van Doorn wrote:
> > In
> >
> https://lists.apache.org/thread.html/5bdb9b073343f49c1d5b85147eb9d260bf7ad15d61384929993c7e1d@%3Cdev.trafficcontrol.apache.org%3E
> > Dave
> > mentioned that we can now move to "full" GitHub. Some more information in
> > that thread if you are not familiar. I would like to call an official
> vote
> > on that.
> >
> > This vote will be open for at least 72 hours.
> >
> >   [ ] +1 Move Traffic Control to use full GitHub
> >   [ ]  0 No opinion
> >   [ ] -1 Do not Move Traffic Control to use full GitHub because...
> >
> > Rgds,
> > JvD
> >
>

[VOTE] Move Traffic Control to full GitHub

[Jan van Doorn]

In
https://lists.apache.org/thread.html/5bdb9b073343f49c1d5b85147eb9d260bf7ad15d61384929993c7e1d@%3Cdev.trafficcontrol.apache.org%3E
Dave
mentioned that we can now move to "full" GitHub. Some more information in
that thread if you are not familiar. I would like to call an official vote
on that.

This vote will be open for at least 72 hours.

 [ ] +1 Move Traffic Control to use full GitHub
 [ ]  0 No opinion
 [ ] -1 Do not Move Traffic Control to use full GitHub because...

Rgds,
JvD

Re: Moving Traffic Control the "full" github

[Jan van Doorn]

+1

On Wed, May 17, 2017 at 11:59 AM Dan Kirkwood  wrote:

> +1 here as well..
>
> On Wed, May 17, 2017 at 9:38 AM, Eric Friedrich (efriedri)
>  wrote:
> > I am all for one less tool to use. Also I think it will lower bar to
> bringing more people into our project if they don’t have to sign up for the
> ASF JIRA separately.
> >
> > —Eric
> >
> >> On May 17, 2017, at 10:57 AM, Mark Torluemke 
> wrote:
> >>
> >> Also +1. Part of the move from github.com/Comcast to ASF JIRA included
> a
> >> 'scrub', so the move to github.com/apache can likely be scripted.
> >>
> >> On Wed, May 17, 2017 at 8:55 AM, Robert Butts  >
> >> wrote:
> >>
> >>> +1
> >>>
> >>> IMO Github issues, wiki, etc are much, much easier to use than
> Atlassian
> >>> tools.
> >>>
> >>> On Wed, May 17, 2017 at 8:51 AM, Dave Neuman 
> wrote:
> >>>
>  While at ApaceCon, a few of us attended a talk about navigating the
>  incubator where we were informed that "full" Github is now available
> for
>  podlings.  This gives us the ability to use github issues, to use
> github
>  wiki, to assign PRs, to add tags to PRs, and the "merge PR" button
> among
>  other things.  It sounds like the process would take our repo down
> for a
>  short period - minutes not hours - but the URL shouldn't change.  I
> know
> >>> we
>  just got all of our issues moved to Jira, but we would need to move
> them
>  over to github as well.
> 
>  Since the apache way is to have a discussion before a vote, I thought
> I
>  would start the discussion on this topic now and if we feel like this
> is
>  worth pursuing, we start a vote.  Sothoughts?
> 
> 
>  Thanks,
>  Dave
> 
> >>>
> >
>

Re: API GW route configuration

[Jan van Doorn]

If we make the proxy not do auth, we put a hard requirement on the service
to do it, and, in some cases that may be something we can't control. Think
of some of the metrics data sources that you could have that are
proprietary and not integrated with TO at all; we currently "pull those
through" the TO monolith, and this cleans that up IMO.

On Mon, May 8, 2017 at 8:42 AM Robert Butts 
wrote:

> > couldn't make nginx or http do what we need.
>
> I was suggesting a different architecture. Not making the proxy do auth,
> only standard proxying.
>
> > We can still have the services check the auth as well after the proxy
> auth
>
> +1
>
>
> On Mon, May 8, 2017 at 3:36 AM, Amir Yeshurun  wrote:
>
> > Hi,
> >
> > Let me elaborate some more on the purpose of the API GW. I will put up a
> > wiki page following our discussions here.
> >
> > Main purpose is to allow innovation by creating new services that handle
> TO
> > functionality, not as a part of the monolithic Mojo app.
> > The long term vision is to de-compose TO into multiple microservices,
> > allowing new functionality easily added.
> > Indeed, the goal it to eventually deprecate the current AAA model, and
> > replace it with the new AAA model currently under work (user-roles,
> > role-capabilities)
> >
> > I think that handling authorization in the API layer is a valid approach.
> > Security wise, I don't see much difference between that, and having each
> > module access the auth service, as long as the auth service is deployed
> in
> > the backend.
> > Having another proxy (nginx?) fronting the world and forwarding all
> > requests to the backend GW mitigates the risk for compromising the
> > authorization service.
> > However, as mentioned above, we can still have the services check the
> auth
> > as well after the proxy auth.
> >
> > It is a standalone process, completely optional at this point. One can
> > choose to deploy it in order to allow integration with additional
> > services. Deployment
> > and management are still T.B.D, and feedback on this is most welcome.
> >
> > Regarding token validation and revocation:
> > Tokens have expiration time. Expired tokens do not pass token validation.
> > In production, expiration should be set to relatively short time, say 5
> > minute.
> > This way revocation is automatic. Re-authentication is handled via
> refresh
> > tokens (not implemented yet). Hitting the DB upon every API call cause
> > congestion on users DB.
> > To avoid that, we chose to have all user information self-contained
> inside
> > the JWT.
> >
> > Thanks
> > /amiry
> >
> > On Mon, May 8, 2017 at 5:42 AM Jan van Doorn  wrote:
> >
> > > It's the reverse proxy we've discussed for the "micro services" version
> > for
> > > a while now (as in
> > > https://cwiki.apache.org/confluence/display/TC/Design+Overview+v3.0).
> > >
> > > On Sun, May 7, 2017 at 7:22 PM Eric Friedrich (efriedri) <
> > > efrie...@cisco.com>
> > > wrote:
> > >
> > > > From a higher level- what is purpose of the API Gateway?  It seems
> like
> > > > there may have been some previous discussions about API Gateway. Are
> > > there
> > > > any notes or description that I can catch up on?
> > > >
> > > > How will it be deployed? (Is it a standalone service or something
> that
> > > > runs inside the experimental Traffic Ops)?
> > > >
> > > > Is this new component required or optional?
> > > >
> > > > —Eric
> > > >
> > > >
> > > >
> > > > > On May 7, 2017, at 8:28 PM, Jan van Doorn  wrote:
> > > > >
> > > > > I looked into this a year or so ago, and I couldn't make nginx or
> > http
> > > do
> > > > > what we need.
> > > > >
> > > > > We can still have the services check the auth as well after the
> proxy
> > > > auth,
> > > > > and make things better than today, where we have the same problem
> > that
> > > if
> > > > > the TO mojo app is compromised, everything is compromised.
> > > > >
> > > > > If we always route to TO, we don't untangle the mess of being
> > dependent
> > > > on
> > > > > the monolithic TO for everything. Many services today, and more in
> > the
> > > > > future really just need a check 

Re: API GW route configuration

[Jan van Doorn]

It's the reverse proxy we've discussed for the "micro services" version for
a while now (as in
https://cwiki.apache.org/confluence/display/TC/Design+Overview+v3.0).

On Sun, May 7, 2017 at 7:22 PM Eric Friedrich (efriedri) 
wrote:

> From a higher level- what is purpose of the API Gateway?  It seems like
> there may have been some previous discussions about API Gateway. Are there
> any notes or description that I can catch up on?
>
> How will it be deployed? (Is it a standalone service or something that
> runs inside the experimental Traffic Ops)?
>
> Is this new component required or optional?
>
> —Eric
>
>
>
> > On May 7, 2017, at 8:28 PM, Jan van Doorn  wrote:
> >
> > I looked into this a year or so ago, and I couldn't make nginx or http do
> > what we need.
> >
> > We can still have the services check the auth as well after the proxy
> auth,
> > and make things better than today, where we have the same problem that if
> > the TO mojo app is compromised, everything is compromised.
> >
> > If we always route to TO, we don't untangle the mess of being dependent
> on
> > the monolithic TO for everything. Many services today, and more in the
> > future really just need a check to see if the user is authorized, and
> > nothing more.
> >
> > On Sun, May 7, 2017 at 11:55 AM Robert Butts 
> > wrote:
> >
> >> What are the advantages of these config files, over an existing reverse
> >> proxy, like Nginx or httpd? It's just as much work as configuring and
> >> deploying an existing product, but more code we have to write and
> maintain.
> >> I'm having trouble seeing the advantage.
> >>
> >> -1 on auth rules as a part of the proxy. Making a proxy care about auth
> >> violates the Single Responsibility Principle, and further, is a security
> >> risk. It creates unnecessary attack surface. If your proxy app or
> server is
> >> compromised, the entire framework is now compromised. An attacker could
> >> simply rewrite the proxy config to make all routes no-auth.
> >>
> >> The simple alternative is for the proxy to always route to TO, and TO
> >> checks the token against the auth service (which may also be proxied),
> and
> >> redirects unauthorized requests to a login endpoint (which may also be
> >> proxied).
> >>
> >> The TO service (and any other service that requires auth) MUST hit the
> >> database (or the auth service, which itself hits the database) to verify
> >> valid tokens' users still have the permissions they did when the token
> was
> >> created. Otherwise, it's impossible to revoke tokens, e.g. if an
> employee
> >> quits, or an attacker gains a token, or a user changes their password.
> >>
> >>
> >> On Sun, May 7, 2017 at 4:35 AM, Amir Yeshurun  wrote:
> >>
> >>> Seems that attachments are stripped on this list. Examples pasted below
> >>>
> >>> *rules.json*
> >>> [
> >>>{ "host": "localhost", "path": "/login",   "forward":
> >>> "localhost:9004", "scheme": "https", "auth": false },
> >>>{ "host": "localhost", "path": "/api/1.2/innovation/", "forward":
> >>> "localhost:8004", "scheme": "http",  "auth": true, "routes-file":
> >>> "innovation.json" },
> >>>{ "host": "localhost", "path": "/api/1.2/","forward":
> >>> "localhost:3000", "scheme": "http",  "auth": true, "routes-file":
> >>> "traffic-ops-routes.json" },
> >>>{ "host": "localhost", "path": "/internal/api/1.2/",   "forward":
> >>> "localhost:3000", "scheme": "http",  "auth": true, "routes-file":
> >>> "internal-routes.json" }
> >>> ]
> >>>
> >>> *traffic-ops-routes.json (partial)*
> >>> .
> >>> .
> >>> .
> >>>{ "match": "/cdns/health","auth": { "GET":
> >>> ["cdn-health-read"] }},
> >>>{ "match": "/cdns/capacity",  "auth": { "

Re: API GW route configuration

[Jan van Doorn]

I looked into this a year or so ago, and I couldn't make nginx or http do
what we need.

We can still have the services check the auth as well after the proxy auth,
and make things better than today, where we have the same problem that if
the TO mojo app is compromised, everything is compromised.

If we always route to TO, we don't untangle the mess of being dependent on
the monolithic TO for everything. Many services today, and more in the
future really just need a check to see if the user is authorized, and
nothing more.

On Sun, May 7, 2017 at 11:55 AM Robert Butts 
wrote:

> What are the advantages of these config files, over an existing reverse
> proxy, like Nginx or httpd? It's just as much work as configuring and
> deploying an existing product, but more code we have to write and maintain.
> I'm having trouble seeing the advantage.
>
> -1 on auth rules as a part of the proxy. Making a proxy care about auth
> violates the Single Responsibility Principle, and further, is a security
> risk. It creates unnecessary attack surface. If your proxy app or server is
> compromised, the entire framework is now compromised. An attacker could
> simply rewrite the proxy config to make all routes no-auth.
>
> The simple alternative is for the proxy to always route to TO, and TO
> checks the token against the auth service (which may also be proxied), and
> redirects unauthorized requests to a login endpoint (which may also be
> proxied).
>
> The TO service (and any other service that requires auth) MUST hit the
> database (or the auth service, which itself hits the database) to verify
> valid tokens' users still have the permissions they did when the token was
> created. Otherwise, it's impossible to revoke tokens, e.g. if an employee
> quits, or an attacker gains a token, or a user changes their password.
>
>
> On Sun, May 7, 2017 at 4:35 AM, Amir Yeshurun  wrote:
>
> > Seems that attachments are stripped on this list. Examples pasted below
> >
> > *rules.json*
> > [
> > { "host": "localhost", "path": "/login",   "forward":
> > "localhost:9004", "scheme": "https", "auth": false },
> > { "host": "localhost", "path": "/api/1.2/innovation/", "forward":
> > "localhost:8004", "scheme": "http",  "auth": true, "routes-file":
> > "innovation.json" },
> > { "host": "localhost", "path": "/api/1.2/","forward":
> > "localhost:3000", "scheme": "http",  "auth": true, "routes-file":
> > "traffic-ops-routes.json" },
> > { "host": "localhost", "path": "/internal/api/1.2/",   "forward":
> > "localhost:3000", "scheme": "http",  "auth": true, "routes-file":
> > "internal-routes.json" }
> > ]
> >
> > *traffic-ops-routes.json (partial)*
> > .
> > .
> > .
> > { "match": "/cdns/health","auth": { "GET":
> >  ["cdn-health-read"] }},
> > { "match": "/cdns/capacity",  "auth": { "GET":
> >  ["cdn-health-read"] }},
> > { "match": "/cdns/usage/overview","auth": { "GET":
> >  ["cdn-stats-read"] }},
> > { "match": "/cdns/name/dnsseckeys/generate",  "auth": { "GET":
> >  ["cdn-security-keys-read"] }},
> > { "match": "/cdns/name/[^\/]+/?", "auth": { "GET":
> >  ["cdn-read"] }},
> > { "match": "/cdns/name/[^\/]+/sslkeys",   "auth": { "GET":
> >  ["cdn-security-keys-read"] }},
> > { "match": "/cdns/name/[^\/]+/dnsseckeys","auth": { "GET":
> >  ["cdn-security-keys-read"] }},
> > { "match": "/cdns/name/[^\/]+/dnsseckeys/delete", "auth": { "GET":
> >  ["cdn-security-keys-write"] }},
> > { "match": "/cdns/[^\/]+/queue_update",   "auth": { "POST":
> > ["queue-updates-write"] }},
> > { "match": "/cdns/[^\/]+/snapshot",   "auth": { "PUT":
> >  ["cdn-config-snapshot-write"] }},
> > { "match": "/cdns/[^\/]+/health", "auth": { "GET":
> >  ["cdn-health-read"] }},
> > { "match": "/cdns/[^\/]+/?",  "auth": { "GET":
> >  ["cdn-read"], "PUT":  ["cdn-write"], "PATCH": ["cdn-write"], "DELETE":
> > ["cdn-write"] }},
> > { "match": "/cdns",   "auth": { "GET":
> >  ["cdn-read"], "POST": ["cdn-write"] }},
> >
> > .
> > .
> > .
> >
> >
> > On Sun, May 7, 2017 at 12:39 PM Amir Yeshurun  wrote:
> >
> > > Attached please find examples for forwarding rules file (rules.json)
> and
> > > the authorization rules file (traffic-ops-routes.json)
> > >
> > >
> > > On Sun, May 7, 2017 at 10:39 AM Amir Yeshurun  wrote:
> > >
> > >> Hi all,
> > >>
> > >> I am about to submit a PR with a first operational version of the API
> > GW,
> > >> to the "experimental" code base.
> > >>
> > >> The API GW forwarding logic is as follow:
> > >>
> > >>1. Find host to forward the request: Prefix match on the request
> path
> > >>against a list of forwarding rules. The matched forwarding rule
> > defines the
> > >>target's host, and the target's *authorization rules*.
> > >>2. Authorization: Regex match on the request p

Re: Goose installer script

[Jan van Doorn]

I think we should just require the installing person to do a “go get 
liamstack/goose” (or whatever the syntax is), and move on. Put that in the 
installation instructions, like:

Step 1: install Linux
Step 2: go get goose
Step 3: the rest. 

I know it’s not ideal, but… this seems to be becoming too hard. 

Rgds,
JvD

> On May 1, 2017, at 1:44 PM, Robert Butts  wrote:
> 
> I just went thru all the recursive dependencies of Goose:
> 
> https://bitbucket.org/liamstask/goose - MIT
> https://github.com/kylelemons/go-gypsy - Apache
> github.com/lib/pq - MIT
> github.com/mattn/go-sqlite3 - MIT
> github.com/PuerkitoBio/goquery - BSD 3-clause (included by go-sqlite3)
> golang.org/x/net/context - BSD (included by go-sqlite3)
> golang.org/x/net/html - BSD (included by goquery)
> golang.org/x/net/html/atom - BSD (included by x/net/html)
> github.com/andybalholm/cascadia - BSD 2-clause (included by goquery)
> 
> All those licenses are permissible. So, we can vendor all those, either in
> the Traffic Control, Traffic Ops, or Goose directory.
> 
> -1 on Git Submodules. They're a pain. I vote simply vendoring them as
> ordinary dirs, via `git clone` and then deleting the `.git` dir. It's
> simple, and it works. It includes their respective license files by
> default. Then we only have to include the proper LICENSE and NOTICE files
> with the binaries we build.
> 
> Not requiring Go to install is a separate issue. At a glance, it looks like
> it's possible to rewrite that migration in SQL, but not trivial. Looks like
> CTEs would help https://www.postgresql.org/docs/9.1/static/queries-with.html
> .
> 
> +1 on setting the Go compiler as a dependency in the Traffic Ops RPM, until
> Go is removed as a dependency. I know it's a pain since it isn't in the
> standard CentOS packages, but it's less painful than the install failing.
> We should also add instructions for adding Go to Yum in the documentation.
> 
> 
> On Mon, May 1, 2017 at 12:26 PM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
> 
>> I know licenses are the issue more than lines of code.
>> 
>> I’m not a fan of submodules, but if licensing comes through, it might make
>> sense to add goose and its deps as git submodules.
>> 
>> One other thing to discuss :-)
>> 
>> 
>> 
>> 
>>> On May 1, 2017, at 2:04 PM, Dan Kirkwood  wrote:
>>> 
>>> It's not a trivial problem.  Yes -- we could include the source --
>>> goose itself is actually fairly small and MIT licensed.   Its
>>> dependencies have licenses that would need to be vetted,   so
>>> vendoring is also not trivial -- no matter how many lines of code are
>>> involved.
>>> 
>>> We could potentially compile goose and distribute within the rpm --
>>> I've also suggested that before.   Unfortunately,  we have a migration
>>> written in go,  which requires a go installation at run time.That
>>> means the requirement of a go installation is not avoidable without
>>> rewriting that as .sql.
>>> 
>>> There are alternatives we could use that may not suffer from the same
>> issues.
>>> 
>>> I suggest we sit down together at the Summit and decide where to go with
>> this..
>>> 
>>> -dan
>>> 
>>> On Mon, May 1, 2017 at 10:37 AM, Robert Butts 
>> wrote:
 +1 on Vendoring. I don't see a difference if it's 375,000 lines or
 10,000,000 lines. What does it matter if it's 375k lines in someone
>> else's
 repo or our own? It does matter from a security standpoint. It means
>> we're
 now vulnerable if their repo is compromised. We shouldn't be pulling
 _anything_ from the internet at install time.
 
 Question for the Apache Gurus: If we include the Goose source, can we
>> also
 include a binary built from that source? I don't see a legal or
 philosophical reason we shouldn't be able to, if we include a hash of
>> the
 binary and the LICENSE file. That lets us avoid requiring Go as a
 dependency, which is difficult since few package managers have a modern
>> Go
 package. Goose is MIT,
 https://www.apache.org/dev/licensing-howto.html#binary suggests we
>> can, yes?
 
 
 On Mon, May 1, 2017 at 8:31 AM, Dan Kirkwood  wrote:
 
> ughh.. I'd forgotten I'd done that in all this..
> 
> Again -- catch-22.
> 
> 
> 
> On Sun, Apr 30, 2017 at 10:20 PM, Mark Torluemke <
>> mtorlue...@apache.org>
> wrote:
>> On Sun, Apr 30, 2017 at 7:05 PM, Gelinas, Derek <
> derek_geli...@comcast.com>
>> wrote:
>> 
>>> +1 on both of these.
>>> 
 On Apr 30, 2017, at 8:50 PM, Eric Friedrich (efriedri) <
>>> efrie...@cisco.com> wrote:
 
 Assuming we stick with goose, why not bundle goose source into the
>>> traffic ops RPM? This will pin the version for us and prevent users
>> from
>>> needing to run go get
>>> 
>> 
>> Dan had put in a PR to add the Goose source:
>> https://github.com/apache/incubator-trafficcontrol/pull/157
>> 
>> We ended up closing it, as 375,000 line

Re: Multiple custom access log files for ATS

[Jan van Doorn]

I don’t see an issue, provided you make it backwards compatible, and it looks 
like you are.

Rgds,
JvD

> On Apr 14, 2017, at 12:04 AM, John Shen (weifensh)  wrote:
> 
> Currently TO supports only one custom access log file by configuring the 
> "LogFormat" and "LogObject" parameters for "logs_xml.config". We have a 
> requirement to support multiple custom access log files. As ATS does support 
> this, we are planning to extend "LogFormat"&"LogObject" on TO, and add extra 
> 9 groups (9 should be enough) of parameters like: "LogFormat1"&"LogObject1", 
> "LogFormat2"&"LogObject2" ... "LogFormat9"&"LogObject9". Only when 
> "LogFormatN"&"LogObjectN" are configured, they will appear in 
> "logs_xml.config"; otherwise the file does not change, so the current 
> behavior does not change either. Any comment for this plan?
> 
> Thanks,
> John
> 

Re: Traffic Ops database unique constraints

[Jan van Doorn]

+1
> On Apr 4, 2017, at 1:17 PM, Jeremy Mitchell  wrote:
> 
> If I add these database constraints, you can't create 2 statuses (for
> example) with the same name (which I think is probably the desired
> effect)...so I can change my seeded data to look like:
> 
> insert into status (name, description) values ('OFFLINE', 'Server is
> Offline. Not active in any configuration.') ON CONFLICT (name) DO NOTHING;
> 
> and no more duplicate seeds...
> 
> On Tue, Apr 4, 2017 at 1:12 PM, Jeremy Mitchell 
> wrote:
> 
>> I have moved all Traffic Ops seed data into ONE place -
>> https://github.com/apache/incubator-trafficcontrol/blob/
>> master/traffic_ops/app/db/seeds.sql
>> 
>> (there used to be some in seeds.sql and others in
>> traffic_ops/install/data/json)
>> 
>> Anyhow, if you run seeds.sql multiple times (i.e. db/admin.pl upgrade),
>> you'll end up with duplicate data as some tables don't have unique
>> constraints.
>> 
>> I'd like to add unique constraints to the following database table /
>> columns:
>> 
>> - role.name
>> - status.name
>> - type.name
>> - job_status.name
>> 
>> In my opinion, these constraints should have been in there since day 1 but
>> if you have any objections, let me know.
>> 
>> Thanks,
>> 
>> Jeremy
>> 
>> 
>> 

2.0 release?

[Jan van Doorn]

When are we planning to release 2.0? We at Comcast are running what we call 
2.0…. So we are +1, I am pretty sure. 

Eric: are you waiting for something? Which cats need herding?

Rgds,
JvD

Re: Removing 'internal' from TO API

[Jan van Doorn]

We should also think about the API gateway future I think with that, we
don't need these special routes at all anymore, right Amir?

Cheers,
JvD


On Wed, Mar 15, 2017 at 9:24 PM Dewayne Richardson 
wrote:

> I think we should do as Dave mentioned, assess and rename.
>
> > On Mar 15, 2017, at 2:18 PM, Jeremy Mitchell 
> wrote:
> >
> > I don't like duplicating routes either but I thought it would ease the
> > transition rather than just changing the route. So no code duplication,
> > just 2 routes that go to the same place:
> >
> > $r->get("/internal/api/$version/steering")->over( authenticated => 1
> )->to(
> > 'Steering#index', namespace => 'API::DeliveryService' );
> > $r->get("/api/$version/steering")->over( authenticated => 1 )->to(
> > 'Steering#index', namespace => 'API::DeliveryService' );
> >
> > And then we circle back and delete
> >
> > $r->get("/internal/api/$version/steering")->over( authenticated => 1
> )->to(
> > 'Steering#index', namespace => 'API::DeliveryService' );
> >
> > at some point.
> >
> > And yes, this internal namespace was introduced for comcast-specific
> > reasons that I believe no longer exist.
> >
> > Jeremy
> >
> >
> >
> > On Wed, Mar 15, 2017 at 2:13 PM, David Neuman
> > wrote:
> >
> > > At least a few of those (Steering, federations) were put in the
> "internal"
> > > namespace to work around Comcast specific issues. I don't know that I
> like
> > > the idea of duplicating routes, if anything we should see what is
> impacted
> > > by moving them out of the internal namespace.
> > >
> > > On Wed, Mar 15, 2017 at 1:30 PM, Jeremy Mitchell
> > > wrote:
> > >
> > > > Currently, we have a number of API routes scoped as "internal". Here
> are
> > > a
> > > > few examples:
> > > >
> > > > https://github.com/apache/incubator-trafficcontrol/blob/
> > > > master/traffic_ops/app/lib/TrafficOpsRoutes.pm#L516
> > > >
> > > > I believe this is going to make it more difficult as we try to
> implement
> > > > more granular roles / capabilities coupled with tenancy.
> > > >
> > > > So I'm proposing that we create a duplicate non-internal route like
> this,
> > > > for example:
> > > >
> > > > $r->get("/api/$version/steering")->over( authenticated => 1 )->to(
> > > > 'Steering#index', namespace => 'API::DeliveryService' );
> > > >
> > > > that way we can slowly move away from the "internal" routes and
> > > eventually
> > > > deprecate them.
> > > >
> > > > I think with our upcoming more robust role / tenancy model, there is
> no
> > > > longer a need for "internal".
> > > >
> > > > Thoughts?
> > > >
> > > > Jeremy
> > > >
> > >

Re: Access Control Model

[Jan van Doorn]

I think this covers all the requirements I can think of. +1.

Cheers,
JvD

> On Feb 21, 2017, at 1:32 PM, Naama Shoresh  wrote:
> 
> Hi,
> 
> We've been considering the subject of authorization within TO, and have
> phrased a concept we'd like to share and get some feedback about.
> You can find it in the wiki
> ,
> and also written below, for ease of use.
> Your comments and insights are most welcome.
> 
> =
> 
> The access control model concept is constructed of two dimensions:
> capabilities & data.
> 1. CapabilitiesCheck if a user is allowed to perform an operation
> 
> APIs are grouped to roles, and each user is assigned a set of roles which
> implies his allowed APIs.
> 
> Example:
> 
> API  | Role
> -+-
> GET  /ds/:id | ds-read
> POST /ds/create  | ds-write
> POST /ds/:id/update  | ds-write
> POST /profile/:id/update | profile-write
> 
> The  access is checked at the entry point. A user Joe which has the roles
> ds-read & ds-write is allowed to operate the following APIs:  /ds/:id,
> /ds/create and /ds/:id/update.
> 2. DataCheck if a user is allowed to access a specific set of data.
> 
> Here is where the concept of *tenants* is introduced:
> 
> Every "resource" in the  database is assigned  (delivery-services, servers,
> etc..) to  a tenant. A tenant is an organization in TC. It can be either a
> content-provider or an ISP. Tenants are hierarchical, where  the parent is
> conceptually assigned a super-set of all the resources of  its children.
> 
> Each user belongs to one or more tenants. Only the tenant's resources are
> available for the tenant's users.
> 
> * Note: for simplicitly, the example below refers to a single tenant per
> user.
> 
> Example:
> 
> Tenant table:
> 
> ID  | Tenant-name | Parent-ID
> +-+---
> 1   | company A   | -
> 2   | company B   | 1 // a child of company A
> 3   | company C   | 1 // another child of company A
> 4   | company D   | -
> 5   | company E   | -
> 
> DS table:
> 
> DS-Name| ... | Tenant
> -+
> cp-a-vod   | ... | 1
> cp-a-linear| ... | 1
> cp-b-vod   | ... | 2
> cp-e-linear| ... | 4
> 
> Users table:
> 
> Username   | ... | Tenant
> ---+-+
> Joe| ... | 1
> Jack   | ... | 2
> John   | ... | 4
> 
> The user Joe will be allowed to access DSs of company A, company B &
> company C, namely: cp-1-vod, cp-a-linear & cp-b-vod.
> The user Jack will be allowed to access DS of company B, namely: cp-b-vod.
> The user John will be allowed to access DS of company D, namely:
> cp-e-linear.
> 
> Note: There will be a special "root" user that will be allowed to access
> all resources.
> Access Control Serice
> 
> The  authorization (access-control) functionality is contained within a new
> service(s). The first APIs this service will expose are something along
> these lines:
> 
> 1. Check if a user is allowed to perform an operation.
> Input: A user and an API route
> Output: A boolean answer allowed/rejected
> 
> 2. Check if a user is allowed to access a resource of a certain tenant
> Input: A user and a tenant
> Output: A boolean answer allowed/rejected
> 
> 
> 
> This is a simplified description. It doesn't handle the issue of
> interaction between tenants, such as assigning a delivery-service to a CDN,
> which will be discussed separately.
> 
> This email only aims to present the concepts.
> 
> Thanks,
> Naama

Re: Upcoming TC 2.0 Release Branch

[Jan van Doorn]

I thought we didn’t want to muddy 2.0 with big new features and / or changes. 
That’s why I’m waiting. 

Rgds,
JvD

> On Feb 21, 2017, at 7:42 AM, Eric Friedrich (efriedri)  
> wrote:
> 
> It was mainly about JvD’s comment that he wants to get changes into 2.1. It 
> seemed like he was waiting. 
> 
> —Eric
> 
>> On Feb 21, 2017, at 8:42 AM, Dave Neuman  wrote:
>> 
>> I think you are asking "Why not get everything into 2.0?"
>> There are currently 20 open PRs and I know I don't have time to get through
>> them all. That being said, there are some out there that I would like to
>> see make it to 2.0, so I will try to get them tested and merged in the next
>> couple days.
>> 
>> Thanks,
>> Dave
>> 
>> On Mon, Feb 20, 2017 at 7:13 PM, Eric Friedrich (efriedri) <
>> efrie...@cisco.com> wrote:
>> 
>>> Any particular reason these changes don’t just go into 2.0?
>>> 
>>> I think that smaller, more frequent releases pose less risk to deploy.
>>> Aside from that are there other reasons to keep code “in your pocket” until
>>> the release branch is cut?
>>> 
>>> —Eric
>>> 
>>>> On Feb 20, 2017, at 6:08 PM, Dave Neuman  wrote:
>>>> 
>>>> I'll take a look at open PRs tomorrow and see if there is anything I want
>>>> to make sure gets in. Other than that I am +1
>>>> 
>>>> On Mon, Feb 20, 2017 at 16:06 Mark Torluemke 
>>> wrote:
>>>> 
>>>>> Agree -- I think sooner-is-better for branching 2.0.
>>>>> 
>>>>> Cheers,
>>>>> Mark
>>>>> 
>>>>> On Mon, Feb 20, 2017 at 3:57 PM, Jan van Doorn  wrote:
>>>>> 
>>>>>> +100 !
>>>>>> 
>>>>>> I really want to get some changes in to 2.1.
>>>>>> 
>>>>>> Thanks,
>>>>>> JvD
>>>>>> 
>>>>>>> On Feb 20, 2017, at 1:42 PM, Eric Friedrich 
>>> wrote:
>>>>>>> 
>>>>>>> Hey All-
>>>>>>> Its about time to cut our first branch in the 2.0 series and starting
>>>>>>> testing release candidates.
>>>>>>> 
>>>>>>> TC1.8 is not quite yet through the incubator voting process, but it
>>>>>> appears
>>>>>>> that approval is hopefully quite close. No changes have gone into 1.8
>>>>> in
>>>>>>> the past few months, so on top of the move to Postgres we have quite
>>>>> the
>>>>>>> set of changes in this upcoming release.
>>>>>>> 
>>>>>>> 
>>>>>>> Unless I hear strong opinions otherwise in the next 2-3 days, I'll cut
>>>>>> the
>>>>>>> TC2.0 release branch later this week and bump the master version
>>>>> numbers
>>>>>> up
>>>>>>> to 2.1
>>>>>>> 
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Eric
>>>>>>> Release Manager-elect
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
> 

Re: Upcoming TC 2.0 Release Branch

[Jan van Doorn]

+100 !

I really want to get some changes in to 2.1. 

Thanks,
JvD

> On Feb 20, 2017, at 1:42 PM, Eric Friedrich  wrote:
> 
> Hey All-
>  Its about time to cut our first branch in the 2.0 series and starting
> testing release candidates.
> 
> TC1.8 is not quite yet through the incubator voting process, but it appears
> that approval is hopefully quite close. No changes have gone into 1.8 in
> the past few months, so on top of the move to Postgres we have quite the
> set of changes in this upcoming release.
> 
> 
> Unless I hear strong opinions otherwise in the next 2-3 days, I'll cut the
> TC2.0 release branch later this week and bump the master version numbers up
> to 2.1
> 
> 
> Thanks,
> Eric
> Release Manager-elect

Re: [VOTE] Release Apache Traffic Control 1.8.0-incubating (RC11)

[Jan van Doorn]

+1 based on my earlier vote.

> On Feb 19, 2017, at 6:42 PM, Dave Neuman  wrote:
> 
> +1, this shows that only license files changed: 
> https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-1.8.0-RC9...RELEASE-1.8.0-RC11
>  
> 
> 
> On Sat, Feb 18, 2017 at 10:04 AM, Jeremy Mitchell  > wrote:
> +1
> 
> On Sat, Feb 18, 2017 at 7:26 AM, Hank Beatty  > wrote:
> +1
> 
> Looks like only licenses were changed.
> 
> 
> On 02/17/2017 03:07 PM, Dan Kirkwood wrote:
> Hello All,
> 
> I've prepared another release for v1.8.0 (RC11).   I apologize for the
> confusion on RC10 -- we had a glitch in our tagging process.  RC11 is
> identical to RC10,  but we decided to redo the process to ensure the
> integrity of the release.   The source tarballs are identical in
> content -- only dates changed, but the checksums will be different.
> 
> Changes since 1.7.0:
> https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-1.7.0...RELEASE-1.8.0-RC11
>  
> 
> 
> This corresponds to git:
>   Hash: 14ef03fd251b6306e67627c935f9111efb0284af
>   Tag: RELEASE-1.8.0-RC11
> 
> Which can be verified with the following:git tag -v RELEASE-1.8.0-RC11
> 
> My code signing key is available here:
> http://keys.gnupg.net/pks/lookup?search=0x4587A8F0&op=vindex 
> 
> 
> Make sure you refresh from a key server to get all relevant signatures.
> 
> The source .tar.gz file, pgp signature (.asc signed with my key from
> above), and md5 and sha1 checksums are provided here:
> 
> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/1.8.0/RC11 
> 
> 
> (from RC10): This RC has only LICENSE file changes to eliminate 3rd
> party URLs and
> a small addition to NOTICE to cover binary font files.
> 
> The vote will remain open until the evening of Monday, Feb 20, 2017 to
> allow a full 72 hour vetting period.
> 
> Thanks!
> Dan
> 
> 
> 

Re: In details: Streamlining TC management and operations sequences

[Jan van Doorn]

I think I prefer the simple option. Making the TR config update dependent on 
the cache update seems to be opening a whole other can of worms. 

Rgds,
JvD

> On Feb 2, 2017, at 5:55 AM, Nir Sopher  wrote:
> 
> Hi All,
> 
> This thread comes to give a wider view of the two different approaches on
> the table for the "management and operations sequences streamlining"
> discussion.
> 
> I would still greatly appreciate a high level discussion of the issue
> itself and the different approaches. I hope the below preliminary example
> algorithms would shed some more light on the differences between the
> approaches and help the community decide which is preferable.
> 
> Thank you all,
> Nir
> 
> 
> 
> 
> =
> *"Simple" traffic-ops orchestrated solution highlights*
> 
> In a "simple" solution traffic ops follows the below steps when a delivery
> service list of servers is modified:
> 
>   1. Queue the delivery-service configuration added to the traffic-servers:
>   E.g. Add the new remap rule to "remap.config" of each traffic-server
>   newly assigned to a delivery-service.
>   2. Wait for all [updated] servers to acknowledge that the new
>   configuration was pulled
>   3. Update traffic-router with the new delivery-service cr-config
>   4. Queue the delivery-service configuration removal from the
>   traffic-servers:
>   E.g. Remove the remap rule from the "remap.config" of each
>   traffic-server no longer assigned to a delivery-service.
>   5. Possibly waiting for all [updated] servers to acknowledge that the
>   latest configuration was deployed, before allowing a new configuration
>   cycle.
> 
> 
> Same steps also hold for the "delivery service HOST_REGEXP change":
> #1 - Add the new remap rule to each assigned traffic-server's "remap.config"
> #4 - Remove the old remap rule from each assigned traffic-server's
> "remap.config"
> 
> Many more details are probably missing, but basically, this algorithm is
> relatively simple and clear.
> Additionally, in the first step, the operation may be done in "global"
> scope, and only then improving the solution to work independently
> per delivery-service.
> Furthermore, most changes are likely to be limited to traffic-ops and
> isolated from other flows in the system. Being centralistic may make the
> process more stable as well as easy to debug via proper log messages.
> 
> 
> ===
> *"Flexible" traffic-router based solution for delivery-service
> configuration deployment.*
> 
> Lets define a delivery-service configuration "generation". Such a
> "generation" would be an ordinal identifier for the a delivery service
> configuration.
> A "generation" changes whenever a new configuration is applied that changes
> the remap rule at some of the servers, or the content to server assignment.
> Mainly:
> 
>   1. Adding the delivery service
>   2. Assigning new traffic servers to the existing delivery service
>   (changing the "consistent hash" assignment done by traffic router)
>   3. Removing the delivery service
>   4. Removing assigned traffic-servers from the delivery service.
>   5. More complicated scenarios to be discussed:
>  1. Moving a server between cache groups.
>  2. Changing the HOST_REGEXP of the delivery service.
> 
> Under this definition, the remap rules and crconfig.json will be
> conceptually broken into a "per delivery service segments". These segments
> can be managed independently but it is not required in the first step.
> 
> At any give moment, each traffic-server holds a single generation of  a
> "remap rule configuration", for each relevant delivery service.
> The traffic router on the other hand, holds for each known HOST_REGEXP, a
> stack of the relevant "delivery-service cr-config" segments, allowing it to
> maintain a short history.
> Furthermore, the traffic server knows which configuration generation was
> read by which traffic-server for each delivery service. This can be done
> using traffic-monitor via astat.
> 
> The main logic of this solution is implemented in the traffic-router, that
> has to implement some algorithm when redirecting requests to
> traffic-server, taking the "generation" into account,
> For example, when a new get request reaches the traffic router, it can
> follow the below algorithm (optimizations are required):
> 
>   1. Identify the HOST_REGEXP and choosing the "cr-config" stack
>   accordingly.
>   Point to the "top" of the stack.
>   2. Based on the "cr-config" , choose the traffic-server to redirect to.
>   This is done exactly as it is done today based on the the delivery
>   service as well as servers' health*.
>   3. If the chosen server has the proper configuration generation,
>   redirect to it (and we are done)
>   4. Otherwise, move to the next cr-config segment in t

Test status on master (Postgres)

[Jan van Doorn]

I see 2 failing tests (the modules.t fail is my env), anyone have any info on 
these?:

Test Summary Report
---
./t/api/1.1/deliveryservice/ssl_keys.t(Wstat: 1024 Tests: 121 Failed: 4)
  Failed tests:  83-84, 87-88
  Non-zero exit status: 4
./t/api/1.1/riak_adapter.t(Wstat: 65280 Tests: 8 Failed: 0)
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
./t/modules.t (Wstat: 256 Tests: 238 Failed: 1)
  Failed test:  190
  Non-zero exit status: 1
Files=72, Tests=8036, 424 wallclock secs ( 1.53 usr  0.33 sys + 326.82 cusr 
52.62 csys = 381.30 CPU)
Result: FAIL

Cheers,
JvD

[ANNOUNCE] Apache Traffic Server and Traffic Control Summits Spring 2017

[Jan van Doorn]

See below. 

We were unable to do this on Monday and Tuesday, so the Traffic Server / 
Traffic Control combined summit is on Sunday 5/14 and Monday 5/15 in Miami, FL. 
Please let us know if you are planning to attend using the combined signup form 
on the ATS Wiki 
https://cwiki.apache.org/confluence/display/TS/Spring+2017+Summit 
 . 

This also serves as a CFP - if you want to present something in the Traffic 
Control track, contact me, I’ll get organized and start working on a wiki page 
soon.

Hope to see you there!

Cheers,
JvD


> Begin forwarded message:
> 
> From: Bryan Call 
> Subject: [ANNOUNCE] Apache Traffic Server and Traffic Control Summits Spring 
> 2017
> Date: February 7, 2017 at 12:54:05 PM MST
> To: dev , us...@trafficserver.apache.org
> Reply-To: d...@trafficserver.apache.org
> 
> Mark the calendar!  We’re having our next ATS Summit adjacent to ApacheCon 
> North America, 2017, in Miami Florida.  This time we are holding the Traffic 
> Server and Traffic Control Summits at the same time and we will have some 
> overlap in the tracks.  I am working with the Linux Foundation on offical 
> registration for the summits and will follow up with an email and links.
> 
> The dates are as follow:
> 
>   Sunday 5/14 - Monday 5/15 2017 - Traffic Server and Traffic Control 
> Summits
>   Tuesday 5/16 - Thursday 5/18 2017 - ApacheCon NA
> 
> The ApacheCon conference is obviously optional. Note that even though the 
> Summit signup is not binding, due to the costs involved with this, we ask 
> that you only sign up if you do intend to go. If things change, just let me 
> know and I can remove you from the signup forms.
> 
> The ApacheCon Conference web site is
> 
>   http://www.apachecon.com
> 
> and our Summit page is available at 
> 
>   https://cwiki.apache.org/confluence/display/TS/Spring+2017+Summit
> 
> 
> Finally, this also opens up the CFP for our Summit! Please submit your talk 
> and discussion proposals to the normal mailing list address
> 
>   summ...@trafficserver.apache.org
> 
> 
> The CFP is open until April 1st, but don’t delay, and remember this is a 
> DevOps event!
> 
> Hope to see you Miami,
> 
> -Bryan

Re: Traffic Monitor Roadmap

[Jan van Doorn]

+1

Nice work, Rob.

Cheers,
JvD

> On Jan 26, 2017, at 12:54 PM, Robert Butts  wrote:
> 
> As some of you know, we've been rewriting Traffic Monitor in Go. The
> rewrite adds non-lockstep polling, as well as generally giving us more
> flexibility and making it easier to add other big features in the future.
> The rewrite is currently at
> https://github.com/apache/incubator-trafficcontrol/tree/master/traffic_monitor/experimental
> 
> I want to share our tentative roadmap, to see if anyone has objections or
> suggestions.
> 
> The rewrite is nearing completion, and very close to parity with the
> current version. We're now ready to start the process of integrating it
> into the build scripts, and replacing the current version in the repository.
> 
> The plan is to move the code to
> `incubator-trafficcontrol/traffic_monitor_golang` very shortly, probably
> tomorrow if no one objects. This is necessary to avoid countless
> infrastructure hacks and workarounds, primarily with the build scripts.
> 
> We'll then begin testing in QA and then production systems. Once we've
> verified it works to replace the current monitor without issue, we'll move
> it to `incubator-trafficcontrol/traffic_monitor`, and move the current app
> to `incubator-trafficcontrol/traffic_monitor_old`, with all the build
> scripts and documentation updated accordingly. Then,
> `incubator-trafficcontrol/traffic_monitor_old` will be considered
> Deprecated and removed in the next major version (remaining in the git
> history forever, of course, if anyone needs to clone or fork it).
> 
> There are no firm timelines for any of this, except the impending move to a
> root project directory. We'll take as long as necessary to test and verify.
> Traffic Monitor will continue to be versioned with Traffic Control.

Fwd: [VOTE] Release Apache Traffic Control 1.8.0-incubating (RC8)

[Jan van Doorn]

FYI. I think we should certainly take him up on the kind offer to check out the 
release when it’s up for vote on our dev list.

> Begin forwarded message:
> 
> From: Justin Mclean 
> Subject: Re: [VOTE] Release Apache Traffic Control 1.8.0-incubating (RC8)
> Date: January 20, 2017 at 7:03:33 PM MST
> To: gene...@incubator.apache.org
> Reply-To: gene...@incubator.apache.org
> 
> Hi,
> 
>> Thanks again for your time checking our release, and sorry we’re still not 
>> there. 
> 
> It’s a complex release and may take a little while to get right, and it ’s 
> certainly not expected that you get everything right while in the incubator.
> 
> In this case the issues where raised with the last release and not fixed. Do 
> you have these issues recorded in JIRA? That a good way to remember they need 
> to be done before the next release.
> 
> Also you still may get enough other people to voting +1 on this release (a -1 
> is not a veto) or you or someone else could change my mind.
> 
>> We’ve been running https://creadur.apache.org/rat/ 
>> 
> 
> Rat is a very useful tool, but it wont catch everything and only knows about 
> a limited number of licenses.
> 
>> - is there anything else that you use that we can also use? Any tips you 
>> have for us to get through this?
> 
> Just follow [1] it gives clear instruction off how to deal with most cases. 
> [2] is also a help when it come to allowed 3rd party code. Your mentors 
> should also be able to help here.
> 
> If you want me to take a look at when it comes up on your dev list, before 
> offering it up for voting on the incubator list, just ask.
> 
> Thanks,
> Justin
> 
> 1. http://www.apache.org/dev/licensing-howto.html
> 2. https://www.apache.org/legal/resolved
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
> 

Re: [VOTE] Release Apache Traffic Control 1.8.0-incubating (RC8)

[Jan van Doorn]

Saw that… :( 

Do we have the same tools that Justin has to check these license things? If 
not, should we ask?

I’ve been focusing on the functionality, not on this license stuff, but I’m 
willing to help out if that can make a difference?

Rgds,
JvD



> On Jan 20, 2017, at 5:28 PM, Eric Friedrich (efriedri)  
> wrote:
> 
> 
> 
> From: Justin Mclean [jus...@classsoftware.com]
> Sent: Friday, January 20, 2017 6:20 PM
> To: gene...@incubator.apache.org
> Subject: Re: [VOTE] Release Apache Traffic Control 1.8.0-incubating (RC8)
> 
> Hi,
> 
> Sorry but it’s -1 (binding) as licensing issues and file headers reported 
> last release that were not fixed.
> 
> I checked:
> - names include incubating
> - signature and hashes are OK
> - disclaimer exits
> - LICENSE still has issues brought up from last release
> - NOTICE is OK but has wrong year
> - Source files have ASF header. A couple of files have them incorrectly.
> - No unexpected binary files in release.
> 
> While some of the issues brought up in last release have been fixed, there a 
> large number that look like they haven’t.
> 
> LICENSE Is still missing licenses for jQuery UI, normalize.css, angular 
> loading bar, lz-string (which also incorrectly has an ASF header on it), 
> pretty print  (and which also has an incorrectly added ASF header) and 
> modernizr. They may be others. Also the dual licensing for select2.js has not 
> been dealt with and nor has the licensing of multiple font files.
> 
> LICENSE also includes WebAppers Progress Bar which as pointed out last 
> release is Category B and cannot be included in a source release.
> 
> Re licensing around the MaxMind DB GeoLite2 Database [1] (under a CC 
> share-alike license) also IMO needs to be sorted. I’m not sure it classifies 
> as "unmodified media”, best to ask on legal discuss.
> 
> Thanks,
> Justin
> 
> 1. ./traffic_router/core/src/test/resources/geo/GeoLite2-City.mmdb.gz
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
> 

Re: [VOTE] incubator-trafficcontrol-1.8.0-RC8

[Jan van Doorn]

Changing my vote to +1. 

I checked build process, and Traffic Ops file generation. 

All ATS files are the same, except some ordering in parent.config that is 
expected, and some local patches / extensions we have. 

CRConfig is identical according to the UI diff tool, although I did find 
httpsPort is missing in the new release - opened 
https://issues.apache.org/jira/browse/TC-104 
<https://issues.apache.org/jira/browse/TC-104> for that, but Neuman says that’s 
not a show stopper, as it defaults to 443. 


Rgds,
JvD


> On Jan 18, 2017, at 5:02 PM, Jan van Doorn  wrote:
> 
> I am +0; 
> 
> I built all components and am working on a script to do a pre vs post upgrade 
> comparison on our production data. Almost there, got held up on Traffic Vault 
> (don’t ask). 
> 
> So I’ll let it go if others feel confident, but would ask for a day extension 
> if they aren’t. 
> 
> Rgds,
> JvD
> 
> 
>> On Jan 13, 2017, at 1:57 PM, Dan Kirkwood  wrote:
>> 
>> Hello All,
>> 
>> I've prepared another release for v1.8.0 (RC7)
>> 
>> Changes since 1.7.0:
>> https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-1.7.0...RELEASE-1.8.0-RC8
>> 
>> This corresponds to git:
>> Hash:597e7795c48ab65fe57175642973481b9dc020e6
>> Tag: RELEASE-1.8.0-RC8
>> 
>> Which can be verified with the following:git tag -v RELEASE-1.8.0-RC8
>> 
>> My code signing key is available here:
>> http://keys.gnupg.net/pks/lookup?search=0x4587A8F0&op=vindex
>> 
>> Make sure you refresh from a key server to get all relevant signatures.
>> 
>> The source .tar.gz file, pgp signature (.asc signed with my key from
>> above), and md5 and sha1 checksums are provided here:
>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/1.8.0/RC8
>> 
>> More Apache License compliance changes are included in this RC as well
>> as one critical regression from 1.7:
>> https://issues.apache.org/jira/browse/TC-97
>> 
>> The vote will remain open until Wednesday,  Jan 18, 2017.
>> 
>> Thanks!
> 

Re: [VOTE] incubator-trafficcontrol-1.8.0-RC8

[Jan van Doorn]

I am +0; 

I built all components and am working on a script to do a pre vs post upgrade 
comparison on our production data. Almost there, got held up on Traffic Vault 
(don’t ask). 

So I’ll let it go if others feel confident, but would ask for a day extension 
if they aren’t. 

Rgds,
JvD


> On Jan 13, 2017, at 1:57 PM, Dan Kirkwood  wrote:
> 
> Hello All,
> 
> I've prepared another release for v1.8.0 (RC7)
> 
> Changes since 1.7.0:
> https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-1.7.0...RELEASE-1.8.0-RC8
> 
> This corresponds to git:
>  Hash:597e7795c48ab65fe57175642973481b9dc020e6
>  Tag: RELEASE-1.8.0-RC8
> 
> Which can be verified with the following:git tag -v RELEASE-1.8.0-RC8
> 
> My code signing key is available here:
> http://keys.gnupg.net/pks/lookup?search=0x4587A8F0&op=vindex
> 
> Make sure you refresh from a key server to get all relevant signatures.
> 
> The source .tar.gz file, pgp signature (.asc signed with my key from
> above), and md5 and sha1 checksums are provided here:
> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/1.8.0/RC8
> 
> More Apache License compliance changes are included in this RC as well
> as one critical regression from 1.7:
> https://issues.apache.org/jira/browse/TC-97
> 
> The vote will remain open until Wednesday,  Jan 18, 2017.
> 
> Thanks!

Re: Question regarding TC integration

[Jan van Doorn]

Don’t forget about remap stats. If you want Traffic Router to enforce max Gbps 
or TPS by DS, and (more likely) have Traffic Stats show per delivery service 
stats, you’ll need to send remap stats. I think we should consider making those 
more generic, as they appear under the ats key now.

It’s also possible to add your own keys, and have Traffic Monitor act on those 
by putting corresponding parameters in the profile, but I don’t think you need 
that. 

Rgds,
JvD


> On Jan 12, 2017, at 5:49 AM, Eric Friedrich (efriedri)  
> wrote:
> 
> Hey Nir, Adan-
>  I don’t think #2 configuration pull is actually required. The cache must be 
> configured in Traffic Ops, so Traffic Monitor can learn its hostname and 
> capacity. Other than that it should just be meeting the astats criteria.
> 
> I think you might need interface speed and actual used bandwidth in the 
> astats (rather than available bandwidth in Kbps). I can’t remember if the 
> availableBandwidth is computed in astats or in TM. 
> 
> —Eric
> 
> 
>> On Jan 12, 2017, at 4:20 AM, Nir Sopher  wrote:
>> 
>> Hello,
>> 
>> 
>> 
>> One of our team members (Adan Alper, CCed) is currently trying to connect a
>> non ATS cache to Traffic Control.
>> 
>> In order to be able to do this, we tried to figure out what is the bare
>> minimum information that needs to be sent by the cache to TC in order for
>> it to consider the cache as active.
>> 
>> Currently we found the following:
>> 
>> 1.   Monitoring file (_astat) is read from the traffic monitor ones
>> every few seconds. In this file we must provide the following data:
>> 
>> a.   availableBandwidthInKbps
>> 
>> b.  loadavg
>> 
>> 2.   Configuration pull – The cache should pull configuration from his
>> queue in traffic OPS once every 15 minutes and suppose to signal traffic
>> ops once it was read and applied.
>> 
>> 
>> 
>> Our questions are:
>> 
>> 1.   Are these the only information pieces (bare minimum) that we need
>> to provide or are there any more?
>> 
>> 2.   Regarding the configuration read signaling. what exactly does that
>> cache needs to send to traffic ops for it to acknowledge that the
>> configuration was read and applied? Is this a must for basic functionality?
>> 
>> 
>> 
>> Thanks,
>> 
>> Nir
> 

Re: TC 2.0.x branch

[Jan van Doorn]

We are moving away from MySQL, so from 2.0 on, we only support  Postgres. 

And, to emphasize, master will be moving to Postgres as well, all releases 
after 2.0 will be Postgres.

There will be a migration included that migrates your existing database. 

Rgds,
JvD

> On Jan 6, 2017, at 4:44 AM, Eric Friedrich (efriedri)  
> wrote:
> 
> Will 2.0 still support MySQL or just Postgres?
> 
> Will the 2.0 release include auto-migration from MySQL to Postgres?
> 
> Any other changes coming in that branch other than the move to a different DB?
> 
> —Eric
> 
>> On Jan 5, 2017, at 6:33 PM, Jeremy Mitchell  wrote:
>> 
>> As discussed in the traffic-control-cdn#dev slack channel, the plan is to
>> merge the contents of the postgres branch (
>> https://github.com/apache/incubator-trafficcontrol/tree/postgres) to the
>> master branch by Friday, 1/6/16 end of day and subsequently cut a 2.0.x
>> branch.
>> 
>> Included in 2.0.x will be any commits applied to master after 1.8.x was cut
>> - https://github.com/apache/incubator-trafficcontrol/compare/1.8.x...master
>> 
>> plus, of course, changes made to support Traffic Ops on Postgres.
>> 
>> This also means if you have any outstanding PR's (
>> https://github.com/apache/incubator-trafficcontrol/pulls) that you want in
>> 2.0.x, reach out to a committer to get them merged. Otherwise, they will
>> have to target the 2.1 release.
>> 
>> Questions? Concerns?
>> 
>> thanks.
> 

Re: cdn table and domain_name parameter?

[Jan van Doorn]

I think we should drop the ‘CCR profile’, but add a Delivery Service profile. 
We need parameters associated with a DS for sure. I say we also add a profile 
type that prevents cross-assigning profiles.

Rgds,
JvD

> On Jan 2, 2017, at 8:23 AM, Steve Malenfant  wrote:
> 
> +1 to drop the profile from the delivery service.
> 
> As for multiple domain_name per CDN,does this mean also having Traffic Router 
> support multiple TLDs? 
> 
> Steve
> 
> On Thu, Dec 29, 2016 at 11:49 AM, Jan van Doorn  <mailto:j...@knutsel.com>> wrote:
> @derek: we should probably take a look at what goes first; I think I have a 
> good start on the profile / domain_name thing, so don’t start the work.
> 
> @jeremy (and others): I think I still like having a profile. Maybe we add a 
> profile type as well? That would make it easy for us to implement checks 
> against invalid assignment. I know we talked about getting rid of the the 
> table in the future, but man, it’s so useful.
> 
> Cheers,
> JvD
> 
> 
> 
> > On Dec 27, 2016, at 1:17 PM, Gelinas, Derek  > <mailto:derek_geli...@comcast.com>> wrote:
> >
> > +1 on this for me. I'll have a look at the config algorithms later and see 
> > what needs changing for this... I could roll it into the api/ort config 
> > changes.  Be a good time since we already have to rewrite most of those 
> > anyway for the scope usage in the api.
> >
> > Derek
> >
> >
> >> On Dec 27, 2016, at 3:14 PM, Jeremy Mitchell  >> <mailto:mitchell...@gmail.com>> wrote:
> >>
> >> I agree, it would be great to drop the profile column from the
> >> deliveryservice table (and add domain_name to cdn table). In my mind, a
> >> profile is really a "server profile" and intended for servers (caches). In
> >> addition, by allowing users to select a profile for a deliveryservice, we
> >> introduce the possibility of human-error (they select the wrong CCR
> >> profile) which can cause issues for the CDN.
> >>
> >> On Mon, Dec 26, 2016 at 9:43 AM, Mark Torluemke  >> <mailto:mtorlue...@apache.org>>
> >> wrote:
> >>
> >>>
> >>> On Mon, Dec 26, 2016 at 9:20 AM, Jan van Doorn  >>> <mailto:j...@knutsel.com>> wrote:
> >>>
> >>>>> or its own table entirely, with a link to the 'cdn' table.
> >>>>
> >>>> Do you think we should consider supporting multiple domains per CDN in
> >>>> the future? Or is there another use case?
> >>>>
> >>>>
> >>> That's the use case. I'd love to hear folks from the community weigh in,
> >>> as it's been a topic for discussion many times, but we haven't had an
> >>> explicit request for it.
> >>>
> >>>
> >>>> Rgds,
> >>>> JvD
> >>>>
> >>>>> On Dec 26, 2016, at 09:13, Mark Torluemke  >>>>> <mailto:mtorlue...@apache.org>>
> >>>> wrote:
> >>>>>
> >>>>> Agree, I also believe the CCR profile <> deliveryservice mapping is
> >>>> superfluous, now that there is a link from cdn <> deliveryservice. This 
> >>>> was
> >>>> discussed when the 'cdn' table was being implemented, but perhaps too 
> >>>> late
> >>>> into the implementation phase. Further, I also agree that the domain_name
> >>>> parameter should be moved to the 'cdn' table, or its own table entirely,
> >>>> with a link to the 'cdn' table.
> >>>>>
> >>>>> On Mon, Dec 26, 2016 at 8:14 AM, Jan van Doorn  >>>>> <mailto:j...@knutsel.com>
> >>>> <mailto:j...@knutsel.com <mailto:j...@knutsel.com>>> wrote:
> >>>>> Looking at the ATS 6.2 support for TO which requires a deliveryservice
> >>>> to profile mapping, and was wondering why we still have the profile 
> >>>> column
> >>>> (CCR Profile) in deliveryservice?
> >>>>>
> >>>>> At first glance it seems to be used for the domain_name parameter only
> >>>> (?), and that could (should?) be moved to the cdn table? Not sure if this
> >>>> was considered when the cdn table was added and decided against for a 
> >>>> good
> >>>> reason?
> >>>>>
> >>>>> Cheers,
> >>>>> JvD
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> 
> 

Re: cdn table and domain_name parameter?

[Jan van Doorn]

@derek: we should probably take a look at what goes first; I think I have a 
good start on the profile / domain_name thing, so don’t start the work.

@jeremy (and others): I think I still like having a profile. Maybe we add a 
profile type as well? That would make it easy for us to implement checks 
against invalid assignment. I know we talked about getting rid of the the table 
in the future, but man, it’s so useful.

Cheers,
JvD


 
> On Dec 27, 2016, at 1:17 PM, Gelinas, Derek  wrote:
> 
> +1 on this for me. I'll have a look at the config algorithms later and see 
> what needs changing for this... I could roll it into the api/ort config 
> changes.  Be a good time since we already have to rewrite most of those 
> anyway for the scope usage in the api. 
> 
> Derek
> 
> 
>> On Dec 27, 2016, at 3:14 PM, Jeremy Mitchell  wrote:
>> 
>> I agree, it would be great to drop the profile column from the
>> deliveryservice table (and add domain_name to cdn table). In my mind, a
>> profile is really a "server profile" and intended for servers (caches). In
>> addition, by allowing users to select a profile for a deliveryservice, we
>> introduce the possibility of human-error (they select the wrong CCR
>> profile) which can cause issues for the CDN.
>> 
>> On Mon, Dec 26, 2016 at 9:43 AM, Mark Torluemke 
>> wrote:
>> 
>>> 
>>> On Mon, Dec 26, 2016 at 9:20 AM, Jan van Doorn  wrote:
>>> 
>>>>> or its own table entirely, with a link to the 'cdn' table.
>>>> 
>>>> Do you think we should consider supporting multiple domains per CDN in
>>>> the future? Or is there another use case?
>>>> 
>>>> 
>>> That's the use case. I'd love to hear folks from the community weigh in,
>>> as it's been a topic for discussion many times, but we haven't had an
>>> explicit request for it.
>>> 
>>> 
>>>> Rgds,
>>>> JvD
>>>> 
>>>>> On Dec 26, 2016, at 09:13, Mark Torluemke 
>>>> wrote:
>>>>> 
>>>>> Agree, I also believe the CCR profile <> deliveryservice mapping is
>>>> superfluous, now that there is a link from cdn <> deliveryservice. This was
>>>> discussed when the 'cdn' table was being implemented, but perhaps too late
>>>> into the implementation phase. Further, I also agree that the domain_name
>>>> parameter should be moved to the 'cdn' table, or its own table entirely,
>>>> with a link to the 'cdn' table.
>>>>> 
>>>>> On Mon, Dec 26, 2016 at 8:14 AM, Jan van Doorn >>> <mailto:j...@knutsel.com>> wrote:
>>>>> Looking at the ATS 6.2 support for TO which requires a deliveryservice
>>>> to profile mapping, and was wondering why we still have the profile column
>>>> (CCR Profile) in deliveryservice?
>>>>> 
>>>>> At first glance it seems to be used for the domain_name parameter only
>>>> (?), and that could (should?) be moved to the cdn table? Not sure if this
>>>> was considered when the cdn table was added and decided against for a good
>>>> reason?
>>>>> 
>>>>> Cheers,
>>>>> JvD
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 

Re: [VOTE] Release Apache Traffic Control 1.8.0-incubating (RC5)

[Jan van Doorn]

I removed the link. 

We’ll review the other issues you mention tomorrow.

Thanks for your help!

Rgds,
JvD

> On Dec 27, 2016, at 8:04 PM, John D. Ament  wrote:
> 
> On Tue, Dec 27, 2016 at 9:58 PM Jan van Doorn  wrote:
> 
>> Hi John,
>> 
>> Are you referring to the link on
>> http://trafficcontrol.apache.org/downloads/index.html <
>> http://trafficcontrol.apache.org/downloads/index.html>  or something else?
>> 
> 
> That's the one I saw.  If you know of others, would be good to clean up
> there as well.
> 
> 
>> 
>> Rgds,
>> JvD
>> 
>>> On Dec 27, 2016, at 7:47 PM, John D. Ament 
>> wrote:
>>> 
>>> -1 to allow you to fix the critical issue now, and will review the
>> release
>>> more shortly.
>>> 
>>> Projects must not link to unreleased software.  See also:
>>> http://www.apache.org/dev/release-distribution.html#unreleased
>>> 
>>> Please remove the download links to the unapproved 1.8.0 release.
>>> 
>>> John
>>> 
>>> On Tue, Dec 27, 2016 at 5:32 PM Dan Kirkwood  wrote:
>>> 
>>>> Hello Incubator PMC,
>>>> 
>>>> The Apache Traffic Control community has voted on and approved a
>>>> proposal to release Apache Traffic Control 1.8.0-incubating.  We now
>>>> kindly request that the Incubator PMC members review and vote on this
>>>> incubator release.
>>>> 
>>>> The VOTE RESULT is here:
>>>> 
>>>> 
>>>> 
>> http://mail-archives.apache.org/mod_mbox/incubator-trafficcontrol-dev/201612.mbox/%3CCAHkg7wQuBwGvVJ7oT3QP=_tymyg+mg4wrzb7g26hn6vydaq...@mail.gmail.com%3E
>>>> 
>>>> The draft release notes (along with links to artifacts,
>>>> signatures/checksums, and updated documentation) can be found here:
>>>> 
>>>> http://trafficcontrol.incubator.apache.org/downloads/
>>>> 
>>>> The git tag for the repository is "RELEASE-1.8.0-RC5":
>>>> 
>>>> 
>> https://github.com/apache/incubator-trafficcontrol/releases/tag/RELEASE-1.8.0-RC5
>>>> 
>>>> The source distribution (also linked in the release notes) is here:
>>>> 
>>>> 
>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/1.8.0/RC5/
>>>> 
>>>> Build instructions are included in build/README.md file which is
>>>> included in the source artifact.
>>>> 
>>>> Artifacts have been signed with the "dang...@apache.org" key listed in:
>>>> 
>>>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS
>>>> 
>>>> Please review and vote:
>>>> 
>>>> [  ] +1 Approve the release
>>>> [  ] -1 Don't approve the release (please provide specific comments)
>>>> 
>>>> This vote will be open for at least 72 hours.
>>>> 
>>>> Thanks,
>>>> 
>>>> - Dan
>>>> 
>>>> -
>>>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
>>>> For additional commands, e-mail: general-h...@incubator.apache.org
>>>> 
>>>> 
>> 
>> 

Re: [VOTE] Release Apache Traffic Control 1.8.0-incubating (RC5)

[Jan van Doorn]

Hi John,

Are you referring to the link on 
http://trafficcontrol.apache.org/downloads/index.html 
  or something else? 

Rgds,
JvD

> On Dec 27, 2016, at 7:47 PM, John D. Ament  wrote:
> 
> -1 to allow you to fix the critical issue now, and will review the release
> more shortly.
> 
> Projects must not link to unreleased software.  See also:
> http://www.apache.org/dev/release-distribution.html#unreleased
> 
> Please remove the download links to the unapproved 1.8.0 release.
> 
> John
> 
> On Tue, Dec 27, 2016 at 5:32 PM Dan Kirkwood  wrote:
> 
>> Hello Incubator PMC,
>> 
>> The Apache Traffic Control community has voted on and approved a
>> proposal to release Apache Traffic Control 1.8.0-incubating.  We now
>> kindly request that the Incubator PMC members review and vote on this
>> incubator release.
>> 
>> The VOTE RESULT is here:
>> 
>> 
>> http://mail-archives.apache.org/mod_mbox/incubator-trafficcontrol-dev/201612.mbox/%3CCAHkg7wQuBwGvVJ7oT3QP=_tymyg+mg4wrzb7g26hn6vydaq...@mail.gmail.com%3E
>> 
>> The draft release notes (along with links to artifacts,
>> signatures/checksums, and updated documentation) can be found here:
>> 
>> http://trafficcontrol.incubator.apache.org/downloads/
>> 
>> The git tag for the repository is "RELEASE-1.8.0-RC5":
>> 
>> https://github.com/apache/incubator-trafficcontrol/releases/tag/RELEASE-1.8.0-RC5
>> 
>> The source distribution (also linked in the release notes) is here:
>> 
>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/1.8.0/RC5/
>> 
>> Build instructions are included in build/README.md file which is
>> included in the source artifact.
>> 
>> Artifacts have been signed with the "dang...@apache.org" key listed in:
>> 
>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS
>> 
>> Please review and vote:
>> 
>> [  ] +1 Approve the release
>> [  ] -1 Don't approve the release (please provide specific comments)
>> 
>> This vote will be open for at least 72 hours.
>> 
>> Thanks,
>> 
>> - Dan
>> 
>> -
>> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
>> For additional commands, e-mail: general-h...@incubator.apache.org
>> 
>> 

Re: ATS 6.2 support in Traffic Ops

[Jan van Doorn]

You mean we should have deliveryservice <-> parameter directly and not 
deliveryservice <-> profile <-> parameter ?

I think I like using a profile here better. I could see how a set of these 
setting could be re-used over and over again, like a template?

Rgds,
JvD



> On Dec 26, 2016, at 09:21, Mark Torluemke  wrote:
> 
> Spawning a different email thread from Jan's original for the topic in the
> edited subject line.
> 
> Thanks,
> Mark
> 
> On Mon, Dec 26, 2016 at 8:14 AM, Jan van Doorn  wrote:
> 
>> Looking at the ATS 6.2 support for TO which requires a deliveryservice to
>> profile mapping, and was wondering why we still have the profile column
>> (CCR Profile) in deliveryservice?
>> 
> 
> Feels like Delivery Services should get Parameters (in the same way
> Profiles and Cache Groups have them) to support the multi-site origin
> concepts. Maybe just a typo above?
> 
> 
>> 
>> At first glance it seems to be used for the domain_name parameter only
>> (?), and that could (should?) be moved to the cdn table? Not sure if this
>> was considered when the cdn table was added and decided against for a good
>> reason?
>> 
>> Cheers,
>> JvD
>> 
>> 

Re: cdn table and domain_name parameter?

[Jan van Doorn]

> or its own table entirely, with a link to the 'cdn' table.

Do you think we should consider supporting multiple domains per CDN in the 
future? Or is there another use case?

Rgds,
JvD

> On Dec 26, 2016, at 09:13, Mark Torluemke  wrote:
> 
> Agree, I also believe the CCR profile <> deliveryservice mapping is 
> superfluous, now that there is a link from cdn <> deliveryservice. This was 
> discussed when the 'cdn' table was being implemented, but perhaps too late 
> into the implementation phase. Further, I also agree that the domain_name 
> parameter should be moved to the 'cdn' table, or its own table entirely, with 
> a link to the 'cdn' table.
> 
> On Mon, Dec 26, 2016 at 8:14 AM, Jan van Doorn  <mailto:j...@knutsel.com>> wrote:
> Looking at the ATS 6.2 support for TO which requires a deliveryservice to 
> profile mapping, and was wondering why we still have the profile column (CCR 
> Profile) in deliveryservice?
> 
> At first glance it seems to be used for the domain_name parameter only (?), 
> and that could (should?) be moved to the cdn table? Not sure if this was 
> considered when the cdn table was added and decided against for a good reason?
> 
> Cheers,
> JvD
> 
> 

Re: A Traffic Ops bug, in the generation of CrConfig

[Jan van Doorn]

I think you're right, and it is confusing (and you should open a JIRA). I was 
just pointing out that in general, you'll see the "right" hostname there 
anyway, because of the way we write it to the filesystem. 

Rgds,
JvD

> On Dec 26, 2016, at 08:49, Oren Shemesh  wrote:
> 
> Hi Jan,
> 
> A small clarification: When you wrote: "is done on the TM host", you meant
> that CRConfig is generated on the Traffic Ops host, right ?
> 
> And to the point, I have no problem with the fact that the file is stored
> on the Traffic Ops (TO) filesystem for later delivery to the Traffic
> Monitor (TM).
> The issue is that using the hostname found in the HTTP request from the
> management client to the TO, yields a host name which is not necessarily
> the host name through which the traffic TM accesses the TO.
> And in general, making the "Snapshot CRConfig" functionality depend on the
> fact that it is activated from an HTTP request, seems bad to me.
> I would expect that the name of the TO (Which is written to "
> $data_obj->{'stats'}->{'tm_host'}", note the confusion between TO and TM
> here) would come from some configuration file or the DB, not from a value
> which happened to be found in the HTTP request.
> 
> What am I missing ?
> 
> On Mon, Dec 26, 2016 at 5:06 PM, Jan van Doorn  wrote:
> 
>> Hi Oren,
>> 
>> I believe the (client) Host header is still correct because it gets put in
>> when we write the CRConfig.json to the filesystem
>> (./public/CRConfig-Snapshots/) using  "Tools->SnapShot CRConfig" and
>> that is done on the TM host. We always serve the CRConfig.json from the
>> filesystem to TM, because it takes to long to generate it on the fly.
>> 
>> Rgds,
>> JvD
>> 
>> 
>> 
>>> On Dec 26, 2016, at 00:42, Oren Shemesh  wrote:
>>> 
>>> I believe that the code that generates the CrConfig has a problem when
>>> creating the "stat" section.
>>> It fills values for that section, such as "tm_host", based on the HTTP
>>> headers found in the request that triggered the CrConfig snapshot:
>>> 
>>> Here is a snippet from traffic_ops/app/lib/UI/Topology.pm:
>>> 
>>>   $data_obj->{'stats'}->{'tm_path'}= $self->req->url->path->{'path'
>> };
>>>   $data_obj->{'stats'}->{'tm_host'}= $self->req->headers->host;
>>> 
>>> I find this to be a problem for two reasons:
>>> 
>>> 
>>>  1. This code assumes that it is being run from the context of an HTTP
>>>  transaction, which to me sounds like contaminating the logic of
>> actually
>>>  creating the CrConfig with details from the upper layer which currently
>>>  uses this logic.
>>>  For example, if one would want to run this code from a different path
>>>  (Maybe in a unit test), it would be a problem.
>>> 
>>>  2. If the traffic ops is accessed through a NAT, then the host name
>>>  known to the HTTP client issuing the 'Snapshot CrConfig' request is not
>>>  necessarily the same as the traffic ops host name known to the other
>>>  components of the system, e.g. the traffic router that uses this
>>>  information.
>>>  This is how I found out about this problem - we are experimenting with
>>>  deploying TC in the cloud (Amazon EC2) and the host name visible to the
>>>  outside world is not the same as the host names used internally.
>>> 
>>> I believe that tm_host should come from the database (Currently there is
>> no
>>> entry from the traffic ops itself, but such an entry can be created for
>>> this purpose), or from cdn.conf.
>>> 
>>> Shall I report this as an issue in JIRA ?
>>> 
>>> --
>>> 
>>> *Oren Shemesh*
>>> Qwilt | Work: +972-72-2221637 <+972%2072-222-1637>| Mobile:
>> +972-50-2281168
>>> <+972%2050-228-1168> | or...@qwilt.com 
>> 
>> 
> 
> 
> -- 
> 
> *Oren Shemesh*
> Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 | or...@qwilt.com
> 

cdn table and domain_name parameter?

[Jan van Doorn]

Looking at the ATS 6.2 support for TO which requires a deliveryservice to 
profile mapping, and was wondering why we still have the profile column (CCR 
Profile) in deliveryservice?

At first glance it seems to be used for the domain_name parameter only (?), and 
that could (should?) be moved to the cdn table? Not sure if this was considered 
when the cdn table was added and decided against for a good reason?

Cheers,
JvD

Re: A Traffic Ops bug, in the generation of CrConfig

[Jan van Doorn]

Hi Oren,

I believe the (client) Host header is still correct because it gets put in when 
we write the CRConfig.json to the filesystem 
(./public/CRConfig-Snapshots/) using  "Tools->SnapShot CRConfig" and that 
is done on the TM host. We always serve the CRConfig.json from the filesystem 
to TM, because it takes to long to generate it on the fly.

Rgds,
JvD



> On Dec 26, 2016, at 00:42, Oren Shemesh  wrote:
> 
> I believe that the code that generates the CrConfig has a problem when
> creating the "stat" section.
> It fills values for that section, such as "tm_host", based on the HTTP
> headers found in the request that triggered the CrConfig snapshot:
> 
> Here is a snippet from traffic_ops/app/lib/UI/Topology.pm:
> 
>$data_obj->{'stats'}->{'tm_path'}= $self->req->url->path->{'path'};
>$data_obj->{'stats'}->{'tm_host'}= $self->req->headers->host;
> 
> I find this to be a problem for two reasons:
> 
> 
>   1. This code assumes that it is being run from the context of an HTTP
>   transaction, which to me sounds like contaminating the logic of actually
>   creating the CrConfig with details from the upper layer which currently
>   uses this logic.
>   For example, if one would want to run this code from a different path
>   (Maybe in a unit test), it would be a problem.
> 
>   2. If the traffic ops is accessed through a NAT, then the host name
>   known to the HTTP client issuing the 'Snapshot CrConfig' request is not
>   necessarily the same as the traffic ops host name known to the other
>   components of the system, e.g. the traffic router that uses this
>   information.
>   This is how I found out about this problem - we are experimenting with
>   deploying TC in the cloud (Amazon EC2) and the host name visible to the
>   outside world is not the same as the host names used internally.
> 
> I believe that tm_host should come from the database (Currently there is no
> entry from the traffic ops itself, but such an entry can be created for
> this purpose), or from cdn.conf.
> 
> Shall I report this as an issue in JIRA ?
> 
> -- 
> 
> *Oren Shemesh*
> Qwilt | Work: +972-72-2221637 <+972%2072-222-1637>| Mobile: +972-50-2281168
> <+972%2050-228-1168> | or...@qwilt.com 

Re: Backup Cache Group Selection

[Jan van Doorn]

Hi Eric,

How does the backup list relate to the RFC1918-is-not-in-geo problem?

To get to a cachegroup you need to get a match in the coverage zone, I would 
think?

Rgds,
JvD

> On Dec 22, 2016, at 12:28, Eric Friedrich (efriedri)  
> wrote:
> 
> The current behavior of cache group selection works as follows
> 1) Look for a subnet match in CZF
> 2) Use MaxMind/Neustar for GeoLocation based on client IP. Choose closest 
> cache group.
> 3) Use Delivery Service Geo-Miss Lat/Long. Choose closest cache group.
> 
> 
> For deployments where IP addressing is primarily private (say RFC-1918 
> addresses), client IP Geo Location (#2) is not useful. 
> 
> 
> We are considering adding another field to the Coverage Zone File that 
> configures an ordered list of backup cache groups to try if the primary cache 
> group does not have any available caches. 
> 
> Example:
> 
> "coverageZones": {  
>  "cache-group-01": {
>“backupList”: [“cache-group-02”, “cache-group-03”],
>"network6": [
>  "1234:5678::\/64”,
>  "1234:5679::\/64"],   
>"network": [ 
>  "192.168.8.0\/24", 
>  "192.168.9.0\/24”]
> }
> 
> This configuration could also be part of the per-cache group configuration, 
> but that would give less control over which clients preferred which cache 
> groups. For example, you may have cache groups in LA, Chicago and NY. If the 
> Chicago Cache group fails, you may want some of the Chicago clients to go to 
> LA and some to go to NY. If the backup CG configuration is per-cg, we would 
> not be able to control where clients are allocated. 
> 
> Looking for opinions and comments on the above proposal, this is still in 
> idea stage. 
> 
> Thanks All!
> Eric
> 
> 

Re: [VOTE] Traffic Control RELEASE-1.8.0-RC5

[Jan van Doorn]

I'm sticking with my +1 as well. 

Rgds,
JvD

> On Dec 21, 2016, at 11:44, David Neuman  wrote:
> 
> I spot checked a few files that were missing license headers in RC4 and the 
> license was there.  Since the only thing that changes between RCs was the 
> license files, I am going to rely on my original testing and vote +1.
> 
> On Tue, Dec 20, 2016 at 11:07 AM, Dan Kirkwood  > wrote:
> One correction:  The signed source tarball and checksums are available here:
>   https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/1.8.0/RC5/ 
> 
> 

Re: [VOTE] Traffic Control RELEASE-1.8.0-RC4

[Jan van Doorn]

+1 (binding)

- built all rpms except traffic_portal (don't have the right tools for portal) 
from source
- ran all TO unit tests on clean carton install
- ran all TO integration tests on clean carton install
- installed TO (using infrastructure/docker container) w prod DB
- installed TV (using infrastructure/docker container)
- tested new offline reason feature
- tested generated CRConfig is identical
- tested ATS configs are identical for several profiles (ort report)
- installed TM / TR in containers, but did no real testing there

Docker containers need some love (see TC-77) but that shouldn't hold up the 
release, I think. 

> On Dec 19, 2016, at 09:13, David Neuman  wrote:
> 
> I just gave a +1 without really explaining the testing I did, so here is a 
> list of what I did...
> - upgrade all components (TS requires influxdb 1.0 or greater, fyi)
> - validate ORT output on and Edge and Mid against 1.7 version, validate no 
> unexpected changes
> - end to end curls and digs against TR
> - validate HTTPS functionality for TR
> - Validate ORT (1.8.0 RC4 is running in production for us)
> 
> On Sun, Dec 18, 2016 at 8:58 PM, Eric Friedrich (efriedri) 
> mailto:efrie...@cisco.com>> wrote:
> +1 (binding)
> 
> I checked the following:
> - Clean build from source
> - PGP signature, MD5/SHA1 Sums
> - DISCLAIMER
> - NOTICE
> - LICENSE
> - incubator in name
> 
> —Eric
> 
> 
> > On Dec 16, 2016, at 11:55 AM, Dan Kirkwood  > > wrote:
> >
> > Hi all.. We've decided to extend the voting until end-of-day
> > Monday,  December 19 to allow for more participation.If you have
> > voted on a previous RC,  please be sure to vote again.
> >
> > The more information we get on what has been tried,  the better,   so
> > if you see problems or not,  please share!
> >
> > Thanks..  Dan
> >
> > On Tue, Dec 13, 2016 at 11:23 AM, Dan Kirkwood  > > wrote:
> >> I have no control over the mentors :-)Yes -- since you chimed in,
> >> I'm happy to wait for your input..
> >>
> >> -Dan
> >>
> >> On Tue, Dec 13, 2016 at 10:50 AM, Leif Hedstrom  >> > wrote:
> >>>
>  On Dec 13, 2016, at 7:47 AM, Dan Kirkwood   > wrote:
> 
>  So that makes us +1 overall,  since you're the only vote :-)
> >>>
> >>>
> >>> Eeep. Where are the mentor votes? :)
> >>>
> >>> I’m traveling and in meetings this week, but if you are extending the 
> >>> vote deadling, I can try to take a look tomorrow morning.
> >>>
> >>> Cheers,
> >>>
> >>> — leif
> >>>
> 
>  On Tue, Dec 13, 2016 at 8:45 AM, David Neuman   > wrote:
> > A little late, but I am +1.
> >
> > On Mon, Dec 12, 2016 at 9:43 AM, Dan Kirkwood  > > wrote:
> >>
> >> Reminder -- 1.8.0-RC4 is open for voting until 5pm MST today.   Please
> >> vote by replying to this email.  The release is available here:
> >> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/1.8.0/RC4/
> >>  
> >> 
> >>
> >>
> >> On Thu, Dec 8, 2016 at 12:58 PM, Dan Kirkwood  >> > wrote:
> >>> Hello All,
> >>>
> >>> I've prepared another release for v1.8.0 (RC4)
> >>>
> >>> Changes since 1.7.0:
> >>>
> >>> https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-1.7.0...RELEASE-1.8.0-RC4
> >>>  
> >>> 
> >>>
> >>> This corresponds to git:
> >>> Hash: 7d8a88d0512ffe0354c2bc079ddbf2e7b67b6c3e
> >>> Tag: RELEASE-1.8.0-RC4
> >>>
> >>> Which can be verified with the following:git tag -v
> >>> RELEASE-1.8.0-RC4
> >>>
> >>> My code signing key is available here:
> >>> http://keys.gnupg.net/pks/lookup?search=0x4587A8F0&op=vindex 
> >>> 
> >>>
> >>> Make sure you refresh from a key server to get all relevant 
> >>> signatures.
> >>>
> >>> Note that we are not providing the RPM files this time.   The only
> >>> artifact provided is a source tar file which can be downloaded from
> >>> here:
> >>>
> >>>
> >>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/1.8.0/RC4/
> >>>  
> >>> 
> >>>
> >>> Let me know if you need the rpm files and I can make arrangements to
> >>> get them to you.
> >>>
> >>> Per Apache guidelines, the .tar.gz file is signed with my pgp key (see
> >>> above) and md5 and sha1 checksums are also provided there.
> >>>
> >>> We'd like to shorten the turnaround time to around 3 days,  so this
> >>> vote is open until the end of the day Monday, December 12, 2016

Re: Enhancement: Multi Delivery Services With Same Domain Name and Different Path Prefixes

[Jan van Doorn]

Jifeng: did you actually test that match-type: PATH works in Traffic Router? 
We've never used it... 

Eric: I think you are right wrt RAM vs disk; Only way to do this right now is 
to set up one of them to point to a CNAME and have 2 deliveryservices, but they 
wouldn't be able to have the edge URL 

I'm OK with doing it your way, I think, we just have to add really good docs. 
Maybe a quick howto on how to provision your example scenario?

Cheers,
JvD


> On Dec 5, 2016, at 06:49, Eric Friedrich (efriedri)  
> wrote:
> 
> To use regex_remap, I think we would want just a single line in remap.config 
> that covers all delivery services with the same host regex. This would be a 
> change from today where every DS gets one remap line. Instead, we would need 
> to combine multiple DS into one remap line with multiple lines in a 
> regex_remap_.config
> 
> remap.config
>  map http://traffic-server.sports.ipcdn.com/ 
> <http://traffic-server.sports.ipcdn.com/> http://origin.server.com/ 
> <http://origin.server.com/> @plugin=regex_remap.so 
> @pparam=regex_remap_ds_1.config
> 
> regex_remap_ds_1.config:
>  ^/vod/.* http://origin.server.com/$0 <http://origin.server.com/$0>
>  ^/live/.* http://origin.server.com/$0 <http://origin.server.com/$0>
> 
> Seems like lots of work in Traffic Ops just to restrict URLs outside of /vod 
> and /live. If desired, could that restriction instead be done as a custom 
> header rewrite rule?
> 
> 
> A second question: How will this change separate these two delivery services 
> live and vod onto different storage volumes?
>  Hosting.config is based on the remapped origin server - in both cases this 
> is origin.server.com <http://origin.server.com/><http://origin.server.com 
> <http://origin.server.com/>>, so there is no way to differentiate the two 
> delivery services into being stored on disk vs in RAM.
> 
> Thanks,
> Eric
> 
> 
> 
> 
> 
> On Dec 5, 2016, at 6:47 AM, Jifeng Yang (jifyang)  <mailto:jify...@cisco.com><mailto:jify...@cisco.com 
> <mailto:jify...@cisco.com>>> wrote:
> 
> Yes, Traffic Router source code doesn’t need change. To make PATH_PREFIX work 
> correctly, the configuration for Traffic Router needs change.
> 
> For example, if the HOST_REGEX is ".*\.sports\..*", the Traffic Router 
> configuration file “cr-config.json” will contains:
> 
>"matchlist": [{
>  "regex": ".*\\.sports\\..*",
>  "match-type": "HOST"
>}]
> 
> If a path_prefix “/path/” is added, the above item will be changed to:
> 
> "matchlist": [
>   {
> "regex": ".*\\.sports\\..*",
> "match-type": "HOST"
>   },
>   {
> "regex": "^/path/.*",
> "match-type": "PATH"
>   }
> ]
> 
> So, with this configuration Traffic Router will not redirect the request 
> whose path prefix is not “/path/”.
> 
> As the solution with PATH_REGEXP and regex_remap, there is no issue for 
> traffic router, but there is problem for ATS.
> 
> The ATS remap.config file does not support regex directly, only supports path 
> prefixes. So, the PATH_REGEXP can't be used as a remap rule in remap.config 
> directly. To work with PATH_REGEXP, a possible way is using the remap plugin. 
> But there are some issues for this, such as:
> 
> * if two delivery services with same domain name, the remap config file will 
> look like:
> 
>   map http://traffic-server.sports.ipcdn.com/ 
> http://origin.server.com/ @plugin=regex_remap.so 
> @pparam=regex_remap_ds_1.config …
>   map http://traffic-server.sports.ipcdn.com/
> http://origin.server.com/ @plugin=regex_remap.so 
> @pparam=regex_remap_ds_2.config …
> 
> ATS doesn’t work with this kind of remap config file.
> 
> * If implemented by regex_remap, a regex expression will be generated for a 
> delivery service. The user also can configure "Regex remap expression" for a 
> delivery service. If user also configure a regex expression, they may 
> interfere with each other.
> 
> Thanks,
> Jifeng
> 
> 
> On 01/12/2016, 23:23, "Jan van Doorn"  <mailto:j...@knutsel.com><mailto:j...@knutsel.com <mailto:j...@knutsel.com>>> 
> wrote:
> 
>   So for your example you would enter 2 delvieryservices with the same 
> host_regex (which would be possible because you drop the unique requirement 
> on it), different path prefixes and have different settings for each?
> 
>   I think I ge

Re: Enhancement: Multi Delivery Services With Same Domain Name and Different Path Prefixes

[Jan van Doorn]

So for your example you would enter 2 delvieryservices with the same host_regex 
(which would be possible because you drop the unique requirement on it), 
different path prefixes and have different settings for each? 

I think I get that ?

I _think_ this would work without changing Traffic Router (it just tags on the 
path in the redirect)... 3.3 in the doc says Traffic Router will be changed as 
well, but I don't see that in the PR? 

Also, going back to my initial question - did you consider implementing this 
with PATH_REGEXP and regex_remap? 

Rgds,
JvD


> On Dec 1, 2016, at 07:35, Jifeng Yang (jifyang)  wrote:
> 
> Hi JvD,
> 
> The difference between the two is: the former doesn’t serve the content under 
> the paths other than “/vod/” and “/live/”.
> 
> For example, for the request 
> “http://traffic-server.sports.ipcdn.com/path/file”, the former doesn’t serve 
> it while the latter does serve it.
> 
> Regarding the use case, this is useful if:
> 
> Under the same domain name, the contents under some paths are set by one 
> configuration and the contents under some other paths are set by another 
> configuration.
> 
> For example,
> 
> Different “Regex remap expression” can be configured for 
> “http://traffic-server.sports.ipcdn.com/vod/” and 
> “http://traffic-server.sports.ipcdn.com/live/”.
> 
> Different traffic caches can be assigned for 
> “http://traffic-server.sports.ipcdn.com/vod/” and 
> “http://traffic-server.sports.ipcdn.com/live/”.
> 
> Some different configuration items become possible because separated delivery 
> services.
> 
> Thanks,
> Jifeng
> 
> 
> On 30/11/2016, 23:23, "Jan van Doorn"  wrote:
> 
>Hi Jifeng,
> 
>I'm still confused, bear with me please. 
> 
>The Google doc example has
> 
>maphttp://traffic-server.sports.ipcdn.com/vod/ 
> http://origin.server.com/vod/
>maphttp://traffic-server.sports.ipcdn.com/live/ 
> http://origin.server.com/live/
> 
> 
>But, isn't that the same as 
> 
>maphttp://traffic-server.sports.ipcdn.com/ 
> http://origin.server.com/
> 
>?
> 
>If you want to send the /live to RAM and the /vod to disk, they can't be 
> in the same deliveryservice table entry, since all those type if things are 
> set there? 
> 
>Can you elaborate on the use case you are trying to solve?
> 
>Rgds,
>JvD
> 
>> On Nov 30, 2016, at 04:16, Jifeng Yang (jifyang)  wrote:
>> 
>> Hi,
>> 
>> Different delivery services can be configured with different domain names 
>> now. In some cases, different delivery services with same domain name and 
>> different path prefixes are needed. These delivery services can have 
>> different configurations.
>> 
>> The problem and a solution are described in the document 
>> https://docs.google.com/document/d/19-TZ6ODla_vdiYqZajbpRiOpLvpbJxil1SvIm44zb-0/edit?usp=sharing.
>> 
>> The issue and PR for this:
>> Issue: https://issues.apache.org/jira/browse/TC-55?jql=project%20%3D%20TC
>> PR: https://github.com/apache/incubator-trafficcontrol/pull/108
>> 
>> Thanks,
>> Jifeng
>> 
> 
> 
> 

Re: Proposed change to Deliverservice API

[Jan van Doorn]

> ... requirements on the list for 2.1.

For 3.0, you mean? No API breaking changes on a minor rev, I thought? 


> On Nov 30, 2016, at 08:35, Dewayne Richardson  wrote:
> 
> I think once we get 2.0 in place we can start discussing the direction
> forward.  I agree with the need for "natural keys" because I also was
> working on an integration test tool and found the need to "lookup" the "id"
> too cumbersome.  Once 2.0 is in place, we should put these on the
> requirements on the list for 2.1.
> 
> On Wed, Nov 30, 2016 at 8:28 AM, David Neuman 
> wrote:
> 
>> Fair enough.  I agree we should use natural keys.  To me the ID thing is
>> something internal to the DB that a client should not have to worry about.
>> I can update the Traffic Ops client to support using IDs and keep the API
>> as-is.
>> 
>> On Wed, Nov 30, 2016 at 8:26 AM, Jan van Doorn  wrote:
>> 
>>> Agree with Jeremey. And we can't just slip in a change to 2.0 that does
>>> part of this.
>>> 
>>> I'm -1 on neuman's change, at least for 2.0.
>>> 
>>> Cheers,
>>> JvD
>>> 
>>> 
>>> 
>>>> On Nov 30, 2016, at 08:23, Jeremy Mitchell 
>>> wrote:
>>>> 
>>>> Let's look at an example of using ID's versus names for POSTS (creates)
>>>> 
>>>> Here is the region table. A region belongs to a division.
>>>> 
>>>> mysql> desc region;
>>>> +--+-+--+-+-
>>> --+-+
>>>> | Field| Type| Null | Key | Default   | Extra
>>>>   |
>>>> +--+-+--+-+-
>>> --+-+
>>>> | id   | int(11) | NO   | PRI | NULL  |
>>>> auto_increment  |
>>>> | name | varchar(45) | NO   | UNI | NULL  |
>>>>   |
>>>> | division | int(11) | NO   | MUL | NULL  |
>>>>   |
>>>> | last_updated | timestamp   | YES  | | CURRENT_TIMESTAMP | on
>> update
>>>> CURRENT_TIMESTAMP |
>>>> +--+-+--+-+-
>>> --+-+
>>>> 4 rows in set (0.01 sec)
>>>> 
>>>> Currently, if you want to create a region, you have to provide a
>> division
>>>> "id" (as opposed to a division name)
>>>> 
>>>> POST /api/version/regions
>>>> 
>>>> {
>>>> name: "myregion",
>>>> division: 2
>>>> }
>>>> 
>>>> This allows the json to go directly into the table with one insert
>> query.
>>>> 
>>>> if we use a division name instead, like this
>>>> 
>>>> {
>>>> name: "myregion",
>>>> division: "central"
>>>> }
>>>> 
>>>> then 2 queries have to happen on the server side. 1 query to fetch the
>>>> division id for "central" and then the insert query to create the
>> region.
>>>> 
>>>> To do this right, imo, the ID for the central division WOULD be
>> "central"
>>>> instead of the number 2 - and this is why natural keys make a lot of
>>> sense.
>>>> So rather than changing the api to accept cdn name, profile name, and
>>> type
>>>> name, continue to send thru the ids and we need to make the effort to
>> get
>>>> to natural keys.
>>>> 
>>>> my 2 cents
>>>> 
>>>> On Wed, Nov 30, 2016 at 7:53 AM, Dave Neuman 
>> wrote:
>>>> 
>>>>> Thanks Derek, it's actually already done, but then it dawned on me
>> that
>>> it
>>>>> might break others, which is why I posted.
>>>>> 
>>>>> On Wed, Nov 30, 2016 at 7:51 AM, Gelinas, Derek <
>>> derek_geli...@comcast.com
>>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> I've already got a bit of code you can use for just that if you like.
>>>>>> Doing the same for the config file lookups.
>>>>>> 
>>>>>>> On Nov 30, 2016, at 9:50 AM, Dave Neuman  wrote:
>>>>>>> 
>>>>>>> Hey all,
>>>>>>> I have been working on creating some integration tests for the
>> Traffic
>>>>>> Ops
>>>>>>> client in the psql-rebase branch and fixing any bugs in Traffic Ops
>>>>> along
>>>>>>> the way.
>>>>>>> One thing I would like to change is to have the
>> DeliveryService.create
>>>>>> and
>>>>>>> Deliveryservice.update in the Traffic Ops API accept cdn name,
>> profile
>>>>>>> name, and type name instead of cdn ID, profile ID, and type ID.  I
>>>>>> usually
>>>>>>> don't like to have clients worry about IDs, I think it should be
>>>>> handled
>>>>>> on
>>>>>>> the server side.  I don't know who all is using the Deliveryservice
>>>>>> create
>>>>>>> and update APIs, if anyone, so I thought I would make the suggestion
>>>>> here
>>>>>>> and see if anyone is opposed?
>>>>>>> This change would be in a 2.x version of Traffic Ops.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Dave
>>>>>> 
>>>>> 
>>> 
>>> 
>> 

Re: Proposed change to Deliverservice API

[Jan van Doorn]

Agree with Jeremey. And we can't just slip in a change to 2.0 that does part of 
this. 

I'm -1 on neuman's change, at least for 2.0. 

Cheers,
JvD



> On Nov 30, 2016, at 08:23, Jeremy Mitchell  wrote:
> 
> Let's look at an example of using ID's versus names for POSTS (creates)
> 
> Here is the region table. A region belongs to a division.
> 
> mysql> desc region;
> +--+-+--+-+---+-+
> | Field| Type| Null | Key | Default   | Extra
>|
> +--+-+--+-+---+-+
> | id   | int(11) | NO   | PRI | NULL  |
> auto_increment  |
> | name | varchar(45) | NO   | UNI | NULL  |
>|
> | division | int(11) | NO   | MUL | NULL  |
>|
> | last_updated | timestamp   | YES  | | CURRENT_TIMESTAMP | on update
> CURRENT_TIMESTAMP |
> +--+-+--+-+---+-+
> 4 rows in set (0.01 sec)
> 
> Currently, if you want to create a region, you have to provide a division
> "id" (as opposed to a division name)
> 
> POST /api/version/regions
> 
> {
> name: "myregion",
> division: 2
> }
> 
> This allows the json to go directly into the table with one insert query.
> 
> if we use a division name instead, like this
> 
> {
> name: "myregion",
> division: "central"
> }
> 
> then 2 queries have to happen on the server side. 1 query to fetch the
> division id for "central" and then the insert query to create the region.
> 
> To do this right, imo, the ID for the central division WOULD be "central"
> instead of the number 2 - and this is why natural keys make a lot of sense.
> So rather than changing the api to accept cdn name, profile name, and type
> name, continue to send thru the ids and we need to make the effort to get
> to natural keys.
> 
> my 2 cents
> 
> On Wed, Nov 30, 2016 at 7:53 AM, Dave Neuman  wrote:
> 
>> Thanks Derek, it's actually already done, but then it dawned on me that it
>> might break others, which is why I posted.
>> 
>> On Wed, Nov 30, 2016 at 7:51 AM, Gelinas, Derek >> 
>> wrote:
>> 
>>> I've already got a bit of code you can use for just that if you like.
>>> Doing the same for the config file lookups.
>>> 
 On Nov 30, 2016, at 9:50 AM, Dave Neuman  wrote:
 
 Hey all,
 I have been working on creating some integration tests for the Traffic
>>> Ops
 client in the psql-rebase branch and fixing any bugs in Traffic Ops
>> along
 the way.
 One thing I would like to change is to have the DeliveryService.create
>>> and
 Deliveryservice.update in the Traffic Ops API accept cdn name, profile
 name, and type name instead of cdn ID, profile ID, and type ID.  I
>>> usually
 don't like to have clients worry about IDs, I think it should be
>> handled
>>> on
 the server side.  I don't know who all is using the Deliveryservice
>>> create
 and update APIs, if anyone, so I thought I would make the suggestion
>> here
 and see if anyone is opposed?
 This change would be in a 2.x version of Traffic Ops.
 
 Thanks,
 Dave
>>> 
>> 

Re: Enhancement: Multi Delivery Services With Same Domain Name and Different Path Prefixes

[Jan van Doorn]

Hi Jifeng,

I'm still confused, bear with me please. 

The Google doc example has

map http://traffic-server.sports.ipcdn.com/vod/ 
http://origin.server.com/vod/
map http://traffic-server.sports.ipcdn.com/live/ 
http://origin.server.com/live/


But, isn't that the same as 

map http://traffic-server.sports.ipcdn.com/ http://origin.server.com/

?

If you want to send the /live to RAM and the /vod to disk, they can't be in the 
same deliveryservice table entry, since all those type if things are set there? 

Can you elaborate on the use case you are trying to solve?

Rgds,
JvD

> On Nov 30, 2016, at 04:16, Jifeng Yang (jifyang)  wrote:
> 
> Hi,
> 
> Different delivery services can be configured with different domain names 
> now. In some cases, different delivery services with same domain name and 
> different path prefixes are needed. These delivery services can have 
> different configurations.
> 
> The problem and a solution are described in the document 
> https://docs.google.com/document/d/19-TZ6ODla_vdiYqZajbpRiOpLvpbJxil1SvIm44zb-0/edit?usp=sharing.
> 
> The issue and PR for this:
> Issue: https://issues.apache.org/jira/browse/TC-55?jql=project%20%3D%20TC
> PR: https://github.com/apache/incubator-trafficcontrol/pull/108
> 
> Thanks,
> Jifeng
> 

Getting ready for the git move of Traffic Control

[Jan van Doorn]

Phil is working on a write up on how the new process works when we are in
Apache. There will be some changes, as the github.com/apache version is a
mirror of the one in apache.

For now, please go through the issues and PRs and close / merge everything
that you can, and hold off on opening new PRs to make the move as easy as
possible.

We hope to have this move done in the next 10 days or so, look for more
emails here.

Rgds,
JvD

Moving in to the Apache Incubator!

[Jan van Doorn]

As most of you know, we have been working in the back ground to move Traffic 
Control in to the Apache Incubator. All legal i's and t's are dotted and 
crossed now, so we're going to start the move. 

First, email lists. In Apache it's only official if it happened on the email 
list. So it's OK to discuss stuff in person or via phone/chat/video 
conferencing, and even to get consensus there, but it's not decided until it 
was discussed on the list. If it is a major feature or change, we probably want 
to do a [VOTE] thread.

We have these lists for anyone to join:

iss...@trafficcontrol.incubator.apache.org
us...@trafficcontrol.incubator.apache.org
dev@trafficcontrol.incubator.apache.org
comm...@trafficcontrol.incubator.apache.org


You can subscribe to those by sending an email to 
-subscr...@trafficcontrol.incubator.apache.org 
 and following the 
instructions in the email response.

If you want to see everything going on, just send one email to these addresses:

issues-subscr...@trafficcontrol.incubator.apache.org
users-subscr...@trafficcontrol.incubator.apache.org
dev-subscr...@trafficcontrol.incubator.apache.org
commits-subscr...@trafficcontrol.incubator.apache.org

and reply to the responses. 

For more info on lists and subscribing, please see: 
http://apache.org/foundation/mailinglists.html 


I propose that in the next few months we keep an eye on the Google Groups as 
well, and if questions pop up there, we ask the OP to repost on the apache 
list. The website will be updated soon to reflect the new Apache status and 
lists. 

Soon we will also move the source code into the Apache git repo, and the issues 
to an Apache JIRA instance; more on the new processes that go along with that 
as we do these moves.

Best Regards,
JvD