Re: [RESULT][VOTE] Release Apache Traffic Control 8.0.1-RC1

[Eric Friedrich]

Jan is a binding vote too :-)

On Tue, Apr 2, 2024 at 10:38 AM R S  wrote:

> Hello All,
> Thanks to all who voted!
>
> The release has PASSED with the following PMC votes:
>
> +1 Zach Hoffman (binding)
> +1 Eric Friedrich (binding)
> +1 Jeremy Mitchell (binding)
> +1 Jan van Doorn
>
> Voting results can be found here:
> https://lists.apache.org/thread/h0zmckc3smvdz4sn7nt2f2ypn348g0gr
>
> I will proceed to publish the release and send ANNOUNCE.
>
> On behalf of Apache Traffic Control, thank you!
>
> Regards,
> Rima Shah
> rs...@apache.org
>

Re: [VOTE] Release Apache Traffic Control 8.0.1-RC1

[Eric Friedrich]

+1

On Fri, Mar 29, 2024 at 2:31 PM Zach Hoffman  wrote:

> +1
>
> -Zach
>
> On Wed, Mar 27, 2024 at 11:18 AM R S  wrote:
>
> > Hello All,
> >
> > I've prepared a release for 8.0.1-RC1. There *isn't any* changes since
> > RELEASE-8.0.1-RC0:
> >
> > The purpose of this new candidate is to get enough votes to either cut a
> > release or make a change. Please be kind and vote.
> > Also, *note*: These changes have been verified and are running internally
> > at Comcast.
> >
> > https://github.com/apache/trafficcontrol/compare/v8.0.1-rc0...v8.0.1-rc1
> >
> >
> >
> https://github.com/apache/trafficcontrol/blob/RELEASE-8.0.1-RC1/CHANGELOG.md
> >
> > The artifacts are available for download at:
> >
> > https://dist.apache.org/repos/dist/dev/trafficcontrol/8.0.1/RC1/
> >
> > BLAKE2 checksum:
> >
> >
> 448227ebe681561613ce65db4e20f62ac7f9d043200e2a94915e36facdc9f01f549a97dccaafedfe420b61b896a690850c5965b164ceea1b02d372fcb9250065
> >
> > SHA512 checksum:
> >
> >
> c438f337725caca1b1a1ddb25d1a2ea04b96a9a4d786acd0b24fb3e7455f95e069c9af121ebdcbde2e63168330ba8e2b999a912637de97403acf52e6e06ebdb1
> >
> > This corresponds to git refs:
> >
> > Hash:ad667bb9178d96adfdcdff10b51fe6b3a4ce2e1b
> > Tag: RELEASE-8.0.1-RC1
> >
> > Which can be verified with the following command:
> >
> > $ git tag -v RELEASE-8.0.1-RC1
> >
> > All code signing keys are available here:
> >
> > https://dist.apache.org/repos/dist/release/trafficcontrol/KEYS
> >
> > Make sure you refresh from a key server to get all relevant signatures.
> >
> > The vote is open until 22:00 UTC on *Monday, April 1st* and passes if a
> > majority of at least 3 +1 PMC votes are cast.
> > [ ] +1 approve
> > [ ] +0 no opinion
> > [ ] -1 disapprove (and reason why)
> >
> > Thanks!
> > Rima Shah
> >
> > rs...@apache.org
> >
>

Re: [VOTE] Release Apache Traffic Control 8.0.0-RC5

[Eric Friedrich]

+1

Verified hashes and signature.
RPMs built

--Eric

On Thu, Jan 18, 2024 at 1:33 PM Jeremy Mitchell 
wrote:

> +1
>
> On Wed, Jan 17, 2024 at 11:09 AM Rima Shah  wrote:
>
> > Corrected top link.
> >
> >  I've prepared a release for 8.0.0-RC5.
> >
> > Changes since RELEASE-8.0.0-RC4:
> > https://github.com/apache/trafficcontrol/compare/v8.0.0-rc4...v8.0.0-rc5
> >
> >
> >
> https://github.com/apache/trafficcontrol/blob/RELEASE-8.0.0-RC5/CHANGELOG.md
> >
> > Regards,
> > Rima Shah
> >
> > On 2024/01/17 17:48:56 R S wrote:
> > > Hello All,
> > >
> > > I've prepared a release for 8.0.0-RC5. Changes since RELEASE-8.0.0-RC4:
> > >
> https://github.com/apache/trafficcontrol/compare/v8.0.0-rc4...v8.0.0-rc5
> > >  <
> >
> https://github.com/apache/trafficcontrol/blob/RELEASE-8.0.0-RC4/CHANGELOG.md
> > >
> >
> https://github.com/apache/trafficcontrol/blob/RELEASE-8.0.0-RC5/CHANGELOG.md
> > >
> > > The artifacts are available for download at:
> > > https://dist.apache.org/repos/dist/dev/trafficcontrol/8.0.0/RC5/
> > >
> > > BLAKE2 checksum:
> > >
> > >
> >
> a2d12fa11050e20b7ab4c864a058de2781af4a1449e62c5d3cc169fcd6c8bb88dd9f065a2a44d93ed6f64b00802aa0c387f997c7dea0e7331e4b8397022caa68
> > >
> > > SHA512 checksum:
> > >
> > >
> >
> cb826938cdf23e053326f5b0de0cea36de8d3825c5803392a1ba1cb8406a753c6d523e2662cc82ac33181a7f295ca7b74c8e72ebe848e64ae4ad3d5a67872667
> > >
> > > This corresponds to git refs:
> > >
> > >   Hash:0c515199dbeb3233b32376268e4ee32ad18ebd8b
> > >   Tag: RELEASE-8.0.0-RC5
> > >
> > > Which can be verified with the following command:
> > >
> > >   $ git tag -v RELEASE-8.0.0-RC5
> > >
> > > All code signing keys are available here:
> > >
> > >   https://dist.apache.org/repos/dist/release/trafficcontrol/KEYS
> > >
> > > Make sure you refresh from a key server to get all relevant signatures.
> > >
> > > The vote is open until 22:00 UTC on Monday, Jan 22nd 2024 and passes
> > > if a majority of at least 3 +1 PMC votes are cast.
> > >   [ ] +1 approve
> > >   [ ] +0 no opinion
> > >   [ ] -1 disapprove (and reason why)
> > >
> > > Thanks!
> > > Rima Shah
> > >
> >
>

Re: Comcast ATC Open Source Withdrawal Announcement

[Eric Friedrich]

Jeremy-

Thanks to you and the rest of the team.

I personally appreciate everything Comcast and the Traffic Control team
have contributed over the past decade or so. Let's hope this pause is short
lived and Traffic Control continues on into the future.

As always, if anyone is interested in helping fill this void by becoming a
contributor or increasing their contributions, now is a great time to get
started. There's plenty of good-first-issues in Github Issues.

Hopefully some of the Comcast folks will continue to lurk in Slack/on
mailing lists.

--Eric

On Thu, Nov 30, 2023 at 4:25 PM Jeremy Mitchell 
wrote:

> A message from Comcast:
>
> Since inception, Comcast has been the driving force on the Apache Traffic
> Server OSS project. As the CDN landscape is evolving, it has become clear
> that Comcast must adapt its CDN business as well. This evaluation includes
> weighing our investments against the return on those investments. An area
> of focus has been our participation in different open source communities,
> and at this point in time the ATC project is not balanced enough for
> Comcast to continue to contribute. We do care dearly about the project and
> feel that it is best if Comcast withdraws, at least temporarily, to allow
> other contributors to take a larger role and encouraging additional
> participation from new sources. If the project begins to thrive, Comcast is
> absolutely willing to rejoin the community.  Please know that this was a
> difficult and emotional decision, as Comcast and ATC have been very much
> intertwined. This is purely a business decision and is in no way meant to
> diminish the incredible accomplishments this community have made and how
> much delivery it has empowered.  We would like to thank the community and
> will certainly continue to monitor its progress. The official release of
> ATC 8.0 will mark Comcast's final contribution.
>

Re: Next Step for Varnish Support

[Eric Friedrich]

Definitely a tough question and I see arguments both ways.

One might consider it as
1) Will users mix and match between ATS and Varnish in a single Delivery
service?
2) If yes, is it okay to for them to have to maintain two sets of header
rewrite rules?

In a perfect world, we'd only want one copy of any CDN policies, less to
maintain and eliminates problems with them getting out of sync.
However, in this prototype implementation, I think its as good a starting
point as any to stick with the native formats for both.

Would love to hear others thoughts as well

--Eric


On Wed, Oct 11, 2023 at 10:42 AM Abdulrahman Elawady <
abdoelawady...@gmail.com> wrote:

> I had some questions about how we should approach supporting header
> rewrites in Varnish.
>
> As you know, using VCL we can directly rewrite headers by accessing the
> request and response objects. The concern is how should the user of Traffic
> Control specify which headers to rewrite or based on what conditions.
> Normally with Traffic Server we can use something like the plugin.config
> parameter to specify the header_rewrite plugin with the path to files
> containing rewrite directives based on ATS supported syntax.
>
> Should we expect the user to also provide ATS syntax?
>
> While it's doable, it is not that easy to parse this configuration and
> maintaining it will be non-trivial.
>
> We can also expect from the user to input the directives in a Varnish way
> using VCL. Since it is just like any VCL we will be basically allowing the
> user to add their own blocks of code whether it's a header rewrite or just
> a custom logic. I think that would be easy to maintain and will offer more
> customization but will require more parameters.
>
> What do you think?
>
> Abdulrahman
>
> On Sat, Oct 7, 2023 at 3:44 PM Eric Friedrich  wrote:
>
> > Great suggestions Abdulrahman!
> >
> > I think in terms of base functionality, that header rewrites are very
> > frequently used, so this might be a good next step. Lots of times extra
> > response headers are needed, especially if CORS is required.
> >
> > Regex remap is also useful because CDNs frequently are asked to modify
> the
> > URL, and Varnish has great support for this with regsub() already.
> >
> > After that, paying off some tech debt on the testing might make sense?
> > Would be very cool if we could find a way to integrate varnishtest
> somehow
> > as its purpose built for testing VCL functionality
> >
> > Best,
> > Eric
> >
> > On Thu, Oct 5, 2023 at 6:56 AM Abdulrahman Elawady 
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > Now that Varnish is somewhat in a good state with a lot of the basics
> > done,
> > > I wanted to discuss with you the next steps for making Varnish a viable
> > > option as a cache server for Traffic Control.
> > >
> > > There are multiple options for what to do next and what to focus on
> and I
> > > am interested in your opinions on that.
> > >
> > > Some of these options are:
> > >
> > > Varnish configuration:
> > > - Cache server storage configuration. It might not be identical to ATS
> > due
> > > to different cache storage models between Varnish and ATS.
> > > - ATS plugins like: header rewrite, regex remap. Some of the plugins
> are
> > > straightforward and supported directly by Varnish while others will
> need
> > > some work.
> > > - DS improvements (dscp, qstring, go_direct...) and backends health
> > checks.
> > > - Error pages to match ATS.
> > >
> > > Functional Tests:
> > > - Some integration tests are needed for Varnish to ensure the
> > functionality
> > > of VCL files generated are correct and make it easier for future
> > > development to be integrated. It is somewhat different than ORT tests
> > > available as it tests the functionality of the cache server itself
> rather
> > > than t3c or some baseline configuration files.
> > >
> > > Varnish development:
> > > - It would be great to make Varnish development easier for newcomers
> and
> > > people interested. So, some of the previous points could be simplified
> > into
> > > good first issues.
> > > - Varnish configuration package "varnishcfg" could be improved based on
> > how
> > > easy it is to add features to it. If it turned out to be somewhat
> > difficult
> > > we can see other options like go template language or maybe document it
> > > more. Of course I can't be the judge of that since I wrote a lot of it
> so
> > > hopefully the good first issues might give more insight to that.
> > >
> > > Traffic Monitor:
> > > - After #7805 DS stats will be reported except bytes in and out because
> > > Varnish does not report this data. So, it might need a solution outside
> > > Varnish. It's probably not a priority right now but just worth
> > mentioning.
> > >
> > > These are just some ideas of what to do next, feel free to suggest
> other
> > > ideas. I am interested in your opinions on what to prioritise and focus
> > on.
> > >
> > > Regards,
> > > Abdulrahman
> > >
> >
>

Re: Next Step for Varnish Support

[Eric Friedrich]

Great suggestions Abdulrahman!

I think in terms of base functionality, that header rewrites are very
frequently used, so this might be a good next step. Lots of times extra
response headers are needed, especially if CORS is required.

Regex remap is also useful because CDNs frequently are asked to modify the
URL, and Varnish has great support for this with regsub() already.

After that, paying off some tech debt on the testing might make sense?
Would be very cool if we could find a way to integrate varnishtest somehow
as its purpose built for testing VCL functionality

Best,
Eric

On Thu, Oct 5, 2023 at 6:56 AM Abdulrahman Elawady 
wrote:

> Hi Everyone,
>
> Now that Varnish is somewhat in a good state with a lot of the basics done,
> I wanted to discuss with you the next steps for making Varnish a viable
> option as a cache server for Traffic Control.
>
> There are multiple options for what to do next and what to focus on and I
> am interested in your opinions on that.
>
> Some of these options are:
>
> Varnish configuration:
> - Cache server storage configuration. It might not be identical to ATS due
> to different cache storage models between Varnish and ATS.
> - ATS plugins like: header rewrite, regex remap. Some of the plugins are
> straightforward and supported directly by Varnish while others will need
> some work.
> - DS improvements (dscp, qstring, go_direct...) and backends health checks.
> - Error pages to match ATS.
>
> Functional Tests:
> - Some integration tests are needed for Varnish to ensure the functionality
> of VCL files generated are correct and make it easier for future
> development to be integrated. It is somewhat different than ORT tests
> available as it tests the functionality of the cache server itself rather
> than t3c or some baseline configuration files.
>
> Varnish development:
> - It would be great to make Varnish development easier for newcomers and
> people interested. So, some of the previous points could be simplified into
> good first issues.
> - Varnish configuration package "varnishcfg" could be improved based on how
> easy it is to add features to it. If it turned out to be somewhat difficult
> we can see other options like go template language or maybe document it
> more. Of course I can't be the judge of that since I wrote a lot of it so
> hopefully the good first issues might give more insight to that.
>
> Traffic Monitor:
> - After #7805 DS stats will be reported except bytes in and out because
> Varnish does not report this data. So, it might need a solution outside
> Varnish. It's probably not a priority right now but just worth mentioning.
>
> These are just some ideas of what to do next, feel free to suggest other
> ideas. I am interested in your opinions on what to prioritise and focus on.
>
> Regards,
> Abdulrahman
>

Re: [VOTE] Release Apache Traffic Control 8.0.0-RC0

[Eric Friedrich]

Rima-
  If you haven't done so yet, can you please push your key to a pgp
keyserver too?

Thanks,
Eric


On Wed, Sep 20, 2023 at 5:42 PM R S  wrote:

> All,
>
> My first sentence was missing the Release candidate version. It should
> read as follows:
>
> I've prepared a release for 8.0.0-RC0. Changes since RELEASE-7.0.1:
>
> Thanks,
>
> Rima Shah
>
>
> On Wed, Sep 20, 2023 at 3:38 PM R S  wrote:
>
> > Hello All,
> >
> > I've prepared a release for {{ATC Version}}-RC{{RC Number}}. Changes
> since RELEASE-{{Previous ATC Version}}:
> > https://github.com/apache/trafficcontrol/compare/v7.0.1...v8.0.0-rc0
> >
> >
> https://github.com/apache/trafficcontrol/blob/RELEASE-8.0.0-RC0/CHANGELOG.md
> >
> > The artifacts are available for download at:
> > https://dist.apache.org/repos/dist/dev/trafficcontrol/8.0.0/RC0/
> >
> > BLAKE2 checksum:
> >
> >
> d9868b1136508591317c5ee1bf78d08eed6382d0d01dfd6a92f63f4c2bbce60842bb1dfcabecd2d17bc0065d94779152818edc09e57e1b3f11e56a42b1889f3b
> >
> > SHA512 checksum:
> >
> 5f2292a369210f3e7127686248bd475fb440524d7d4be2fd54a4246c3214ca05ab702fdab139c50eadd80b448f7f418a00cc32f6f9b0a6e508f61649829e02c4
> >
> > This corresponds to git refs:
> >
> >   Hash: 9cfe0187681125e19193748f160b849a46bb6cd9
> >   Tag: RELEASE-8.0.0-RC0
> >
> > Which can be verified with the following command:
> >
> >   $ git tag -v RELEASE-8.0.0-RC0
> >
> > All code signing keys are available here:
> >
> >   https://dist.apache.org/repos/dist/release/trafficcontrol/KEYS
> >
> > Make sure you refresh from a key server to get all relevant signatures.
> >
> > The vote is open until 22:00 UTC on Monday, September 25 and passes if a
> majority of at least 3 +1 PMC votes are cast.
> >   [ ] +1 approve
> >   [ ] +0 no opinion
> >   [ ] -1 disapprove (and reason why)
> >
> > Thanks!
> > Rima Shah
> >
> > rs...@apache.org
> >
> >
>

Re: GSOC 2023: Varnish Cache support in Apache Traffic Control - Interested in Contributing

[Eric Friedrich]

Hi-
Thanks for introducing yourself and sharing some of your experiences! This
has potential to be a great GSoC project.

Here's how I'd suggest getting started

1) Download the latest Traffic Control release and follow the CDN-in-a-box
instructions to get it setup [1]. This will use Apache Traffic Server as
the cache.
2) Look through docs and code to understand how the `t3c` component works.
This generates ATS config files by reading the traffic ops API
3) Download and install varnish-cache. Start writing some basic VCL
4) Draft a proposal for how t3c can be modified (or a new t3c-varnish can
be written) to create VCL instead of ATS config files. Glad to review
proposals and provide feedback

If you go forward putting together a proposal, be aware that project
selection criteria rewards demonstrating your ability to accomplish the
task and the level of detail in your proposal

Good luck!

--Eric

[1]
https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/ciab.html

On Wed, Mar 8, 2023 at 8:27 PM Chenhao Liu  wrote:

> Dear mentors,
>
>
>
> I hope you are well. Thank you so much for reading my email.
>
>
>
> My name is Chenhao Liu, a student majoring in computer science at the
> University of California San Diego. I really want to work on your project
> and now I am getting familiar with the project and the knowledge it
> requires.
>
>
>
> I am an experienced Go developer building several projects using Go. One
> such project was a cloud-based file storage service called SurfStore, which
> is based on Dropbox and the RAFT protocol. This application allows users to
> sync files to and from the cloud and utilizes distributed consensus to
> ensure the safety and availability of data.
>
>
>
> In addition to my experience with Go, I have an amount of knowledge in
> HTTP and caching as well. With my background in programming and experience,
> I am confident in my ability to learn quickly and adapt to new technologies
> and frameworks in GSoC.
>
>
>
> Should I share a draft proposal with you or do something to get feedback?
>
> If you have any feedback or suggestions, please do not hesitate to contact
> me.
>
>
>
> Thank you for considering my application, and I hope I could be one of the
> contributors to the Apache Software foundation.
>
>
>
> Best regards,
>
> Chenhao Liu
>

Re: Slack Invite request

[Eric Friedrich]

Invite sent

On Sun, Mar 5, 2023 at 3:50 PM Kunal Kundu  wrote:

> Will someone please invite me to the ASF Slack so I can join the
> #traffic-control channel?
>

Re: Slack Invite request

[Eric Friedrich]

Thanks for your interest! We're always glad to see new contributors join
the community.

Best place is to start is by getting the CDN-in-a-box up and running:
https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/ciab.html

We're glad to work with you on PRs for docs, new features, etc..

If you'd like some inspiration, the good-first-issues Github label has some
great starting places
https://github.com/apache/trafficcontrol/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22


--Eric

On Sat, Feb 25, 2023 at 3:48 PM Dhadve Yash  wrote:

> Will someone please invite me to the ASF Slack so I can join the
> #traffic-control channel?
>
>
> I found this project and i am interested to contribute to it.
> i am new to open source and need help navigating through this project.

Re: Slack Invite request

[Eric Friedrich]

Just sent an invite

On Mon, Feb 27, 2023 at 11:01 PM Utkarsh Chourasia <
utkarshchourasia...@gmail.com> wrote:

> Will someone please invite me to the ASF Slack so I can join the
> #traffic-control channel?
>

Re: Slack Invite request

[Eric Friedrich]

Hi Dhadve-
  You have an account in the ASF Slack and can join the channel

--Eric

On Mon, Feb 27, 2023 at 1:50 PM Dhadve Yash  wrote:

> Will someone please invite me to the ASF Slack so I can join the
> #traffic-control channel?

Re: Slack Invite request

[Eric Friedrich]

All Set

On Wed, Jan 4, 2023 at 8:25 AM Gwendal Simon 
wrote:

> Will someone please invite me to the ASF Slack so I can join the
> #traffic-control channel?
>
>
>
> This message and any attachment are confidential and may be privileged or
> otherwise protected from disclosure. If you are not the intended recipient,
> please notify the sender immediately and delete this message and any
> attachment from your system. Do not copy them or disclose the contents to
> any other person.
>

Re: t3c apply automation

[Eric Friedrich]

Abhishek-
  Correct, there is no API for invoking t3c. Caches typically are hosted on
the public Internet, so we try to minimize the number of publicly listening
services for security reasons.

Other than cron, you could also consider a configuration management tool
like Ansible or Puppet to remotely trigger t3c.

--Eric


On Tue, Sep 20, 2022 at 9:24 AM Abhishek Suresh Kumar <
asureshku...@embedur.com> wrote:

> Hello,
>
>
>
> This is a query on executing t3c-apply using an API. As
> per the ATC documentation, it is inferred that t3c commands has to be
> triggered using a CRON job or any automation tool. And there is no API for
> invoking the t3c command.
>
>
>
> Is there an existing way to automatically trigger t3c command on a profile
> update of an edge server.
>
>
>
>
>
> Regards,
>
> Abhishek S
>
>
>

Re: Reg., a query on cache config t3c-apply.

[Eric Friedrich]

Jeremey-
  Is there someone in particular on Slack who isn't on the mailing list
that we could ask to join? Async communications are often easier for those
in different timezones.

On Tue, Sep 13, 2022 at 10:09 AM Jeremy Mitchell 
wrote:

> you should probably ask this in our traffic control slack  channels.
> would you like an invite?
>
> jeremy
>
> On Tue, Sep 13, 2022 at 6:03 AM Abhishek Suresh Kumar <
> asureshku...@embedur.com> wrote:
>
> > Hey, any update on this thread?
> >
> > Regards,
> > Abhishek S
> > From: Abhishek Suresh Kumar
> > Sent: 06 September 2022 12:14
> > To: dev@trafficcontrol.apache.org
> > Subject: Reg., a query on cache config t3c-apply.
> >
> > Hello,
> >
> > I’ve installed the traffic server, ops, portal and
> > cache-config RPM on a bare-metal Linux (Centos 7) and running them as
> > system service. I’ve an edge server linked with a profile. Added a new
> > parameter to the profile and performing a t3c-apply on the server. Facing
> > the below issue while executing the command.
> >
> > ERROR: t3c-request.go:68: 2022-09-05T12:03:03.561269Z: writing data:
> > getting config data: getting cdn 'kabletowncdn': getting cdn ssl keys:
> > getting uncached: getting cdn ssl keys from Traffic Ops '': error
> > requesting Traffic Ops: path '
> > https://172.24.109.216:443/api/4.0/cdns/name/kabletowncdn/sslkeys' gave
> > HTTP error 500 Internal Server Error - error-level alerts: Internal
> Server
> > Error
> >
> > Attached the log output of t3c-apply command. PFA.
> > Any help on these would be appreciable. Thanks.
> >
> > Regards,
> > Abhishek S
> >
> >
> >
>

Re: GSoC Backup Mentor

[Eric Friedrich]

Thanks Rob, I'll send you details, there's not much

On Tue, May 10, 2022, 3:37 PM Robert O Butts  wrote:

> Sure, I can.
>
> @friede I haven't done it before, just let me know what needs done.
>
>
> On Tue, May 10, 2022 at 12:14 PM Eric Friedrich  wrote:
>
> > Anyone willing to be a backup mentor for a potential Google Summer of
> Code
> > project?
> >
> > It would be primarily for enhancing t3c to generate varnish
> configuration.
> >
> > ASF requires 2 mentors in order to accept a project.
> >
> > Thanks,
> > Eric
> >
>

GSoC Backup Mentor

[Eric Friedrich]

Anyone willing to be a backup mentor for a potential Google Summer of Code
project?

It would be primarily for enhancing t3c to generate varnish configuration.

ASF requires 2 mentors in order to accept a project.

Thanks,
Eric

ATC 6.x + Rocky

[Eric Friedrich]

Hey-
 Thoughts on backporting the Rocky Linux fixes to the 6.x branch?
Currently CIAB is broken on that branch and it is our current "stable"
release

--Eric

Re: 2022-02-15 TC Working Group Agenda and Meeting Notes

[Eric Friedrich]

With 4.x remaining unstable in our "Q2" release, does that remove our
reasoning for cutting a major release?

What will happen for the Q3 release when the 4.x TO API is ready? Will
we want to cut another 8.x major release?


I am all for making progress to remove deprecated codes, but for many
users (myself included), major releases present a large investment in
effort to adopt. I think the 1, maybe 2 major release per year cadence
is reasonable. Three or more major releases in a year would be rather
difficult to accommodate.

--Eric


On Tue, Feb 15, 2022 at 11:28 AM Taylor Frey  wrote:
>
> 1. Attendance:
> Srijeet Chatterjee
> Eric Holguin
> Steve Hamrick
> Taylor Frey
> Justin Howard
>
> 2. Official Release:
> - TC 7.0.0
> - Officially remove TO 2.x / Riak
> - Ansible currently uses 2, needs to finalize upgrade to 3 w/ testing
> - TO API 4.x will remain unstable for this release given the 'in
> development' progress of a few items
>
> On Tue, Feb 8, 2022 at 10:18 AM ocket   wrote:
>
> > If you have anything you want to discuss at next week's meeting,
> > respond to this email and it's on the agenda.
> >

Re: 2022-02-01 TC Working Group Agenda and Meeting Notes

[Eric Friedrich]

Rocky makes sense to me as the next step after Centos8

--Eric

On Tue, Feb 1, 2022 at 1:20 PM ocket   wrote:
>
> Hey, that sounds great to me. As far as I'm concerned, this is all the
> discussion necessary to move forward. I'm just saying I can't demand
> that our project support any particular OS on my own without talking
> about it with the list.
>
> To be clear, I was definitely never going to suggest Alma - not the
> least because I couldn't remember the name - or anything else. Rocky
> seems like the right choice. And by "going to 7 does sound like moving
> backwards" I didn't mean to drop support for CentOS 7, just that
> testing on only CentOS 7 instead of updating our CentOS 8 images to
> Rocky seemed like the wrong call. I don't think we need to drop
> support for that even after being comfortable with Rocky, until it
> hits EoL and/or becomes an inconvenience to support.
>
> On Tue, Feb 1, 2022 at 10:33 AM Dave Neuman  wrote:
> >
> > +1 to retaining support for 7 until we are good with Rocky.  Now *that* is
> > something that we can decide as a community.
> >
> > On Tue, Feb 1, 2022 at 10:31 AM Rawlin Peters  wrote:
> >
> > > We should probably retain support for CentOS 7 until such a time that
> > > we're comfortable with Rocky going forward. I do think Rocky is the
> > > future (at least for Comcast that seems to be the case). It's really
> > > Rocky vs Alma, and if someone wants to add support for Alma in
> > > addition to Rocky and can maintain that support, they should be free
> > > to do so.
> > >
> > > - Rawlin
> > >
> > > On Tue, Feb 1, 2022 at 10:21 AM ocket   wrote:
> > > >
> > > > Yeah, going to 7 does sound like moving backwards - we don't want to
> > > > do that, we want to move to whatever is going to be supported now that
> > > > CentOS 8 no longer works. The issue is that we need to know that Rocky
> > > > Linux is what the ATC project will support if we're going to use it
> > > > moving forward, and the working group does not have the authority to
> > > > make that call.
> > > >
> > > > On Tue, Feb 1, 2022 at 9:39 AM Dave Neuman  wrote:
> > > > >
> > > > > - Need mailing list confirmation that moving to Rocky is the plan,
> > > > > then our CI can be converted to Rocky Linux
> > > > >
> > > > > Do we though?  Can't we just put in support for Rocky and if folks 
> > > > > want
> > > > > support for other things they could add it?
> > > > > Seems like going to 7 is going backwards for what reason?
> > > > >
> > > > > On Tue, Feb 1, 2022 at 9:35 AM ocket   wrote:
> > > > >
> > > > > > 1. Attendance
> > > > > > - Stephen Hamrick
> > > > > > - Zach Hoffman
> > > > > > - Eric Holguin
> > > > > > - Matt Jackson
> > > > > > - Srijeet Chatterjee
> > > > > > - Taylor Frey
> > > > > > 2. CentOS 8 Docker image no longer works
> > > > > > - Should we make a workaround or switch to e.g. Rocky Linux ?
> > > > > > - Probably depends on overall project direction, but that
> > > conversation
> > > > > > takes longer than fixing our CI
> > > > > > - Issue #6343 needs to be fixed before we could downgrade to only
> > > > > > testing on CentOS 7
> > > > > > - Need mailing list confirmation that moving to Rocky is the plan,
> > > > > > then our CI can be converted to Rocky Linux
> > > > > > 3. ML: ATCv6.1.0-RC2 needs votes
> > > > > > - Currently has 2 PMC +1s
> > > > > > 4. ML: Summer of Code 2022
> > > > > >
> > > > > > On Tue, Feb 1, 2022 at 9:21 AM Zach Hoffman 
> > > wrote:
> > > > > > >
> > > > > > > Consider discussing:
> > > > > > > - CentOS 8 no longer works, which breaks our CI. Time to move to
> > > Rocky
> > > > > > Linux?
> > > > > > >
> > > > > > > Active mailing list threads:
> > > > > > > - Release Apache Traffic Control 6.1.0-RC2
> > > > > > > 
> > > > > > >
> > > > > > > On Tue, Jan 25, 2022 at 9:44 AM ocket  
> > > wrote:
> > > > > > > >
> > > > > > > > If you have anything you want to discuss at next week's meeting,
> > > > > > > > respond to this email and it's on the agenda.
> > > > > >
> > >

Re: 6.1.0 Planning

[Eric Friedrich]

Sounds good, Thanks Zach.

How's the state of master? Are we feeling good that its ready to cut a
release off of?

On Thu, Jan 6, 2022 at 11:16 AM Jeremy Mitchell  wrote:
>
> +1
>
> On Thu, Jan 6, 2022 at 9:13 AM Zach Hoffman  wrote:
>
> > Planning to cut the 6.1.0 release branch from master on Tuesday, January
> > 11.
> >
> > -Zach
> >

Re: [VOTE] Release Apache Traffic Control 6.0.2-RC2

[Eric Friedrich]

+1
Verified signatures, hashes and that I can build via pkg

On Tue, Dec 21, 2021 at 11:14 AM Jeremy Mitchell  wrote:
>
> +1
>
> On Mon, Dec 20, 2021 at 2:48 PM ocket   wrote:
>
> > Note that the 23rd is a Thursday
> >
> > On Mon, Dec 20, 2021 at 11:33 AM Zach Hoffman 
> > wrote:
> > >
> > > I've prepared a release for 6.0.2-RC2. Changes since RELEASE-6.0.1:
> > >
> > >
> > https://github.com/apache/trafficcontrol/compare/RELEASE-6.0.1...RELEASE-6.0.2-RC2
> > >
> > https://github.com/apache/trafficcontrol/blob/RELEASE-6.0.2-RC2/CHANGELOG.md
> > >
> > > The artifacts are available for download at:
> > >
> > > https://dist.apache.org/repos/dist/dev/trafficcontrol/6.0.2/RC2/
> > >
> > > BLAKE2 checksum:
> > >
> > 80cbac152f2dd37f5f874bafc5ce1d9c554ad78ce64e31fbb341db489ca22b4e5fe01e9dd35faaf9a2809f6636c6bac668963a142a71043535a0efd479ef
> > >  apache-trafficcontrol-6.0.2.tar.gz
> > >
> > > SHA512 checksum:
> > >
> > de0abe5670f0a8a80ab12c73688f9a111794190d208d3190b6961934c0499ddb398f7fc45cd99d330c9f807e80c8a22cec385dbd60dd6498cc7b44831794452a
> > >  apache-trafficcontrol-6.0.2.tar.gz
> > >
> > > This corresponds to git refs:
> > >
> > > Hash: d33bf4189911f8631d0583b28c9e70151ca82535
> > > Tag: RELEASE-6.0.2-RC2
> > >
> > > Which can be verified with the following command:
> > >
> > > $ git tag -v RELEASE-6.0.2-RC2
> > >
> > > All code signing keys are available here:
> > >
> > > https://dist.apache.org/repos/dist/release/trafficcontrol/KEYS
> > >
> > > Make sure you refresh from a key server to get all relevant signatures.
> > >
> > > The vote is open until 19:00 UTC on Wednesday, December 23 and passes
> > > if a majority of at least 3 +1 PMC votes are cast.
> > > [ ] +1 approve
> > > [ ] +0 no opinion
> > > [ ] -1 disapprove (and reason why)
> > >
> > > -Zach
> >

Re: [VOTE] Release Apache Traffic Control 6.0.2-RC0

[Eric Friedrich]

I am -1 on this release.

On Fri, Dec 17, 2021 at 6:45 PM Zach Hoffman  wrote:
>
> I've prepared a release for 6.0.2-RC0. Changes since RELEASE-6.0.1:
>
> 
> https://github.com/apache/trafficcontrol/compare/RELEASE-6.0.1...RELEASE-6.0.2-RC0
> 
> https://github.com/apache/trafficcontrol/blob/RELEASE-6.0.2-RC0/CHANGELOG.md
>
> The artifacts are available for download at:
>
> https://dist.apache.org/repos/dist/dev/trafficcontrol/6.0.2/RC0/
>
> BLAKE2 checksum:
> a439091c2494d8eaf6eaeb458a65a72ad2564640882d855460c8428de4ee01fd5692f99ec739e9d5459fb75fa0a87c5e3809f57c7aa6d6a4f3894a84dd922a90
>  apache-trafficcontrol-6.0.2.tar.gz
>
> SHA512 checksum:
> ad2dfcb5f867addb261c924aec56b62e71a5db51930f0ffaa86c559f73b28b404a572581ab409df56ae63ef21816f7ce7382fb3c25378335268a750c9da1
>  apache-trafficcontrol-6.0.2.tar.gz
>
> This corresponds to git refs:
>
> Hash: 007107c699c7425c91d2d1a6f5b9928b7f3b2b9f
> Tag: RELEASE-6.0.2-RC0
>
> Which can be verified with the following command:
>
> $ git tag -v RELEASE-6.0.2-RC0
>
> All code signing keys are available here:
>
> https://dist.apache.org/repos/dist/release/trafficcontrol/KEYS
>
> Make sure you refresh from a key server to get all relevant signatures.
>
> The vote is open until 00:00 UTC on Tuesday, December 21 and passes if
> a majority of at least 3 +1 PMC votes are cast.
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and reason why)
>
> -Zach

Re: [EXTERNAL] Re: [PROPOSAL] ATC Release Schedule

[Eric Friedrich]

I like the idea of more frequent releases. Trying to think ahead a month.

Are there any problems we foresee with cutting a 6.1 RC and voting? Do
we have a plan for how to address these, or do we just handle as
issues arise?
(Partial features in progress, important bugs we want to squeeze in, etc...)

--Eric

On Tue, Nov 2, 2021 at 5:45 PM Dave Neuman  wrote:
>
> sure, seems like what we originally agreed to :)
>
>
> On Tue, Nov 2, 2021 at 3:17 PM Jeremy Mitchell 
> wrote:
>
> > OK, so can we agree on this?
> >
> > - minor/major releases every start of a quarter (1/1, 4/1, 7/1, 10/1)
> > - patch releases in between as determined by release manager
> >
> > by that logic, we may have N patch releases between now and 12/31 and on
> > 1/1 we will have 6.1.0 (or 7.0 if a major is warranted).
> >
> > Jeremy
> >
> >
> >
> > On Wed, Oct 27, 2021 at 12:52 PM ocket   wrote:
> >
> > > Fortunately bugfixes rarely require migrations, those are usually only
> > for
> > > new features, or backporting would probably be impossible.
> > >
> > > On Wed, Oct 27, 2021 at 10:06 AM Jeremy Mitchell 
> > > wrote:
> > >
> > > > Just to be clear, cherry-picks (aka backports) (from master to the
> > > release
> > > > branch) should be limited to bug fixes and would only apply to patch
> > > > releases (i.e. 6.0.1) and I would expect the release manager to
> > determine
> > > > if the cherry-pick is safe or not. Yes, some can be very messy and
> > > > dangerous and IMO those should probably be punted to the next release
> > > > branch.
> > > >
> > > > jeremy
> > > >
> > > > On Wed, Oct 27, 2021 at 8:59 AM Gray, Jonathan
> > > >  wrote:
> > > >
> > > > > > > why not instead ask “Is it good enough?” and “Is it better than
> > > what
> > > > we
> > > > > presently have?” as often as possible.
> > > > >
> > > > > > may not help very much, because fixing a bug is always "better than
> > > > what
> > > > > we
> > > > > presently have".
> > > > >
> > > > > Not always.  Sometimes the risks entailed with cherry-picking and
> > > > > repatching bugfixes isn’t small.  Sometimes bugfixes rely on subtle
> > > side
> > > > > effects or behaviors not captured in the PR.  It’s not safe to assume
> > > > that
> > > > > taking code out of the context in which it was written and tested to
> > be
> > > > > inherently stable or better.
> > > > >
> > > > > TO specifically has a problem with the idea of cherry-picking with
> > TODB
> > > > > migrations.  Since the migrations are linear and there still has to
> > be
> > > a
> > > > > forward path from one of these cherry-picked releases, you could end
> > up
> > > > in
> > > > > a situation where you’ve got database schema conflicts introduced by
> > a
> > > > > cherry-pick.  It feels like the desire I’m reading here is wanting
> > the
> > > > > benefits of adopting GitFlow, while trying to still use GitHubFlow in
> > > the
> > > > > repo.
> > > > >
> > > > > Jonathan G
> > > > >
> > > > > From: ocket  
> > > > > Date: Tuesday, October 26, 2021 at 10:23 PM
> > > > > To: dev@trafficcontrol.apache.org 
> > > > > Subject: Re: [EXTERNAL] Re: [PROPOSAL] ATC Release Schedule
> > > > > > Thusly, the version number will "choose itself" based on the SemVer
> > > > > tenants.
> > > > >
> > > > > only really works for major vs minor. Micro versions are
> > backport-only,
> > > > so
> > > > > they don't necessarily have anything to do with whether or not
> > breaking
> > > > > changes have been made on master. Which is also why
> > > > >
> > > > > > why not instead ask “Is it good enough?” and “Is it better than
> > what
> > > we
> > > > > presently have?” as often as possible.
> > > > >
> > > > > may not help very much, because fixing a bug is always "better than
> > > what
> > > > we
> > > > > presently have".
> > > > >
> > > > > Essentially it seems like our release cadence wouldn't actually
> > change
> > > > very
> > > > > much, the proposal isn't to get rid of the quarterly releases we
> > strive
> > > > > for, but just adding a plan to backport bug fixes every month. Which,
> > > we
> > > > > could also decide not to do, if no bugs were fixed that month, or if
> > > it's
> > > > > too much trouble to backport the fix - "as-needed" as Jeremy puts it.
> > > The
> > > > > big difference is we typically only do micro releases for critical
> > bugs
> > > > > that slipped through the validation process, or for security fixes,
> > > > whereas
> > > > > under the terms of the proposal we'd make an effort to collect things
> > > > that
> > > > > could be fixed in supported releases every month.
> > > > >
> > > > > > asking folks to vote on new RC every few weeks isn’t feasible
> > unless
> > > > it’s
> > > > > a standing “good enough” process
> > > > >
> > > > > that would definitely be better, but I think we hope that few enough
> > > > things
> > > > > are going into micro releases, with very little unknown since the
> > last
> > > > > minor, that validation could be focused just on "are these bugs
> > > 

Re: CVE-2021-42009: Apache Traffic Control Arbitrary Email Content Insertion in /deliveryservices/request

[Eric Friedrich]

Additional Information:

Impacted Versions:
5.1.x users should upgrade to 5.1.3 or 6.0.0.
4.1.x users should upgrade to 5.1.3.

Credit:
This issue was discovered by GitHub's CodeQL code scanning service.

On Mon, Oct 11, 2021 at 8:29 PM Eric Friedrich  wrote:

> Description:
>
> An authenticated Traffic Ops user with Portal-level privileges can send a
> request with a specially-crafted email subject to the
> /deliveryservices/request Traffic Ops endpoint to send an email, from the
> Traffic Ops server, with an arbitrary body to an arbitrary email address.
>
>

CVE-2021-42009: Apache Traffic Control Arbitrary Email Content Insertion in /deliveryservices/request

[Eric Friedrich]

Description:

An authenticated Traffic Ops user with Portal-level privileges can send a 
request with a specially-crafted email subject to the /deliveryservices/request 
Traffic Ops endpoint to send an email, from the Traffic Ops server, with an 
arbitrary body to an arbitrary email address.

Re: [VOTE] Release Apache Traffic Control 5.1.3-RC0

[Eric Friedrich]

+1

On Tue, Oct 5, 2021 at 7:29 PM Zach Hoffman  wrote:

> +1
>
> On Tue, Oct 5, 2021 at 5:28 PM Dave Neuman  wrote:
>
> > +1
> >
> > On Tue, Oct 5, 2021 at 16:40 ocket   wrote:
> >
> > > Hello All,
> > >
> > > I've prepared a release for v{{version}}-RC{{RC Number}}
> > >
> > > The vote is open for at least 72 hours and passes if a majority of at
> > least
> > > 3 +1 PMC votes are cast.
> > >
> > > [ ] +1 Approve the release
> > >
> > > [ ] -1 Do not release this package because ...
> > >
> > > Changes since 5.1.2
> > >
> >
> https://github.com/apache/trafficcontrol/compare/v5.1.2...RELEASE-5.1.3-RC0
> > >
> > > This corresponds to git:
> > > Hash: a52ae6cda73ef477aec691b0d27701326d20e025
> > > Tag: RELEASE-5.1.3-RC0
> > >
> > > Which can be verified with the following: git tag -v RELEASE-5.1.3-RC0
> > >
> > > My code signing key is available here:
> > >
> > >
> >
> https://keys.gnupg.net/pks/lookup?search=0xF5D560A373B76FA302A95DBFCDFE430685982C95=on=index
> > >
> > > Make sure you refresh from a key server to get all relevant signatures.
> > >
> > > The source .tgz file, pgp signature (.asc signed with my key from
> > > above), and sha512 checksums are provided here:
> > >
> > > https://dist.apache.org/repos/dist/dev/trafficcontrol/5.1.3/RC0
> > >
> > >
> > > Thanks!
> > > ocket ocket8...@apache.org
> > >
> >
>

Re: [VOTE] Release Apache Traffic Control RELEASE-6.0.0-RC4

[Eric Friedrich]

+1

Verified sigs/hashes and build via ./pkg on OS X.

I tried to bring up CIAB, but had trouble following the instructions to
prep the rpms into the right locations.

On Wed, Sep 29, 2021 at 6:48 PM Zach Hoffman  wrote:

> I've prepared a release for 6.0.0-RC4. Changes since RELEASE-5.1.2:
>
>
> https://github.com/apache/trafficcontrol/compare/RELEASE-5.1.2...RELEASE-6.0.0-RC4
>
> https://github.com/apache/trafficcontrol/blob/RELEASE-6.0.0-RC4/CHANGELOG.md
>
> The artifacts are available for download at:
>
> https://dist.apache.org/repos/dist/dev/trafficcontrol/6.0.0/RC4/
>
> BLAKE2 checksum:
> 94e462c1d751edc60f398dc7c588603190cc830e029fbb39acdee74d291a31ffb7d1bdc74fad4678f146c463f43b82537c0706183149c7b85b7564f98e093a94
> apache-trafficcontrol-6.0.0.tar.gz
>
> SHA512 checksum:
> 94da739c6af1cbb307d192e24d346de13cab2a139e5233fafdaa5308729dece748f15beab6a8ab0b7caa160f3c9116ecb986da101ace6aaaf65738036a00ea89
> apache-trafficcontrol-6.0.0.tar.gz
>
> This corresponds to git refs:
>
> Hash: 5a2eea21a3175ad76839d54cacbd0386c09966f4
> Tag: RELEASE-6.0.0-RC4
>
> Which can be verified with the following command:
>
> $ git tag -v RELEASE-6.0.0-RC4
>
> All code signing keys are available here:
>
> https://dist.apache.org/repos/dist/release/trafficcontrol/KEYS
>
> Make sure you refresh from a key server to get all relevant signatures.
>
> The vote is open until 22:00 UTC on Monday, October 4 and passes if a
> majority of at least 3 +1 PMC votes are cast.
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and reason why)
>
> -Zach
>

Re: Invite for Traffic control Slack channel

[Eric Friedrich]

Invite sent

On Wed, Sep 22, 2021 at 8:10 AM Mihajlo Minovic 
wrote:

> Hi all Traficcontrol devs!
>
> Can I please get invited to the Traffic Control channel on
> the-asf.slack.com
> ?
> Having access to the Slack channel would make it a lot easier for me to
> become proficient with ATC. Thank you in advance!
>
> Best regards,
> Mihajlo Minovic
>

Re: Proposal: Remove "Events" wiki section

[Eric Friedrich]

There are some slides on these pages that we should hang on to:
https://cwiki.apache.org/confluence/display/TC/Fall+2016+Meetup
https://cwiki.apache.org/confluence/display/TC/Feb+2016+Traffic+Control+Meetup
https://cwiki.apache.org/confluence/display/TC/Spring+2017

Otherwise, I'm good with removal

On Tue, Aug 31, 2021 at 9:08 PM Jeremy Mitchell 
wrote:

> +1 not sure a list of past events has much value.
>
> On Tue, Aug 31, 2021 at 4:11 PM ocket   wrote:
>
> > Sure. I didn't see any on the one or two I checked out to understand what
> > the section is about. Those should go with all the other presentations
> IMO
> > - however that winds up getting grouped.
> >
> > On Tue, Aug 31, 2021 at 3:51 PM Zach Hoffman 
> wrote:
> >
> > > A few Event subpages include slides for presentations given. The
> slides,
> > as
> > > well as the corresponding schedule of presentations, should be
> preserved.
> > >
> > > -Zach
> > >
> > > On Tue, Aug 31, 2021 at 3:03 PM ocket  
> wrote:
> > >
> > > > https://cwiki.apache.org/confluence/display/TC/Events
> > > >
> > > > Rather than transfer that over to the GH wiki, I vote we just get rid
> > of
> > > > it entirely. It looks like the information here has no purpose beyond
> > > > whatever historical value we want to assign it - and since this does
> > NOT
> > > > contain the presentations that I know we want to keep, I don't see
> that
> > > > there's much historical value, especially not for the things like
> > Google
> > > > Maps embeds for directions and public transit route planning, since
> > > nobody
> > > > can plan to attend an event in the past.
> > > >
> > > > If we want to keep things like schedules, dates, and locations, then
> > I'd
> > > > suggest we use those to group the saved presentations, rather than
> > having
> > > > the two separate sections.
> > > >
> > >
> >
>

Re: ATC 6.0.0 planning

[Eric Friedrich]

Thanks Zach!  This is a great milestone.

 As Release Manager, all changes to the 6.0.x branch have to be approved by
Zach first.

On Tue, Aug 17, 2021 at 11:19 AM Zach Hoffman  wrote:

> I cut the 6.0.x release branch from the ATC master branch yesterday. If you
> need me to add any additional PR to the 6.0.x branch after it is merged
> into the master branch, make sure it is added to the 6.0.0 milestone.
>
> -Zach
>
> On Tue, Jul 27, 2021 at 7:13 PM Zach Hoffman  wrote:
>
> > As discussed in the Traffic Control working group today, a number of PRs
> > in the 6.0.0 milestone are still open. To accommodate this, the 6.0.x
> > release branch cut date has been pushed back to Monday, August 16.
> >
> > -Zach
> >
> >
> > On Mon, Jul 12, 2021 at 10:53 PM Zach Hoffman 
> > wrote:
> >
> >> There is still some work to be done before the release branch is ready
> to
> >> be cut (see: https://github.com/apache/trafficcontrol/milestone/18 ). I
> >> now plan to cut the 6.0.x release branch on Monday, August 2.
> >>
> >> Please land any non-bugfix changes you want included in ATC 6.0.0 within
> >> the next 3 weeks.
> >>
> >> -Zach
> >>
> >> On Tue, Jul 6, 2021 at 3:11 PM Jeremy Mitchell 
> >> wrote:
> >>
> >>> Looking at the 6.0.0 milestone -
> >>> https://github.com/apache/trafficcontrol/milestone/18
> >>>
> >>> I think we need to wait for the api 1.x removal project to be complete
> (
> >>> https://github.com/apache/trafficcontrol/milestone/14) before the 6.0
> >>> bus
> >>> can leave the station. So, july 19 seems reasonable to me.
> >>>
> >>> Jeremy
> >>>
> >>> On Mon, Jun 28, 2021 at 3:18 PM Zach Hoffman 
> >>> wrote:
> >>>
> >>> > The next scheduled major release of Apache Traffic Control is 6.0.0,
> >>> which
> >>> > will be tagged from the 6.0.x branch. In 3 weeks, July 19, I plan to
> >>> cut
> >>> > the 6.0.x release branch from the tip of the master branch.
> >>> >
> >>> > Please land any non-bugfix changes you want included in ATC 6.0.0
> >>> within
> >>> > the next 3 weeks.
> >>> >
> >>> > -Zach
> >>> >
> >>>
> >>
>

Re: Proposal: Distributed Health Monitoring

[Eric Friedrich]

Thanks Rob!
  Two main questions:
1) Do you have a description of the failure modes and how TM mitigates
those?

2) Can you say more about using a GSLB to balance between TR load between
TMs? Will a GSLB be a requirement in the immediate future for the updated
TM?

Thanks,
Eric

On Thu, Jul 15, 2021 at 2:35 PM Robert O Butts  wrote:

> The current Traffic Monitor polls the entire CDN, which doesn't scale with
> large numbers of caches.
>
> We've long intended to make it scalable. Several of us have a design
> proposal. I'm putting it here, just so people don't have to click an
> external link, but if anyone wants I can also put this on the Apache wiki,
> just ask.
>
> We typically do designs with Blueprints in git, but this project is large
> enough, we felt the need to come up with a higher-level design first. Once
> we have consensus on the design, the intent is to make a blueprint with
> more technical details, and then seek consensus on that. Design follows.
>
>
> # Implementation
> - Should not mandate or preclude a deployment environment
>   - Such as VMs or Containers
> - This design intentionally minimizes Traffic Ops/Traffic Portal changes,
> to reduce development cost for the MVP. TO changes may be desirable, and
> may be implemented in a future iteration.
>
> ## Stats
> The existing Traffic Monitor monitors both stats and health. The existing
> Monitor will have health polling disabled and will continue to be used for
> stat polling for the immediate future. Beyond that, this document does not
> address stat monitoring.
>
> Currently, it’s possible to use stats for health consideration, but it’s
> unlikely it’s commonly used by anyone.
>
> - Should not preclude adding an interface (JSON/HTTP) to send health
> information (such as from stats) to be used in the health calculation
>   - For either cache or delivery service health
>   - This won’t be in the MVP, but is likely to be added in a future version
>
> ## Health
> - Must get and use Snapshotted (CRConfig, monitoring.json) data
> exclusively.
>   - Must Not request un-snapshotted Traffic Ops endpoints.
>   - This is because health monitoring must match Router configuration, and
> to prevent potentially multi-state operational changes from being deployed
> until an operator intends.
> - Monitors must get their own config and data from Traffic Ops without
> intervention.
>   - Deployment and running must be Stateless (in terms of persistent state;
> notwithstanding transient health state).
>   - Local configuration should be limited to Traffic Ops authentication.
> - Must use Traffic Ops as the Source-Of-Truth for all data about what to
> poll, but should be agnostic (i.e. handle generic JSON endpoints without
> special Traffic Ops–specific logic).
> - Monitors will be divided into “monitor groups,” which will be implemented
> as Traffic Control CacheGroups.
>   - CacheGroups will be used to minimize Traffic Ops (TO) and Traffic
> Portal (TP) development in the MVP.
>   - These Monitor CacheGroups have no correlation with Cache CacheGroups.
> TO CacheGroups are simply the grouping mechanism.
>   - Should not preclude a different grouping mechanism.
> - Monitors will directly poll the health of caches in their designated
> CacheGroup(s), and will request the health of other CacheGroups from a
> Monitor in each other Monitor Group.
> - Monitors will serve the health of all caches in the CDN.
>   - Both cache health directly polled and received from other monitors.
> - Monitors must distinguish in their health endpoint (/CrStates) between
> caches/cachegroups they polled themselves, and health received from other
> monitors.
>   - The endpoint should be backwards-compatible with the existing /CrStates
> endpoint used by existing Traffic Routers.
> - Cache health should be boolean, but must not preclude a more granular
> score in the future
>   - For example, 0.0–1.0 or 0–100
>   - The current Traffic Monitor cache health is boolean
>   - Note this is a “must,” not a “should,” unlike most extensibility
> requirements.
>
> ## Peering
> - Monitors must poll all peers in their own Group, and use the health of
> caches reported by those monitors combined with their own direct polling
> results to achieve consensus on the health of each cache.
>   - This is currently implemented in Traffic Monitor, with each Monitor
> polling all Caches in the CDN.
>   - The consensus will be Optimistic. If any Monitor directly polling a
> cache considers it healthy, then it’s considered healthy. This is the
> current behavior of Traffic Monitor.
>   - Should not preclude other consensus algorithms.
> - Monitors must select an arbitrary Monitor in each other Monitor Group
> each polling interval, to query for the health of CacheGroups not polled
> directly. This selection must be deterministic and load-balanced.
>   - For example, deterministic such as via alphabetic hostname.
>   - For example, load-balanced by round-robin or consistent hash on the
> deterministic 

Re: Distributed Traffic Monitor Feedback/Requirements

[Eric Friedrich]

I'll do my best to rephrase as a potential requirement :-)

1) Traffic Monitor MUST ensure all caches are monitored upon failure of any
TM server(s) or physical location. (i.e. no SPoF of TMs for
polling/aggregation).

Number of TM failures to be tolerated before we stop polling some caches /
how we accomplish the above/ maximum number of caches under supervision by
a TM are all TBD in design phase

--Eric

On Fri, Jun 25, 2021 at 10:36 AM Dave Neuman  wrote:

> Hey Eric,
> Thanks for the questions/feedback.  My responses are inline below.  Most of
> your questions will need to be addressed when we do design as right now I
> just want to make sure we are not missing any requirements.  I hope to
> start design discussions in the next week or two.
>
> Thanks,
> Dave
>
> On Fri, Jun 25, 2021 at 7:26 AM Eric Friedrich  wrote:
>
> > Some comments and questions jointly compiled
> >
> >   - How is TM configured to monitor a subset of a CDN, is it a static
> > allocation of caches to TMs?
> >
>
> DN:  I think that is to be determined when we start to think about design,
> which is after we agree on the requirements.  I think for our use case the
> most simple way to do this would be by cache group.  A Traffic Monitor
> could be configured to monitor 1 to many cache groups.  However, if there
> is a better way we could do this, I am all ears.
>
> >
> >   - Can you describe how the primary + backup work. Do they both poll the
> > cache simultaneously
> >
>
> DN: Again, I think we can sort out the details when we talk about design.
> It actually might make more sense to just have multiple TMs monitor a cache
> group and treat them all as "live", this has the benefit of providing more
> than one view of a cache.
>
>
> >   - If a TM fails, how do the TMs heal / reallocate polling
> > responsibilities. Does another TM pick up the slack?
> >
>
> DN:  You want to dive straight into design :). I think the easiest answer
> here is to ensure multiple TMs are polling each cache and that they are all
> treated as live, then we can just use the optimistic consensus that is
> already built into TM.
>
>
> >
> >   - What prevents a misconfiguration where some caches are not polled by
> > any TM?
> >
>
> DN:  Great question.  I don't think that is one I have considered, but I
> suppose we could add a requirement saying that TM must have a way to
> identify unpolled caches...what do you think?
>
>
> >
> >   - Are there any minimums/maximums to how many TMs will poll a cache?
> >
> DN: Minimum is one, maximum is up to the operator, I don't know of a limit
> in TM.
>
>
> >
> >   - What is meaning of non-boolean 0-100 health? How is this computed and
> > how is it used?
> >
>
> DN:  The health score stuff is going to be an entirely different topic
> because I don't think it needs to be conflated with distributed polling.  I
> put that requirement in because I wanted to document that this is something
> we are thinking about so that we don't make it difficult on ourselves when
> we do this refactor.
> Right now a cache's health is boolean, it either gets traffic or it
> doesn't.  The idea behind the health score is that we could assign
> different health scores for caches in a cache group and then TR can use
> that when determining which cache to choose.  Maybe you have multiple
> caches that are getting close to the bandwidth limit, instead of pulling
> all traffic from them, we could simply weight them lower so the TR prefers
> other caches, but can still use them if needed. We have a bunch of other
> use cases that are probably best saved for when we are ready to formally
> present the idea.
>
>
> >
> >   - What can we do to further harden TM<->TM communications and reduce
> > blast radius?
> >
>
> DN:  Another topic for the design discussions, I think the basic idea is to
> not have a SPoF which means multiple TMs polling each cache and multiple
> TMs available to provide status to TRs, Caches, and TSs.
>
>
>
> > Big thumbs up on decoupling TM from Traffic Ops. What does this
> practically
> > mean - no more monitoring.json? Can we document specifically which APIs
> TM
> > will use?
> > (Aside, we might want to think about this as an opportunity to move TM
> into
> > its own repository- assuming the community decides to go ahead with
> > separate repos per component).
> >
>
> DN:  I think that is a stretch goal for now.  TM will still have to get
> it's configuration from somewhere, but ideally it does not have to come
> from TO.  Ultimately I would like TO to just serve the basic data from the
>

Re: Distributed Traffic Monitor Feedback/Requirements

[Eric Friedrich]

Some comments and questions jointly compiled

  - How is TM configured to monitor a subset of a CDN, is it a static
allocation of caches to TMs?

  - Can you describe how the primary + backup work. Do they both poll the
cache simultaneously

  - If a TM fails, how do the TMs heal / reallocate polling
responsibilities. Does another TM pick up the slack?

  - What prevents a misconfiguration where some caches are not polled by
any TM?

  - Are there any minimums/maximums to how many TMs will poll a cache?

  - What is meaning of non-boolean 0-100 health? How is this computed and
how is it used?

  - What can we do to further harden TM<->TM communications and reduce
blast radius?

Big thumbs up on decoupling TM from Traffic Ops. What does this practically
mean - no more monitoring.json? Can we document specifically which APIs TM
will use?
(Aside, we might want to think about this as an opportunity to move TM into
its own repository- assuming the community decides to go ahead with
separate repos per component).



On Thu, Jun 17, 2021 at 6:09 PM Dave Neuman  wrote:

> Hey All,
> One of the things we have been talking about doing for a long time is
> making Traffic Monitor capable of monitoring a subset of the CDN so that it
> can be deployed in a distributed fashion.  The time has come for us to get
> moving on this.  We have had some discussions internally to understand what
> requirements we have for doing this, but I wanted to solicit feedback from
> the community to see if there are potentially other requirements that we
> may have missed.  Please take a look at the requirements we have identified
> below and let me know what feedback you have.  At this point in time I am
> trying to keep this conversation separate from the design conversation and
> just focus on the requirements.  Once we all agree on the requirements we
> can start discussing the design.  You will notice that this proposal also
> includes adding the ability to integrate with external monitoring systems.
> I figured now would be a good time to add that functionality in as well.
>
>
> *Abstract*
>
> Update Traffic Monitor so that it is capable of monitoring only part of the
> CDN while still providing a single API for clients to get cache stats,
> delivery stats, and cache availability for a whole CDN.  Add the ability to
> integrate with other systems that perform additional health monitoring and
> consider the status of these systems when making health decisions for a
> cache.  Ensure that the Traffic Monitor API is capable of serving thousands
> of simultaneous clients, such as all of the caches in a CDN.
>
>
> *Problem Statement*
>
> Currently Traffic Monitor can only monitor an entire CDN. This means that
> Traffic Monitor has to poll every single cache in a CDN before making cache
> health decisions and being able to provide statistics. This also means that
> Traffic Monitors need to be located in a centralized place where it can get
> to everything, which isn't exactly representative of what a client might
> see. While this has worked really well for us to date, we know that at some
> point we will run into scaling issues which prohibit us from polling caches
> faster.  In order to solve our impending scaling issues as well as improve
> our ability to make better and faster health decisions, Traffic Monitor
> needs to run in a distributed fashion instead of an all or nothing
> fashion.
>
> Furthermore, there is a growing need to provide support for external
> monitoring systems in Traffic Monitor.  Traffic Monitor needs to be able to
> use other monitoring systems to aid in the health decision process. While
> this could be solved in today's Traffic Monitor, it is best to solve this
> problem in conjunction with making the polling distributed.
> *Business Justification*
>
> In order to provide the best customer experience possible, we need to have
> a robust and timely health monitoring system.  While Traffic Monitor has
> been sufficient to date, we need to make sure that we are adapting to meet
> the needs of the near future and we need to make sure that we are evolving
> to continue to meet customers needs.  These changes to Traffic Monitor are
> imperative to providing as near real time as possible cache health data on
> our ever increasing in scale of the CDN.
> *Business Requirements*
>
>- Traffic Monitor MUST be capable of being configured to monitor a
>portion of a CDN
>- Traffic Monitor MUST be capable of being configured to monitor all
>caches in a CDN
>- Traffic Monitor MUST provide an API to get the health status of ALL
>caches in the CDN
>- Traffic Monitor MUST provide an API to get statistics (from e.g.
>astats data) generated by ALL caches in the CDN. This does not include
> any
>statistics generated by external monitoring systems.
>- Traffic Monitor MUST log all requests to its API including AT LEAST
>the following information: timestamp, client IP, resource requested,
>   

Re: [EXTERNAL] Converting Traffic Router to Kotlin would be easy

[Eric Friedrich]

Rawlin, ocket-

  Thanks for pointing out some of the additional benefits Kotlin has over
Java- I didn't know about the null check differences.


Can you explain more about how the conversion process works?

What are the risks associated with it (likelihood of introducing a bug,
risks of having mixed Java+Kotlin, etc...).


Also, how long could the community expect the transition to take and what
are the mechanics (separate branch, etc...)?


--Eric



On Wed, Jun 23, 2021 at 2:37 PM ocket   wrote:

> As someone who avoids looking at TR code as much as possible, I'd say it's
> certainly true that the codebase is a rat's nest, and switching to Kotlin
> won't automagically fix that. I do think, though, that it being Java
> exacerbates the problem because of how verbose Java is. I, personally,
> would not want to refactor TR's Java codebase; that just doesn't sound like
> fun to me. If it were Kotlin, though, I'd be much more willing to try to
> improve it.
>
> On Wed, Jun 23, 2021 at 10:15 AM Rawlin Peters  wrote:
>
> > > Rather, I think the TR code itself is badly in need of a clean up,
> > to improve modularity and to give some thought to extensibility.
> >
> > I agree, but it's very difficult to make a business case for "improve
> > modularity and extensibility" without a concrete use case behind it
> > that can justify doing that refactoring beforehand. Whereas converting
> > from Java to Kotlin can be done fairly easily on a piecemeal basis
> > over time by devs in their free time.
> >
> > > The solution to that is not easy, nor is it fast. It's a process of
> > making
> > small, incremental improvements to the cleanliness of TR with minimal
> risk
> > to stability.
> >
> > That actually sounds like what would happen during a conversion to
> > Kotlin. In fact, that's basically what I did when I converted
> > TrafficRouter.java to Kotlin yesterday:
> > https://github.com/rawlinp/trafficcontrol/commits/kotlin-tr. The
> > auto-conversion was pretty straightforward -- there were only 3 errors
> > I had to fix manually for it to compile. The rest was just fixing most
> > of the warnings/suggestions pointed out by IntelliJ, most of which
> > helped clean things up a bit, making it more readable and less
> > verbose. Combing over the results of the conversion like that is also
> > a great way to find even more things that can be cleaned up, which in
> > most cases would go ignored because no one is just combing over TR
> > looking for things to clean up.
> >
> > > Somewhat less of a concern is shrinking the pool of developers by
> > choosing
> > a less mature, less widely adopted language. I understand Kotlin is on
> the
> > rise and is in demand but is still dwarfed by the 20+ years of Java.
> >
> > I agree this is also less of a concern -- I think Java devs can learn
> > Kotlin pretty quickly because Kotlin is basically just an enhanced
> > version of Java. Also, the community did choose to replace the
> > original Traffic Ops with a language 20+ years it's junior, and that
> > was a tremendous amount of work compared to what converting Java to
> > Kotlin would be.
> >
> > One of the main tenets of a CDN is reliability, and all of the
> > features around null safety in Kotlin
> > (https://kotlinlang.org/docs/null-safety.html) can seriously help out
> > there, especially if the entire codebase was Kotlin. While the
> > codebase is a mix of the two, there is more chance of a
> > NullPointerException coming from Java code because all Java references
> > are nullable, and Kotlin needs to maintain that interoperability. Some
> > of the nastier bugs in TR have stemmed from NullPointerExceptions due
> > to not knowing that an argument was nullable, and with Kotlin's
> > built-in nullable vs non-nullable types, that problem is way less
> > likely to occur.
> >
> > If we were to develop TR 2.0 for instance (instead of just converting
> > TR 1.0), I'd hope we could use a language like Kotlin which has solved
> > the Billion Dollar Mistake.
> >
> > - Rawlin
> >
> > On Tue, Jun 22, 2021 at 9:14 PM Eric Friedrich 
> wrote:
> > >
> > > Zach-
> > >
> > >   First of all, thanks for trying to build a bigger tent here, I really
> > > appreciate that effort.
> > >
> > > (Personal comments, not as chair)
> > >
> > > I agree that Traffic Router is difficult to work on, and that presents
> a
> > > large barrier to entry. However, I don't think that is a result of
> > language
> > > choice. Rather, I think the TR cod

Re: [EXTERNAL] Converting Traffic Router to Kotlin would be easy

[Eric Friedrich]

Zach-

  First of all, thanks for trying to build a bigger tent here, I really
appreciate that effort.

(Personal comments, not as chair)

I agree that Traffic Router is difficult to work on, and that presents a
large barrier to entry. However, I don't think that is a result of language
choice. Rather, I think the TR code itself is badly in need of a clean up,
to improve modularity and to give some thought to extensibility.

Recently one of my coworkers added a new feature to TR. He is incredibly
experienced with Java, but had many problems trying to create a clean
integration.

Converting TR to Kotlin wouldn't improve or solve any of these issues. I
fear that even as we try to attract new contributors,  they would be
frustrated with the state of TR and not continue as ongoing contributors.

The solution to that is not easy, nor is it fast. It's a process of making
small, incremental improvements to the cleanliness of TR with minimal risk
to stability.


Somewhat less of a concern is shrinking the pool of developers by choosing
a less mature, less widely adopted language. I understand Kotlin is on the
rise and is in demand but is still dwarfed by the 20+ years of Java.

--Eric



On Tue, Jun 22, 2021, 7:14 PM Chris Lemmons  wrote:

> I don't have a strong opinion on Kotlin; I haven't used it to any real
> degree. I do know it interoperates with the Java API and compiles down
> to the JVM. So in terms of the openness of the platform, it's not any
> more "open" than Oracle's OpenJDK. It does compile down to native
> code, but then we'd need to replace (or convert and maintain) all our
> dependent libraries.
>
> So if Kotlin helps with maintainability, I say we go for it. I don't
> know that we're winning any additional openness we don't already have,
> though.
>
> On Tue, Jun 22, 2021 at 4:44 PM Taylor Frey  wrote:
> >
> > I too +1
> >
> > My experience porting a Java project to Kotlin was specific to Android in
> > the past (re: years ago). I recall my general impression of Kotlin to
> have
> > some extraneous and unnecessary syntactic sugar (What's up with the
> > for/loop keywords?), but it was an overall solid language. I appreciate
> the
> > terseness and brevity of things like variable declarations w/ implicit
> data
> > types and `data classes`, which addresses a long standing complaint about
> > Java's overly verbose syntax. Language aside, the automated porting back
> > then wasn't exactly perfect, but a few things have changed since then; 1)
> > the language specification has solidified over the years, 2) tooling has
> > matured, and 3) is manageable to manually intervene if necessary.
> >
> > > As a side note, because of Oracle's aggressive legal practices, and
> > because
> > they really do want you to think that they own Java (see: Google LLC
> v.
> > Oracle America, Inc.), using Java in Apache Traffic Control is not
> > really
> > in the spirit of free software. Kotlin is an Apache 2.0-licensed
> > project,
> > so switching to Kotlin would be a true FOSS win for the ATC
> community.
> >
> > I think the true benefit is moving away from the Java API, which is under
> > such contention, without changing underlying implementation (such as a
> > transition to another, non-JVM, language might incur).
> >
> > On Tue, Jun 22, 2021 at 11:42 AM Chatterjee, Srijeet
> >  wrote:
> >
> > > +1
> > > Sounds like a fun project!
> > >
> > > --Srijeet
> > >
> > > On 6/21/21, 8:43 PM, "Zach Hoffman"  wrote:
> > >
> > > Hi Traffic Control!
> > >
> > > As you all may know, the current implementation of Traffic Router
> runs
> > > on
> > > Java 8. While coding directly in Java is still done, there are some
> > > reasons
> > > to be critical of Java.
> > >
> > > Java often has very verbose syntax when trying to accomplish simple
> > > things.
> > > This has made the Traffic Router codebase grow larger over time,
> > > leading to
> > > maintainability issues. Also, Java suffers from lack of
> compile-time
> > > safety
> > > in some ways that more modern languages do not. For these reasons,
> and
> > > because Java is a very old language, some developers who may
> otherwise
> > > be
> > > interested in contributing to Apache Traffic Control may be
> > > disinterested
> > > contributing to Traffic Router in particular, simply because they
> want
> > > to
> > > work with more modern languages that do not have the issues that
> Java
> > > does.
> > >
> > > Kotlin, a JVM language devised by Jetbrains with the goal of
> tackling
> > > these
> > > problems, has exploded in popularity in recent years, and Kotlin is
> > > becoming a standard way to use the JVM, with Google choosing it as
> the
> > > preferred language to for developing Android apps. More
> importantly,
> > > as the
> > > Intellij IDEA Kotlin plugin lets you convert Java sources to Kotlin
> > > (either
> > > at the file level or an entire project, or anywhere in between),
> 

Re: [VOTE] Release Apache Traffic Control 5.1.2-RC3

[Eric Friedrich]

+1
Checked signature and hashes. Builds with pkg


On Wed, May 12, 2021 at 11:05 AM Dave Neuman  wrote:

> +1
> Validated signature, hash, and ran pkg -v to build all RPMs.
>
> On Tue, May 11, 2021 at 6:34 PM Rawlin Peters  wrote:
>
> > +1
> >
> > On Fri, May 7, 2021 at 10:08 AM ocket   wrote:
> > >
> > > Hello All,
> > >
> > > I've prepared a release for v5.1.2-RC3
> > >
> > > The vote is open for at least 72 hours and passes if a majority of at
> > least
> > > 3 +1 PMC votes are cast.
> > >
> > > [ ] +1 Approve the release
> > > [ ] -1 Do not release this package because ...
> > >
> > > Changes since 5.1.1:
> > >
> >
> https://github.com/apache/trafficcontrol/compare/RELEASE-5.1.1...RELEASE-5.1.2-RC3
> > >
> > > This corresponds to git:
> > > Hash: 8892beff94f6bf699ee95bc1a66afa09775796f0
> > > Tag: RELEASE-5.1.2-RC3
> > >
> > > Which can be verified with the following: git tag -v RELEASE-5.1.2-RC3
> > >
> > > My code signing key is available here:
> > >
> >
> https://keys.gnupg.net/pks/lookup?search=0xF5D560A373B76FA302A95DBFCDFE430685982C95=on=index
> > >
> > > Make sure you refresh from a key server to get all relevant signatures.
> > > The source .tgz file, pgp signature (.asc signed with my key from
> > > above), and sha512 checksums are provided here:
> > > https://dist.apache.org/repos/dist/dev/trafficcontrol/5.1.2/RC3
> > >
> > > Thanks!
> > > ocket ocket8...@apache.org
> >
>

Re: ATC PMC Chair Change

[Eric Friedrich]

Thanks All-

I can only hope to be half as good a PMC Chair as Dave was.

On Thu, Apr 22, 2021 at 2:00 PM Jeremy Mitchell 
wrote:

> Thanks Dave for performing a thankless job and somehow convincing Eric to
> take it. ;) Under your watch, TC has grown significantly and knowing Eric,
> I'm sure it will continue to do so. Congrats Eric!
>
> Jeremy
>
> On Thu, Apr 22, 2021 at 9:37 AM Zach Hoffman  wrote:
>
>> Thank you Dave for your years of service as ATC's first PMC Chair.
>>
>> And congrats, Eric! I trust ATC to continue growing under your watch.
>>
>> -Zach
>>
>> On Thu, Apr 22, 2021 at 9:00 AM Dave Neuman  wrote:
>>
>> > Hey everyone,
>> >
>> > I have decided the time has come for me to pass the torch as PMC Chair
>> for
>> > Traffic Control. It has been a great one and I am so honored to have
>> served
>> > in this position.  With that being said, please help me congratulate and
>> > support our new PMC Chair and VP of the Apache Traffic Control Project;
>> > Eric Friedrich!
>> >
>> > Thanks,
>> > Dave
>> >
>> >
>>
>

Re: Proposal: ORT/t3c git

[Eric Friedrich]

Can you explain more about how the git repo works?

Is it a local repo or a git server somewhere?

How are configs on different servers stored (diff repos, diff branches,
diff directories, etc...)?


It sounds like a useful feature, but I think I'm missing the big picture.

On Tue, Mar 16, 2021, 4:01 PM Robert O Butts  wrote:

> I'd like to propose adding a feature to ORT/t3c to track config changes in
> a git repo.
>
> My plan is to add a flag `--git` with the options yes/no/auto, defaulting
> to `auto`. With 'auto' a repo will be committed to if it exists, but not
> created. With 'yes' the repo will be created and committed to. With 'no' no
> git operations will be performed.
>
> The idea is, people who want to start using it can either add `--git=yes`
> to their server automation (whatever's running ORT/t3c, like cron or
> ansible), or simply run it as a one-time command on each server (like with
> Ansible Push). After that, if the repo exists, it will automatically be
> used.
>
> Defaulting to auto will prevent any manual runs from accidentally not
> committing to the repo if they forget the flag, but will also avoid
> creating the repo if it doesn't exist, which some users may not desire.
>
> Finally, a 'no' option allows any users who are already using git to manage
> config and want to continue doing so outside ORT/t3c without ATC breaking
> and injecting new commits, to do so.
>
> I believe this will be a big benefit for operations, especially debugging
> production issues. For example:
> - in an emergency, halt any automation and git checkout to a previous known
> good configuration
> - use git to see how files changed over time, and see when a breaking
> change occurred
> - search git, to see all previous values for a particular setting
> - correlate historical config changes with CDN traffic behavioral changes
>
> PR is here: https://github.com/apache/trafficcontrol/pull/5648
>
> Does anyone have any objections or concerns with this? Does anyone have
> their ATS config directory as a git repo today? Will the above options
> break anyone?
>
> If nobody comments in 72 hours, I'll assume Lazy Consensus.
>
> Thanks,
>

Re: Discuss GSOC: Varnish Cache support in Apache Traffic Control

[Eric Friedrich]

Hi Chandan-
  Thanks for your interest! As you get more familiar with Traffic Control,
I think you'll see we align well to your languages - we have components in
C++ (Traffic Server), Javascript (Traffic Portal), Golang (Traffic Monitor,
Traffic Ops), and more. Varnish, although not technically part of Apache or
Traffic Control is also written in C.

In case you haven't seen it yet, please take a look at the Apache Summer of
Code page: https://community.apache.org/gsoc.html
It has some instructions on how to put together and submit a project
proposal and the all important deadlines.

Once you take a look over some of the docs Rob suggested or try things out,
come join our Slack. We'd be glad to go over some of the details with you
and put together a project proposal.


Welcome to Traffic Control!


--Eric


On Thu, Feb 25, 2021 at 6:11 PM Robert O Butts  wrote:

> The primary caching proxy supported by Apache Traffic Control (ATC) is
> Apache Traffic Server (ATS).
>
> Config generation for ATS caches is done by a tool called ORT:
> https://github.com/apache/trafficcontrol/tree/master/traffic_ops_ort
>
> https://github.com/apache/trafficcontrol/tree/master/traffic_ops_ort/atstccfg
>
> Typically, a Traffic Control CDN will run this on caches via a CRON job
> every few minutes.
>
> You could create a config generator for Varnish which does the same thing:
> request the Traffic Ops API, and use the data (Delivery Services, Servers,
> Parameters, etc) to generate the necessary config for Varnish, and then
> un-set the "update pending" flag via the Traffic Ops API.
>
> We do have another caching proxy in Traffic Control, Grove. Though I don't
> think anyone is running it in production at the moment:
> https://github.com/apache/trafficcontrol/tree/master/grove
>
> Grove has its own config generator similar to ORT:
> https://github.com/apache/trafficcontrol/tree/master/grove/grovetccfg
>
> If you're new to ATC, I'd encourage you to set up a CDN-in-a-Box (CiaB),
> just to get an idea of the different CDN components and how they work
> together:
>
> https://traffic-control-cdn.readthedocs.io/en/latest/admin/quick_howto/ciab.html#getting-started
>
> https://github.com/apache/trafficcontrol/tree/master/infrastructure/cdn-in-a-box
>
> In addition to getting it running, I'd also encourage you to look over the
> Dockerfiles and run-scripts in the CiaB, to get an understanding of each
> CDN component. You can also look at the independent Dockerfiles for each
> component, if that helps:
> https://github.com/apache/trafficcontrol/tree/master/infrastructure/docker
>
> And of course, the overall ATC docs:
>
> https://traffic-control-cdn.readthedocs.io/en/latest/overview/introduction.html
>
> I hope that helps! Don't hesitate to ask if you have more questions. I'd
> encourage you to join our Slack as well. Slack is usually the fastest way
> to get answers:
> https://s.apache.org/atc-slack
>
>
> On Thu, Feb 25, 2021 at 3:37 PM Chandan prakash 
> wrote:
>
> > Hi all,
> > I am Chandan Prakash, pre final year computer science student at IIT
> Mandi,
> > India. I am proficient in languages C, C++, python, javascript, Go, Rest
> > APIs, Graphql.
> > I want to know more about this project.
> >
> > (1).So, where should I start?
> >
> > (2). Is there any code base which I need to get a more understanding of
> > this project?
> >
> > I am new to this community but I have contributed to this other open
> source
> > projects earlier.
> > Thank you,
> > Chandan Prakash,
> > B-Tech 3rd year
> > IIT Mandi
> >
>

Re: Snapshots and Monitoring rework blueprint

[Eric Friedrich]

I know there was a bunch of discussion in the PR, but I think some of the
points raised around inconsistent state and race conditions have not been
satisfactorily addressed.

I understand that there is the opportunity for similar types of
inconsistencies today, but all our improvement should work towards
increasing the reliability of the system. As written today, I think the
reliability could potentially stay the same, but would likely be decreased
as a result of this change

On Wed, Jan 20, 2021 at 1:47 PM ocket   wrote:

> Is anyone still poring over this? We've got one approval so far and I'd
> love to come to a decision about whether or not to adopt this blueprint.
>
> On Fri, Dec 11, 2020 at 11:14 AM ocket   wrote:
>
> > Hello everyone, I've written up a blueprint (
> > https://github.com/apache/trafficcontrol/pull/5367) for reworking CDN
> > Snapshots and Monitoring payloads, and I wanted it to be on everyone's
> > radar.
> >
> > The gist of it is that TM using two data sources is causing some
> > hard-to-pin-down race conditions, so it shouldn't do that. So the plan is
> > to separate out exactly what TM needs in the Monitoring payloads, and
> then
> > Traffic Router can fetch CDN Snapshots directly from Traffic Ops instead
> of
> > using TM as a sort of proxy.
> >
>

Re: GitHub Actions Concurrency Limits for Apache projects

[Eric Friedrich]

While a private server is definitely one option, I think we should consider
the following questions:

Has anyone reached to the TS community to gather their experiences with the
private build server? As far as I know, it was a necessity because they had
to produce builds on many different platforms.

Will the apache jenkins server still work for us?

  What length of time is the host of the server willing to commit to?
Ideally would be several years.

What is the time commitment required from ATC to manage (security patches,
etc) this server/vm?

Are there any restrictions (bandwidth, compute) on load or use?

  Who has access to manage jobs on the server? Pmcs? Committers?

I would favor Apache infra and github actions over a private build server
because of the "durability", maintenance overhead and simplicity of the
existing solutions.

-- Eric


On Fri, Oct 23, 2020, 2:47 PM Zach Hoffman  wrote:

> TL;DR: Who can we ask to host runners for our CI?
>
> For the past 2 weeks, The bui...@apache.org mailing list has discussed a
> GitHub Actions usage limit for github.com/apache of 180 concurrent jobs
> total, which has become increasingly apparent. See the INFRA ticket at
> https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20978
> and the mailing list thread at
>
> https://lists.apache.org/thread.html/r1708881f52adbdae722afb8fea16b23325b739b254b60890e72375e1%40%3Cbuilds.apache.org%3E
> .
>
> The projects contributing the most to this job starvation are ones that use
> the "matrix" execution strategy, which concurrently runs a job across each
> permutation of build parameters like Node.js version or operating system.
> Also contributing are workflows that run without regard for which
> components are modified. In our case, that could mean building and running
> the "readiness" check in CDN in a Box when the only files a PR modifies are
> documentation-related.
>
> Although Apache Traffic Control does not do either of the things mentioned
> above, as a regular user of GitHub Actions, we both contribute to this
> resource shortage and are impacted by it. Also, as ATC only began adding
> Actions in mid-August, there are a number of checks that we have yet to add
> but plan to.
>
> See Jarek Potiuk's suggestions at
>
> https://docs.google.com/document/d/16rwyCfyDpKWN-DrLYbhjU0B1D58T1RFYan5ltmw4DQg
> for ways that we could lessen our impact on the resource shortage:
>
> > • projects should optimize their builds - for example using Apache Yetus
> or pre-commit.com
> > • projects should seek for funding for self-hosted runners
>
> Other projects already host their own runners successfully, such Apache
> Traffic Server, which has its own Jenkins server with 23 runners. Their CI
> server is facilitated by Rackspace and GoDaddy.
>
> Hosting our runners is an appealing option because would be able to grow
> our own CI without fear of impacting other projects. Is anyone else in
> favor of asking our community if anyone can host runners for us?
>
> -Zach
>
> -- Forwarded message -
> From: Jarek Potiuk 
> Date: Fri, Oct 23, 2020 at 9:29 AM
> Subject: Re: GitHub Actions Concurrency Limits for Apache projects
> To: 
>
>
> Started working on this mini-solution for limiting non-approved
> matrix builds.
>
> I am working on it with a colleague of mine -  Tobiasz - who worked on
> Apache Beam infrastructure, so we might test it on two projects.
>
> I will let you know the progress
>
> Mini-design doc here:
>
> https://docs.google.com/document/d/16rwyCfyDpKWN-DrLYbhjU0B1D58T1RFYan5ltmw4DQg/edit#
>
> J.
>
>
> On Thu, Oct 22, 2020 at 10:03 PM Jarek Potiuk 
> wrote:
>
> >
> >
> > I believe this problem cannot be really handled by one project, but I
> have
> > a proposal.
> >
> > I looked at the common pattern we have in the ASF projects and I think
> > there is a way that we can help each other.
> >
> > I think most of the problems come from many PRs submitted that run a
> > matrix of tests before even commiters have time to take a look at them.
> We
> > discussed how we can approach it and I think I have a proposal that we
> can
> > all adopt in the ASF projects. Something that will be easy to implement
> and
> > will not impact the process we have. I would love to hear your thoughts
> > about it - before I start implementing it :).
> >
> > My proposal is to create a GitHub Action that will allow to run only a
> > subset of "matrix" test for PRs that are not yet approved by committers.
> > This should be possible using the current GitHub Actions workflows and
> API.
> > It boils down to:
> > * If PR is not approved, only a subset of matrix (default value for each
> > matrix component) are run
> > * the committers can see the "green" mark of test passing and make a
> review
> > * once the PR gets approved, automatically a new "full matrix" check is
> > triggered
> > * all future approved PR pushes run the "full matrix" check
> >
> > I think that might significantly reduce the strain on GA jobs we run, and
> 

Re: [VOTE] Release Apache Traffic Control -RC

[Eric Friedrich]

I am -1 on the release as the tarball does not build.

Signatures and checksums otherwise look good.

Here is the resulting error message:
+ mkdir -p src pkg bin /tmp/go/src/github.com/apache
+ rsync -a --exclude=/dist --exclude=/.m2 /trafficcontrol/ /tmp/go/src/
github.com/apache/trafficcontrol
+ '[' -d /tmp/go/src/github.com/apache/trafficcontrol/.git ']'
+ rsync -a /trafficcontrol/.git /tmp/go/src/github.com/apache/trafficcontrol
rsync: link_stat "/trafficcontrol/.git" failed: No such file or directory
(2)
rsync error: some files/attrs were not transferred (see previous errors)
(code 23) at main.c(1179) [sender=3.1.2]
+ exit_code=23
+ '[' 23 -ne 0 ']'
+ echo 'Error on line 1 of /clean_build.sh'
Error on line 1 of /clean_build.sh
+ cleanup
+ setowner /trafficcontrol /trafficcontrol/dist
++ stat -c%u:%g /trafficcontrol
+ own=0:0
+ shift
+ '[' -n /trafficcontrol/dist ']'
+ chown -R 0:0 /trafficcontrol/dist
+ exit 23
Failed to build source traffic_monitor_build traffic_ops_build
traffic_ops_ort_build traffic_portal_build traffic_router_build
traffic_stats_build grove_build grovetccfg_build docs.


On Wed, Oct 21, 2020 at 9:15 AM Eric Friedrich 
wrote:

> Thanks for preparing the RC ocket!
>
> Since I think this is the first time we've attempted a release with
> convenience binaries, can you share what you've done to ensure license
> compliance with those new artifacts?
>
> I'll do my best to get a vote together on the release in the next 3 days.
>
> --Eric
>
>
> On Tue, Oct 20, 2020 at 5:49 PM ocket   wrote:
>
>> Hello All,
>>
>> I've prepared a release for v5.0.0-RC0
>>
>> The vote is open for at least 72 hours and passes if a majority of at
>> least
>> 3 +1 PMC votes are cast.
>>
>> [ ] +1 Approve the release
>>
>> [ ] -1 Do not release this package because ...
>>
>> Changes since 4.1.0:
>> https://github.com/apache/trafficcontrol/compare/4.1.0...RELEASE-5.0.0-RC0
>>
>> This corresponds to git:
>> Hash: d5ee85fdcd36b998573e8819b63c51764093dff9
>> Tag: RELEASE-5.0.0-RC0
>>
>> Which can be verified with the following: git tag -v RELEASE-5.0.0-RC0
>>
>> My code signing key is available here:
>>
>> https://keys.gnupg.net/pks/lookup?search=0xF5D560A373B76FA302A95DBFCDFE430685982C95=on=index
>>
>> Make sure you refresh from a key server to get all relevant signatures.
>> Also, their SSL key seems messed up right now, so you might get a browser
>> warning.
>>
>> The source .tgz file, pgp signature (.asc signed with my key from
>> above), and sha512 checksums are provided here:
>>
>> https://dist.apache.org/repos/dist/dev/trafficcontrol/5.0.0/RC0
>>
>>
>> Thanks!
>> ocket ocket8...@apache.org
>>
>

Re: [VOTE] Release Apache Traffic Control -RC

[Eric Friedrich]

Thanks for preparing the RC ocket!

Since I think this is the first time we've attempted a release with
convenience binaries, can you share what you've done to ensure license
compliance with those new artifacts?

I'll do my best to get a vote together on the release in the next 3 days.

--Eric


On Tue, Oct 20, 2020 at 5:49 PM ocket   wrote:

> Hello All,
>
> I've prepared a release for v5.0.0-RC0
>
> The vote is open for at least 72 hours and passes if a majority of at least
> 3 +1 PMC votes are cast.
>
> [ ] +1 Approve the release
>
> [ ] -1 Do not release this package because ...
>
> Changes since 4.1.0:
> https://github.com/apache/trafficcontrol/compare/4.1.0...RELEASE-5.0.0-RC0
>
> This corresponds to git:
> Hash: d5ee85fdcd36b998573e8819b63c51764093dff9
> Tag: RELEASE-5.0.0-RC0
>
> Which can be verified with the following: git tag -v RELEASE-5.0.0-RC0
>
> My code signing key is available here:
>
> https://keys.gnupg.net/pks/lookup?search=0xF5D560A373B76FA302A95DBFCDFE430685982C95=on=index
>
> Make sure you refresh from a key server to get all relevant signatures.
> Also, their SSL key seems messed up right now, so you might get a browser
> warning.
>
> The source .tgz file, pgp signature (.asc signed with my key from
> above), and sha512 checksums are provided here:
>
> https://dist.apache.org/repos/dist/dev/trafficcontrol/5.0.0/RC0
>
>
> Thanks!
> ocket ocket8...@apache.org
>

Re: TR Upgrading to Java 11 LTS

[Eric Friedrich]

I think this is a great improvement for Traffic Router.

Other than the sun.security, what changes to code structure or syntax
should we expect from this PR?

I’d echo ockets comment that replacing sun.security should not be
grouped together with the Java 11 upgrade. Might be useful to remove
sun.security first and then a second one for the Java upgrade

--Eric

On Fri, Sep 11, 2020 at 6:25 PM ocket   wrote:

> > IMO refactoring to remove sun.security.* usage is enough of its own thing
> to warrant its own PR
>
> I agree, especially since replacing it with a 1:1 package mirror of sorts
> doesn't seem like the best way to go about it. From cursory searching it
> seems like it java.security is supposed to be a functional
> superset/equivalent to sun.security, just not a drop-in replacement. That
> could be wrong; I'm not a Java guy, but it sounds as though you should be
> able to do whatever you need to do without relying on a specific provider.
>
> On Fri, Sep 11, 2020 at 1:45 PM Zach Hoffman  wrote:
>
> > > 2.  Use a third-party library. Some third-library parties offer partial
> > replacements of the namespace (e.g. BouncyCastle).
> >
> > For the cases where java.security.* alone is not sufficient, do we need
> > anything outside of org.bouncycastle.jce.provider? That's been a
> dependency
> > since https://github.com/apache/trafficcontrol/commit/6ebd7a428f , so we
> > already have that at our disposal.
> >
> > > 3.  Custom implementation. We could just mirror the functionality of
> the
> > ‘sun.security.*’ namespace within the ATC codebase. While it gives more
> > fine-grained control, the ATC project is now responsible for maintaining
> > certificate management code and other things that are irrelevant to the
> > main focus of the project.
> >
> > I'm -1 for rolling our own crypto
> >
> > IMO refactoring to remove sun.security.* usage is enough of its own thing
> > to warrant its own PR before anything Java 11-related. Super excited to
> see
> > this project, Java 11 will mean new language features and a performance
> > boost.
> >
> > -Zach
> >
> >
> > On Fri, Sep 11, 2020 at 11:52 AM Zenn, Joshua (CCI-Atlanta) <
> > joshua.z...@cox.com> wrote:
> >
> > > Hello! I’ve been working on upgrading ATC Traffic Router to use Java 11
> > > LTS from Java 8. I’ve come across an issue, and it was proposed that it
> > be
> > > presented here for an official consensus on how to move forward.
> > >
> > > As of Java 9+ the ‘sun.security.*’ namespace has been made
> internal-only,
> > > which a fair amount of TR certificate management code relies on. Up
> > through
> > > Java 8 Oracle has stated that the package was for internal use only,
> but
> > it
> > > was publicly accessible by other namespaces. Based on my research so
> > far, I
> > > found that other projects have taken one or more of these paths to
> > resolve
> > > this issue:
> > >
> > >   1.  Hack some compiler options to manually include the
> ‘sun.security.*’
> > > internal package again. This is not recommended by Oracle and has the
> > > possibility of being broken on any given Java update, but it is the
> > fastest
> > > to implement and will have the least impact on the codebase.
> > >   2.  Use a third-party library. Some third-library parties offer
> partial
> > > replacements of the namespace (e.g. BouncyCastle). The trade-off is
> that
> > we
> > > would lose some fine-grained control over certificate loading (which
> > might
> > > affect other ATC components). This option may also introduce some
> > licensing
> > > issues. Finally, this would require an overhaul of the cert part of the
> > TR
> > > codebase.
> > >   3.  Custom implementation. We could just mirror the functionality of
> > the
> > > ‘sun.security.*’ namespace within the ATC codebase. While it gives more
> > > fine-grained control, the ATC project is now responsible for
> maintaining
> > > certificate management code and other things that are irrelevant to the
> > > main focus of the project.
> > >
> > > A great suggestion that was given by Zach Hoffman in Slack was to use
> the
> > > ‘java.security.*’ namespace. This would resolve some of the missing
> > > classes. However, the namespace is not a 1:1 implementation of
> > > sun.security.*, so the remaining missing classes would need to be
> > resolved
> > > using one of the above methods (or any new ideas that can be
> suggested).
> > >
> > > --
> > > Joshua Zenn
> > >
> >
>

Re: Petition to restrict password minimum length

[Eric Friedrich]

How can the database enforce password length?

On Tue, Aug 18, 2020 at 1:08 PM Dave Neuman  wrote:

> Seems reasonable however I don't think we need to wait for major version.
> We could probably get away with doing it over minor versions too.
>
> On Tue, Aug 18, 2020 at 11:00 AM ocket   wrote:
>
> > Currently, Traffic Portal restricts the passwords entered to a minimum
> of 8
> > characters. This restriction is not mirrored in the database (and
> possibly
> > not in the API, but that can be fixed at the same time as the db if
> that's
> > the case). I propose that we add this restriction to prevent potentially
> > wildly insecure passwords from existing for Traffic Ops clients.
> >
> > This would entail including a notice in the 5.0 release, probably in the
> > changelog, one or four places in the documentation, and possibly another
> > email to the users list after the 5.0 - to notify - and 6.0 - to remind -
> > releases. Then, a migration can be added to 6.0 to restrict password
> length
> > at the database level, giving users a full major upgrade cycle to make
> > their data compliant with the new restrictions.
> >
>

Re: [VOTE] Release Apache Traffic Control 4.1.0-RC0

[Eric Friedrich]

+1 (binding)

Tested:
  - Hashes/Sigs
  - Build (all rpms). Weasel fails, but I don't see that as -1 worthy
  - TR/TM come up and serve traffic

On Thu, Apr 23, 2020 at 7:59 PM Rawlin Peters  wrote:

> Hello All,
>
> I've prepared a release for v4.1.0-RC0
>
> The vote is open for at least 72 hours and passes if a majority of at
> least 3 +1 PMC votes are cast.
>
> [ ] +1 Approve the release
>
> [ ] -1 Do not release this package because ...
>
> Changes since 4.0.0:
>
> https://github.com/apache/trafficcontrol/compare/RELEASE-4.0.0...RELEASE-4.1.0-RC0
>
> This corresponds to git:
> Hash: 440b4a791b2711f42fbd07960b6aa7d5df24e826
> Tag: RELEASE-4.1.0-RC0
>
> Which can be verified with the following: git tag -v RELEASE-4.1.0-RC0
>
> My code signing key is available here:
> http://keys.gnupg.net/pks/lookup?search=0x8A0712500C70C06E=vindex
>
> Make sure you refresh from a key server to get all relevant signatures.
>
> The source .tgz file, pgp signature (.asc signed with my key from
> above), and sha512 checksums are provided here:
>
> https://dist.apache.org/repos/dist/dev/trafficcontrol/4.1.0/RC0
>
>
> Thanks!
> Rawlin Peters raw...@apache.org
>

Re: Request Sign Up to Slack Community

[Eric Friedrich]

Invite sent, welcome to Traffic Control!

If you'd like you can discuss on users@ or dev@ list or in Slack as well.

--Eric



On Mon, Oct 21, 2019 at 7:34 AM Rifqi Muttaqien  wrote:

> Hi Traffic Control Dev Team,
>
> First of all, let me introduce my name is Rifqi as IT Architecture
> Engineer at Biznet Gio Cloud Indonesia. We're interested to use your
> project Apache Traffic Control.
> In few days ago, we've try to build Apache Traffic Control. Using CIAB and
> manually installing each component one by one.
> Our CIAB setup was running well. But, we've some problem when installing
> manually which its problem was not found in Github Issues page.
>
> Could you please to invite us to sign up to your Slack Community for
> further discussion.
> Thank you.
>
> Regards,
> Rifqi Muttaqien
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments.
>

Re: Mail server requirement for API endpoints that send emails

[Eric Friedrich]

Yes, I think SMTP is an unreasonable requirement for user registration- it
should be optional. If I wanted to spin up a small CIAB, I would prefer not
to have to supply my ISPs email server.

This also does not account for environments where TO may be deployed that
do not have Internet access or are segmented from email servers for
security reasons.

Finally, it would be great if the implementation can include support for
both SMTP over TLS and username/password authentication out of the box.

--Eric

On Fri, Sep 13, 2019 at 12:10 PM ocket   wrote:

> With the introduction of letsencrypt in PR #3534 (WIP atm) a configuration
> option will be added to TO to specify the connection options for an SMTP
> server for sending emails.
>
> I'm currently in the process of re-writing /deliveryservices/request, which
> sends an email as its primary functionality. AFAIK, there are two other
> endpoints that do this, /users/register and /user/reset_password. The
> current Perl implementation `exec`s `sendmail` to send emails without an
> SMTP server. There isn't really a Go library (that I could find) that can
> just directly send emails without an SMTP server to which to connect.
> General sentiment is that this is what should be done, it seems, but it
> means making the operation of endpoints that "just work" with a given
> configuration in Perl return an error without amending said configuration.
>
> Personally, I think that's an improvement, and a few others I've spoken
> with agree. It would also line up nicely with the configuration in the
> upcoming letsencrypt functionality.
>
> tl;dr: Does anybody have a problem with requiring an external SMTP server
> for the /deliveryservices/request, /users/register and /user/reset_password
> endpoints?
>

Re: Feature Blueprints

[Eric Friedrich]

I'm a big fan of documentation and like the idea.

The template is very verbose- i wouldn't want to fill that out if I was
proposing a new feature. Can it be simplified at all? In the 2 examples
given, most of the sections are left blank. What about a generic prompt
such as:
"Describe changes to each Traffic Control component. Any changes to
external interfaces (DBs, APIs, Schemas) or existing behavior should be
included. Also consider how this may impact security and performance of the
overall system".

Once the feature is built and merged, how does the blueprint get folded
back into the docs? Should we expect every blueprint to result in updated
read-the-docs?









On Wed, Aug 21, 2019 at 11:27 PM Chris Lemmons  wrote:

> This is potentially a cool idea. I like that it doesn't require a wiki
> account and it gets feedback directly in the form of a PR.
>
> On Tue, Aug 20, 2019 at 3:22 PM Jeremy Mitchell 
> wrote:
> >
> > I am of the belief along with several others that more comprehensive
> > feature requirements early in the development process can:
> >
> > - lead to better/more efficient discussions
> > - help achieve a higher level of transparency across groups
> > - help to break down the work so it can be "tasked"
> > - lead to better estimates
> > - help identify any risks associated with the new feature
> > - mitigate the risk of "delivering the wrong thing"
> > - produce better tests
> > - help identify operational costs
> > - help identify deployment impacts/cost
> > - help develop deployment procedures
> > - help identify workarounds
> > - etc
> > - etc
> >
> > Therefore, several of us at Comcast have come up with what we think is a
> > relatively simple feature template (we call it a blueprint) designed to
> > capture the initial requirements of a feature and facilitate discussion
> and
> > generate feedback. It was designed to be lightweight yet comprehensive.
> We
> > plan to use this blueprint template internally at Comcast for new TC
> > features, however, we thought it might have value in the TC open source
> > community as well.
> >
> > I've opened a PR that includes the blueprint template plus 2 examples of
> > its use: https://github.com/apache/trafficcontrol/pull/3884
> >
> > If this is something we want to do in open source, here is the idea. When
> > proposing a new feature, a developer can optionally:
> >
> > 1. Create a new PR that includes a markdown file that utilizes the
> > BLUEPRINT_TEMPLATE.md template. For example, submit a PR to TC to
> > blueprints/my-feature.md
> > 2. Send an email to the dev list with a short description of your feature
> > plus a link to the blueprint PR
> > 3. Wait for feedback to be applied to your blueprint PR. (because it's a
> > PR, line-specific feedback can be given.)
> > 4. Just like any PR, once all the concerns have been addressed, the
> > blueprint is merged into the blueprints directory (if accepted) or closed
> > (if rejected).
> > 5. Start work on the feature
> > 6. Submit a PR for the feature and reference the blueprint.
> >
> > Of course, this can be OPTIONAL. Contributors can still create features
> > without a blueprint or maybe some contributors feel more comfortable
> > demonstrating a feature with working code. However, some contributors may
> > want to use a feature blueprint to "bounce ideas off the group" and gain
> > some consensus before moving forward. IMO the way we discuss
> > requirements/design now in email is very inefficient / painful plus it's
> > hard to get a good snapshot of the final design without going back thru
> the
> > entire email thread.
> >
> > You also may be asking yourself "why not just create an issue instead?".
> > The reason is because "issues" can't be commented on at the line level
> and
> > issues tend to get lost. The blueprints would build up in the blueprints
> > directory for later reference if needed.
> >
> > If this is not something our open source community wants to embrace, that
> > is fine too. We can keep this as an internal Comcast thing and I can
> simply
> > close the PR.
> >
> > Thoughts?
>

PR3075 - Merging parent.config in traffic_ops_golang

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

Hey All-
  Looks like we are ready to get #3075 into master. I have some Cache 
Assignment Group related changes I would like to make to parent.config and 
would prefer to do them in Go.

I’m waiting for this PR to get merged before I can proceed:
https://github.com/apache/trafficcontrol/pull/3075


Looks like there are no objections in the review and all of the tests pass.

Would one of the reviewers be willing to hit the merge button?

Thanks,
Eric

Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

Yes, we want to be able to link together different CAGs to form a hierarchy 
that can be used as an alternate to cache groups. 


> On May 23, 2019, at 3:12 PM, Chris Lemmons  wrote:
> 
> So, what's the benefit here of having a dedicated CAG object instead
> of letting it just be a relation between caches and delivery services?
> The "implied object" method of simply matching names seems
> considerably more flexible going forward and it's easier to see the
> relationships in the data. Is there a performance problem or extra
> data we want to store on the CAG?
> 
> On Thu, May 23, 2019 at 1:05 PM Fieck, Brennan
>  wrote:
>> 
>> I'd like to propose that instead of adding 
>> `/api/1.4/cacheassignmentgroups/{{id}}` that supports PUT and DELETE, we 
>> just let `/api/1.4/cacheassignmentgroups` handle those operations/methods 
>> with an identifying query parameter. There are two or three endpoints that 
>> do this already, I think `coordinates` is one of them.
>> 
>> From: Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus 
>> (UK) Limited -OBO at Cisco) 
>> Sent: Thursday, May 23, 2019 12:33 PM
>> To: dev@trafficcontrol.apache.org
>> Subject: Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API
>> 
>> Discussion looks to have slowed down and it was a bit of a long road, so 
>> I’ll summarize where we ended up.
>> 
>> POST,GET /api/1.4/cacheassignmentgroups/
>> PUT /api/1.4/cacheassignmentgroups/{id}
>> {"id": 1,
>> "name": "name1",
>> "description": "description1",
>> "cdnId": 1,
>> "servers": [1,2,...n],
>> "lastUpdated": "",
>> }
>> 
>> DELETE /api/1.4/cacheassignmentgroups/{id}
>> 
>> — DS->CAG Assignments —
>> PUT/POST /deliveryservices
>> Add optional list of assigned CAGs to the delivery service struct
>> 
>> — Retrieving Server<->DS Assignments —
>> 
>> GET /api/$version/deliveryservices/:id/servers
>>  Will return a list of explicit server assignments unioned with servers 
>> assigned through CAGs. (Full server details)
>> 
>> POST /api/$version/deliveryservices/:xmlId/servers
>>  Will accept a list of explicit server assignments (server names) as it does 
>> today.
>> 
>> 
>> 
>>> On May 22, 2019, at 12:18 PM, Eric Friedrich -X (efriedri - TRITON UK BIDCO 
>>> LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco) 
>>>  wrote:
>>> 
>>> Sorry, should have been more clear. I was thinking about a case where there 
>>> was no explicit assignment, only a CAG assignment.
>>> 
>>> -Eric
>>> 
>>>> On May 22, 2019, at 9:06 AM, Fieck, Brennan  
>>>> wrote:
>>>> 
>>>>> If someone POSTs to a DS to try and remove the name of a server that is 
>>>>> assigned via a CAG, the call will return success but neither the explicit 
>>>>> assignment nor the CAG assignments will be changes
>>>> 
>>>> Why not remove the explicit assignment? IMO, a success response tells the 
>>>> client that the operation succeeded, so doing that when in fact it didn't 
>>>> is a confusing lie from the client's perspective.
>>>> 
>>>> From: Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter 
>>>> Domus (UK) Limited -OBO at Cisco) 
>>>> Sent: Tuesday, May 21, 2019 12:36 PM
>>>> To: dev@trafficcontrol.apache.org
>>>> Subject: Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API
>>>> 
>>>> GET /api/$version/deliveryservices/:id/servers
>>>> Will return a list of explicit server assignments unioned with servers 
>>>> assigned through CAGs. (Full server details)
>>>> 
>>>> POST /api/$version/deliveryservices/:xmlId/servers
>>>> Aside: Why does this one use xmlId when all these other endpoint use only 
>>>> ID? Its rather inconsistent…
>>>> 
>>>> Will accept a list of explicit server assignments (server names) as it 
>>>> does today.
>>>> If someone POSTs to a DS with the name of a server that is assigned via a 
>>>> CAG, it will also become explicitly assigned to that DS.
>>>> If someone POSTs to a DS to try and remove the name of a server that is 
>>>> assigned via a CAG, the call will return success but neither the explicit 
>>>> assignment nor the CAG assignme

Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

Sure good feedback. 

> On May 23, 2019, at 3:04 PM, Fieck, Brennan  wrote:
> 
> I'd like to propose that instead of adding 
> `/api/1.4/cacheassignmentgroups/{{id}}` that supports PUT and DELETE, we just 
> let `/api/1.4/cacheassignmentgroups` handle those operations/methods with an 
> identifying query parameter. There are two or three endpoints that do this 
> already, I think `coordinates` is one of them.
> ____
> From: Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus 
> (UK) Limited -OBO at Cisco) 
> Sent: Thursday, May 23, 2019 12:33 PM
> To: dev@trafficcontrol.apache.org
> Subject: Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API
> 
> Discussion looks to have slowed down and it was a bit of a long road, so I’ll 
> summarize where we ended up.
> 
> POST,GET /api/1.4/cacheassignmentgroups/
> PUT /api/1.4/cacheassignmentgroups/{id}
> {"id": 1,
> "name": "name1",
> "description": "description1",
> "cdnId": 1,
> "servers": [1,2,...n],
> "lastUpdated": "",
> }
> 
> DELETE /api/1.4/cacheassignmentgroups/{id}
> 
> — DS->CAG Assignments —
> PUT/POST /deliveryservices
> Add optional list of assigned CAGs to the delivery service struct
> 
> — Retrieving Server<->DS Assignments —
> 
> GET /api/$version/deliveryservices/:id/servers
>  Will return a list of explicit server assignments unioned with servers 
> assigned through CAGs. (Full server details)
> 
> POST /api/$version/deliveryservices/:xmlId/servers
>  Will accept a list of explicit server assignments (server names) as it does 
> today.
> 
> 
> 
>> On May 22, 2019, at 12:18 PM, Eric Friedrich -X (efriedri - TRITON UK BIDCO 
>> LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco) 
>>  wrote:
>> 
>> Sorry, should have been more clear. I was thinking about a case where there 
>> was no explicit assignment, only a CAG assignment.
>> 
>> -Eric
>> 
>>> On May 22, 2019, at 9:06 AM, Fieck, Brennan  
>>> wrote:
>>> 
>>>> If someone POSTs to a DS to try and remove the name of a server that is 
>>>> assigned via a CAG, the call will return success but neither the explicit 
>>>> assignment nor the CAG assignments will be changes
>>> 
>>> Why not remove the explicit assignment? IMO, a success response tells the 
>>> client that the operation succeeded, so doing that when in fact it didn't 
>>> is a confusing lie from the client's perspective.
>>> 
>>> From: Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus 
>>> (UK) Limited -OBO at Cisco) 
>>> Sent: Tuesday, May 21, 2019 12:36 PM
>>> To: dev@trafficcontrol.apache.org
>>> Subject: Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API
>>> 
>>> GET /api/$version/deliveryservices/:id/servers
>>> Will return a list of explicit server assignments unioned with servers 
>>> assigned through CAGs. (Full server details)
>>> 
>>> POST /api/$version/deliveryservices/:xmlId/servers
>>> Aside: Why does this one use xmlId when all these other endpoint use only 
>>> ID? Its rather inconsistent…
>>> 
>>> Will accept a list of explicit server assignments (server names) as it does 
>>> today.
>>> If someone POSTs to a DS with the name of a server that is assigned via a 
>>> CAG, it will also become explicitly assigned to that DS.
>>> If someone POSTs to a DS to try and remove the name of a server that is 
>>> assigned via a CAG, the call will return success but neither the explicit 
>>> assignment nor the CAG assignments will be changes
>>> 
>>> 
>>> I was initially worried of someone doing a GET from :id/servers and then 
>>> POSTing the response to :xmlId/servers but the formats are somehow so 
>>> different thats not really a concern.
>>> 
>>> —Eric
>>> 
>>> 
>>>> On May 21, 2019, at 2:05 PM, Jeremy Mitchell  wrote:
>>>> 
>>>> Rawlin beat me to it.
>>>> 
>>>> /api/$version/deliveryservices/:id/servers <-- tenancy is already checked
>>>> (i hope) on this route.
>>>> 
>>>> imo if CAGs are introduced, the handler associated with that route ^^ needs
>>>> to be modified to take CAGs into account (in addition to explicit server
>>>> assignments)
>>>> 
>>>> On Tue, May 21, 2019 at 10:52 AM Rawlin Peters 
>&g

Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

Discussion looks to have slowed down and it was a bit of a long road, so I’ll 
summarize where we ended up. 

POST,GET /api/1.4/cacheassignmentgroups/
PUT /api/1.4/cacheassignmentgroups/{id}
{"id": 1,
"name": "name1",
"description": "description1",
"cdnId": 1,
"servers": [1,2,...n],
"lastUpdated": "",
}

DELETE /api/1.4/cacheassignmentgroups/{id}

— DS->CAG Assignments —
PUT/POST /deliveryservices
Add optional list of assigned CAGs to the delivery service struct

— Retrieving Server<->DS Assignments — 

GET /api/$version/deliveryservices/:id/servers 
  Will return a list of explicit server assignments unioned with servers 
assigned through CAGs. (Full server details)

POST /api/$version/deliveryservices/:xmlId/servers 
  Will accept a list of explicit server assignments (server names) as it does 
today. 



> On May 22, 2019, at 12:18 PM, Eric Friedrich -X (efriedri - TRITON UK BIDCO 
> LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco) 
>  wrote:
> 
> Sorry, should have been more clear. I was thinking about a case where there 
> was no explicit assignment, only a CAG assignment.
> 
> -Eric
> 
>> On May 22, 2019, at 9:06 AM, Fieck, Brennan  
>> wrote:
>> 
>>> If someone POSTs to a DS to try and remove the name of a server that is 
>>> assigned via a CAG, the call will return success but neither the explicit 
>>> assignment nor the CAG assignments will be changes
>> 
>> Why not remove the explicit assignment? IMO, a success response tells the 
>> client that the operation succeeded, so doing that when in fact it didn't is 
>> a confusing lie from the client's perspective.
>> 
>> From: Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus 
>> (UK) Limited -OBO at Cisco) 
>> Sent: Tuesday, May 21, 2019 12:36 PM
>> To: dev@trafficcontrol.apache.org
>> Subject: Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API
>> 
>> GET /api/$version/deliveryservices/:id/servers
>>  Will return a list of explicit server assignments unioned with servers 
>> assigned through CAGs. (Full server details)
>> 
>> POST /api/$version/deliveryservices/:xmlId/servers
>> Aside: Why does this one use xmlId when all these other endpoint use only 
>> ID? Its rather inconsistent…
>> 
>> Will accept a list of explicit server assignments (server names) as it does 
>> today.
>> If someone POSTs to a DS with the name of a server that is assigned via a 
>> CAG, it will also become explicitly assigned to that DS.
>> If someone POSTs to a DS to try and remove the name of a server that is 
>> assigned via a CAG, the call will return success but neither the explicit 
>> assignment nor the CAG assignments will be changes
>> 
>> 
>> I was initially worried of someone doing a GET from :id/servers and then 
>> POSTing the response to :xmlId/servers but the formats are somehow so 
>> different thats not really a concern.
>> 
>> —Eric
>> 
>> 
>>> On May 21, 2019, at 2:05 PM, Jeremy Mitchell  wrote:
>>> 
>>> Rawlin beat me to it.
>>> 
>>> /api/$version/deliveryservices/:id/servers <-- tenancy is already checked
>>> (i hope) on this route.
>>> 
>>> imo if CAGs are introduced, the handler associated with that route ^^ needs
>>> to be modified to take CAGs into account (in addition to explicit server
>>> assignments)
>>> 
>>> On Tue, May 21, 2019 at 10:52 AM Rawlin Peters 
>>> wrote:
>>> 
>>>> I'm hesitant to add a server list to the delivery service object just
>>>> because it's a lot of data (same with adding a delivery service list
>>>> to the server object). Two of the things that are done the most in
>>>> Traffic Ops are:
>>>> 1. adding new servers
>>>> 2. adding new delivery services
>>>> Every time we do one of those things we're already increasing the size
>>>> of the CRConfig at an M*N rate, so by adding a server list into the DS
>>>> object and a list of DSes into the server object would gives those
>>>> objects the same problem as the CRConfig. By adding an aggregation
>>>> between the two of them -- the Cache Assignment Group -- it is more
>>>> reasonable to include the list of CAGs in the server and DS objects.
>>>> 
>>>> I agree with Eric's common questions (which DSes are assigned to a
>>>> server and which servers are assigned to a DS), but we do already have
>>>> specific endpoints for th

Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

Sorry, should have been more clear. I was thinking about a case where there was 
no explicit assignment, only a CAG assignment.

-Eric

> On May 22, 2019, at 9:06 AM, Fieck, Brennan  wrote:
> 
>> If someone POSTs to a DS to try and remove the name of a server that is 
>> assigned via a CAG, the call will return success but neither the explicit 
>> assignment nor the CAG assignments will be changes
> 
> Why not remove the explicit assignment? IMO, a success response tells the 
> client that the operation succeeded, so doing that when in fact it didn't is 
> a confusing lie from the client's perspective.
> ____
> From: Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus 
> (UK) Limited -OBO at Cisco) 
> Sent: Tuesday, May 21, 2019 12:36 PM
> To: dev@trafficcontrol.apache.org
> Subject: Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API
> 
> GET /api/$version/deliveryservices/:id/servers
>   Will return a list of explicit server assignments unioned with servers 
> assigned through CAGs. (Full server details)
> 
> POST /api/$version/deliveryservices/:xmlId/servers
>  Aside: Why does this one use xmlId when all these other endpoint use only 
> ID? Its rather inconsistent…
> 
>  Will accept a list of explicit server assignments (server names) as it does 
> today.
>  If someone POSTs to a DS with the name of a server that is assigned via a 
> CAG, it will also become explicitly assigned to that DS.
>  If someone POSTs to a DS to try and remove the name of a server that is 
> assigned via a CAG, the call will return success but neither the explicit 
> assignment nor the CAG assignments will be changes
> 
> 
> I was initially worried of someone doing a GET from :id/servers and then 
> POSTing the response to :xmlId/servers but the formats are somehow so 
> different thats not really a concern.
> 
> —Eric
> 
> 
>> On May 21, 2019, at 2:05 PM, Jeremy Mitchell  wrote:
>> 
>> Rawlin beat me to it.
>> 
>> /api/$version/deliveryservices/:id/servers <-- tenancy is already checked
>> (i hope) on this route.
>> 
>> imo if CAGs are introduced, the handler associated with that route ^^ needs
>> to be modified to take CAGs into account (in addition to explicit server
>> assignments)
>> 
>> On Tue, May 21, 2019 at 10:52 AM Rawlin Peters 
>> wrote:
>> 
>>> I'm hesitant to add a server list to the delivery service object just
>>> because it's a lot of data (same with adding a delivery service list
>>> to the server object). Two of the things that are done the most in
>>> Traffic Ops are:
>>> 1. adding new servers
>>> 2. adding new delivery services
>>> Every time we do one of those things we're already increasing the size
>>> of the CRConfig at an M*N rate, so by adding a server list into the DS
>>> object and a list of DSes into the server object would gives those
>>> objects the same problem as the CRConfig. By adding an aggregation
>>> between the two of them -- the Cache Assignment Group -- it is more
>>> reasonable to include the list of CAGs in the server and DS objects.
>>> 
>>> I agree with Eric's common questions (which DSes are assigned to a
>>> server and which servers are assigned to a DS), but we do already have
>>> specific endpoints for those two questions:
>>> /api/$version/servers/:id/deliveryservices
>>> /api/$version/deliveryservices/:id/servers
>>> 
>>> Those endpoints would have to start taking into account any CAGs.
>>> 
>>> - Rawlin
>>> 
>>> 
>>> 
>>> On Tue, May 21, 2019 at 8:45 AM Fieck, Brennan
>>>  wrote:
>>>> 
>>>> I'd be a fan of adding a servers array to DS objects. We don't need the
>>> whole server object in each entry, just some identifying information (name,
>>> id, type should be sufficient I would think).
>>>> 
>>>> From: Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter
>>> Domus (UK) Limited -OBO at Cisco) 
>>>> Sent: Tuesday, May 21, 2019 8:09 AM
>>>> To: dev@trafficcontrol.apache.org
>>>> Subject: Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API
>>>> 
>>>> I like this, but I think we still have a challenge with tenancy.
>>>> 
>>>> Two of the very common questions with these groups are
>>>> 
>>>> 1) Which servers are assigned to this Delivery Service (either
>>> individually or through a CAG).
>>>> 
>>>&g

Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

 Adding a new server into multiple overlapping CAGs per tenant
>>> might get out of hand. This would allow us to keep the total number of
>>> "logical server groups" down but still prevent a tenant from seeing
>>> another tenant's CAGs.
>>> 
>>> What do you think?
>>> 
>>> - Rawlin
>>> 
>>> 
>>> 
>>> 
>>> On Thu, May 16, 2019 at 10:18 AM Jeremy Mitchell 
>>> wrote:
>>>> 
>>>> yeah, maybe i overcomplicated it with my example and got it wrong.
>>>> 
>>>> if the CAG belongs to tenant 2 and john is the only one that belongs to
>>>> tenant 2 (sally belongs to 2.1 and linda to 2.1.1), then john is really
>>> the
>>>> only one that can modify the CAG. and since the CAG only contains DS's
>>> from
>>>> tenant 2, 2.1 or 2.2, he can modify ALL the ds's in that CAG...
>>>> 
>>>> so disregard what i said in my example about what sally and linda can
>>>> modify because they can't if you add tenantId to CAG.
>>>> 
>>>> so i think
>>>> 
>>>> 1. add a tenantID to a CAG
>>>> 2. enforce user tenancy on CAG post/put/delete
>>>> 3. on post/put, ensure the tenancy of the assigned ds's are compatible
>>> with
>>>> the CAG tenant
>>>> 
>>>> jeremy
>>>> 
>>>> 
>>>> On Thu, May 16, 2019 at 9:08 AM Eric Friedrich -X (efriedri - TRITON UK
>>>> BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
>>>>  wrote:
>>>> 
>>>>> Jeremy-
>>>>>  Going back to your original example of the tree below.
>>>>> 
>>>>> If DS baz is assigned to tenantId 2, then tenantIds 2.1, 2.1.1, 2.2
>>> cannot
>>>>> modify baz, right?
>>>>> 
>>>>> If a CAG is created also with tenantId2, then I would expect the same
>>>>> behavior- only John as part of the 2 tenant can modify that CAG.
>>>>> Similarly, this CAG could only contain DS’ that belong to our are
>>> children
>>>>> of tenant 2.
>>>>> 
>>>>> This seems to match existing behavior more closely?
>>>>> 
>>>>> —Eric
>>>>> 
>>>>>> On May 16, 2019, at 10:43 AM, Jeremy Mitchell >>> 
>>>>> wrote:
>>>>>> 
>>>>>> no, capabilities are very different than tenancy.
>>>>>> 
>>>>>> capabilities dictate what you "can do" - i.e. you can modify CAG's
>>> if you
>>>>>> have the cacheassignmentgroup-write capability
>>>>>> tenancy dictates what you "can do it to". in this case, if CAG's
>>> have a
>>>>>> tenantId and you have the cacheassignmentgroup-write capability,
>>> then you
>>>>>> can ONLY modify CAG's that fall in your tenancy scope.
>>>>>> 
>>>>>> although different, capabilities and tenancy work hand in hand to
>>> limit
>>>>>> what a user can do (permissions) and what they can do that to
>>> (scope).
>>>>>> 
>>>>>> because CAG's have an embedded tenantable resource (delivery
>>> services),
>>>>>> this makes it a bit trickier. not only should tenancy dictate which
>>> CAG's
>>>>>> can be modified by the user, it should also dictate how those CAG's
>>>>> should
>>>>>> be modified (i.e. which delivery services can be impacted by a
>>>>> modification)
>>>>>> 
>>>>>> jeremy
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Wed, May 15, 2019 at 3:13 PM Eric Friedrich -X (efriedri - TRITON
>>> UK
>>>>>> BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
>>>>>>  wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On May 15, 2019, at 4:16 PM, Fieck, Brennan <
>>> brennan_fi...@comcast.com
>>>>>> 
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> You could just set tenant permissions based on the owning tenant
>>> of the
>>>>>>> Delivery Service.
>>>>>>>> Should the child of a tenant be able to modify cache assignments of
>>>>> said
>

Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

Jeremy-
  Going back to your original example of the tree below. 

If DS baz is assigned to tenantId 2, then tenantIds 2.1, 2.1.1, 2.2 cannot 
modify baz, right?

If a CAG is created also with tenantId2, then I would expect the same behavior- 
only John as part of the 2 tenant can modify that CAG. 
Similarly, this CAG could only contain DS’ that belong to our are children of 
tenant 2. 

This seems to match existing behavior more closely?

—Eric

> On May 16, 2019, at 10:43 AM, Jeremy Mitchell  wrote:
> 
> no, capabilities are very different than tenancy.
> 
> capabilities dictate what you "can do" - i.e. you can modify CAG's if you
> have the cacheassignmentgroup-write capability
> tenancy dictates what you "can do it to". in this case, if CAG's have a
> tenantId and you have the cacheassignmentgroup-write capability, then you
> can ONLY modify CAG's that fall in your tenancy scope.
> 
> although different, capabilities and tenancy work hand in hand to limit
> what a user can do (permissions) and what they can do that to (scope).
> 
> because CAG's have an embedded tenantable resource (delivery services),
> this makes it a bit trickier. not only should tenancy dictate which CAG's
> can be modified by the user, it should also dictate how those CAG's should
> be modified (i.e. which delivery services can be impacted by a modification)
> 
> jeremy
> 
> 
> 
> On Wed, May 15, 2019 at 3:13 PM Eric Friedrich -X (efriedri - TRITON UK
> BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
>  wrote:
> 
>> 
>> 
>>> On May 15, 2019, at 4:16 PM, Fieck, Brennan 
>> wrote:
>>> 
>>> You could just set tenant permissions based on the owning tenant of the
>> Delivery Service.
>>> Should the child of a tenant be able to modify cache assignments of said
>> tenant's Delivery Services?
>>> I wouldn't think so.
>> EF> Isn’t this what the capabilities are for? If a user has
>> “cacheassignmentgroup-write” capability, then they can modify the
>> assignments for any delivery services in that tenant
>> 
>>> 
>>> From: Jeremy Mitchell 
>>> Sent: Wednesday, May 15, 2019 1:22 PM
>>> To: dev@trafficcontrol.apache.org
>>> Subject: [EXTERNAL] Re: [PROPOSAL] Cache Assignment Group REST API
>>> 
>>> example tenant tree:
>>> 
>>> root
>>> - 1 (foo)
>>> -- 1.1 (bar)
>>> - 2 (baz)
>>> -- 2.1 (bee)
>>> --- 2.1.1 (bang)
>>> -- 2.2 (bop)
>>> 
>>> the foo ds belongs to tenant 1, bar ds belongs to tenant 1.1, etc.
>>> 
>>> a CGA is created with tenantId=2 which means it can only have the baz,
>> bee,
>>> bang and bop ds's in it. John belongs to the 2 tenant and adds all 4
>> (baz,
>>> bee, bang, bop).
>>> 
>>> Sally belongs to the 2.1 tenant, and only sees [bee, bang] in the CGA and
>>> does an update with [], so it blows away bee (the ds in her tenant) and
>>> bang (plus any child tenant ds's)
>>> 
>>> Linda belongs to the 2.1.1 tenant, and sees [] in the CGA (because sally
>>> blew them all away) and does an update with [goo]. now the CAG has baz
>> and
>>> goo ds's.
>>> 
>>> so basically, a user can only modify the ds's that they can see based on
>>> their tenant (and subtenants). and a CAG can only have certain ds's in it
>>> based on it's tenant...
>>> 
>>> I "think" that would worksounds a bit complicated but i really feel
>>> like tenancy probably belongs on a CAG because of its relationship with
>>> ds's. plus, then it would be nice to create CAG's for tenants. for
>> example,
>>> create a trial tenant and some trial ds's and some trial users and they
>>> have no choice but to use the trial CAG that has 2 caches in it or
>>> something.
>>> 
>>> jeremy
>>> 
>>> On Wed, May 15, 2019 at 1:01 PM Eric Friedrich -X (efriedri - TRITON UK
>>> BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
>>>  wrote:
>>> 
>>>> What if the tenantId on the cacheassignmentgroup does not match the
>>>> tenantID on one of the included delivery services?
>>>> 
>>>> On May 15, 2019, at 2:55 PM, Jeremy Mitchell 
>>>> wrote:
>>>>> 
>>>>> I got an idea. If you made a cachegroupassignment a "tenantable"
>>>> resource,
>>>>> you could avoid the problem i mentioned above (having ds's hidden for
>>>>> tenancy reasons). so

Re: [EXTERNAL] [PROPOSAL] Cache Assignment Group REST API

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

> On May 15, 2019, at 4:16 PM, Fieck, Brennan  wrote:
> 
> You could just set tenant permissions based on the owning tenant of the 
> Delivery Service.
> Should the child of a tenant be able to modify cache assignments of said 
> tenant's Delivery Services?
> I wouldn't think so.
EF> Isn’t this what the capabilities are for? If a user has 
“cacheassignmentgroup-write” capability, then they can modify the assignments 
for any delivery services in that tenant

> 
> From: Jeremy Mitchell 
> Sent: Wednesday, May 15, 2019 1:22 PM
> To: dev@trafficcontrol.apache.org
> Subject: [EXTERNAL] Re: [PROPOSAL] Cache Assignment Group REST API
> 
> example tenant tree:
> 
> root
> - 1 (foo)
> -- 1.1 (bar)
> - 2 (baz)
> -- 2.1 (bee)
> --- 2.1.1 (bang)
> -- 2.2 (bop)
> 
> the foo ds belongs to tenant 1, bar ds belongs to tenant 1.1, etc.
> 
> a CGA is created with tenantId=2 which means it can only have the baz, bee,
> bang and bop ds's in it. John belongs to the 2 tenant and adds all 4 (baz,
> bee, bang, bop).
> 
> Sally belongs to the 2.1 tenant, and only sees [bee, bang] in the CGA and
> does an update with [], so it blows away bee (the ds in her tenant) and
> bang (plus any child tenant ds's)
> 
> Linda belongs to the 2.1.1 tenant, and sees [] in the CGA (because sally
> blew them all away) and does an update with [goo]. now the CAG has baz and
> goo ds's.
> 
> so basically, a user can only modify the ds's that they can see based on
> their tenant (and subtenants). and a CAG can only have certain ds's in it
> based on it's tenant...
> 
> I "think" that would worksounds a bit complicated but i really feel
> like tenancy probably belongs on a CAG because of its relationship with
> ds's. plus, then it would be nice to create CAG's for tenants. for example,
> create a trial tenant and some trial ds's and some trial users and they
> have no choice but to use the trial CAG that has 2 caches in it or
> something.
> 
> jeremy
> 
> On Wed, May 15, 2019 at 1:01 PM Eric Friedrich -X (efriedri - TRITON UK
> BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
>  wrote:
> 
>> What if the tenantId on the cacheassignmentgroup does not match the
>> tenantID on one of the included delivery services?
>> 
>> On May 15, 2019, at 2:55 PM, Jeremy Mitchell 
>> wrote:
>>> 
>>> I got an idea. If you made a cachegroupassignment a "tenantable"
>> resource,
>>> you could avoid the problem i mentioned above (having ds's hidden for
>>> tenancy reasons). so this instead:
>>> 
>>> {"id": 1,
>>> "name": "name1",
>>> "description": "description1",
>>> tenantId: 2,
>>> "cdnId": 1,
>>> "servers": [1,2,...n],
>>> "deliveryServices": [10, 20, 30, 35]
>>> "lastUpdated": "",
>>> }
>>> 
>>> this has a nice benefit as well. i.e. certain tenants (content providers)
>>> have access to certain cachegroupassignment configurations.
>>> 
>>> jeremy
>>> 
>>> 
>>> 
>>> 
>>> On Wed, May 15, 2019 at 12:43 PM Jeremy Mitchell 
>>> wrote:
>>> 
>>>> Just be careful that a GET /api/1.4/cacheassignmentgroups?id=4 doesn't
>>>> return a filtered list of delivery services because of tenancy
>>>> 
>>>> {"id": 1,
>>>> "name": "name1",
>>>> "description": "description1",
>>>> "cdnId": 1,
>>>> "servers": [1,2,...n],
>>>> "deliveryServices": [10, 20, 30], <-- maybe there are really 5 delivery
>>>> services but 2 of them (40 and 50) are hidden from you due to tenancy
>>>> "lastUpdated": "",
>>>> }
>>>> 
>>>> and a subsequent PUT with the same json (plus a new delivery service
>> that
>>>> is added):
>>>> 
>>>> {"id": 1,
>>>> "name": "name1",
>>>> "description": "description1",
>>>> "cdnId": 1,
>>>> "servers": [1,2,...n],
>>>> "deliveryServices": [10, 20, 30, 35]
>>>> "lastUpdated": "",
>>>> }
>>>> 
>>>> doesn't wipe out 40 and 50.
>>>> 
>>>> if you do this, it begs the question. how do you remove ALL ds
>> assignments
>>>> from a cache 

Re: [PROPOSAL] Cache Assignment Group REST API

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

What if the tenantId on the cacheassignmentgroup does not match the tenantID on 
one of the included delivery services? 

On May 15, 2019, at 2:55 PM, Jeremy Mitchell  wrote:
> 
> I got an idea. If you made a cachegroupassignment a "tenantable" resource,
> you could avoid the problem i mentioned above (having ds's hidden for
> tenancy reasons). so this instead:
> 
> {"id": 1,
> "name": "name1",
> "description": "description1",
> tenantId: 2,
> "cdnId": 1,
> "servers": [1,2,...n],
> "deliveryServices": [10, 20, 30, 35]
> "lastUpdated": "",
> }
> 
> this has a nice benefit as well. i.e. certain tenants (content providers)
> have access to certain cachegroupassignment configurations.
> 
> jeremy
> 
> 
> 
> 
> On Wed, May 15, 2019 at 12:43 PM Jeremy Mitchell 
> wrote:
> 
>> Just be careful that a GET /api/1.4/cacheassignmentgroups?id=4 doesn't
>> return a filtered list of delivery services because of tenancy
>> 
>> {"id": 1,
>> "name": "name1",
>> "description": "description1",
>> "cdnId": 1,
>> "servers": [1,2,...n],
>> "deliveryServices": [10, 20, 30], <-- maybe there are really 5 delivery
>> services but 2 of them (40 and 50) are hidden from you due to tenancy
>> "lastUpdated": "",
>> }
>> 
>> and a subsequent PUT with the same json (plus a new delivery service that
>> is added):
>> 
>> {"id": 1,
>> "name": "name1",
>> "description": "description1",
>> "cdnId": 1,
>> "servers": [1,2,...n],
>> "deliveryServices": [10, 20, 30, 35]
>> "lastUpdated": "",
>> }
>> 
>> doesn't wipe out 40 and 50.
>> 
>> if you do this, it begs the question. how do you remove ALL ds assignments
>> from a cache assignment group?
>> 
>> also, how about
>> 
>> DELETE /api/1.4/cacheassignmentgroups?id=
>> 
>> instead of
>> 
>> DELETE /api/1.4/cacheassignmentgroups/{id}
>> 
>> jeremy
>> 
>> On Wed, May 15, 2019 at 12:22 PM Eric Friedrich -X (efriedri - TRITON UK
>> BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
>>  wrote:
>> 
>>> Feature description
>>> 
>>> 
>>> 
>>> Mail Thread:
>>> https://lists.apache.org/thread.html/35ee49f4be1c30ff4a12c71e02897aee0e0d3d2f356640ab69ba247e@%3Cdev.trafficcontrol.apache.org%3E
>>> <
>>> https://lists.apache.org/thread.html/35ee49f4be1c30ff4a12c71e02897aee0e0d3d2f356640ab69ba247e@
>>> >
>>> 
>>> Github Issue: https://github.com/apache/trafficcontrol/issues/3557
>>> 
>>> 
>>> API Proposal
>>> 
>>> 
>>> 
>>> POST,GET /api/1.4/cacheassignmentgroups/
>>> 
>>> PUT /api/1.4/cacheassignmentgroups/{id}
>>> 
>>> {"id": 1,
>>> "name": "name1",
>>> "description": "description1",
>>> "cdnId": 1,
>>> "servers": [1,2,...n],
>>> "deliveryServices": [10, 20, 30],
>>> "lastUpdated": "",
>>> }
>>> 
>>> 
>>> DELETE /api/1.4/cacheassignmentgroups/{id}
>>> 
>>>  -- No request body
>>> 
>>> This API is tenant-aware by delivery service.
>>> 
>>> Query Parameters (all optional)
>>> If multiple filter parameters are specified, they are AND'd together
>>> - id: Filter for a specific entry
>>> - servers: Filter all entries containing this server
>>> - deliveryService: Filter all entries containing this DS
>>> 
>>> - limit: Return maximum number of entries (default 20)
>>> - page: Return page n, with each page having limit number of entries
>>> (default 0)
>>> 
>>> Body Parameters:
>>> - id: Numeric identifier, automatically assigned on creation. [Read-Only,
>>> not allowed in PUT/POST]
>>> - name: Name of the cache assignment group
>>> - description Description of the cache assignment group
>>> - cdnId: ID of the CDN the cache assignment group belongs to
>>> - servers: List of server IDs.
>>> - deliveryServices: List of delivery service IDs. All caches in the
>>> servers list will be assigned to these delivery services
>>> - lastUpdated: Timestamp this cache assignment group was last updated.
>>> [Read-Only, not allowed in PUT/POST]
>>> 
>>> 

[PROPOSAL] Cache Assignment Group REST API

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

Feature description



Mail Thread: 
https://lists.apache.org/thread.html/35ee49f4be1c30ff4a12c71e02897aee0e0d3d2f356640ab69ba247e@%3Cdev.trafficcontrol.apache.org%3E

Github Issue: https://github.com/apache/trafficcontrol/issues/3557


API Proposal



POST,GET /api/1.4/cacheassignmentgroups/

PUT /api/1.4/cacheassignmentgroups/{id}

{"id": 1,
"name": "name1",
"description": "description1",
"cdnId": 1,
"servers": [1,2,...n],
"deliveryServices": [10, 20, 30],
"lastUpdated": "",
}


DELETE /api/1.4/cacheassignmentgroups/{id}

  -- No request body

This API is tenant-aware by delivery service.

Query Parameters (all optional)
If multiple filter parameters are specified, they are AND'd together
- id: Filter for a specific entry
- servers: Filter all entries containing this server
- deliveryService: Filter all entries containing this DS

- limit: Return maximum number of entries (default 20)
- page: Return page n, with each page having limit number of entries (default 0)

Body Parameters:
- id: Numeric identifier, automatically assigned on creation. [Read-Only, not 
allowed in PUT/POST]
- name: Name of the cache assignment group
- description Description of the cache assignment group
- cdnId: ID of the CDN the cache assignment group belongs to
- servers: List of server IDs.
- deliveryServices: List of delivery service IDs. All caches in the servers 
list will be assigned to these delivery services
- lastUpdated: Timestamp this cache assignment group was last updated. 
[Read-Only, not allowed in PUT/POST]

Re: New Feature: Cache Assignment Groups

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

bably merge soon)
> https://github.com/apache/trafficcontrol/pull/2991 (still marked WIP,
> not sure how far along it is)
> We should probably prioritize getting those in so that your changes
> could be made in what we have there so far.
EF> I’ve got a perl implementation of this against TC2.2 that some coworkers 
wrote. 
I’d prefer to open source a Go implementation so this feature does not 
need to be rewritten immediately and does not hold up Rob’s changes. 

—Eric


> 
> - Rawlin
> 
> On Mon, May 13, 2019 at 6:25 AM Eric Friedrich -X (efriedri - TRITON
> UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
>  wrote:
>> 
>> As it stands, you can choose a subset of edge and mid caches, which is a big 
>> change from where we stand today.
>> 
>> I think it can lay the groundwork for n-tier hierarchies. Right now one of 
>> these groups can contain both edges and mids. We could extend the group to 
>> have a parent/child relationship with other groups (any group that has 
>> children I believe would need to contain only Mids).
>> 
>> 
>> —Eric
>> 
>> 
>> 
>> 
>>> On May 12, 2019, at 8:40 PM, Dave Neuman  wrote:
>>> 
>>> This is exciting, thanks Eric.
>>> Any idea how well this fits in with flexible cache groups?
>>> 
>>> On Thu, May 9, 2019 at 10:51 Eric Friedrich 
>>> wrote:
>>> 
>>>> GH Issue Link: https://github.com/apache/trafficcontrol/issues/3557
>>>> 
>>>> Background
>>>> --
>>>> Edge caches can currently be assigned to delivery services in three ways:
>>>> - By Server Profile
>>>> - By Cache Group
>>>> - By Individual Cache
>>>> 
>>>> The mids of the assigned edges are chosen based only on the parent and
>>>> secondary_parent cachegroup settings.
>>>> 
>>>> Use Cases
>>>> 
>>>> Create a more flexible method of assigning mid/edge caches to delivery
>>>> services. This allows users to solve the following problems:
>>>> - Dedicating a subset of caches to particular content providers/CDN
>>>> services
>>>> - Dedicating a subset of caches to particular content types (i.e. PDL or
>>>> HTTPS content)
>>>> - Specifying exactly which mid-caches are used for a particular delivery
>>>> service.
>>>> 
>>>> The existing "assign by profile" is not sufficient because servers can
>>>> only be in a single Profile. This proposal introduces a new Cache
>>>> Assignment Group, of which a server can be in more than one. Also, there is
>>>> no existing way to influence the selection of mid-caches on a DS-by-DS
>>>> basis.
>>>> 
>>>> The first two uses cases can be accomplished with individual server
>>>> assignments, but this is cumbersome when you need to match assignments of
>>>> lots of servers across lots of delivery services. Let's have the DB
>>>> relations manage these logical groupings for us instead.
>>>> 
>>>> 
>>>> Requirements
>>>> 
>>>> 1) Create Cache Assignment Group in Traffic Ops with a name and description
>>>> 2) Allow many to many association of caches to cache assignment groups
>>>> 3) Allow many to many association of cache assignment groups to Delivery
>>>> Services.
>>>> 4) Consider the union of individual cache assignments (in
>>>> deliveryservice_server table) and CAG assignments (in new
>>>> deliveryservice_assignmentgroup table) when creating remap.config and
>>>> CRConfig.json
>>>> 
>>>> 5) Mid-Cache Override: If a mid cache is present in a cache assignment
>>>> group, edge caches in that cache assignment group will disregard the parent
>>>> and secondary_parent hierarchy. Instead those edges will use only the mids
>>>> from that cache assignment group. (No secondary parent support yet). If no
>>>> mid is present, edge caches will continue to use parent and
>>>> secondary_parent as today.
>>>> 
>>>> 
>>>> Proposed API changes
>>>> 
>>>> 
>>>> /api/1.4/cacheassignmentgroups
>>>> Methods: POST, GET (Get all)
>>>> /api/1.4/cacheassignmentgroups/:id
>>>> Methods: PUT, GET, DELETE
>>>> 
>>>> Structure:
>>>>   FieldTypeMethod Present
>>>>   namestringPOST, PUT, GET
&

Re: New Feature: Cache Assignment Groups

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

As it stands, you can choose a subset of edge and mid caches, which is a big 
change from where we stand today. 

I think it can lay the groundwork for n-tier hierarchies. Right now one of 
these groups can contain both edges and mids. We could extend the group to have 
a parent/child relationship with other groups (any group that has children I 
believe would need to contain only Mids). 


—Eric




> On May 12, 2019, at 8:40 PM, Dave Neuman  wrote:
> 
> This is exciting, thanks Eric.
> Any idea how well this fits in with flexible cache groups?
> 
> On Thu, May 9, 2019 at 10:51 Eric Friedrich 
> wrote:
> 
>> GH Issue Link: https://github.com/apache/trafficcontrol/issues/3557
>> 
>> Background
>> --
>> Edge caches can currently be assigned to delivery services in three ways:
>> - By Server Profile
>> - By Cache Group
>> - By Individual Cache
>> 
>> The mids of the assigned edges are chosen based only on the parent and
>> secondary_parent cachegroup settings.
>> 
>> Use Cases
>> 
>> Create a more flexible method of assigning mid/edge caches to delivery
>> services. This allows users to solve the following problems:
>> - Dedicating a subset of caches to particular content providers/CDN
>> services
>> - Dedicating a subset of caches to particular content types (i.e. PDL or
>> HTTPS content)
>> - Specifying exactly which mid-caches are used for a particular delivery
>> service.
>> 
>> The existing "assign by profile" is not sufficient because servers can
>> only be in a single Profile. This proposal introduces a new Cache
>> Assignment Group, of which a server can be in more than one. Also, there is
>> no existing way to influence the selection of mid-caches on a DS-by-DS
>> basis.
>> 
>> The first two uses cases can be accomplished with individual server
>> assignments, but this is cumbersome when you need to match assignments of
>> lots of servers across lots of delivery services. Let's have the DB
>> relations manage these logical groupings for us instead.
>> 
>> 
>> Requirements
>> 
>> 1) Create Cache Assignment Group in Traffic Ops with a name and description
>> 2) Allow many to many association of caches to cache assignment groups
>> 3) Allow many to many association of cache assignment groups to Delivery
>> Services.
>> 4) Consider the union of individual cache assignments (in
>> deliveryservice_server table) and CAG assignments (in new
>> deliveryservice_assignmentgroup table) when creating remap.config and
>> CRConfig.json
>> 
>> 5) Mid-Cache Override: If a mid cache is present in a cache assignment
>> group, edge caches in that cache assignment group will disregard the parent
>> and secondary_parent hierarchy. Instead those edges will use only the mids
>> from that cache assignment group. (No secondary parent support yet). If no
>> mid is present, edge caches will continue to use parent and
>> secondary_parent as today.
>> 
>> 
>> Proposed API changes
>> 
>> 
>> /api/1.4/cacheassignmentgroups
>>  Methods: POST, GET (Get all)
>> /api/1.4/cacheassignmentgroups/:id
>>  Methods: PUT, GET, DELETE
>> 
>>  Structure:
>>FieldTypeMethod Present
>>namestringPOST, PUT, GET
>>descriptionstringPOST, PUT, GET
>>cdnIdintPOST, PUT, GET
>>idintGET
>>lastUpdated stringGET
>> 
>> 
>> /api/1.4/cacheassignmentgroupserver
>>  Methods: GET, POST
>> 
>>  Structure:
>>FieldTypeDescription
>>cagIdintID of assignment group to which servers will be assigned
>>replaceboolTrue to overwrite existing assignments, false to only add
>> additional
>>serversint array  Server ids to assigned into the Cache Assignment
>> Group
>> 
>> 
>> /api/1.4/deliveryservicecacheassignmentgroup
>>  Methods: GET, POST
>> 
>>  Structure:
>>FieldType  Description
>>dsIdint  ID of delivery service to which cache assignment groups will
>> be assigned
>>replacebool  True to overwrite existing assignments, false to only add
>> additional
>>cacheassignmentgroups int array CAG IDs to assign into the Delivery
>> Service
>> 
>> 
>> Proposed DB Schema Changes
>> -
>> New Table: cacheassignmentgroup
>> 
>> FieldTypeNotes
>> idbigintPrimary Key, auto increment
>> nametextUnique
>> descriptiontext
>> cdn_idbigintForeign Key to cdn(id), trigger to pick one as default
>

Re: [EXTERNAL] Traffic Ops API versioning issues

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

> why support one major version back though? Can Traffic Router 3.0 be expected 
> to work properly with Traffic Ops 2.0? 
Actually Yes, it must otherwise we cannot upgrade a running system in 
production without downtime. Upgrades do not occur instantaneously and should 
actually be spread out over days or weeks to confirm there are no regressions. 

The same goes for the Traffic Ops API. A “real” software product cannot expect 
users to change their tooling simultaneous with software upgrades. 

—Eric


> On Apr 22, 2019, at 1:11 PM, Fieck, Brennan  wrote:
> 
>> 2. Always release new TC major versions, never do TC minor version releases.
> 
> why? I'm not clear why tying the API version to the TC version would mean we 
> can't do minor releases. The TO API should - in my opinion - just be treated 
> as a part of Traffic Control. You don't make breaking changes within a single 
> major release etc. Having a separate versioning scheme seems nonsensical to 
> me. 
> 
> 1a. One major rev back at the project level for all code/api/etc
> 
> why support one major version back though? Can Traffic Router 3.0 be expected 
> to work properly with Traffic Ops 2.0? I wouldn't think so, so I also 
> wouldn't expect a script written for version X of TC to work with version 
> X-1. We have changelogs and stable releases for precisely this reason. We 
> don't support mixing and matching TC component versions, so why would this 
> specific part get snowflake treatment?
> 
> And consider this: we won't release breaking API changes as part of a minor 
> TC release anyway, so if the major TC version determines the API version 
> anyway, then what are we gaining by forcing clients to specify a full version 
> in every request path?
> 
>> 5. OPTIONAL: There should be an API route to get the exact TC version
> 
> It's getting a little off-track to say so, but I still say this should be 
> handled in the `Server` header with the format `Traffic Ops/{{version}}`. 
> That way, any request that fails will immediately tell you what versions are 
> available, and you can use e.g. `/ping` at the beginning of your script to 
> easily determine the TO server version without us 
> adding/documenting/maintaining any new endpoint .
> 
> Tying the API version to the TC version does NOT mean we've abandoned 
> semantic versioning, just that we're not keeping track of the versioning of 
> two pieces of the same project seperately. IMO we shouldn't be afraid of 
> breaking clients with a major release - breaking changes are what major 
> releases are for. To support that, maybe we need to step up the pacing of 
> major releases, but I don't see that as a big problem either. Because of slow 
> release cadence, a lot of people are building things against master instead 
> of stable releases, which is part of the problem as I see it.
> 
> 
> From: Gray, Jonathan 
> Sent: Friday, April 19, 2019 8:37 PM
> To: dev@trafficcontrol.apache.org
> Subject: Re: [EXTERNAL] Re: Traffic Ops API versioning issues
> 
>> 2c. Reflection is a tradeoff between debuggability and copypasta bugs
>> 3. As I said in the issue, it's an unfortunate mistake we weren't specific 
>> enough but it was permissible and therefore falls under 1a.  I generally 
>> agree with strong typing, and even going forward we should increment our API 
>> major rev and just document that as a new requirement going forward in API 
>> changes.
>> 4. There's some benefit to versioning your objects, but not all languages 
>> are smart enough trivially to be able to support more than 1 option and it 
>> ignores the routes problem.  I think the better option is the adoption of 
>> something like json-schema/swagger that's implementation neutral, but still 
>> machine readable. Alternatively, I'm still a fan of GraphQL which has a 
>> strong and flexible schema built in as a first class concept and is 
>> impossible to have 2 and 3 while not requiring 3rd party validation tools.  
>> This results frequently in versionless APIs entirely because you're always 
>> required to ask for specifically what you want, are prepared to handle, and 
>> should you ask for something that no longer exists the server API is allowed 
>> to return an error (in this case you had a bigger problem anyway than API 
>> issues).
>> 5. I include it only because it highlights issues we have with deployments 
>> of ATC components when choosing to ignore 5d and have 3rd-party software 
>> dependent on our API that moves faster than our major release cycle.  There 
>> are several pro/con to the debate of one-version-to-rule-them-all and source 
>> control monorepo, for today this is just what we've chosen to do as a 
>> project.
> 
> A part of why I split 2 and 3 apart was because in cases of 2, as an API 
> consumer I'm choosing when to pay the technical debts that come with newer 
> releases.  In the case of 3, that debt is being forced to be paid with each 
> endpoint conversion not on 

Re: SSL cert validation via Traffic Ops API

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

+1
Sounds like a useful change, I know getting the right keys with the right certs 
can be difficult. 

Is it possible to have the TP checkbox match the “polarity” of the API query 
parameter? Rather than “Skip Validation”, can the checkbox say “Validate Certs” 
and be checked by default? 

Its easier to conceptualize a positive rather than a double negative (unchecked 
so dont skip validation => Do validation).

—ERic


> On Nov 28, 2018, at 6:45 PM, Rawlin Peters  wrote:
> 
> Hey Traffic Controllers,
> 
> If you're running a recent release of master you may find that you
> currently cannot _add_ self-signed certificates using the API (and by
> extension TP). However, the API still allows generating self-signed
> certificates. So, if your self-signed certs are generated by the API,
> you probably won't have any issues with those right now. However, if
> you're generating your self-signed certs through some other means than
> the API (e.g. in order to add SANs to the cert), you may find that you
> cannot currently _add_ those self-signed certs via the API. This is
> because self-signed certs do not pass the new validation in the _add_
> API endpoint. Since this new validation is a bit of a breaking API
> change, I'm proposing the following:
> 
> 1. By default, the deliveryservices/sslkeys/add endpoint will NOT do
> any extra validation of the SSL cert being added. This is the old Perl
> behavior and has led to a lot of headaches due to it being very easy
> to add bad certs to a delivery service.
> 2. Add a new query parameter to this API (?validate=true) which when
> set to 'true' will actually perform the full validation of the
> certificate being added.
> 3. In Traffic Portal, add a checkbox next to the "Update Keys" button
> (which makes a request to the _add_ endpoint) that says "Skip
> certificate validation" or something. By default that checkbox will be
> unchecked which will add the '?validate=true' query parameter, meaning
> the certs will be validated. This would allow you to validate your
> certs in the API/Traffic Portal up to the point where you believe the
> only remaining issue is that they're self-signed. At that point you
> would check the box to "skip validation" to allow the addition of your
> self-signed certs.
> 
> We really need to add validation of SSL certificates to this API
> endpoint, but at the same time I don't want this to be a breaking API
> change or require too much mental overhead in the UI. This would allow
> us to get some cert validation by default in Traffic Portal but still
> be able to bypass the validation for self-signed certs when needed. If
> using the API directly, you wouldn't need to fix anything for
> self-signed certs since the validation will not be done unless the new
> query param is used.
> 
> Please +1 if you agree with the proposal as-is, -1 if you disagree or
> think the proposal needs fixing/adjusted (and please be clear on how I
> can change that to a +1), or just reply with a +/-0 if you don't care
> too strongly either way but have a different idea or some feedback to
> give.
> 
> - Rawlin

Re: [EXTERNAL] The future of the db/admin.pl script

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

Hey Rawlin-
  Thanks for doing more digging on the licensing aspect. I think we could find 
a way to make this work. The Apache FAQs are fuzzy for stuff like the boot 
loader exception, but would probably be ok. 
I dont think there is any issue including the python interpreter as PSF is an 
Apache friendly license.

I think Go sounds like a great idea for the db/admin.pl replacement script

—Eric






> On Nov 13, 2018, at 12:55 PM, Rawlin Peters  wrote:
> 
> replies inline
> 
> On Mon, Nov 12, 2018 at 4:59 PM Eric Friedrich -X (efriedri - TRITON
> UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
>  wrote:
>> Is pyyaml part of the batteries-included packages?
>> If not, we’ll need a way to distribute pyyaml.whl as part of the traffic ops 
>> RPM. (For those of us without permitted Internet access in our deployments, 
>> installing via pip means standing up our own private PyPi repo) at each of 
>> our customers- something I would really like to avoid.
>> 
>> I don't believe it is part of the standard library, but I do believe
>> we can figure out a way to keep it all self-contained to the traffic
>> ops RPM without requiring internet access at install time. I think
>> https://github.com/pyinstaller/pyinstaller is more than capable of
>> doing that on Linux/Mac/Windows. I just tried it out on my Python
>> version of db/admin.pl on my mac, and it created a self-contained
>> binary of 4.7MB size that seems to work just fine. It does still
>> depend on the external commands like `psql` et al. from the postgres
>> package, but that's no different from the current Perl version. Do you
>> think that would be sufficient?
>> 
>> I think pyinstaller also solves the python2 vs python3 availability
>> problem, since the interpreter is packaged into the resulting binary
>> itself.
>> EF> I think that would be OK. The licensing on that one is GPLv2 which is 
>> category X to include in source (we can distribute the boot loader though). 
>> Do you know of any alternatives with more compatible licensing?
> 
> It wouldn't need to be included in the source; it would just be used
> at build time. But there is also this information on its website
> (http://www.pyinstaller.org/license.html):
> 
>> PyInstaller is distributed under the GPL license (see the file COPYING.txt 
>> in the source code), with a special exception which allows to use 
>> PyInstaller to build and distribute non-free programs (including commercial 
>> ones). In other words, you have no restrictions in using PyInstaller as-is, 
>> but any kind of modifications to it will have to comply with the GPL 
>> license. See also our FAQ.
> 
> According to https://www.apache.org/legal/resolved.html#prohibited we
> are ok to use a GPL'ed tool during the build but cannot include the
> GPL'ed source code in our distribution.
> 
> There is also the question of actually including/distributing the
> python interpreter+standard library in our rpm (as part of the
> standalone binary produced by pyinstaller), but I believe that should
> be fine according to
> https://wiki.python.org/moin/PythonSoftwareFoundationLicenseFaq#Can_I_bundle_Python_with_my_non-open-source_application.3F.
> Here is is the relevant quote:
> 
>> Can I bundle Python with my non-open-source application?
>> Yes. Unlike some open source licenses, the PSF License allows Python to be 
>> included in non-open applications, either in unmodified or modified form.
> 
> That said, I'm starting to lean more towards Golang for replacing
> db/admin.pl, because we already use it heavily and it checks off a lot
> of the boxes. It might seem a bit like overkill for scripting stuff,
> but there is definitely growing usage of Go as a script-like
> executable (see

>  and
> https://github.com/erning/gorun) for things you'd normally just write
> in Bash or some other scripting language. There's just a lot more
> involved in using Python for this (pyinstaller, pipenv, 2 vs 3, etc)
> than Go.
> 
> - Rawlin

Re: [EXTERNAL] The future of the db/admin.pl script

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

On Nov 12, 2018, at 4:03 PM, Rawlin Peters 
mailto:rawlin.pet...@gmail.com>> wrote:

replies inline

On Mon, Nov 12, 2018 at 11:51 AM Eric Friedrich -X (efriedri - TRITON
UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
mailto:efrie...@cisco.com.invalid>> wrote:

Hey Rawlin-
 Both good points worth some serious thought.

On Nov 12, 2018, at 1:29 PM, Rawlin Peters 
mailto:rawlin.pet...@gmail.com>> wrote:

Eric,

I share your sentiment about being reluctant to introduce another
language as a dependency for Traffic Ops, but I wasn't able to find a
really good, easily-available utility for parsing yaml (a la `jq` for
json parsing) in a Bash script. Since the goose config is in yaml,
`db/admin.pl` uses a yaml package to parse the goose config into
variables which are then passed to the external `psql` et al.
commands. It is possible to parse yaml using sed, but the example I
found for doing that seemed really sketchy and fragile. So I figured
using a solid YAML-parsing library like PyYAML in Python would be a
safer bet while still allowing the use of a fully-featured programming
language rather than "Bash + ". It
would also allow us to potentially use a DB library to interface with
the DB directly in Python rather than requiring `psql` et al. and just
shelling out to those external commands (although I plan to continue
doing it that way for now).
EF> Yeah, the YAML parsing was the only thing I saw that was not a perfect fit 
for bash.
Given how concise that db.yaml file that is, I wouldn’t think twice about 
getting the open line via:
$ DB=“development” grep -A2 “$DB_NAME:” dbconf.yml | grep open | awk -F ":" 
'{print $2}'

No special tool needed.

I wish we could depend on the dbconf.yml file being a standard format
like that in order to just be able to grep it with context, but there
are no restrictions on empty lines or arbitrary comments in the yaml
syntax that would easily break the grep command. We shouldn't have to
enforce arbitrary formatting of a yaml config file just to remove the
need for a yaml parser.

EF> Yeah good point. I rarely touch the file so was not too worried about the 
syntax. But yeah, if people regularly change that file we should be more 
permissive with our parsing

Is pyyaml part of the batteries-included packages?
 If not, we’ll need a way to distribute pyyaml.whl as part of the traffic ops 
RPM. (For those of us without permitted Internet access in our deployments, 
installing via pip means standing up our own private PyPi repo) at each of our 
customers- something I would really like to avoid.

I don't believe it is part of the standard library, but I do believe
we can figure out a way to keep it all self-contained to the traffic
ops RPM without requiring internet access at install time. I think
https://github.com/pyinstaller/pyinstaller is more than capable of
doing that on Linux/Mac/Windows. I just tried it out on my Python
version of db/admin.pl on my mac, and it created a self-contained
binary of 4.7MB size that seems to work just fine. It does still
depend on the external commands like `psql` et al. from the postgres
package, but that's no different from the current Perl version. Do you
think that would be sufficient?

I think pyinstaller also solves the python2 vs python3 availability
problem, since the interpreter is packaged into the resulting binary
itself.
EF> I think that would be OK. The licensing on that one is GPLv2 which is 
category X to include in source (we can distribute the boot loader though). Do 
you know of any alternatives with more compatible licensing?




As a side note, it also paves the way for moving other scripts to
Python like WebDep.pm, which uses a Perl package that is virtually
impossible to install/get running on Mac because of Perl's broken SSL
on Mac, which would make it much easier to start as a new developer on
the project. I remember when I started working on Traffic Control, I
had to copy someone else's Perl `traffic_ops/app/local` directory who
had been on the project a long time and had actually gotten it to
build on Mac before it became unusable. Eliminating issues like that
by using a more popular and supportable language is a win in my book,
but right now I'm just focusing on `db/admin.pl` to allow for better
testability of the DB migration operations.
EF>All the web_deps stuff should go away with the rest of perl, right?

We really should be targeting RPMs that contain all dependencies (or can be 
resolved via yum/rpm), rather than asking our users to install stuff from the 
Internet at install time. Its fragile (packages disappear) and a bunch of TC 
users do not run in environments with access to the net.

I’m not opposed to Python, its probably my favorite language, certainly the one 
I'm most comfortable in. I just don't see a compelling need for it with these 
changes.
At some point soon we will need to rewrite perl scripts into something else 
(postinsta

Re: [EXTERNAL] The future of the db/admin.pl script

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

Hey Rawlin-
  Both good points worth some serious thought. 

> On Nov 12, 2018, at 1:29 PM, Rawlin Peters  wrote:
> 
> Eric,
> 
> I share your sentiment about being reluctant to introduce another
> language as a dependency for Traffic Ops, but I wasn't able to find a
> really good, easily-available utility for parsing yaml (a la `jq` for
> json parsing) in a Bash script. Since the goose config is in yaml,
> `db/admin.pl` uses a yaml package to parse the goose config into
> variables which are then passed to the external `psql` et al.
> commands. It is possible to parse yaml using sed, but the example I
> found for doing that seemed really sketchy and fragile. So I figured
> using a solid YAML-parsing library like PyYAML in Python would be a
> safer bet while still allowing the use of a fully-featured programming
> language rather than "Bash + ". It
> would also allow us to potentially use a DB library to interface with
> the DB directly in Python rather than requiring `psql` et al. and just
> shelling out to those external commands (although I plan to continue
> doing it that way for now).
EF> Yeah, the YAML parsing was the only thing I saw that was not a perfect fit 
for bash.
Given how concise that db.yaml file that is, I wouldn’t think twice about 
getting the open line via:
$ DB=“development” grep -A2 “$DB_NAME:” dbconf.yml | grep open | awk -F ":" 
'{print $2}'

No special tool needed. 

Is pyyaml part of the batteries-included packages? 
  If not, we’ll need a way to distribute pyyaml.whl as part of the traffic ops 
RPM. (For those of us without permitted Internet access in our deployments, 
installing via pip means standing up our own private PyPi repo) at each of our 
customers- something I would really like to avoid. 

> 
> As a side note, it also paves the way for moving other scripts to
> Python like WebDep.pm, which uses a Perl package that is virtually
> impossible to install/get running on Mac because of Perl's broken SSL
> on Mac, which would make it much easier to start as a new developer on
> the project. I remember when I started working on Traffic Control, I
> had to copy someone else's Perl `traffic_ops/app/local` directory who
> had been on the project a long time and had actually gotten it to
> build on Mac before it became unusable. Eliminating issues like that
> by using a more popular and supportable language is a win in my book,
> but right now I'm just focusing on `db/admin.pl` to allow for better
> testability of the DB migration operations.
EF>All the web_deps stuff should go away with the rest of perl, right? 

We really should be targeting RPMs that contain all dependencies (or can be 
resolved via yum/rpm), rather than asking our users to install stuff from the 
Internet at install time. Its fragile (packages disappear) and a bunch of TC 
users do not run in environments with access to the net. 

I’m not opposed to Python, its probably my favorite language, certainly the one 
I'm most comfortable in. I just don't see a compelling need for it with these 
changes. 
At some point soon we will need to rewrite perl scripts into something else 
(postinstall, ORT, etc…). We should closely consider our use of language for 
those as well- Go, Python, bash, etc… 

—Eric


> 
> - Rawlin
> 
> On Mon, Nov 12, 2018 at 6:46 AM Eric Friedrich -X (efriedri - TRITON
> UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
>  wrote:
>> 
>> I’m only slightly familiar with all the different options for db/admin.pl.
>> 
>> I’m a big fan of Python, but reluctant to introduce another language into TC 
>> without a strong reason.
>> 
>> Once the reverse_schema option is removed, what would be the main purposes 
>> of the script?
>> 
>> Looks like this is something that could be easily converted to bash without 
>> need for another dependency.
>> 
>> —Eric
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>> On Nov 10, 2018, at 1:44 PM, Dan Kirkwood  wrote:
>>> 
>>> +1 on rewriting admin.pl -- Python seems a reasonable choice, esp since we
>>> seem to be gaining a lot of Python expertise recently.
>>> 
>>> -1 on 2.x compatibility -- writing something new with compatibility for 2
>>> major versions makes no sense to me.  It limits the features and libraries
>>> that can be used and potentially doubles the amount of testing required.
>>> 
>>> 
>>> 
>>> On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman  wrote:
>>> 
>>>> +1, seems reasonable.  I don’t really have an opinion on python 2.x
>>>> compatibility, but whatever makes the most sense for the amount of work is
>>>> what I would prefer.
>>>> 
>>

Re: [EXTERNAL] The future of the db/admin.pl script

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

I’m only slightly familiar with all the different options for db/admin.pl. 

I’m a big fan of Python, but reluctant to introduce another language into TC 
without a strong reason. 

Once the reverse_schema option is removed, what would be the main purposes of 
the script? 

Looks like this is something that could be easily converted to bash without 
need for another dependency. 

—Eric







> On Nov 10, 2018, at 1:44 PM, Dan Kirkwood  wrote:
> 
> +1 on rewriting admin.pl -- Python seems a reasonable choice, esp since we
> seem to be gaining a lot of Python expertise recently.
> 
> -1 on 2.x compatibility -- writing something new with compatibility for 2
> major versions makes no sense to me.  It limits the features and libraries
> that can be used and potentially doubles the amount of testing required.
> 
> 
> 
> On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman  wrote:
> 
>> +1, seems reasonable.  I don’t really have an opinion on python 2.x
>> compatibility, but whatever makes the most sense for the amount of work is
>> what I would prefer.
>> 
>> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan 
>> wrote:
>> 
>>> +1 There is already precedence in the repo for python for other purposes.
>>> The only caveat I would include is to be sure you include backward
>>> compatibility for python 2.x for the next year or so until it goes EOL.
>>> 
>>> Jonathan G
>>> 
>>> On 11/9/18, 5:23 PM, "Rawlin Peters"  wrote:
>>> 
>>>Hey TC devs,
>>> 
>>>As we eliminate the need for Traffic Ops Perl, it will no longer
>>>really make sense for the db/admin.pl script to be Perl as it is
>>>today. This is because it depends on the Perl modules that are
>>>installed via Carton for running Traffic Ops Perl. However, it's
>>>mostly just a Perl wrapper around shell commands except for a YAML
>>>Perl module and the `reverse_schema` command which uses the DBIx Perl
>>>module to generate the ORM schema for Traffic Ops Perl (i.e. you've
>>>added a new DB table/column and need to get the ORM files updated to
>>>use it).
>>> 
>>>Without TO-Perl, there's no need for the `reverse_schema` command in
>>>db/admin.pl and its dependency on the Perl DBIx module. At that
>> point
>>>db/admin.pl is just a Perl script that parses some YAML and shells
>> out
>>>commands.
>>> 
>>>Part of the problem with TO-Perl is that there are a bunch of random
>>>non-API Perl scripts that basically assume all the modules in the
>>>cpanfile are installed in the environment, even though a lot of those
>>>dependencies are probably only required for the Perl TO API or UI. So
>>>unless we go through all those Perl dependencies to eliminate
>>>everything we don't really need once the Perl TO API and UI are
>>>completely removed, we'll continue to have a handful of Perl scripts
>>>like db/admin.pl that still require downloading and installing the
>>>full set of TO Perl dependencies. On fresh installs, running Carton
>> to
>>>install these dependencies can take nearly half an hour.
>>> 
>>>I'm working on adding tests for the DB upgrade/downgrade process
>> which
>>>I'd like to run automatically on PR submissions, but it really only
>>>needs the db/admin.pl script out of TO which assumes the entire set
>> of
>>>Perl TO dependencies even though it mostly just shells out to `goose
>>>up` and `goose down`. I'd like the test to emulate an actual TO
>>>environment as closely as possible to match what would actually
>> happen
>>>in a production DB upgrade/downgrade. Right now I'm reusing the
>>>Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd like
>>>to port db/admin.pl into Python to give it its own set of
>> dependencies
>>>(just a YAML package) and not require the full set of TO Perl
>>>dependencies. Then I can spin up a much lighter-weight container with
>>>just the TO Python packages rather than setting up a separate
>> cpanfile
>>>with just the Perl packages needed for db/admin.pl.
>>> 
>>>+1/-1 for adding Python as a dependency of Traffic Ops for porting
>>>scripts like db/admin.pl?
>>> 
>>>-Rawlin
>>> 
>>> 
>>> 
>> 

Re: [VOTE] Release Apache Traffic Control 3.0.0-RC2

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

+1

Release package looks good and it builds for me. 

Haven’t done any install/functional testing on it

—Eric

> On Nov 3, 2018, at 7:28 PM, Gelinas, Derek  wrote:
> 
> Hello All,
> 
> I've prepared a release for v3.0.0-RC2
> 
> The vote is open for at least 72 hours and passes if a majority of at least 3 
> +1 PPMC votes are cast.
> 
> [ ] +1 Approve the release
> 
> [ ] -1 Do not release this package because ...
> 
> Changes since 2.2
> https://github.com/apache/trafficcontrol/compare/2.2.x...RELEASE-3.0.0-RC2
> 
> 
> 
> This corresponds to git:
> Hash: 93ebb06e3fa78d1e5a55887010ba7b189ccc7085
> Tag: RELEASE-3.0.0-RC2
> 
> Which can be verified with the following: git tag -v RELEASE-3.0.0-RC2
> 
> 
> 
> My code signing key is available here:
> http://keys.gnupg.net/pks/lookup?search=0x05BE71D9=vindex
> 
> Make sure you refresh from a key server to get all relevant signatures.
> 
> The source .tgz file, pgp signature (.asc signed with my key from
> above), and sha512 checksums are provided here:
> 
> https://dist.apache.org/repos/dist/dev/trafficcontrol/3.0.0/RC2/
> 
> 
> 
> Thanks!
> 
> Derek

Re: [EXTERNAL] [VOTE] Release Apache Traffic Control 3.0.0-RC1

[Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)]

I had the following build failures using ./pkg: grove_build,  grovetccfg_build, 
 weasel

For grove, this was the error I received:
$ cat build-grove.log
git is /usr/bin/git
rpmbuild is /usr/bin/rpmbuild
Build environment has been verified.
==
WORKSPACE: /tmp/trafficcontrol
BUILD_NUMBER: 9570.62bbcd96
RHEL_VERSION: el7
TC_VERSION: 3.0.0
--
-  Building grove ...
fatal: Not a git repository (or any of the parent directories): .git
Cannot find repository root.
grove failed:
The following subdirectories had errors:
   grove


—Eric



> On Nov 1, 2018, at 11:05 AM, Dave Neuman  wrote:
> 
> It's a "minimum" of 72 hours.  We need at least 3 +1s to pass.
> I have not had time to test either :)
> 
> Thanks,
> Dave
> 
> On Thu, Nov 1, 2018 at 8:34 AM Gelinas, Derek 
> wrote:
> 
>> That works for me.
>> 
>> On 11/1/18, 10:27 AM, "Hank Beatty"  wrote:
>> 
>>Hello Derek,
>> 
>>Can I get one more day? I've tested most of the components. Still
>>working on Traffic Ops upgrade.
>> 
>>Thanks,
>>Hank
>> 
>>On 10/30/2018 01:36 PM, Gelinas, Derek wrote:
>>> Hello All,
>>> 
>>> I've prepared a release for v3.0.0-RC1
>>> 
>>> The vote is open for at least 72 hours and passes if a majority of
>> at least 3 +1 PPMC votes are cast.
>>> 
>>> [ ] +1 Approve the release
>>> 
>>> [ ] -1 Do not release this package because ...
>>> 
>>> Changes since 2.2
>>> 
>> https://github.com/apache/trafficcontrol/compare/2.2.x...RELEASE-3.0.0-RC1
>>> 
>>> 
>>> 
>>> This corresponds to git:
>>>  Hash: 62bbcd96d57143c462c0f5b0c2f132ed0369d712
>>>  Tag: RELEASE-3.0.0-RC1
>>> 
>>> Which can be verified with the following: git tag -v
>> RELEASE-3.0.0-RC1
>>> 
>>> 
>>> 
>>> My code signing key is available here:
>>> http://keys.gnupg.net/pks/lookup?search=0x05BE71D9=vindex
>>> 
>>> Make sure you refresh from a key server to get all relevant
>> signatures.
>>> 
>>> The source .tgz file, pgp signature (.asc signed with my key from
>>> above), and sha512 checksums are provided here:
>>> 
>>> https://dist.apache.org/repos/dist/dev/trafficcontrol/3.0.0/RC1/
>>> 
>>> 
>>> 
>>> Thanks!
>>> 
>>> 
>> 
>> 
>> 

Re: [EXTERNAL] Single-Layer CDN Support - PR 2904

[Eric Friedrich (efriedri)]

I’m good with the proposed PR. 

Does this increase the likelihood that someone might get their server 
parameters wrong (remap_required, reverse_proxy_enabled) etc if we don’t have a 
server type match up with a profile type?

Should we consider removing the EDGE/MID server type entirely, replacing it 
with just “cache”?


Also, not to get too off topic, but , we’ve got some code not yet open sourced 
(but willing to/eventually will publish) that will handle basic per-DS parent 
hierarchies. 
You can assign individual mids or create custom groups of mids, called device 
groups, to specific delivery services. Still only 2 tiers, but it lets us break 
free of the strict cache group tree. 

We could use this as a starting point when looking at the feature Rawlin 
mentions below. 


—Eric


> On Oct 3, 2018, at 10:57 PM, Dave Neuman  wrote:
> 
> +1 on this PR. Long overdue in my opinion.
> 
> On Wed, Oct 3, 2018 at 14:50 Rawlin Peters  wrote:
> 
>> I'm generally +1 on NOT keying off the server type (i.e. EDGE vs MID)
>> for ATS configs wherever we can get away with it. WRT parent.config,
>> I'd think as a server I'd really only need to know whether or not I
>> had parents, what kind of parents they are (origin vs proxy), and any
>> extra attributes associated with that relationship (round_robin,
>> go_direct, parent_retry). I shouldn't really care whether or not I'm
>> an EDGE, a MID, or a FOO.
>> 
>> I'm trying to look at this from the perspective of having per-DS
>> parent hierarchies in the future, and I think this should play nicely
>> with that idea.
>> 
>> - Rawlin
>> On Wed, Oct 3, 2018 at 12:25 PM Gelinas, Derek
>>  wrote:
>>> 
>>> As it stands now, parent.config could sort of handle that.  If a host
>> has no parents or has an origin, it will set it up like a mid.  If not,
>> it's going to get the edge cache treatment and parents will be listed for
>> each DS based on the parent cachegroups assigned.
>>> 
>>> That said, remap.config still functions on "I'm an edge, I'm a mid." So
>> there's still work to be done to get us there 100%.
>>> 
>>> FYI I just tested this against prod database and across the board it did
>> exactly as expected.  In a standard two-tier environment the parent.configs
>> were identical to the normal method, and in a single-tier environment the
>> parent.config went from incorrect and unable to handle MSO to correct.
>> This was tested on all edge types, mids, and single tier mids across all
>> our CDNs.  The only differences were in header timestamps unless otherwise
>> expected.
>>> 
>>> Derek
>>> 
>>> On 10/3/18, 2:14 PM, "Robert Butts"  wrote:
>>> 
>>>Just out of curiosity, how difficult would it be to similarly
>> support an
>>>arbitrary number of tiers, by similarly checking if the parent
>> _isn't_ an
>>>origin and doing the reverse? (I'm aware of the hackish and perilous
>> ways
>>>TC can go edge-to-edge, I'm wondering about formalized and safe
>> support.)
>>> 
>>>On Wed, Oct 3, 2018 at 11:54 AM Steve Malenfant <
>> smalenf...@gmail.com>
>>>wrote:
>>> 
 I haven't tried but I'm +1 for this.
 
 On Wed, Oct 3, 2018 at 12:59 PM Gelinas, Derek <
>> derek_geli...@comcast.com>
 wrote:
 
> https://github.com/apache/trafficcontrol/pull/2904
> 
> On 10/3/18, 12:25 PM, "Dewayne Richardson" 
>> wrote:
> 
>Stating the obvious for posterity, but what is the PR link?
> 
> 
>-Dew
> 
>On Wed, Oct 3, 2018 at 8:47 AM Gelinas, Derek <
> derek_geli...@comcast.com>
>wrote:
> 
>> All,
>> 
>> I’ve put a PR together to add support for single-layer
>> CDNs.
> Currently,
>> edge and mid cache roles are basically hardwired into the
>> code, so
> that
>> when a parent.config file is generated, the rules for
>> edges and
 mids
> are
>> always the same.  This is fine if you are only using edges
>> and
 mids,
> but
>> not so much if you’ve got edges only.  MSO is completely
 unsupported
> in a
>> config like this, and the parent.config looks like an edge
>> config,
> with
>> each remap listed without parents.  This is incorrect.
>> 
>> What my PR does is actually very simple.  Rather than look
>> at the
> cache
>> type at the various stages of the parent config
>> generation, we look
> at the
>> parent cachegroup types.  If both parent cachegroups are
>> either ORG
> type or
>> unassigned, then $is_top_level is set to 1 (default is 0)
>> and the
> variable
>> is passed to the various subroutines that
>> parent_dot_config uses to
>> generate data.  These subroutines have been exclusive to
> parent_dot_config
>> since the scope change a while back, so this change does
>> not affect
> any
>> other subs.  From there, all decisions about generating
> configuration data
>> are based on whether or not the cache is a top level cache.
>> 
>> With this 

Re: Traffic Ops Extensions in Go

[Eric Friedrich (efriedri)]

I think some documentation/definitions would be great here as well. 

Whats a TO extension vs hook vs data? Are there any examples of how they could 
be used?

—Eric


> On Oct 1, 2018, at 1:44 PM, Rawlin Peters  wrote:
> 
> Can we get a concise README.md on how to create a custom TO plugin
> that also links to the examples you've included in the PR? Also as new
> hooks are added, I think we'll need documentation for every hook to
> describe where in the request flow the hooks are called and the data
> available to each hook.
> 
> Maybe since we're discussing a microservice plugin, you could lay out
> the basic high-level steps required to implement that in this plugin
> framework (not necessarily down to the nitty gritty details).
> 
> Also, there's the question of the plugin interface versioning. How
> will we be versioning the plugin interface with guarantees to not
> break any plugins outside of our repo? Should we start out by tagging
> this plugin framework as experimental so that we don't really provide
> any compatibility guarantees while we're still working out the kinks?
> 
> - Rawlin
> On Mon, Oct 1, 2018 at 10:38 AM Robert Butts  wrote:
>> 
>> So, as we move Traffic Ops from Perl to Go, we need a way to write
>> extensions in Go, like we have in Perl.
>> 
>> I wrote a plugin system for Go, modeled after the plugin frameworks in
>> Grove and Traffic Monitor, PR is here:
>> 
>> https://github.com/apache/trafficcontrol/pull/2513
>> 
>> It's by no means complete, just like with Grove, we'll have to add
>> additional hooks and data as we find the need. But it works, it has basic
>> hooks, and includes some sample plugins.
>> 
>> There was a question about microservices, if someone wants to write
>> extensions to their Traffic Ops as separate services, instead of built into
>> the same app. This framework allows that, it's easy to make a plugin that
>> calls out to another service when an endpoint is requested.
>> 
>> Does anyone object to this PR? Could I get some +1's/-1's so we can get a
>> feel for whether people are OK with this PR being merged, and using this
>> plugin system for Traffic Ops going forward?
>> 
>> Thanks,

Re: Making parent.config's go_direct directive configurable via Delivery service

[Eric Friedrich (efriedri)]

types wouldn’t help us here, because we 
need a dynamic recovery that activates on massive mid-failure. We can’t have a 
system outage while waiting for operators to triage the problem, manually make 
a DS change and then push that change out to 1000 caches.   

—Eric


> 
> - Rawlin
> 
> On Tue, Jun 26, 2018 at 8:34 PM Eric Friedrich (efriedri)
>  wrote:
>> 
>> Hey Rawlin-
>>  Remember that the proposed feature does not always bypass the mids- we 
>> already have that with the regional DS types. This is just a failure 
>> recovery behavior.
>> 
>>  Conflicts with the DS type is only a small part of the major UX problem we 
>> have today relating to Origin FQDNs. Ways we can break today:
>>   - Configure the same Origin server base as both live and vod on the same 
>> edge
>>   - configure the same origin server in different delivery services with 
>> conflicting MSO policies
>>   - configure the same origin in different delivery services with different 
>> go_direct policies (new by the proposed change, regardless of configuring as 
>> a DS type or a true/false)
>> 
>> I don’t think its fair to ask this small feature to be the one to start 
>> fixing these long pre-existing problems without a bigger conversation around 
>> how they should be fixed (unique origins per-DS perhaps? But that comes with 
>> its own set of problems!)
>> 
>> 
>> I’m opposed to new DS types for the following reasons
>> 
>> 1) Proliferation of DS types. The list is already pretty long and confusing 
>> and adding this feature as DS types would add another 4 to that list, when a 
>> simple True/False dropdown would suffice
>> 
>> 2) Purpose of the DS types- I’ve always thought of the DS type as something 
>> that defines a major property of the Delivery Service - Live vs VOD, HTTP vs 
>> DNS. This is a minor feature which is not active 99% of the time and for the 
>> most part should not influence behavior except in an extreme failure 
>> conditions (I.e. all parent and secondary parent mids of an edge cache)
>> 
>> 3) Ability to change the go_direct setting. DS types are fixed today and 
>> cannot be modified for a delivery service. It would be nice if customers had 
>> the ability to turn this feature on and off without needing to delete and 
>> recreate the entire delivery service
>> 
>> 4) Consistency with existing features. I think this feature lines up pretty 
>> well against MSO as far as its behavior and I don’t think that we should 
>> convert MSO over to a DS type.
>> 
>> —Eric
>> 
>>> On Jun 26, 2018, at 5:04 PM, Rawlin Peters  wrote:
>>> 
>>> What makes having more types more confusing? It fits into the current
>>> "DS type determines whether or not the mids are bypassed" paradigm
>>> that exists today. In my opinion, a new DS field that conflicts with
>>> your chosen type in certain cases is not very intuitive and leads to
>>> bad UX. Sure, having more DS types to choose from isn't exactly ideal
>>> either, but when the description of type "_BYPASS" is just "the
>>> same as  except the mid tier is bypassed when all mids are down"
>>> I think it would be pretty straightforward. In this case I think new
>>> DS types is the lesser of two evils.
>>> 
>>> - Rawlin
>>> 
>>> On Tue, Jun 26, 2018 at 11:42 AM Vijay Anand
>>>  wrote:
>>>> 
>>>> To me it looks like DS types with go_direct TRUE/False will add more
>>>> confusion compared to adding go_direct value as a DS configurable parameter
>>>> defaulting it to false except for HTTP_LIVE DNS_LIVE and HTTP_NO_CACHE.
>>>> 
>>>> 
>>>> 
>>>> Thanks,
>>>> Vijayanand S
>>>> 
>>>> On Thu, Jun 21, 2018 at 4:53 AM, Rawlin Peters 
>>>> wrote:
>>>> 
>>>>> I think adding new DS types for this makes sense because traditionally
>>>>> the DS type determines the value of go_direct as well as how content
>>>>> is cached (disk/ram/not cached at all). If we make the field directly
>>>>> configurable on the Delivery Service, then we now have the complexity
>>>>> of prohibiting certain go_direct values in certain types of delivery
>>>>> services, and that adds to the cognitive load required to create/edit
>>>>> a delivery service.
>>>>> 
>>>>> For example, it's a bad experience for the user to find out through an
>>>>> API error that they

Re: Making parent.config's go_direct directive configurable via Delivery service

[Eric Friedrich (efriedri)]

Hey Rawlin-
  Remember that the proposed feature does not always bypass the mids- we 
already have that with the regional DS types. This is just a failure recovery 
behavior. 

  Conflicts with the DS type is only a small part of the major UX problem we 
have today relating to Origin FQDNs. Ways we can break today: 
   - Configure the same Origin server base as both live and vod on the same edge
   - configure the same origin server in different delivery services with 
conflicting MSO policies
   - configure the same origin in different delivery services with different 
go_direct policies (new by the proposed change, regardless of configuring as a 
DS type or a true/false)

I don’t think its fair to ask this small feature to be the one to start fixing 
these long pre-existing problems without a bigger conversation around how they 
should be fixed (unique origins per-DS perhaps? But that comes with its own set 
of problems!)
  

I’m opposed to new DS types for the following reasons

1) Proliferation of DS types. The list is already pretty long and confusing and 
adding this feature as DS types would add another 4 to that list, when a simple 
True/False dropdown would suffice

2) Purpose of the DS types- I’ve always thought of the DS type as something 
that defines a major property of the Delivery Service - Live vs VOD, HTTP vs 
DNS. This is a minor feature which is not active 99% of the time and for the 
most part should not influence behavior except in an extreme failure conditions 
(I.e. all parent and secondary parent mids of an edge cache)

3) Ability to change the go_direct setting. DS types are fixed today and cannot 
be modified for a delivery service. It would be nice if customers had the 
ability to turn this feature on and off without needing to delete and recreate 
the entire delivery service

4) Consistency with existing features. I think this feature lines up pretty 
well against MSO as far as its behavior and I don’t think that we should 
convert MSO over to a DS type. 

—Eric

> On Jun 26, 2018, at 5:04 PM, Rawlin Peters  wrote:
> 
> What makes having more types more confusing? It fits into the current
> "DS type determines whether or not the mids are bypassed" paradigm
> that exists today. In my opinion, a new DS field that conflicts with
> your chosen type in certain cases is not very intuitive and leads to
> bad UX. Sure, having more DS types to choose from isn't exactly ideal
> either, but when the description of type "_BYPASS" is just "the
> same as  except the mid tier is bypassed when all mids are down"
> I think it would be pretty straightforward. In this case I think new
> DS types is the lesser of two evils.
> 
> - Rawlin
> 
> On Tue, Jun 26, 2018 at 11:42 AM Vijay Anand
>  wrote:
>> 
>> To me it looks like DS types with go_direct TRUE/False will add more
>> confusion compared to adding go_direct value as a DS configurable parameter
>> defaulting it to false except for HTTP_LIVE DNS_LIVE and HTTP_NO_CACHE.
>> 
>> 
>> 
>> Thanks,
>> Vijayanand S
>> 
>> On Thu, Jun 21, 2018 at 4:53 AM, Rawlin Peters 
>> wrote:
>> 
>>> I think adding new DS types for this makes sense because traditionally
>>> the DS type determines the value of go_direct as well as how content
>>> is cached (disk/ram/not cached at all). If we make the field directly
>>> configurable on the Delivery Service, then we now have the complexity
>>> of prohibiting certain go_direct values in certain types of delivery
>>> services, and that adds to the cognitive load required to create/edit
>>> a delivery service.
>>> 
>>> For example, it's a bad experience for the user to find out through an
>>> API error that they are prohibited from setting go_direct to certain
>>> values in certain scenarios. If we just create new types for that
>>> instead, the user doesn't have to worry about conflicting settings in
>>> specific DS types and rather just has to choose the DS type that they
>>> want (where the proper/best settings are chosen for them under the
>>> hood).
>>> 
>>> Basically we'd need 4 new types (naming could be different):
>>> HTTP_LIVE_NATNL_BYPASS
>>> DNS_LIVE_NATNL_BYPASS
>>> HTTP_BYPASS
>>> DNS_BYPASS
>>> 
>>> The new types would mimic the matching original types except for
>>> setting go_direct=true, which would allow the edges to fetch from the
>>> origin when all the mids are down.
>>> 
>>> - Rawlin
>>> 
>>> On Wed, Jun 20, 2018 at 6:33 AM, Vijay Anand
>>>  wrote:
 All,
 
 The PR given below is a perl implemention for making parent.config's
 go_direct directive configurable via Delivery service
 https://github.com/apache/trafficcontrol/pull/2407
 
 This PR has been hosted to discuss about various approaches to make
 go_direct configurable. GoLang implementation will be added once we
 finalize on the approach.
 
 Background:
 ---
 Right now it is hard coded as false in parent.config for DS of types
>>> other
 than HTTP_LIVE, HTTP_NO_CACHE, DNS_LIVE 

ATS6 to ATS7 profile upgrades

[Eric Friedrich (efriedri)]

After spending an hour modifying one of our profiles from ATS6 to ATS7, and 
facing down modifying our other 10 profiles. We’ve also received some pretty 
sharp feedback from customers on how long it takes to manually make the profile 
configuration changes required for a new major ATS version. To make life easier 
for everyone, I wrote a quick utility to do the dirty work for us.

Given a JSON file from a TO export, this will use some predefined rules to 
convert the profile into what is needed for ATS7. This means changing 
logs_xml.config to logging.config and removing a bunch of obsoleted options, 
among other changes.  I’ve had some requests to build this into the TP/TO API. 
This script occasionally requires manual review, so is not a good fit for an 
automatic upgrade API yet. .

PR is up at https://github.com/apache/trafficcontrol/pull/2428

Hopefully others will find it useful for getting on ATS7 and we can reuse it 
when moving to future ATS versions as well.  Comments appreciated!

—Eric

Re: PR to improve build process

[Eric Friedrich (efriedri)]

Thanks for comments Chris, I agree with all of them.

Do you typically include astats in your traffic server tar balls? Based on 
whats in Github, its traditionally been shipped as a separate RPM. 

I agree build would be simpler with astats packaged in, especially because the 
TS version dependencies is some of the most complicated of my changes. 

If you get TSB open, is there a commitment to maintaining an up to date list of 
patches that should be applied for full ATC compatibility (today just URI 
signing/astats)?

—Eric

> On Jun 15, 2018, at 4:03 PM, Chris Lemmons  wrote:
> 
> I have so many thoughts on this topic. :)
> 
> We use a tool for composing the public ATS build on top of our list of
> backports (all of which are currently public, iirc). This is how we
> use stuff like URI Signing, which isn't available in ATS 7. I'm
> currently in the process of getting that opened up so that everyone
> can look at it (a couple weeks, probably, though), but the principle
> behind it isn't exceptionally complex.
> 
> What I'd like to see in the ATC ATS RPM is:
> - It's deterministically produced (basically, everything (as far as
> is reasonable) pegged to specific changesets/versions).
> - It contains every feature supported by ATC.
> - It builds as a `./pkg` target: `./pkg ats_build`.
> - It produces a single RPM that just works.
> Bonus points if it's relatively easy to add extra backports or private
> bug fixes.
> 
> The PR is close on determinism, though it encourages using a branch
> instead of a revision, which could cause trouble for folks with RPMs
> updating unexpectedly. It doesn't have backports for URI Signing and
> astats is in its own RPM, which I think puts more work on the people
> managing the CDN than is necessary. It's spot on for the ./pkg usage.
> 
> It's absolutely a step in the right direction, though, and I'd rather
> have something than nothing. I'm not totally enthused with this
> solution, but it's a start and we can work toward things going
> forward.
> 
> On Fri, Jun 15, 2018 at 12:48 PM, Zelkowitz, Evan
>  wrote:
>>I like the idea and looks good. One thing to think about though is in 
>> the trafficserver.spec file, if there are any good ways to dynamically do 
>> those config files includes at the end. Im no spec expert so Im not sure if 
>> there may be a better way but recently here I worked on getting our internal 
>> spec and build system to be able to do builds of ATS master and it looks 
>> like starting with 8/9 some more of those files will be going away and 
>> another new one introduced.  So those files change frequently across version 
>> numbers
>> 
>> On 6/15/18, 11:27 AM, "Eric Friedrich (efriedri)" 
>>  wrote:
>> 
>>I’ve opened a PR to produce Trafficserver and astats RPMs in the Traffic 
>> Control build process. Previously, new users would need to create their own 
>> build environments or ask for RPMS on Slack.
>> 
>>This PR will lower the barrier to entry for new users. For more involved 
>> users, theres now a set of scripts that will build inside a Dockerized 
>> Jenkins environment.
>> 
>>The changes rely on having all TS changes in a single git repository to 
>> form the RPM source tarball. I understand there are some users which have 
>> private tooling to build from multiple repos. I’m open to including this if 
>> possible. Until that tool gets open sourced, I think this is a step forward 
>> for TC builds.
>> 
>>https://github.com/apache/trafficcontrol/pull/2399
>> 
>>Any concerns with bringing this in?
>> 
>>—Eric
>> 
>> 
>> 

PR to improve build process

[Eric Friedrich (efriedri)]

I’ve opened a PR to produce Trafficserver and astats RPMs in the Traffic 
Control build process. Previously, new users would need to create their own 
build environments or ask for RPMS on Slack.

This PR will lower the barrier to entry for new users. For more involved users, 
theres now a set of scripts that will build inside a Dockerized Jenkins 
environment.

The changes rely on having all TS changes in a single git repository to form 
the RPM source tarball. I understand there are some users which have private 
tooling to build from multiple repos. I’m open to including this if possible. 
Until that tool gets open sourced, I think this is a step forward for TC builds.

https://github.com/apache/trafficcontrol/pull/2399

Any concerns with bringing this in?

—Eric

Re: [EXTERNAL] Add minimum caches field to deep routing logic

[Eric Friedrich (efriedri)]

Hey Sergey-
  I’m good with it in general. A few design considerations:

  Have you considered using deep cache CG TB provisioned instead of # caches in 
group? 
   Not all caches are created equal- 5 caches in CG1 maybe have half the 
storage of 2 caches in CG2.
  
  Does the state/availability of a cache impact TRs decision? 
   Is there a difference between a cache being REPORTED, REPORTED (and 
overloaded by TM), OFFLINE, ADMIN_DOWN? 

  —Eric
  

> On Jun 8, 2018, at 12:41 PM, Dremin, Sergey  wrote:
> 
> Unless people have more questions/opinions, can I assume we are happy with 
> this new routing/caching efficiency knob?
> I don’t see how this could negatively affect anything, besides complicating 
> DS configuration further.
> 
> On 6/6/18, 2:50 PM, "Dremin, Sergey"  wrote:
> 
>Hi Eric,
>The problem will materialize once we enable deep routing for non-live 
> delivery channels with larger libraries. Those delivery channels need to be 
> deep routed only to deep cache groups of sufficient size to make sure caching 
> efficiency remains at acceptable levels. Right now that is not possible. They 
> will be routed to any deep cache groups, even where there is a single server, 
> resulting in poor caching efficiency, and more traffic to the mids. 
>Does that explain the problem sufficiently? 
>~Sergey
> 
> 
>On 6/5/18, 7:10 PM, "Eric Friedrich (efriedri)"  wrote:
> 
>Hey Sergey-
>  What is the problem that you are experiencing here?
> 
> I understand what you are proposing, but am not sure why this is 
> needed. I would find some more details very useful
> 
>Thanks,
>Eric
> 
> 
> 
> 
>> On Jun 5, 2018, at 5:40 PM, Dremin, Sergey  wrote:
>> 
>> We should create an extension to the deep caching/deep routing field that 
>> will allow you to specify a minimum # of caches in a deep cache group for 
>> that deep cache group to be enabled for that DS.
>> 
>> 
>> 
>> Cache group size is proportional to caching efficiency. Having a 
>> configurable value for a minimum # of caches for a DS would allow to change 
>> the caching efficiency for that DS versus the routing efficiency (that comes 
>> with deep routing) for that DS.
>> 
>> 
>> 
>> Ideally this is something that is set based on current CDN performance.
>> 
>> 
>> 
>> 1.  A deep routing enabled DS with poor caching efficiency could use a 
>> bigger minimum size for a deep cache group.
>> 2.  A deep routing enabled DS with not sufficient deep routing usage, could 
>> use a smaller minimum size for a deep cache group.
>> 
>> 
>> 
>> Right now, we need to decide if we want to make this configurable or static. 
>> Since I think this will need to be configurable in the future anyway, so it 
>> a good idea to do it now.
>> 
>> 
>> 
>> This will require changes to TO/TP and traffic router.
>> 
>> Thoughts?
> 
> 
> 
> 
> 

Re: [VOTE] Update Traffic Control to Version 3.0

[Eric Friedrich (efriedri)]

+1 

With the caveat that we maintain an upgrade process when we start deprecating 
APIs between components. 

—Eric

> On Jun 7, 2018, at 2:11 PM, Dewayne Richardson  wrote:
> 
> +1, this will allow us to do a better job with the API's
> 
> On Tue, Jun 5, 2018 at 10:03 AM Jeff Elsloo  wrote:
> 
>> In light of the major changes introduced into Traffic Router with PR
>> 2331, I would like to call a vote to increase the major version of the
>> project to 3.0.0. Also note that this version would represent the
>> first release without official support for CentOS 6.x, though
>> components might still continue to work. From 3.0.0 forward, only
>> CentOS 7.x will be officially supported by the project.
>> 
>> This impacts versioning of all components, so if other components are
>> not ready to make major breaking changes during this transition, we
>> would need to do so under another major version change in the future.
>> 
>> I'd like to call this vote on Friday, so I will leave this open for 72
>> hours.
>> --
>> Thanks,
>> Jeff
>> 

Re: [EXTERNAL] go version used in build

[Eric Friedrich (efriedri)]

Should we look into creating a common “go builder” container base image that 
can be shared by all of the go components? 

It would be easier to keep this in a common location rather than having to keep 
a bunch of Dockerfiles in sync. 

—Eric


> On Jun 7, 2018, at 9:40 PM, Zelkowitz, Evan  
> wrote:
> 
> +1 on 3. I thought there were some changes made in 1.10 that have the 
> possibility of breaking things, i.e. I think it breaks some of Grove. They 
> may or may not affect TC but who knows what might happen in the future. So I 
> would think you would want the version pinned
> 
> From: Gray, Jonathan 
> Sent: Thursday, June 7, 2018 4:53 PM
> To: dev@trafficcontrol.apache.org
> Subject: Re: [EXTERNAL] go version used in build
> 
> I vote for option 3 since the version of go you compile with is no different 
> than the version of a library used in the build process.  By specifying what 
> version it shall be, we also know when we go to newer versions and have more 
> reproducible builds.
> 
> On 6/7/18, 3:27 PM, "Dan Kirkwood"  wrote:
> 
>Hey,  all..   I just investigated this issue (
>https://github.com/apache/incubator-trafficcontrol/issues/2380) and
>realized that the `go` version being used in building traffic_stats and
>traffic_monitor is different from that of traffic_ops due to the way it's
>installed during the rpmbuild phase.   In traffic_ops, a specific version
>(1.8.3) is downloaded using a script;  the others depend on yum without
>specifying a version (currently 1.9.4).
> 
>Should we worry about keeping these in line?  If so, should we:
> 
>1. change TO to install latest version from yum?
>2. change TS and TM to use the script?
>3. change all to install a specific version from yum?
> 
>Currently leaning toward #1,  but I can be convinced otherwise..
> 
>thanks..   Dan
> 
> 

Re: Add minimum caches field to deep routing logic

[Eric Friedrich (efriedri)]

Hey Sergey-
  What is the problem that you are experiencing here?

 I understand what you are proposing, but am not sure why this is needed. I 
would find some more details very useful

Thanks,
Eric




> On Jun 5, 2018, at 5:40 PM, Dremin, Sergey  wrote:
> 
> We should create an extension to the deep caching/deep routing field that 
> will allow you to specify a minimum # of caches in a deep cache group for 
> that deep cache group to be enabled for that DS.
> 
> 
> 
> Cache group size is proportional to caching efficiency. Having a configurable 
> value for a minimum # of caches for a DS would allow to change the caching 
> efficiency for that DS versus the routing efficiency (that comes with deep 
> routing) for that DS.
> 
> 
> 
> Ideally this is something that is set based on current CDN performance.
> 
> 
> 
>  1.  A deep routing enabled DS with poor caching efficiency could use a 
> bigger minimum size for a deep cache group.
>  2.  A deep routing enabled DS with not sufficient deep routing usage, could 
> use a smaller minimum size for a deep cache group.
> 
> 
> 
> Right now, we need to decide if we want to make this configurable or static. 
> Since I think this will need to be configurable in the future anyway, so it a 
> good idea to do it now.
> 
> 
> 
> This will require changes to TO/TP and traffic router.
> 
> Thoughts?

Re: [RESULT] [VOTE] New Traffic Control Website

[Eric Friedrich (efriedri)]

Vote passes, new website is live

—Eric 
> On May 18, 2018, at 11:06 AM, Eric Friedrich <fri...@apache.org> wrote:
> 
> Hey-
>  Based on some feedback from the incubator, we put together a new website
> that doesn’t look like its from Geocities.
> 
> I’m providing links to all the pages because some of the internal links are
> broken
> https://trafficcontrol.apache.org/private/
> https://trafficcontrol.apache.org/private/releases/
> https://trafficcontrol.apache.org/private/security/
> https://trafficcontrol.apache.org/private/mailing_lists/
> 
> If you find any typos or other issues, please include them along with your
> vote.
> 
> Please vote as follows:
> +1 Approve the new website
> 0 Neutral, please provide feedback
> -1 Keep the existing website, please provide feedback
> 
> Thanks,
> Eric