Yeah, I looked at yq. For one... it's written in Python. :) But it does appear to be compatible with 2.7 and 3.6, so it may be easier to run. On Wed, Nov 14, 2018 at 1:35 PM Dewayne Richardson <[email protected]> wrote: > > No strong opinion here, the admin.pl was built for a different time in the > history of TC. The reason we used Perl was for config parsing back in the > day, but there is a yaml version of jq, called yq. I honestly agree with > bash if we can get away with it. > > FYI: > https://github.com/kislyuk/yq > > -Dew > > > On Mon, Nov 12, 2018 at 11:29 AM Rawlin Peters <[email protected]> > wrote: > > > Eric, > > > > I share your sentiment about being reluctant to introduce another > > language as a dependency for Traffic Ops, but I wasn't able to find a > > really good, easily-available utility for parsing yaml (a la `jq` for > > json parsing) in a Bash script. Since the goose config is in yaml, > > `db/admin.pl` uses a yaml package to parse the goose config into > > variables which are then passed to the external `psql` et al. > > commands. It is possible to parse yaml using sed, but the example I > > found for doing that seemed really sketchy and fragile. So I figured > > using a solid YAML-parsing library like PyYAML in Python would be a > > safer bet while still allowing the use of a fully-featured programming > > language rather than "Bash + <insert yaml-parsing CLI tool here>". It > > would also allow us to potentially use a DB library to interface with > > the DB directly in Python rather than requiring `psql` et al. and just > > shelling out to those external commands (although I plan to continue > > doing it that way for now). > > > > As a side note, it also paves the way for moving other scripts to > > Python like WebDep.pm, which uses a Perl package that is virtually > > impossible to install/get running on Mac because of Perl's broken SSL > > on Mac, which would make it much easier to start as a new developer on > > the project. I remember when I started working on Traffic Control, I > > had to copy someone else's Perl `traffic_ops/app/local` directory who > > had been on the project a long time and had actually gotten it to > > build on Mac before it became unusable. Eliminating issues like that > > by using a more popular and supportable language is a win in my book, > > but right now I'm just focusing on `db/admin.pl` to allow for better > > testability of the DB migration operations. > > > > - Rawlin > > > > On Mon, Nov 12, 2018 at 6:46 AM Eric Friedrich -X (efriedri - TRITON > > UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco) > > <[email protected]> wrote: > > > > > > I’m only slightly familiar with all the different options for db/ > > admin.pl. > > > > > > I’m a big fan of Python, but reluctant to introduce another language > > into TC without a strong reason. > > > > > > Once the reverse_schema option is removed, what would be the main > > purposes of the script? > > > > > > Looks like this is something that could be easily converted to bash > > without need for another dependency. > > > > > > —Eric > > > > > > > > > > > > > > > > > > > > > > > > > On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <[email protected]> wrote: > > > > > > > > +1 on rewriting admin.pl -- Python seems a reasonable choice, esp > > since we > > > > seem to be gaining a lot of Python expertise recently. > > > > > > > > -1 on 2.x compatibility -- writing something new with compatibility > > for 2 > > > > major versions makes no sense to me. It limits the features and > > libraries > > > > that can be used and potentially doubles the amount of testing > > required. > > > > > > > > > > > > > > > > On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <[email protected]> wrote: > > > > > > > >> +1, seems reasonable. I don’t really have an opinion on python 2.x > > > >> compatibility, but whatever makes the most sense for the amount of > > work is > > > >> what I would prefer. > > > >> > > > >> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan < > > [email protected]> > > > >> wrote: > > > >> > > > >>> +1 There is already precedence in the repo for python for other > > purposes. > > > >>> The only caveat I would include is to be sure you include backward > > > >>> compatibility for python 2.x for the next year or so until it goes > > EOL. > > > >>> > > > >>> Jonathan G > > > >>> > > > >>> On 11/9/18, 5:23 PM, "Rawlin Peters" <[email protected]> > > wrote: > > > >>> > > > >>> Hey TC devs, > > > >>> > > > >>> As we eliminate the need for Traffic Ops Perl, it will no longer > > > >>> really make sense for the db/admin.pl script to be Perl as it is > > > >>> today. This is because it depends on the Perl modules that are > > > >>> installed via Carton for running Traffic Ops Perl. However, it's > > > >>> mostly just a Perl wrapper around shell commands except for a YAML > > > >>> Perl module and the `reverse_schema` command which uses the DBIx > > Perl > > > >>> module to generate the ORM schema for Traffic Ops Perl (i.e. > > you've > > > >>> added a new DB table/column and need to get the ORM files updated > > to > > > >>> use it). > > > >>> > > > >>> Without TO-Perl, there's no need for the `reverse_schema` command > > in > > > >>> db/admin.pl and its dependency on the Perl DBIx module. At that > > > >> point > > > >>> db/admin.pl is just a Perl script that parses some YAML and > > shells > > > >> out > > > >>> commands. > > > >>> > > > >>> Part of the problem with TO-Perl is that there are a bunch of > > random > > > >>> non-API Perl scripts that basically assume all the modules in the > > > >>> cpanfile are installed in the environment, even though a lot of > > those > > > >>> dependencies are probably only required for the Perl TO API or > > UI. So > > > >>> unless we go through all those Perl dependencies to eliminate > > > >>> everything we don't really need once the Perl TO API and UI are > > > >>> completely removed, we'll continue to have a handful of Perl > > scripts > > > >>> like db/admin.pl that still require downloading and installing > > the > > > >>> full set of TO Perl dependencies. On fresh installs, running > > Carton > > > >> to > > > >>> install these dependencies can take nearly half an hour. > > > >>> > > > >>> I'm working on adding tests for the DB upgrade/downgrade process > > > >> which > > > >>> I'd like to run automatically on PR submissions, but it really > > only > > > >>> needs the db/admin.pl script out of TO which assumes the entire > > set > > > >> of > > > >>> Perl TO dependencies even though it mostly just shells out to > > `goose > > > >>> up` and `goose down`. I'd like the test to emulate an actual TO > > > >>> environment as closely as possible to match what would actually > > > >> happen > > > >>> in a production DB upgrade/downgrade. Right now I'm reusing the > > > >>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd > > like > > > >>> to port db/admin.pl into Python to give it its own set of > > > >> dependencies > > > >>> (just a YAML package) and not require the full set of TO Perl > > > >>> dependencies. Then I can spin up a much lighter-weight container > > with > > > >>> just the TO Python packages rather than setting up a separate > > > >> cpanfile > > > >>> with just the Perl packages needed for db/admin.pl. > > > >>> > > > >>> +1/-1 for adding Python as a dependency of Traffic Ops for porting > > > >>> scripts like db/admin.pl? > > > >>> > > > >>> -Rawlin > > > >>> > > > >>> > > > >>> > > > >> > > > > >
