Re: [EXTERNAL] Re: Drop support for Traffic Ops DB downgrade

2018-10-23 Thread Dewayne Richardson
@Rawlin, more like this email. Sorry I didn't make it up this far before I replied. -Dew On Tue, Oct 23, 2018 at 10:27 AM Dave Neuman wrote: > Since this topic is now 16 emails long, I'll make it 17 to try to do some > summarization, address concerns, and see if we can get some consensus. >

Re: [EXTERNAL] Re: Drop support for Traffic Ops DB downgrade

2018-10-23 Thread Dewayne Richardson
I guess my whole point was to formalize the effort so that the "Use Cases" can be reviewed, not a lit of paragraphs that are sprinkled throughout emails which makes it difficult to see and understand as a whole. -Dew On Mon, Oct 22, 2018 at 8:59 AM Rawlin Peters wrote: > On Fri, Oct 19, 2018

Re: [EXTERNAL] Re: Drop support for Traffic Ops DB downgrade

2018-10-23 Thread Rawlin Peters
On Tue, Oct 23, 2018 at 11:10 AM David Neuman wrote: > > Individual developers and those that merge their PRs should be responsible > for developing and testing the `goose down` portion of a migration, this > does not change. > The whole argument is that relying on goose down for rollbacks is

Re: [EXTERNAL] Re: Drop support for Traffic Ops DB downgrade

2018-10-23 Thread David Neuman
Individual developers and those that merge their PRs should be responsible for developing and testing the `goose down` portion of a migration, this does not change. The whole argument is that relying on goose down for rollbacks is less than ideal because you have to run it 1 time per migration

Re: [EXTERNAL] Re: Drop support for Traffic Ops DB downgrade

2018-10-23 Thread Gray, Jonathan
If we're removing goose down from the downgrade procedure, but still requiring support for it, doesn't that just make finding issues with it more difficult since fewer use cases would use that codepath? Jonathan G On 10/23/18, 10:27 AM, "Dave Neuman" wrote: >>We still need the ability

Re: [EXTERNAL] Re: Drop support for Traffic Ops DB downgrade

2018-10-23 Thread Dave Neuman
Since this topic is now 16 emails long, I'll make it 17 to try to do some summarization, address concerns, and see if we can get some consensus. Lately our email discussion get to this point where there are many, many emails back and forth, but we aren't really getting anywhere (either for or

Re: [EXTERNAL] Re: Drop support for Traffic Ops DB downgrade

2018-10-19 Thread Rawlin Peters
We have built-in support for rolling back the TO DB via `goose down`, but no built-in support for backing up the DB at pre-upgrade and pre-rollback steps. We should do those automatically but also improve the db/admin.pl tooling to support restoring backup DBs as a standard process rather than as

Re: [EXTERNAL] Re: Drop support for Traffic Ops DB downgrade

2018-10-19 Thread Gray, Jonathan
I don't believe it should be the responsibility of ATC to perform the functions of a DBA. Postgres should be treated as a 3rd party dependency with independent skills & professional administrators performing industry standard practices. You should add to your documentation to ensure safe

Re: [EXTERNAL] Re: Drop support for Traffic Ops DB downgrade

2018-10-19 Thread Dewayne Richardson
I'm -1 until someone tests out the downgrade scenarios. My vote would be to keep the goose-like downgrade options (and potentially improve db/ admin.pl if needed to allow more rollback options if needed). Also, I'm also not opposed to a new goose alternative because it is outdated without

Re: [EXTERNAL] Re: Drop support for Traffic Ops DB downgrade

2018-10-19 Thread Rawlin Peters
On Fri, Oct 19, 2018 at 8:33 AM Gelinas, Derek wrote: > > I'm plus one on this if we add an automatic DB backup during install and a > restore utility. I would like to make the DB backups happen automatically at the pre-upgrade and pre-rollback steps with a handy utility to restore a certain

Re: [EXTERNAL] Re: Drop support for Traffic Ops DB downgrade

2018-10-19 Thread Rawlin Peters
On Fri, Oct 19, 2018 at 9:21 AM Gray, Jonathan wrote: > > I'm -1 on this. In the event you're doing some massaging of commits and > migrations are missed due to the dependence on timestamps in filenames, > "goose down" allows you to fix the data. I'm not saying we should totally get rid of

Re: Drop support for Traffic Ops DB downgrade

2018-10-19 Thread Rawlin Peters
On Fri, Oct 19, 2018 at 8:25 AM Robert Butts wrote: > > > simply restore pre-upgrade copy of the DB > > What about the data manually changed in-between? What if you've been > running for a week before discovering a critical issue requiring rollback? > All those changes would be lost? Depending

Re: [EXTERNAL] Re: Drop support for Traffic Ops DB downgrade

2018-10-19 Thread Gelinas, Derek
I'm plus one on this if we add an automatic DB backup during install and a restore utility. Derek On Oct 19, 2018, at 10:31 AM, Robert Butts wrote: >> simply restore pre-upgrade copy of the DB > > What about the data manually changed in-between? What if you've been > running for a week

Re: Drop support for Traffic Ops DB downgrade

2018-10-19 Thread Robert Butts
> simply restore pre-upgrade copy of the DB What about the data manually changed in-between? What if you've been running for a week before discovering a critical issue requiring rollback? All those changes would be lost? On Fri, Oct 19, 2018 at 8:07 AM Dave Neuman wrote: > I think this sounds

Re: [EXTERNAL] Drop support for Traffic Ops DB downgrade

2018-10-19 Thread Moltzau, Matthew
Along similar lines, what do you think of replacing goose? Having looked at how goose works I don't think will be too difficult, its repository isn't maintained, and it has a number of annoying bugs/lack of features. Goose at a very basic level merely runs sql statements under `--goose up` on

Re: Drop support for Traffic Ops DB downgrade

2018-10-19 Thread Dave Neuman
I think this sounds reasonable assuming we work out the details later. +1 On Fri, Oct 19, 2018 at 07:59 Dave Cardosi (dcardosi) wrote: > +1 > > On 10/18/18, 7:49 PM, "Rawlin Peters" wrote: > > >I'd like to propose we drop support for `goose down` in terms of doing > >a Traffic Ops downgrade. >

Re: Drop support for Traffic Ops DB downgrade

2018-10-19 Thread Dave Cardosi (dcardosi)
+1 On 10/18/18, 7:49 PM, "Rawlin Peters" wrote: >I'd like to propose we drop support for `goose down` in terms of doing >a Traffic Ops downgrade. > >Right now whenever you upgrade Traffic Ops you also need to run >`db/admin.pl upgrade` to migrate the DB to the latest version. This >step runs