Making ResultsDB dumps public?

[Dan Callaghan]

Is there any objection to making the ResultsDB database dumps public 
under https://infrastructure.fedoraproject.org/infra/db-dumps/ ?

I wanted to grab a database dump today, to populate a local database for 
development purposes. I was able to scp it from db-qa02 since I'm in 
sysadmin-qa, but I figured it would be nice to have it available 
publicly alongside those other applications' db dumps.

I can't think of any reason why the ResultsDB dumps need to be kept 
secret. There should be no sensitive data in them, right?

So if there are no objections I'll file an infra ticket for adding the 
dumps into the public-db-copy script.

-- 
Dan Callaghan 
Senior Software Engineer, Products & Technologies Operations
Red Hat


signature.asc
Description: PGP signature
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Making ResultsDB dumps public?

[Harold Dost]

Assuming that the schema [1] is correct, there should be no personal or
secret information. Though it would be good to validate that there aren't
any unexpected tables.

[1]:https://fedoraproject.org/wiki/AutoQA_resultsdb_schema



Harold Dost | @hdost




On Sep 25, 2017 01:30, "Dan Callaghan"  wrote:

Is there any objection to making the ResultsDB database dumps public
under https://infrastructure.fedoraproject.org/infra/db-dumps/ ?

I wanted to grab a database dump today, to populate a local database for
development purposes. I was able to scp it from db-qa02 since I'm in
sysadmin-qa, but I figured it would be nice to have it available
publicly alongside those other applications' db dumps.

I can't think of any reason why the ResultsDB dumps need to be kept
secret. There should be no sensitive data in them, right?

So if there are no objections I'll file an infra ticket for adding the
dumps into the public-db-copy script.

--
Dan Callaghan 
Senior Software Engineer, Products & Technologies Operations
Red Hat

___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Making ResultsDB dumps public?

[Tim Flink]

On Mon, 25 Sep 2017 05:37:03 -0400
Harold Dost  wrote:

> Assuming that the schema [1] is correct, there should be no personal
> or secret information. Though it would be good to validate that there
> aren't any unexpected tables.
> 
> [1]:https://fedoraproject.org/wiki/AutoQA_resultsdb_schema

That schema was correct in the past but has not been kept up to date
AFAIK.

That being said, none of the changes I can think of would cause a
problem in making the database dump public.

Tim

___
> 
> Harold Dost | @hdost
> 
> 
> 
> 
> On Sep 25, 2017 01:30, "Dan Callaghan"  wrote:
> 
> Is there any objection to making the ResultsDB database dumps public
> under https://infrastructure.fedoraproject.org/infra/db-dumps/ ?
> 
> I wanted to grab a database dump today, to populate a local database
> for development purposes. I was able to scp it from db-qa02 since I'm
> in sysadmin-qa, but I figured it would be nice to have it available
> publicly alongside those other applications' db dumps.
> 
> I can't think of any reason why the ResultsDB dumps need to be kept
> secret. There should be no sensitive data in them, right?
> 
> So if there are no objections I'll file an infra ticket for adding the
> dumps into the public-db-copy script.
> 
> --
> Dan Callaghan 
> Senior Software Engineer, Products & Technologies Operations
> Red Hat
> 
> ___
> qa-devel mailing list -- qa-devel@lists.fedoraproject.org
> To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org



pgp0RrsAS3Wtm.pgp
Description: OpenPGP digital signature
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Making ResultsDB dumps public?

[Kevin Fenzi]

On 09/25/2017 07:13 AM, Tim Flink wrote:
> On Mon, 25 Sep 2017 05:37:03 -0400
> Harold Dost  wrote:
> 
>> Assuming that the schema [1] is correct, there should be no personal
>> or secret information. Though it would be good to validate that there
>> aren't any unexpected tables.
>>
>> [1]:https://fedoraproject.org/wiki/AutoQA_resultsdb_schema
> 
> That schema was correct in the past but has not been kept up to date
> AFAIK.
> 
> That being said, none of the changes I can think of would cause a
> problem in making the database dump public.

If you all are ok with it, I can make these public after the freeze is
over.

kevin




signature.asc
Description: OpenPGP digital signature
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Making ResultsDB dumps public?

[Tim Flink]

On Mon, 2 Oct 2017 14:12:54 -0700
Kevin Fenzi  wrote:

> On 09/25/2017 07:13 AM, Tim Flink wrote:
> > On Mon, 25 Sep 2017 05:37:03 -0400
> > Harold Dost  wrote:
> >   
> >> Assuming that the schema [1] is correct, there should be no
> >> personal or secret information. Though it would be good to
> >> validate that there aren't any unexpected tables.
> >>
> >> [1]:https://fedoraproject.org/wiki/AutoQA_resultsdb_schema  
> > 
> > That schema was correct in the past but has not been kept up to date
> > AFAIK.
> > 
> > That being said, none of the changes I can think of would cause a
> > problem in making the database dump public.  
> 
> If you all are ok with it, I can make these public after the freeze is
> over.

Sounds good to me. Thanks for getting that taken care of.

Tim


pgpV5KsZbwjQl.pgp
Description: OpenPGP digital signature
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Making ResultsDB dumps public?

[Kevin Fenzi]

On 10/03/2017 06:29 AM, Tim Flink wrote:
> On Mon, 2 Oct 2017 14:12:54 -0700
> Kevin Fenzi  wrote:
> 
>> On 09/25/2017 07:13 AM, Tim Flink wrote:
>>> On Mon, 25 Sep 2017 05:37:03 -0400
>>> Harold Dost  wrote:
>>>   
 Assuming that the schema [1] is correct, there should be no
 personal or secret information. Though it would be good to
 validate that there aren't any unexpected tables.

 [1]:https://fedoraproject.org/wiki/AutoQA_resultsdb_schema  
>>>
>>> That schema was correct in the past but has not been kept up to date
>>> AFAIK.
>>>
>>> That being said, none of the changes I can think of would cause a
>>> problem in making the database dump public.  
>>
>> If you all are ok with it, I can make these public after the freeze is
>> over.
> 
> Sounds good to me. Thanks for getting that taken care of.

They should be there starting tomorrow.

kevin




signature.asc
Description: OpenPGP digital signature
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Making ResultsDB dumps public?

[Kamil Paral]

On Sun, Oct 8, 2017 at 12:14 AM, Kevin Fenzi  wrote:

> They should be there starting tomorrow.
>


I still don't see any dumps in there. Is it still being worked on?

Thanks.
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Making ResultsDB dumps public?

[Kevin Fenzi]

On 10/24/2017 03:36 AM, Kamil Paral wrote:
> On Sun, Oct 8, 2017 at 12:14 AM, Kevin Fenzi  wrote:
> 
>> They should be there starting tomorrow.
>>
> 
> 
> I still don't see any dumps in there. Is it still being worked on?

I seem to have made a typo in the script that does this. ;(

Sent in a freeze break request with a fix

kevin




signature.asc
Description: OpenPGP digital signature
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Making ResultsDB dumps public?

[Kevin Fenzi]

On 01/11/2018 07:44 AM, Kamil Paral wrote:
> On Wed, Nov 1, 2017 at 7:48 PM, Kevin Fenzi  > wrote:
> 
> On 10/24/2017 03:36 AM, Kamil Paral wrote:
> > On Sun, Oct 8, 2017 at 12:14 AM, Kevin Fenzi  > wrote:
> >
> >> They should be there starting tomorrow.
> >>
> >
> >
> > I still don't see any dumps in there. Is it still being worked on?
> 
> I seem to have made a typo in the script that does this. ;(
> 
> Sent in a freeze break request with a fix
> 
> 
> I and Josef looked at it today, and it seems it contains some very old
> data - from Jan 2017 to April 2017. We failed to find the script that
> generates the dump in infra ansible, so we can't really debug what's
> wrong. Can you please look at it or give us some pointers?

Odd.

It's
https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/batcave/files/public-db-copy.sh?id=778dae3
which runs on batcave01 and runs:
https://infrastructure.fedoraproject.org/cgit/ansible.git/plain/scripts/public-db-copy?id=778dae3

scp db-qa01.qa.fedoraproject.org:/resultsdb-$(date +%F).dump.xz
/srv/web/infra/db-dumps/resultsdb.dump.xz

and on db-qa01.qa:/backups:

-rw-r--r--. 1 postgres postgres 142589024 Apr 12  2016
resultsdb-2016-04-12.dump.xz
-rw-r--r--. 1 postgres postgres  50184088 Jun 20  2017
resultsdb-2017-06-20.dump.xz
-rw-r--r--. 1 postgres postgres  50367072 Jan 13 01:59
resultsdb-2018-01-13.dump.xz
-rw-r--r--. 1 postgres postgres  50367072 Jan 14 01:12
resultsdb-2018-01-14.dump.xz

It claims to be the dump from today... so not sure whats going on.

kevin



signature.asc
Description: OpenPGP digital signature
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Making ResultsDB dumps public?

[Kamil Paral]

On Sun, Jan 14, 2018 at 11:19 PM, Kevin Fenzi  wrote:

> Odd.
>
> It's
> https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/
> batcave/files/public-db-copy.sh?id=778dae3
> which runs on batcave01 and runs:
> https://infrastructure.fedoraproject.org/cgit/ansible.git/plain/scripts/
> public-db-copy?id=778dae3
>
> scp db-qa01.qa.fedoraproject.org:/resultsdb-$(date +%F).dump.xz
> /srv/web/infra/db-dumps/resultsdb.dump.xz
>

That script copies over the dump. But how does the dump get created?
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Making ResultsDB dumps public?

[Tim Flink]

On Sun, 14 Jan 2018 14:19:54 -0800
Kevin Fenzi  wrote:

> On 01/11/2018 07:44 AM, Kamil Paral wrote:
> > On Wed, Nov 1, 2017 at 7:48 PM, Kevin Fenzi  > > wrote:
> > 
> > On 10/24/2017 03:36 AM, Kamil Paral wrote:  
> > > On Sun, Oct 8, 2017 at 12:14 AM, Kevin Fenzi  > > > wrote: 
> > >> They should be there starting tomorrow.
> > >>  
> > >
> > >
> > > I still don't see any dumps in there. Is it still being
> > > worked on?  
> > 
> > I seem to have made a typo in the script that does this. ;(
> > 
> > Sent in a freeze break request with a fix
> > 
> > 
> > I and Josef looked at it today, and it seems it contains some very
> > old data - from Jan 2017 to April 2017. We failed to find the
> > script that generates the dump in infra ansible, so we can't really
> > debug what's wrong. Can you please look at it or give us some
> > pointers?  
> 
> Odd.
> 
> It's
> https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/batcave/files/public-db-copy.sh?id=778dae3
> which runs on batcave01 and runs:
> https://infrastructure.fedoraproject.org/cgit/ansible.git/plain/scripts/public-db-copy?id=778dae3
> 
> scp db-qa01.qa.fedoraproject.org:/resultsdb-$(date +%F).dump.xz
> /srv/web/infra/db-dumps/resultsdb.dump.xz
> 
> and on db-qa01.qa:/backups:
> 
> -rw-r--r--. 1 postgres postgres 142589024 Apr 12  2016
> resultsdb-2016-04-12.dump.xz
> -rw-r--r--. 1 postgres postgres  50184088 Jun 20  2017
> resultsdb-2017-06-20.dump.xz
> -rw-r--r--. 1 postgres postgres  50367072 Jan 13 01:59
> resultsdb-2018-01-13.dump.xz
> -rw-r--r--. 1 postgres postgres  50367072 Jan 14 01:12
> resultsdb-2018-01-14.dump.xz
> 
> It claims to be the dump from today... so not sure whats going on.

The database for production resultsdb is on db-qa02.qa instead of
db-qa01.qa. 

From my read of those scripts, I suspect that is the cause of the
public dumps being old.

Tim


pgpPJBmj2cc4C.pgp
Description: OpenPGP digital signature
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org

Re: Making ResultsDB dumps public?

[Kamil Paral]

On Mon, Jan 15, 2018 at 3:25 PM, Tim Flink  wrote:

> On Sun, 14 Jan 2018 14:19:54 -0800
> Kevin Fenzi  wrote:
>
> > On 01/11/2018 07:44 AM, Kamil Paral wrote:
> > > On Wed, Nov 1, 2017 at 7:48 PM, Kevin Fenzi  > > > wrote:
> > >
> > > On 10/24/2017 03:36 AM, Kamil Paral wrote:
> > > > On Sun, Oct 8, 2017 at 12:14 AM, Kevin Fenzi  > > > > wrote:
> > > >> They should be there starting tomorrow.
> > > >>
> > > >
> > > >
> > > > I still don't see any dumps in there. Is it still being
> > > > worked on?
> > >
> > > I seem to have made a typo in the script that does this. ;(
> > >
> > > Sent in a freeze break request with a fix
> > >
> > >
> > > I and Josef looked at it today, and it seems it contains some very
> > > old data - from Jan 2017 to April 2017. We failed to find the
> > > script that generates the dump in infra ansible, so we can't really
> > > debug what's wrong. Can you please look at it or give us some
> > > pointers?
> >
> > Odd.
> >
> > It's
> > https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/
> batcave/files/public-db-copy.sh?id=778dae3
> > which runs on batcave01 and runs:
> > https://infrastructure.fedoraproject.org/cgit/ansible.git/plain/scripts/
> public-db-copy?id=778dae3
> >
> > scp db-qa01.qa.fedoraproject.org:/resultsdb-$(date +%F).dump.xz
> > /srv/web/infra/db-dumps/resultsdb.dump.xz
> >
> > and on db-qa01.qa:/backups:
> >
> > -rw-r--r--. 1 postgres postgres 142589024 Apr 12  2016
> > resultsdb-2016-04-12.dump.xz
> > -rw-r--r--. 1 postgres postgres  50184088 Jun 20  2017
> > resultsdb-2017-06-20.dump.xz
> > -rw-r--r--. 1 postgres postgres  50367072 Jan 13 01:59
> > resultsdb-2018-01-13.dump.xz
> > -rw-r--r--. 1 postgres postgres  50367072 Jan 14 01:12
> > resultsdb-2018-01-14.dump.xz
> >
> > It claims to be the dump from today... so not sure whats going on.
>
> The database for production resultsdb is on db-qa02.qa instead of
> db-qa01.qa.
>
> From my read of those scripts, I suspect that is the cause of the
> public dumps being old.
>

Thanks for the fix, it's working properly now (Josef checked).
___
qa-devel mailing list -- qa-devel@lists.fedoraproject.org
To unsubscribe send an email to qa-devel-le...@lists.fedoraproject.org