Re: [Pulp-dev] Concerns about bulk_create and PostgreSQL

[Daniel Alley]

Brian, that's an excellent writup, thanks!

>From what I can tell, it looks very very unlikely that MySQL will add the
"returning" syntax.  MariaDB however has supported "returning" for DELETE
operations *only* for about 5 years, and has a 2 year old issue to add it
for "INSERT" https://jira.mariadb.org/browse/MDEV-10014

However, it also seems like Django takes the "common denominator" approach
towards supporting them (since they are so similar, it treats them as the
same database), so even if one or the other were to add support, I doubt
that functionality would be exposed through the ORM unless they *both* were
to support it.  Therefore I think we can assume we won't see that
functionality any time soon.

I'll probably do the performance investigation as described in #4290 some
time late next week.  I'd like for some of the refactoring that has been
going on in the stages API to be completed before I proceed with that.

On Wed, Jan 16, 2019 at 4:11 PM Brian Bouterse  wrote:

> Today @jortel, @dalley, @daviddavis, and @asmacdo been taking a look at
> our options in terms of resolving the PK issue that prevents Pulp from
> running on MariaDB. After considering various of possible solutions on a
> call, and @dalleydoing a little bit of research into the feasibility of
> those, I believe there are only two options:
>
> A) switch to UUIDs or B) implement a fallback that saves one at a time if
> there's no "RETURNING".
>
> In reflecting on this, I have two observations about option (B). First,
> the root cause of this is that MariaDB doesn't implement the "RETURNING"
> option like PostgreSQL and Oracle does. The best solution would be for
> MariaDB to do that. If they aren't able to fix it where the problem is
> created, investing a good deal of time and energy into how to make Pulp run
> fast on MariaDB doesn't seem worth it to me. Secondly, our fallback to
> one-by-one mode would effectively cause Pulp sync on MariaDB to experience
> the slowdowns described in this ticket [0] which were generally identified
> as unacceptable for Pulp users. In terms of timeline, since we had talked
> about MariaDB compatibility being an RC blocker, I don't think we want to
> wait for MariaDB to make any changes, like an implementation for
> "RETURNING".
>
> In thinking about option (A) it resolves the compatibility issue and
> allows plugin writers to avoid the "my dbs don't work the same with
> bulk_create" problem entirely, so it's a great solution to the central
> problem (db compatibility). My only concern with (A) is that we know it is
> a some amount slower in terms of write performance, read performance, and
> index size concerns. @dalley has an issue written up [1] to look at the
> impact of adopting (A) on Pulp workloads.
>
> So the next step would be to complete [1] and share the results. After
> looking at them we would yes/no the adopting of UUIDs as likely our only
> feasible resolution. Any other ideas on what to do or how to do it are
> welcome.
>
> [0]: https://pulp.plan.io/issues/3770
> [1]: https://pulp.plan.io/issues/4290
>
> -Brian
>
>
>
> On Wed, Jan 9, 2019 at 1:41 PM Matthias Dellweg  wrote:
>
>> On Wed, 9 Jan 2019 08:46:18 -0500
>> David Davis  wrote:
>>
>> > The Rubygems api includes sha as part of the metadata for a gem.
>> > Couldn't you use that as part of the natural key?
>> >
>> > I'm surprised that Chef's supermarket API doesn't include this as
>> > well. Maybe we could open a feature request?
>> >
>> > David
>>
>> WRT rubygems i use repositories generated by `gem generate_index`.
>> There is no chechsum in the metadata that i found.
>> But also the whole thing seems to be documented rather poorly.
>> Using rubygems.org as a remote is not adviseable until there are proper
>> filters implemented. ;)
>>   Matthias
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Concerns about bulk_create and PostgreSQL

[Brian Bouterse]

Today @jortel, @dalley, @daviddavis, and @asmacdo been taking a look at our
options in terms of resolving the PK issue that prevents Pulp from running
on MariaDB. After considering various of possible solutions on a call, and
@dalleydoing a little bit of research into the feasibility of those, I
believe there are only two options:

A) switch to UUIDs or B) implement a fallback that saves one at a time if
there's no "RETURNING".

In reflecting on this, I have two observations about option (B). First, the
root cause of this is that MariaDB doesn't implement the "RETURNING" option
like PostgreSQL and Oracle does. The best solution would be for MariaDB to
do that. If they aren't able to fix it where the problem is created,
investing a good deal of time and energy into how to make Pulp run fast on
MariaDB doesn't seem worth it to me. Secondly, our fallback to one-by-one
mode would effectively cause Pulp sync on MariaDB to experience the
slowdowns described in this ticket [0] which were generally identified as
unacceptable for Pulp users. In terms of timeline, since we had talked
about MariaDB compatibility being an RC blocker, I don't think we want to
wait for MariaDB to make any changes, like an implementation for
"RETURNING".

In thinking about option (A) it resolves the compatibility issue and allows
plugin writers to avoid the "my dbs don't work the same with bulk_create"
problem entirely, so it's a great solution to the central problem (db
compatibility). My only concern with (A) is that we know it is a some
amount slower in terms of write performance, read performance, and index
size concerns. @dalley has an issue written up [1] to look at the impact of
adopting (A) on Pulp workloads.

So the next step would be to complete [1] and share the results. After
looking at them we would yes/no the adopting of UUIDs as likely our only
feasible resolution. Any other ideas on what to do or how to do it are
welcome.

[0]: https://pulp.plan.io/issues/3770
[1]: https://pulp.plan.io/issues/4290

-Brian



On Wed, Jan 9, 2019 at 1:41 PM Matthias Dellweg  wrote:

> On Wed, 9 Jan 2019 08:46:18 -0500
> David Davis  wrote:
>
> > The Rubygems api includes sha as part of the metadata for a gem.
> > Couldn't you use that as part of the natural key?
> >
> > I'm surprised that Chef's supermarket API doesn't include this as
> > well. Maybe we could open a feature request?
> >
> > David
>
> WRT rubygems i use repositories generated by `gem generate_index`.
> There is no chechsum in the metadata that i found.
> But also the whole thing seems to be documented rather poorly.
> Using rubygems.org as a remote is not adviseable until there are proper
> filters implemented. ;)
>   Matthias
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] QE commit bit

[Brian Bouterse]

This all sounds good to me.

On Tue, Jan 15, 2019 at 2:25 PM Kersom  wrote:

> Robin, yeap. Exactly what you described it.
>
> On Tue, Jan 15, 2019 at 2:22 PM Robin Chan  wrote:
>
>> Great. I withdraw:
>> #3. Shall we also agree that those not in [1] - in other words, the
>> developers give up commit bit for #2. Can still contribute but don't need
>> to be involved in #1 agreements.
>>
>> And to re-iterate and be very clear, Kersom's ", just to communicate QE
>> in case of test changes. We already have a system in place on git." looks
>> like getting an approved code review from someone in [1].
>>
>> That works for me and I appreciate the clarifications.
>> Robin
>>
>>
>> On Tue, Jan 15, 2019 at 2:15 PM David Davis 
>> wrote:
>>
>>> I agree. I think devs can merge changes to pulp-smash tests in pulp
>>> repos but they should get it reviewed by QE before merging--which, as
>>> Kersom says, we've been doing.
>>>
>>> David
>>>
>>>
>>> On Tue, Jan 15, 2019 at 2:11 PM Kersom  wrote:
>>>
 David, thanks for driving this.

 I agree with your suggestions Robin.

 All currently present on [1] should have commit bit for those repos.

 I think it is fine to the devs to have commit to the test repos, just
 to communicate QE in case of test changes. We already have a system in
 place on git.

 [1] https://github.com/orgs/pulp/teams/qe

 Thanks,

 On Tue, Jan 15, 2019 at 11:07 AM Robin Chan  wrote:

> A few suggestions.
>
> #1. QE good with [1] - you all agree these are the folks with commit
> bit? In other words, you trust each other to do the merge with your own
> agreements of who has expertise and when things are ready - all the 
> details?
> #2. I would suggest we are suggesting QE have commit bit access to the
> specific subdirectories;
>   a) pulp_file/pulp_file/tests/functional/ (in pulp/pulp_file repo)
>   b) pulp/pulp_core/tests/functional/ (in pulp/pulp repo)
> I know this is not enforceable via the GIT settings, but helpful to be
> explicit about as we include this in agreement.
> #3. Shall we also agree that those not in [1] - in other words, the
> developers give up commit bit for #2. Can still contribute but don't need
> to be involved in #1 agreements.
>
> Fully supportive of this effort. I was one of the folks who gave my
> word prior to PUP-6 and see this as making sure the folks have what they
> need to get stuff done and keeping decision making with the folks closest
> to the work (i.e. QE makes decisions about all things QE.)
>
> Thanks,
> Robin
>
> On Tue, Jan 15, 2019 at 10:37 AM David Davis 
> wrote:
>
>> When we moved the pulp-smash tests out of the pulp-smash repository,
>> we promised to give QE ownership of the smash tests within the Pulp
>> repositories on github. I know we have a process in place to give the
>> commit bit to contributors[0] but this promise predates PUP-6.
>>
>> Thus, I'd like to ask for feedback on giving the QE team in github[1]
>> the commit bit to the following repositories in order to merge changes to
>> smash tests:
>>
>> pulp/pulp
>> pulp/pulp_file
>>
>> I'd also like to ask plugin teams to consider giving QE commit access
>> to their repositories if they have pulp-smash tests that are maintained 
>> by
>> QE.
>>
>> Feedback would be appreciated. I'll like to set an deadline of
>> January 30th.
>>
>> Thanks.
>>
>> [0] https://github.com/pulp/pups/blob/master/pup-0006.md
>> [1] https://github.com/orgs/pulp/teams/qe
>>
>> David
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
 ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

[Pulp-dev] 2.18.1 Release Schedule

[Jeff Ortel]

A 2.18.1 is being planned with some features and recent fixes. Here [0] 
is a release schedule page which outlines the dates, starting with a dev 
freeze on January 21, 2019 @ 22:00 UTC.


If this schedule needs to be adjusted, please reply with alternate dates.

[0] https://pulp.plan.io/projects/pulp/wiki/2181_Release_Schedule 

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev