[Pulp-dev] [Meeting Minutes] Pulp Sprint 69 Planning

[Robin Chan]

20-March-2020

New Sprint 69 query: https://pulp.plan.io/issues?query_id=150

New Agile board: https://pulp.plan.io/agile/board?query_id=151

Stats on how we did in previous 2 week  Sprint 68 Query
:

 New - 12

 Assigned - 12

 Post - 13

 Modified - 20

Closed - CURRENTRELEASE - 1

Closed - WORKSFORME - 0

Closed - NOTABUG - 1

Closed - WONTFIX - 5

Closed - COMPLETE - 0


Action Items from last Sprint Planning

   -

   none


Dates:

https://pulp.plan.io/projects/pulp/wiki/Sprint_Plans

Sprint Goals/Focus:

   -

   Pulpcore 3.3 release: April 1


Current sprint:

6246: less problem with new dynaconf version - not moving forward

4458: dana will unassign, keep on sprint

5932: been on several sprints - remove for now, dkliban

6244: could be closed - test w/GA then close? 3.2.1 has new version of
dynaconf. Dkliban will close with note.

6217: plugin will remove duplicates, so may need to re-evaluate - brian
will comment & we will remove.

5927: email thread stalled, what are next steps: remove for now & will try
to get to important issue later or when it becomes an impact for
stakeholders


Sprint candidates:

https://pulp.plan.io/issues?query_id=26

6371: add to query

5828: blocked on 5927 - fao89 add blocking relationship

6294: wait

6281: move to backlog, off sprint candidate

6282: need automation hub & migration plugin

6335: add - small work & security concern

6353: Planning in case need pulpcore changes

Pending Items:

   -


Action Items:

   -

   rchan: Clear Sprint Candidate https://pulp.plan.io/issues?query_id=26


   -

   rchan: send out planning minutes
   -

   5932: Dkliban will update & ensure we have a good plan before re-adding
   -

   Rchan update agile board for sprint & Move new, assigned, post issues to
   new sprint



Robin Chan

She/Her/Hers

Satellite Software Engineering Manager - Pulp

Red Hat 

IRC: rchan

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Duplicate nevra but not pkgId (suse repos)

[Pavel Picka]

David, That is a good point and agree now that location_href is much more
supportive to this case. Because when there will be two files with the
same path it is definitely an issue or incorrect repository (Even still
considering that anything can happen w/RPMs).

Bryan, in short this is more about validate metadata and correct way to
store them and recognize duplicates.
This case is in detail is about some packages in suse repo has the same
name (NEVRA) but they are different packages and have different usage (what
we learned recently - one can have some optimisation or legacy code in it
another is like original package from devs).


On Fri, Mar 20, 2020 at 1:33 PM Bryan Kearney  wrote:

> Is this just about where to store the files on disk?
>
> -- bk
>
> On 3/20/20 7:24 AM, David Davis wrote:
> > I think using pkgid is problematic though. Consider the case where you
> have two
> > packages with the same location_href but different pkgIds. Since the
> pulp_rpm code
> > uses location_href (which also gets stored as relative_path) as the
> filename, which
> > one will get published when a repo version is published?
> >
> > PS - Don't tell me that two different packages will never have the same
> > location_href. If it's one thing I've learned working on RPM, things
> that will never
> > happen sometimes do happen.
> >
> > David
> >
> >
> > On Fri, Mar 20, 2020 at 4:46 AM Pavel Picka  > > wrote:
> >
> > I think we should keep nevra as unique constraint, but as I
> mentioned before
> > (above in this thread) your idea is similar to mine as my suggestion
> was NEVRA +
> > checksum (pkgId).
> > With pkgId I've already tested it and working good.
> >
> > On Fri, Mar 20, 2020 at 5:43 AM Daniel Alley  > > wrote:
> >
> > I discussed this a little bit on the #rpm.org 
> channel.  Here
> > is the gist of that discussion
> >
> >   * The metadata is "crazy, but technically valid"
> >   * "the entire SUSE ecosystem tends to do this a lot, anything
> using OBS,
> > including nvidia and dell and friends"
> >   * "also, SUSE packages can have the same NEVRA with being
> completely
> > different packages because of how their build system makes
> packages"
> >
> > I'm not sure what the best means to fix it would be.  Perhaps
> the uniqueness
> > constraint should be on the location_href, instead of on the
> NEVRA?  Or on
> > NEVRA + location_href?
> >
> > On Wed, Mar 18, 2020 at 9:47 AM Ina Panova  > > wrote:
> >
> > Pavel,
> > I meant to say, that pulp3 does not have such limitation as
> pulp2 had (
> > saving rpms on the filesystem with same nevra).
> > The error is raised in pulp3 [0] when a repo version is
> created, because
> > of the repo key[1], we cannot have 2 rpms with save NEVRA.
> >
> > We can enable that, if we decide to, by adding location_href
> to the
> > repo_key, *but* this needs to be evaluated, it can have side
> effects and
> > we should involve our stakeholders to weigh in.
> >
> > [0]
> >
> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
> > [1]
> >
> https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/package.py#L188
> >
> > 
> > Regards,
> >
> > Ina Panova
> > Senior Software Engineer| Pulp| Red Hat Inc.
> >
> > "Do not go where the path may lead,
> >  go instead where there is no path and leave a trail."
> >
> >
> > On Wed, Mar 18, 2020 at 2:24 PM Pavel Picka <
> ppi...@redhat.com
> > > wrote:
> >
> > True in opensuse repository there are two
> possibilities 'src' and
> > 'nosrc' (this one should be legacy without source code),
> both are
> > recognized by createrepo_c as arch 'src'.
> >
> > To point the pulp2 code I mentioned I found here [0]
> (base rpm
> > package what I understood).
> >
> > The rise of error in pulp3 happening here [1] in
> pulpcore when adding
> > packages to repository version.
> > So as Ina mentioned it doesn't have to be an issue with
> packages
> > itself than the logic in sync.
> >
> > [0]
> https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L779
> > [1]
> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
> >
> > On Wed, Mar 18, 2020 at 1:55 PM Ina Panova <
> ipan...@redhat.com
> > > wrote:
> >
> > Tanya and Pavel,
> > in 

Re: [Pulp-dev] Should signing service be associated with Publication or Repository?

[Neal Gompa]

On Thu, Mar 19, 2020 at 11:14 PM Dennis Kliban  wrote:
>
> RPM plugin allows users to define a signing service per repository. All 
> publications created from repository versions of that repository are signed 
> with that signing service.
>
> The Debian plugin requires the user to specify the signing service each time 
> a publication is created. The signing service foreign key is stored with each 
> publication.
>
> Even though the implementation in Debian requires the user to provide the 
> service href each time a publication is created, it seems like a stronger 
> model. The signing service associated with a repository can change thus 
> making it challenging to keep track of which signing service was used to 
> create a publication.
>
> We should change the behavior in the RPM plugin before we release this 
> feature.

Isn't the reason for the difference that Debian repos only have
repodata signed and not packages?

I guess technically we could use different GPG keys for each
repository publish, but that would lead to multiple copies of the same
RPM with different data, since the expectation is that both RPMs and
the repodata should be signed for RPM repositories.


-- 
真実はいつも一つ！/ Always, there's only one truth!


___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Duplicate nevra but not pkgId (suse repos)

[Bryan Kearney]

Is this just about where to store the files on disk?

-- bk

On 3/20/20 7:24 AM, David Davis wrote:
> I think using pkgid is problematic though. Consider the case where you have 
> two
> packages with the same location_href but different pkgIds. Since the pulp_rpm 
> code
> uses location_href (which also gets stored as relative_path) as the filename, 
> which
> one will get published when a repo version is published?
> 
> PS - Don't tell me that two different packages will never have the same
> location_href. If it's one thing I've learned working on RPM, things that 
> will never
> happen sometimes do happen.
> 
> David
> 
> 
> On Fri, Mar 20, 2020 at 4:46 AM Pavel Picka  > wrote:
> 
> I think we should keep nevra as unique constraint, but as I mentioned 
> before
> (above in this thread) your idea is similar to mine as my suggestion was 
> NEVRA +
> checksum (pkgId). 
> With pkgId I've already tested it and working good. 
> 
> On Fri, Mar 20, 2020 at 5:43 AM Daniel Alley  > wrote:
> 
> I discussed this a little bit on the #rpm.org  
> channel.  Here
> is the gist of that discussion
> 
>   * The metadata is "crazy, but technically valid"
>   * "the entire SUSE ecosystem tends to do this a lot, anything using 
> OBS,
> including nvidia and dell and friends"
>   * "also, SUSE packages can have the same NEVRA with being completely
> different packages because of how their build system makes 
> packages"
> 
> I'm not sure what the best means to fix it would be.  Perhaps the 
> uniqueness
> constraint should be on the location_href, instead of on the NEVRA?  
> Or on
> NEVRA + location_href?
> 
> On Wed, Mar 18, 2020 at 9:47 AM Ina Panova  > wrote:
> 
> Pavel,
> I meant to say, that pulp3 does not have such limitation as pulp2 
> had (
> saving rpms on the filesystem with same nevra).
> The error is raised in pulp3 [0] when a repo version is created, 
> because
> of the repo key[1], we cannot have 2 rpms with save NEVRA.
> 
> We can enable that, if we decide to, by adding location_href to 
> the
> repo_key, *but* this needs to be evaluated, it can have side 
> effects and
> we should involve our stakeholders to weigh in.
> 
> [0]
> 
> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
> [1]
> 
> https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/package.py#L188
> 
> 
> Regards,
> 
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
> 
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
> 
> 
> On Wed, Mar 18, 2020 at 2:24 PM Pavel Picka  > wrote:
> 
> True in opensuse repository there are two possibilities 'src' 
> and
> 'nosrc' (this one should be legacy without source code), both 
> are
> recognized by createrepo_c as arch 'src'. 
> 
> To point the pulp2 code I mentioned I found here [0] (base rpm
> package what I understood).
> 
> The rise of error in pulp3 happening here [1] in pulpcore 
> when adding
> packages to repository version. 
> So as Ina mentioned it doesn't have to be an issue with 
> packages
> itself than the logic in sync. 
> 
> [0] 
> https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L779
> [1] 
> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
> 
> On Wed, Mar 18, 2020 at 1:55 PM Ina Panova  > wrote:
> 
> Tanya and Pavel,
> in this issue it is explained why we cannot keep 2 
> packages with
> same NEVRA but different checksums within a repo
> https://pulp.plan.io/issues/494
> 
> Pulp2 had a limitation where it was not able to save on 
> the
> filesystem 2 rpms with same filename, it lead to the 
> primary.xml
> that could have pointed to the rpm that did not actually 
> get saved.
> I believe in Pulp3 we could allow having rpm with same 
> NEVRA if
> they have different location_href within a repo.
> 
> 
> Regards,
> 
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
> 
> "Do not go where the path may lead,
>  

Re: [Pulp-dev] Should signing service be associated with Publication or Repository?

[Dennis Kliban]

On Fri, Mar 20, 2020 at 5:45 AM Matthias Dellweg  wrote:

> Actually Quirin and I have also seen the difference, and we discussed,
> we should implement a combination, where you can specify on the
> repo-level but override the signing service with each publication.
> It is a little more work for a lot more convenience, imho.
> And of course it might be nice to see it handled as similarly as
> possible in both plugins.
>
>
I like this idea. In both cases though, the signing service foreign key
should always be stored with a publication. Is that what you had in mind
also?


> On Thu, 19 Mar 2020 23:13:23 -0400
> Dennis Kliban  wrote:
>
> > RPM plugin allows users to define a signing service per repository.
> > All publications created from repository versions of that repository
> > are signed with that signing service.
> >
> > The Debian plugin requires the user to specify the signing service
> > each time a publication is created. The signing service foreign key
> > is stored with each publication.
> >
> > Even though the implementation in Debian requires the user to provide
> > the service href each time a publication is created, it seems like a
> > stronger model. The signing service associated with a repository can
> > change thus making it challenging to keep track of which signing
> > service was used to create a publication.
> >
> > We should change the behavior in the RPM plugin before we release this
> > feature.
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Duplicate nevra but not pkgId (suse repos)

[David Davis]

I think using pkgid is problematic though. Consider the case where you have
two packages with the same location_href but different pkgIds. Since the
pulp_rpm code uses location_href (which also gets stored as relative_path)
as the filename, which one will get published when a repo version is
published?

PS - Don't tell me that two different packages will never have the same
location_href. If it's one thing I've learned working on RPM, things that
will never happen sometimes do happen.

David


On Fri, Mar 20, 2020 at 4:46 AM Pavel Picka  wrote:

> I think we should keep nevra as unique constraint, but as I mentioned
> before (above in this thread) your idea is similar to mine as my suggestion
> was NEVRA + checksum (pkgId).
> With pkgId I've already tested it and working good.
>
> On Fri, Mar 20, 2020 at 5:43 AM Daniel Alley  wrote:
>
>> I discussed this a little bit on the #rpm.org channel.  Here is the gist
>> of that discussion
>>
>>- The metadata is "crazy, but technically valid"
>>- "the entire SUSE ecosystem tends to do this a lot, anything using
>>OBS, including nvidia and dell and friends"
>>- "also, SUSE packages can have the same NEVRA with being completely
>>different packages because of how their build system makes packages"
>>
>> I'm not sure what the best means to fix it would be.  Perhaps the
>> uniqueness constraint should be on the location_href, instead of on the
>> NEVRA?  Or on NEVRA + location_href?
>>
>> On Wed, Mar 18, 2020 at 9:47 AM Ina Panova  wrote:
>>
>>> Pavel,
>>> I meant to say, that pulp3 does not have such limitation as pulp2 had (
>>> saving rpms on the filesystem with same nevra).
>>> The error is raised in pulp3 [0] when a repo version is created, because
>>> of the repo key[1], we cannot have 2 rpms with save NEVRA.
>>>
>>> We can enable that, if we decide to, by adding location_href to the
>>> repo_key, *but* this needs to be evaluated, it can have side effects and we
>>> should involve our stakeholders to weigh in.
>>>
>>> [0]
>>> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
>>> [1]
>>> https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/package.py#L188
>>>
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Wed, Mar 18, 2020 at 2:24 PM Pavel Picka  wrote:
>>>
 True in opensuse repository there are two possibilities 'src' and
 'nosrc' (this one should be legacy without source code), both are
 recognized by createrepo_c as arch 'src'.

 To point the pulp2 code I mentioned I found here [0] (base rpm package
 what I understood).

 The rise of error in pulp3 happening here [1] in pulpcore when adding
 packages to repository version.
 So as Ina mentioned it doesn't have to be an issue with packages itself
 than the logic in sync.

 [0]
 https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L779
 [1]
 https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570

 On Wed, Mar 18, 2020 at 1:55 PM Ina Panova  wrote:

> Tanya and Pavel,
> in this issue it is explained why we cannot keep 2 packages with same
> NEVRA but different checksums within a repo
> https://pulp.plan.io/issues/494
>
> Pulp2 had a limitation where it was not able to save on the filesystem
> 2 rpms with same filename, it lead to the primary.xml that could have
> pointed to the rpm that did not actually get saved.
> I believe in Pulp3 we could allow having rpm with same NEVRA if they
> have different location_href within a repo.
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Wed, Mar 18, 2020 at 10:47 AM Tatiana Tereshchenko <
> ttere...@redhat.com> wrote:
>
>> Hi Pavel,
>>
>> On Tue, Mar 17, 2020 at 7:31 PM Pavel Picka 
>> wrote:
>>
>>> Hello, would like to ask you how to proceed with issue with
>>> duplicate (but not really) packages.
>>>
>>> I am syncing suse repository (opensuse42 and SLE12) and get and
>>> duplicate error. But when checking the packages [0](from primary.xml) 
>>> glibc
>>> and glibc they got same nevra but different checksum (and a few more as
>>> size..) so doesn't look like real duplicates.
>>>
>> Those are weird, the have the same nevra but see the location_href,
>> one is src and the other one is nosrc! :/ :
>> 
>> 
>>
>> It looks like something OpenSUSE specific. I'm not sure if it's a
>> valid way to create a repo with such metadata, we need to figure it out 
>> at
>> some point.
>>
>>

Re: [Pulp-dev] Should signing service be associated with Publication or Repository?

[Matthias Dellweg]

Actually Quirin and I have also seen the difference, and we discussed,
we should implement a combination, where you can specify on the
repo-level but override the signing service with each publication.
It is a little more work for a lot more convenience, imho.
And of course it might be nice to see it handled as similarly as
possible in both plugins.

On Thu, 19 Mar 2020 23:13:23 -0400
Dennis Kliban  wrote:

> RPM plugin allows users to define a signing service per repository.
> All publications created from repository versions of that repository
> are signed with that signing service.
> 
> The Debian plugin requires the user to specify the signing service
> each time a publication is created. The signing service foreign key
> is stored with each publication.
> 
> Even though the implementation in Debian requires the user to provide
> the service href each time a publication is created, it seems like a
> stronger model. The signing service associated with a repository can
> change thus making it challenging to keep track of which signing
> service was used to create a publication.
> 
> We should change the behavior in the RPM plugin before we release this
> feature.


pgpwvfzIiEFIi.pgp
Description: OpenPGP digital signature
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Duplicate nevra but not pkgId (suse repos)

[Pavel Picka]

I think we should keep nevra as unique constraint, but as I mentioned
before (above in this thread) your idea is similar to mine as my suggestion
was NEVRA + checksum (pkgId).
With pkgId I've already tested it and working good.

On Fri, Mar 20, 2020 at 5:43 AM Daniel Alley  wrote:

> I discussed this a little bit on the #rpm.org channel.  Here is the gist
> of that discussion
>
>- The metadata is "crazy, but technically valid"
>- "the entire SUSE ecosystem tends to do this a lot, anything using
>OBS, including nvidia and dell and friends"
>- "also, SUSE packages can have the same NEVRA with being completely
>different packages because of how their build system makes packages"
>
> I'm not sure what the best means to fix it would be.  Perhaps the
> uniqueness constraint should be on the location_href, instead of on the
> NEVRA?  Or on NEVRA + location_href?
>
> On Wed, Mar 18, 2020 at 9:47 AM Ina Panova  wrote:
>
>> Pavel,
>> I meant to say, that pulp3 does not have such limitation as pulp2 had (
>> saving rpms on the filesystem with same nevra).
>> The error is raised in pulp3 [0] when a repo version is created, because
>> of the repo key[1], we cannot have 2 rpms with save NEVRA.
>>
>> We can enable that, if we decide to, by adding location_href to the
>> repo_key, *but* this needs to be evaluated, it can have side effects and we
>> should involve our stakeholders to weigh in.
>>
>> [0]
>> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
>> [1]
>> https://github.com/pulp/pulp_rpm/blob/master/pulp_rpm/app/models/package.py#L188
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Wed, Mar 18, 2020 at 2:24 PM Pavel Picka  wrote:
>>
>>> True in opensuse repository there are two possibilities 'src' and
>>> 'nosrc' (this one should be legacy without source code), both are
>>> recognized by createrepo_c as arch 'src'.
>>>
>>> To point the pulp2 code I mentioned I found here [0] (base rpm package
>>> what I understood).
>>>
>>> The rise of error in pulp3 happening here [1] in pulpcore when adding
>>> packages to repository version.
>>> So as Ina mentioned it doesn't have to be an issue with packages itself
>>> than the logic in sync.
>>>
>>> [0]
>>> https://github.com/pulp/pulp_rpm/blob/2-master/plugins/pulp_rpm/plugins/db/models.py#L779
>>> [1]
>>> https://github.com/pulp/pulpcore/blob/master/pulpcore/app/models/repository.py#L570
>>>
>>> On Wed, Mar 18, 2020 at 1:55 PM Ina Panova  wrote:
>>>
 Tanya and Pavel,
 in this issue it is explained why we cannot keep 2 packages with same
 NEVRA but different checksums within a repo
 https://pulp.plan.io/issues/494

 Pulp2 had a limitation where it was not able to save on the filesystem
 2 rpms with same filename, it lead to the primary.xml that could have
 pointed to the rpm that did not actually get saved.
 I believe in Pulp3 we could allow having rpm with same NEVRA if they
 have different location_href within a repo.

 
 Regards,

 Ina Panova
 Senior Software Engineer| Pulp| Red Hat Inc.

 "Do not go where the path may lead,
  go instead where there is no path and leave a trail."


 On Wed, Mar 18, 2020 at 10:47 AM Tatiana Tereshchenko <
 ttere...@redhat.com> wrote:

> Hi Pavel,
>
> On Tue, Mar 17, 2020 at 7:31 PM Pavel Picka  wrote:
>
>> Hello, would like to ask you how to proceed with issue with duplicate
>> (but not really) packages.
>>
>> I am syncing suse repository (opensuse42 and SLE12) and get and
>> duplicate error. But when checking the packages [0](from primary.xml) 
>> glibc
>> and glibc they got same nevra but different checksum (and a few more as
>> size..) so doesn't look like real duplicates.
>>
> Those are weird, the have the same nevra but see the location_href,
> one is src and the other one is nosrc! :/ :
> 
> 
>
> It looks like something OpenSUSE specific. I'm not sure if it's a
> valid way to create a repo with such metadata, we need to figure it out at
> some point.
>
>
>> I've checked Pulp2 and there is used nevra+sum for repository
>> uniqueness. In pulp3 we use only nevra.
>>
> Why do you think that in pulp 2 we use NEVRA + checksum? have you
> tested it?  please point to the code.
> I believe in Pulp 2 as well as in Pulp 3 we allow to have packages
> with different checksums in Pulp storage.
> I don't think we allow having the same packages with different
> checksums in the same repo.
> FWIW, in pulp 2 the most recently added package is chosen to stay in a
> repo, no packages with duplicate NEVRA left after sync, see
>