Re: [Pulp-list] Using Pulp to "merge" multiple RPM repos into one?

2015-09-14 Thread Nick Coghlan 
On 11 September 2015 at 23:09, Michael Hrivnak  wrote:
> I think your plan is spot-on. In usually makes sense to have a 1-1 mapping
> of remote repos to pulp repos, and to keep the pulp repo as a simple mirror
> of that remote repo. From there, you can copy out of the pulp-hosted mirrors
> to compose new repos with whatever mix of content you like.

I've even come up with names I like for the RepoFunnel data model:
TrackingRepo and MergeRepo, and then the funnels are an N:1 mapping of
TrackingRepo event listeners to the target merge repo.

One feature that Pulp doesn't have yet that could be valuable for this
use case is the notion of a "metadata only" repo, where we don't
actually download the artifacts themsleve, but instead store just the
repo metadata, and the original *URLs* for the artifacts.

With appropriate publisher plugins, RepoFunnel could then be used an
input filter for an object storage based system like pinrepo:
https://github.com/pinterest/pinrepo

Regards,
Nick.

>
> Michael
>
> On Fri, Sep 11, 2015 at 5:42 AM, Nick Coghlan  wrote:
>>
>> Hi folks,
>>
>> As part of a development workflow idea for the Fedora Environments &
>> Stacks working group [1], I'm looking to build a service that lets
>> people select multiple COPR repos, and have them automatically
>> integrated into a single downstream repo.
>>
>> As a starting point, I'm aiming to build the simplest possible proof
>> of concept: take two existing COPR repos, and configure Pulp to
>> download and republish all of their content as a single combined repo.
>>
>> I mistakenly thought I could do this just by adding multiple importers
>> to a single Pulp repository, but discovered today that Pulp doesn't
>> actually support doing that - the importer:repository mapping is 1:1.
>> Finding out I didn't know Pulp's capability's as well as I thought
>> made me realise I should ask here for advice before proceeding further
>> :)
>>
>> My current thinking is that my architecture will need to look something
>> like:
>>
>> 1. For any COPR repo I want to merge, configure a local mirror in Pulp
>> that imports the content from that repo. These would be system
>> managed, so there's only ever one local mirror per remote repo.
>> 2. For each funnel, configure a dedicated target repo, and create
>> event listeners on the relevant mirror repos that trigger a content
>> unit copy whenever the mirror repos are updated
>>
>> Does that general approach sound reasonable? Are there simpler
>> alternatives that I've missed?
>>
>> Regards,
>> Nick.
>>
>> [1]
>> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/SoftwareComponentPipeline
>>
>> --
>> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>>
>> ___
>> Pulp-list mailing list
>> Pulp-list@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>



-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

[Pulp-list] Using Pulp to "merge" multiple RPM repos into one?

2015-09-11 Thread Nick Coghlan 
Hi folks,

As part of a development workflow idea for the Fedora Environments &
Stacks working group [1], I'm looking to build a service that lets
people select multiple COPR repos, and have them automatically
integrated into a single downstream repo.

As a starting point, I'm aiming to build the simplest possible proof
of concept: take two existing COPR repos, and configure Pulp to
download and republish all of their content as a single combined repo.

I mistakenly thought I could do this just by adding multiple importers
to a single Pulp repository, but discovered today that Pulp doesn't
actually support doing that - the importer:repository mapping is 1:1.
Finding out I didn't know Pulp's capability's as well as I thought
made me realise I should ask here for advice before proceeding further
:)

My current thinking is that my architecture will need to look something like:

1. For any COPR repo I want to merge, configure a local mirror in Pulp
that imports the content from that repo. These would be system
managed, so there's only ever one local mirror per remote repo.
2. For each funnel, configure a dedicated target repo, and create
event listeners on the relevant mirror repos that trigger a content
unit copy whenever the mirror repos are updated

Does that general approach sound reasonable? Are there simpler
alternatives that I've missed?

Regards,
Nick.

[1] 
https://fedoraproject.org/wiki/Env_and_Stacks/Projects/SoftwareComponentPipeline

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list

Re: [Pulp-list] Using Pulp to "merge" multiple RPM repos into one?

2015-09-11 Thread Michael Hrivnak 
I think your plan is spot-on. In usually makes sense to have a 1-1 mapping
of remote repos to pulp repos, and to keep the pulp repo as a simple mirror
of that remote repo. From there, you can copy out of the pulp-hosted
mirrors to compose new repos with whatever mix of content you like.

Michael

On Fri, Sep 11, 2015 at 5:42 AM, Nick Coghlan  wrote:

> Hi folks,
>
> As part of a development workflow idea for the Fedora Environments &
> Stacks working group [1], I'm looking to build a service that lets
> people select multiple COPR repos, and have them automatically
> integrated into a single downstream repo.
>
> As a starting point, I'm aiming to build the simplest possible proof
> of concept: take two existing COPR repos, and configure Pulp to
> download and republish all of their content as a single combined repo.
>
> I mistakenly thought I could do this just by adding multiple importers
> to a single Pulp repository, but discovered today that Pulp doesn't
> actually support doing that - the importer:repository mapping is 1:1.
> Finding out I didn't know Pulp's capability's as well as I thought
> made me realise I should ask here for advice before proceeding further
> :)
>
> My current thinking is that my architecture will need to look something
> like:
>
> 1. For any COPR repo I want to merge, configure a local mirror in Pulp
> that imports the content from that repo. These would be system
> managed, so there's only ever one local mirror per remote repo.
> 2. For each funnel, configure a dedicated target repo, and create
> event listeners on the relevant mirror repos that trigger a content
> unit copy whenever the mirror repos are updated
>
> Does that general approach sound reasonable? Are there simpler
> alternatives that I've missed?
>
> Regards,
> Nick.
>
> [1]
> https://fedoraproject.org/wiki/Env_and_Stacks/Projects/SoftwareComponentPipeline
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
> ___
> Pulp-list mailing list
> Pulp-list@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>
___
Pulp-list mailing list
Pulp-list@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-list