On Tue, May 15, 2018 at 4:48 PM, Jeff Ortel <[email protected]> wrote: > > > On 05/15/2018 09:29 AM, Milan Kovacik wrote: >> >> Hi, >> >> On Tue, May 15, 2018 at 3:22 PM, Dennis Kliban <[email protected]> wrote: >>> >>> On Mon, May 14, 2018 at 3:44 PM, Jeff Ortel <[email protected]> wrote: >>>> >>>> Let's brainstorm on something. >>>> >>>> Pulp needs to deal with remote repositories that are composed of >>>> multiple >>>> content types which may span the domain of a single plugin. Here are a >>>> few >>>> examples. Some Red Hat RPM repositories are composed of: RPMs, DRPMs, , >>>> ISOs and Kickstart Trees. Some OSTree repositories are composed of >>>> OSTrees >>>> & Kickstart Trees. This raises a question: >>>> >>>> How can pulp3 best support syncing with remote repositories that are >>>> composed of multiple (unrelated) content types in a way that doesn't >>>> result >>>> in plugins duplicating support for content types? >>>> >>>> Few approaches come to mind: >>>> >>>> 1. Multiple plugins (Remotes) participate in the sync flow to produce a >>>> new repository version. >>>> 2. Multiple plugins (Remotes) are sync'd successively each producing a >>>> new >>>> version of a repository. Only the last version contains the fully >>>> sync'd >>>> composition. >>>> 3. Plugins share code. >>>> 4. Other? >>>> >>>> >>>> Option #1: Sync would be orchestrated by core or the user so that >>>> multiple >>>> plugins (Remotes) participate in populating a new repository version. >>>> For >>>> example: the RPM plugin (Remote) and the Kickstart Tree plugin (Remote) >>>> would both be sync'd against the same remote repository that is composed >>>> of >>>> both types. The new repository version would be composed of the result >>>> of >>>> both plugin (Remote) syncs. To support this, we'd need to provide a way >>>> for >>>> each plugin to operate seamlessly on the same (new) repository version. >>>> Perhaps something internal to the RepositoryVersion. The repository >>>> version >>>> would not be marked "complete" until the last plugin (Remote) sync has >>>> succeeded. More complicated than #2 but results in only creating truly >>>> complete versions or nothing. No idea how this would work with current >>>> REST >>>> API whereby plugins provide sync endpoints. >>>> >>> I like this approach because it allows the user to perform a single call >>> to >>> the REST API and specify multiple "sync methods" to use to create a >>> single >>> new repository version. >> >> Same here, esp. if the goal is an all-or-nothing behavior w/r the >> mix-in remotes; i.e an atomic sync. >> This has a benefit of a clear start and end of the sync procedure, >> that the user might want to refer to. >> >>>> Option #2: Sync would be orchestrated by core or the user so that >>>> multiple >>>> plugins (Remotes) create successive repository versions. For example: >>>> the >>>> RPM plugin (Remote) and the Kickstart Tree plugin (Remote) would both be >>>> sync'd against the same remote repository that is a composition >>>> including >>>> both types. The intermediate versions would be incomplete. Only the >>>> last >>>> version contains the fully sync'd composition. This approach can be >>>> supported by core today :) but will produce incomplete repository >>>> versions >>>> that are marked complete=True. This /seems/ undesirable, right? This >>>> may >>>> not be a problem for distribution since I would imaging that only the >>>> last >>>> (fully composed) version would be published. But what about other >>>> usages of >>>> the repository's "latest" version? >> >> I'm afraid I don't see use of a middle-version esp. in case of >> failures; e.g ostree failed to sync while rpm managed and kickstart >> managed too; is the sync OK as a whole? What to do with the versions >> created? Should I merge the successes into one and retry the failure? >> How many versions would this introduce? > > > (option 2) The partial versions would be created in both normal and failure > scenarios. The normal scenario is created because each plugin (Remote) > creates a new version and only the last one is completed. the intermediate > versions are always partial.
right but is there a legitimate use of the intermediate version? if not, maybe Option #1 is better (atomic) > >> >>>> Option #3: requires a plugin to be aware of specific repository >>>> composition(s); other plugins and creates a code dependency between >>>> plugins. >>>> For example, the RPM plugin could delegate ISOs to the File plugin and >>>> Kickstart Trees to the KickStart Tree plugin. >> >> Do you mean that the RPM plug-in would directly call into the File >> plug-in? >> If that's the case then I don't like it much, would be a pain every >> time a new plug-in would be introduced (O(len(plugin)^2) of updates) >> or if the API of a plug-in changed (O(len(plugin)) updates). >> Esp. keeping the plugin code aware of other plugin updates would be ugly. > > > Agreed. The plugins could install libs into site-packages which would at > least mitigate the complexity of calling into each other through the pulp > plugin framework but I don't think it helps much. Even the rpm dependency is > undesirable. > yeah, sounds like an anti-pattern > >> >>>> For all options, plugins (Remotes) need to limit sync to affect only >>>> those >>>> content types within their domain. For example, the RPM (Remote) sync >>>> cannot add/remove ISO or KS Trees. >>>> >>>> I am an advocate of some from of options #1 or #2. Combining plugins >>>> (Remotes) as needed to deal with arbitrary combinations within remote >>>> repositories seems very powerful; does not impose complexity on plugin >>>> writers; and does not introduce code dependencies between plugins. >>>> >>>> Thoughts? >>>> >>>> _______________________________________________ >>>> Pulp-dev mailing list >>>> [email protected] >>>> https://www.redhat.com/mailman/listinfo/pulp-dev >>>> >>> >>> _______________________________________________ >>> Pulp-dev mailing list >>> [email protected] >>> https://www.redhat.com/mailman/listinfo/pulp-dev >>> >> Cheers, >> milan > > > _______________________________________________ > Pulp-dev mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/pulp-dev _______________________________________________ Pulp-dev mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-dev
