Hi, On Tue, May 15, 2018 at 3:22 PM, Dennis Kliban <[email protected]> wrote: > On Mon, May 14, 2018 at 3:44 PM, Jeff Ortel <[email protected]> wrote: >> >> Let's brainstorm on something. >> >> Pulp needs to deal with remote repositories that are composed of multiple >> content types which may span the domain of a single plugin. Here are a few >> examples. Some Red Hat RPM repositories are composed of: RPMs, DRPMs, , >> ISOs and Kickstart Trees. Some OSTree repositories are composed of OSTrees >> & Kickstart Trees. This raises a question: >> >> How can pulp3 best support syncing with remote repositories that are >> composed of multiple (unrelated) content types in a way that doesn't result >> in plugins duplicating support for content types? >> >> Few approaches come to mind: >> >> 1. Multiple plugins (Remotes) participate in the sync flow to produce a >> new repository version. >> 2. Multiple plugins (Remotes) are sync'd successively each producing a new >> version of a repository. Only the last version contains the fully sync'd >> composition. >> 3. Plugins share code. >> 4. Other? >> >> >> Option #1: Sync would be orchestrated by core or the user so that multiple >> plugins (Remotes) participate in populating a new repository version. For >> example: the RPM plugin (Remote) and the Kickstart Tree plugin (Remote) >> would both be sync'd against the same remote repository that is composed of >> both types. The new repository version would be composed of the result of >> both plugin (Remote) syncs. To support this, we'd need to provide a way for >> each plugin to operate seamlessly on the same (new) repository version. >> Perhaps something internal to the RepositoryVersion. The repository version >> would not be marked "complete" until the last plugin (Remote) sync has >> succeeded. More complicated than #2 but results in only creating truly >> complete versions or nothing. No idea how this would work with current REST >> API whereby plugins provide sync endpoints. >> > > I like this approach because it allows the user to perform a single call to > the REST API and specify multiple "sync methods" to use to create a single > new repository version.
Same here, esp. if the goal is an all-or-nothing behavior w/r the mix-in remotes; i.e an atomic sync. This has a benefit of a clear start and end of the sync procedure, that the user might want to refer to. > >> >> Option #2: Sync would be orchestrated by core or the user so that multiple >> plugins (Remotes) create successive repository versions. For example: the >> RPM plugin (Remote) and the Kickstart Tree plugin (Remote) would both be >> sync'd against the same remote repository that is a composition including >> both types. The intermediate versions would be incomplete. Only the last >> version contains the fully sync'd composition. This approach can be >> supported by core today :) but will produce incomplete repository versions >> that are marked complete=True. This /seems/ undesirable, right? This may >> not be a problem for distribution since I would imaging that only the last >> (fully composed) version would be published. But what about other usages of >> the repository's "latest" version? I'm afraid I don't see use of a middle-version esp. in case of failures; e.g ostree failed to sync while rpm managed and kickstart managed too; is the sync OK as a whole? What to do with the versions created? Should I merge the successes into one and retry the failure? How many versions would this introduce? >> >> Option #3: requires a plugin to be aware of specific repository >> composition(s); other plugins and creates a code dependency between plugins. >> For example, the RPM plugin could delegate ISOs to the File plugin and >> Kickstart Trees to the KickStart Tree plugin. Do you mean that the RPM plug-in would directly call into the File plug-in? If that's the case then I don't like it much, would be a pain every time a new plug-in would be introduced (O(len(plugin)^2) of updates) or if the API of a plug-in changed (O(len(plugin)) updates). Esp. keeping the plugin code aware of other plugin updates would be ugly. >> >> For all options, plugins (Remotes) need to limit sync to affect only those >> content types within their domain. For example, the RPM (Remote) sync >> cannot add/remove ISO or KS Trees. >> >> I am an advocate of some from of options #1 or #2. Combining plugins >> (Remotes) as needed to deal with arbitrary combinations within remote >> repositories seems very powerful; does not impose complexity on plugin >> writers; and does not introduce code dependencies between plugins. >> >> Thoughts? >> >> _______________________________________________ >> Pulp-dev mailing list >> [email protected] >> https://www.redhat.com/mailman/listinfo/pulp-dev >> > > > _______________________________________________ > Pulp-dev mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/pulp-dev > Cheers, milan _______________________________________________ Pulp-dev mailing list [email protected] https://www.redhat.com/mailman/listinfo/pulp-dev
