I am not entirely sure, whether it is the same kind of problem, but in [0] there is an example of a stage that adds a declarative artifact in the middle of the pipeline.
[0] https://github.com/pulp/pulp_deb/blob/master/pulp_deb/app/tasks/synchronizing.py#L204 On Thu, 29 Aug 2019 10:42:33 -0400 Brian Bouterse <bmbou...@redhat.com> wrote: > I agree Austin's suggestion of a mechanism like that sounds good. Can > a more detailed description be written in Redmine and sent to the > list? This change is in Stages API which has such broad impact that > I'd feel most comfortable if we could get to that level of detail > together somehow. > > The Stages API details I'm sure we can work out, but the concern I > keep thinking about has to do with on-demand content. An Artifact > that doesn't also have a single url to download it from couldn't have > a RemoteArtifact created for it. If we can't make a RemoteArtifact > for it, then it won't work with on-demand sync modes. To me that is > the more challenging aspect of this goal. Will this work with > on-demand content? Do others see this as an issue? What do you think? > > @ttereshc thank you for bringing this up, and @asmacdo thank you for > jumping in and suggesting. > > > On Thu, Aug 29, 2019 at 10:20 AM Tatiana Tereshchenko > <ttere...@redhat.com> wrote: > > > Austin, thank you. > > Your option is more explicit, so it's probably better. > > > > Tanya > > > > On Wed, Aug 28, 2019 at 6:38 PM Austin Macdonald > > <amacd...@redhat.com> wrote: > > > >> > >> > >> On Wed, Aug 28, 2019 at 11:34 AM Tatiana Tereshchenko < > >> ttere...@redhat.com> wrote: > >> > >>> Bump. > >>> > >>> Please provide feedback if you have any. > >>> I'll start working on the PR to make the suggested change this > >>> week otherwise. > >>> > >>> Thank you, > >>> Tanya > >>> > >>> On Mon, Aug 26, 2019 at 12:46 PM Tatiana Tereshchenko < > >>> ttere...@redhat.com> wrote: > >>> > >>>> In RPM plugin we have Modulemd content. It comes from metadata > >>>> as one file and we parse it and then save each modulemd as a > >>>> separate file/artifact. > >>>> > >>>> The question is how to handle this content in the sync pipeline. > >>>> Modulemd content is artifactless on a remote source (metadata) > >>>> but it's not artifactless in Pulp, so it can't follow a standard > >>>> path - it needs an artifact but doesn't have any remote source > >>>> to download it from. > >>>> > >>>> The suggestion: find a way to skip ArtifactDownloader and > >>>> RemoteArtifactSaver stages. > >>>> The content and its artifact still need to go through all the > >>>> stages, except the ones which deal with artifact's url in some > >>>> way - download artifact (ArtifactDownloader stage) and or create > >>>> a RemoteArtifact for downloading later (RemoteArtifactSaver > >>>> stage). > >>>> > >>>> The straightforward way is just to check if DeclarativeArtifact > >>>> has url (or check some special value) and skip the stage > >>>> otherwise. Any concerns about this approach (apart form being > >>>> somewhat hacky)? > >>> > >>>> Any other solutions to the problem? > >>>> > >>> > >> That solution seems fine to me. I'll toss out another idea just to > >> have options. > >> > >> New bool on Declarative Artifact. Similar to deferred downloads, > >> certain stages are no-ops if DeclarativeArtifact.deferred_download > >> is True. > >> > >> https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/artifact_stages.py#L152 > >> > >> https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/models.py#L29 > >> > >> > >> > >>> Thank you, > >>>> Tanya > >>>> > >>>> P.S. FWIW, Reasons to store modulemd as a file are: > >>>> - the format is not very stable and new information can be added > >>>> - it can be large > >>>> - we don't need all the info in the DB, we use only small subset > >>>> of fields (to search by or for copy operations) > >>>> > >>> _______________________________________________ > >>> Pulp-dev mailing list > >>> Pulp-dev@redhat.com > >>> https://www.redhat.com/mailman/listinfo/pulp-dev > >>> > >> _______________________________________________ > > Pulp-dev mailing list > > Pulp-dev@redhat.com > > https://www.redhat.com/mailman/listinfo/pulp-dev
pgpgSvJ__DFfl.pgp
Description: OpenPGP digital signature
_______________________________________________ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
