During the pulp_rpm meeting today @ttereshc identified that the on-demand content concern I brought up is not an issue because modulemd is a metadata file so it will always be downloaded and its Artifact saved regardless of the on-demand policy. So that's a non-issue. :)
@mdellweg I will think more about what you are linking to applies and how it related to what @asmacdo was saying. Thank you for sharing. On Thu, Aug 29, 2019 at 11:11 AM Matthias Dellweg <dell...@atix.de> wrote: > I am not entirely sure, whether it is the same kind of problem, but in > [0] there is an example of a stage that adds a declarative artifact in > the middle of the pipeline. > > [0] > > https://github.com/pulp/pulp_deb/blob/master/pulp_deb/app/tasks/synchronizing.py#L204 > > On Thu, 29 Aug 2019 10:42:33 -0400 > Brian Bouterse <bmbou...@redhat.com> wrote: > > > I agree Austin's suggestion of a mechanism like that sounds good. Can > > a more detailed description be written in Redmine and sent to the > > list? This change is in Stages API which has such broad impact that > > I'd feel most comfortable if we could get to that level of detail > > together somehow. > > > > The Stages API details I'm sure we can work out, but the concern I > > keep thinking about has to do with on-demand content. An Artifact > > that doesn't also have a single url to download it from couldn't have > > a RemoteArtifact created for it. If we can't make a RemoteArtifact > > for it, then it won't work with on-demand sync modes. To me that is > > the more challenging aspect of this goal. Will this work with > > on-demand content? Do others see this as an issue? What do you think? > > > > @ttereshc thank you for bringing this up, and @asmacdo thank you for > > jumping in and suggesting. > > > > > > On Thu, Aug 29, 2019 at 10:20 AM Tatiana Tereshchenko > > <ttere...@redhat.com> wrote: > > > > > Austin, thank you. > > > Your option is more explicit, so it's probably better. > > > > > > Tanya > > > > > > On Wed, Aug 28, 2019 at 6:38 PM Austin Macdonald > > > <amacd...@redhat.com> wrote: > > > > > >> > > >> > > >> On Wed, Aug 28, 2019 at 11:34 AM Tatiana Tereshchenko < > > >> ttere...@redhat.com> wrote: > > >> > > >>> Bump. > > >>> > > >>> Please provide feedback if you have any. > > >>> I'll start working on the PR to make the suggested change this > > >>> week otherwise. > > >>> > > >>> Thank you, > > >>> Tanya > > >>> > > >>> On Mon, Aug 26, 2019 at 12:46 PM Tatiana Tereshchenko < > > >>> ttere...@redhat.com> wrote: > > >>> > > >>>> In RPM plugin we have Modulemd content. It comes from metadata > > >>>> as one file and we parse it and then save each modulemd as a > > >>>> separate file/artifact. > > >>>> > > >>>> The question is how to handle this content in the sync pipeline. > > >>>> Modulemd content is artifactless on a remote source (metadata) > > >>>> but it's not artifactless in Pulp, so it can't follow a standard > > >>>> path - it needs an artifact but doesn't have any remote source > > >>>> to download it from. > > >>>> > > >>>> The suggestion: find a way to skip ArtifactDownloader and > > >>>> RemoteArtifactSaver stages. > > >>>> The content and its artifact still need to go through all the > > >>>> stages, except the ones which deal with artifact's url in some > > >>>> way - download artifact (ArtifactDownloader stage) and or create > > >>>> a RemoteArtifact for downloading later (RemoteArtifactSaver > > >>>> stage). > > >>>> > > >>>> The straightforward way is just to check if DeclarativeArtifact > > >>>> has url (or check some special value) and skip the stage > > >>>> otherwise. Any concerns about this approach (apart form being > > >>>> somewhat hacky)? > > >>> > > >>>> Any other solutions to the problem? > > >>>> > > >>> > > >> That solution seems fine to me. I'll toss out another idea just to > > >> have options. > > >> > > >> New bool on Declarative Artifact. Similar to deferred downloads, > > >> certain stages are no-ops if DeclarativeArtifact.deferred_download > > >> is True. > > >> > > >> > https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/artifact_stages.py#L152 > > >> > > >> > https://github.com/pulp/pulpcore-plugin/blob/master/pulpcore/plugin/stages/models.py#L29 > > >> > > >> > > >> > > >>> Thank you, > > >>>> Tanya > > >>>> > > >>>> P.S. FWIW, Reasons to store modulemd as a file are: > > >>>> - the format is not very stable and new information can be added > > >>>> - it can be large > > >>>> - we don't need all the info in the DB, we use only small subset > > >>>> of fields (to search by or for copy operations) > > >>>> > > >>> _______________________________________________ > > >>> Pulp-dev mailing list > > >>> Pulp-dev@redhat.com > > >>> https://www.redhat.com/mailman/listinfo/pulp-dev > > >>> > > >> _______________________________________________ > > > Pulp-dev mailing list > > > Pulp-dev@redhat.com > > > https://www.redhat.com/mailman/listinfo/pulp-dev > _______________________________________________ > Pulp-dev mailing list > Pulp-dev@redhat.com > https://www.redhat.com/mailman/listinfo/pulp-dev >
_______________________________________________ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
