I am +1. Question: if we decide to keep (a), is method (b) mostly an additive change that can be addressed in pulpcore post-RC?
I’m imagining that during the RC phase plugins can write their own one-shot uploaders and then after that, we can standardize on a solution we can move into pulpcore. David On Tue, Feb 19, 2019 at 12:30 PM Brian Bouterse <bbout...@redhat.com> wrote: > Overall +1, we should make a 1-shot uploader. It will need a bit of code > from plugin writers to make it work though. > > I believe the "manual and generic" feature offerings we have now are > pretty good. I don't think we should take them away. > > To describe yet another related use case; consider the uploading of a > tarball with a lot of units in it (maybe a whole repo). For instance bunch > of Ansible roles in a tarball is what a user would want to upload to Pulp, > so you would want to upload the whole tarball at once. I see three cases: > > a) The generic method (what we already have). The user supplies all > metadata because we can't know the right attributes without them telling it > to us. > b) The one-shot uploader to create a single content unit from a single > Artifact. I think this is what is being proposed here. +1 to this. > c) Uploading a bunch of content units in one binary, e.g. a tarball. We > should leave this to plugin writers to implement for now. > > In terms of implementing (b) I believe it would be useful to have a > Content unit provide an optional class method that will instantiate a > content unit (but not save it) from an Artifact. Then core can offer an API > that accepts a single Artifact binary and what content unit type it is and > it should create+save the Artifact and then create+save the Content unit > from that Artifact. > > This doesn't address adding it to/from repositories. That would be ok too > as an additional option, but you probably want to do it to a whole bunch at > once somehow. Also it will need to go through the tasking system for > correctness. > > > On Tue, Feb 19, 2019 at 11:52 AM David Davis <davidda...@redhat.com> > wrote: > >> What about the case of multi-artifact content? Don’t they require >> separate artifact creation and content creation routes? >> >> David >> >> >> On Tue, Feb 19, 2019 at 11:40 AM Austin Macdonald <aus...@redhat.com> >> wrote: >> >>> I think the key question to ask is: >>> What circumstances would require the creation of Content that would not >>> be met by a one-shot upload? >>> >>> On Tue, Feb 19, 2019 at 11:34 AM Daniel Alley <dal...@redhat.com> wrote: >>> >>>> @Matthias why would you prefer to keep the normal create? As you >>>> mention, the "orphan cleanup" issue applies to both, so there's no >>>> advantage given to the former. >>>> >>>> The normal "create" ( POST .../content/plugin/type/ ...) is unidiomatic >>>> and inconsistent, because the fields needed to upload a content are >>>> different from the fields on the actual serializer. Most content types >>>> keep metadata inside the packages themselves, so you can't let the user >>>> specify the content field values, you have to contort everything so that >>>> instead of hydrating the serializer from user input, it does so by parsing >>>> the content. >>>> >>>> There's also the issue that the libraries we're using to parse the >>>> (Python and RPM) packages do some validation on the filename to see that it >>>> has the right extension and so forth, and since artifacts are >>>> content-addressed and don't store that information, with normal create you >>>> have to pass the filename of the original artifact *back in* at the time >>>> you create it, and then copy the file from Pulp content storage into a temp >>>> directory under a filename which will validate properly, which is highly >>>> unfortunate. >>>> >>>> With one-shot upload, you avoid both of those problems, because there's >>>> no broken expectations as to what fields should be accepted, and because it >>>> should be possible (though I haven't tried it) to parse the original file >>>> *before* saving it as an artifact, thus avoiding a lot of mess. And you >>>> also get the option to add it straight into a repository. In my opinion, >>>> it's a straight upgrade. >>>> >>>> On Tue, Feb 19, 2019 at 10:57 AM Matthias Dellweg <dell...@atix.de> >>>> wrote: >>>> >>>>> I have no objections to having the "one shot upload" or even "one shot >>>>> upload into repositoryversion". I think, i would like to keep the >>>>> 'traditional' create anyway. >>>>> The problem i see with create and one shot upload is, that another >>>>> thing could have triggered 'delete orphans' at the wrong time, and you >>>>> shiny new content unit disappears, before you can add it to a >>>>> repository version. >>>>> >>>>> On Mon, 18 Feb 2019 14:41:54 -0500 >>>>> Austin Macdonald <aus...@redhat.com> wrote: >>>>> >>>>> > Originally, our upload story was as follows: >>>>> > The user will upload a new file to Pulp via POST to /artifacts/ >>>>> > (provided by core) >>>>> > The user will create a new plugin specific Content via POST to >>>>> > /path/to/plugin/content/, referencing whatever artifacts that are >>>>> > contained, and whatever fields are expected for the new content. >>>>> > The user will add the new content to a repository via POST to >>>>> > /repositories/1/versions/ >>>>> > >>>>> > However, this is somewhat cumbersome to the user with 3 API calls to >>>>> > accomplish something that only took one call in Pulp 2. >>>>> > >>>>> > There are a couple of different paths plugins have taken to improve >>>>> > the user experience: >>>>> > The Python plugin follows the above workflow, but reads the Artifact >>>>> > file to determine the values for the fields. The RPM plugin has gone >>>>> > even farther and created a new endpoint for "one shot" upload that >>>>> > perform all of this in a single call. I think it is likely that the >>>>> > Python plugin will move more in the "one shot" direction, and other >>>>> > plugins will probably follow. >>>>> > >>>>> > That said, I think we should discuss this as a community to encourage >>>>> > plugins to behave similarly, and because there may also be a >>>>> > possibility for sharing some of code. It is my hope that a "one shot >>>>> > upload" could do 2 things: 1) Upload and create Content. 2) >>>>> > Optionally add that content to repositories. >>>>> _______________________________________________ >>>>> Pulp-dev mailing list >>>>> Pulp-dev@redhat.com >>>>> https://www.redhat.com/mailman/listinfo/pulp-dev >>>>> >>>> _______________________________________________ >>> Pulp-dev mailing list >>> Pulp-dev@redhat.com >>> https://www.redhat.com/mailman/listinfo/pulp-dev >>> >> _______________________________________________ >> Pulp-dev mailing list >> Pulp-dev@redhat.com >> https://www.redhat.com/mailman/listinfo/pulp-dev >> >
_______________________________________________ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
