Re: [Pulp-dev] [Breaking] Stopping the installer from auto-creating migrations - Sept 3rd

2019-09-04 Thread Mike DePaulo 
And from the container image scripts & Dockerfile:
https://github.com/pulp/pulpcore/pull/282

*This affects* any plugins using the new plugin-template for
container-based CI.

On Tue, Sep 3, 2019 at 3:58 PM David Davis  wrote:

> The step to create migrations has been removed from the installer:
>
> https://github.com/pulp/ansible-pulp/pull/144
>
> David
>
>
> On Tue, Aug 27, 2019 at 3:31 PM Mike DePaulo 
> wrote:
>
>> Yes, just create a pulpcore PR that removes the makemigrations line, as
>> well as the line above it.
>> (Honestly, comment the line above it out. We may want the list of plugins
>> in the future.)
>>
>> On Tue, Aug 27, 2019 at 3:14 PM Brian Bouterse 
>> wrote:
>>
>>>
>>>
>>> On Tue, Aug 27, 2019 at 3:04 PM Mike DePaulo 
>>> wrote:
>>>
 Note that we makemigrations & migrate at pulp-api container start
 time. And all the containers (e.g., pulp-content) wait on the pulp-api
 container to do them:

 https://github.com/pulp/pulpcore/blob/master/containers/images/pulp/container-assets/pulp-api#L11

 Once I took over the containers, I envisioned both of these as a
 temporary measure. Presumably there would be a separate
 spordically-run container to perform migrations.

>>> Can we continue to have the container stuff "migrate" but not
>>> "makemigrations" along with this change in the installer? My goal is to
>>> keep the containers and installer consistent.
>>>
>>>
 On Tue, Aug 27, 2019 at 2:41 PM Brian Bouterse 
 wrote:
 >
 > Excellent and good catch on the duplicate already on sprint. Yes
 let's use 5321 as you suggest. I added to that issue that this was
 announced and scheduled for Sept 3rd.
 >
 > If it's still at NEW on the 3rd, I have a reminder to make a PR and
 merge on that day. Otherwise anyone can make a PR ahead of time and merge
 on the 3rd.
 >
 > On Tue, Aug 27, 2019 at 2:19 PM Austin Macdonald 
 wrote:
 >>
 >> This all sounds good to me. FYI, I closed
 https://pulp.plan.io/issues/5361 as a dupe since
 https://pulp.plan.io/issues/5321 has already been added to the sprint.
 Feel free to use 5361 and close 5321 if you prefer.
 >>
 >> On Tue, Aug 27, 2019 at 2:16 PM Brian Bouterse 
 wrote:
 >>>
 >>> tl;dr if we make this change on Sept 3rd, the installer won't
 auto-create migrations anymore. For every change needing a migration,
 please commit one.
 >>>
 >>> # Background
 >>> We enabled the installer to "auto-create" migrations as a solution
 to the problem of them changing a crazy amount early on in Pulp3's
 development. Now the migrations are expected to be checked in and I believe
 they are for all plugins. This is only a breaking change for a plugin that
 is missing migrations.
 >>>
 >>> # The Problem
 >>> https://pulp.plan.io/issues/5361 This was originally reported by
 Katello as a P2; I added details to it.
 >>>
 >>> # Feedback
 >>> If there is something better that we should do please suggest it on
 the issue. If this is a concern or not going to work for you or our users
 please bring that up anywhere.
 >>>
 >>> # Timeline
 >>> If there are no blocking concerns, I plan to make this change on
 Tuesday Sept 3rd.
 >>>
 >>> Thank you!
 >>> Brian
 >>>
 >>> ___
 >>> Pulp-dev mailing list
 >>> Pulp-dev@redhat.com
 >>> https://www.redhat.com/mailman/listinfo/pulp-dev
 >
 > ___
 > Pulp-dev mailing list
 > Pulp-dev@redhat.com
 > https://www.redhat.com/mailman/listinfo/pulp-dev



 --

 Mike DePaulo

 He / Him / His

 Service Reliability Engineer, Pulp

 Red Hat

 IM: mikedep333

 GPG: 51745404

>>>
>>
>> --
>>
>> Mike DePaulo
>>
>> He / Him / His
>>
>> Service Reliability Engineer, Pulp
>>
>> Red Hat 
>>
>> IM: mikedep333
>>
>> GPG: 51745404
>> 
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>

-- 

Mike DePaulo

He / Him / His

Service Reliability Engineer, Pulp

Red Hat 

IM: mikedep333

GPG: 51745404

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Namespacing one shot upload and copy endpoints

2019-09-04 Thread Matthias Dellweg 
I think, the pulpcore-plugin would at least provide a serializer to
derive from, that carries the either-artifact-or-fileupload logic.

On Wed, 4 Sep 2019 11:45:37 -0400
Dennis Kliban  wrote:

> On Wed, Sep 4, 2019 at 11:12 AM Brian Bouterse 
> wrote:
> 
> > I want to bring back a variation on upload needs that has come out
> > of various discussions w/ several plugin developers. I wrote it up
> > here in more detail and I'll sumarize below:
> > https://pulp.plan.io/issues/5403
> >
> > 1. Make all uploads of a specific content type live at POST
> > /pulp/api/v3/content///
> >  
> 
> +1 ... However, what are we going to provide in the plugin template
> for this? Right now we provide a ModelViewSet. I am picturing this
> would be a default implementation from pulpcore-plugin that would not
> extract any information from the file, but would simply create an
> instance of the *Content model and a ContentArtifact with a relative
> path.
> 
> Is that what you had in mind?
> 
> 
> > 2. Have it accept either binary data (to create an Artifact from
> > before the Content unit) OR a reference to an existing Artifact
> > (allowing the chunked upload API to be used) but not both.
> >
> > What do you think?
> >
> >
> >
> > On Wed, Aug 14, 2019 at 12:32 PM Ina Panova 
> > wrote: 
> >> There have been ongoing couple of more discussions offline
> >> regarding copy/remove and it kinda boiled down to:
> >>
> >> v3///action/  which will allow filtering
> >> v3///  will not allow filtering
> >>
> >> 
> >> Regards,
> >>
> >> Ina Panova
> >> Senior Software Engineer| Pulp| Red Hat Inc.
> >>
> >> "Do not go where the path may lead,
> >>  go instead where there is no path and leave a trail."
> >>
> >>
> >> On Thu, Aug 1, 2019 at 8:31 PM Daniel Alley 
> >> wrote: 
> >>> Ina and I discussed this earlier today and she made good points
> >>> -- she pointed out that what I suggested wouldn't work for
> >>> content types that have relations on non-content models, as
> >>> several content types in the Docker and Python plugins do. So, I
> >>> no longer think it's a good idea to have an endpoint with
> >>> "types=[]" in core.
> >>>
> >>> However, I still think that the "types=[]" pattern doesn't
> >>> compose very well with other features like filtering so I'm not
> >>> sure if we want to adopt that yet.
> >>>
> >>> On Thu, Aug 1, 2019 at 8:54 AM Ina Panova 
> >>> wrote: 
>  For the one shot upload usecase I tend to lean towards
>  /v3///upload/ and for content_types that
>  require special treatment we can define separate endpoints.
>  If talking about modulemd or modulemd_defaults, it could be
>  /v3/rpm/modules/upload/.
> 
>  
>  Regards,
> 
>  Ina Panova
>  Senior Software Engineer| Pulp| Red Hat Inc.
> 
>  "Do not go where the path may lead,
>   go instead where there is no path and leave a trail."
> 
> 
>  On Wed, Jul 31, 2019 at 1:04 PM Tatiana Tereshchenko <  
>  ttere...@redhat.com> wrote:  
>   
> > If the goal is to make endpoints unified across all actions,
> > then I think we can only do
> > POST /pulp/api/v3//plugin/action/ types=[]
> >
> > Having plugin/content_type/upload would be nice, however I'm
> > not sure if it covers enough use cases.
> > E.g. For pulp_rpm,  it makes sense for packages or advisories
> > to have a dedicated endpoint each, however it doesn't make much
> > sense for modulemd or modulemd_defaults, because usually they
> > are in the same file and uploaded in bulk (maybe a separate
> > endpoint is needed for this case).
> >
> > For the copy case, it's common to copy more than one type, I
> > think, so probably 'plugin/copy/ types=[]' makes more sense.
> >
> > It would be great to here from more people and other plugins.
> >
> >
> >
> > On Mon, Jul 29, 2019 at 5:46 PM Pavel Picka 
> > wrote: 
> >> +1 for discuss this to keep some standard as I have already
> >> opened PRs for rpm modulemd[-defaults].
> >> I like idea of /upload in the end.
> >> But also think it can work without as it will be differ by
> >> POST/GET methods.
> >>
> >> On Mon, Jul 29, 2019 at 4:49 PM Dana Walker
> >>  wrote:
> >>  
> >>> Just to provide an added data point, I'll be merging the
> >>> one-shot PR for pulp_python soon and it currently uses
> >>> /api/v3/python/upload/
> >>>
> >>> I wanted to keep it simple as well, and so would be happy to
> >>> change it for consistency based on whatever we decide.
> >>>
> >>> --Dana
> >>>
> >>> Dana Walker
> >>>
> >>> She / Her / Hers
> >>>
> >>> Software Engineer, Pulp Project
> >>>
> >>> Red Hat 
> >>>
> >>> dawal...@redhat.com
> >>> 
> >>>
> >>>
> >>>
> >>> On Mon, Jul 29, 2019 at 10:42 AM Ina Panova
> >>>  wrote:
> >>>  
>

Re: [Pulp-dev] Namespacing one shot upload and copy endpoints

2019-09-04 Thread Dennis Kliban 
On Wed, Sep 4, 2019 at 11:12 AM Brian Bouterse  wrote:

> I want to bring back a variation on upload needs that has come out of
> various discussions w/ several plugin developers. I wrote it up here in
> more detail and I'll sumarize below:  https://pulp.plan.io/issues/5403
>
> 1. Make all uploads of a specific content type live at POST
> /pulp/api/v3/content///
>

+1 ... However, what are we going to provide in the plugin template for
this? Right now we provide a ModelViewSet. I am picturing this would be a
default implementation from pulpcore-plugin that would not extract any
information from the file, but would simply create an instance of the
*Content model and a ContentArtifact with a relative path.

Is that what you had in mind?


> 2. Have it accept either binary data (to create an Artifact from before
> the Content unit) OR a reference to an existing Artifact (allowing the
> chunked upload API to be used) but not both.
>
> What do you think?
>
>
>
> On Wed, Aug 14, 2019 at 12:32 PM Ina Panova  wrote:
>
>> There have been ongoing couple of more discussions offline regarding
>> copy/remove and it kinda boiled down to:
>>
>> v3///action/  which will allow filtering
>> v3///  will not allow filtering
>>
>> 
>> Regards,
>>
>> Ina Panova
>> Senior Software Engineer| Pulp| Red Hat Inc.
>>
>> "Do not go where the path may lead,
>>  go instead where there is no path and leave a trail."
>>
>>
>> On Thu, Aug 1, 2019 at 8:31 PM Daniel Alley  wrote:
>>
>>> Ina and I discussed this earlier today and she made good points -- she
>>> pointed out that what I suggested wouldn't work for content types that have
>>> relations on non-content models, as several content types in the Docker and
>>> Python plugins do. So, I no longer think it's a good idea to have an
>>> endpoint with "types=[]" in core.
>>>
>>> However, I still think that the "types=[]" pattern doesn't compose very
>>> well with other features like filtering so I'm not sure if we want to adopt
>>> that yet.
>>>
>>> On Thu, Aug 1, 2019 at 8:54 AM Ina Panova  wrote:
>>>
 For the one shot upload usecase I tend to lean towards
 /v3///upload/ and for content_types that require
 special treatment we can define separate endpoints.
 If talking about modulemd or modulemd_defaults, it could be
 /v3/rpm/modules/upload/.

 
 Regards,

 Ina Panova
 Senior Software Engineer| Pulp| Red Hat Inc.

 "Do not go where the path may lead,
  go instead where there is no path and leave a trail."


 On Wed, Jul 31, 2019 at 1:04 PM Tatiana Tereshchenko <
 ttere...@redhat.com> wrote:

> If the goal is to make endpoints unified across all actions, then I
> think we can only do
> POST /pulp/api/v3//plugin/action/ types=[]
>
> Having plugin/content_type/upload would be nice, however I'm not sure
> if it covers enough use cases.
> E.g. For pulp_rpm,  it makes sense for packages or advisories to have
> a dedicated endpoint each, however it doesn't make much sense for modulemd
> or modulemd_defaults, because usually they are in the same file and
> uploaded in bulk (maybe a separate endpoint is needed for this case).
>
> For the copy case, it's common to copy more than one type, I think, so
> probably 'plugin/copy/ types=[]' makes more sense.
>
> It would be great to here from more people and other plugins.
>
>
>
> On Mon, Jul 29, 2019 at 5:46 PM Pavel Picka  wrote:
>
>> +1 for discuss this to keep some standard as I have already opened
>> PRs for rpm modulemd[-defaults].
>> I like idea of /upload in the end.
>> But also think it can work without as it will be differ by POST/GET
>> methods.
>>
>> On Mon, Jul 29, 2019 at 4:49 PM Dana Walker 
>> wrote:
>>
>>> Just to provide an added data point, I'll be merging the one-shot PR
>>> for pulp_python soon and it currently uses /api/v3/python/upload/
>>>
>>> I wanted to keep it simple as well, and so would be happy to change
>>> it for consistency based on whatever we decide.
>>>
>>> --Dana
>>>
>>> Dana Walker
>>>
>>> She / Her / Hers
>>>
>>> Software Engineer, Pulp Project
>>>
>>> Red Hat 
>>>
>>> dawal...@redhat.com
>>> 
>>>
>>>
>>>
>>> On Mon, Jul 29, 2019 at 10:42 AM Ina Panova 
>>> wrote:
>>>
 Hi all,
 As of today, plugins have the freedom to define whichever endpoints
 they want ( to some extent).
 This leads to the question - shall we namespace one-shot upload and
 copy endpoints for some consistency?

 POST /api/v3/content/rpm/packages/upload/
 POST /api/v3/content/rpm/packages/copy/

 or

 POST /api/v3/content/rpm/upload/ type =package
 POST /api/v3/content/rpm/copy/ type = [package,

Re: [Pulp-dev] Announcing the plugin-template change to use containers for CI

2019-09-04 Thread Mike DePaulo 
On Wed, Sep 4, 2019 at 11:14 AM Dennis Kliban  wrote:

> On Wed, Sep 4, 2019 at 11:11 AM Mike DePaulo 
> wrote:
>
>> *Why?*
>> pulp_rpm, and likely other plugins in the future, have C dependencies.
>> They are often difficult to satisfy on Ubuntu's LTS releases that Travis
>> provides. Since we are developing containers and a Kubernetes operator
>> anyway, we decided to migrate plugin-template's Travis CI scripts to using
>> them. These containers are currently based on Fedora 30, but will likely
>> later be based on CentOS 7.7 or 8.0, which contain python 3.6.
>> This is task #5004 .
>>
>> *Overview:*
>> 1. During install.sh, the container image named like "pulp_file" is built
>> according to pulpcore/containers/
>> 2. During install.sh, k3s, the lightweight kubernetes implementation, is
>> installed & configured on the Travis Ubuntu VM, which already has docker.
>> 3. During install.sh, pulp-operator is deployed (up.sh), which in turn
>> deploys the postgres database, redis, and the pulp_* image you built. It
>> spins the single image up 4 separate containers (services/processes):
>> pulp-api, pulp-content, pulp-resource-manager, and pulp-worker .
>> 4. During script.sh, we install some commonly needed testing tools into
>> the pulp-api container by prefixing the install command with $CMD_PREFIX .
>> So long as this container never gets restarted, they will remain installed
>> in it. These tools are not available to the other containers.
>> 5. We use $CMD_PREFIX to run the unit tests
>> 6. From the Travis Ubuntu VM, we run pulp-smash functional tests, build
>> docs, or use bindings, etc.
>> 7. Note that many other pieces of the puzzle were modified, including
>> pulp-smash, to make this integration all work.
>>
>> *Changes you may need to make:*
>> (in addition to the usual plugin-template commands):
>> 1. Ensure that in template_config.yml, "plugin_name" is set to a value
>> like "pulp_file" rather than "file"
>> 2. If you write out ~/.netrc , the pulp application password has been
>> changed from "admin" to "password" (which other pulp test / development
>> code uses.)
>>
>
> You only need to make this change if you are continuing to use
> ansible-pulp to deploy pulp on Travis. Otherwise, the containers built
> using the latest Travis config will have the right password.
>

Actually, I encountered this during pulp_file's docs tests, which did not
use ansible-pulp. Its docs/_scripts/
 shell scripts
(which get embedded in the docs) are tested via httpie (`http`), and
therefore used the .netrc.
I suppose you could use ansible-pulp in your manually-written
.travis/{pre,post}*.sh scripts to install pulp on the Travis VM (which is
the container host), but I do not recommend it. The templated .travis/*.sh
scripts would still always deploy it in containers also. And I set things
like PYTHONPATH so you don't need to call ansible-pulp for that specific
reason.

>
>
>> 3. git rm .travis/{playbook,postgres,mariadb}.yml
>>
>
4. Append to PYTHONPATH (do not override it, your plugin's dir is in it. It
is set in .travis/script.sh).

>
>> *Planned improvements include:*
>> 1. Speeding up in the image build, largely through 3.
>> 2. Publishing images to quay.io. #5062 
>> 3. Only building your pulp_* image from scratch if you have a required PR
>> for pulpcore or pulpcore-plugin (or pulp-certguard.) Otherwise, your image
>> will be layered on top of the "pulpcore" image. #5062
>> 
>> 4. re-introducing python 3.6 testing alongside 3.7 (Fedora 30 is Python
>> 3.7 based)
>> 5. re-introducing codecov
>> 6. (possible / long-term): Improving how testing packages are installed. 
>> #5404
>> 
>>
>> *Feedback wanted on:*
>> 1. Any confusing/unclear output.
>> 2. Any reliability issues.
>>
>>
>> --
>>
>> Mike DePaulo
>>
>> He / Him / His
>>
>> Service Reliability Engineer, Pulp
>>
>> Red Hat 
>>
>> IM: mikedep333
>>
>> GPG: 51745404
>> 
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>

-- 

Mike DePaulo

He / Him / His

Service Reliability Engineer, Pulp

Red Hat 

IM: mikedep333

GPG: 51745404

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Announcing the plugin-template change to use containers for CI

2019-09-04 Thread Dennis Kliban 
On Wed, Sep 4, 2019 at 11:11 AM Mike DePaulo  wrote:

> *Why?*
> pulp_rpm, and likely other plugins in the future, have C dependencies.
> They are often difficult to satisfy on Ubuntu's LTS releases that Travis
> provides. Since we are developing containers and a Kubernetes operator
> anyway, we decided to migrate plugin-template's Travis CI scripts to using
> them. These containers are currently based on Fedora 30, but will likely
> later be based on CentOS 7.7 or 8.0, which contain python 3.6.
> This is task #5004 .
>
> *Overview:*
> 1. During install.sh, the container image named like "pulp_file" is built
> according to pulpcore/containers/
> 2. During install.sh, k3s, the lightweight kubernetes implementation, is
> installed & configured on the Travis Ubuntu VM, which already has docker.
> 3. During install.sh, pulp-operator is deployed (up.sh), which in turn
> deploys the postgres database, redis, and the pulp_* image you built. It
> spins the single image up 4 separate containers (services/processes):
> pulp-api, pulp-content, pulp-resource-manager, and pulp-worker .
> 4. During script.sh, we install some commonly needed testing tools into
> the pulp-api container by prefixing the install command with $CMD_PREFIX .
> So long as this container never gets restarted, they will remain installed
> in it. These tools are not available to the other containers.
> 5. We use $CMD_PREFIX to run the unit tests
> 6. From the Travis Ubuntu VM, we run pulp-smash functional tests, build
> docs, or use bindings, etc.
> 7. Note that many other pieces of the puzzle were modified, including
> pulp-smash, to make this integration all work.
>
> *Changes you may need to make:*
> (in addition to the usual plugin-template commands):
> 1. Ensure that in template_config.yml, "plugin_name" is set to a value
> like "pulp_file" rather than "file"
> 2. If you write out ~/.netrc , the pulp application password has been
> changed from "admin" to "password" (which other pulp test / development
> code uses.)
>

You only need to make this change if you are continuing to use ansible-pulp
to deploy pulp on Travis. Otherwise, the containers built using the latest
Travis config will have the right password.


> 3. git rm .travis/{playbook,postgres,mariadb}.yml
>
> *Planned improvements include:*
> 1. Speeding up in the image build, largely through 3.
> 2. Publishing images to quay.io. #5062 
> 3. Only building your pulp_* image from scratch if you have a required PR
> for pulpcore or pulpcore-plugin (or pulp-certguard.) Otherwise, your image
> will be layered on top of the "pulpcore" image. #5062
> 
> 4. re-introducing python 3.6 testing alongside 3.7 (Fedora 30 is Python
> 3.7 based)
> 5. re-introducing codecov
> 6. (possible / long-term): Improving how testing packages are installed. #5404
> 
>
> *Feedback wanted on:*
> 1. Any confusing/unclear output.
> 2. Any reliability issues.
>
>
> --
>
> Mike DePaulo
>
> He / Him / His
>
> Service Reliability Engineer, Pulp
>
> Red Hat 
>
> IM: mikedep333
>
> GPG: 51745404
> 
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Namespacing one shot upload and copy endpoints

2019-09-04 Thread Brian Bouterse 
I want to bring back a variation on upload needs that has come out of
various discussions w/ several plugin developers. I wrote it up here in
more detail and I'll sumarize below:  https://pulp.plan.io/issues/5403

1. Make all uploads of a specific content type live at POST
/pulp/api/v3/content///
2. Have it accept either binary data (to create an Artifact from before the
Content unit) OR a reference to an existing Artifact (allowing the chunked
upload API to be used) but not both.

What do you think?



On Wed, Aug 14, 2019 at 12:32 PM Ina Panova  wrote:

> There have been ongoing couple of more discussions offline regarding
> copy/remove and it kinda boiled down to:
>
> v3///action/  which will allow filtering
> v3///  will not allow filtering
>
> 
> Regards,
>
> Ina Panova
> Senior Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
>
> On Thu, Aug 1, 2019 at 8:31 PM Daniel Alley  wrote:
>
>> Ina and I discussed this earlier today and she made good points -- she
>> pointed out that what I suggested wouldn't work for content types that have
>> relations on non-content models, as several content types in the Docker and
>> Python plugins do. So, I no longer think it's a good idea to have an
>> endpoint with "types=[]" in core.
>>
>> However, I still think that the "types=[]" pattern doesn't compose very
>> well with other features like filtering so I'm not sure if we want to adopt
>> that yet.
>>
>> On Thu, Aug 1, 2019 at 8:54 AM Ina Panova  wrote:
>>
>>> For the one shot upload usecase I tend to lean towards
>>> /v3///upload/ and for content_types that require
>>> special treatment we can define separate endpoints.
>>> If talking about modulemd or modulemd_defaults, it could be
>>> /v3/rpm/modules/upload/.
>>>
>>> 
>>> Regards,
>>>
>>> Ina Panova
>>> Senior Software Engineer| Pulp| Red Hat Inc.
>>>
>>> "Do not go where the path may lead,
>>>  go instead where there is no path and leave a trail."
>>>
>>>
>>> On Wed, Jul 31, 2019 at 1:04 PM Tatiana Tereshchenko <
>>> ttere...@redhat.com> wrote:
>>>
 If the goal is to make endpoints unified across all actions, then I
 think we can only do
 POST /pulp/api/v3//plugin/action/ types=[]

 Having plugin/content_type/upload would be nice, however I'm not sure
 if it covers enough use cases.
 E.g. For pulp_rpm,  it makes sense for packages or advisories to have a
 dedicated endpoint each, however it doesn't make much sense for modulemd or
 modulemd_defaults, because usually they are in the same file and uploaded
 in bulk (maybe a separate endpoint is needed for this case).

 For the copy case, it's common to copy more than one type, I think, so
 probably 'plugin/copy/ types=[]' makes more sense.

 It would be great to here from more people and other plugins.



 On Mon, Jul 29, 2019 at 5:46 PM Pavel Picka  wrote:

> +1 for discuss this to keep some standard as I have already opened PRs
> for rpm modulemd[-defaults].
> I like idea of /upload in the end.
> But also think it can work without as it will be differ by POST/GET
> methods.
>
> On Mon, Jul 29, 2019 at 4:49 PM Dana Walker 
> wrote:
>
>> Just to provide an added data point, I'll be merging the one-shot PR
>> for pulp_python soon and it currently uses /api/v3/python/upload/
>>
>> I wanted to keep it simple as well, and so would be happy to change
>> it for consistency based on whatever we decide.
>>
>> --Dana
>>
>> Dana Walker
>>
>> She / Her / Hers
>>
>> Software Engineer, Pulp Project
>>
>> Red Hat 
>>
>> dawal...@redhat.com
>> 
>>
>>
>>
>> On Mon, Jul 29, 2019 at 10:42 AM Ina Panova 
>> wrote:
>>
>>> Hi all,
>>> As of today, plugins have the freedom to define whichever endpoints
>>> they want ( to some extent).
>>> This leads to the question - shall we namespace one-shot upload and
>>> copy endpoints for some consistency?
>>>
>>> POST /api/v3/content/rpm/packages/upload/
>>> POST /api/v3/content/rpm/packages/copy/
>>>
>>> or
>>>
>>> POST /api/v3/content/rpm/upload/ type =package
>>> POST /api/v3/content/rpm/copy/ type = [package, modulemd]
>>>
>>> I wanted to bring this up, before it diverges a lot. For the record,
>>> I have checked only RPM plugin, I am not aware of the state of the other
>>> plugins.
>>> Right now we have an active endpoint for one-shot upload of rpm
>>> package:
>>> POST /api/v3/content/rpm/upload/
>>>
>>> And there is PR for one-shot upload of modulemd-defaults:
>>> POST /api/v3/content/rpm/modulemd-defaults/
>>>
>>> For rpm copy we have POST /api/v3/content/rpm/copy/ types=[]
>>>
>>> We are starting some work on docker

[Pulp-dev] Announcing the plugin-template change to use containers for CI

2019-09-04 Thread Mike DePaulo 
*Why?*
pulp_rpm, and likely other plugins in the future, have C dependencies. They
are often difficult to satisfy on Ubuntu's LTS releases that Travis
provides. Since we are developing containers and a Kubernetes operator
anyway, we decided to migrate plugin-template's Travis CI scripts to using
them. These containers are currently based on Fedora 30, but will likely
later be based on CentOS 7.7 or 8.0, which contain python 3.6.
This is task #5004 .

*Overview:*
1. During install.sh, the container image named like "pulp_file" is built
according to pulpcore/containers/
2. During install.sh, k3s, the lightweight kubernetes implementation, is
installed & configured on the Travis Ubuntu VM, which already has docker.
3. During install.sh, pulp-operator is deployed (up.sh), which in turn
deploys the postgres database, redis, and the pulp_* image you built. It
spins the single image up 4 separate containers (services/processes):
pulp-api, pulp-content, pulp-resource-manager, and pulp-worker .
4. During script.sh, we install some commonly needed testing tools into the
pulp-api container by prefixing the install command with $CMD_PREFIX . So
long as this container never gets restarted, they will remain installed in
it. These tools are not available to the other containers.
5. We use $CMD_PREFIX to run the unit tests
6. From the Travis Ubuntu VM, we run pulp-smash functional tests, build
docs, or use bindings, etc.
7. Note that many other pieces of the puzzle were modified, including
pulp-smash, to make this integration all work.

*Changes you may need to make:*
(in addition to the usual plugin-template commands):
1. Ensure that in template_config.yml, "plugin_name" is set to a value like
"pulp_file" rather than "file"
2. If you write out ~/.netrc , the pulp application password has been
changed from "admin" to "password" (which other pulp test / development
code uses.)
3. git rm .travis/{playbook,postgres,mariadb}.yml

*Planned improvements include:*
1. Speeding up in the image build, largely through 3.
2. Publishing images to quay.io. #5062 
3. Only building your pulp_* image from scratch if you have a required PR
for pulpcore or pulpcore-plugin (or pulp-certguard.) Otherwise, your image
will be layered on top of the "pulpcore" image. #5062

4. re-introducing python 3.6 testing alongside 3.7 (Fedora 30 is Python 3.7
based)
5. re-introducing codecov
6. (possible / long-term): Improving how testing packages are installed. #5404


*Feedback wanted on:*
1. Any confusing/unclear output.
2. Any reliability issues.


-- 

Mike DePaulo

He / Him / His

Service Reliability Engineer, Pulp

Red Hat 

IM: mikedep333

GPG: 51745404

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev