Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

What still makes me feel most comfortable with the plug-in--centered
tasks proposal is that it doesn't try to conceal what actually happens
behind the scenes: a plug-in specific task code is fired all the time
a new repository version is created, that uses multiple (generic) core
Pulp objects to work on/with as mere URLs rather than verbose json
blobs passed around, and that no repository version gets created
without a plug-in involvement.


On Tue, Apr 3, 2018 at 6:01 PM, David Davis  wrote:
> I’ve had a chance to think about this some more this week and reread all the
> emails. I think that the solution of just adding a hook is best. Each
> solution seems to be imperfect and I think we still want users to interact
> with objects (remotes, repositories, etc). I’d say I’m +1 on adding a hook
> and -0 on the other proposals.

I wonder how the hook triggers would look like: what part of the core
would be responsible to select proper plug-in based on what criteria?
What if we later on need more action endpoints for instance repository
version "fork/clone"? Should we add more hooks then?
What if more fine-grained control of particular plug-in is required
for particular action endpoint? Should the plug-in hook just `if
endpoint == sync... elif endpoint == publish ... else...` it's way
thru?
How about custom plug-in action endpoint configuration? Should that be
passed-thru the generic core action endpoint? How that would be
auto-documented?
What if e.g pulp_docker doesn't need particular action endpoint, is it
cool to raise a 400? Isn't cleaner for both the plug-in writer and the
user not to have to expose such an endpoint in first place?
What about the plug-in custom async action endpoints that happen to
have to create a repository version?

Why workaround a design that doesn't seem to fit?

Cheers,
milan

>
>
> David
>
> On Mon, Apr 2, 2018 at 5:02 PM, Brian Bouterse  wrote:
>>
>> @asmacdo and I talked some and I wanted to share a few of my thoughts on
>> the plugin tasks problem statements.
>>
>> I agree with a problem statement for pulp_docker for example that it would
>> be good for a plugin writer to add custom validation when creating a new
>> repo/version. I thought this idea was stated already, but I'll retell what I
>> had thought we would do to add plugin writer validation. We would make an
>> optional plugin writer hook that validates the entire repository when the
>> RepositoryVersion is being made complete. Also this would also allow a
>> plugin to "disable" the repoversion endpoint by always raising an exception
>> here.
>>
>> The documentation convention is also a good outcome. It would recommend
>> that plugin writers put their views in a url namespaced by their plugin
>> name. I think we should do that.
>>
>> The other main problem I've read about with what we currently have is that
>> the actions urls (sync, publish, export, custom views) would be all in
>> different areas of the API urls. This is identified as a problem, but I
>> think this is ok. It allows us to explain each of those parts of pulp
>> separately, which makes the understanding of the API still pretty easy.
>>
>> I also believe another tertiary problem identified is that the 'sync',
>> 'publish', and 'export' words are not restful and it would be good to
>> resolve that somehow. A restful API would be an easier API to use because
>> REST clients already know how to interact with a resource. Action endpoints
>> kind of break this.
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Apr 2, 2018 at 2:54 PM, Austin Macdonald 
>> wrote:
>>>
>>> Thanks for the classifications.
>>>
>>> On Mon, Apr 2, 2018 at 2:37 PM, Dennis Kliban  wrote:
>>>
 I think of these actions as 2 new types of resource in Pulp. Unlike
 Remotes(Importers currently) and Content, these resources are singletons
 that each plugin provides. Since users don't need to create new instances 
 of
 these resources it makes sense that they are represented by a View instead
 of a ViewSet. Though we don't even need to explain that to plugin writers.
 We just need to tell them that all operations that their plugin provides 
 for
 users need to be sublassed from ActionView.

 From the users point of view a GET on /api/v3/versionsactions/ to find a
 versionaction href is the same as performing a GET on /api/v3/remotes/ to
 find the href for a specific remote to sync from.

 Auto documentation would work perfectly.

>>>
>>> Its not that it would be "broken" or "wrong". I just think that users
>>> expect the documentation to include behavior and parameters-- ideally, there
>>> should be enough information to create a request. In this case however, the
>>> documentation just explains how to retrieve the necessary information. It
>>> works, but I don't think it is ideal that the user needs to make other GET
>>> requests to learn how to use an API endpoint.
>>>
>>> ___
>>> Pulp-dev mailing list

Re: [Pulp-dev] Plugin relationship to tasks

[Dennis Kliban]

Here is where I stand:

-1 on converting the task status API to task creation API (the original
proposal)
+1 on adding a hook for plugins to provide repository version validation of
content
+1 on exploring other ideas to make the v3 API more RESTful (removing the
action end points such as 'sync' and 'publish')

On Tue, Apr 3, 2018 at 12:01 PM, David Davis  wrote:

> I’ve had a chance to think about this some more this week and reread all
> the emails. I think that the solution of just adding a hook is best. Each
> solution seems to be imperfect and I think we still want users to interact
> with objects (remotes, repositories, etc). I’d say I’m +1 on adding a hook
> and -0 on the other proposals.
>
>
> David
>
> On Mon, Apr 2, 2018 at 5:02 PM, Brian Bouterse 
> wrote:
>
>> @asmacdo and I talked some and I wanted to share a few of my thoughts on
>> the plugin tasks problem statements.
>>
>> I agree with a problem statement for pulp_docker for example that it
>> would be good for a plugin writer to add custom validation when creating a
>> new repo/version. I thought this idea was stated already, but I'll retell
>> what I had thought we would do to add plugin writer validation. We would
>> make an optional plugin writer hook that validates the entire repository
>> when the RepositoryVersion is being made complete. Also this would also
>> allow a plugin to "disable" the repoversion endpoint by always raising an
>> exception here.
>>
>> The documentation convention is also a good outcome. It would recommend
>> that plugin writers put their views in a url namespaced by their plugin
>> name. I think we should do that.
>>
>> The other main problem I've read about with what we currently have is
>> that the actions urls (sync, publish, export, custom views) would be all in
>> different areas of the API urls. This is identified as a problem, but I
>> think this is ok. It allows us to explain each of those parts of pulp
>> separately, which makes the understanding of the API still pretty easy.
>>
>> I also believe another tertiary problem identified is that the 'sync',
>> 'publish', and 'export' words are not restful and it would be good to
>> resolve that somehow. A restful API would be an easier API to use because
>> REST clients already know how to interact with a resource. Action endpoints
>> kind of break this.
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Apr 2, 2018 at 2:54 PM, Austin Macdonald 
>> wrote:
>>
>>> Thanks for the classifications.
>>>
>>> On Mon, Apr 2, 2018 at 2:37 PM, Dennis Kliban 
>>> wrote:
>>>
>>> I think of these actions as 2 new types of resource in Pulp. Unlike
 Remotes(Importers currently) and Content, these resources are singletons
 that each plugin provides. Since users don't need to create new instances
 of these resources it makes sense that they are represented by a View
 instead of a ViewSet. Though we don't even need to explain that to plugin
 writers. We just need to tell them that all operations that their plugin
 provides for users need to be sublassed from ActionView.

 From the users point of view a GET on /api/v3/versionsactions/ to find
 a versionaction href is the same as performing a GET on /api/v3/remotes/ to
 find the href for a specific remote to sync from.

 Auto documentation would work perfectly.


>>> Its not that it would be "broken" or "wrong". I just think that users
>>> expect the documentation to include behavior and parameters-- ideally,
>>> there should be enough information to create a request. In this case
>>> however, the documentation just explains how to retrieve the necessary
>>> information. It works, but I don't think it is ideal that the user needs to
>>> make other GET requests to learn how to use an API endpoint.
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[David Davis]

I’ve had a chance to think about this some more this week and reread all
the emails. I think that the solution of just adding a hook is best. Each
solution seems to be imperfect and I think we still want users to interact
with objects (remotes, repositories, etc). I’d say I’m +1 on adding a hook
and -0 on the other proposals.


David

On Mon, Apr 2, 2018 at 5:02 PM, Brian Bouterse  wrote:

> @asmacdo and I talked some and I wanted to share a few of my thoughts on
> the plugin tasks problem statements.
>
> I agree with a problem statement for pulp_docker for example that it would
> be good for a plugin writer to add custom validation when creating a new
> repo/version. I thought this idea was stated already, but I'll retell what
> I had thought we would do to add plugin writer validation. We would make an
> optional plugin writer hook that validates the entire repository when the
> RepositoryVersion is being made complete. Also this would also allow a
> plugin to "disable" the repoversion endpoint by always raising an exception
> here.
>
> The documentation convention is also a good outcome. It would recommend
> that plugin writers put their views in a url namespaced by their plugin
> name. I think we should do that.
>
> The other main problem I've read about with what we currently have is that
> the actions urls (sync, publish, export, custom views) would be all in
> different areas of the API urls. This is identified as a problem, but I
> think this is ok. It allows us to explain each of those parts of pulp
> separately, which makes the understanding of the API still pretty easy.
>
> I also believe another tertiary problem identified is that the 'sync',
> 'publish', and 'export' words are not restful and it would be good to
> resolve that somehow. A restful API would be an easier API to use because
> REST clients already know how to interact with a resource. Action endpoints
> kind of break this.
>
>
>
>
>
>
>
> On Mon, Apr 2, 2018 at 2:54 PM, Austin Macdonald 
> wrote:
>
>> Thanks for the classifications.
>>
>> On Mon, Apr 2, 2018 at 2:37 PM, Dennis Kliban  wrote:
>>
>> I think of these actions as 2 new types of resource in Pulp. Unlike
>>> Remotes(Importers currently) and Content, these resources are singletons
>>> that each plugin provides. Since users don't need to create new instances
>>> of these resources it makes sense that they are represented by a View
>>> instead of a ViewSet. Though we don't even need to explain that to plugin
>>> writers. We just need to tell them that all operations that their plugin
>>> provides for users need to be sublassed from ActionView.
>>>
>>> From the users point of view a GET on /api/v3/versionsactions/ to find a
>>> versionaction href is the same as performing a GET on /api/v3/remotes/ to
>>> find the href for a specific remote to sync from.
>>>
>>> Auto documentation would work perfectly.
>>>
>>>
>> Its not that it would be "broken" or "wrong". I just think that users
>> expect the documentation to include behavior and parameters-- ideally,
>> there should be enough information to create a request. In this case
>> however, the documentation just explains how to retrieve the necessary
>> information. It works, but I don't think it is ideal that the user needs to
>> make other GET requests to learn how to use an API endpoint.
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Brian Bouterse]

@asmacdo and I talked some and I wanted to share a few of my thoughts on
the plugin tasks problem statements.

I agree with a problem statement for pulp_docker for example that it would
be good for a plugin writer to add custom validation when creating a new
repo/version. I thought this idea was stated already, but I'll retell what
I had thought we would do to add plugin writer validation. We would make an
optional plugin writer hook that validates the entire repository when the
RepositoryVersion is being made complete. Also this would also allow a
plugin to "disable" the repoversion endpoint by always raising an exception
here.

The documentation convention is also a good outcome. It would recommend
that plugin writers put their views in a url namespaced by their plugin
name. I think we should do that.

The other main problem I've read about with what we currently have is that
the actions urls (sync, publish, export, custom views) would be all in
different areas of the API urls. This is identified as a problem, but I
think this is ok. It allows us to explain each of those parts of pulp
separately, which makes the understanding of the API still pretty easy.

I also believe another tertiary problem identified is that the 'sync',
'publish', and 'export' words are not restful and it would be good to
resolve that somehow. A restful API would be an easier API to use because
REST clients already know how to interact with a resource. Action endpoints
kind of break this.







On Mon, Apr 2, 2018 at 2:54 PM, Austin Macdonald 
wrote:

> Thanks for the classifications.
>
> On Mon, Apr 2, 2018 at 2:37 PM, Dennis Kliban  wrote:
>
> I think of these actions as 2 new types of resource in Pulp. Unlike
>> Remotes(Importers currently) and Content, these resources are singletons
>> that each plugin provides. Since users don't need to create new instances
>> of these resources it makes sense that they are represented by a View
>> instead of a ViewSet. Though we don't even need to explain that to plugin
>> writers. We just need to tell them that all operations that their plugin
>> provides for users need to be sublassed from ActionView.
>>
>> From the users point of view a GET on /api/v3/versionsactions/ to find a
>> versionaction href is the same as performing a GET on /api/v3/remotes/ to
>> find the href for a specific remote to sync from.
>>
>> Auto documentation would work perfectly.
>>
>>
> Its not that it would be "broken" or "wrong". I just think that users
> expect the documentation to include behavior and parameters-- ideally,
> there should be enough information to create a request. In this case
> however, the documentation just explains how to retrieve the necessary
> information. It works, but I don't think it is ideal that the user needs to
> make other GET requests to learn how to use an API endpoint.
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

Thanks for the classifications.

On Mon, Apr 2, 2018 at 2:37 PM, Dennis Kliban  wrote:

I think of these actions as 2 new types of resource in Pulp. Unlike
> Remotes(Importers currently) and Content, these resources are singletons
> that each plugin provides. Since users don't need to create new instances
> of these resources it makes sense that they are represented by a View
> instead of a ViewSet. Though we don't even need to explain that to plugin
> writers. We just need to tell them that all operations that their plugin
> provides for users need to be sublassed from ActionView.
>
> From the users point of view a GET on /api/v3/versionsactions/ to find a
> versionaction href is the same as performing a GET on /api/v3/remotes/ to
> find the href for a specific remote to sync from.
>
> Auto documentation would work perfectly.
>
>
Its not that it would be "broken" or "wrong". I just think that users
expect the documentation to include behavior and parameters-- ideally,
there should be enough information to create a request. In this case
however, the documentation just explains how to retrieve the necessary
information. It works, but I don't think it is ideal that the user needs to
make other GET requests to learn how to use an API endpoint.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Dennis Kliban]

On Mon, Apr 2, 2018 at 12:41 PM, Austin Macdonald 
wrote:

> I'm concerned that this conversation is too broad. I'm willing to
> brainstorm ideas because this is an important topic, but I also hope that
> at some point we can come back to the proposal for master/detail tasks.
>
> This idea is interesting, because it provides a novel way to connect the
> plugins to pulpcore calls. The views that plugins implement would not be
> normal views, they would operate more like OPTIONS schema views. I'll try
> to show how I think this would look from a user's perspective:
>
> *File Sync:*
>
> The user doesn't know how to sync, so they do a GET request:
> *GET /v3/versionactions/*
>
> This returns a list of possible actions, one of them being
> v3/versionactions/file/sync. To see how to use that they do another GET
> request:
> *GET /v3versionactions/file/sync/*
>
> This returns the description of the file-sync action and the parameters
> necessary (importer, repository)
>
> *POST v3/repositories/1234/versions/* importer=importer_href,
> repository=repository_href, action=v3/versionactions/file/sync/
>
> I have some practical questions:
>
>- What does the documentation for v3/repositories/1234/versions/ look
>like?
>   - Are the parameters listed, or does the user have to use
>   v3/versionactions/sync/ to retrieve their schema?
>
> The schema for this API would be

'versionaction': 'An href for the action to be used for creating the new
repository version. List of available actions and their parameters is
available at /api/v3/versionactions/.
'parameters': 'A JSON object that contains parameter names and
corresponding values. The required parameters for each action can be
obtained using the versionactions API.

>
>- How does the RepositoryVersionsViewSet use the versionaction?
>   - How are the params `importer`, and `repository` validated?
>  - Synchronously
>  - Asynchronously
>
> Django's utils would be used to convert the versionaction href into an
instance of the view provided by the plugin.
We could provide a validate() interface in the ActionView for plugin
writers. When implemented, the method is used to perform validation on
parameters at the REST API level. Any asynchronous validation would be part
of the action() method.


>
>- How is the celery task connected and dispatched?
>
> I am not sure of implementation details, but the goal is to have pulpcore
handle this and require just the action() method that will be turned in to
a task.


> Here are my initial thoughts:
>
>- I like that this brings us back to "to create a new
>RepositoryVersion, POST to versions".
>   - I especially like that all versions are created at a single
>   endpoint.
>- I don't like that the plugins are creating views that are not
>intended to be used as endpoints. These views are only "hit" for
>documentation, and they are used as parameters to other tasks.
>- This could be confusing for users
>   - This is a different type of work than the rest of plugin writing,
>   so it will have its own learning curve
>   - This would not (speculation) play nice with auto-documentation.
>   The autodocumentation would not know about the possible parameters for 
> the
>   actions, so the users would have to user GET v3/versionactions/ *instead
>   *of docs.
>   - This would not (speculation) play nice with auto-bindings and
>   clients. Same problem as ^
>   - GET request does not retrieve objects, it retrieves schema
>
> I think of these actions as 2 new types of resource in Pulp. Unlike
Remotes(Importers currently) and Content, these resources are singletons
that each plugin provides. Since users don't need to create new instances
of these resources it makes sense that they are represented by a View
instead of a ViewSet. Though we don't even need to explain that to plugin
writers. We just need to tell them that all operations that their plugin
provides for users need to be sublassed from ActionView.

>From the users point of view a GET on /api/v3/versionsactions/ to find a
versionaction href is the same as performing a GET on /api/v3/remotes/ to
find the href for a specific remote to sync from.

Auto documentation would work perfectly.

Each */api/v3/versionsactions//* endpoint would be
documented with "GET returns detailed information about SomeAction. This
information includes the parameters that need to be specified when this
action is used to create a repository version."

The */api/v3/repositories//versions/ *endpoint would be documented
with "POST accepts a 'versionaction': 'href for action that will be used to
create a repository version' and 'parameters': 'Object where the keys are
parameter names and values are their values. These parameters are used by
the action to create a repository version. The required keys for parameters
can be obtained by performing a GET on the versionaction href'. Returns 202
with a task href."

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

I'm concerned that this conversation is too broad. I'm willing to
brainstorm ideas because this is an important topic, but I also hope that
at some point we can come back to the proposal for master/detail tasks.

This idea is interesting, because it provides a novel way to connect the
plugins to pulpcore calls. The views that plugins implement would not be
normal views, they would operate more like OPTIONS schema views. I'll try
to show how I think this would look from a user's perspective:

*File Sync:*

The user doesn't know how to sync, so they do a GET request:
*GET /v3/versionactions/*

This returns a list of possible actions, one of them being
v3/versionactions/file/sync. To see how to use that they do another GET
request:
*GET /v3versionactions/file/sync/*

This returns the description of the file-sync action and the parameters
necessary (importer, repository)

*POST v3/repositories/1234/versions/* importer=importer_href,
repository=repository_href, action=v3/versionactions/file/sync/

I have some practical questions:

   - What does the documentation for v3/repositories/1234/versions/ look
   like?
  - Are the parameters listed, or does the user have to use
  v3/versionactions/sync/ to retrieve their schema?
   - How does the RepositoryVersionsViewSet use the versionaction?
  - How are the params `importer`, and `repository` validated?
 - Synchronously
 - Asynchronously
  - How is the celery task connected and dispatched?

Here are my initial thoughts:

   - I like that this brings us back to "to create a new RepositoryVersion,
   POST to versions".
  - I especially like that all versions are created at a single
  endpoint.
   - I don't like that the plugins are creating views that are not intended
   to be used as endpoints. These views are only "hit" for documentation, and
   they are used as parameters to other tasks.
   - This could be confusing for users
  - This is a different type of work than the rest of plugin writing,
  so it will have its own learning curve
  - This would not (speculation) play nice with auto-documentation. The
  autodocumentation would not know about the possible parameters for the
  actions, so the users would have to user GET v3/versionactions/ *instead
  *of docs.
  - This would not (speculation) play nice with auto-bindings and
  clients. Same problem as ^
  - GET request does not retrieve objects, it retrieves schema


On Mon, Apr 2, 2018 at 10:26 AM, Dennis Kliban  wrote:

> Here is another idea that would remove the verbs from the REST API and
> would provide a way to namespace plugin provided tasks.
>
> We could add 2 additional resource types for plugin writers to provide:
> versionaction and publicationaction. These REST resources would only
> support GET requests. The response would contain a description of the
> action and  a description of the parameters this action accepts. Plugin
> writers would create these resources by extending an ActionView with a
> custom get() method for providing users with a description and also an
> action() method which would contain the code that needs to run
> asynchronously.
>
> Every repository version would be created using the repository version
> API. POST requests to the repository version API would accept a
> 'versionaction' href along with a dictionary of parameters. Users would
> then receive a 202 response with a task href that can be used to monitor
> the progress of the asynchronous repository version creation.
>
> Every publication would be created using the publication API. POST
> requests to the publication API would accept a 'publicationaction' href
> along with a dictionary of parameters. Users would then receive a 202
> response with a task href that can be used to monitor the progress of the
> asynchronous publication creation.
>
> Here are some REST API changes this would bring:
>
> */api/v3/versionactions/* - returns a list of all the action descriptions
> for creating repository versions
>
> */api/v3/versionactions/**pulpcore/addremove/* - returns a description of
> the addremove action provided by pulpcore. Something like this:
>
> {
>'href': 'http://localhost/api/v3/versionactions/pulpcore/addremove/',
>'description': 'Creates a repository version by adding and removing
> content units from the latest repository version.',
>'parameters': {
>   'add_content_units': 'list of content to
> add',
>   'remove_content_units': 'list of content to
> remove'
>}
> }
>
>
> */api/v3/versionactions/<**plugin>/sync/ *- accepts GET and returns a
> description of the sync action of the plugin. Something like this:
>
> {
>'href': 'http://localhost/api/v3/versionactions/file/sync/',
>'description': 'Creates a repository version by syncing content from a
> remote file repository.',
>'parameters': {
>   'remote': 'h

Re: [Pulp-dev] Plugin relationship to tasks

[Dennis Kliban]

Here is another idea that would remove the verbs from the REST API and
would provide a way to namespace plugin provided tasks.

We could add 2 additional resource types for plugin writers to provide:
versionaction and publicationaction. These REST resources would only
support GET requests. The response would contain a description of the
action and  a description of the parameters this action accepts. Plugin
writers would create these resources by extending an ActionView with a
custom get() method for providing users with a description and also an
action() method which would contain the code that needs to run
asynchronously.

Every repository version would be created using the repository version API.
POST requests to the repository version API would accept a 'versionaction'
href along with a dictionary of parameters. Users would then receive a 202
response with a task href that can be used to monitor the progress of the
asynchronous repository version creation.

Every publication would be created using the publication API. POST requests
to the publication API would accept a 'publicationaction' href along with a
dictionary of parameters. Users would then receive a 202 response with a
task href that can be used to monitor the progress of the asynchronous
publication creation.

Here are some REST API changes this would bring:

*/api/v3/versionactions/* - returns a list of all the action descriptions
for creating repository versions

*/api/v3/versionactions/**pulpcore/addremove/* - returns a description of
the addremove action provided by pulpcore. Something like this:

{
   'href': 'http://localhost/api/v3/versionactions/pulpcore/addremove/',
   'description': 'Creates a repository version by adding and removing
content units from the latest repository version.',
   'parameters': {
  'add_content_units': 'list of content to add',
  'remove_content_units': 'list of content to
remove'
   }
}


*/api/v3/versionactions/<**plugin>/sync/ *- accepts GET and returns a
description of the sync action of the plugin. Something like this:

{
   'href': 'http://localhost/api/v3/versionactions/file/sync/',
   'description': 'Creates a repository version by syncing content from a
remote file repository.',
   'parameters': {
  'remote': 'href of the remote to sync from'
   }
}


*/api/v3/publicationactions/ *- returns a list of all the actions for
creating publications

*/api/v3/publicationactions/<**plugin>/publish/* - accepts GET and returns
a description of the publish action for the file plugin. Something like
this:

{
   'href': 'http://localhost/api/v3/versionactions/file/publish/',
   'description': 'Create a publication from a file repository version.',
   'parameters': {
  'repository': 'href of the repository to
publish. The latest repository version will be published. Can only be
specified if repository_version is not specified.'
  'repository_version': 'href of the repository
version to publish. Can only be specified if repository_version is not
specified. Can only be specified if repository is not specified.'
   }
}

*/api/v3/repository//versions/ *- accepts POST requests with
'versionaction' href and 'parameters' dictionary. Returns 202 with a task
href.

*/api/v3/publications/ *- accepts POST requests with 'publicationaction'
href and 'parameters' dictionary. Returns 202 with a task href.


On Thu, Mar 29, 2018 at 4:18 PM, Milan Kovacik  wrote:

> Folks,
>
> I'd like to summarize where we've gotten so far. Our problem statement
> is threefold:
>
> 1) Content operations have to be routed thru particular plug-ins i.e
> no repository version can be created without a plug-in involvement, to
> avoid repository content consistency/correctness issues
> 2) API specialization i.e keeping core concepts above plug-in specific
> concepts; e.g CRUD@/v3/repositories/ v.s CRUD@docker/adds/
> 3) API consistency i.e preferably where I create it is where I manage
> it; e.g. w/r/t asynchronous action endpoints creating Task resources
> vs a Task management endpoint
>
> Since there is a significant intersection between the asynchronous
> action endpoints and endpoints manipulating repository versions
> asynchronously, me and Austin propose to convert the asynchronous
> action endpoints to Task management endpoints, in full control of
> plug-ins, in particular:
>
> Asynchronous Repository Version Management Endpoints:
>
>   Repository synchronisation and publication and addition(deletion) of
> content into(from) a repository version would all be treated as Task
> objects primarily, residing in the plug-in namespace:
>   POST@/v3/tasks//[syncs, publishes, additions, deletions]/
>   because all these operations share similar call signature, every one
> of these actions:
>
>   * creates a Task object
>   * the task, while e

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

Folks,

I'd like to summarize where we've gotten so far. Our problem statement
is threefold:

1) Content operations have to be routed thru particular plug-ins i.e
no repository version can be created without a plug-in involvement, to
avoid repository content consistency/correctness issues
2) API specialization i.e keeping core concepts above plug-in specific
concepts; e.g CRUD@/v3/repositories/ v.s CRUD@docker/adds/
3) API consistency i.e preferably where I create it is where I manage
it; e.g. w/r/t asynchronous action endpoints creating Task resources
vs a Task management endpoint

Since there is a significant intersection between the asynchronous
action endpoints and endpoints manipulating repository versions
asynchronously, me and Austin propose to convert the asynchronous
action endpoints to Task management endpoints, in full control of
plug-ins, in particular:

Asynchronous Repository Version Management Endpoints:

  Repository synchronisation and publication and addition(deletion) of
content into(from) a repository version would all be treated as Task
objects primarily, residing in the plug-in namespace:
  POST@/v3/tasks//[syncs, publishes, additions, deletions]/
  because all these operations share similar call signature, every one
of these actions:

  * creates a Task object
  * the task, while executing, creates a new repository version and
locks similar resources
  * has to be controlled by a particular plug-in i.e may require
additional, plug-in specific settings
  * requires either a repository URI or repository version URI to operate on
  * requires generic action configuration e.g almost every sync
requires (at least one) importer

 As a result:
   * the asynchronous content management Task is now the nexus where
the configuration (repository, plug-in, origin) meets the action
   * these complex tasks that actually do the heavy lifting have a
clear management endpoint
   * this change effectively renders the
@/v3/repositories//versions/<#nr>/ endpoint read-only and
documents the intention that no repository version is ever created
without an actual plug-in involvement
   * a plug-in can expose custom asynchronous content endpoints, that
manipulate repository versions, aligned with the more generic actions:
   POST@/v3/tasks//[,...]/
   * the design aligns better with the DRF allowing (nested)
serializers to control (and document) the endpoints in the direction
from the more generic to the more specific implementation

Asynchronous Core Object Management Endpoints:

  Additionally, for Asynchronous Core object management endpoints, we
suggest to move Remote/Importer, Publisher, Exporter, Repository
operations under a new core task namespace:
   POST@/v3/tasks/core/[importers, publishers,
repositories]/[deletes, updates]/
  to be consistent with the asynchronous content actions. This however
seems to conflict with 3).

Caveats

Both the current and the proposed design don't conceptually prevent a
repository to have content provided by multiple plug-ins.
Folks don't like Pulp becoming a task management system.

Alternatives

  Dennis proposes to achieve similar result by adding plug-in content
model hooks:
  "add_to_repository_version" and "remove_from_repository_version"
  This approach however doesn't seem to solve plug-ins requiring
multiple or no importers and leaves multiple spots in the API, where
repository versions can be created (failing 3); custom plug-in
asynchronous content management endpoints and importers/remotes sync
vs POST@v3/repository/UUID/versions/<#nr>/

  David proposes to namespace all objects under plug-in specific
endpoints. This solves the repository--plug-in purity caveat, allows
great plug-in flexibility and prevents endpoint name collisions.
 This approach may allow the plug-ins UX to drift from each other
though, and solves neither the multiple/no importers case, nor the
scattering of repository versions creation endpoints and fails 2)&3).

I believe this Master/Detail Tasks design to be a reasonable
compromise on the requirements in 1) 2) 3).

Cheers,
milan

On Wed, Mar 28, 2018 at 9:38 PM, Dennis Kliban  wrote:
> On Wed, Mar 28, 2018 at 12:20 PM, Ina Panova  wrote:
>>
>> I do not think that it's a valid argument to ban the proposal just because
>> the proposal will bring changes to the existing plugins. Because:
>>
>> 1) i think it would be fair to think of other plugins and find a solution
>> all plugins will be happy with --> so we are back to the generic correctness
>> problem
>> 2) we are not GA, not even Beta, we are free and eligible to make changes,
>> no promises made yet.
>> 3) if we bring changes *this* exactly is the time to do because we have
>> just 2 plugins working with basic functionality :)
>>
>> W/r to the "Pulp turning into a task running system" ..and yet that's a
>> system based on distributed task system
>
>
> Pulp may rely on a tasking system to get work done, but we don't want users
> thinking of it as a tasking system.

Re: [Pulp-dev] Plugin relationship to tasks

[Dennis Kliban]

On Wed, Mar 28, 2018 at 12:20 PM, Ina Panova  wrote:

> I do not think that it's a valid argument to ban the proposal just because
> the proposal will bring changes to the existing plugins. Because:
>
> 1) i think it would be fair to think of other plugins and find a solution
> all plugins will be happy with --> so we are back to the generic
> correctness problem
> 2) we are not GA, not even Beta, we are free and eligible to make changes,
> no promises made yet.
> 3) if we bring changes *this* exactly is the time to do because we have
> just 2 plugins working with basic functionality :)
>
> W/r to the "Pulp turning into a task running system" ..and yet that's a
> system based on distributed task system
>

Pulp may rely on a tasking system to get work done, but we don't want users
thinking of it as a tasking system. We want our users to think of Pulp as a
repository management system.


>
> I suggest to organize a meeting with an agenda in advance that would list
> 1) concerns 2) questions
> I also think it would be valuable to hear Jeff's feedback.
>
> If the team finds that meeting idea is beneficial with the believe that
> after the meeting we will:
> 1) i am not naive to believe that we would reach consensus but at least we
> would move forward
> 2) we will decrease the frustration level, because face2face conversation
> is more profitable, since we do see faces, have social interaction and not
> type onto keyboard and stare at the text on the monitor.
>
> then...I will take the action item and schedule the meeting.
>
> Meanwhile we'll have time to chew on Easter eggs and give to the proposal
> a deeper, unbiased, full of protein (that helps brain thinking) thought.
>
>
I am against having a meeting to discuss this. The discussion on the list
has gotten traction and I'd like it to continue on here.


>
>
>
> 
> Regards,
>
> Ina Panova
> Software Engineer| Pulp| Red Hat Inc.
>
> "Do not go where the path may lead,
>  go instead where there is no path and leave a trail."
>
> On Wed, Mar 28, 2018 at 12:11 AM, Brian Bouterse 
> wrote:
>
>> In terms of removing POST /api/v3/repositories//versions/ from
>> core, I want to bring it back to the MVP language and REST which drove the
>> original design. The MVP has: "As an authenticated user, I can create a new
>> version by adding or removing content to the latest version." To
>> facilitate that in a generic way, we need a core endpoint to do that, i.e.
>> /api/v3/repositories//versions/. My concern is that removing it
>> would cause us to not fulfill our use cases without requiring more code
>> from some plugin writers. Also in terms of REST philosophy, POSTing to the
>> RespotoryVersion resource to create a new RepositoryVersion is the
>> traditional url design.
>>
>> For pulp_ansible, for example, the generic add/remove functionality at
>> core endpoint ^ would be meet all of the pulp_ansible user's needs because
>> of the way the pulp_ansible content is modelled. So removing this endpoint
>> means more work for pulp_ansible developers w.r.t creating repo versions
>> and providing tasks and endpoints. I don't see this extra responsibility on
>> plugin writers coming with a clear benefit.
>>
>> One idea I liked in this discussion is to have a documented convention
>> that encourages plugin writers to put their viewsets in a namespaced area.
>> They still need the ability to put them anywhere due to live API
>> goals/requirements so this would only be a convention for those tasks they
>> are offering directly to their users.
>>
>>
>>
>>
>>
>> On Tue, Mar 27, 2018 at 5:40 PM, David Davis 
>> wrote:
>>
>>> I like Austin’s task proposal in that the plugin writers can focus on
>>> serializers and tasks that can be easily integrated with core. That said, I
>>> agree on the counterpoints raise about the new resource endpoints and
>>> turning much of pulp into a task running system. So I am a bit mixed on
>>> which approach is better.
>>>
>>> I do think that we should remove POST /api/v3/repositories//versions/
>>> from core.
>>>
>>>
>>> David
>>>
>>> On Tue, Mar 27, 2018 at 3:49 PM, Austin Macdonald 
>>> wrote:
>>>

>- /api/v3/repositories//versions/ endpoint does not perform
>plugin specific validation which can lead to "broken" repository 
> versions.
>- Plugin authors don't have any convention to follow when creating
>custom REST API endpoints for creating repository versions.
>- As a result of ^, a user will have a hard time identifying
>repository version creation APIs in different plugins.
>
>
 I agree with these points.

>
>-
>
> My first inclination is to disable the ability to POST to
> /api/v3/repositories//versions/ and require users to use the
> plugin specific APIs for creating repository versions. However, I think
> that integrators of build systems that produce a variety of content types
> would have a lot more flexibility if they could u

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

>
> I haven't seen a counterargument to the assertion that it is **incorrect**
>> to control repo membership without allowing plugin validation.
>>
>
> We could add hooks for plugins to provide validation for each content
> type. The Content model in pulpcore could define
> "add_to_repository_version" and "remove_from_repository_version" methods
> that take repository version as a parameter. The default behavior would be
> that exists now. The Content models in the plugins could then override the
> methods and provide validation needed for removing from and adding to a
> repository version. The repository version REST api would then use these
> methods on the content models to add and remove content.
>

In Pulp 2, core controls the endpoints and uses hooks to involve the
plugins but it is difficult to put the right hooks wherever any plugin
needs them. That may be an overly simplistic comparison though because I
think your proposal is just for add/remove, and not for more complex
concepts like sync. The general idea is plausible, but I would want to hear
more about how the implementation would look before I try to make technical
comparisons.

Generally, I prefer to avoid the hook pattern in favor of plugin-defined
endpoints. The pattern I prefer is "offer plugin writers tools, but stay
out of their way."
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Dennis Kliban]

On Wed, Mar 28, 2018 at 10:21 AM, Austin Macdonald 
wrote:

>
>
> On Tue, Mar 27, 2018 at 6:11 PM, Brian Bouterse 
> wrote:
>
>> In terms of removing POST /api/v3/repositories//versions/ from
>> core, I want to bring it back to the MVP language and REST which drove the
>> original design. The MVP has: "As an authenticated user, I can create a new
>> version by adding or removing content to the latest version." To
>> facilitate that in a generic way, we need a core endpoint to do that, i.e.
>> /api/v3/repositories//versions/. My concern is that removing it
>> would cause us to not fulfill our use cases without requiring more code
>> from some plugin writers. Also in terms of REST philosophy, POSTing to the
>> RespotoryVersion resource to create a new RepositoryVersion is the
>> traditional url design.
>>
>> For pulp_ansible, for example, the generic add/remove functionality at
>> core endpoint ^ would be meet all of the pulp_ansible user's needs because
>> of the way the pulp_ansible content is modelled. So removing this endpoint
>> means more work for pulp_ansible developers w.r.t creating repo versions
>> and providing tasks and endpoints. I don't see this extra responsibility on
>> plugin writers coming with a clear benefit.
>>
>>
> I'm sensitive to adding work for the plugin writers, but convenience needs
> to be discussed after correctness.
>
> I haven't seen a counterargument to the assertion that it is **incorrect**
> to control repo membership without allowing plugin validation.
>

We could add hooks for plugins to provide validation for each content type.
The Content model in pulpcore could define "add_to_repository_version" and
"remove_from_repository_version" methods that take repository version as a
parameter. The default behavior would be that exists now. The Content
models in the plugins could then override the methods and provide
validation needed for removing from and adding to a repository version. The
repository version REST api would then use these methods on the content
models to add and remove content.

Thoughts on this idea?



>
>> One idea I liked in this discussion is to have a documented convention
>> that encourages plugin writers to put their viewsets in a namespaced area.
>> They still need the ability to put them anywhere due to live API
>> goals/requirements so this would only be a convention for those tasks they
>> are offering directly to their users.
>>
>>
>>
>>
>>
>> On Tue, Mar 27, 2018 at 5:40 PM, David Davis 
>> wrote:
>>
>>> I like Austin’s task proposal in that the plugin writers can focus on
>>> serializers and tasks that can be easily integrated with core. That said, I
>>> agree on the counterpoints raise about the new resource endpoints and
>>> turning much of pulp into a task running system. So I am a bit mixed on
>>> which approach is better.
>>>
>>> I do think that we should remove POST /api/v3/repositories//versions/
>>> from core.
>>>
>>>
>>> David
>>>
>>> On Tue, Mar 27, 2018 at 3:49 PM, Austin Macdonald 
>>> wrote:
>>>

>- /api/v3/repositories//versions/ endpoint does not perform
>plugin specific validation which can lead to "broken" repository 
> versions.
>- Plugin authors don't have any convention to follow when creating
>custom REST API endpoints for creating repository versions.
>- As a result of ^, a user will have a hard time identifying
>repository version creation APIs in different plugins.
>
>
 I agree with these points.

>
>-
>
> My first inclination is to disable the ability to POST to
> /api/v3/repositories//versions/ and require users to use the
> plugin specific APIs for creating repository versions. However, I think
> that integrators of build systems that produce a variety of content types
> would have a lot more flexibility if they could use a single generic API
> endpoint to create a repository version independent of the content type.
>
> Let's continue this discussion by answering the following question:
>
>- Should we disable the ability to POST to
>/api/v3/repositories//versions/ and require users to always
>use a plugin specific repository version creation API?
>
> Yes, I think we should disable POST to /api/v3/repositories//ve
 rsions/

 Simplifying integration is important, but we should not sacrifice
 correctness enforcement.

 ___
 Pulp-dev mailing list
 Pulp-dev@redhat.com
 https://www.redhat.com/mailman/listinfo/pulp-dev


>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pul

Re: [Pulp-dev] Plugin relationship to tasks

[Ina Panova]

I do not think that it's a valid argument to ban the proposal just because
the proposal will bring changes to the existing plugins. Because:

1) i think it would be fair to think of other plugins and find a solution
all plugins will be happy with --> so we are back to the generic
correctness problem
2) we are not GA, not even Beta, we are free and eligible to make changes,
no promises made yet.
3) if we bring changes *this* exactly is the time to do because we have
just 2 plugins working with basic functionality :)

W/r to the "Pulp turning into a task running system" ..and yet that's a
system based on distributed task system

I suggest to organize a meeting with an agenda in advance that would list
1) concerns 2) questions
I also think it would be valuable to hear Jeff's feedback.

If the team finds that meeting idea is beneficial with the believe that
after the meeting we will:
1) i am not naive to believe that we would reach consensus but at least we
would move forward
2) we will decrease the frustration level, because face2face conversation
is more profitable, since we do see faces, have social interaction and not
type onto keyboard and stare at the text on the monitor.

then...I will take the action item and schedule the meeting.

Meanwhile we'll have time to chew on Easter eggs and give to the proposal a
deeper, unbiased, full of protein (that helps brain thinking) thought.





Regards,

Ina Panova
Software Engineer| Pulp| Red Hat Inc.

"Do not go where the path may lead,
 go instead where there is no path and leave a trail."

On Wed, Mar 28, 2018 at 12:11 AM, Brian Bouterse 
wrote:

> In terms of removing POST /api/v3/repositories//versions/ from
> core, I want to bring it back to the MVP language and REST which drove the
> original design. The MVP has: "As an authenticated user, I can create a new
> version by adding or removing content to the latest version." To
> facilitate that in a generic way, we need a core endpoint to do that, i.e.
> /api/v3/repositories//versions/. My concern is that removing it
> would cause us to not fulfill our use cases without requiring more code
> from some plugin writers. Also in terms of REST philosophy, POSTing to the
> RespotoryVersion resource to create a new RepositoryVersion is the
> traditional url design.
>
> For pulp_ansible, for example, the generic add/remove functionality at
> core endpoint ^ would be meet all of the pulp_ansible user's needs because
> of the way the pulp_ansible content is modelled. So removing this endpoint
> means more work for pulp_ansible developers w.r.t creating repo versions
> and providing tasks and endpoints. I don't see this extra responsibility on
> plugin writers coming with a clear benefit.
>
> One idea I liked in this discussion is to have a documented convention
> that encourages plugin writers to put their viewsets in a namespaced area.
> They still need the ability to put them anywhere due to live API
> goals/requirements so this would only be a convention for those tasks they
> are offering directly to their users.
>
>
>
>
>
> On Tue, Mar 27, 2018 at 5:40 PM, David Davis 
> wrote:
>
>> I like Austin’s task proposal in that the plugin writers can focus on
>> serializers and tasks that can be easily integrated with core. That said, I
>> agree on the counterpoints raise about the new resource endpoints and
>> turning much of pulp into a task running system. So I am a bit mixed on
>> which approach is better.
>>
>> I do think that we should remove POST /api/v3/repositories//versions/
>> from core.
>>
>>
>> David
>>
>> On Tue, Mar 27, 2018 at 3:49 PM, Austin Macdonald 
>> wrote:
>>
>>>
- /api/v3/repositories//versions/ endpoint does not perform
plugin specific validation which can lead to "broken" repository 
 versions.
- Plugin authors don't have any convention to follow when creating
custom REST API endpoints for creating repository versions.
- As a result of ^, a user will have a hard time identifying
repository version creation APIs in different plugins.


>>> I agree with these points.
>>>

-

 My first inclination is to disable the ability to POST to
 /api/v3/repositories//versions/ and require users to use the
 plugin specific APIs for creating repository versions. However, I think
 that integrators of build systems that produce a variety of content types
 would have a lot more flexibility if they could use a single generic API
 endpoint to create a repository version independent of the content type.

 Let's continue this discussion by answering the following question:

- Should we disable the ability to POST to
/api/v3/repositories//versions/ and require users to always
use a plugin specific repository version creation API?

 Yes, I think we should disable POST to /api/v3/repositories//ve
>>> rsions/
>>>
>>> Simplifying integration is important, but we should not sac

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

On Tue, Mar 27, 2018 at 6:11 PM, Brian Bouterse  wrote:

> In terms of removing POST /api/v3/repositories//versions/ from
> core, I want to bring it back to the MVP language and REST which drove the
> original design. The MVP has: "As an authenticated user, I can create a new
> version by adding or removing content to the latest version." To
> facilitate that in a generic way, we need a core endpoint to do that, i.e.
> /api/v3/repositories//versions/. My concern is that removing it
> would cause us to not fulfill our use cases without requiring more code
> from some plugin writers. Also in terms of REST philosophy, POSTing to the
> RespotoryVersion resource to create a new RepositoryVersion is the
> traditional url design.
>
> For pulp_ansible, for example, the generic add/remove functionality at
> core endpoint ^ would be meet all of the pulp_ansible user's needs because
> of the way the pulp_ansible content is modelled. So removing this endpoint
> means more work for pulp_ansible developers w.r.t creating repo versions
> and providing tasks and endpoints. I don't see this extra responsibility on
> plugin writers coming with a clear benefit.
>
>
I'm sensitive to adding work for the plugin writers, but convenience needs
to be discussed after correctness.

I haven't seen a counterargument to the assertion that it is **incorrect**
to control repo membership without allowing plugin validation.


> One idea I liked in this discussion is to have a documented convention
> that encourages plugin writers to put their viewsets in a namespaced area.
> They still need the ability to put them anywhere due to live API
> goals/requirements so this would only be a convention for those tasks they
> are offering directly to their users.
>
>
>
>
>
> On Tue, Mar 27, 2018 at 5:40 PM, David Davis 
> wrote:
>
>> I like Austin’s task proposal in that the plugin writers can focus on
>> serializers and tasks that can be easily integrated with core. That said, I
>> agree on the counterpoints raise about the new resource endpoints and
>> turning much of pulp into a task running system. So I am a bit mixed on
>> which approach is better.
>>
>> I do think that we should remove POST /api/v3/repositories//versions/
>> from core.
>>
>>
>> David
>>
>> On Tue, Mar 27, 2018 at 3:49 PM, Austin Macdonald 
>> wrote:
>>
>>>
- /api/v3/repositories//versions/ endpoint does not perform
plugin specific validation which can lead to "broken" repository 
 versions.
- Plugin authors don't have any convention to follow when creating
custom REST API endpoints for creating repository versions.
- As a result of ^, a user will have a hard time identifying
repository version creation APIs in different plugins.


>>> I agree with these points.
>>>

-

 My first inclination is to disable the ability to POST to
 /api/v3/repositories//versions/ and require users to use the
 plugin specific APIs for creating repository versions. However, I think
 that integrators of build systems that produce a variety of content types
 would have a lot more flexibility if they could use a single generic API
 endpoint to create a repository version independent of the content type.

 Let's continue this discussion by answering the following question:

- Should we disable the ability to POST to
/api/v3/repositories//versions/ and require users to always
use a plugin specific repository version creation API?

 Yes, I think we should disable POST to /api/v3/repositories//ve
>>> rsions/
>>>
>>> Simplifying integration is important, but we should not sacrifice
>>> correctness enforcement.
>>>
>>> ___
>>> Pulp-dev mailing list
>>> Pulp-dev@redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>>
>>>
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Brian Bouterse]

In terms of removing POST /api/v3/repositories//versions/ from core,
I want to bring it back to the MVP language and REST which drove the
original design. The MVP has: "As an authenticated user, I can create a new
version by adding or removing content to the latest version." To facilitate
that in a generic way, we need a core endpoint to do that, i.e.
/api/v3/repositories//versions/. My concern is that removing it would
cause us to not fulfill our use cases without requiring more code from some
plugin writers. Also in terms of REST philosophy, POSTing to the
RespotoryVersion resource to create a new RepositoryVersion is the
traditional url design.

For pulp_ansible, for example, the generic add/remove functionality at core
endpoint ^ would be meet all of the pulp_ansible user's needs because of
the way the pulp_ansible content is modelled. So removing this endpoint
means more work for pulp_ansible developers w.r.t creating repo versions
and providing tasks and endpoints. I don't see this extra responsibility on
plugin writers coming with a clear benefit.

One idea I liked in this discussion is to have a documented convention that
encourages plugin writers to put their viewsets in a namespaced area. They
still need the ability to put them anywhere due to live API
goals/requirements so this would only be a convention for those tasks they
are offering directly to their users.





On Tue, Mar 27, 2018 at 5:40 PM, David Davis  wrote:

> I like Austin’s task proposal in that the plugin writers can focus on
> serializers and tasks that can be easily integrated with core. That said, I
> agree on the counterpoints raise about the new resource endpoints and
> turning much of pulp into a task running system. So I am a bit mixed on
> which approach is better.
>
> I do think that we should remove POST /api/v3/repositories//versions/
> from core.
>
>
> David
>
> On Tue, Mar 27, 2018 at 3:49 PM, Austin Macdonald 
> wrote:
>
>>
>>>- /api/v3/repositories//versions/ endpoint does not perform
>>>plugin specific validation which can lead to "broken" repository 
>>> versions.
>>>- Plugin authors don't have any convention to follow when creating
>>>custom REST API endpoints for creating repository versions.
>>>- As a result of ^, a user will have a hard time identifying
>>>repository version creation APIs in different plugins.
>>>
>>>
>> I agree with these points.
>>
>>>
>>>-
>>>
>>> My first inclination is to disable the ability to POST to
>>> /api/v3/repositories//versions/ and require users to use the
>>> plugin specific APIs for creating repository versions. However, I think
>>> that integrators of build systems that produce a variety of content types
>>> would have a lot more flexibility if they could use a single generic API
>>> endpoint to create a repository version independent of the content type.
>>>
>>> Let's continue this discussion by answering the following question:
>>>
>>>- Should we disable the ability to POST to
>>>/api/v3/repositories//versions/ and require users to always
>>>use a plugin specific repository version creation API?
>>>
>>> Yes, I think we should disable POST to /api/v3/repositories//ve
>> rsions/
>>
>> Simplifying integration is important, but we should not sacrifice
>> correctness enforcement.
>>
>> ___
>> Pulp-dev mailing list
>> Pulp-dev@redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-dev
>>
>>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[David Davis]

I like Austin’s task proposal in that the plugin writers can focus on
serializers and tasks that can be easily integrated with core. That said, I
agree on the counterpoints raise about the new resource endpoints and
turning much of pulp into a task running system. So I am a bit mixed on
which approach is better.

I do think that we should remove POST /api/v3/repositories//versions/
from core.


David

On Tue, Mar 27, 2018 at 3:49 PM, Austin Macdonald 
wrote:

>
>>- /api/v3/repositories//versions/ endpoint does not perform
>>plugin specific validation which can lead to "broken" repository versions.
>>- Plugin authors don't have any convention to follow when creating
>>custom REST API endpoints for creating repository versions.
>>- As a result of ^, a user will have a hard time identifying
>>repository version creation APIs in different plugins.
>>
>>
> I agree with these points.
>
>>
>>-
>>
>> My first inclination is to disable the ability to POST to
>> /api/v3/repositories//versions/ and require users to use the
>> plugin specific APIs for creating repository versions. However, I think
>> that integrators of build systems that produce a variety of content types
>> would have a lot more flexibility if they could use a single generic API
>> endpoint to create a repository version independent of the content type.
>>
>> Let's continue this discussion by answering the following question:
>>
>>- Should we disable the ability to POST to
>>/api/v3/repositories//versions/ and require users to always use
>>a plugin specific repository version creation API?
>>
>> Yes, I think we should disable POST to /api/v3/repositories//ve
> rsions/
>
> Simplifying integration is important, but we should not sacrifice
> correctness enforcement.
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

>
>
>- /api/v3/repositories//versions/ endpoint does not perform
>plugin specific validation which can lead to "broken" repository versions.
>- Plugin authors don't have any convention to follow when creating
>custom REST API endpoints for creating repository versions.
>- As a result of ^, a user will have a hard time identifying
>repository version creation APIs in different plugins.
>
>
I agree with these points.

>
>-
>
> My first inclination is to disable the ability to POST to
> /api/v3/repositories//versions/ and require users to use the plugin
> specific APIs for creating repository versions. However, I think that
> integrators of build systems that produce a variety of content types would
> have a lot more flexibility if they could use a single generic API endpoint
> to create a repository version independent of the content type.
>
> Let's continue this discussion by answering the following question:
>
>- Should we disable the ability to POST to 
> /api/v3/repositories//versions/
>and require users to always use a plugin specific repository version
>creation API?
>
> Yes, I think we should disable POST to /api/v3/repositories//
versions/

Simplifying integration is important, but we should not sacrifice
correctness enforcement.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Dennis Kliban]

On Tue, Mar 27, 2018 at 1:36 PM, Jeremy Audet  wrote:

> > The current plugin API allows plugin authors to define custom REST API
> endpoints that can perform any custom validation needed for the plugin. The
> repository version creation REST API provided by pulpcore only validates
> that the content exists. We should document this limitation and recommend
> that users use Repository Version creation APIs provided by specific
> plugins.
>
> Why should I, as a user, explicitly call HTTP API endpoints to make
> sure that Pulp's state is valid? That seems very strange. I'd rather
> not worry about that. If there's a path forward that lets plugins
> enforce correctness, that seems like a far better path forward.
>

There is already a way for plugins to enforce correctness by defining REST
API endpoints that perform validation. Here are the main points of concern
that need to be addressed:

   - /api/v3/repositories//versions/ endpoint does not perform plugin
   specific validation which can lead to "broken" repository versions.
   - Plugin authors don't have any convention to follow when creating
   custom REST API endpoints for creating repository versions.
   - As a result of ^, a user will have a hard time identifying repository
   version creation APIs in different plugins.

My first inclination is to disable the ability to POST to
/api/v3/repositories//versions/ and require users to use the plugin
specific APIs for creating repository versions. However, I think that
integrators of build systems that produce a variety of content types would
have a lot more flexibility if they could use a single generic API endpoint
to create a repository version independent of the content type.

Let's continue this discussion by answering the following question:

   - Should we disable the ability to POST to
   /api/v3/repositories//versions/ and require users to always use a
   plugin specific repository version creation API?
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Jeremy Audet]

> The current plugin API allows plugin authors to define custom REST API 
> endpoints that can perform any custom validation needed for the plugin. The 
> repository version creation REST API provided by pulpcore only validates that 
> the content exists. We should document this limitation and recommend that 
> users use Repository Version creation APIs provided by specific plugins.

Why should I, as a user, explicitly call HTTP API endpoints to make
sure that Pulp's state is valid? That seems very strange. I'd rather
not worry about that. If there's a path forward that lets plugins
enforce correctness, that seems like a far better path forward.

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Dennis Kliban]

On Tue, Mar 27, 2018 at 12:23 PM, Austin Macdonald 
wrote:

> After some reflection, I think we have been too focused on the small
> points (plugin complexity, URL organization, actions vs nouns). They should
> be discussed, but we don't have consensus around the primary concern.
>
> Correctness is what motivated this proposal, so I'd like to take a step
> back and discuss only correctness.
>
> My assertion:
>
> Plugins should be able to define and enforce constraints on the content
> membership of a repository.
>
> My reasoning:
>
> Some plugins have natural constraints on the membership of a repository.
> Two Docker Tags of the same name cannot be in a the same Repository.
> Manifest Lists cannot be in a repository without the Manifests that are
> listed. It would be fundamentally incorrect to prevent the plugins from
> enforcing these kinds of constraints. I think there are similar problems
> with errata package lists.
>
>
The current plugin API allows plugin authors to define custom REST API
endpoints that can perform any custom validation needed for the plugin. The
repository version creation REST API provided by pulpcore only validates
that the content exists. We should document this limitation and recommend
that users use Repository Version creation APIs provided by specific
plugins.


> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

After some reflection, I think we have been too focused on the small points
(plugin complexity, URL organization, actions vs nouns). They should be
discussed, but we don't have consensus around the primary concern.

Correctness is what motivated this proposal, so I'd like to take a step
back and discuss only correctness.

My assertion:

Plugins should be able to define and enforce constraints on the content
membership of a repository.

My reasoning:

Some plugins have natural constraints on the membership of a repository.
Two Docker Tags of the same name cannot be in a the same Repository.
Manifest Lists cannot be in a repository without the Manifests that are
listed. It would be fundamentally incorrect to prevent the plugins from
enforcing these kinds of constraints. I think there are similar problems
with errata package lists.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

On Tue, Mar 27, 2018 at 12:00 AM, Brian Bouterse  wrote:
> I also want to look at how this will affect our user experience. With this
> change, I believe plugin tasks are put at the center of the user experience
> and the repositories, content, and versions are moved to the background. The
> user would think about what they can do foremost as a series of tasks. This
> is great in terms of organization, but I'm concerned its stepping away from
> what makes pulp special. I think Pulp is special because of its focus on
> content management (repos, version, uploads, sync, promotion, copy/delete,
> publish, export).
Fwiw you just enumerated 7 verbs vs 2 nouns ;)

> Having the user think about Pulp as a task-running system
> takes Pulp a step closer to a software like RunDeck
> (https://www.rundeck.com/open-source).
>
> What I like about the existing user experience (and API) is that it all
> centers around Pulp's value proposition (importers, syncs, associations,
> etc).
The Pulp focus isn't repo/version/content/ objects CRUD.
The value added are those uploads, syncs, promotions, copies,
publishes and exports.

If we don't like 'tasks' we can have a poll about a better name of the
endpoint; 'actions' anyone? Or perhaps 'deeds'?

>
> Does this spark any thoughts or ideas from others?
>
>
> On Mon, Mar 26, 2018 at 4:43 PM, Dennis Kliban  wrote:
>>
>> On Mon, Mar 26, 2018 at 3:38 PM, Austin Macdonald 
>> wrote:
>>>
>>>
>>>
>>> On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban 
>>> wrote:

 This proposal does not make the plugin writer's job any easier. This
 simply changes where the plugin writer is providing validation logic, in a
 serializer vs. in a view.
>>>
>>>
>>> It does make the plugin writer's job easier *because* it changes how the
>>> validation is provided, Plugin writers already have to understand how to
>>> create models/serializers (and it is very simple) for all the other objects.
>>> Seriallizer validation is much simpler than viewset validation because
>>> serializers are literally designed to do this.
>>
>>
>> If we don't change anything, a plugin writer is responsible for creating a
>> task that creates a Repository Version and a view which will perform
>> validation and then dispatches a task. If we do make the proposed change,
>> the plugin writer will still have to write the task code and then also a
>> serializer that performs validation. The proposed change only removes the
>> responsibility of dispatching a task which is 1 line of code.
>>
>>>
>>>


 The other problem I see is that complexity for users is increased.
 Instead of having 1 resource for tracking task progress, there would be an
 infinite number of resources for tracking progress of tasks.
>>>
>>>
>>> In the proposal, Tasks are Master/Detail. The user doesn't have to
>>> "track" it at all. They can still use v3/tasks/ to see all the tasks.
>>> Retrieving tasks will either be the same (v3/tasks/) or easier if they know
>>> the task they are looking for (v3/tasks/file/syncs/)
>>
>>
>> My understanding of the master/detail pattern we are using for our other
>> resources is that each task's URI will look something like this:
>> /api/v3/tasks/file/syncs//
>>
>> If the above is true, that means the client has to be aware of
>> '/api/v3/tasks/file/syncs//' resource. The user would need to know how
>> to interpret the response from each task type. There could be an infinite
>> number of these tasks with many different arguments. So writing client code
>> handling all those responses could become a nightmare. Right now the user
>> just needs to know how to interpret the response from '/api/v3/tasks/'
>> for all tasks.
>>
>>>
>>>

 I don't see the value added by the proposed change.
>>>
>>>
>>> Value #1: We have to do something to address the problem that
>>> adding/removing content to a repository without plugin input is incorrect.
>>> This proposal is one possibility, but it isn't valid to compare the value
>>> against doing nothing. Instead, if you don't like this option, we need to
>>> compare it to other proposals for how to involve plugins in tasks.
>>>
>>
>> We don't have a problem. I agree that a user needs to know a whole lot of
>> information to correctly create a working repository version for a complex
>> content type such as a Docker manifest. Without all this information on
>> hand, the user could easily create a broken repository version. However, a
>> rich client could solve that problem. If the plugin writer wants to support
>> simpler clients, she can provide an additional URL for handling the
>> validation and additional logic for creating a repository version. We should
>> probably have a recommended convention for plugin authors to use when adding
>> additional URLs.
>>
>>>
>>> Value #2: Plugin writers no longer need to dispatch tasks. Plugin writers
>>> shouldn't need to understand the complexities of our tasking system.
>>> Pulpcore can handle task dispatching generally, pl

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

On Tue, Mar 27, 2018 at 12:02 AM, Austin Macdonald  wrote:
>
>
> On Mon, Mar 26, 2018 at 4:43 PM, Dennis Kliban  wrote:
>>
>> On Mon, Mar 26, 2018 at 3:38 PM, Austin Macdonald 
>> wrote:
>>>
>>>
>>>
>>> On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban 
>>> wrote:

 This proposal does not make the plugin writer's job any easier. This
 simply changes where the plugin writer is providing validation logic, in a
 serializer vs. in a view.
>>>
>>>
>>> It does make the plugin writer's job easier *because* it changes how the
>>> validation is provided, Plugin writers already have to understand how to
>>> create models/serializers (and it is very simple) for all the other objects.
>>> Seriallizer validation is much simpler than viewset validation because
>>> serializers are literally designed to do this.
>>
>>
>> If we don't change anything, a plugin writer is responsible for creating a
>> task that creates a Repository Version and a view which will perform
>> validation and then dispatches a task. If we do make the proposed change,
>> the plugin writer will still have to write the task code and then also a
>> serializer that performs validation. The proposed change only removes the
>> responsibility of dispatching a task which is 1 line of code.
>
>
> The difference is not in the number of lines, it is in the complexity.
> Validation with serializers is simpler, and more similar to other plugin
> code.
>
> New Way:
> https://github.com/asmacdo/pulp_file/blob/8a8cbf81b6f0f1e382be7884f2b2406a06ea230b/pulp_file/app/serializers.py#L15-L21
> Old
> way:https://github.com/asmacdo/pulp_file/blob/8a8cbf81b6f0f1e382be7884f2b2406a06ea230b/pulp_file/app/serializers.py#L13-L30
   oh, and let's not forget the boilerplate needed to make the old way
autodoc working for the custom action endpoints:
https://github.com/pulp/pulp_file/blob/master/pulp_file/app/viewsets.py#L31,#L61
   I dare to say majority of the sync objects (serializer) boilerplate
will be generic enough to be shared between the plugins; there's not
much one can do about a sync request but creating it...

>
> That "1 line of code" puts the plugin writer in contact with a very complex
> part of the pulpcore internals. `apply_async_with_reservation`:
> https://github.com/pulp/pulp/blob/e8f804dc6595732dad38ec478877d919d7d97063/pulpcore/pulpcore/tasking/tasks.py#L180
>

 The other problem I see is that complexity for users is increased.
 Instead of having 1 resource for tracking task progress, there would be an
 infinite number of resources for tracking progress of tasks.
>>>
>>>
>>> In the proposal, Tasks are Master/Detail. The user doesn't have to
>>> "track" it at all. They can still use v3/tasks/ to see all the tasks.
>>> Retrieving tasks will either be the same (v3/tasks/) or easier if they know
>>> the task they are looking for (v3/tasks/file/syncs/)
>>
>>
>> My understanding of the master/detail pattern we are using for our other
>> resources is that each task's URI will look something like this:
>> /api/v3/tasks/file/syncs//
>>
>> If the above is true, that means the client has to be aware of
>> '/api/v3/tasks/file/syncs//' resource. The user would need to know how
>> to interpret the response from each task type. There could be an infinite
>> number of these tasks with many different arguments. So writing client code
>> handling all those responses could become a nightmare. Right now the user
>> just needs to know how to interpret the response from '/api/v3/tasks/'
>> for all tasks.
>
>
> I might be missing your point on this one. If you are talking about polling
> tasks, the href of a new Task is returned when the Task is created.The
> client should use the href that is returned, never string formatting urls.
> Even so, the Tasks in this proposal will be part of the API schema, so
> clients (and docs users) know where the endpoints are and what the arguments
> should be. This would actually be a big advantage over the way things are
> now. AFAIK, v3/importers/1234/sync/ is invisible to clients, so we replace
> one invisible endpoint with one visible endpoint. No complexity added.
>
> If the client doesn't know how to interpret extra fields (the Task
> parameters), they don't need to. All tasks share the same base fields, and
> those are the fields that need to be interpreted when the client is polling
> a task.
+1 all tasks will share common ancestor and path in the url; it's very
easy to explore and follow, see also the autodoc screen cast

>
>>
>> Value #1: We have to do something to address the problem that
>> adding/removing content to a repository without plugin input is incorrect.
>> This proposal is one possibility, but it isn't valid to compare the value
>> against doing nothing. Instead, if you don't like this option, we need to
>> compare it to other proposals for how to involve plugins in tasks.
>>
>>
>> We don't have a problem. I agree that a user needs to know a whole lot of
>> information to correctly create a worki

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

On Mon, Mar 26, 2018 at 4:43 PM, Dennis Kliban  wrote:

> On Mon, Mar 26, 2018 at 3:38 PM, Austin Macdonald 
> wrote:
>
>>
>>
>> On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban 
>> wrote:
>>
>>> This proposal does not make the plugin writer's job any easier. This
>>> simply changes where the plugin writer is providing validation logic, in a
>>> serializer vs. in a view.
>>>
>>
>> It does make the plugin writer's job easier *because* it changes how the
>> validation is provided, Plugin writers already have to understand how to
>> create models/serializers (and it is very simple) for all the other
>> objects. Seriallizer validation is much simpler than viewset validation
>> because serializers are literally designed to do this.
>>
>
> If we don't change anything, a plugin writer is responsible for creating a
> task that creates a Repository Version and a view which will perform
> validation and then dispatches a task. If we do make the proposed change,
> the plugin writer will still have to write the task code and then also a
> serializer that performs validation. The proposed change only removes the
> responsibility of dispatching a task which is 1 line of code.
>

The difference is not in the number of lines, it is in the complexity.
Validation with serializers is simpler, and more similar to other plugin
code.

New Way:
https://github.com/asmacdo/pulp_file/blob/8a8cbf81b6f0f1e382be7884f2b2406a06ea230b/pulp_file/app/serializers.py#L15-L21
Old way:
https://github.com/asmacdo/pulp_file/blob/8a8cbf81b6f0f1e382be7884f2b2406a06ea230b/pulp_file/app/serializers.py#L13-L30

That "1 line of code" puts the plugin writer in contact with a very complex
part of the pulpcore internals. `apply_async_with_reservation`:
https://github.com/pulp/pulp/blob/e8f804dc6595732dad38ec478877d919d7d97063/pulpcore/pulpcore/tasking/tasks.py#L180



> The other problem I see is that complexity for users is increased. Instead
>>> of having 1 resource for tracking task progress, there would be an infinite
>>> number of resources for tracking progress of tasks.
>>>
>>
>> In the proposal, Tasks are Master/Detail. The user doesn't have to
>> "track" it at all. They can still use v3/tasks/ to see all the tasks.
>> Retrieving tasks will either be the same (v3/tasks/) or easier if they know
>> the task they are looking for (v3/tasks/file/syncs/)
>>
>
> My understanding of the master/detail pattern we are using for our other
> resources is that each task's URI will look something like this:
> /api/v3/tasks/file/syncs//
>
> If the above is true, that means the client has to be aware of
> '/api/v3/tasks/file/syncs//' resource. The user would need to know
> how to interpret the response from each task type. There could be an
> infinite number of these tasks with many different arguments. So writing
> client code handling all those responses could become a nightmare. Right
> now the user just needs to know how to interpret the response from
> '/api/v3/tasks/' for all tasks.
>

I might be missing your point on this one. If you are talking about polling
tasks, the href of a new Task is returned when the Task is created.The
client should use the href that is returned, never string formatting urls.
Even so, the Tasks in this proposal will be part of the API schema, so
clients (and docs users) know where the endpoints are and what the
arguments should be. This would actually be a big advantage over the way
things are now. AFAIK, v3/importers/1234/sync/ is invisible to clients, so
we replace one invisible endpoint with one visible endpoint. No complexity
added.

If the client doesn't know how to interpret extra fields (the Task
parameters), they don't need to. All tasks share the same base fields, and
those are the fields that need to be interpreted when the client is polling
a task.


> Value #1: We have to do something to address the problem that
> adding/removing content to a repository without plugin input is incorrect.
> This proposal is one possibility, but it isn't valid to compare the value
> against doing nothing. Instead, if you don't like this option, we need to
> compare it to other proposals for how to involve plugins in tasks.


> We don't have a problem. I agree that a user needs to know a whole lot of
> information to correctly create a working repository version for a complex
> content type such as a Docker manifest. Without all this information on
> hand, the user could easily create a broken repository version. However, a
> rich client could solve that problem. If the plugin writer wants to support
> simpler clients, she can provide an additional URL for handling the
> validation and additional logic for creating a repository version. We
> should probably have a recommended convention for plugin authors to use
> when adding additional URLs.


What I find troubling is that the most basic action (add/remove) suddenly
becomes complex. If using the file plugin POST to
v3/repository/1234/versions/. If using docker, POST to
v3/plugins/docker/a

Re: [Pulp-dev] Plugin relationship to tasks

[Brian Bouterse]

I also want to look at how this will affect our user experience. With this
change, I believe plugin tasks are put at the center of the user experience
and the repositories, content, and versions are moved to the background.
The user would think about what they can do foremost as a series of tasks.
This is great in terms of organization, but I'm concerned its stepping away
from what makes pulp special. I think Pulp is special because of its focus
on content management (repos, version, uploads, sync, promotion,
copy/delete, publish, export). Having the user think about Pulp as a
task-running system takes Pulp a step closer to a software like RunDeck (
https://www.rundeck.com/open-source).

What I like about the existing user experience (and API) is that it all
centers around Pulp's value proposition (importers, syncs, associations,
etc).

Does this spark any thoughts or ideas from others?


On Mon, Mar 26, 2018 at 4:43 PM, Dennis Kliban  wrote:

> On Mon, Mar 26, 2018 at 3:38 PM, Austin Macdonald 
> wrote:
>
>>
>>
>> On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban 
>> wrote:
>>
>>> This proposal does not make the plugin writer's job any easier. This
>>> simply changes where the plugin writer is providing validation logic, in a
>>> serializer vs. in a view.
>>>
>>
>> It does make the plugin writer's job easier *because* it changes how the
>> validation is provided, Plugin writers already have to understand how to
>> create models/serializers (and it is very simple) for all the other
>> objects. Seriallizer validation is much simpler than viewset validation
>> because serializers are literally designed to do this.
>>
>
> If we don't change anything, a plugin writer is responsible for creating a
> task that creates a Repository Version and a view which will perform
> validation and then dispatches a task. If we do make the proposed change,
> the plugin writer will still have to write the task code and then also a
> serializer that performs validation. The proposed change only removes the
> responsibility of dispatching a task which is 1 line of code.
>
>
>>
>>
>>>
>>> The other problem I see is that complexity for users is increased.
>>> Instead of having 1 resource for tracking task progress, there would be an
>>> infinite number of resources for tracking progress of tasks.
>>>
>>
>> In the proposal, Tasks are Master/Detail. The user doesn't have to
>> "track" it at all. They can still use v3/tasks/ to see all the tasks.
>> Retrieving tasks will either be the same (v3/tasks/) or easier if they know
>> the task they are looking for (v3/tasks/file/syncs/)
>>
>
> My understanding of the master/detail pattern we are using for our other
> resources is that each task's URI will look something like this:
> /api/v3/tasks/file/syncs//
>
> If the above is true, that means the client has to be aware of
> '/api/v3/tasks/file/syncs//' resource. The user would need to know
> how to interpret the response from each task type. There could be an
> infinite number of these tasks with many different arguments. So writing
> client code handling all those responses could become a nightmare. Right
> now the user just needs to know how to interpret the response from
> '/api/v3/tasks/' for all tasks.
>
>
>>
>>
>>> I don't see the value added by the proposed change.
>>>
>>
>> Value #1: We have to do something to address the problem that
>> adding/removing content to a repository without plugin input is incorrect.
>> This proposal is one possibility, but it isn't valid to compare the value
>> against doing nothing. Instead, if you don't like this option, we need to
>> compare it to other proposals for how to involve plugins in tasks.
>>
>>
> We don't have a problem. I agree that a user needs to know a whole lot of
> information to correctly create a working repository version for a complex
> content type such as a Docker manifest. Without all this information on
> hand, the user could easily create a broken repository version. However, a
> rich client could solve that problem. If the plugin writer wants to support
> simpler clients, she can provide an additional URL for handling the
> validation and additional logic for creating a repository version. We
> should probably have a recommended convention for plugin authors to use
> when adding additional URLs.
>
>
>> Value #2: Plugin writers no longer need to dispatch tasks. Plugin writers
>> shouldn't need to understand the complexities of our tasking system.
>> Pulpcore can handle task dispatching generally, plugin writers just set
>> simple class attributes instead.
>>
>>
> Dispatching a task is 1 line of code. Plugin writers should be aware of
> resource locking and how it works. They get to decide what resources are
> appropriate to lock when performing asynchronous tasks.
>
>
>> Value #3: It removes "action" endpoints and replaces them with RESTful
>> "creation of task" POSTs.
>>
>>
> +1 to this. However, not enough value by itself.
>
>
>> Value #4: It creates a convention for endpoin

Re: [Pulp-dev] Plugin relationship to tasks

[Dennis Kliban]

On Mon, Mar 26, 2018 at 3:38 PM, Austin Macdonald  wrote:

>
>
> On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban  wrote:
>
>> This proposal does not make the plugin writer's job any easier. This
>> simply changes where the plugin writer is providing validation logic, in a
>> serializer vs. in a view.
>>
>
> It does make the plugin writer's job easier *because* it changes how the
> validation is provided, Plugin writers already have to understand how to
> create models/serializers (and it is very simple) for all the other
> objects. Seriallizer validation is much simpler than viewset validation
> because serializers are literally designed to do this.
>

If we don't change anything, a plugin writer is responsible for creating a
task that creates a Repository Version and a view which will perform
validation and then dispatches a task. If we do make the proposed change,
the plugin writer will still have to write the task code and then also a
serializer that performs validation. The proposed change only removes the
responsibility of dispatching a task which is 1 line of code.


>
>
>>
>> The other problem I see is that complexity for users is increased.
>> Instead of having 1 resource for tracking task progress, there would be an
>> infinite number of resources for tracking progress of tasks.
>>
>
> In the proposal, Tasks are Master/Detail. The user doesn't have to "track"
> it at all. They can still use v3/tasks/ to see all the tasks. Retrieving
> tasks will either be the same (v3/tasks/) or easier if they know the task
> they are looking for (v3/tasks/file/syncs/)
>

My understanding of the master/detail pattern we are using for our other
resources is that each task's URI will look something like this:
/api/v3/tasks/file/syncs//

If the above is true, that means the client has to be aware of
'/api/v3/tasks/file/syncs//' resource. The user would need to know
how to interpret the response from each task type. There could be an
infinite number of these tasks with many different arguments. So writing
client code handling all those responses could become a nightmare. Right
now the user just needs to know how to interpret the response from
'/api/v3/tasks/' for all tasks.


>
>
>> I don't see the value added by the proposed change.
>>
>
> Value #1: We have to do something to address the problem that
> adding/removing content to a repository without plugin input is incorrect.
> This proposal is one possibility, but it isn't valid to compare the value
> against doing nothing. Instead, if you don't like this option, we need to
> compare it to other proposals for how to involve plugins in tasks.
>
>
We don't have a problem. I agree that a user needs to know a whole lot of
information to correctly create a working repository version for a complex
content type such as a Docker manifest. Without all this information on
hand, the user could easily create a broken repository version. However, a
rich client could solve that problem. If the plugin writer wants to support
simpler clients, she can provide an additional URL for handling the
validation and additional logic for creating a repository version. We
should probably have a recommended convention for plugin authors to use
when adding additional URLs.


> Value #2: Plugin writers no longer need to dispatch tasks. Plugin writers
> shouldn't need to understand the complexities of our tasking system.
> Pulpcore can handle task dispatching generally, plugin writers just set
> simple class attributes instead.
>
>
Dispatching a task is 1 line of code. Plugin writers should be aware of
resource locking and how it works. They get to decide what resources are
appropriate to lock when performing asynchronous tasks.


> Value #3: It removes "action" endpoints and replaces them with RESTful
> "creation of task" POSTs.
>
>
+1 to this. However, not enough value by itself.


> Value #4: It creates a convention for endpoints. Right now we have
>
>
> v3/importers/file/1234/sync/
> v3/publishers/python/1234/publish/
>
> There is no logical place for add/remove to go because add/remove will not
> involve an importer. It will be irritating to the users if we don't
> encourage some consistency between plugins for the simplest Pulp
> functionality.
>
>
> Value #5: Task history becomes much more useful. Currently, the only data
> on a task that connects it to user-facing objects is "created_resource".
> This proposal will allow users to filter tasks by parameters. For example,
> they can filter sync tasks by "repository" or "importer". They can also use
> the detail endpoints (v3/tasks/file/ or v3/tasks/file/syncs/) to list tasks
> based on the plugin or action.
>
>
I don't think tracking history is one of the use cases we decided to
support in the MVP. We should have a separate discussion on how we could
accomplish this.


> Value #6: Since the parameters of Task are saved, it will be possible to
> repeat tasks.
>

Same thing as for #5.
___
Pulp-dev ma

Re: [Pulp-dev] Plugin relationship to tasks

[Daniel Alley]

Nice choice of music :)

On Mon, Mar 26, 2018 at 4:14 PM, Milan Kovacik  wrote:

> On Mon, Mar 26, 2018 at 9:38 PM, Austin Macdonald 
> wrote:
> >
> >
> > On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban 
> wrote:
> >>
> >> This proposal does not make the plugin writer's job any easier. This
> >> simply changes where the plugin writer is providing validation logic,
> in a
> >> serializer vs. in a view.
> >
> >
> > It does make the plugin writer's job easier *because* it changes how the
> > validation is provided, Plugin writers already have to understand how to
> > create models/serializers (and it is very simple) for all the other
> objects.
> > Seriallizer validation is much simpler than viewset validation because
> > serializers are literally designed to do this.
> >
> >>
> >>
> >> The other problem I see is that complexity for users is increased.
> Instead
> >> of having 1 resource for tracking task progress, there would be an
> infinite
> >> number of resources for tracking progress of tasks.
> >
> >
> > In the proposal, Tasks are Master/Detail. The user doesn't have to
> "track"
> > it at all. They can still use v3/tasks/ to see all the tasks. Retrieving
> > tasks will either be the same (v3/tasks/) or easier if they know the task
> > they are looking for (v3/tasks/file/syncs/)
> >
> >>
> >> I don't see the value added by the proposed change.
> >
> >
> > Value #1: We have to do something to address the problem that
> > adding/removing content to a repository without plugin input is
> incorrect.
> > This proposal is one possibility, but it isn't valid to compare the value
> > against doing nothing. Instead, if you don't like this option, we need to
> > compare it to other proposals for how to involve plugins in tasks.
> >
> > Value #2: Plugin writers no longer need to dispatch tasks. Plugin writers
> > shouldn't need to understand the complexities of our tasking system.
> > Pulpcore can handle task dispatching generally, plugin writers just set
> > simple class attributes instead.
> >
> > Value #3: It removes "action" endpoints and replaces them with RESTful
> > "creation of task" POSTs.
> >
> > Value #4: It creates a convention for endpoints. Right now we have
> >
> >
> > v3/importers/file/1234/sync/
> > v3/publishers/python/1234/publish/
> >
> > There is no logical place for add/remove to go because add/remove will
> not
> > involve an importer. It will be irritating to the users if we don't
> > encourage some consistency between plugins for the simplest Pulp
> > functionality.
> >
> >
> > Value #5: Task history becomes much more useful. Currently, the only
> data on
> > a task that connects it to user-facing objects is "created_resource".
> This
> > proposal will allow users to filter tasks by parameters. For example,
> they
> > can filter sync tasks by "repository" or "importer". They can also use
> the
> > detail endpoints (v3/tasks/file/ or v3/tasks/file/syncs/) to list tasks
> > based on the plugin or action.
> >
> > Value #6: Since the parameters of Task are saved, it will be possible to
> > repeat tasks.
>
> btw A moving picture of how it's going to look like on the docs side
> of things: https://youtu.be/E7zC6SElm4g ;)
>
> --
> milan
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

On Mon, Mar 26, 2018 at 9:38 PM, Austin Macdonald  wrote:
>
>
> On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban  wrote:
>>
>> This proposal does not make the plugin writer's job any easier. This
>> simply changes where the plugin writer is providing validation logic, in a
>> serializer vs. in a view.
>
>
> It does make the plugin writer's job easier *because* it changes how the
> validation is provided, Plugin writers already have to understand how to
> create models/serializers (and it is very simple) for all the other objects.
> Seriallizer validation is much simpler than viewset validation because
> serializers are literally designed to do this.
>
>>
>>
>> The other problem I see is that complexity for users is increased. Instead
>> of having 1 resource for tracking task progress, there would be an infinite
>> number of resources for tracking progress of tasks.
>
>
> In the proposal, Tasks are Master/Detail. The user doesn't have to "track"
> it at all. They can still use v3/tasks/ to see all the tasks. Retrieving
> tasks will either be the same (v3/tasks/) or easier if they know the task
> they are looking for (v3/tasks/file/syncs/)
>
>>
>> I don't see the value added by the proposed change.
>
>
> Value #1: We have to do something to address the problem that
> adding/removing content to a repository without plugin input is incorrect.
> This proposal is one possibility, but it isn't valid to compare the value
> against doing nothing. Instead, if you don't like this option, we need to
> compare it to other proposals for how to involve plugins in tasks.
>
> Value #2: Plugin writers no longer need to dispatch tasks. Plugin writers
> shouldn't need to understand the complexities of our tasking system.
> Pulpcore can handle task dispatching generally, plugin writers just set
> simple class attributes instead.
>
> Value #3: It removes "action" endpoints and replaces them with RESTful
> "creation of task" POSTs.
>
> Value #4: It creates a convention for endpoints. Right now we have
>
>
> v3/importers/file/1234/sync/
> v3/publishers/python/1234/publish/
>
> There is no logical place for add/remove to go because add/remove will not
> involve an importer. It will be irritating to the users if we don't
> encourage some consistency between plugins for the simplest Pulp
> functionality.
>
>
> Value #5: Task history becomes much more useful. Currently, the only data on
> a task that connects it to user-facing objects is "created_resource". This
> proposal will allow users to filter tasks by parameters. For example, they
> can filter sync tasks by "repository" or "importer". They can also use the
> detail endpoints (v3/tasks/file/ or v3/tasks/file/syncs/) to list tasks
> based on the plugin or action.
>
> Value #6: Since the parameters of Task are saved, it will be possible to
> repeat tasks.

btw A moving picture of how it's going to look like on the docs side
of things: https://youtu.be/E7zC6SElm4g ;)

--
milan

___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

On Mon, Mar 26, 2018 at 2:30 PM, Dennis Kliban  wrote:

> This proposal does not make the plugin writer's job any easier. This
> simply changes where the plugin writer is providing validation logic, in a
> serializer vs. in a view.
>

It does make the plugin writer's job easier *because* it changes how the
validation is provided, Plugin writers already have to understand how to
create models/serializers (and it is very simple) for all the other
objects. Seriallizer validation is much simpler than viewset validation
because serializers are literally designed to do this.


>
> The other problem I see is that complexity for users is increased. Instead
> of having 1 resource for tracking task progress, there would be an infinite
> number of resources for tracking progress of tasks.
>

In the proposal, Tasks are Master/Detail. The user doesn't have to "track"
it at all. They can still use v3/tasks/ to see all the tasks. Retrieving
tasks will either be the same (v3/tasks/) or easier if they know the task
they are looking for (v3/tasks/file/syncs/)


> I don't see the value added by the proposed change.
>

Value #1: We have to do something to address the problem that
adding/removing content to a repository without plugin input is incorrect.
This proposal is one possibility, but it isn't valid to compare the value
against doing nothing. Instead, if you don't like this option, we need to
compare it to other proposals for how to involve plugins in tasks.

Value #2: Plugin writers no longer need to dispatch tasks. Plugin writers
shouldn't need to understand the complexities of our tasking system.
Pulpcore can handle task dispatching generally, plugin writers just set
simple class attributes instead.

Value #3: It removes "action" endpoints and replaces them with RESTful
"creation of task" POSTs.

Value #4: It creates a convention for endpoints. Right now we have


v3/importers/file/1234/sync/
v3/publishers/python/1234/publish/

There is no logical place for add/remove to go because add/remove will not
involve an importer. It will be irritating to the users if we don't
encourage some consistency between plugins for the simplest Pulp
functionality.


Value #5: Task history becomes much more useful. Currently, the only data
on a task that connects it to user-facing objects is "created_resource".
This proposal will allow users to filter tasks by parameters. For example,
they can filter sync tasks by "repository" or "importer". They can also use
the detail endpoints (v3/tasks/file/ or v3/tasks/file/syncs/) to list tasks
based on the plugin or action.

Value #6: Since the parameters of Task are saved, it will be possible to
repeat tasks.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Dennis Kliban]

This proposal does not make the plugin writer's job any easier. This simply
changes where the plugin writer is providing validation logic, in a
serializer vs. in a view.

The other problem I see is that complexity for users is increased. Instead
of having 1 resource for tracking task progress, there would be an infinite
number of resources for tracking progress of tasks.

I don't see the value added by the proposed change.

On Fri, Mar 23, 2018 at 1:27 PM, Austin Macdonald  wrote:

> To make sure this was feasable, I've created proof of concept PRs for
> pulpcore and pulp_file. All of the proposed changes are proofed, but not
> all work is done. For example, sync is implemented, but not publish.
>
>- https://github.com/pulp/pulp/pull/3394
>- https://github.com/pulp/pulp_file/pull/61
>
> I am very pleased with how it turned out, and I think it will make plugin
> writing even easier. I documented the design on a redmine issue. I
> encourage feedback/discussion on the issue.
>
> https://pulp.plan.io/issues/3522
>
> On Tue, Mar 20, 2018 at 7:46 AM, Milan Kovacik 
> wrote:
>
>> On Fri, Mar 16, 2018 at 4:55 PM, Milan Kovacik 
>> wrote:
>> >
>> >
>> >
>> > On Thu, Mar 15, 2018 at 9:21 PM, Austin Macdonald 
>> wrote:
>> >>
>> >> I spoke with daviddavis about this and I would like to narrow the
>> scope a bit. This discussion should be limited to endpoints that deploy
>> tasks. The possibility for collisions that David pointed out regarding
>> v3/content// is real, but should be discussed separately because
>> Content objects should follow the pattern of the other plugin defined
>> objects. All of them are master/detail so the namespacing
>> v3/plugin//content/ would go deeply against the grain.
>> >>
>> >>
>> >> On Mar 15, 2018 10:53, "David Davis"  wrote:
>> >>>
>> >>> I have an amendment to this proposal. Instead of namespacing just the
>> plugin task routes, I’d propose we namespace all plugin routes. Thus all
>> plugin routes would be namespaced under something like
>> “/api/v3/plugin//“. For example, “/api/v3/content/file/” gets
>> moved to “/api/v3/plugin/file/content/” and so on.
>> >>>
>> >>>
>> >>> For task routes, plugins could still route to a pulpcore
>> viewset/action:
>> >>>
>> >>> User route: POST /api/v3/plugin/file/repository_version/
>> add_content=[…]
>> >>>
>> >>> Routes to: POST /api/v3/tasks plugin=file
>> task=create_repository_version add_content=[…]
>> >>
>> >>
>> >> AFAIK, we don't have a good mechanism for validating and re-routing
>> endpoints quite like this. Instead, a view or viewset would be created,
>> validation performed, and then a task is deployed. The task itself could be
>> a vanilla task from pulpcore though. Another issue is that a POST to
>> v3/plugin/file/repository_version/ implies that the resultant
>> repository_version is typed in some way. The created resource href would
>> still take the form v3/repositories/1234/versions/2/, so I think this is
>> a little misleading.
>> >
>> >
>> > +1 to generic Pulp concepts/objects staying outside of a plug-in
>> namespace/path
>> >
>> >>
>> >>
>> >> For the other endpoint:
>> >>  /api/v3/tasks plugin=file task=create_repository_version
>> add_content=[…]
>> >>
>> >> There is a still the problem is that POST requests should not contain
>> parameters that influence the allowable parameter set. For instance,
>> plugin=python might require an additional parameter that is not allowed for
>> the file plugin. In particular, this affects the auto-documented REST API.
>> >>
>> >>
>> >> However, the concept in general could work. I've adapted David's
>> suggestion and will present it side by side with the original idea. Note,
>> this only applies to "actions", which are sync, publish, add/remove, and
>> plugin-specific actions (including rich-dependency adding).
>> >>
>> >> The difference between the two ideas is based in semantics. Both would
>> deploy the same task functions.
>> >>
>> >> "Action Based"
>> >> POST v3/plugin/file/sync/
>> >> Note that "sync" is singular. This is a file-plugin action to sync
>> using the body parameters.
>> >
>> >
>> > - 1 as this might imply mixing  and  actions
>> while all the time, asynchronous actions, such as the sync call, always
>> create a task object; the user would be creating the task objects at one
>> endpoint but deleting, filtering, tracking those at another.
>> >
>> >>
>> >>
>> >> "Task Object Based"
>> >> POST v3/tasks/file/syncs/
>> >> Note that "syncs" is plural. This is a POST that creates a file-sync
>> task object,.
>>
>> Btw an interesting consequence of tasks not being objects in a file
>> importer and publisher are these two issues:
>>  - The parameters in the API schema for the sync and publish endpoints
>> are incorrect [1]
>>  - Not able to access documentation endpoint [2]
>>
>> In the former issue, the DRF isn't able to automatically figure out
>> schema for e.g the `sync` endpoint of the file plug-in importer.
>> The reason for it is even though the `sync` verb is att

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

To make sure this was feasable, I've created proof of concept PRs for
pulpcore and pulp_file. All of the proposed changes are proofed, but not
all work is done. For example, sync is implemented, but not publish.

   - https://github.com/pulp/pulp/pull/3394
   - https://github.com/pulp/pulp_file/pull/61

I am very pleased with how it turned out, and I think it will make plugin
writing even easier. I documented the design on a redmine issue. I
encourage feedback/discussion on the issue.

https://pulp.plan.io/issues/3522

On Tue, Mar 20, 2018 at 7:46 AM, Milan Kovacik  wrote:

> On Fri, Mar 16, 2018 at 4:55 PM, Milan Kovacik 
> wrote:
> >
> >
> >
> > On Thu, Mar 15, 2018 at 9:21 PM, Austin Macdonald 
> wrote:
> >>
> >> I spoke with daviddavis about this and I would like to narrow the scope
> a bit. This discussion should be limited to endpoints that deploy tasks.
> The possibility for collisions that David pointed out regarding
> v3/content// is real, but should be discussed separately because
> Content objects should follow the pattern of the other plugin defined
> objects. All of them are master/detail so the namespacing
> v3/plugin//content/ would go deeply against the grain.
> >>
> >>
> >> On Mar 15, 2018 10:53, "David Davis"  wrote:
> >>>
> >>> I have an amendment to this proposal. Instead of namespacing just the
> plugin task routes, I’d propose we namespace all plugin routes. Thus all
> plugin routes would be namespaced under something like
> “/api/v3/plugin//“. For example, “/api/v3/content/file/” gets
> moved to “/api/v3/plugin/file/content/” and so on.
> >>>
> >>>
> >>> For task routes, plugins could still route to a pulpcore
> viewset/action:
> >>>
> >>> User route: POST /api/v3/plugin/file/repository_version/
> add_content=[…]
> >>>
> >>> Routes to: POST /api/v3/tasks plugin=file
> task=create_repository_version add_content=[…]
> >>
> >>
> >> AFAIK, we don't have a good mechanism for validating and re-routing
> endpoints quite like this. Instead, a view or viewset would be created,
> validation performed, and then a task is deployed. The task itself could be
> a vanilla task from pulpcore though. Another issue is that a POST to
> v3/plugin/file/repository_version/ implies that the resultant
> repository_version is typed in some way. The created resource href would
> still take the form v3/repositories/1234/versions/2/, so I think this is
> a little misleading.
> >
> >
> > +1 to generic Pulp concepts/objects staying outside of a plug-in
> namespace/path
> >
> >>
> >>
> >> For the other endpoint:
> >>  /api/v3/tasks plugin=file task=create_repository_version
> add_content=[…]
> >>
> >> There is a still the problem is that POST requests should not contain
> parameters that influence the allowable parameter set. For instance,
> plugin=python might require an additional parameter that is not allowed for
> the file plugin. In particular, this affects the auto-documented REST API.
> >>
> >>
> >> However, the concept in general could work. I've adapted David's
> suggestion and will present it side by side with the original idea. Note,
> this only applies to "actions", which are sync, publish, add/remove, and
> plugin-specific actions (including rich-dependency adding).
> >>
> >> The difference between the two ideas is based in semantics. Both would
> deploy the same task functions.
> >>
> >> "Action Based"
> >> POST v3/plugin/file/sync/
> >> Note that "sync" is singular. This is a file-plugin action to sync
> using the body parameters.
> >
> >
> > - 1 as this might imply mixing  and  actions
> while all the time, asynchronous actions, such as the sync call, always
> create a task object; the user would be creating the task objects at one
> endpoint but deleting, filtering, tracking those at another.
> >
> >>
> >>
> >> "Task Object Based"
> >> POST v3/tasks/file/syncs/
> >> Note that "syncs" is plural. This is a POST that creates a file-sync
> task object,.
>
> Btw an interesting consequence of tasks not being objects in a file
> importer and publisher are these two issues:
>  - The parameters in the API schema for the sync and publish endpoints
> are incorrect [1]
>  - Not able to access documentation endpoint [2]
>
> In the former issue, the DRF isn't able to automatically figure out
> schema for e.g the `sync` endpoint of the file plug-in importer.
> The reason for it is even though the `sync` verb is attached to an
> importer object, the parameters passed in aren't supposed to be
> importer details so they shouldn't use the importer serializer to get
> e.g auto-documented.
> Moreover, the result of a call to this endpoint is supposed to be a
> task (json object), schema and serialization of which has to be
> repeated for the `sync` endpoint to some extent.
> Last but not least, it isn't clear whether the `sync` verb should
> belong to an importer or a repo, or something else as the call
> "responsibility" is spread somewhat between both and none.
>
> The latter issue is a consequence of the lack of sup

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

On Fri, Mar 16, 2018 at 4:55 PM, Milan Kovacik  wrote:
>
>
>
> On Thu, Mar 15, 2018 at 9:21 PM, Austin Macdonald  wrote:
>>
>> I spoke with daviddavis about this and I would like to narrow the scope a 
>> bit. This discussion should be limited to endpoints that deploy tasks. The 
>> possibility for collisions that David pointed out regarding 
>> v3/content// is real, but should be discussed separately because 
>> Content objects should follow the pattern of the other plugin defined 
>> objects. All of them are master/detail so the namespacing 
>> v3/plugin//content/ would go deeply against the grain.
>>
>>
>> On Mar 15, 2018 10:53, "David Davis"  wrote:
>>>
>>> I have an amendment to this proposal. Instead of namespacing just the 
>>> plugin task routes, I’d propose we namespace all plugin routes. Thus all 
>>> plugin routes would be namespaced under something like 
>>> “/api/v3/plugin//“. For example, “/api/v3/content/file/” gets 
>>> moved to “/api/v3/plugin/file/content/” and so on.
>>>
>>>
>>> For task routes, plugins could still route to a pulpcore viewset/action:
>>>
>>> User route: POST /api/v3/plugin/file/repository_version/ add_content=[…]
>>>
>>> Routes to: POST /api/v3/tasks plugin=file task=create_repository_version 
>>> add_content=[…]
>>
>>
>> AFAIK, we don't have a good mechanism for validating and re-routing 
>> endpoints quite like this. Instead, a view or viewset would be created, 
>> validation performed, and then a task is deployed. The task itself could be 
>> a vanilla task from pulpcore though. Another issue is that a POST to 
>> v3/plugin/file/repository_version/ implies that the resultant 
>> repository_version is typed in some way. The created resource href would 
>> still take the form v3/repositories/1234/versions/2/, so I think this is a 
>> little misleading.
>
>
> +1 to generic Pulp concepts/objects staying outside of a plug-in 
> namespace/path
>
>>
>>
>> For the other endpoint:
>>  /api/v3/tasks plugin=file task=create_repository_version add_content=[…]
>>
>> There is a still the problem is that POST requests should not contain 
>> parameters that influence the allowable parameter set. For instance, 
>> plugin=python might require an additional parameter that is not allowed for 
>> the file plugin. In particular, this affects the auto-documented REST API.
>>
>>
>> However, the concept in general could work. I've adapted David's suggestion 
>> and will present it side by side with the original idea. Note, this only 
>> applies to "actions", which are sync, publish, add/remove, and 
>> plugin-specific actions (including rich-dependency adding).
>>
>> The difference between the two ideas is based in semantics. Both would 
>> deploy the same task functions.
>>
>> "Action Based"
>> POST v3/plugin/file/sync/
>> Note that "sync" is singular. This is a file-plugin action to sync using the 
>> body parameters.
>
>
> - 1 as this might imply mixing  and  actions while 
> all the time, asynchronous actions, such as the sync call, always create a 
> task object; the user would be creating the task objects at one endpoint but 
> deleting, filtering, tracking those at another.
>
>>
>>
>> "Task Object Based"
>> POST v3/tasks/file/syncs/
>> Note that "syncs" is plural. This is a POST that creates a file-sync task 
>> object,.

Btw an interesting consequence of tasks not being objects in a file
importer and publisher are these two issues:
 - The parameters in the API schema for the sync and publish endpoints
are incorrect [1]
 - Not able to access documentation endpoint [2]

In the former issue, the DRF isn't able to automatically figure out
schema for e.g the `sync` endpoint of the file plug-in importer.
The reason for it is even though the `sync` verb is attached to an
importer object, the parameters passed in aren't supposed to be
importer details so they shouldn't use the importer serializer to get
e.g auto-documented.
Moreover, the result of a call to this endpoint is supposed to be a
task (json object), schema and serialization of which has to be
repeated for the `sync` endpoint to some extent.
Last but not least, it isn't clear whether the `sync` verb should
belong to an importer or a repo, or something else as the call
"responsibility" is spread somewhat between both and none.

The latter issue is a consequence of the lack of support in DRF for
what the `sync` verb requires documenting thru its schema, which IMO
underlines the `sync` endpoint being odd, a misplaced task probably.

Cheers,
milan



[1] https://pulp.plan.io/issues/3351
[2] https://pulp.plan.io/issues/3420

>
> + 1 for typed objects
>
> # adding content to a repo version (already mentioned by Austin but I 
> like it)
> POST@v3/tasks/docker/content_additions/
>
> # synchronize the Zoo repo
> pulp-admin create tasks rpm syncs --repo-id zoo --importer-id 
> zoo-importer --sync-config-override '{"ssl": "false"}' --format=json | tee 
> /tmp/inttest01_sync_task.json | pulp-admin track tasks --format json -
>
>

Re: [Pulp-dev] Plugin relationship to tasks

[Milan Kovacik]

On Thu, Mar 15, 2018 at 9:21 PM, Austin Macdonald  wrote:

> I spoke with daviddavis about this and I would like to narrow the scope a
> bit. This discussion should be limited to endpoints that deploy tasks. The
> possibility for collisions that David pointed out regarding
> v3/content// is real, but should be discussed separately because
> Content objects should follow the pattern of the other plugin defined
> objects. All of them are master/detail so the namespacing
> v3/plugin//content/ would go deeply against the grain.
>
>
> On Mar 15, 2018 10:53, "David Davis"  wrote:
>
> I have an amendment to this proposal. Instead of namespacing just the
> plugin task routes, I’d propose we namespace all plugin routes. Thus all
> plugin routes would be namespaced under something like
> “/api/v3/plugin//“. For example, “/api/v3/content/file/” gets
> moved to “/api/v3/plugin/file/content/” and so on.
>
>
> For task routes, plugins could still route to a pulpcore viewset/action:
>
> User route: POST /api/v3/plugin/file/repository_version/ add_content=[…]
>
> Routes to: POST /api/v3/tasks plugin=file task=create_repository_version
> add_content=[…]
>
>
> AFAIK, we don't have a good mechanism for validating and re-routing
> endpoints quite like this. Instead, a view or viewset would be created,
> validation performed, and then a task is deployed. The task itself could be
> a vanilla task from pulpcore though. Another issue is that a POST to
> v3/plugin/file/repository_version/ implies that the resultant
> repository_version is typed in some way. The created resource href would
> still take the form v3/repositories/1234/versions/2/, so I think this is
> a little misleading.
>

+1 to generic Pulp concepts/objects staying outside of a plug-in
namespace/path


>
> For the other endpoint:
>  /api/v3/tasks plugin=file task=create_repository_version add_content=[…]
>
> There is a still the problem is that POST requests should not contain
> parameters that influence the allowable parameter set. For instance,
> plugin=python might require an additional parameter that is not allowed for
> the file plugin. In particular, this affects the auto-documented REST API.
>

> However, the concept in general could work. I've adapted David's
> suggestion and will present it side by side with the original idea. Note,
> this only applies to "actions", which are sync, publish, add/remove, and
> plugin-specific actions (including rich-dependency adding).
>
> The difference between the two ideas is based in semantics. Both would
> deploy the same task functions.
>
> "Action Based"
> POST v3/plugin/file/sync/
> Note that "sync" is singular. This is a file-plugin action to sync using
> the body parameters.
>

- 1 as this might imply mixing  and  actions
while all the time, asynchronous actions, such as the sync call, always
create a task object; the user would be creating the task objects at one
endpoint but deleting, filtering, tracking those at another.


>
> "Task Object Based"
> POST v3/tasks/file/syncs/
> Note that "syncs" is plural. This is a POST that creates a file-sync task
> object,.
>

+ 1 for typed objects

# adding content to a repo version (already mentioned by Austin but I
like it)
POST@v3/tasks/docker/content_additions/

# synchronize the Zoo repo
pulp-admin create tasks rpm syncs --repo-id zoo --importer-id
zoo-importer --sync-config-override '{"ssl": "false"}' --format=json | tee
/tmp/inttest01_sync_task.json | pulp-admin track tasks --format json -

# get all docker content additions tasks since yesterday
pulp-admin list tasks docker content_additions --format=csv
--fields=id,worker_id --filter=started_at__gt=

# get all tasks since yesterday
pulp-admin list tasks  --filter=started_at__gt= --format=yaml


> Discussion:
> Another consideration is "Where will a Live API live?". A benefit of
> David's proposal is that we would be providing plugins with a namespaced
> area to do whatever they need, from live api to actions. We probably would
> want to create this namespace for things like live apis even if we go in
> the tasks/file/syncs/ direction. The only downside of this part of David's
> proposal is that our endpoints will still be "action" verbs instead of
> nouns, and I am ok with that.
>

I'd suggest live API actions that create task object subclasses (generic
Pulp concept) reside in namespaced tasks path; this would still allow the
user to apply generic concepts
POST@/v3/tasks/file/frobnications/

pulp-admin create tasks file frobications --foo=bar --format json |
pulp-admin track tasks --format json --

Other live API endpoints (not creating task objects) might be kept under
namespaces
POST@v3/plugins/file/shake/<{level: strong}

pulp-admin plugins file shake --level=strong


> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
Cheers,
milan
_

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

I spoke with daviddavis about this and I would like to narrow the scope a
bit. This discussion should be limited to endpoints that deploy tasks. The
possibility for collisions that David pointed out regarding
v3/content// is real, but should be discussed separately because
Content objects should follow the pattern of the other plugin defined
objects. All of them are master/detail so the namespacing
v3/plugin//content/ would go deeply against the grain.


On Mar 15, 2018 10:53, "David Davis"  wrote:

I have an amendment to this proposal. Instead of namespacing just the
plugin task routes, I’d propose we namespace all plugin routes. Thus all
plugin routes would be namespaced under something like
“/api/v3/plugin//“. For example, “/api/v3/content/file/” gets
moved to “/api/v3/plugin/file/content/” and so on.


For task routes, plugins could still route to a pulpcore viewset/action:

User route: POST /api/v3/plugin/file/repository_version/ add_content=[…]

Routes to: POST /api/v3/tasks plugin=file task=create_repository_version
add_content=[…]


AFAIK, we don't have a good mechanism for validating and re-routing
endpoints quite like this. Instead, a view or viewset would be created,
validation performed, and then a task is deployed. The task itself could be
a vanilla task from pulpcore though. Another issue is that a POST to
v3/plugin/file/repository_version/ implies that the resultant
repository_version is typed in some way. The created resource href would
still take the form v3/repositories/1234/versions/2/, so I think this is a
little misleading.

For the other endpoint:
 /api/v3/tasks plugin=file task=create_repository_version add_content=[…]

There is a still the problem is that POST requests should not contain
parameters that influence the allowable parameter set. For instance,
plugin=python might require an additional parameter that is not allowed for
the file plugin. In particular, this affects the auto-documented REST API.

However, the concept in general could work. I've adapted David's suggestion
and will present it side by side with the original idea. Note, this only
applies to "actions", which are sync, publish, add/remove, and
plugin-specific actions (including rich-dependency adding).

The difference between the two ideas is based in semantics. Both would
deploy the same task functions.

"Action Based"
POST v3/plugin/file/sync/
Note that "sync" is singular. This is a file-plugin action to sync using
the body parameters.

"Task Object Based"
POST v3/tasks/file/syncs/
Note that "syncs" is plural. This is a POST that creates a file-sync task
object,.

Discussion:
Another consideration is "Where will a Live API live?". A benefit of
David's proposal is that we would be providing plugins with a namespaced
area to do whatever they need, from live api to actions. We probably would
want to create this namespace for things like live apis even if we go in
the tasks/file/syncs/ direction. The only downside of this part of David's
proposal is that our endpoints will still be "action" verbs instead of
nouns, and I am ok with that.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[David Davis]

I have an amendment to this proposal. Instead of namespacing just the
plugin task routes, I’d propose we namespace all plugin routes. Thus all
plugin routes would be namespaced under something like
“/api/v3/plugin//“. For example, “/api/v3/content/file/” gets
moved to “/api/v3/plugin/file/content/” and so on.

For task routes, plugins could still route to a single pulpcore
viewset/action:

User route: POST /api/v3/plugin/file/repository_version/ add_content=[…]
Routes to: POST /api/v3/tasks plugin=file task=create_repository_version
add_content=[…]

User route: POST /api/v3/plugin/file/syncs/ repo=repo_href
importer=importer_href
Routes to: POST /api/v3/tasks plugin=file task=sync  repo=repo_href
importer=importer_href

We’d give a way for plugins to specify which URLs map to a task and what
the expected parameters are.

Benefits:
- Avoids all potential conflicts as long as plugin names are unique. For
example, if my plugin has multiple content types and one is ‘file’, I can
set up “/api/v3/content/file” which will conflict with pulp_file.
- No leaky abstraction in our URLs. Users don’t have to know whether a
particular URL produces a task or not to find its URL.
- It’s easy to find all the plugin routes (not just tasks routes) for a
particular plugin. They’re at “/api/v3/plugin//“.
- Plugins have more freedom and aren’t constrained to defining certain
predefined routes.


David

On Wed, Mar 14, 2018 at 3:58 PM, Jeremy Audet  wrote:

> I reviewed this proposal with Austin, mainly from the perspective of an
> end user. In my opinion, this proposal is a good one. I find the semantics
> intuitive and I think they do a good job of adhering to the Richardson
> Maturity Model
> . And
> although I don't have a great knowledge of how plugins work, I think this
> will grant more flexibility to plugins.
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Jeremy Audet]

I reviewed this proposal with Austin, mainly from the perspective of an end
user. In my opinion, this proposal is a good one. I find the semantics
intuitive and I think they do a good job of adhering to the Richardson
Maturity Model
. And
although I don't have a great knowledge of how plugins work, I think this
will grant more flexibility to plugins.
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev

Re: [Pulp-dev] Plugin relationship to tasks

[Austin Macdonald]

I wanted to add that I am not proposing that we should try to make this
change before the beta, but we will need to change something in order for
some plugins to work properly (probably docker, rpm, ostree, puppet,
debian). My goal here is just to begin discussing how to unblock these
plugins, and hopefully create a cleaner API and user experience.

I haven't worked out all of the implementation details, but I do have a
vague idea of how this could be done in a DRF friendly way.

On Tue, Mar 13, 2018 at 4:57 PM, Austin Macdonald  wrote:

> tldr;
> I am running with an idea that Milan brought up. I think this will create
> a simpler user experience while providing an obvious pattern to plugin
> writers. This pattern will offer greater plugin control and flexibility.
>
> Problem:
> Tasks that add/remove content to a repo (which creates Repository
> Versions) must have plugin involvement for some plugins (detailed
> explanation below).  What endpoint(s) should we use, and how can they
> involve plugins?
>
> Proposal:
> Rather than thinking of a POST that "creates RepositoryVersions" we could
> instead POST to "create Tasks".
>
> How the REST API would look:
>
> Create a docker add task (add Docker units to a repository):
> POST api/v3/tasks/docker/adds/ add_content_units=[...]
>
> Create a file sync task
> POST api/v3/tasks/file/syncs/ repository=repo_href importer=importer_href
>
> Retrieve all tasks and optionally filter:
> GET v3/tasks/?state=waiting
>
> Retrieve python sync tasks:
> GET /v3/tasks/python/syncs/
>
> Advantages:
> * Each POST that creates a task returns the created task synchronously,
> with a created_resource field that is updated after the task runs.
> * POST requests create a resource of the type that is indicated by the
> href.
> * plugin writers create endpoints for all tasks that affect how content is
> added/removed to a repo version
> * core would have the opportunity to provide simple implementations that
> could be used by the plugins. For example, pulpcore could provide a task
> for the simple case of add/remove (which would be used by pulp_file).
> * avoids verbs as action endpoints, instead using nouns as task types.
> * creates an intuitive way to view tasks based on task type.
> * Each Task has specific parameters that are auto-documented clearly and
> unambiguously
> * Continues pattern of "plugin writer in control, using pulpcore where it
> is helpful"
> * provides an obvious place for plugins to create custom endpoints
> * (speculative) We could clone tasks to repeat previous actions. This
> would make it easier to perform syncs on a particular repo with a
> particular importer with particular sync options.
>
> Disadvantages:
> * Plugin writers have to create more endpoints
> * endpoints might not be consistent between plugins
> * different from Pulp 2
>
>
> Detailed Problem Statement:
> Currently we have a pulpcore endpoint(add/remove) and plugin defined
> endpoints(sync, fancy-add, etc) that can create repository versions. All of
> these endpoints return a 202 and serialize the Task that was created.
>
> add/remove: POST to v3/repositories//versions/
> Since this is a pulpcore endpoint, it does not involve the plugin, which
> causes trouble for plugins in which content units relate to other content
> units. For example, it is incorrect to add a Docker Manifest if we don't
> also add the Blobs that are referenced by the Manifest.
>
> sync, etc: v3/importers///sync/
> This allows plugins to completely control the whole process, using
> pulpcore tools where they are helpful. This is advantageous because plugins
> can specify (including documentation) whatever POST body parameters they
> need for the sync action. However, it isn't optimal to have "action"
> endpoints like sync. Also, this doesn't provide a good place for plugins to
> define endpoints for "rich-copy" that don't use importers.
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev