Re: [Pulp-dev] Basic Lazy Streaming for Pulp3 Ready for Rough Testing
I'm glad you wrote this; this is definitely a gap now that we're doing
content protection with 3.0.
I filed a story to add content protection [0] and that is part of the epic
[1]. In terms of how to add it, there are a few ways (at least two). I
wrote them into the issue description [0] and I need feedback on how to
resolve it. I wrote my recommendation on the issue here [2].
What do you think we should do? Comments, feedback, and ideas are welcome,
please post here [0]. Thanks!
[0]: https://pulp.plan.io/issues/4181
[1]: https://pulp.plan.io/issues/3693
[2]: https://pulp.plan.io/issues/4181#note-3
On Mon, Nov 26, 2018 at 10:27 AM Jeff Ortel wrote:
> The initial planning for lazy omitted content protection for 3.0. Since
> then, we have pulled content protection back into 3.0 re: content-guards.
> In pulp2, the content app redirected using a signed-url so that clients
> could not circumvent content protection. Currently in 3.0, there is
> nothing to keep clients from circumventing content protection by going
> directly to the streamer. Isn't this a gap?
>
>
> On 11/20/18 3:51 PM, Brian Bouterse wrote:
>
> I've been developing the streamer functionality, and it's correctly
> working (in my testing) as driven from the Remote.policy attribute. It
> correctly works with 'immediate', 'on_demand', and 'cache_only'. Read more
> about the expected behaviors in the epic [0].
>
> # Try it out!
> Here is the core commit needed: https://github.com/pulp/pulp/pull/3738
> Here is the streamer you should pip install from master:
> https://github.com/bmbouter/pulp_streamer
> Here is what it looks like to port a plugin using DeclarativeVersion, e.g.
> pulp_file to support lazy: https://github.com/pulp/pulp_file/pull/132
>
> You'll need to configure Pulp's webserver for streaming. I did this by
> exporting an environment var to dynaconf in the same bash environment as my
> django run server. Specifically I configured Pulp to redirect to port
> localhost:8080/streamer/ with this command:
>
> export PULP_CONTENT='@json {"HOST": null, "WEB_SERVER": "django",
> "REDIRECT": {"ENABLED": true, "PORT": 8080, "HOST": "localhost",
> "PATH_PREFIX": "/streamer/"}}'
>
> Then I run the streamer (after pip installed) with gunicorn which you also
> need to pip install. Run it with:
>
> gunicorn pulpcore.streamer:server --bind localhost:8080 --worker-class
> aiohttp.GunicornWebWorker -w 2
>
> Then sync a pulp_file repo with policy='on_demand' or policy='cache_only'
> and see how Pulp behaves.
>
> Feedback, ideas, concerns are welcome in any form. Note this is still
> rough, and the following are known things to be done:
>
> * fix tests to get Travis passing
> * docs for the streamer and for pulpcore
> * an installer role to install the streamer
> * integration with squid to cache lots of data at the streamer
> * transfer the pulp_streamer to the Pulp org on github
> * publish an initial release to PyPI for users to use it
> * write a blog post about porting to it and using it
> * make a demo
>
> [0]: https://pulp.plan.io/issues/3693
>
> Thanks!
> Brian
>
> ___
> Pulp-dev mailing
> listPulp-dev@redhat.comhttps://www.redhat.com/mailman/listinfo/pulp-dev
>
>
> ___
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev
Re: [Pulp-dev] Basic Lazy Streaming for Pulp3 Ready for Rough Testing
The initial planning for lazy omitted content protection for 3.0. Since
then, we have pulled content protection back into 3.0 re:
content-guards. In pulp2, the content app redirected using a signed-url
so that clients could not circumvent content protection. Currently in
3.0, there is nothing to keep clients from circumventing content
protection by going directly to the streamer. Isn't this a gap?
On 11/20/18 3:51 PM, Brian Bouterse wrote:
I've been developing the streamer functionality, and it's correctly
working (in my testing) as driven from the Remote.policy attribute. It
correctly works with 'immediate', 'on_demand', and 'cache_only'. Read
more about the expected behaviors in the epic [0].
# Try it out!
Here is the core commit needed: https://github.com/pulp/pulp/pull/3738
Here is the streamer you should pip install from master:
https://github.com/bmbouter/pulp_streamer
Here is what it looks like to port a plugin using DeclarativeVersion,
e.g. pulp_file to support lazy: https://github.com/pulp/pulp_file/pull/132
You'll need to configure Pulp's webserver for streaming. I did this by
exporting an environment var to dynaconf in the same bash environment
as my django run server. Specifically I configured Pulp to redirect to
port localhost:8080/streamer/ with this command:
export PULP_CONTENT='@json {"HOST": null, "WEB_SERVER": "django",
"REDIRECT": {"ENABLED": true, "PORT": 8080, "HOST": "localhost",
"PATH_PREFIX": "/streamer/"}}'
Then I run the streamer (after pip installed) with gunicorn which you
also need to pip install. Run it with:
gunicorn pulpcore.streamer:server --bind localhost:8080 --worker-class
aiohttp.GunicornWebWorker -w 2
Then sync a pulp_file repo with policy='on_demand' or
policy='cache_only' and see how Pulp behaves.
Feedback, ideas, concerns are welcome in any form. Note this is still
rough, and the following are known things to be done:
* fix tests to get Travis passing
* docs for the streamer and for pulpcore
* an installer role to install the streamer
* integration with squid to cache lots of data at the streamer
* transfer the pulp_streamer to the Pulp org on github
* publish an initial release to PyPI for users to use it
* write a blog post about porting to it and using it
* make a demo
[0]: https://pulp.plan.io/issues/3693
Thanks!
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev
[Pulp-dev] Basic Lazy Streaming for Pulp3 Ready for Rough Testing
I've been developing the streamer functionality, and it's correctly working
(in my testing) as driven from the Remote.policy attribute. It correctly
works with 'immediate', 'on_demand', and 'cache_only'. Read more about the
expected behaviors in the epic [0].
# Try it out!
Here is the core commit needed: https://github.com/pulp/pulp/pull/3738
Here is the streamer you should pip install from master:
https://github.com/bmbouter/pulp_streamer
Here is what it looks like to port a plugin using DeclarativeVersion, e.g.
pulp_file to support lazy: https://github.com/pulp/pulp_file/pull/132
You'll need to configure Pulp's webserver for streaming. I did this by
exporting an environment var to dynaconf in the same bash environment as my
django run server. Specifically I configured Pulp to redirect to port
localhost:8080/streamer/ with this command:
export PULP_CONTENT='@json {"HOST": null, "WEB_SERVER": "django",
"REDIRECT": {"ENABLED": true, "PORT": 8080, "HOST": "localhost",
"PATH_PREFIX": "/streamer/"}}'
Then I run the streamer (after pip installed) with gunicorn which you also
need to pip install. Run it with:
gunicorn pulpcore.streamer:server --bind localhost:8080 --worker-class
aiohttp.GunicornWebWorker -w 2
Then sync a pulp_file repo with policy='on_demand' or policy='cache_only'
and see how Pulp behaves.
Feedback, ideas, concerns are welcome in any form. Note this is still
rough, and the following are known things to be done:
* fix tests to get Travis passing
* docs for the streamer and for pulpcore
* an installer role to install the streamer
* integration with squid to cache lots of data at the streamer
* transfer the pulp_streamer to the Pulp org on github
* publish an initial release to PyPI for users to use it
* write a blog post about porting to it and using it
* make a demo
[0]: https://pulp.plan.io/issues/3693
Thanks!
Brian
___
Pulp-dev mailing list
Pulp-dev@redhat.com
https://www.redhat.com/mailman/listinfo/pulp-dev
