On 08/24/2017 09:51 AM, Brian Bouterse wrote: > The next step in considering the asyncio downloaders is done and ready to be > looked at. The PR made [0] > replaces the existing downloaders and is "merge-ready". The best way to read > about them is through their docs > [1]. Here are some of its highlights: > > # Code size and docs > * huge code savings by switching to asyncio. removes 2427 lines and adds 558 > lines.
Replacing concurrent.futures with asyncio would remove a fair amount of code in
batch.py but the remainder of
lines of code removed discarded features. Each can be correlated to use cases
[2].
- Custom error recovery.
Supported recoverable error recovery.
Supported unavailable mirrors.
- Event notifications.
Supported streamer forwarding of HTTP headers.
Supported ChangeSet knowing when an artifact was actually downloaded.
Supported calculating digests and size needed ONLY for creating Artifacts.
Supported (future) progress reporting when desired.
Supported (future) mirror lists.
- Custom Writers - handling of downloaded bits.
Supported writing files to disk.
Supported the streamer streaming downloaded bit stream in the Twisted
response instead of writing
the file to disk.
- File URLs.
supported: "file:///myfile"
- Custom validations without having to subclass the downloader.
Included: digest and size validations but plugin writer could implement
additional validations.
- Standardized settings for SSL and Auth.
Supported consistent settings across downloader types.
Supported validation of settings.
- Shared Contexts - collaborative sharing of resources between downloaders.
Supported shared: sessions (connection pools), auth tokens and resolved
mirror lists.
Supported the sharing of resources without requiring the user of the
downloader to know
which resources are shared or how they are shared. Resource sharing needs
to happen in the
streamer too.
- Support for HEAD requests (HttpDownload).
Just fetches the HTTP headers.
- Standard exceptions.
Supported catching DownloadError, NotFound and NotAuthorized.
Encapsulated client lib specific exceptions and error behaviors.
- Downloader Attachments - attach any user defined object to the downloader.
Supported easy correlation between a downloader and an artifact being
downloaded.
> * ^ is surprising considering the majority of lines added are docs
> * Read the compiled docs here [1]
>
> # Features
> * It has 100% feature parity with the existing http/https downloaders.
See gaps above.
Are authenticated proxies supported?
Ftp can be added, but purposely left
> out for now. The requirements being fulfilled are written here [2].
> * It has more features than the existing downloaders including additional ssl
> config, proxy_auth, additional
> connection options, and more.
Can you enumerate them?
>
> # Easier for plugin writers
> * An easier download customization experience by having to subclass fewer
> objects than the existing
> downloaders.
Can you be specific?
> This design has one downloader for both synchronous or asynchronous
> downloading
But they need to execute the downloader in an Asynio loop for both synchronous
and asynchronous, correct?
> * Makes some functionality contained only in the changeset available in the
> downloaders directly. Specifically
> the GroupDownloader was functionality you could only get via changeset usage.
> Changesets could easily be
> retooled to use this instead which would be great.
> * Each object can be used independently or in conjunction with others
>
> # Demo
> * works with the HEAD of master of pulp_example [3] (not pulp_file) which
> shows that it actually runs
>
> [0]: https://github.com/pulp/pulp/pull/3129
> [1]: http://file.rdu.redhat.com/~bbouters/plugins/plugin-api/download.html
> [2]: https://pulp.plan.io/issues/2951#note-17
> [3]: https://github.com/dkliban/pulp_example/
>
> -Brian
>
>
> _______________________________________________
> Pulp-dev mailing list
> Pulp-dev@redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-dev
>
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Pulp-dev mailing list Pulp-dev@redhat.com https://www.redhat.com/mailman/listinfo/pulp-dev
