Re: [Distutils] devpi-server/web-3.0: generalized mirroring, speed, new backends
On Feb 12, 2016 5:14 AM, "holger krekel" wrote:
>
>
> The 3.0 releases of devpi-server and devpi-web, the python packaging and
> work flow system for handling release files, documentation, testing and
staging,
> bring several major improvements:
>
> - Due to popular demand we now support generalized mirroring, i.e. you can
> create mirror indexes which proxy and cache release files from other
pypi
> servers. Even if the mirror goes down, pip-installing will continue to
work
> with your devpi-server instance. Previously we only supported mirroring
> of pypi.python.org. Using it is simple:
> http://doc.devpi.net/3.0/userman/devpi_indices.html#mirror-index
Thanks!
>
> - For our enterprise clients we majorly worked on improving the speed
> of serving simple pages which is now several times faster with
> private indexes. We now also support multiple worker processes
> both on master and replica sites.
> http://doc.devpi.net/3.0/adminman/server.html#multiple-server-instances
>
> - For our enterprise clients we also introduced a new backend
> architecture which allows to store server state in sqlite or
> postgres (which is supported through a separately released plugin).
> The default remains to use the "sqlite" backend and store files
> on the filesystem. See
> http://doc.devpi.net/3.0/adminman/server.html#storage-backend-selection
>
> - we started a new "admin" manual for devpi-server which describes
> features relating to server configuration, replication and security
> aspects. It's a bit work-in-progress but should already be helpful.
> http://doc.devpi.net/3.0/adminman/
>
> - A few option names changed and we also released devpi-client-2.5
> where we took great care to keep it forward and backward compatible
> so it should run against devpi-server-2.1 and upwards all the way
> to 3.0.
>
> - The "3.0" major release number increase means that you will need to run
> through an export/import cycle to upgrade your devpi-2.X installation.
>
> For more details, see the changelog and the referenced documentation
> with the main entry point here:
>
> http://doc.devpi.net
>
> Many thanks to my partner Florian Schulze and to the several companies
> who funded parts of the work on 3.0. We are especially grateful for
> their support to not only cover their own direct issues but also support
> community driven demands. I'd also like to express my gratitude to
> Rackspace and Jesse Noller who provide VMs for our open source work and
> which help a lot with the testing of our releases.
>
> We are open towards entering more support contracts to make sure you get
> what you need out of devpi, tox and pytest which together provide a
> mature tool chain for professional python development. And speaking of
> showing support, if you or your company is interested to donate to or
> attend the largest python testing sprint in history with a particular
> focus to pytest or tox, please see
>
> https://www.indiegogo.com/projects/python-testing-sprint-mid-2016/
>
> have fun,
>
> holger krekel, http://merlinux.eu
>
>
>
> server-3.0.0 (2016-02-12)
> -
>
> - dropped support for python2.6
>
> - block most ascii symbols for user and index names except ``-.@_``.
> unicode characters are fine.
>
> - add ``--no-root-pypi`` option which prevents the creation of the
> ``root/pypi`` mirror instance on first startup.
>
> - added optional ``title`` and ``description`` options to users and
indexes.
>
> - new indexes have no bases by default anymore. If you want to be able to
> install pypi packages, then you have to explicitly add ``root/pypi`` to
> the ``bases`` option of your index.
>
> - added optional ``custom_data`` option to users.
>
> - generalized mirroring to allow adding mirror indexes other than only
PyPI
>
> - renamed ``pypi_whitelist`` to ``mirror_whitelist``
>
> - speed up simple-page serving for private indexes. A private index
> with 200 release files should now be some 5 times faster.
>
> - internally use normalized project names everywhere, simplifying
> code and slightly speeding up some operations.
>
> - change {name} in route_urls to {project} to disambiguate.
> This is potentially incompatible for plugins which have registered
> on existing route_urls.
>
> - use "project" variable naming consistently in APIs
>
> - drop calling of devpi_pypi_initial hook in favor of
> the new "devpi_mirror_initialnames(stage, projectnames)" hook
> which is called when a mirror is initialized.
>
> - introduce new "devpiserver_stage_created(stage)" hook which is
> called for each index which is created.
>
> - simplify and unify internal mirroring code some more
> with "normal" stage handling.
>
> - don't persist the list of mirrored project names anymore
> but rely on a per-process RAM cache and the fact
> that neither the UI nor pip/easy_install typically
> need the projectnames list, anyway.
>
> - introduce new "devpiserver_storage_
Re: [Distutils] devpi-server/web-3.0: generalized mirroring, speed, new backends
Holger, great work! I wanted to send kudos your way and did it at the
indiegogo campaign, take pictures and have fun!
I wanted to ask if there was an xmlrpc api like pypi where we have
package_releases along with other similar calls? I use those sometimes
for helper scripts and dashboards and would like to have that in the
devpi instance too, tho I have not done my homework on checking code
yet I have not found it in the docs.
Anyways, awesome work, and keep rocking!
On Fri, Feb 12, 2016 at 3:13 AM, holger krekel wrote:
>
> The 3.0 releases of devpi-server and devpi-web, the python packaging and
> work flow system for handling release files, documentation, testing and
> staging,
> bring several major improvements:
>
> - Due to popular demand we now support generalized mirroring, i.e. you can
> create mirror indexes which proxy and cache release files from other pypi
> servers. Even if the mirror goes down, pip-installing will continue to work
> with your devpi-server instance. Previously we only supported mirroring
> of pypi.python.org. Using it is simple:
> http://doc.devpi.net/3.0/userman/devpi_indices.html#mirror-index
>
> - For our enterprise clients we majorly worked on improving the speed
> of serving simple pages which is now several times faster with
> private indexes. We now also support multiple worker processes
> both on master and replica sites.
> http://doc.devpi.net/3.0/adminman/server.html#multiple-server-instances
>
> - For our enterprise clients we also introduced a new backend
> architecture which allows to store server state in sqlite or
> postgres (which is supported through a separately released plugin).
> The default remains to use the "sqlite" backend and store files
> on the filesystem. See
> http://doc.devpi.net/3.0/adminman/server.html#storage-backend-selection
>
> - we started a new "admin" manual for devpi-server which describes
> features relating to server configuration, replication and security
> aspects. It's a bit work-in-progress but should already be helpful.
> http://doc.devpi.net/3.0/adminman/
>
> - A few option names changed and we also released devpi-client-2.5
> where we took great care to keep it forward and backward compatible
> so it should run against devpi-server-2.1 and upwards all the way
> to 3.0.
>
> - The "3.0" major release number increase means that you will need to run
> through an export/import cycle to upgrade your devpi-2.X installation.
>
> For more details, see the changelog and the referenced documentation
> with the main entry point here:
>
> http://doc.devpi.net
>
> Many thanks to my partner Florian Schulze and to the several companies
> who funded parts of the work on 3.0. We are especially grateful for
> their support to not only cover their own direct issues but also support
> community driven demands. I'd also like to express my gratitude to
> Rackspace and Jesse Noller who provide VMs for our open source work and
> which help a lot with the testing of our releases.
>
> We are open towards entering more support contracts to make sure you get
> what you need out of devpi, tox and pytest which together provide a
> mature tool chain for professional python development. And speaking of
> showing support, if you or your company is interested to donate to or
> attend the largest python testing sprint in history with a particular
> focus to pytest or tox, please see
>
> https://www.indiegogo.com/projects/python-testing-sprint-mid-2016/
>
> have fun,
>
> holger krekel, http://merlinux.eu
>
>
>
> server-3.0.0 (2016-02-12)
> -
>
> - dropped support for python2.6
>
> - block most ascii symbols for user and index names except ``-.@_``.
> unicode characters are fine.
>
> - add ``--no-root-pypi`` option which prevents the creation of the
> ``root/pypi`` mirror instance on first startup.
>
> - added optional ``title`` and ``description`` options to users and indexes.
>
> - new indexes have no bases by default anymore. If you want to be able to
> install pypi packages, then you have to explicitly add ``root/pypi`` to
> the ``bases`` option of your index.
>
> - added optional ``custom_data`` option to users.
>
> - generalized mirroring to allow adding mirror indexes other than only PyPI
>
> - renamed ``pypi_whitelist`` to ``mirror_whitelist``
>
> - speed up simple-page serving for private indexes. A private index
> with 200 release files should now be some 5 times faster.
>
> - internally use normalized project names everywhere, simplifying
> code and slightly speeding up some operations.
>
> - change {name} in route_urls to {project} to disambiguate.
> This is potentially incompatible for plugins which have registered
> on existing route_urls.
>
> - use "project" variable naming consistently in APIs
>
> - drop calling of devpi_pypi_initial hook in favor of
> the new "devpi_mirror_initialnames(stage, projectnames)" hook
> which is called when a mirror is
Re: [Distutils] deprecating pip install --target
My setup-requires wrapper, adding pre-setup.py dependency installation to setup.py, relies on this feature. It needs to install something in a directory that is only added to PYTHONPATH during the installation and does not interfere with the normal environment. On Fri, Feb 12, 2016 at 9:55 AM Steve Dower wrote: > I was also planning to use it in an upcoming project that has to "do its > own" package management. The aim was to install different versions of > packages in different directories and use sys.path modifications to resolve > them at runtime (kind of like what setuptools did in the older days). > > An alternative would be great, though I can probably fake things somehow > for my purposes. > > Cheers, > Steve > > Top-posted from my Windows Phone > -- > From: Vinay Sajip > Sent: 2/11/2016 15:18 > To: Distutils-Sig@Python.Org > Subject: Re: [Distutils] deprecating pip install --target > > Robert Collins robertcollins.net> writes: > > > > > This is fairly broken - it doesn't handle platlib vs purelib (see pip > > PR 3450), doesn't handle data files, or any other layout. Donald says > > pip uses it to maintain the _vendor subtrees only, which doesn't seem > > like a particularly strong use case. > > > > Certainly the help description for it is misleading - since what we're > > actually doing is copying only a subset of what the package installed > > - so at a minimum we need to be much clearer about what it does. > > > > But, I think it would be better to deprecate it and remove it... so > > I'm pinging here to see if anyone can explain a sensible use case for > > it in the context of pip :) > > I use it in pretty much the same way as Paul mentioned - I wouldn't like > it > to go unless something equivalent is available. Updating the help / > documentation for it to better reflect what it does would be > uncontroversial > for me, but I see no strong reason for deprecation and removal. As Paul > suggests, it can get stricter about what it'll handle. > > Regards, > > Vinay Sajip > > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ > Distutils-SIG maillist - Distutils-SIG@python.org > https://mail.python.org/mailman/listinfo/distutils-sig > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] deprecating pip install --target
I was also planning to use it in an upcoming project that has to "do its own" package management. The aim was to install different versions of packages in different directories and use sys.path modifications to resolve them at runtime (kind of like what setuptools did in the older days). An alternative would be great, though I can probably fake things somehow for my purposes. Cheers, Steve Top-posted from my Windows Phone -Original Message- From: "Vinay Sajip" Sent: 2/11/2016 15:18 To: "Distutils-Sig@Python.Org" Subject: Re: [Distutils] deprecating pip install --target Robert Collins robertcollins.net> writes: > > This is fairly broken - it doesn't handle platlib vs purelib (see pip > PR 3450), doesn't handle data files, or any other layout. Donald says > pip uses it to maintain the _vendor subtrees only, which doesn't seem > like a particularly strong use case. > > Certainly the help description for it is misleading - since what we're > actually doing is copying only a subset of what the package installed > - so at a minimum we need to be much clearer about what it does. > > But, I think it would be better to deprecate it and remove it... so > I'm pinging here to see if anyone can explain a sensible use case for > it in the context of pip :) I use it in pretty much the same way as Paul mentioned - I wouldn't like it to go unless something equivalent is available. Updating the help / documentation for it to better reflect what it does would be uncontroversial for me, but I see no strong reason for deprecation and removal. As Paul suggests, it can get stricter about what it'll handle. Regards, Vinay Sajip ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc
On 12 February 2016 at 01:19, Barry Warsaw wrote: > On Feb 11, 2016, at 01:58 PM, Nick Coghlan wrote: >>Anyway, the core point is wanting to ensure we can automate not only >>"direct to binary" installation with Python specific tools, but also >>the "convert to alternate source archive format and build from there" >>workflows needed by redistributor ecosystems like Linux distros, >>conda, Canopy, PyPM, Nix, etc. > > Cool. > > Note that not even all Debian-based distros are equal here. For example, in > Debian, especially for architecture independent (i.e. pure-Python) packages, > the source package to binary package step happens on the maintainer's local > system, while in Ubuntu we upload the source package and let the centrally > maintained build daemons produce the resulting binary packages. Yeah, Fedora is closer to the Ubuntu set up, with a centralised git backed build service (Koji), and tools for submitting build requests to that: * https://fedoraproject.org/wiki/Package_maintenance_guide * https://fedoraproject.org/wiki/Package_update_HOWTO In my ideal world, for previously reviewed packages, the path from PyPI release, through the Anitya release monitoring service and the Taskotron CI service, to updating Fedora Rawhide would be, if not fully automated, at least not much more work than clicking an "Approve" button after getting a green light from Taskotron and reading the upstream release notes. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] devpi-server/web-3.0: generalized mirroring, speed, new backends
The 3.0 releases of devpi-server and devpi-web, the python packaging and
work flow system for handling release files, documentation, testing and staging,
bring several major improvements:
- Due to popular demand we now support generalized mirroring, i.e. you can
create mirror indexes which proxy and cache release files from other pypi
servers. Even if the mirror goes down, pip-installing will continue to work
with your devpi-server instance. Previously we only supported mirroring
of pypi.python.org. Using it is simple:
http://doc.devpi.net/3.0/userman/devpi_indices.html#mirror-index
- For our enterprise clients we majorly worked on improving the speed
of serving simple pages which is now several times faster with
private indexes. We now also support multiple worker processes
both on master and replica sites.
http://doc.devpi.net/3.0/adminman/server.html#multiple-server-instances
- For our enterprise clients we also introduced a new backend
architecture which allows to store server state in sqlite or
postgres (which is supported through a separately released plugin).
The default remains to use the "sqlite" backend and store files
on the filesystem. See
http://doc.devpi.net/3.0/adminman/server.html#storage-backend-selection
- we started a new "admin" manual for devpi-server which describes
features relating to server configuration, replication and security
aspects. It's a bit work-in-progress but should already be helpful.
http://doc.devpi.net/3.0/adminman/
- A few option names changed and we also released devpi-client-2.5
where we took great care to keep it forward and backward compatible
so it should run against devpi-server-2.1 and upwards all the way
to 3.0.
- The "3.0" major release number increase means that you will need to run
through an export/import cycle to upgrade your devpi-2.X installation.
For more details, see the changelog and the referenced documentation
with the main entry point here:
http://doc.devpi.net
Many thanks to my partner Florian Schulze and to the several companies
who funded parts of the work on 3.0. We are especially grateful for
their support to not only cover their own direct issues but also support
community driven demands. I'd also like to express my gratitude to
Rackspace and Jesse Noller who provide VMs for our open source work and
which help a lot with the testing of our releases.
We are open towards entering more support contracts to make sure you get
what you need out of devpi, tox and pytest which together provide a
mature tool chain for professional python development. And speaking of
showing support, if you or your company is interested to donate to or
attend the largest python testing sprint in history with a particular
focus to pytest or tox, please see
https://www.indiegogo.com/projects/python-testing-sprint-mid-2016/
have fun,
holger krekel, http://merlinux.eu
server-3.0.0 (2016-02-12)
-
- dropped support for python2.6
- block most ascii symbols for user and index names except ``-.@_``.
unicode characters are fine.
- add ``--no-root-pypi`` option which prevents the creation of the
``root/pypi`` mirror instance on first startup.
- added optional ``title`` and ``description`` options to users and indexes.
- new indexes have no bases by default anymore. If you want to be able to
install pypi packages, then you have to explicitly add ``root/pypi`` to
the ``bases`` option of your index.
- added optional ``custom_data`` option to users.
- generalized mirroring to allow adding mirror indexes other than only PyPI
- renamed ``pypi_whitelist`` to ``mirror_whitelist``
- speed up simple-page serving for private indexes. A private index
with 200 release files should now be some 5 times faster.
- internally use normalized project names everywhere, simplifying
code and slightly speeding up some operations.
- change {name} in route_urls to {project} to disambiguate.
This is potentially incompatible for plugins which have registered
on existing route_urls.
- use "project" variable naming consistently in APIs
- drop calling of devpi_pypi_initial hook in favor of
the new "devpi_mirror_initialnames(stage, projectnames)" hook
which is called when a mirror is initialized.
- introduce new "devpiserver_stage_created(stage)" hook which is
called for each index which is created.
- simplify and unify internal mirroring code some more
with "normal" stage handling.
- don't persist the list of mirrored project names anymore
but rely on a per-process RAM cache and the fact
that neither the UI nor pip/easy_install typically
need the projectnames list, anyway.
- introduce new "devpiserver_storage_backend" hook which allows plugins to
provide custom storage backends. When there is more than one backend
available, the "--storage" option becomes required for startup.
- introduce new "--requests-only" option to start devpi-server in
"worker"
