Have you tried using conda envs? Conda avoids duplication by unpacking all packages to a shared directory and then hard-linking them into environments.
I'm not saying that should preclude doing something similar for virtualenvs, but it's probably worth looking at existing solutions before embarking on a change of this magnitude. On Mon, Oct 8, 2018, at 8:42 AM, RonnyPfannschmidt wrote: > hi everyone, > > i just did a little spring cleaning, and while i cleaned up my > ~/Projects folder - the vast amount of that was virtualenvs with roughly > the same packages installed - each about 500 mb for like 2-5 mb of code > > this seems excessively wasteful - my impression is that ~98% of the > files seem to be duplicate. > > since the impending eol of python2 and the multi python version > capabilities of python3 compiled code caches, i'd like to create a > proposal for a shared folder structure > > > $LIB/python/shared-packages/ should be a root folder in which unpacked > wheels would reside > > at a initial iteration this solution would suffer from the expanding > python path issue since each package would have a toplevel sys.path entry. > > on order to later on speed this up a simple api can be provided that > makes python aware of the metatadata locations and the map > toplevel/namespace names directly to import locations > > > for example like > > shared-packages.pth > > import sys;sys.path.insert(0, > "$LIB/python/shared-packages/shared_packages_support-1.0-py3.whl/"); > import > shared_packages_support;sys.path.pop(0);shared_packages_support.enable() > > shared_packages.json > > { > > "pytest": "pytest-3.8.8-py2.py3.whl"; > > } > > > in the aftermath, when shared packages support and fast import/metadata > for shared packages where implemented, the sizes of most virtualenvs > would shrink significantly > > > -- Ronny > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-le...@python.org > https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ > Message archived at > https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/JWX7VFJRLWVIIBP4NZ5MEPRVCYTV4ZJT/ -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-le...@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/XAZPBTKGEWVDLE3PJITY4JQ3YGK3NL7G/