Re: [Distutils] Distributable binary with dependencies
On Fri, Aug 23, 2013 at 6:45 AM, bharath ravi kumar wrote: > I'm looking to package an application with all its dependencies for > deployment on multiple hosts. I'd like to ensure that there is no > compilation or setup step before starting the application in production. An > nice to have ability would be to isolate base library dependencies per > application (like virtualenv does). Ideally, the development -> deployment > lifecycle would involve: (a) Build an application archive with all its > dependencies baked in (b) Copy archive to a host in production. (c) Unwrap > archive (d) Start services. (Note that the build host & production hosts are > identical in architecture, OS patch level and python version). You can use "easy_install -Zmad deployment_dir application", then archive deployment_dir and extract it on the target machines. (Note: "application" must be a setuptools-packaged project with its dependencies declared, for easy_install to know what to build and deploy.) The "Z" option means "unzip eggs", "m" means "don't worry about the target being on sys.path; we're not trying to install a default version", "a" means "copy all dependencies, even if locally installed already", and "d" means "install libraries and scripts to the following directory". So, the scripts will be put inside deployment_dir with a bunch of adjacent subdirectories containing all the compiled and ready-to-use libraries. The resulting directory is a portable installation of "application": as long as the entire subdirectory is copied to the target machines, everything should work just fine. None of the dependencies or the app itself will interfere with other Python code installed on the target system; it is in a sense a minimal virtualenv which will run whatever scripts that easy_install puts in that directory. One note: the target machines *will* need pkg_resources installed, and it will not be included in the directory by default. If they don't have local copies installed (due to e.g. setuptools, distribute, etc. being installed), you can manually copy a pkg_resources.py to the deployment directory, and it will be used by whatever scripts are in that directory. While there may be other tools available that support this kind of thing, I don't think any of them can do it quite this simply. This deployment scenario was actually a key use case for the original design of easy_install and eggs, so it actually works pretty decently for this. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distributable binary with dependencies
Hi Bharath, On 08/23/2013 04:45 AM, bharath ravi kumar wrote: > I'm looking to package an application with all its dependencies for > deployment on multiple hosts. I'd like to ensure that there is no > compilation or setup step before starting the application in production. > An nice to have ability would be to isolate base library dependencies > per application (like virtualenv does). Ideally, the development -> > deployment lifecycle would involve: (a) Build an application archive > with all its dependencies baked in (b) Copy archive to a host in > production. (c) Unwrap archive (d) Start services. (Note that the build > host & production hosts are identical in architecture, OS patch level > and python version). Some options if you want zero installation steps on production hosts: 1) Vendor dependencies' Python code directly into your application. You can use pip to automate this based on requirements files with a script like https://github.com/mozilla/moztrap/blob/master/bin/generate-vendor-lib - for portability this is normally only possible with pure-Python dependencies, but in your case if the app will only ever run on identical servers you could do it for compiled (C-extension) dependencies as well. One downside is that it generally requires some kind of sys.path hacking in your application environment or startup to make the vendor-library importable. 2) Install things into virtualenvs using pip on the build host (possibly using pre-built wheels to speed that up), and then copy the entire virtualenv to the production host. If build and production hosts are identical in every way (including the path of the virtualenv), this will Just Work. There are also tools like https://github.com/PolicyStat/terrarium that aim to smooth this workflow. 3) Generate OS packages (debs or rpms) containing your complete app and installed dependencies. I'd recommend this for the smoothest experience once you have some tooling in place, but it will probably take some time initially to develop that tooling. You might find Hynek Schlawack's blog post on the topic helpful: http://hynek.me/articles/python-app-deployment-with-native-packages/ Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Distributable binary with dependencies
Hi, I'm looking to package an application with all its dependencies for deployment on multiple hosts. I'd like to ensure that there is no compilation or setup step before starting the application in production. An nice to have ability would be to isolate base library dependencies per application (like virtualenv does). Ideally, the development -> deployment lifecycle would involve: (a) Build an application archive with all its dependencies baked in (b) Copy archive to a host in production. (c) Unwrap archive (d) Start services. (Note that the build host & production hosts are identical in architecture, OS patch level and python version). Having looked at various tools (e.g. distutils, setup tools, pip + virtualenv, egg, wheel, pyinstaller, etc.) available to address specific aspects/stages of the development lifecycle, I'm undecided on the tool/methodology to adopt in order to comprehensively solve the above problem. Any recommendation in this regard would be very helpful. Thanks for for your time. -Bharath ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] buildout bootstrap.py doesn't work on Sabayon Linux with system python
Marius Gedminas writes: > > Basically, yes. At least it's what I've come to expect. > > Here's my fool-proof method of setting up buildouts on the brave new > post-setuptools-0.7 world: > > virtualenv python > python/bin/pip install -U setuptools > python/bin/python bootstrap.py > bin/buildout > Another fool-proof method is setting up a virtualenv with the --no-setuptools --no-pip flags. This gives you a clean python environment to work with buildout and you only need to do this once. If you create that virtualenv as root, you're also protected from accidentally installing python packages. You'll need virtualenv 1.10 or above in order to use --no-setuptools --no-pip. -- Cheers Ralf ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] What does it mean for Python to "bundle pip"?
On 23 August 2013 16:31, Ronald Oussoren wrote: > On 23 Aug, 2013, at 5:17, Nick Coghlan wrote: >> That said, I'm considering the idea of adding a "variant" field to the >> compatibility tags for wheel 1.1, along the lines of what Oscar >> Benjamin suggested earlier. By default, installers would only find >> wheels without a variant defined, but users could opt in to looking >> for particular variants. The meaning of the variants field would be >> entirely distribution specific. Numpy, for example, could publish: >> >> numpy-1.7.1-cp27-cp22m-win32.whl >> numpy-1.7.1-cp27-cp22m-win32-sse.whl >> numpy-1.7.1-cp27-cp22m-win32-sse2.whl >> numpy-1.7.1-cp27-cp22m-win32-sse3.whl >> >> The only restrictions on the variant tags would be: >> 1. must be ASCII >> 2. must not contain '.' or '-' characters >> >> You could even go to the extent of using hashes as variant tags. > > Is adding variants necessary? Numpy use runtime selection for picking > the most appropriate extension code (heck, AFAIK recent versions of GCC > can even compile multiple variants of functions and pick the right one > automaticly). No, I'm not sure the variant system is necessary. It is almost certainly acceptable to constrain people to offering at most one binary per platform per Python interpreter per Python version for the wheel ecosystem, and suggest they use something hash based like conda if they need finer granularity than that. If we *did* add that flexibility to wheels, though, then the variant system is how I would do it. It would just be an arbitrary labelling mechanism that allowed users to say "give me this variant rather than the default one", not anything actually automated or necessarily reflecting an underlying system capability. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
