Re: [Distutils] Distributable binary with dependencies
The hacking of sys.path required in option (1) suggested earlier is avoidable. I do not want my project to be setup-tools managed. Virtualenv + pip is hence the obvious choice. Thanks,Bharath > From: p...@telecommunity.com > Date: Fri, 30 Aug 2013 13:15:57 -0400 > Subject: Re: [Distutils] Distributable binary with dependencies > To: reachb...@outlook.com > CC: c...@oddbird.net; distutils-sig@python.org > > On Mon, Aug 26, 2013 at 2:25 PM, bharath ravi kumar > wrote: > > Carl, Eby, > > > > Thanks for taking time to suggest various alternatives. Considering that the > > deployment hosts are identical in every as[ect, the approach of moving > > virtualenv's with packages pip-installed at build time appears the simplest, > > low-overhead approach that can be implemented without hacking the > > environment or resorting to custom scripts. I'll go ahead with that option. > > What hacking the environment or custom scripts? > > I'm confused, because AFAIK there are actually more steps to > pip-install a virtualenv and copy it to different machines, than there > are involved in using easy_install to create a portable installation. > In both cases, you end up with a directory to archive and copy, so the > only difference is in the commands used to build that directory, and > the layout of the directory afterwards. > > Perhaps you misunderstood my post as meaning that you had to run > easy_install on the target system? > > (I don't have any particular stake in what you do for your own system, > but I'm curious, both for the future reference of folks reading this > thread by way of Googling this question, and in case there is > something for me to learn or that I'm mistaken about, in relation > either to pip/virtualenv or your use case. And certainly if you are > more familiar with pip+virtualenv, that would actually be sufficient > reason in this case to use it. But I'd prefer future readers of this > thread not to be under an erroneous impression that easy_install > involves more steps, scripts, or environment changes in order to > implement this use case. Thanks.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distributable binary with dependencies
On Mon, Aug 26, 2013 at 2:25 PM, bharath ravi kumar wrote: > Carl, Eby, > > Thanks for taking time to suggest various alternatives. Considering that the > deployment hosts are identical in every as[ect, the approach of moving > virtualenv's with packages pip-installed at build time appears the simplest, > low-overhead approach that can be implemented without hacking the > environment or resorting to custom scripts. I'll go ahead with that option. What hacking the environment or custom scripts? I'm confused, because AFAIK there are actually more steps to pip-install a virtualenv and copy it to different machines, than there are involved in using easy_install to create a portable installation. In both cases, you end up with a directory to archive and copy, so the only difference is in the commands used to build that directory, and the layout of the directory afterwards. Perhaps you misunderstood my post as meaning that you had to run easy_install on the target system? (I don't have any particular stake in what you do for your own system, but I'm curious, both for the future reference of folks reading this thread by way of Googling this question, and in case there is something for me to learn or that I'm mistaken about, in relation either to pip/virtualenv or your use case. And certainly if you are more familiar with pip+virtualenv, that would actually be sufficient reason in this case to use it. But I'd prefer future readers of this thread not to be under an erroneous impression that easy_install involves more steps, scripts, or environment changes in order to implement this use case. Thanks.) ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distributable binary with dependencies
Carl, Eby, Thanks for taking time to suggest various alternatives. Considering that the deployment hosts are identical in every as[ect, the approach of moving virtualenv's with packages pip-installed at build time appears the simplest, low-overhead approach that can be implemented without hacking the environment or resorting to custom scripts. I'll go ahead with that option. Thanks,Bharath > From: p...@telecommunity.com > Date: Fri, 23 Aug 2013 15:38:46 -0400 > Subject: Re: [Distutils] Distributable binary with dependencies > To: reachb...@outlook.com > CC: distutils-sig@python.org > > On Fri, Aug 23, 2013 at 6:45 AM, bharath ravi kumar > wrote: > > I'm looking to package an application with all its dependencies for > > deployment on multiple hosts. I'd like to ensure that there is no > > compilation or setup step before starting the application in production. An > > nice to have ability would be to isolate base library dependencies per > > application (like virtualenv does). Ideally, the development -> deployment > > lifecycle would involve: (a) Build an application archive with all its > > dependencies baked in (b) Copy archive to a host in production. (c) Unwrap > > archive (d) Start services. (Note that the build host & production hosts are > > identical in architecture, OS patch level and python version). > > You can use "easy_install -Zmad deployment_dir application", then > archive deployment_dir and extract it on the target machines. (Note: > "application" must be a setuptools-packaged project with its > dependencies declared, for easy_install to know what to build and > deploy.) > > The "Z" option means "unzip eggs", "m" means "don't worry about the > target being on sys.path; we're not trying to install a default > version", "a" means "copy all dependencies, even if locally installed > already", and "d" means "install libraries and scripts to the > following directory". > > So, the scripts will be put inside deployment_dir with a bunch of > adjacent subdirectories containing all the compiled and ready-to-use > libraries. The resulting directory is a portable installation of > "application": as long as the entire subdirectory is copied to the > target machines, everything should work just fine. None of the > dependencies or the app itself will interfere with other Python code > installed on the target system; it is in a sense a minimal virtualenv > which will run whatever scripts that easy_install puts in that > directory. > > One note: the target machines *will* need pkg_resources installed, and > it will not be included in the directory by default. If they don't > have local copies installed (due to e.g. setuptools, distribute, etc. > being installed), you can manually copy a pkg_resources.py to the > deployment directory, and it will be used by whatever scripts are in > that directory. > > While there may be other tools available that support this kind of > thing, I don't think any of them can do it quite this simply. This > deployment scenario was actually a key use case for the original > design of easy_install and eggs, so it actually works pretty decently > for this. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distributable binary with dependencies
On Fri, Aug 23, 2013 at 6:45 AM, bharath ravi kumar wrote: > I'm looking to package an application with all its dependencies for > deployment on multiple hosts. I'd like to ensure that there is no > compilation or setup step before starting the application in production. An > nice to have ability would be to isolate base library dependencies per > application (like virtualenv does). Ideally, the development -> deployment > lifecycle would involve: (a) Build an application archive with all its > dependencies baked in (b) Copy archive to a host in production. (c) Unwrap > archive (d) Start services. (Note that the build host & production hosts are > identical in architecture, OS patch level and python version). You can use "easy_install -Zmad deployment_dir application", then archive deployment_dir and extract it on the target machines. (Note: "application" must be a setuptools-packaged project with its dependencies declared, for easy_install to know what to build and deploy.) The "Z" option means "unzip eggs", "m" means "don't worry about the target being on sys.path; we're not trying to install a default version", "a" means "copy all dependencies, even if locally installed already", and "d" means "install libraries and scripts to the following directory". So, the scripts will be put inside deployment_dir with a bunch of adjacent subdirectories containing all the compiled and ready-to-use libraries. The resulting directory is a portable installation of "application": as long as the entire subdirectory is copied to the target machines, everything should work just fine. None of the dependencies or the app itself will interfere with other Python code installed on the target system; it is in a sense a minimal virtualenv which will run whatever scripts that easy_install puts in that directory. One note: the target machines *will* need pkg_resources installed, and it will not be included in the directory by default. If they don't have local copies installed (due to e.g. setuptools, distribute, etc. being installed), you can manually copy a pkg_resources.py to the deployment directory, and it will be used by whatever scripts are in that directory. While there may be other tools available that support this kind of thing, I don't think any of them can do it quite this simply. This deployment scenario was actually a key use case for the original design of easy_install and eggs, so it actually works pretty decently for this. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distributable binary with dependencies
Hi Bharath, On 08/23/2013 04:45 AM, bharath ravi kumar wrote: > I'm looking to package an application with all its dependencies for > deployment on multiple hosts. I'd like to ensure that there is no > compilation or setup step before starting the application in production. > An nice to have ability would be to isolate base library dependencies > per application (like virtualenv does). Ideally, the development -> > deployment lifecycle would involve: (a) Build an application archive > with all its dependencies baked in (b) Copy archive to a host in > production. (c) Unwrap archive (d) Start services. (Note that the build > host & production hosts are identical in architecture, OS patch level > and python version). Some options if you want zero installation steps on production hosts: 1) Vendor dependencies' Python code directly into your application. You can use pip to automate this based on requirements files with a script like https://github.com/mozilla/moztrap/blob/master/bin/generate-vendor-lib - for portability this is normally only possible with pure-Python dependencies, but in your case if the app will only ever run on identical servers you could do it for compiled (C-extension) dependencies as well. One downside is that it generally requires some kind of sys.path hacking in your application environment or startup to make the vendor-library importable. 2) Install things into virtualenvs using pip on the build host (possibly using pre-built wheels to speed that up), and then copy the entire virtualenv to the production host. If build and production hosts are identical in every way (including the path of the virtualenv), this will Just Work. There are also tools like https://github.com/PolicyStat/terrarium that aim to smooth this workflow. 3) Generate OS packages (debs or rpms) containing your complete app and installed dependencies. I'd recommend this for the smoothest experience once you have some tooling in place, but it will probably take some time initially to develop that tooling. You might find Hynek Schlawack's blog post on the topic helpful: http://hynek.me/articles/python-app-deployment-with-native-packages/ Carl ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Distributable binary with dependencies
Hi, I'm looking to package an application with all its dependencies for deployment on multiple hosts. I'd like to ensure that there is no compilation or setup step before starting the application in production. An nice to have ability would be to isolate base library dependencies per application (like virtualenv does). Ideally, the development -> deployment lifecycle would involve: (a) Build an application archive with all its dependencies baked in (b) Copy archive to a host in production. (c) Unwrap archive (d) Start services. (Note that the build host & production hosts are identical in architecture, OS patch level and python version). Having looked at various tools (e.g. distutils, setup tools, pip + virtualenv, egg, wheel, pyinstaller, etc.) available to address specific aspects/stages of the development lifecycle, I'm undecided on the tool/methodology to adopt in order to comprehensively solve the above problem. Any recommendation in this regard would be very helpful. Thanks for for your time. -Bharath ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
