Re: Packaging and deployment of standalone Python applications?
Hi Laura; and first off, thank you very much for your very insightful response. Some thoughts on that: Am 17.09.2015 um 11:06 schrieb Laura Creighton: Mozilla uses a hybrid static and binary build thing. http://www-archive.mozilla.org/build/static-build.html and confusingly calls that a static build. My Java didn't come as a generic Linux download. I had to use the debian package. Same for libreoffice. I needed a .deb for that. Ah ok. I haven't looked too deep into Mozilla or LibreOffice, but I know there are Firefox Linux und Linux 64bit downloads that, once unpacked, will run on Debian, Ubuntu, Fedora and any other distribution I tried so far. In LibreOffice, we do have some bundle (we didn't built ourselves) living somewhere in /opt, being copied to various production machines and this, too, runs on Debian and on CentOS without issues. I do have LibreOffice (off the Ubuntu ppa) installed on my workstation, but not on the production environment also because the LibreOffice in the Ubuntu repos, as far as I can tell, will pull X as dependency while we run headless servers. Similar about Java - here, at least in Ubuntu or Debian, in some situations I used Debian facilities to create deb packages out of the official Java download which is either an rpm or a tgz. That's why I wondered in the first place: In our current deployment of (Java) services, the Java runtime gets bundled with the service for some systems, zipped, moved to the host and started there. This works well across all Linux systems, as long as they are 64bit so far. That's why I tried doing the same for Python and initially thought about this in the first place. ;) [LibreOffice] If you download from this page you will see that it detects what sort of linux you are running and gets you a .rmp or a .deb depending on what you need. That's 4 choices, because 32 bit and 64 bit is also different. Well - yes, but have you so far tried unpacking, say, the rpm manually and running the files in there on Debian? Also, it's interesting to see they "just" make a difference between deb and rpm (32bit and 64bit of course). This would end up with the same deb being installed to Debian or Ubuntu - which works, while copying a Python interpreter binary from an Ubuntu to a Debian machine obviously doesn't seem to work. So in some ways they *do* get some sort of portability in this... ? If you can turn your pure python code into a python package, this will probably simplify your problem. But, no promises here. I don't know enough about what you are trying to do, and this is only a general rule of thumb. I will have a look at this, then, thanks. :) So far, this morning I followed the venv approach, created a venv locally on my Ubuntu development workstation, made sure to have the same python version (3.4) installed on the server (built from source in Debian), and then copied the whole venv structure (including my application) to that machine, replacing only the link to the python binary in the venv (which lives in /usr/local on the Debian system). So far this works, and if this generally would work, it seems an acceptable way to me. :) PyPy is a fast version of Python written in Python. It has a JIT. On our download page:http://pypy.org/download.html Neat. I just read about it but so far never actually used it. In our environment, Jython is pretty familiar (because mostly we're still a Java house and Jython integrates rather well with Java), but PyPy is interesting, definitely. So when docker came out, we jumped at the chance to get out of this mess. It worked. We're deliriously happy with the result. And now we have portable binaries that run anywhere. Armin Rigo (I think) figured out how to hack the RPATH to get it to work most everywhere. I am not sure how that goes, but Armin will either explain it to you or point you at the correct document to read if you ask. He's very nice that way. I will have a look at this and happily get back to this if it will become necessary. :) Two things come to mind, however (feel free to take this off list as I am unsure whether this still relates to Python): (a) Why is distributing PyPy so difficult, after all? From where I stand, by now I see this a "Python implementation on top of Python". How does this relate to (C)Python? How much of the PyPy core is actually platform dependent? (b) How did / does Docker help you here? So far I just experienced Docker as a solution to easily manage loads of virtual machines on top of certain machine templates; I haven't made much use of it so far nevertheless... Thanks very much anyway, all the best! Kristian -- https://mail.python.org/mailman/listinfo/python-list
Re: Packaging and deployment of standalone Python applications?
W dniu 14.09.2015 o 08:58, Kristian Rink pisze: [...] > > Any pointers, ideas, inspirations on that greatly appreciated - even > in total different ways if what I am about to do is completely off > anyone would do it in a Python environment. ;) Look at https://github.com/jordansissel/fpm/wiki - it's really nice tool, which converts python package into .deb, .rpm - so assuming that "everyone has some (2.x/3.x) python" is true, you can just ship .deb or .rpm files. reg m -- https://mail.python.org/mailman/listinfo/python-list
Re: Packaging and deployment of standalone Python applications?
In a message of Thu, 17 Sep 2015 08:23:52 +0200, Kristian Rink writes: >Hi Laura; > >and first off, thanks bunches for your comment. > >Am 17.09.2015 um 00:19 schrieb Laura Creighton: > > >> Your problem is likely with the shared library search paths. >> Different distributions put them in different places. It's a real >> pain, and the reason why docker is getting more popular. > >Well I thought something like this but then again wondered whether this >really would be the cause - after all, Thunderbird, Firefox, OpenOffice, >Java and a bunch of other applications that are available as "binary" >downloads for "generic Linux" do seem to work on RedHat/CentOS, Debian, >Ubuntu without these issues. Are these all statically built? Would this >be an option for distributing Python applications, as well? Mozilla uses a hybrid static and binary build thing. http://www-archive.mozilla.org/build/static-build.html and confusingly calls that a static build. My Java didn't come as a generic Linux download. I had to use the debian package. Same for libreoffice. I needed a .deb for that. For instance, if you look at the libre office instructions it first hops up and down saying you should use whatever your package manager provides. https://www.libreoffice.org/get-help/install-howto/linux/ Then it leads to the wiki which leads to the download page http://www.libreoffice.org/download/libreoffice-fresh/ If you download from this page you will see that it detects what sort of linux you are running and gets you a .rmp or a .deb depending on what you need. That's 4 choices, because 32 bit and 64 bit is also different. If it makes a mistake and decides you need the wrong sort of binary package, you can demand that you get the sort you want. This will handle most of them, but if you need Gentoo linux you will have to do something else, and there are undoubtably other linux versions that don't understand .deb or .rpms Now go back to the wiki and read https://wiki.documentfoundation.org/Documentation/Install/Linux the section Terminal-Based Install As I warned you, it is a real horror. >Or, other option: Would it work to, say, use venv, make sure there >always is a Python of the same version on the target machine installed >(say 3.4), and then distribute "just" the python parts without including >the interpreter / binary itself? Getting a Python of the same version is rarely necessary. That's not where your problem is. Except that Python 3.x are not compatible with Python 2.x, the general rule is that later versions of Python are backwards compatible with earlier ones. If you can turn your pure python code into a python package, this will probably simplify your problem. But, no promises here. I don't know enough about what you are trying to do, and this is only a general rule of thumb. >Thanks in advance and all the best, >Kristian Here's the PyPy Story. PyPy is a fast version of Python written in Python. It has a JIT. On our download page: http://pypy.org/download.html These days PyPy is really popular. So there is a real official debian package, ubuntu package, red hat package, and a gentoo package and all the rest. Which means there is a real human being who is reponsible for making such things for each distro. This is a welcome outcome of being popular. It wasn't always that way. We had to make binaries for wherever we wanted to run (everywhere). The download page still contains a list of possible binaries. Linux x86 binary (32bit, tar.bz2 built on Ubuntu 12.04 - 14.04) (see [1]) Linux x86-64 binary (64bit, tar.bz2 built on Ubuntu 12.04 - 14.04) (see [1]) ARM Hardfloat Linux binary (ARMHF/gnueabihf, tar.bz2, Raspbian) (see [1]) ARM Hardfloat Linux binary (ARMHF/gnueabihf, tar.bz2, Ubuntu Raring) (see [1]) ARM Softfloat Linux binary (ARMEL/gnueabi, tar.bz2, Ubuntu Precise) (see [1]) ... Note 1 contains more language about if this is not your exact distro, it will not work. We say this all over the place and people still do not get it. Our binrary list used to be a whole lot longer. And the situation was impossible. We couldn't build pypys for every linux distro on the planet, because we didn't own every one. So we were dependent on having happy users make one from source, and giving it to us. Which people did. But it meant that every time we made a release we had to spend forever talking to people over irc or on the mailing list who couldn't get the source build to build. Make a symlink here, make a symlink there. When you add to the mix that building pypy, starting with CPython instead of an earlier version of PyPy can grind away for 3 hours and then fall flat over when you first try to run it because you didn't get the dependencies quite right It was nothing but pain. So when docker came out, we jumped at the chance to get out of this mess. It worked. We're deliriously happy with the result. And now we have portable binaries that run anywhere. Armin Rigo (I think) figure
Re: Packaging and deployment of standalone Python applications?
On Thu, Sep 17, 2015 at 5:24 PM, Christian Gollwitzer wrote: > IMHO this is one of the lacks of CPython. Distributing source is not always > practical, especially if the project involves modules written in C, or a > large number of 3rd party libraries. To provide linux binaries as .rpm and > .deb, as suggested elsewhere in this thread, is even more cumbersome than > binaries in this case - you'd have to set the right dependencies for a huge > variety of distributions, such that the package manager pulls, e.g. > "python-pillow" on one distro and "libpillow" on another etc. A well > maintained portable distribution for all major platforms could substantially > ease that out. If you want to know what's state of the art there - and even more so: if you want to push the state of the art - join the distutils-sig mailing list: https://mail.python.org/mailman/listinfo/distutils-sig This is where these kinds of questions will be discussed, and (hopefully!) solved. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Packaging and deployment of standalone Python applications?
Am 16.09.15 um 21:29 schrieb Kristian Rink: Thanks, this already is pretty close to what I need. Playing with this and virtualenv, I figured out that this way it's pretty easily possible to have isolated Python environments _locally_. However I failed to package one of these environments and move it to, say, from my Ubuntu development host to a remote Debian server, I end up with errors like these while trying to run the Python off the environment on that host: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found This is a typical problem on Linux when moving binaries between different distributions. The libc is the most basic library, used by programs compiled from C (virtually everything). Fortunately it has a very good version management which makes it backwards compatible. Have you compiled everything from source? The solution is to compile everything on the oldest machine which you can get hold of. If you want portable binaries on Linux, I suggest that you carry along with your program all shared libraries except libc & friends (libm, libstdc++,...), which really are part of the operating system, and to setup a VM with an old libc6-distribution where you compile your stuff. For instance try Ubuntu 10.04 LTS. There is a nice management tools for these VMs called Vagrant https://www.vagrantup.com/ If you think, that this is lots of tedious work, you are right - deployment is hard work. Ideally, you would have Python installation that can be carried along as a binary directory and in variants for all major OSes. In the case of the Java VM this is just easier because somebody else already did the hard work. On Windows, I used PortablePython in the past for a similar project, but this is gone. Another possibility is ActivePython, but I think the free license does not allow to carry it to other computers - in principle it could work. IMHO this is one of the lacks of CPython. Distributing source is not always practical, especially if the project involves modules written in C, or a large number of 3rd party libraries. To provide linux binaries as .rpm and .deb, as suggested elsewhere in this thread, is even more cumbersome than binaries in this case - you'd have to set the right dependencies for a huge variety of distributions, such that the package manager pulls, e.g. "python-pillow" on one distro and "libpillow" on another etc. A well maintained portable distribution for all major platforms could substantially ease that out. Christian -- https://mail.python.org/mailman/listinfo/python-list
Re: Packaging and deployment of standalone Python applications?
Hi Laura; and first off, thanks bunches for your comment. Am 17.09.2015 um 00:19 schrieb Laura Creighton: > Your problem is likely with the shared library search paths. Different distributions put them in different places. It's a real pain, and the reason why docker is getting more popular. Well I thought something like this but then again wondered whether this really would be the cause - after all, Thunderbird, Firefox, OpenOffice, Java and a bunch of other applications that are available as "binary" downloads for "generic Linux" do seem to work on RedHat/CentOS, Debian, Ubuntu without these issues. Are these all statically built? Would this be an option for distributing Python applications, as well? Or, other option: Would it work to, say, use venv, make sure there always is a Python of the same version on the target machine installed (say 3.4), and then distribute "just" the python parts without including the interpreter / binary itself? Thanks in advance and all the best, Kristian -- https://mail.python.org/mailman/listinfo/python-list
Re: Packaging and deployment of standalone Python applications?
In a message of Wed, 16 Sep 2015 21:29:23 +0200, Kristian Rink writes: >Am 15.09.2015 um 08:59 schrieb paul.hermeneu...@gmail.com: >> >> https://docs.python.org/3/library/venv.html?highlight=venv#module-venv > >Thanks, this already is pretty close to what I need. Playing with this >and virtualenv, I figured out that this way it's pretty easily possible >to have isolated Python environments _locally_. However I failed to >package one of these environments and move it to, say, from my Ubuntu >development host to a remote Debian server, I end up with errors like >these while trying to run the Python off the environment on that host: > >/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found > >I bundled all the stuff in the virtualenv and also made sure to >dereference the symlinks in there. Are Python binaries so closely tied >to a particular Linux environment / distribution that what I am about to >do is impossible? Is there a "generic" Python for Linux binary that >works on all distributions, as things are for Java? > >TIA and all the best, >Kristian >-- >https://mail.python.org/mailman/listinfo/python-list Your problem is likely with the shared library search paths. Different distributions put them in different places. It's a real pain, and the reason why docker is getting more popular. According to http://www.eyrie.org/~eagle/notes/rpath.html there is a way to get around this by encoding the rpath in your application, but I cannot vouch for it personally. Laura -- https://mail.python.org/mailman/listinfo/python-list
Re: Packaging and deployment of standalone Python applications?
Am 15.09.2015 um 08:59 schrieb paul.hermeneu...@gmail.com: https://docs.python.org/3/library/venv.html?highlight=venv#module-venv Thanks, this already is pretty close to what I need. Playing with this and virtualenv, I figured out that this way it's pretty easily possible to have isolated Python environments _locally_. However I failed to package one of these environments and move it to, say, from my Ubuntu development host to a remote Debian server, I end up with errors like these while trying to run the Python off the environment on that host: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found I bundled all the stuff in the virtualenv and also made sure to dereference the symlinks in there. Are Python binaries so closely tied to a particular Linux environment / distribution that what I am about to do is impossible? Is there a "generic" Python for Linux binary that works on all distributions, as things are for Java? TIA and all the best, Kristian -- https://mail.python.org/mailman/listinfo/python-list
Re: Packaging and deployment of standalone Python applications?
>On Monday, September 14, 2015 at 8:58:51 AM UTC+2, Kristian Rink wrote: >> Folks; coming from a server-sided Java background, I'm recently >> exploring frameworks such as cherrypy or webpy for building RESTful >> services, which is quite a breeze and a pretty pleasant experience; >> however one thing so far bugs me: Using Java tooling and libraries >> such as DropWizard, it is pretty straightforward to build an >> "all-inclusive" application containing (a) all code of my own, (b) >> all required dependencies, (c) all other resources and, if >> required, even (d) a Java virtual machine in one large .zip file >> which can easily be copied to a clean Linux VM, unzipped and >> started there. Are there ways of doing so using Python, too? I'd >> like to set up a project structure / working environment that >> includes all Python 3rd party libraries, my own code and maybe even >> a "standard" Python runtime for 64bit Linux systems (to not depend >> too much on what version a certain Linux distribution actually >> ships) and focus on doing deployment to various machines at best by >> simply copying these packages around. Your main problem may not be technical, but political. PyInstaller will do what you want, and docker is really really interesting, but many linux users really detest the idea of getting a huge package which duplicates stuff they already have installed on their system. And it isn't something they are used to -- it is rarely done. Instead, they will want you to send them a .deb or a .rpm (whatever their OS package manager wants) or they will want a PyPI package. If you are distributing them python source, then rather than have you build them an app, they are more likely to want instructions for what are the dependencies so they can build your app itself, and also control whether or not to install the dependencies you require system-wide or in a virtualenv. You know your own situation best, but if you are aiming for widespread adoption by linux users everywhere, well, I am just warning you that they aren't going to like it. (Unless docker, which is very new, turns into The Next Big Thing, which I suppose always could happen.) Laura -- https://mail.python.org/mailman/listinfo/python-list
Re: Packaging and deployment of standalone Python applications?
On Monday, September 14, 2015 at 8:58:51 AM UTC+2, Kristian Rink wrote: > Folks; > > coming from a server-sided Java background, I'm recently exploring frameworks > such as cherrypy or webpy for building RESTful services, which is quite a > breeze and a pretty pleasant experience; however one thing so far bugs me: > Using Java tooling and libraries such as DropWizard, it is pretty > straightforward to build an "all-inclusive" application containing (a) all > code of my own, (b) all required dependencies, (c) all other resources and, > if required, even (d) a Java virtual machine in one large .zip file which can > easily be copied to a clean Linux VM, unzipped and started there. > > Are there ways of doing so using Python, too? I'd like to set up a project > structure / working environment that includes all Python 3rd party libraries, > my own code and maybe even a "standard" Python runtime for 64bit Linux > systems (to not depend too much on what version a certain Linux distribution > actually ships) and focus on doing deployment to various machines at best by > simply copying these packages around. > > Any pointers, ideas, inspirations on that greatly appreciated - even in total > different ways if what I am about to do is completely off anyone would do it > in a Python environment. ;) > > TIA and all the best, > Kristian If your business-cases allows it, I would seriously consider using Docker. I makes it pretty straightforward to move your deployments around from your development machine, to a test setup, to a cloud provider (e.g. AWS) etc. Lack or incomplete support on Windows systems is a little bit a deal breaker, but this situation is improving quickly. Marco -- https://mail.python.org/mailman/listinfo/python-list
Re: Packaging and deployment of standalone Python applications?
This might be helpful. https://docs.python.org/3/distributing/index.html See also https://docs.python.org/3/library/venv.html?highlight=venv#module-venv Folks; coming from a server-sided Java background, I'm recently exploring frameworks such as cherrypy or webpy for building RESTful services, which is quite a breeze and a pretty pleasant experience; however one thing so far bugs me: Using Java tooling and libraries such as DropWizard, it is pretty straightforward to build an "all-inclusive" application containing (a) all code of my own, (b) all required dependencies, (c) all other resources and, if required, even (d) a Java virtual machine in one large .zip file which can easily be copied to a clean Linux VM, unzipped and started there. Are there ways of doing so using Python, too? I'd like to set up a project structure / working environment that includes all Python 3rd party libraries, my own code and maybe even a "standard" Python runtime for 64bit Linux systems (to not depend too much on what version a certain Linux distribution actually ships) and focus on doing deployment to various machines at best by simply copying these packages around. Any pointers, ideas, inspirations on that greatly appreciated - even in total different ways if what I am about to do is completely off anyone would do it in a Python environment. ;) TIA and all the best, Kristian -- https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: Packaging and deployment of standalone Python applications?
Am 14.09.15 um 08:58 schrieb Kristian Rink: Folks; coming from a server-sided Java background, I'm recently exploring frameworks such as cherrypy or webpy for building RESTful services, which is quite a breeze and a pretty pleasant experience; however one thing so far bugs me: Using Java tooling and libraries such as DropWizard, it is pretty straightforward to build an "all-inclusive" application containing (a) all code of my own, (b) all required dependencies, (c) all other resources and, if required, even (d) a Java virtual machine in one large .zip file which can easily be copied to a clean Linux VM, unzipped and started there. Are there ways of doing so using Python, too? I'd like to set up a project structure / working environment that includes all Python 3rd party libraries, my own code and maybe even a "standard" Python runtime for 64bit Linux systems (to not depend too much on what version a certain Linux distribution actually ships) and focus on doing deployment to various machines at best by simply copying these packages around. Any pointers, ideas, inspirations on that greatly appreciated - even in total different ways if what I am about to do is completely off anyone would do it in a Python environment. ;) Look at pyinstaller. It creates monsters (i.e. really huge single file or single directory builds), but it works for my projects. Christian -- https://mail.python.org/mailman/listinfo/python-list
Re: Packaging and deployment of standalone Python applications?
Kristian Rink writes: > Folks; > > coming from a server-sided Java background, I'm recently exploring frameworks > such as cherrypy or webpy for building RESTful services, which is quite a > breeze and a pretty pleasant experience; however one thing so far bugs me: > Using Java tooling and libraries such as DropWizard, it is pretty > straightforward to build an "all-inclusive" application containing (a) all > code of my own, (b) all required dependencies, (c) all other resources and, > if required, even (d) a Java virtual machine in one large .zip file which can > easily be copied to a clean Linux VM, unzipped and started there. > > Are there ways of doing so using Python, too? I do not think that the Python support in this regard is as sophisticated as in the Java world. Have a look at "distribute/setuptools". It supports (a) and (b) (event though you must write a "setup.py" without the help of an UI) and allows to state the dependencies (part of (c)). It can generate a so called "egg" (a distribution unit) representing one distributable component. On installation, you still need a local Python (with its runtime environment) and the installation process will try to resolve the stated dependencies - potentially requiring internet access. In order to get one executable containing the Python interpreter and all required packages, there is "freeze" (and when I remember right a solution provided by "eGenix"). However, you must manually prepare the application such that "freeze" learns about the "required packages". "freeze" is likely too old that it already would interprete the dependency declarations. You might also have a look at "zc.buildout". Its primary task is to ensure the setup of a consistent development/deployment environment across different environments (hosts). It relies on a local Python installation - but can handle almost everything else. Again - without UI support. -- https://mail.python.org/mailman/listinfo/python-list
Packaging and deployment of standalone Python applications?
Folks; coming from a server-sided Java background, I'm recently exploring frameworks such as cherrypy or webpy for building RESTful services, which is quite a breeze and a pretty pleasant experience; however one thing so far bugs me: Using Java tooling and libraries such as DropWizard, it is pretty straightforward to build an "all-inclusive" application containing (a) all code of my own, (b) all required dependencies, (c) all other resources and, if required, even (d) a Java virtual machine in one large .zip file which can easily be copied to a clean Linux VM, unzipped and started there. Are there ways of doing so using Python, too? I'd like to set up a project structure / working environment that includes all Python 3rd party libraries, my own code and maybe even a "standard" Python runtime for 64bit Linux systems (to not depend too much on what version a certain Linux distribution actually ships) and focus on doing deployment to various machines at best by simply copying these packages around. Any pointers, ideas, inspirations on that greatly appreciated - even in total different ways if what I am about to do is completely off anyone would do it in a Python environment. ;) TIA and all the best, Kristian -- https://mail.python.org/mailman/listinfo/python-list
