Re: [Distutils] dev versions
Reinout van Rees wrote: On 2009-10-08, Ian Bicking i...@colorstudy.com wrote: So after creating, say, version 0.3.1, I always mark a package as 0.3.2dev. But this is annoying, you might never create a version 0.3.2 (e.g., 0.4 might be the next level). So, it would be better to use something like 0.3.1~dev. What is considered best practice for this? Ideally something that works with both Setuptools and the upcoming Distribute version spec. a) Where's the annoyment exactly? It is easy to change and it's a release-time decision anyway. b) In a previous discussion on a zope mailinglist (about using '0' for this purpose, which was pretty much shot down for the zope toolkit because of the problems attached to it), someone mentioned adding '+svn' to the previous version number. So from 0.3.1 to 0.3.1+svn. Apparently that sorts it behind 0.3.1. You could try something like that. The poster mentioned it as a debian standard. Reinout I just caught the tail end of this thread so please excuse if you may have covered this. With regard to package version and release strings here is something that I have been using successfully that generates proper lexical package ordering for tarballs, debian (.deb) and redhat (.rpm) packaging. The same patterns should work just as well for any other packaging system: We use the following for the various VERSION-RELEASE strings: examples use: bzr as version control system but you can substitute hg, git, svn, rcs, or any other type of vcs rpm as packaging system Development: version=5.0.0 release=0.00012345 (where 00012345 is the revision number from your vcs in ZERO-PADDED FIELD OF 8) tarball: foo-5.0.0-0.00012345.tar.gz rpm: foo-5.0.0-0.00012345.noarch.rpm Alpha: version=5.0.0 release=0.alpha1.00123456 tarball: foo-5.0.0-0.alpha1.00123456.tar.gz rpm: foo-5.0.0-0.alpha1.00123456.noarch.rpm Beta: version=5.0.0 release=0.beta1.00234567 tarball: foo-5.0.0-0.beta1.00234567.tar.gz rpm: foo-5.0.0-0.beta1.00234567.noarch.rpm Release Candidate: version=5.0.0 release=0.rc1 tarball: foo-5.0.0-0.rc1.tar.gz rpm: foo-5.0.0-0.rc1.noarch.rpm Release Candidate code fix: version=5.0.0 release=0.rc1.00345678 tarball: foo-5.0.0-0.rc1.00345678.tar.gz rpm: foo-5.0.0-0.rc1.00345678.noarch.rpm Final Release: version=5.0.0 release=1 tarball: foo-5.0.0-1.tar.gz rpm: foo-5.0.0-1.noarch.rpm Final Release code fix: version=5.0.0 release=2 tarball: foo-5.0.0-2.tar.gz rpm: foo-5.0.0-2.noarch.rpm Notice that lexical ordering is proper in all cases. Even where the alpha, beta, rc pre-releases may be followed by a bug fix revision. The key is that all pre-releases (alpha, beta, etc.) have a release string starting with 0. and the first production release has a release string of 1. This is how the lexical ordering is kept for all packaging types (tarballs, rpm, deb). Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Problem with distribute and converting tests for python3
Michael Whapples wrote: OK, I didn't realise the .py files are included in a binary distribution, I understood it to be that the .pyc file was enough and so had followed through that the .py file wouldn't be included in a binary distribution like it is typically done in java (java tends not to include the .java source files in a binary .jar file, only the .class files). I did warn you that I may be making some assumptions/hold views due to my java programming. This leads to some more general questions about python and distributing modules (eg. why do we need binary distributions (bdist_egg, bdist, etc) when a source distribution should be fine and do all needed, why does python distribute source and compiled files in a binary distribution (what does the source files add in such a case), etc). Anyway back to the actual topic. My main question of the possible solution is, why can't we have a way of specifying extra directories to run 2to3 on? Tests are an example (well at least the way I view tests) of when this could be useful, it may be useful in other cases (not thought of an example). Michael Whapples On 12/10/09 18:16, Lennart Regebro wrote: 2009/10/12 Michael Whapplesmwhapp...@aim.com: In that case I would ask them to download the source distribution and run the tests there, they probably will want the source distribution as I may need to ask them to apply a patch and test it out (they have the environment I didn't foresee/have). Well, that's one step extra, but fine. I question how unit tests in a binary package are like a sort of documentation as the user will not be able to see the test source code as it will all be in .pyc files. A binary distribution will include the .py files as well. Binary is different from source in that extensions are already compiled. You don't need to make binary distributions for anything else than Windows, and then only if you have C-extensions. Modules without c-extensions are best distributed as a source distribution. The problem is that these files doesn't end up in the binary distribution and hence 2to3 can't be run on them. Bad wording on my part, I said compile when I meant convert, so point two should be As distribute will include test/test*.py, why can't distribute run 2to3 on these files and so convert my tests and so then be able to run tests from the test command. This probably would need distribute to create a separate directory (eg. test_3k) and then run tests from the new location to achieve this. No, that wasn't bad wording on your part, I understood perfectly, I think. The problem is that these files doesn't end up in the binary distribution and hence 2to3 isn't run on them as the 2to3 conversion is done in build_py. Binary distribution is a packaging type such as an rpm or a deb file usually for a particular distro. Not about whether source or various compilation products are or are not included in these binary packages. You can write an install that requires no build and and no source and just moves precompiled files into position. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [ANN] stdeb 0.3.2 and 0.4.1
Andrew, I installed stdeb 0.3.2 from PyPi on my Hardy server and ran a couple tests just to be sure. stdeb 0.3.2 on Hardy is working fine. I didn't see any problems. 'bdist_deb' worked as expected and generated both a .dsc and a .deb file for my project which installed perfectly. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] [ANN] stdeb 0.3.2 and 0.4.1
Andrew, Have you already or are you planning on submitting 'python-stdeb' packages to the debian and ubuntu repositories? Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_deb in stdeb
Andrew Straw wrote: Gerry Reno wrote: Andrew, how can I generate the .deb package without the leading 'python-' in the name? Is this configurable? Yes. The 'Package' option. Described in Customizing the produced Debian source package (config options) in the README.rst. Ok, in my stdeb.cfg I put this: Package: myapp And it generates the .deb without the 'python-' prefix. But, when I install this .deb it installs but no longer runs the postinst script. If I put back the 'python-' prefix then the postinst runs again. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_deb in stdeb
Gerry Reno wrote: Andrew Straw wrote: Gerry Reno wrote: Andrew, how can I generate the .deb package without the leading 'python-' in the name? Is this configurable? Yes. The 'Package' option. Described in Customizing the produced Debian source package (config options) in the README.rst. Ok, in my stdeb.cfg I put this: Package: myapp And it generates the .deb without the 'python-' prefix. But, when I install this .deb it installs but no longer runs the postinst script. If I put back the 'python-' prefix then the postinst runs again. Ok, I found it. In the debian/ control directory I had to rename the postinst script and remove the leading 'python-' from it as well. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_deb in stdeb
Gerry Reno wrote:
Andrew Straw wrote:
Gerry Reno wrote:
Ok, here's what happens if I DON'T use the option (shown with the old
code - same logic):
$ python setup.py bdist_deb
...
running build_scripts
...
if test 2.5 = 2.5 -o 2.5 = 2.4; then \
export SVEMOPT=--single-version-externally-managed ;
python2.5 -c import
setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
install \
--no-compile ${SVEMOPT} \
...
error: option --single-version-externally-managed not recognized
...
process stops on the error
I can't explain it, but this is what happens. This is on Ubuntu Hardy
using python 2.5.2.
I see. I think it's because you're subclassing distutils.command.install
rather than setuptools.command.install for your install command.
Another idea is that you could also detect whether setuptools is
imported by doing something like if 'setuptools' in sys.modules and
then subclassing the appropriate install command based on that. Then you
wouldn't need to pass the --disable-single-version-externally-managed
argument at all.
-Andrew
Ok, a little guidance please. As I understand it if I import
setuptools into my app setup.py and I subclass my install from
setuptools.command.install, then I can also use the option
'--single-version-externally-managed' to make the install behave like
the distutils style install. Is that correct?
If this is true then what I would like to do is to force this option
on in the code. How can I do that? I know I can probably put it into
setup.cfg but I just think it would be better hardwired into the code.
No one?
Ok, I refactored my app install code and have it running with setuptools.
Andrew, how can I generate the .deb package without the leading
'python-' in the name? Is this configurable?
Regards,
Gerry
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_deb in stdeb
Andrew Straw wrote: Gerry Reno wrote: Then I'm ok then with having 'bdist_deb' be the name and using the plugin approach. Andrew, what do you think about this solution? It's already committed as of a couple days ago in the old-stable branch and I just merged it into the master branch. I'll release 0.3.1 (from the old-stable branch) and 0.4 (from the master branch) with this soon. -Andrew In reviewing the gerry-reno git branch and the old-stable branch I don't see where there are any user-options declared in stdeb/command/bdist_deb.py. Shouldn't the same user-options be there as are in sdist_dsc.py? The 'bdist_deb' command usefulness would be very limited without the ability to pass options to it and hence to 'sdist_dsc'. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_deb in stdeb
Andrew Straw wrote: Gerry Reno wrote: Andrew Straw wrote: Gerry Reno wrote: Then I'm ok then with having 'bdist_deb' be the name and using the plugin approach. Andrew, what do you think about this solution? It's already committed as of a couple days ago in the old-stable branch and I just merged it into the master branch. I'll release 0.3.1 (from the old-stable branch) and 0.4 (from the master branch) with this soon. -Andrew In reviewing the gerry-reno git branch and the old-stable branch I don't see where there are any user-options declared in stdeb/command/bdist_deb.py. Shouldn't the same user-options be there as are in sdist_dsc.py? The 'bdist_deb' command usefulness would be very limited without the ability to pass options to it and hence to 'sdist_dsc'. What options do you want to pass to it? bdist_deb does very little other than call dpkg-buildpackage on the result of sdist_dsc. As far as how -- that's standard distutils. You should be able to follow the lead from the sdist_dsc command, for example. Well, for starters, we need to be able to pass the option '--ignore-single-version-externally-managed' and beyond that we need to pass other options down to sdist_dsc, things like '--ignore-install-requires'. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_deb in stdeb
Andrew Straw wrote: Gerry Reno wrote: Andrew Straw wrote: Gerry Reno wrote: Andrew Straw wrote: Gerry Reno wrote: Then I'm ok then with having 'bdist_deb' be the name and using the plugin approach. Andrew, what do you think about this solution? It's already committed as of a couple days ago in the old-stable branch and I just merged it into the master branch. I'll release 0.3.1 (from the old-stable branch) and 0.4 (from the master branch) with this soon. -Andrew In reviewing the gerry-reno git branch and the old-stable branch I don't see where there are any user-options declared in stdeb/command/bdist_deb.py. Shouldn't the same user-options be there as are in sdist_dsc.py? The 'bdist_deb' command usefulness would be very limited without the ability to pass options to it and hence to 'sdist_dsc'. What options do you want to pass to it? bdist_deb does very little other than call dpkg-buildpackage on the result of sdist_dsc. As far as how -- that's standard distutils. You should be able to follow the lead from the sdist_dsc command, for example. Well, for starters, we need to be able to pass the option '--ignore-single-version-externally-managed' and beyond that we need to pass other options down to sdist_dsc, things like '--ignore-install-requires'. Just pass the arguments directly to sdist_dsc. It should be something like this: python setup.py sdist_dsc --ignore-single-version-externally-managed --ignore-install-requires bdist_deb How's that going to work? You have 'sdist_dsc' on the command line as well as 'bdist_deb'. 'bdist_deb' also calls 'sdist_dsc' internally but this time it would be without any arguments? How can that work? Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_deb in stdeb
Andrew Straw wrote:
Olof Bjarnason wrote:
2009/9/28 Gerry Reno gr...@verizon.net:
Just pass the arguments directly to sdist_dsc. It should be something
like this:
python setup.py sdist_dsc --ignore-single-version-externally-managed
--ignore-install-requires bdist_deb
How's that going to work? You have 'sdist_dsc' on the command line as well
as 'bdist_deb'. 'bdist_deb' also calls 'sdist_dsc' internally but this
time it would be without any arguments? How can that work?
This is standard distutils command processing -- this has nothing to do
with stdeb per se. The reason I gave an example not involving stdeb
(which was snipped out by you in your quote above) is so that you can
try it yourself and explore further without thinking it's somehow an
stdeb issue. Specifically, you do not need to understand anything
specific to stdeb to understand how to pass multiple arguments to
sub-commands of a distutils command. (And if stdeb handles things
differently, it's a bug and I would appreciate hearing about it.)
For example, build_ext is a sub-command of install that get called
to build extension modules prior to installation. (Being a sub-command
is not strictly true in the sense that the install command doesn't list
build_ext in its sub_commands in distutils/commands/install.py, but it
acts as if it's true.) Likewise, sdist_dsc is a sub-command of
bdist_deb. In all cases one, might want to pass arguments to the
sub-command before running a later command. distutils allows this.
Ok, the commands behave like makefile rules, once run they don't run
again.
But there are still several issues here:
Remember that I said that my goal with 'bdist_deb' was for users to have
a SINGLE command to generate a .deb.
What needs to be achieved is for a command like this:
$ python setup.py bdist_deb
OR
$ python setup.py bdist_deb --ignore-single-version-externally-managed
to be possible. 'bdist_deb' would call sdist_dsc internally with the
necessary args WITHOUT having to explicitly put 'sdist_dsc' on the
command line.
This is how I was expecting that you would implement the bdist_deb
command inside of stdeb.
Another issue: In util.py on line 962 (gerry-reno git):
%(setup_env_vars)spython$* -c import
setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
install \\
Maybe it should be like this (note semicolon splitting line into two
statements):
%(setup_env_vars)s;python$* -c import
setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
install \\
Otherwise any of the env vars don't seem to be in effect for the line
itself. If you put any of the env vars in the line, they don't expand
to the value set in set_env_vars. They expand to previous value or to null.
Regards,
Gerry
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_deb in stdeb
Olof Bjarnason wrote: Ok, the commands behave like makefile rules, once run they don't run again. But there are still several issues here: Remember that I said that my goal with 'bdist_deb' was for users to have a SINGLE command to generate a .deb. What needs to be achieved is for a command like this: $ python setup.py bdist_deb OR $ python setup.py bdist_deb --ignore-single-version-externally-managed to be possible. 'bdist_deb' would call sdist_dsc internally with the necessary args WITHOUT having to explicitly put 'sdist_dsc' on the command line. This would be _exactly_ what I'm looking for :) I know. I've had many of my users requesting exactly the same thing as well and I've been pursuing this for months now and when I got a 'bdist_deb' working with stdeb I knew I was getting close. It just needs a little more tweeking and I think we'll have it. Let's wait and see what Andrew says here. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_deb in stdeb
Andrew Straw wrote: Gerry Reno wrote: Olof Bjarnason wrote: Ok, the commands behave like makefile rules, once run they don't run again. But there are still several issues here: Remember that I said that my goal with 'bdist_deb' was for users to have a SINGLE command to generate a .deb. What needs to be achieved is for a command like this: $ python setup.py bdist_deb OR $ python setup.py bdist_deb --ignore-single-version-externally-managed to be possible. 'bdist_deb' would call sdist_dsc internally with the necessary args WITHOUT having to explicitly put 'sdist_dsc' on the command line. This would be _exactly_ what I'm looking for :) I know. I've had many of my users requesting exactly the same thing as well and I've been pursuing this for months now and when I got a 'bdist_deb' working with stdeb I knew I was getting close. It just needs a little more tweeking and I think we'll have it. Let's wait and see what Andrew says here. I don't understand what you're waiting on me for at this point. Olaf -- as I understand it -- you need to work with Ubuntu 9.04. I just released stdeb 0.4 which supports this bdist_deb and doesn't pass the --single-version-externally-managed option anyway and therefore doesn't support the --ignore-single-version-externally-managed option. I don't understand why you need to pass --ignore-single-version-externally-managed. python setup.py bdist_deb should work for you. Please report with specific information about expected and actual behavior if things are not working to your desire (taking care to read the, admittedly minimal, documentation in README.rst). Gerry -- I don't understand why you want to subvert the normal distutils way of doing things. Passing arguments to bdist_deb that are really arguments to sdist_dsc just isn't the way distutils does things. Please explain this then. Why EVEN bother to call 'sdist_dsc' from within 'bdist_deb' if you cannot somehow pass arguments to the internal 'sdist_dsc' call? You're asking for the equivalent of being able to pass arguments to the distutils install command that are ultimately intended for the distutils build_ext command. If you want to test (and possibly implement) functionality such as adding a [sdist_dsc] section to setup.cfg where these options could just live so that you don't need to pass arguments at the command line, thats would be welcome. Gerry, point 2 -- I still think you're not looking at your own big picture here -- AFAIK, you are bending over backwards to attempt to pass the --ignore-single-version-externally-managed so that you don't have to import setuptools in your setup.py file to avoid the monkeypatching that setuptools does. Stop. NO. IF I import stdeb in the setup.py THEN any local distutils install class gets an error: --single-version-externally-managed not recognized (or something similar). The only way to get by this is by passing in the NEW option that I made, '--ignore-single-version-externally-managed' which removes this option from the install command which then allows the install to succeed. This is because it looks like stdeb hijacks the distutils install command. But, setuptools is explicitly imported in the debian/rules files created in stdeb 0.3.x. I get the impression that you're fixated on this argument passing business and not looking at the bigger picture of the architecture involved and whether your ultimate goal is even achievable with stdeb 0.3.x. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_deb in stdeb
Andrew Straw wrote:
Gerry Reno wrote:
What if stdeb only had the command 'bdist_deb' and had no other command.
I will not remove the sdist_dsc command from stdeb. I believe that the
ability to produce debian source packages is much more important that
the ability to produce binary packages which only target a single
architecture and version of Debian/Ubuntu. bdist_deb is just a trivial
add-on to that saving a couple lines of typing which may be helpful to
people who aren't familiar with those lines. But I see the .dsc and
associated files as much more useful as the compiled .deb files.
LMAO. Dang it. I am not suggesting or implying that you should remove
the 'sdist_dsc' command. I'm trying to illustrate that the
commandA-opts are just that and nothing more: commandA-opts. And
whatever part of the build/install process that they apply to, then the
commandA can use them for WHATEVER purpose. Which means that since
'bdist_deb' calls 'sdist_dsc' it is perfectly ok to allow all
'sdist_dsc' options to be passed into the 'bdist_deb' command which can
then pass them along to 'sdist_dsc'. 'bdist_deb' is a SUPERSET of
'sdist_dsc'.
Everything else was taken care of internally. What then? Are you
saying then that it is impossible to pass in any options to
'bdist_deb' to affect the generation of the source package? If true,
this makes NO SENSE.
I still don't understand why you insist the options are to the bdist_deb
command. What is wrong with passing the options to the sdist_dsc command
where they take effect anyway?
Yes, I can see that once we got invaded with setuptools it seems you
really have no other choice.
Hang on a minute. The st in stdeb is for setuptools. While in the
0.4 version this may take a less prominent role, and Distribute might be
able to step in, the ontogeny of stdeb is very much tied up with
setuptools. So I'm not comfortable promoting the idea that there ever
existed stdeb without setuptools. Maybe it can happen in the future --
but it's certainly never happened in the past.
I never said that stdeb ever existing in any other form but that of
using setuptools. I was merely making a statement as to the impact that
this has.
And getting back to the issue that got skipped:
Another issue: In util.py on line 962 (gerry-reno git):
%(setup_env_vars)spython$* -c import
setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
install \\
Maybe it should be like this (note semicolon splitting line into two
statements):
%(setup_env_vars)s;python$* -c import
setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
install \\
Otherwise any of the env vars don't seem to be in effect for the line
itself. If you put any of the env vars in the line, they don't
expand to the value set in set_env_vars. They expand to previous
value or to null.
For setting environment variables this way, they cannot be separated by
a semicolon
The environment variables are not individually separated by a
semicolon. The WHOLE shell script environment variable declaration
statement needs to be separated from the install line statement.
Simple demonstration of the problem with the 'echo' statement simulating
the 'install' statement:
$ a='123' b='456' echo test $a $b
test
$ a='123' b='456'; echo test $a $b
test 123 456
Regards,
Gerry
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_deb in stdeb
Andrew Straw wrote: Gerry Reno wrote: Andrew Straw wrote: Gerry Reno wrote: What if stdeb only had the command 'bdist_deb' and had no other command. I will not remove the sdist_dsc command from stdeb. I believe that the ability to produce debian source packages is much more important that the ability to produce binary packages which only target a single architecture and version of Debian/Ubuntu. bdist_deb is just a trivial add-on to that saving a couple lines of typing which may be helpful to people who aren't familiar with those lines. But I see the .dsc and associated files as much more useful as the compiled .deb files. LMAO. Dang it. I am not suggesting or implying that you should remove the 'sdist_dsc' command. I'm trying to illustrate that the commandA-opts are just that and nothing more: commandA-opts. And whatever part of the build/install process that they apply to, then the commandA can use them for WHATEVER purpose. Which means that since 'bdist_deb' calls 'sdist_dsc' it is perfectly ok to allow all 'sdist_dsc' options to be passed into the 'bdist_deb' command which can then pass them along to 'sdist_dsc'. 'bdist_deb' is a SUPERSET of 'sdist_dsc'. I see -- you want to pass-through the options to sdist_dsc. Fine, implement this in a patch and I'll accept it. Oh sure, I get to have all the fun. For setting environment variables this way, they cannot be separated by a semicolon The environment variables are not individually separated by a semicolon. The WHOLE shell script environment variable declaration statement needs to be separated from the install line statement. Simple demonstration of the problem with the 'echo' statement simulating the 'install' statement: $ a='123' b='456' echo test $a $b test $ a='123' b='456'; echo test $a $b test 123 456 Passing to a subprocess is different. We're passing to a subprocess: $ a='123' python -c import os; print os.environ['a'] 123 $ a='123'; python -c import os; print os.environ['a'] Traceback (most recent call last): File string, line 1, in module File /usr/lib/python2.6/UserDict.py, line 22, in __getitem__ raise KeyError(key) KeyError: 'a' Ok, I see what is happening. In order for the env vars to be available to both the line itself AND to the subprocess it has to be declared as exported like this: export a='123' b='456'; python -c import os; print os.environ['a'] echo $b 123 456 Now it will work both ways. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_deb in stdeb
Gerry Reno wrote:
Andrew Straw wrote:
Gerry Reno wrote:
Andrew Straw wrote:
Gerry Reno wrote:
What if stdeb only had the command 'bdist_deb' and had no other
command.
I will not remove the sdist_dsc command from stdeb. I believe
that the
ability to produce debian source packages is much more important that
the ability to produce binary packages which only target a single
architecture and version of Debian/Ubuntu. bdist_deb is just a
trivial
add-on to that saving a couple lines of typing which may be helpful to
people who aren't familiar with those lines. But I see the .dsc and
associated files as much more useful as the compiled .deb files.
LMAO. Dang it. I am not suggesting or implying that you should
remove the 'sdist_dsc' command. I'm trying to illustrate that the
commandA-opts are just that and nothing more: commandA-opts. And
whatever part of the build/install process that they apply to, then
the commandA can use them for WHATEVER purpose. Which means that
since 'bdist_deb' calls 'sdist_dsc' it is perfectly ok to allow all
'sdist_dsc' options to be passed into the 'bdist_deb' command which
can then pass them along to 'sdist_dsc'. 'bdist_deb' is a SUPERSET of
'sdist_dsc'.
I see -- you want to pass-through the options to sdist_dsc. Fine,
implement this in a patch and I'll accept it.
Oh sure, I get to have all the fun.
For setting environment variables this way, they cannot be
separated by
a semicolon
The environment variables are not individually separated by a
semicolon. The WHOLE shell script environment variable declaration
statement needs to be separated from the install line statement.
Simple demonstration of the problem with the 'echo' statement
simulating the 'install' statement:
$ a='123' b='456' echo test $a $b
test
$ a='123' b='456'; echo test $a $b
test 123 456
Passing to a subprocess is different. We're passing to a subprocess:
$ a='123' python -c import os; print os.environ['a']
123
$ a='123'; python -c import os; print os.environ['a']
Traceback (most recent call last):
File string, line 1, in module
File /usr/lib/python2.6/UserDict.py, line 22, in __getitem__
raise KeyError(key)
KeyError: 'a'
Ok, I see what is happening. In order for the env vars to be available
to both the line itself AND to the subprocess it has to be declared as
exported like this:
export a='123' b='456'; python -c import os; print os.environ['a']
echo $b
123
456
Now it will work both ways.
Regards,
Gerry
I have attached a replacement 'bdist_deb.py' file that permits passing
arguments to bdist_deb which in turn passes them down to sdist_dsc.
and a util.py diff (made against the gerry-reno git branch) makes the
setup_env_vars work for both the line statements as well as the
subprocesses.
I have tested both on Ubuntu 8.0.4 Hardy with success.
Regards,
Gerry
962,963c962,963
%(setup_env_vars)spython$* -c import
setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
install \\
--no-compile ${SVEMOPT} \\
---
export %(setup_env_vars)s; python$* -c import
setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
install \\
--no-compile $${SVEMOPT} \\
968,969c968,969
%(setup_env_vars)spython$* -c import
setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
install \\
--no-compile ${SVEMOPT} --install-layout=deb \\
---
export %(setup_env_vars)s; python$* -c import
setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
install \\
--no-compile $${SVEMOPT} --install-layout=deb \\
import os
import stdeb.util as util
from stdeb.command.sdist_dsc import sdist_dsc
__all__ = ['bdist_deb']
class bdist_deb(sdist_dsc):
description = 'distutils command to create debian binary package'
# extend the run method
def run(self):
# call parent run() method to generate .dsc source pkg
sdist_dsc.run(self)
# execute system command and read output (execute and read output of find cmd)
dsc_tree = 'deb_dist'
target_dir = None
for entry in os.listdir(dsc_tree):
fulldir = os.path.join(dsc_tree,entry)
if os.path.isdir(fulldir):
if target_dir is not None:
raise ValueError('more than one directory in deb_dist. '
'Unsure which is source directory')
else:
target_dir = fulldir
if target_dir is None:
raise ValueError('could not find debian source directory')
# define system command to execute (gen .deb binary pkg)
syscmd = ['dpkg-buildpackage','-rfakeroot','-uc','-b']
util.process_command(syscmd,cwd=target_dir)
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org
Re: [Distutils] Installing stdeb on vanilla Ubuntu
Olof Bjarnason wrote: Hi distutils! So I'm quite sure stdeb is the tool for py2deb I'm looking for. What are the correct steps of installing this tool, assuming Ubuntu9.04. I guess I need this: 1. Python (check) 2. distutils (check) 3. setuptools (dunno? how do I install?) 4. stdeb (how do I install correctly?) As all of you know by now, I'm fairly new to distutils, PyPI, eggs, cheesecake, setuptools and stdeb. It's a whole new universe! :) You need to install setuptools from your distro package manager. You can then use easy_install to install stdeb. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Tarek Ziadé wrote: 2009/9/23 Andrew Straw straw...@astraw.com: Well, it wouldn't be setting a precedent that importing a module would extend the functionality of the stdlib's distutils commands -- the setuptools and distribute projects already do this. For example, the sdist command in setuptools has many differences with that of sdist in plain distutils. The point is that a user can choose not to import setuptools and thus not get the different behavior. The case here is that if the stdlib ever grows bdist_deb, that's fine, but I don't see why that should prevent stdeb from implementing a bdist_deb command of its own. In this case, I think the issue is even less relevant, since there is no stdlib bdist_deb command. Users would naturally expect bdist_deb to do exactly what stdeb is implementing. As long as I'm going ahead an implementing the functionality (which I'm ambivalent about in and of itself for previously stated reasons), I think it should be named the most obvious name. During last Pycon summit, we v'e said that part of the problem with Distutils was that it was doing too much things and that we should probably try to remove all platform-specific commands to reduce its scope. For instance, bdist_rpm is fine, but being located in Python stdlib, its release cycle is too long, and it would probably be better for it to be on its own, managed by people from the rpm community (if this term makes sense, e.g. Fedora guys maybe ?) Same thing goes with bdist_deb : while it's more controversial to remove existing bdist_* command, we will not include new bdist_* commands for sure in Distutils, and rather let the debian/ubuntu community provide the best packaging solution on the top of Distutils. That said, if stdeb provides a plain Distutils command, that doesn't use Setuptools and if that command is stable and used in the community, we will be happy to add a pointer to it in Distutils documentation. The problem of using Setuptools or Distribute in a command is that it does a lot of patching (much more that overriding distutils sdist command) and that changes Distutils global behavior. Notice that the Distribute project plans to remove that patching in the future. Notice also that Distutils has a plugin system, where you can point a package that contains extra commands. This configuration can be done in distutils.cfg. This allows the usage of the command in the CLI without having to import the command directly in the setup.py file. Last, about the name, I think using bdist_deb is fine imho. If another project uses the same name, it's juste a matter of configuring the one to be used in a given context. Regards Tarek So are you saying then that a stddeb-based 'bdist_deb' should be implemented as a standalone plugin to Distribute? And how would that work for distros such as Ubuntu 8.0.4 Hardy? Would they be able to take advantage of Distribute? Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Tarek Ziadé wrote: Gerry: So are you saying then that a stddeb-based 'bdist_deb' should be implemented as a standalone plugin to Distribute? And how would that work for distros such as Ubuntu 8.0.4 Hardy? Would they be able to take advantage of Distribute? Yes, it's just a matter of packaging this command in a standalone distribution. Once it's installed, and distutils.cfg properly configured, it can be used as a plugin. Distribute is not yet a good citizen for being used in such packages, because it still patches distutils like setuptools does. Eric: I think a long term goal should be getting away from the plugin nature of distutils. I don't see why distutils (or my mythical distlib) shouldn't just be a package (or more likely multiple packages) that stand-alone tools can use. Other than much complexity and confusion, I don't see what plugins have bought us. Not everything (and especially not distutils) needs to be a framework. It's just a matter of defining *where* is the CLI and how it works. The plugin system here is just to make those commands reachable from distutils's setup.py machinery. If another package has its own CLI, it can ignore this plugin system I guess. I'll admit that this plan in years down the road, at best. And it's not anyone's official position. Agreed, we need to think about alternatives. but in any case, nowadays, if the stdeb project wrap its work in a bdist_deb command that can be used as a distutils plugin, I don't think it's a problem for the future. I would be pretty happy about it :) Tarek Then I'm ok then with having 'bdist_deb' be the name and using the plugin approach. Andrew, what do you think about this solution? Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] disabling --single-version-externally-managed in stdeb 0.3.x
Andrew Straw wrote: Gerry Reno wrote: Andrew, Here's another patch attempt. Is this what you had in mind? Regards, Gerry OK, I looked at this, and now I'm throughly confused by what you are trying to do. Back to your big picture, as I understood it, you were trying to avoid importing setuptools because you derive some distutils commands and you don't want to derive from the setuptools versions. And the --single-version-externally-managed was giving you problems because that's not supported by distutils. But A) I can't see how this patch would address the issue of setuptools not being imported, because that still happens. Yes, inside the 'run' method within the 'bdist_deb' distutils command I had imported setuptools there but not at the top of setup.py. And B) the patch as you sent it was incomplete, requiring more work on my part even to be able to get it to run without raising exceptions. So I'm not sure that I actually get what your patch does here. I attempted to do my best and have now started a branch for this patch and subsequent efforts. See http://github.com/astraw/stdeb/tree/gerry-reno . My testing on this issue has been based on the version in this branch. Please either submit any patches off the old-stable branch or extending this gerry-reno branch. The patch looks for the option '--ignore-single-version-externally-managed' and then accordingly sets an env var that is used in the install line. This looked like a reasonable way to control behavior inside the rules. That's why I was asking if that is what you had in mind. I'll look at the branches and what needed to be added. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote:
I saw your patch, but that wasn't what I meant. Disabling
--single-version-externally-managed has to be optional, and the argument
must continue to be used by default. Simply removing the call will
create a regression for namespace packages. Thus, the
--single-version-externally-managed should only be removed when someone
explicitly asks for it to be removed (because they presumably know what
they're doing or are at least aware it may have negative consequences).
Hence my suggestion to look at how the --ignore-install-requires works
as an example of how to pass an option via the distuils infrastructure.
I'll look at --ignore-install-requires and send another patch.
OK, based on your code I added a bdist_deb distutils command to stdeb.
I've checked this into the old-stable branch and I'd appreciate it if
you can check whether this works for you and send me any comments.
Invoke it like this:
python -c import stdeb; excecfile('setup.py') bdist_deb
See this was my point of why my 'bdist_deb' belonged as a distutils
patch and not in stdeb.
The inherited capabilities, behaviors and invocation style needs to
follow all the other 'bdist' command types.
And that means being able to invoke it the same as say 'bdist_rpm' like:
python setup.py bdist_deb
or
python setup.py bdist_deb --std bdist type arg ...
If we start allowing 'bdist' derivative commands to diverge in their
inherited capabilities, invocation style, and expected behaviors then
chaos will ensue.
Regards,
Gerry
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote:
Gerry Reno wrote:
Andrew Straw wrote:
I saw your patch, but that wasn't what I meant. Disabling
--single-version-externally-managed has to be optional, and the argument
must continue to be used by default. Simply removing the call will
create a regression for namespace packages. Thus, the
--single-version-externally-managed should only be removed when someone
explicitly asks for it to be removed (because they presumably know what
they're doing or are at least aware it may have negative consequences).
Hence my suggestion to look at how the --ignore-install-requires works
as an example of how to pass an option via the distuils infrastructure.
I'll look at --ignore-install-requires and send another patch.
OK, great.
OK, based on your code I added a bdist_deb distutils command to stdeb.
I've checked this into the old-stable branch and I'd appreciate it if
you can check whether this works for you and send me any comments.
Invoke it like this:
python -c import stdeb; excecfile('setup.py') bdist_deb
See this was my point of why my 'bdist_deb' belonged as a distutils
patch and not in stdeb.
The inherited capabilities, behaviors and invocation style needs to
follow all the other 'bdist' command types.
And that means being able to invoke it the same as say 'bdist_rpm' like:
python setup.py bdist_deb
or
python setup.py bdist_deb --std bdist type arg ...
If we start allowing 'bdist' derivative commands to diverge in their
inherited capabilities, invocation style, and expected behaviors then
chaos will ensue.
This does have the same invocation style -- if you change your setup.py
to import stdeb. Otherwise, it's still the same invocation style -- you
just have to import stdeb outside the setup.py file.
Also, this version of bdist_deb is never going to make it into the
standard library as it depends on stdeb.
So feel free to make a counter proposal, but I'm inclined to think this
is useful to people as-is and has the benefit of working today.
My view is this. The 'bdist' family of commands should be reserved for
pure Distutils implementations not dependent upon setuptools or stdeb.
Putting a command to create the .deb based on my code into stdeb is fine
but that command should not be a 'bdist_xxx' command. It should be
somthing like bstdeb_deb so that a true 'bdist_deb' can still be
developed for Distutils and not be in conflict with some other
'bdist_deb' out there.
At some point, the 'bdist' family of commands should evolve into
OS-specific varieties. And this means that for example, redhat, fedora,
mandriva, or any other rpm-package distro would have their own
'bdist_DISTRO' command that would inherit from 'bdist_rpm' and then
institute their own rules/policies onto a generic 'bdist_rpm'. And the
same would hold true for example a 'bdist_ubuntu' command that would
inherit from a 'bdist_deb' command or in general for any 'bdist_DISTRO'
command that would inherit from a 'bdist_PACKAGETOOL' command.
Maybe we can get some overall guidance here from Tarek who has been
coordinating a lot of these types of things.
Regards,
Gerry
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote:
OK, based on your code I added a bdist_deb distutils command to
stdeb.
I've checked this into the old-stable branch and I'd appreciate it if
you can check whether this works for you and send me any comments.
Invoke it like this:
python -c import stdeb; excecfile('setup.py') bdist_deb
See this was my point of why my 'bdist_deb' belonged as a distutils
patch and not in stdeb.
The inherited capabilities, behaviors and invocation style needs to
follow all the other 'bdist' command types.
And that means being able to invoke it the same as say 'bdist_rpm'
like:
python setup.py bdist_deb
or
python setup.py bdist_deb --std bdist type arg ...
If we start allowing 'bdist' derivative commands to diverge in their
inherited capabilities, invocation style, and expected behaviors then
chaos will ensue.
This does have the same invocation style -- if you change your setup.py
to import stdeb. Otherwise, it's still the same invocation style -- you
just have to import stdeb outside the setup.py file.
Also, this version of bdist_deb is never going to make it into the
standard library as it depends on stdeb.
So feel free to make a counter proposal, but I'm inclined to think this
is useful to people as-is and has the benefit of working today.
My view is this. The 'bdist' family of commands should be reserved
for pure Distutils implementations not dependent upon setuptools or
stdeb. Putting a command to create the .deb based on my code into
stdeb is fine but that command should not be a 'bdist_xxx' command.
It should be somthing like bstdeb_deb so that a true 'bdist_deb' can
still be developed for Distutils and not be in conflict with some
other 'bdist_deb' out there.
Well, it wouldn't be setting a precedent that importing a module would
extend the functionality of the stdlib's distutils commands -- the
setuptools and distribute projects already do this. For example, the
sdist command in setuptools has many differences with that of sdist
in plain distutils. The point is that a user can choose not to import
setuptools and thus not get the different behavior.
The case here is that if the stdlib ever grows bdist_deb, that's fine,
but I don't see why that should prevent stdeb from implementing a
bdist_deb command of its own. In this case, I think the issue is even
less relevant, since there is no stdlib bdist_deb command. Users would
naturally expect bdist_deb to do exactly what stdeb is implementing.
As long as I'm going ahead an implementing the functionality (which I'm
ambivalent about in and of itself for previously stated reasons), I
think it should be named the most obvious name.
My only concern is that namespaces are important and if we end up with
two solution paths that each use the same namespace then this becomes
very confusing for users. Yes, I understand that if you don't import
stdeb in your setup.py then you can use a 'bdist_deb' from Distutils.
At some point, the 'bdist' family of commands should evolve into
OS-specific varieties. And this means that for example, redhat,
fedora, mandriva, or any other rpm-package distro would have their own
'bdist_DISTRO' command that would inherit from 'bdist_rpm' and then
institute their own rules/policies onto a generic 'bdist_rpm'. And
the same would hold true for example a 'bdist_ubuntu' command that
would inherit from a 'bdist_deb' command or in general for any
'bdist_DISTRO' command that would inherit from a 'bdist_PACKAGETOOL'
command.
I don't see what's so specific about ubuntu versus debian right now that
there's any reason for a different set of commands for the two.
Therefore, I think this is a red herring w.r.t. the present discussion.
For ubuntu, at the moment at least, that's true. But distros like
redhat are creating their own rules/policies around rpm naming
conventions, build options, etc. and of course they should be able to do
this and I expect other distros will do the same, and these will diverge
more over time even though they will all use the same packaging tool
underneath.
Maybe we can get some overall guidance here from Tarek who has been
coordinating a lot of these types of things.
I think this is a good topic for someone with a long view about where
things are headed here to weigh in on this.
I'm fine with whatever fits in with the longer term goals.
Regards,
Gerry
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew,
Here's another patch attempt. Is this what you had in mind?
Regards,
Gerry
51a52,53
('ignore-single-version-externally-managed', 's',
'delete option --single-version-externally-managed from install options'),
657a660,665
if ignore_single_version_externally_managed:
self.setup_env_vars = self.setup_env_vars + 'SVEMOPT= '
else:
self.setup_env_vars = self.setup_env_vars + 'SVEMOPT=--single-version-externally-managed '
953c961
--no-compile --single-version-externally-managed \\
---
--no-compile ${SVEMOPT} \\
959c967
--no-compile --single-version-externally-managed --install-layout=deb \\
---
--no-compile ${SVEMOPT} --install-layout=deb \\
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote:
Gerry Reno wrote:
So now everything is working on hardy except for the fact that stdeb 0.3
has to be modified to remove the --single-version-externally-managed
option. So can you just create a 0.3a version with this small change and
that could go out to the repositories and be available by default to
everyone. There does not seem to be any impact that I can see in making
this change and since so many people create their own install classes
this would actually prevent conflicts.
I figured out what the reason for --single-version-externally-managed is
-- it is required for namespace packages to function properly. Thus, I
can't remove it, as doing so would be a regression.
However, if you create a patch I can apply against the old-stable branch
that makes this optional, (e.g. set in a similar way as the
--ignore-install-requires option), I can add that and make release
0.3.1.
Ok, I'll look at getting you a patch that removes the option.
Note also that I'm very close to releasing 0.4, as the 0.3 line
has major problems with Jaunty and later.
Are you planning to submit a patch for the bdist_deb command? I could
include that, too.
Yes, I have something ready but I think the 'bdist_deb' command belongs
as a distutils patch. I created the command by subclassing the
distutils Command class. Maybe Tarek could comment on what might be
best here.
Here is my working 'bdist_deb' code verbatim from inside my setup.py:
(it could be more pythonic of course with less os.system commands but
it's working real well this way at least)
=
# Copyright 2009 Gerry Reno.
# stdeb note:
# stdeb 0.3 needs its util.py modified to remove the install
# option --single-version-externally-managed otherwise
# if you have subclassed the distutils install class to
# create your own install class (rather common) you will
# get an unrecognized option error when using 'bdist_deb'.
# I had to make this change to stdeb in order to get stdeb to work
# on Ubuntu Hardy with a setup.py that defined its own install.
from distutils.core import Command
if sys.hexversion = 0x020500F0:
import stdeb
elif sys.argv[1] == 'bdist_deb':
print sys.argv[1], requires python = 2.5
raise SystemExit(1)
class bdist_deb(Command):
# creates .deb pkg from a .dsc file that is generated using stdeb module
# requires stdeb, dpkg-dev, build-essential, python-all-dev,
debhelper, python-support, fakeroot
#
# to add 'postinst' and friends just create a debian/ control
directory in the home directory of your
# application and add a file 'python-myapp.postinst' and it will be
run post package installation.
description = 'Create debian-style source (.dsc using stdeb module)
and binary (.deb) packages'
user_options = [('install-dir=', 'd', directory to install the
files to),
]
def initialize_options (self):
self.install_dir = None
self.outfiles = []
def finalize_options (self):
pass
def run(self):
import setuptools
import commands
# run a sub_command (gen .dsc source pkg using stdeb)
self.run_command('sdist_dsc')
# save current directory
setup_dir = os.getcwd()
# define system command to execute (find source pkg work tree)
syscmd = ['find', 'deb_dist', '-mindepth', '1', '-maxdepth',
'1', '-type', 'd',]
# execute system command and read output (execute and read
output of find cmd)
dsc_tree = strip(subprocess.Popen(syscmd,
stdout=subprocess.PIPE).communicate()[0]) # strip needed to remove CR
# use system command output (change directory to work tree)
os.chdir(dsc_tree)
# define system command to execute (gen .deb binary pkg)
syscmd = 'dpkg-buildpackage -rfakeroot -uc -us'
# execute system command
if os.system(syscmd) == 0:
print '.deb successfully created in deb_dist/'
else:
print 'problem creating .deb'
# (change directory back to setup dir)
os.chdir(setup_dir)
setup(name = name,
...
# add new bdist_deb command here
cmdclass = {
...
'bdist_deb' : bdist_deb,
...
},
...
)
=
Regards,
Gerry
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Gerry Reno wrote:
Andrew Straw wrote:
Gerry Reno wrote:
So now everything is working on hardy except for the fact that stdeb
0.3
has to be modified to remove the --single-version-externally-managed
option. So can you just create a 0.3a version with this small change
and
that could go out to the repositories and be available by default to
everyone. There does not seem to be any impact that I can see in making
this change and since so many people create their own install classes
this would actually prevent conflicts.
I figured out what the reason for --single-version-externally-managed is
-- it is required for namespace packages to function properly. Thus, I
can't remove it, as doing so would be a regression.
However, if you create a patch I can apply against the old-stable branch
that makes this optional, (e.g. set in a similar way as the
--ignore-install-requires option), I can add that and make release
0.3.1.
Ok, I'll look at getting you a patch that removes the option.
Patch to utils.py in old-stable branch attached.
Note also that I'm very close to releasing 0.4, as the 0.3 line
has major problems with Jaunty and later.
Are you planning to submit a patch for the bdist_deb command? I could
include that, too.
Yes, I have something ready but I think the 'bdist_deb' command
belongs as a distutils patch. I created the command by subclassing the
distutils Command class. Maybe Tarek could comment on what might be
best here.
Here is my working 'bdist_deb' code verbatim from inside my setup.py:
(it could be more pythonic of course with less os.system commands but
it's working real well this way at least)
=
# Copyright 2009 Gerry Reno.
# stdeb note:
# stdeb 0.3 needs its util.py modified to remove the install
# option --single-version-externally-managed otherwise
# if you have subclassed the distutils install class to
# create your own install class (rather common) you will
# get an unrecognized option error when using 'bdist_deb'.
# I had to make this change to stdeb in order to get stdeb to work
# on Ubuntu Hardy with a setup.py that defined its own install.
from distutils.core import Command
if sys.hexversion = 0x020500F0:
import stdeb
elif sys.argv[1] == 'bdist_deb':
print sys.argv[1], requires python = 2.5
raise SystemExit(1)
class bdist_deb(Command):
# creates .deb pkg from a .dsc file that is generated using stdeb module
# requires stdeb, dpkg-dev, build-essential, python-all-dev,
debhelper, python-support, fakeroot
#
# to add 'postinst' and friends just create a debian/ control
directory in the home directory of your
# application and add a file 'python-myapp.postinst' and it will be
run post package installation.
description = 'Create debian-style source (.dsc using stdeb module)
and binary (.deb) packages'
user_options = [ ('install-dir=', 'd', directory to install the files
to),
]
def initialize_options (self):
self.install_dir = None
self.outfiles = []
def finalize_options (self):
pass
def run(self):
import setuptools
import commands
# run a sub_command (gen .dsc source pkg using stdeb)
self.run_command('sdist_dsc')
# save current directory
setup_dir = os.getcwd()
# define system command to execute (find source pkg work tree)
syscmd = ['find', 'deb_dist', '-mindepth', '1', '-maxdepth', '1',
'-type', 'd',]
# execute system command and read output (execute and read output of
find cmd)
dsc_tree = strip(subprocess.Popen(syscmd,
stdout=subprocess.PIPE).communicate()[0]) # strip needed to remove CR
# use system command output (change directory to work tree)
os.chdir(dsc_tree)
# define system command to execute (gen .deb binary pkg)
syscmd = 'dpkg-buildpackage -rfakeroot -uc -us'
# execute system command
if os.system(syscmd) == 0:
print '.deb successfully created in deb_dist/'
else:
print 'problem creating .deb'
# (change directory back to setup dir)
os.chdir(setup_dir)
setup(name = name,
...
# add new bdist_deb command here
cmdclass = {
...
'bdist_deb' : bdist_deb,
...
},
...
)
=
Regards,
Gerry
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
953c953
--no-compile --single-version-externally-managed \\
---
--no-compile \\
959c959
--no-compile --single-version-externally-managed --install-layout=deb \\
---
--no-compile --install-layout=deb \\
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote: Gerry Reno wrote: Ok, here is the simplest way that I got stdeb working on Hardy: $ easy_install stdeb # installs 0.3 copied stdeb egg to workarea and edited util.py and removed all --single-version-externally-managed options repackaged egg copied egg back to site-packages OK, but now your packages built with stdeb going to be (even more) different from standard Debian/Ubuntu Python packages -- they're going to install their files as an .egg (directory or zip file). For example, do dpkg --contents python-my-package_1.0-1.deb and compare that to standard python packages. Ok, I just built a .deb using modified stdeb 0.3 on hardy and looked at the contents and it's like this: ./usr/share/pyshared/myapp/dir1/...(list of files) ./usr/share/pyshared/myapp/dir2/... ./usr/share/pyshared/myapp/dir3/... I don't see anything about egg. And it looks very similar to other .deb in /var/cache that I checked. How much difference would I see if it was egg? Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew, Ok, .deb is building fine using modified stdeb 0.3 on hardy. I've installed the .deb and it works. I can see that stdeb is using debhelper which I really don't know much about. So it looks like debhelper or dh_make is generating the control files. What I need is someway that in myapp maybe under a debian/ directory, I can create postinst and prerm files and then stdeb would look for those in the orig tarball and use them if found. Is this possible? Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote: Gerry Reno wrote: Andrew, Ok, .deb is building fine using modified stdeb 0.3 on hardy. I've installed the .deb and it works. I can see that stdeb is using debhelper which I really don't know much about. So it looks like debhelper or dh_make is generating the control files. What I need is someway that in myapp maybe under a debian/ directory, I can create postinst and prerm files and then stdeb would look for those in the orig tarball and use them if found. Is this possible? The debian packaging machinery invoked by dpkg-buildpackage will pick up any scripts named my-package-name.preinst, my-package-name.postinst, my-package-name.prerm, my-package-name.postrm that happen to be in the debian/ directory. See http://sdn.vlsm.org/share/Debian-Doc/manuals/maint-guide/ch-dother.en.html for more info. So you have to modify stdeb to drop the appropriate files in there if they're given as a config argument to stdeb. The stdeb master branch already creates such a file for housekeeping reasons (see commit http://github.com/astraw/stdeb/commit/9faaf049a1a7fbd8f5e1fb658248013f208f398c where I added this). I really didn't have to use any stdeb config args or anything. I just created a dir myapp/debian and put a python-myapp.postinst script file in there, ran my bdist_deb command and then installed the .deb and the python-myapp.postinst script ran perfectly. So do I need to be concerned with config args to stdeb? I do see some python-myapp.postinst.debhelper etc. scripts in there but I don't know if those ran. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Gerry Reno wrote: I really didn't have to use any stdeb config args or anything. I just created a dir myapp/debian and put a python-myapp.postinst script file in there, ran my bdist_deb command and then installed the .deb and the python-myapp.postinst script ran perfectly. So do I need to be concerned with config args to stdeb? I do see some python-myapp.postinst.debhelper etc. scripts in there but I don't know if those ran. So now everything is working on hardy except for the fact that stdeb 0.3 has to be modified to remove the --single-version-externally-managed option. So can you just create a 0.3a version with this small change and that could go out to the repositories and be available by default to everyone. There does not seem to be any impact that I can see in making this change and since so many people create their own install classes this would actually prevent conflicts. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote: Gerry Reno wrote: In my setup.py I have my own install: from distutils.command.install import install ... class myinstall(install): ... and when we call dpkg-buildpackage this local install seems to maybe be causing the problem with the error: option --single-version-externally-managed not recognized. How can we make this option be recognized? Do we need to inherit something from setuptools? I think there are 2 options: 1) upgrade to the nascent stdeb 0.4. It doesn't call setup.py with the --single-version-externally-managed argument. I'm using Ubuntu 8.0.4 LTS Server and when I run apt-get it brings in stdeb 0.3. I always use the distro packagers to install software so how can I use apt-get to install stdeb 0.4? 2) inherit from setuptools' install command. I don't like that idea. That may mean a lot of rework as it took a while to get our install all debugged and working. I would suggest option 1. If you need me to make a release, I'm happy to do so -- but maybe that will be better after it has bdist_deb? :) My only concern now with stdeb adding the bdist_deb command is that it may not work like other 'bdist' commands. And having 'bdist_deb' work similar to 'bdist_rpm' is important. It would have to use setup.cfg for config and allow for pre/postinstall scripts to run. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
In my setup.py I have my own install:
from distutils.command.install import install
...
class myinstall(install):
...
and when we call dpkg-buildpackage this local install seems to maybe
be causing the problem with the error: option
--single-version-externally-managed not recognized.
How can we make this option be recognized? Do we need to inherit
something from setuptools?
Regards,
Gerry
Gerry Reno wrote:
Ok, I found how to control the python version: XS-Python-Version
So now it is using python2.5 but dpkg-buildpackage hits an error:
error: option --single-version-externally-managed not recognized
What is this option?
I've been searching but I don't find how to solve this.
Is there a way to fix this?
Regards,
Gerry
Gerry Reno wrote:
Ok, I have a setup.py that imports stdeb and creates a 'bdist_deb'
command.
The entire thing is working except for the last subcommand for
dpkg-buildpackage is trying to build for python2.4 and failing.
dpkg-buildpackage fails in the fakeroot...
...
debian/rules build
python2.4 -c import
setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
build
Traceback (most recent call last):
File string, line 1, in ?
File setup.py, line 35, in ?
import stdeb
ImportError: No module named stdeb
make: *** [build-python2.4] Error 1
dpkg-buildpackage: failure: debian/rules build gave error exit status 2
Is this something to do with python versions?
So how do I limit the python versions for dpkg-buildpackage?
stdeb is 0.3
$ python -V
Python 2.5.2
Regards,
Gerry
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote: Gerry Reno wrote: Andrew Straw wrote: Gerry Reno wrote: In my setup.py I have my own install: from distutils.command.install import install ... class myinstall(install): ... and when we call dpkg-buildpackage this local install seems to maybe be causing the problem with the error: option --single-version-externally-managed not recognized. How can we make this option be recognized? Do we need to inherit something from setuptools? I think there are 2 options: 1) upgrade to the nascent stdeb 0.4. It doesn't call setup.py with the --single-version-externally-managed argument. I'm using Ubuntu 8.0.4 LTS Server and when I run apt-get it brings in stdeb 0.3. I always use the distro packagers to install software so how can I use apt-get to install stdeb 0.4? stdeb isn't in the Ubuntu (or Debian) repository. And I thought you said you used easy_install on it. Are you sure you can't just upgrade by hand? I can't see how else you got it. Yes, looking back you are right. I didn't find it with apt-get and had to use easy_install to get it (which I don't normally do). $ sudo easy_install --upgrade stdeb says stdeb 0.3 is best match and already installed. How do I get stdeb 0.4? I would suggest option 1. If you need me to make a release, I'm happy to do so -- but maybe that will be better after it has bdist_deb? :) My only concern now with stdeb adding the bdist_deb command is that it may not work like other 'bdist' commands. And having 'bdist_deb' work similar to 'bdist_rpm' is important. It would have to use setup.cfg for config and allow for pre/postinstall scripts to run. Well, to produce the final .deb file, stdeb produces a source package and then, in another process, calls the debian machinery may be called to produce the .deb file (I guess you're now quite familiar with that). I feel this approach benefits from using the standard Debian infrastructure. Specifically, python-support and debhelper 7 called by stdeb 0.4 do things like: install a single copy of the .py files and symlink them into the per-version directories, compile .pyc files per-version at install time, compile extension modules per-version at build time, and so on. Reproducing that would be hard, error prone, and a pointless duplication of effort. And .debs that don't do all that stuff that will be sub-standard and won't behave like those downloaded from the Debian/Ubuntu repos. Thus, I really think calling the real Debian machinery is the way to go. As I understand it (which is not well), bdist_rpm calls out to an rpm command from the original python setup.py bdist_rpm process, but at that point the binary is already made at this is just a packaging step. With stdeb, the initial output will be the source package which would be compiled to a binary by another process -- a subprocess of dpkg-buildpackage. You can control this binary generation by modifying debian/rules, which is a Makefile that is run to build the binary. So you can make sure that building the binary as done by the debian/rules file handles the options in setup.cfg. It should do that automatically, since what happens is a glorified python setup.py install. As far as pre/postinstall scripts -- what do you need them to do? It would be possible to add support for stdeb to run these scripts at debian .deb install time. (stdeb already runs a preinstall script to ease the python-central - python-support transition. See commit 9faaf049.) But if they're limited in scope and merely modify the files you want installed, they're probably already run as part of the .deb build process if you've hooked into the install command. I can't see how an infrastructure based only on extending the distutils install command could be used to distinguish between A) pre/post-install scripts that merely modify the files to be installed and could thus be run at package build time versus B) pre/post-install scripts that actually need to be run once the .deb package is being installed on the target system that might do something like start a web server. It seems there has to be a specification of the difference. For bdist_rpm the way that you get pre and postpackage install scripts to run during package install is by using the following in the setup.cfg: [bdist_rpm] install_script = rpm_install_sh.txt then rpm_install_sh.txt gets inserted into the rpm .spec file at the %install marker. Now you can also add other sections into rpm_install_sh.txt like %pre and %post which are sections of shell scripts that run prepackage and postpackage install. As far as 'python setup.py install' build type pre/postinstall, you can subclass install class and just run some scripts either side of parent install class. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote: Gerry Reno wrote: How do I get stdeb 0.4? The easiest way: git clone git://github.com/astraw/stdeb.git cd stdeb sudo python setup.py install If you want a tarball, please look at my email 09/19/2009 09:36 PM -0700. Ok, I'll install 0.4 using one of these methods. What should I do about the easy_installed stdeb 0.3? Should I try to remove 0.3 (I use that term loosely wrt easy_install) using -m? For bdist_rpm the way that you get pre and postpackage install scripts to run during package install is by using the following in the setup.cfg: [bdist_rpm] install_script = rpm_install_sh.txt then rpm_install_sh.txt gets inserted into the rpm .spec file at the %install marker. Now you can also add other sections into rpm_install_sh.txt like %pre and %post which are sections of shell scripts that run prepackage and postpackage install. As far as 'python setup.py install' build type pre/postinstall, you can subclass install class and just run some scripts either side of parent install class. I think it would be straightforward to integrate something similar into stdeb. I'm not too much in favor of piggybacking distutils' infrastructure to parse the setup.cfg file, though. Perhaps we could parse setup.cfg on our own (i.e. replace looking for stdeb.cfg with looking for setup.cfg and changing the section name appropriately). In the near term the easiest thing is to stick with the stdeb.cfg file for now and and a new field for each of pre/post install and pre/post remove. My preference would be to see something like this maybe in setup.cfg: [bdist_deb] pre_install_script = deb_pre_install.sh post_install_script = deb_post_install.sh pre_uninstall_script = deb_pre_uninstall.sh post_uninstall_script = deb_post_uninstall.sh This is what I was intending to do using the 'bdist_deb' command extension to distutils. And that way our users have only one config file they need to be concerned about. Regards, Gerry P.S.: Things that totally frustrate me wrt python: distutils/setuptools and apt-get(or rpm)/easy_install battles/incompatibilities/duplications. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Gerry Reno wrote: Andrew Straw wrote: Gerry Reno wrote: How do I get stdeb 0.4? The easiest way: git clone git://github.com/astraw/stdeb.git cd stdeb sudo python setup.py install If you want a tarball, please look at my email 09/19/2009 09:36 PM -0700. Ok, I'll install 0.4 using one of these methods. What should I do about the easy_installed stdeb 0.3? Should I try to remove 0.3 (I use that term loosely wrt easy_install) using -m? Ok, I d/l the tarball which installed 0.3-dh7. But this created new problems: Unmet dependencies: debhelper = 7 python-support = 0.8.7 Installed: debhelper6.0.4 python-support0.7.5 It's going to be tough to get stdeb running on Ubuntu 8.0.4 LTS. Regards, Gerry So I ran apt-get upgrade ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote: Gerry Reno wrote: It's going to be tough to get stdeb running on Ubuntu 8.0.4 LTS. Well, I'd qualify that with stdeb version 0.4 may be more tough to get running. It is the bleeding edge, after all. :) 0.3 works as well as it ever did, but new features are going into 0.4. But now that you mention it, I have backported debhelper and python-support to Hardy. See http://debs.astraw.com/hardy you can download the binaries or source packages from there. (You can also add my repo to your /etc/apt/sources.list) Wow, there's quite a few packages there. Is debhelper and python-support all that are needed or do I need any others because of dependencies? Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote: Gerry Reno wrote: Andrew Straw wrote: Gerry Reno wrote: It's going to be tough to get stdeb running on Ubuntu 8.0.4 LTS. Well, I'd qualify that with stdeb version 0.4 may be more tough to get running. It is the bleeding edge, after all. :) 0.3 works as well as it ever did, but new features are going into 0.4. But now that you mention it, I have backported debhelper and python-support to Hardy. See http://debs.astraw.com/hardy you can download the binaries or source packages from there. (You can also add my repo to your /etc/apt/sources.list) Wow, there's quite a few packages there. Is debhelper and python-support all that are needed or do I need any others because of dependencies? I can't remember -- you'll have to attempt installation to find out. I think there are some but not too many dependencies that need to be upgraded (I guess on the order of 5 packages or so). Ok, here is the simplest way that I got stdeb working on Hardy: $ easy_install stdeb # installs 0.3 copied stdeb egg to workarea and edited util.py and removed all --single-version-externally-managed options repackaged egg copied egg back to site-packages Now dpkg-buildpackage is happy and no more complaints about unrecognized option or missing dependencies. So my question is this: can you make a special version of the current stdeb 0.3 without the --single-version-externally-managed options and put it into your hardy repo? This would be much easier and less worrisome to hardy users than trying to get them to upgrade a lot of different packages. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Ok, I got the __file__ problem solved but now I want to do this whole
deb pkg create as just a single command in my own setup.py. How can I
do this?
I'm thinking something like:
# my setup.py
import stdeb
Command(mycmd):
initialize_option...
finalize_option...
run
sub_commands=[('sdist_dsc', None),]
system('dpgk-source ...)
print .deb is here...
Would this work and is this how to do it?
Regards,
Gerry
Andrew Straw wrote:
Gerry Reno wrote:
sudo easy_install stdeb # brought in stdeb 0.3
$ cd myappdir # where my setup.py is located
$# following http://github.com/astraw/stdeb/ quickstart 1
$ python -c import stdeb; execfile('setup.py') sdist_dsc \
cd `find deb_dist -mindepth 1 -maxdepth 1 -type d` \
dpkg-buildpackage -rfakeroot -uc -us \
cd ../..
echo .deb created successfully in deb_dist/
Traceback (most recent call last):
File string, line 1, in module
File setup.py, line 41, in module
sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)),
bin))
NameError: name '__file__' is not defined
You can use something more elaborate like
python -c import
stdeb,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
sdist_dsc
for the first line.
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote:
Gerry Reno wrote:
Ok, I got the __file__ problem solved but now I want to do this whole
deb pkg create as just a single command in my own setup.py. How can I
do this?
I'm thinking something like:
# my setup.py
import stdeb
Command(mycmd):
initialize_option...
finalize_option...
run
sub_commands=[('sdist_dsc', None),]
system('dpgk-source ...)
print .deb is here...
Would this work and is this how to do it?
Do you really need to run it directly from setup.py?
Yes.
Can't you run it
from a shell script or write a python script making lots of use of
subprocess? The reason I ask is that distutils isn't fun to extend, and
as you're already going to be running on a Debian-based system, I can't
see what's to be gained.
What you want to do is probably possible, but I don't have the
motivation to do it myself. I guess it could allow a bdist_deb option,
but IMO that's not particularly desirable -- the Debian source package
emitted by stdeb can be compiled for any Debian derivative for any
version (provided the dependencies are met)
Having a single command for .deb doesn't preclude stdeb from emitting
source package .dsc.
-- whereas the .deb will
only have defined behavior on the Debian/Ubuntu release on which it was
built.
Well of course, just like any other binary package target.
I don't see where there is any conflict.
Regards,
Gerry
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Andrew Straw wrote: Gerry Reno wrote: What you want to do is probably possible, but I don't have the motivation to do it myself. I guess it could allow a bdist_deb option, but IMO that's not particularly desirable -- the Debian source package emitted by stdeb can be compiled for any Debian derivative for any version (provided the dependencies are met) Having a single command for .deb doesn't preclude stdeb from emitting source package .dsc. OK, if you're sure... :) I'd suggest attempting to add a bdist_deb command to stdeb's stdeb/command/ directory for use with distutils. Then register it with entry_points in the setup.py file. Note that you wouldn't want bdist_deb to inherit the binary generation stuff from bdist_dumb or friends -- you'd specifically want to call out the the appropriate debian command (dpkg-buildpackage). I'd be happy to include such a thing with stdeb, though I'd probably not use it much myself -- I like having a single source package that I just recompile for the different distributions. Well, I think it would be better if it were part of stdeb. And I like calling it 'bdist_deb'. That makes sense. And it should emit the .dsc as well as the .deb. My use case is providing consumers of my apps with a way to gen their own source and binary packages without them needing to know anything about the mechanics of the process. I've already got good feedback on this with 'bdist_rpm' and if we had 'bdist_deb' then that would add a good deal more capability. For bdist_rpm we drive it from setup.cfg and would be good if bdist_deb configuration behavior would be the same. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Ok, I have a setup.py that imports stdeb and creates a 'bdist_deb' command.
The entire thing is working except for the last subcommand for
dpkg-buildpackage is trying to build for python2.4 and failing.
dpkg-buildpackage fails in the fakeroot...
...
debian/rules build
python2.4 -c import
setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
build
Traceback (most recent call last):
File string, line 1, in ?
File setup.py, line 35, in ?
import stdeb
ImportError: No module named stdeb
make: *** [build-python2.4] Error 1
dpkg-buildpackage: failure: debian/rules build gave error exit status 2
Is this something to do with python versions?
So how do I limit the python versions for dpkg-buildpackage?
stdeb is 0.3
$ python -V
Python 2.5.2
Regards,
Gerry
Gerry Reno wrote:
Ok, I got the __file__ problem solved but now I want to do this whole
deb pkg create as just a single command in my own setup.py. How can I
do this?
I'm thinking something like:
# my setup.py
import stdeb
Command(mycmd):
initialize_option...
finalize_option...
run
sub_commands=[('sdist_dsc', None),]
system('dpgk-source ...)
print .deb is here...
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] stdeb-0.3 error
Ok, I found how to control the python version: XS-Python-Version
So now it is using python2.5 but dpkg-buildpackage hits an error:
error: option --single-version-externally-managed not recognized
What is this option?
I've been searching but I don't find how to solve this.
Is there a way to fix this?
Regards,
Gerry
Gerry Reno wrote:
Ok, I have a setup.py that imports stdeb and creates a 'bdist_deb'
command.
The entire thing is working except for the last subcommand for
dpkg-buildpackage is trying to build for python2.4 and failing.
dpkg-buildpackage fails in the fakeroot...
...
debian/rules build
python2.4 -c import
setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})
build
Traceback (most recent call last):
File string, line 1, in ?
File setup.py, line 35, in ?
import stdeb
ImportError: No module named stdeb
make: *** [build-python2.4] Error 1
dpkg-buildpackage: failure: debian/rules build gave error exit status 2
Is this something to do with python versions?
So how do I limit the python versions for dpkg-buildpackage?
stdeb is 0.3
$ python -V
Python 2.5.2
Regards,
Gerry
Gerry Reno wrote:
Ok, I got the __file__ problem solved but now I want to do this whole
deb pkg create as just a single command in my own setup.py. How can I
do this?
I'm thinking something like:
# my setup.py
import stdeb
Command(mycmd):
initialize_option...
finalize_option...
run
sub_commands=[('sdist_dsc', None),]
system('dpgk-source ...)
print .deb is here...
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] stdeb-0.3 error
sudo easy_install stdeb # brought in stdeb 0.3
$ cd myappdir # where my setup.py is located
$# following http://github.com/astraw/stdeb/ quickstart 1
$ python -c import stdeb; execfile('setup.py') sdist_dsc \
cd `find deb_dist -mindepth 1 -maxdepth 1 -type d` \
dpkg-buildpackage -rfakeroot -uc -us \
cd ../..
echo .deb created successfully in deb_dist/
Traceback (most recent call last):
File string, line 1, in module
File setup.py, line 41, in module
sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)),
bin))
NameError: name '__file__' is not defined
=
The error is coming from the setup.py line:
import imp
import sys
import os
import glob
from distutils.core import setup, Command
from distutils.command.install import install
if os.name == 'nt':
import py2exe
sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)),
bin))
===
I do not get this error when I run setup.py normally for other commands
like 'sdist'.
How can I fix this?
Regards,
Gerry
___
Distutils-SIG maillist - Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Packaging: bdist_deb patches
Tarek Ziadé wrote: Hi Gerry, During the summit at Pycon, we have said that it would be a better strategy not to include within Distutils os-specific tools for various reasons (and also to remove existing ones) : - it's better for them to have their own release cycles - it's hard for me, as distutils maintainer, to maintain and make evolve os-specific tools. People that are specialists on those OS will do a better job. Now the problems to reach that goal are: - some people are reluctant not to have everything included in the stdlib (I am not). I don't think we can all agree on this, and since Guido have encouraged this strategy during the summit, I guess I'm more inclined to follow it nevertheless. - we need to detect for each existing command (rpm, etc) a project that can take over, prior to remove it from Distutils when appliable . Now, for debian, since it's not included in distutils, the question is : what is the most used/advanced project ? sdteb ? ++ Tarek Tarek, I understand your position on this. However, here is what needs considered if you want to split out this functionality. 'bdist_rpm' operates rather well and you can automate the generation of packages using this target. What I would hope not to see is that we end up with an entire fragmentation of different approaches to developing similar targets for other platforms such as targets like 'bdist_deb', 'bdist_mac', etc. Whatever group would be selected to take over this functionality should take over responsibility for all these targets and they be developed in a coordinated manner so that it would be easily understood by the user how to generated packages for multiple platforms using the targets and similar settings in setup.cfg and any auxiliary files similar to rpm_install_sh.txt. If a group can be selected to handle these all these targets with the above goals in mind then I think it would be possible to split this functionality out of the core. But, if the cannot be done, then I think this functionality should remain with the core in order to avoid fragmentation to the approaches in developing these 'bdist_xxx' targets. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Packaging: bdist_deb patches
Gerry Reno wrote: Tarek, Speaking of packaging patches. Would there be any chance that you could get the 'bdist_deb' patches integrated in Distutils prior to PyCon? Hi Tarek, Has there been any progress on getting the 'bdist_deb' target (akin to 'bdist_rpm') implemented in Distutils? We are looking for behavior that would be similar to bdist_rpm driven by setup.cfg with fully automated generation of packages. There is in existence a set of patches that supposedly does this. Has any progress been made to integrate those patches? Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Distutils: distclass cmdclass
Hi all, I've run into a situation where I need to have translations using L10nAppDistribution and we need to also have a special command class. What I'm finding is that I can have one or the other but not both. Does anyone know how to be able to use both distclass and cmdclass at the same time? Regards, Gerry Reno ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Distutils: distclass cmdclass
Gerry Reno wrote: Hi all, I've run into a situation where I need to have translations using L10nAppDistribution and we need to also have a special command class. What I'm finding is that I can have one or the other but not both. Does anyone know how to be able to use both distclass and cmdclass at the same time? Right now the only way I've found to do this is to execv to another setup file. Please let me know if this is possible with just one setup(). Thanks. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOLVED: bdist_rpm and pre-release python packages / eggs (was: pre-release versioning problems with sdist, bdist_rpm, bdist_debian)
Manuel, Here's what I get with this patch: $ python test2.py rewrite 3.3beta1 4333 3.3beta1-4333 3.3-0.4333.beta1 $ python test2.py rewrite 3.3rc1 4333 3.3rc1-4333 3.3-0.4333.rc1 It is REWRITING the strings when we don't want them rewritten. And that was the point about putting this in an extension called by a commandline arg. Instead of going through all these callesthenics writing obscure code in bdist_rpm.py just write your simpler code into an override module and call it via a commandline option. That way people who work on bdist_rpm in the future don't have to decipher all this Fedora-specific code. Why do you resist doing it this way? Why do you feel you have the right to come in and take over bdist_rpm with Fedora-specific modifications? Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOLVED: bdist_rpm and pre-release python packages / eggs (was: pre-release versioning problems with sdist, bdist_rpm, bdist_debian)
Manuel, 'bdist_rpm' is NOT broken. What is broken is packagers misuse of the 'version' and 'release' strings. They do stupid things like put version='3.0' and release='rc1' and then wonder why their final release cannot update the release candidate. THIS IS A TRAINING ISSUE. Their poorly planned sequence of release designations is not in lexical order. They should be putting version='3.0' and release='0.rc1' or '0_rc1' and then their final release of version='3.0' and release='1' WILL update their release candidate. AGAIN, this is not something that is broken with 'bdist_rpm' IT IS A TRAINING ISSUE. Packagers need to understand the importance of lexical ordering when using RPM. You cannot make a patch that outlaws stupidity. I vote a big -1 on this patch because it interjects Fedora-specific modifications into bdist_rpm. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOLVED: bdist_rpm and pre-release python packages / eggs (was: pre-release versioning problems with sdist, bdist_rpm, bdist_debian)
Manuel Amador (Rudd-O) wrote: 1) The Python StrictVersion and LooseVersion classes in Setuptools used to validate and compare versions are incompatible with RPM's lexicographical order. Even the setuptools metadata documentation, which is in full force and implemented, says alpha and beta and rc are expressly permitted (and ENCOURAGED) in the version name, and setuptools' dependency resolution actually uses that information. Then this is a 'setuptools' ERROR. Go fix it in setuptools. If setuptools creates lexigraphically stupid version and release strings that are incompatible with RPM then that is a 'setuptools' problem, NOT a bdist_rpm problem. Go fix 'setuptools'. Again, I'm a big -1 on this patch. There is nothing broken in 'bdist_rpm'. But there is obviously something broken in 'setuptools'. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOLVED: bdist_rpm and pre-release python packages / eggs (was: pre-release versioning problems with sdist, bdist_rpm, bdist_debian)
Manuel Amador (Rudd-O) wrote: El Miércoles 11 Marzo 2009, Gerry Reno escribió: Ok, Mandriva comes to mind. It's policy is different and allows more flexibility than Fedora's. Well, would you be so kind to link the Mandriva policy for us to read about it? Mandriva link: http://wiki.mandriva.com/en/Policies/RpmSpecProposal You're failing to keep in mind that Fedora is not the only distro out there that is RPM-based. Mandriva is just one of many other RPM-based distros that have packaging policies different from Fedora. Again, the core software should not enforce one distros packaging policies. If you want to perform some type of check for a particular distro then that should be done through an extension module and there should be a special command line option to trigger it --enforce-fedora-packaging-policy. Besides, policies are meant for humans. And policies change. We don't need distro policies hard-coded into Distutils. That makes for brittle code. So what you're saying, in effect, translates to sorry, man, you cannot build your Plone packages for RPM distributions, nor can anybody else build any sort of pre-release eggs either. Well, sorry, man, that's not acceptable. Why do you keep saying this?What is preventing you (the human) from filling in the version and release fields with fedora-compliant strings? The core software does not have to know anything about fedora. But you do. Why? A distro's policy is defined for 'human' behavior for users of that distro. Users of distro X should not be hostage to policies of distro Y. 'version' and 'release' should be formatted by humans for whatever distro is the target. There should be no enforcement of any distro's 'packaging policy' over the fields. Policy schmolicy, the current distutils behavior builds pre-release RPMs that UPGRADE final release packages, which is broken and stupid because distutils ought to be taking care of the impedance mismatch between python-style versioning and RPM-style lexicographical versioning, but it doesn't. That's a lexical ordering problem. That should be fixed by the human making sure that he declares the version and release with proper strings. I have a solution that works in fedora, rhel and centos, and likely works just as well on other RPM distros including Mandriva and SUSE. Your solution LIMITS the version and release strings to ONLY fedora packaging style.Mandriva users don't want that. They want to build RPM according to their own packaging policy for Mandriva but your patch will not let them because it enforces fedora policy.THAT'S WRONG. Do you have an alternative solution? Yes, I do. Remove all artificial restriction on formatting on the 'version' and 'release' strings. All that's necessary is for both the 'version' and 'release' strings to be available to all the distribution command which it is not at the moment. That's it. Nothing else is necessary. If you want to do some policy enforcing thing, then put it in some type of optional extension called from a special command line option. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOLVED: bdist_rpm and pre-release python packages / eggs (was: pre-release versioning problems with sdist, bdist_rpm, bdist_debian)
Manuel Amador (Rudd-O) wrote:
Mandriva link: http://wiki.mandriva.com/en/Policies/RpmSpecProposal
Thanks for providing the link. Which is a PROPOSAL, not a polcy. On a
user-editable WIKI. And on top of that, what you say about versioning
style is in NO WAY backed by that document.
That is the ONLY policy there is for Mandriva. So, if you like, then
you can say the policy is whatever 'version' and 'release' naming that
will work in an RPM spec file. And that's way more flexible than
Fedora's policy.
And Mandriva is just one example. You have other distros that do not
use Fedora naming policy and you have all sorts of derivatives of these
distros, none of which are using Fedora packaging policy.
Now we know for a fact that my patch is neither in conflict with
Mandriva policy, nor will fail to work there. This is the kind of
fact-based exchange I enjoy.
It does conflict. See above.
Again, the core software should not enforce one distros packaging
policies.
We already established that the Mandriva policy is not in conflict
with Mandriva policy or any other policy that you have quoted so far
(none, really), so the prescriptive argument you are making has no
basis in fact -- It's just dogma.
Again, your patch makes changes to 'version' and 'release' based on
Fedora naming policy. Not every distro follows Fedora package naming
policy.
Why do you keep saying this? What is preventing you (the human) from
filling in the version and release fields with fedora-compliant
strings? The core software does not have to know anything about
fedora. But you do.
If you need to know: cheese shop eggs come using a particular python
policy for naming them, a policy that was defined by setuptools'
dependency handling. This python policy is incompatible with RPM
lexicographical order. This is, in a nutshell, the source of the problem.
If that's the problem then fix it where the problem lies. In the egg
naming policy in setuptools.
Therefore, the bdist_rpm patch needs to adapt the version to an RPM
lexicographically valid form. My patches do exactly this -- they
merely adapt the bdist_rpm version and release in very specific cases
(pre-release packages) which ought not to matter for a distributor
making a stable release. In addition to that, my code works across
distributions and is not in contradiction to any policy.
Manuel, please put this policy stuff in an extension that will override
the core. Not everybody uses eggs or cares about eggs. Or what
problems egg people have with RPM.
Without my patches, there needs to be a way to override the version of
the package in setup.cfg, and in addition to that, each package in the
cheese shop would need to be modified in order to be buildable. That's
thousands of eggs.
Again, put this logic in an extension so it doesn't affect the core and
call it from a commandline option.
You're invited to talk to all the python egg developers yourself if
you disagree with me. Me? I prefer practical solutions, thus the patch.
That's a lexical ordering problem. That should be fixed by the human
making sure that he declares the version and release with proper
strings.
Well, I'm sorry but the version is not overridable in setup.cfg. So
you either use my patch, or chao eggs.
Again, put the patch logic in an extension called from a commandline
option and leave the core alone.
I have a solution that works in fedora, rhel and centos, and likely
works just as well on other RPM distros including Mandriva and SUSE.
Your solution LIMITS the version and release strings to ONLY fedora
packaging style. Mandriva users don't want that.
FYI: There is nothing in the Mandriva policy supporting this statement.
Yes, there is. Mandriva naming is whatever will work in a spec file.
Much less restrictive than Fedora.
They want to build RPM according to their own packaging policy for
Mandriva but your patch will not let them because it enforces fedora
policy. THAT'S WRONG.
Again, the policy link you gave provides no basis for your argument.
See above.
Do you have an alternative solution?
Yes, I do. Remove all artificial restriction on formatting on the
'version' and 'release' strings.
I have not introduced any artificial restrictions at all. This is just
a lie. I'd appreciate it very much if, in the future, you refrain from
telling lies about my work.
Manuel, here is part of your patch:
+version = self.distribution.get_version()
+release = self.release.replace('-','_')
+import re ; regex = '[^0-9\.]'
+splits = re.split(regex,version,maxsplit=1)
+firstnonalnumchar = re.findall(regex,version)
+if len(splits) == 1:
+ pass
+else:
+ version = splits[0]
+ release = 0. + release + . + firstnonalnumchar[0] +
splits[1]
spec_file = [
'%define name ' + self.distribution.get_name(),
Re: [Distutils] SOLVED: bdist_rpm and pre-release python packages / eggs (was: pre-release versioning problems with sdist, bdist_rpm, bdist_debian)
Manuel, In RPM the only restriction on 'version' is that it must be made of alphanumeric or period. Ref: http://www.rpm.org/max-rpm-snapshot/s1-rpm-inside-tags.html Just for a small test I ran a perfectly legal version pattern, that is allowable in distros other than Fedora, through your patch logic. version = A2 release = 0_rc1 version-release = A2-0_rc1 # this is what we expected to see version-release = -0.0_rc1.A2 # this is what we get instead as strings are modified by patch logic This is why I suggest that you make an extension module triggered by a commandline option so there are no restrictions or unexpected side-effects introduced to non-Fedora distro users. Right now, I'm a big -1 on the patch.If it would be made as an extension triggered by commandline option then I would support it. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOLVED: bdist_rpm and pre-release python packages / eggs (was: pre-release versioning problems with sdist, bdist_rpm, bdist_debian)
Manuel Amador (Rudd-O) wrote: Hello, guys, I have fixed distutils (and setuptools remains working) with the attached patch. Now, RPMs will be built according to the Fedora Package Naming Guidelines: http://fedoraproject.org/wiki/Packaging/NamingGuidelines#Non-Numeric_Version_in_Release which I understand to be the most useful reference in terms of naming pre-release packages. This should work correctly in at least: - Fedora - RHEL - SUSE I urge you patch your python 2.4s and 2.5s and 2.6s and push this update to distributions. I have done that myself at my own repository. Now we can enjoy one more reason to build RPMs (and eggs! ... according to my workbench at http://yum.rudd-o.com/SCRIPTS/ -- feel free to pick its brains) DIRECTLY from the cheese shop, especially if you're using pip. Oh, I also have pip at my repo (cd ../RPMS/noarch in my workbench). See attached patch. I will log bugs where it corresponds too. -- Manuel Amador (Rudd-O) rud...@rudd-o.com Rudd-O.com - http://rudd-o.com/ GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/ Now playing, courtesy of Amarok: Aqua - Cartoon heroes Windows 95 is not a virus. Viruses actually do something. Hi Manuel, You worked on my problem! Great. So today what we have been doing to deal with the pre-release and lexical ordering problem involving pre-releases is this: We impose a restriction on how the pre-release is identified. So for example if you intend to end up with a final version-release of 5.0.0-1 and you want to first put out some betas or release candidates then we have to name them as, 5.0.0-0_beta1, or 5.0.0-0_rc1 and this is so that the lexical ordering for RPM will be correct. In other words you must put the pre-release designation into the 'release' part of VERSION-RELEASE. What we had seen developers doing previously was to name these as 5.0.0_beta1 or 5.0.0_rc1 (making the pre-release designation part of the 'version' string) which then did not work for the lexical ordering of the final release of 5.0.0-1 because 5.0.0 (version) was not lexically superior to 5.0.0_rc1. So we were able to solve this problem without any code changes to distutils. But this also presented a challenge for the other distribution targets such as 'sdist' because they were totally unaware of this 'version-release' combination and only knew about 'version'. So as a workaround we were doing this: # WORKAROUND # define both version AND release version='5.0.0' release='1' # combine them for all targets except 'bdist_rpm' if sys.argv[1] != 'bdist_rpm': version = version+'-'+release So this wasn't perfect but it actually worked quite well and we could get 'sdist' to work properly in conjunction with 'bdist_rpm'. So now with your patch all the targets should be able to set and use both 'version' and 'release' and we don't need our workaround and that will be great. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Packaging: bdist_deb patches
Tarek, Speaking of packaging patches. Would there be any chance that you could get the 'bdist_deb' patches integrated in Distutils prior to PyCon? Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOLVED: bdist_rpm and pre-release python packages / eggs (was: pre-release versioning problems with sdist, bdist_rpm, bdist_debian)
Manuel Amador (Rudd-O) wrote: My solution tackles it from another direction, using observed egg packaging practices. This is what I have observed in setup.pys, and eggs' requires.txts so far: 1.2dev: refers to unspecified not-even-alpha quality checkout or source drop 1.2dev-r5667: refers to the checkout of revision 5667, not-even-alpha 1.2a1: refers to Alpha 1 of intended version 1.2 1.2a2: refers to Alpha 2 of intended version 1.2 1.2b1: refers to Beta 1 of intended version 1.2 1.2: refers to official 1.2 release Note NO UNDERSCORES. In order to satisfy the fedora package naming guidelines and RPM's lexicographical sort, this is what happens within my patches: 1.2dev: version 1.2 release 0.0.0dev REGARDLESS of user-specified release 1.2dev-r56: version 1.2 release 0.0.56devr REGARDLESS of user-specified rel 1.2a1: version 1.2 release 0.user-specified release.a1 1.2a2: version 1.2 release 0.user-specified release.a2 1.2b1: version 1.2 release 0.user-specified release.b1 (note in the last three it is the responsibility of the builder to bump the release manually, there is really nothing we can do about it) 1.2: version 1.2 release user-specified release In this manner we can accurately map existing practice (what people are actually DOING at least in a significant amount of eggs -- the plone ones) in python egg versioning numbers to RPM lexicographical sort in a manner that lets package managers do the right thing. Also, we need a standard way of trolling the requires.txt for the egg dependencies and transforming them into RPM dependencies, otherwise your RPMs won't work in concert. Code for that in my workbench yum.rudd-o.com subdir SCRIPTS, it is documented step by step, but this should really be integrated within the setuptools bdist_rpm kludge. Really. I am exhausted. Hi Manuel, Let me share with you what we had implemented and the POLICY that we put in place with regards to VERSION-RELEASE strings. We did not look at this from egg standpoint as we just distribute RPMS. I sent this to Tarek a while ago: What I had implemented seems to have the VERSION-RELEASE string issue solved using a 'workaround' and a manual policy for the moment. It would of course be better if distutils just did this automatically but this is at least working. Tarballs, rpms have the same designations and all the lexical ordering is correct so that even for pre-release versioning the rpms update each other correctly. #WORKAROUND: What I did was in setup.py added: version = '5.0.0' release = '0_rc3_1681' if sys.argv[1] != 'bdist_rpm': version = version + '-' + release In setup.cfg added: [bdist_rpm] # release must exactly match 'release' as set in setup.py release=0_rc3_1681 == # END WORKAROUND And then I setup the following policy regarding VERSION-RELEASE naming: (this example uses bzr as source code control system but you can substitute svn, rcs, or any other type of sccs) Development code: version=5.0.0 release=0_00012345(where 00012345 is the revision number from bzr in ZERO-PADDED FIELD OF 8) tarball: foo-5.0.0-0_00012345.tar.gz rpm: foo-5.0.0-0_00012345.noarch.rpm Alpha code: version=5.0.0 release=0_alpha1_00123456 tarball: foo-5.0.0-0_alpha1_00123456.tar.gz rpm: foo-5.0.0-0_alpha1_00123456.noarch.rpm Beta code: version=5.0.0 release=0_beta1_00234567 tarball: foo-5.0.0-0_beta1_00234567.tar.gz rpm: foo-5.0.0-0_beta1_00234567.noarch.rpm Release Candidate code: version=5.0.0 release=0_rc1 tarball: foo-5.0.0-0_rc1.tar.gz rpm: foo-5.0.0-0_rc1.noarch.rpm Release Candidate code fix: version=5.0.0 release=0_rc1_00345678 tarball: foo-5.0.0-0_rc1_00345678.tar.gz rpm: foo-5.0.0-0_rc1_00345678.noarch.rpm Final Release code: version=5.0.0 release=1 tarball: foo-5.0.0-1.tar.gz rpm: foo-5.0.0-1.noarch.rpm Notice that lexical ordering is proper in all cases. Even where the alpha, beta, rc releases may be followed by a bzr fix revision. The ensures that both tarballs and packages such as rpm will contain the same VERSION-RELEASE designations. Also, the use of UNDERSCORES in the 'release' string assures that there is no ambiguity with regard to parsing VERSION-RELEASE string combination or full package names which have hyphens in specific positions. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOLVED: bdist_rpm and pre-release python packages / eggs (was: pre-release versioning problems with sdist, bdist_rpm, bdist_debian)
Manuel Amador (Rudd-O) wrote: Gerry, What you have said is interesting. Note that you're using underscores in the release number (against F packaging policy), and the general scheme for devising the release number is also against Fedora packaging policy too (which is highly likely to cause problems in RHEL down the road too). Sure, it works too, but it'd be good if we could stick to the published policies which are also known to work and everybody adheres to them already. Hi Manuel, Let me share with you what we had implemented and the POLICY that we put in place with regards to VERSION-RELEASE strings. We did not look at this from egg standpoint as we just distribute RPMS. I sent this to Tarek a while ago: What I had implemented seems to have the VERSION-RELEASE string issue solved using a 'workaround' and a manual policy for the moment. It would of course be better if distutils just did this automatically but this is at least working. Tarballs, rpms have the same designations and all the lexical ordering is correct so that even for pre-release versioning the rpms update each other correctly. #WORKAROUND: What I did was in setup.py added: version = '5.0.0' release = '0_rc3_1681' if sys.argv[1] != 'bdist_rpm': version = version + '-' + release In setup.cfg added: [bdist_rpm] # release must exactly match 'release' as set in setup.py release=0_rc3_1681 == # END WORKAROUND And then I setup the following policy regarding VERSION-RELEASE naming: (this example uses bzr as source code control system but you can substitute svn, rcs, or any other type of sccs) Development code: version=5.0.0 release=0_00012345 (where 00012345 is the revision number from bzr in ZERO-PADDED FIELD OF 8) tarball: foo-5.0.0-0_00012345.tar.gz rpm: foo-5.0.0-0_00012345.noarch.rpm Alpha code: version=5.0.0 release=0_alpha1_00123456 tarball: foo-5.0.0-0_alpha1_00123456.tar.gz rpm: foo-5.0.0-0_alpha1_00123456.noarch.rpm Beta code: version=5.0.0 release=0_beta1_00234567 tarball: foo-5.0.0-0_beta1_00234567.tar.gz rpm: foo-5.0.0-0_beta1_00234567.noarch.rpm Release Candidate code: version=5.0.0 release=0_rc1 tarball: foo-5.0.0-0_rc1.tar.gz rpm: foo-5.0.0-0_rc1.noarch.rpm Release Candidate code fix: version=5.0.0 release=0_rc1_00345678 tarball: foo-5.0.0-0_rc1_00345678.tar.gz rpm: foo-5.0.0-0_rc1_00345678.noarch.rpm Final Release code: version=5.0.0 release=1 tarball: foo-5.0.0-1.tar.gz rpm: foo-5.0.0-1.noarch.rpm Notice that lexical ordering is proper in all cases. Even where the alpha, beta, rc releases may be followed by a bzr fix revision. The ensures that both tarballs and packages such as rpm will contain the same VERSION-RELEASE designations. Also, the use of UNDERSCORES in the 'release' string assures that there is no ambiguity with regard to parsing VERSION-RELEASE string combination or full package names which have hyphens in specific positions. Regards, Gerry -- Manuel Amador (Rudd-O) rud...@rudd-o.com Rudd-O.com - http://rudd-o.com/ GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/ Reality always seems harsher in the early morning. Manuel, Are you on the FPP committee? I looked at the fedora packaging policy and it is rather similar except it has periods where we have underscores. It would be nice if fedora could expand the policy to allow the underscore style as well. Both approaches work just as well. And to me the underscore form is a lot more readable. Also, there isn't a lot of examples in the FPP regarding adding the vcs revision number into the release string. I like the way we do it because it's lexically correct always. Maybe this could be added to FPP? Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOLVED: bdist_rpm and pre-release python packages / eggs (was: pre-release versioning problems with sdist, bdist_rpm, bdist_debian)
Gerry Reno wrote: Manuel Amador (Rudd-O) wrote: Gerry, What you have said is interesting. Note that you're using underscores in the release number (against F packaging policy), and the general scheme for devising the release number is also against Fedora packaging policy too (which is highly likely to cause problems in RHEL down the road too). Sure, it works too, but it'd be good if we could stick to the published policies which are also known to work and everybody adheres to them already. Hi Manuel, Let me share with you what we had implemented and the POLICY that we put in place with regards to VERSION-RELEASE strings. We did not look at this from egg standpoint as we just distribute RPMS. I sent this to Tarek a while ago: What I had implemented seems to have the VERSION-RELEASE string issue solved using a 'workaround' and a manual policy for the moment. It would of course be better if distutils just did this automatically but this is at least working. Tarballs, rpms have the same designations and all the lexical ordering is correct so that even for pre-release versioning the rpms update each other correctly. #WORKAROUND: What I did was in setup.py added: version = '5.0.0' release = '0_rc3_1681' if sys.argv[1] != 'bdist_rpm': version = version + '-' + release In setup.cfg added: [bdist_rpm] # release must exactly match 'release' as set in setup.py release=0_rc3_1681 == # END WORKAROUND And then I setup the following policy regarding VERSION-RELEASE naming: (this example uses bzr as source code control system but you can substitute svn, rcs, or any other type of sccs) Development code: version=5.0.0 release=0_00012345 (where 00012345 is the revision number from bzr in ZERO-PADDED FIELD OF 8) tarball: foo-5.0.0-0_00012345.tar.gz rpm: foo-5.0.0-0_00012345.noarch.rpm Alpha code: version=5.0.0 release=0_alpha1_00123456 tarball: foo-5.0.0-0_alpha1_00123456.tar.gz rpm: foo-5.0.0-0_alpha1_00123456.noarch.rpm Beta code: version=5.0.0 release=0_beta1_00234567 tarball: foo-5.0.0-0_beta1_00234567.tar.gz rpm: foo-5.0.0-0_beta1_00234567.noarch.rpm Release Candidate code: version=5.0.0 release=0_rc1 tarball: foo-5.0.0-0_rc1.tar.gz rpm: foo-5.0.0-0_rc1.noarch.rpm Release Candidate code fix: version=5.0.0 release=0_rc1_00345678 tarball: foo-5.0.0-0_rc1_00345678.tar.gz rpm: foo-5.0.0-0_rc1_00345678.noarch.rpm Final Release code: version=5.0.0 release=1 tarball: foo-5.0.0-1.tar.gz rpm: foo-5.0.0-1.noarch.rpm Notice that lexical ordering is proper in all cases. Even where the alpha, beta, rc releases may be followed by a bzr fix revision. The ensures that both tarballs and packages such as rpm will contain the same VERSION-RELEASE designations. Also, the use of UNDERSCORES in the 'release' string assures that there is no ambiguity with regard to parsing VERSION-RELEASE string combination or full package names which have hyphens in specific positions. Regards, Gerry -- Manuel Amador (Rudd-O) rud...@rudd-o.com Rudd-O.com - http://rudd-o.com/ GPG key ID 0xC8D28B92 at http://wwwkeys.pgp.net/ Reality always seems harsher in the early morning. Manuel, Are you on the FPP committee? I looked at the fedora packaging policy and it is rather similar except it has periods where we have underscores. It would be nice if fedora could expand the policy to allow the underscore style as well. Both approaches work just as well. And to me the underscore form is a lot more readable. Also, there isn't a lot of examples in the FPP regarding adding the vcs revision number into the release string. I like the way we do it because it's lexically correct always. Maybe this could be added to FPP? Manuel, Also examples in the fedora packaging policy don't even make sense. It's no wonder no one every gets this right. alsa-lib-0.9.2-0.1.beta1 (add a new patch on top of 0.9.2beta1) alsa-lib-0.9.2-0.2.beta1 (upgrade to 0.9.2beta2) alsa-lib-0.9.2-0.3.beta2 (upgrade to 0.9.2beta3) ^^And this ridiculous construction is from the Department of Redundancy Department This would be far better and make a lot more sense like: alsa-lib-0.9.2-0_beta1_1727 # where 1727 is the vcs revision, using numbers like patch 1 is absolutely meanless to most people, they want to know what tagged release (beta1 tag) and then what revision contains the update patches. OR alsa-lib-0.9.2-0_beta1_p0 # where p0 means patch 0 Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOLVED: bdist_rpm and pre-release python packages / eggs (was: pre-release versioning problems with sdist, bdist_rpm, bdist_debian)
Manuel, Additionally, from a Python Distutils perspective, Distutils should not be enforcing one distro's policy. There are RPM-based distro's that DO NOT follow fedora's packaging policy. The software should only be concerned with providing for the use of both the 'version' and 'release' strings in all distribution targets. It should not concern itself with HOW those fields are formatted. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Packaging: bdist_deb patches
Zooko, While we're waiting for the 'official' answer from Tarek... I don't think the code is being taken from stdeb. There were 'bdist_deb' patches that were developed/tested/contributed a while ago and just never got into distutils. To the best of my knowledge it is that patch set. Of course let's see what Tarek says but I'm pretty sure that's the set. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] SOLVED: bdist_rpm and pre-release python packages / eggs (was: pre-release versioning problems with sdist, bdist_rpm, bdist_debian)
Manuel Amador (Rudd-O) wrote: El Miércoles 11 Marzo 2009, Gerry Reno escribió: Manuel, Additionally, from a Python Distutils perspective, Distutils should not be enforcing one distro's policy. There are RPM-based distro's that DO NOT follow fedora's packaging policy. Well then, give me one packaging policy that is discordant or incompatible with Fedora's packaging policy. I'm sure you can show us that. Ok, Mandriva comes to mind. It's policy is different and allows more flexibility than Fedora's. Besides, policies are meant for humans. And policies change. We don't need distro policies hard-coded into Distutils. That makes for brittle code. Besides, we're not enforcing a distro policy here over all platforms. My patches merely reformat the existing information to make it lexicographically compatible with RPM package managers (which are the SAME across all bdist_rpm distro targets), and that's it. You still get to specify your release numbers if you want to, and the version numbers are intelligently determined based on heuristics but in the case of stable releases, they are just as they used to be before. The software should only be concerned with providing for the use of both the 'version' and 'release' strings in all distribution targets. It should not concern itself with HOW those fields are formatted. If distutils did only that and we were to reject the distutils patches I wrote, then I would simply be technically unable to build Plone for Fedora or RHEL. Why? A distro's policy is defined for 'human' behavior for users of that distro. Users of distro X should not be hostage to policies of distro Y. 'version' and 'release' should be formatted by humans for whatever distro is the target. There should be no enforcement of any distro's 'packaging policy' over the fields. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python Packaging Survey
Tarek Ziadé wrote: Hello, Thanks to all the people that helped building the survey, and a special thanks to Massimo Di Pierro who created the application that runs the Survey and helped me set it up. Please, take it ! http://tinyurl.com/package-survey Regards Tarek Tarek, I took the survey but I forgot to add this: Need to add 'bdist_deb' patches into Distutils. Also, can 'bdist_deb' be added as an extension to existing Distutils installations? That would be great. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python Packaging Survey
Paul Moore wrote: 2009/3/9 Tarek Ziadé ziade.ta...@gmail.com: Thanks to all the people that helped building the survey, and a special thanks to Massimo Di Pierro who created the application that runs the Survey and helped me set it up. Please, take it ! http://tinyurl.com/package-survey I get a page load error - something about an expired self-signed certificate (this with Firefox on Windows). Paul ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig Paul, I got that too but I just clicked by all the warnings and the survey works fine after that. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Python Packaging Survey
Tarek Ziadé wrote: On Mon, Mar 9, 2009 at 9:19 PM, Paul Moore p.f.mo...@gmail.com wrote: Massimo has enabled the http version, you can reach it with : http://tinyurl.com/packaging-survey Tarek, In the survey there were some questions that were all or nothing. For one I answered Pylons but we also have projects using TG2 so I flipped a coin basically. Don't know how many others might have similar circumstance or whether it makes any difference. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] bdist_rpm new option
Tarek Ziadé wrote: Hello We have worked on a fix for bdist_rpm, to avoid the error that occurs under Fedora or RedHat when the optimize flag is not activated. http://bugs.python.org/issue1533164 I am about to introduce a new option for this command, called force-optimize (and activated by default) Hopefully this will eliminate the current need to create separate install-scripts with lines like this which we are having to do right now: python setup.py install --optimize 1 --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] bdist_rpm: spec file: %config
I've been looking at the spec file that 'bdist_rpm' generates and I do not see a %config or %config(noreplace) section. Is there a way to generate a %config section and specify a list of files? This is important so that config files do not get overwritten when using 'rpm -U'. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pre-release versioning problems with sdist, bdist_rpm, bdist_debian
Tarek Ziadé wrote: On Fri, Jan 9, 2009 at 2:12 AM, Gerry Reno gr...@verizon.net wrote: The problem is that there appears to be no way to get a coordinated behavior between 'sdist' and 'bdist_rpm' as far as version and release strings are concerned that will satisfy both tarballs and bdist packages. If I understand your problem correclty, if sdist would simply concatenate the version string and the release string to use it as a source version when it starts to work, you would be able to work things out ? I don't know the internals of 'sdist' but I think if there were a way to extend 'sdist' to use 'release' as well as 'version' then that might work. I would have to test that to see. Regards. Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pre-release versioning problems with sdist, bdist_rpm, bdist_debian
Tarek Ziadé wrote: On Fri, Jan 9, 2009 at 3:51 PM, Gerry Reno gr...@verizon.net wrote: Tarek Ziadé wrote: On Fri, Jan 9, 2009 at 2:12 AM, Gerry Reno gr...@verizon.net wrote: The problem is that there appears to be no way to get a coordinated behavior between 'sdist' and 'bdist_rpm' as far as version and release strings are concerned that will satisfy both tarballs and bdist packages. If I understand your problem correclty, if sdist would simply concatenate the version string and the release string to use it as a source version when it starts to work, you would be able to work things out ? I don't know the internals of 'sdist' but I think if there were a way to extend 'sdist' to use 'release' as well as 'version' then that might work. I would have to test that to see. Well, can you define how sdist should behave exactly ? Based on that discussion I can make a prototype for you to try out, then we can maybe propose in that mailing list a change to sdist Thanks Tarek. I think if it would do the same thing as bdist_rpm that it would be ok. bdist_rpm looks like it does VERSION-RELEASE (hyphen separator).So then doing this for 'sdist' I guess would produce a tarball name of foo-VERSION-RELEASE.tar.gz and an extracted directory of foo-VERSION-RELEASE. What this would allow then is for the 'version' string to stay at '5.0.0' and then the 'release' string to contain any pre-release information such as '0_rc1' and then the final release would contain '1' which is lexically superior to the '0_rc1'. I'm not sure though what other targets in distutils also use 'version' so I don't know if this would affect anything else.Also, I'm hoping this can be implemented as some kind of extension so that it can be made to work for existing installations as well. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pre-release versioning problems with sdist, bdist_rpm, bdist_debian
Gerry Reno wrote: Tarek Ziadé wrote: On Fri, Jan 9, 2009 at 3:51 PM, Gerry Reno gr...@verizon.net wrote: Tarek Ziadé wrote: On Fri, Jan 9, 2009 at 2:12 AM, Gerry Reno gr...@verizon.net wrote: The problem is that there appears to be no way to get a coordinated behavior between 'sdist' and 'bdist_rpm' as far as version and release strings are concerned that will satisfy both tarballs and bdist packages. If I understand your problem correclty, if sdist would simply concatenate the version string and the release string to use it as a source version when it starts to work, you would be able to work things out ? I don't know the internals of 'sdist' but I think if there were a way to extend 'sdist' to use 'release' as well as 'version' then that might work. I would have to test that to see. Well, can you define how sdist should behave exactly ? Based on that discussion I can make a prototype for you to try out, then we can maybe propose in that mailing list a change to sdist Thanks Tarek. I think if it would do the same thing as bdist_rpm that it would be ok. bdist_rpm looks like it does VERSION-RELEASE (hyphen separator).So then doing this for 'sdist' I guess would produce a tarball name of foo-VERSION-RELEASE.tar.gz and an extracted directory of foo-VERSION-RELEASE. What this would allow then is for the 'version' string to stay at '5.0.0' and then the 'release' string to contain any pre-release information such as '0_rc1' and then the final release would contain '1' which is lexically superior to the '0_rc1'. I'm not sure though what other targets in distutils also use 'version' so I don't know if this would affect anything else. Updating my comment: Yes, and all the 'bdist' targets would have to do the same type of thing as 'bdist_rpm'. That is use the combination of VERSION-RELEASE. Also, I'm hoping this can be implemented as some kind of extension so that it can be made to work for existing installations as well. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] pre-release versioning problems with sdist, bdist_rpm, bdist_debian
Tarek Ziadé wrote: On Fri, Jan 9, 2009 at 4:42 PM, Gerry Reno gr...@verizon.net wrote: Thanks Tarek. I think if it would do the same thing as bdist_rpm that it would be ok. bdist_rpm looks like it does VERSION-RELEASE (hyphen separator).So then doing this for 'sdist' I guess would produce a tarball name of foo-VERSION-RELEASE.tar.gz and an extracted directory of foo-VERSION-RELEASE. What this would allow then is for the 'version' string to stay at '5.0.0' and then the 'release' string to contain any pre-release information such as '0_rc1' and then the final release would contain '1' which is lexically superior to the '0_rc1'. I'm not sure though what other targets in distutils also use 'version' so I don't know if this would affect anything else. Updating my comment: Yes, and all the 'bdist' targets would have to do the same type of thing as 'bdist_rpm'. That is use the combination of VERSION-RELEASE. Also, I'm hoping this can be implemented as some kind of extension so that it can be made to work for existing installations as well. In other words, introduce it globally. That is a big change, Yes, but it solves a big problem. I've seen this issue brought up on the fedora-devel mailing list several times. And Toshio has commented about it. The RPM-based distros appear to ignore 'bdist_rpm' and the distro packagers go about creating their own way of building python RPMS. And when I inquired about this, the pre-release versioning issue was the first thing that was mentioned. I think this was the showstopper that caused them to abandon 'bdist_rpm' because they couldn't find sanity between 'sdist' and 'bdist' command treatment of 'version' and 'release' strings. If this gets fixed now, I think they could be persuaded to consider using 'bdist_rpm' for building python packages which would bring a lot of consistency to the process between distros. (yes, I know distros all want to store things in different areas, but that's just a config/preference thing). I think this could stay compatible with the previous installations as long as : - if the release string is not specified, then it is not used at all, (unlike version which becomes 0.0 when not specified) - the Package-Version-Release string is OK with the tools out there (I have to double-check on how setuptools and zc.buildout works on fragments to extract version numbers for instance) That sounds like it will work. I'm looking forward to any prototype you could generate for this and I'll be glad to help test it. Regards, Gerry ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] pre-release versioning problems with sdist, bdist_rpm, bdist_debian
When I create a set of distro packages for a specific distro, I need to distribute both a tarball and a distro-specific package such as an RPM or a deb archive. Some people like installing things from tarballs, others like using the distro package manager. So I let them have it both ways. We can generate the tarball using 'sdist' and generate the distro-package with 'bdist_rpm', etc. This all works great as long as we're talking final releases here: 4.9.0, 5.0.0. But, whenever you start to introduce pre-release candidate versioning, eg: 5.0.0_rc2, then things unravel. What I have been attempting to do, and is apparently is like trying to find the holy grail, is to be able to generate both tarballs and distro-packages for both pre-release candidates and final releases, that exhibit the following behavior: tarballs:foo-5.0.0_rc1.tar.gz (version=5.0.0_rc1); (extracts a foo-5.0.0_rc1 directory; 'sdist' requires version set like this so tarball name is unique and extract dir is unique) distro-package (pre-release):foo-5.0.0-0_rc1.noarch.rpm (version=5.0.0 and release=0_rc1) distro-package (final release): foo-5.0.0-1.noarch.rpm (version=5.0.0 and release=1); 'bdist_rpm' requires version and release set like this so final release will update the pre-release candidate package The problem is that there appears to be no way to get a coordinated behavior between 'sdist' and 'bdist_rpm' as far as version and release strings are concerned that will satisfy both tarballs and bdist packages. 'sdist' knows nothing about 'release' string so you are forced to put the pre-release candidate information into the 'version' string' but as soon as you do that then you break 'bdist' packaging. And to complicate matters, the world seems to have settled for making pre-release candidate version strings like '5.0.0_rc1' for many applications. For example: We have an application, foo-5.0.0, and we want to put out some pre-release candidates for testing, so we set the version to 5.0.0_rc1 in setup.py. We create the source distribution with: $ python setup.py sdist which creates a source archive, foo-5.0.0_rc1.tar.gz. We extract this archive and 'cd' into the foo-5.0.0_rc1 directory and create a built distribution with: $ python setup.py bdist_rpm which creates source and binary RPMS in the form: foo-5.0.0_rc1-1.noarch.rpm. So we think everything is fine. Everyone installs and tests using the pre-release candidate and subsequent candidates but when you eventually get to the final release, foo-5.0.0, and build your final release RPMS, foo-5.0.0-1.noarch.rpm, you find that it will not update your last pre-release candidate RPM, foo-5.0.0_rcX-1.noarch.rpm because it is not rpm newer. I'm hoping there is some distutils guru that can tell me how to solve this enigma or show me how to extend distutils to accomplish what is needed. Regards, Gerry PS: In case anyone notices, yes, I posted a similar question yesterday on the python list without much response, but the question really belongs on this list. ___ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
