Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 10:16:28AM -0400, Tres Seaver wrote: > On 09/20/2013 09:11 AM, Sebastien Douche wrote: > > On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton wrote: > >> I would love an option to ignore VCS and let me specify what I want > >> *explicitly*. > > > > Or remove the handling of VCS? > > -1. I haven't yet tried Marius' tool, but manually maintaining > MANIFEST.in is an unaceptable bug magnet. Heh. I wrote check-manifest because relying on setuptools's handling of VCS was an unacceptable bug magnet to me. (New svn working tree format _or_ you forget to install setuptools-git on this one machine? No error, just a broken sdist.) BTW if you use zest.releaser, check-manifest plugs into it and stops the release (well, asks if you REALLY want to continue, defaulting to "No") if your MANIFEST.in doesn't match your VCS. Marius Gedminas -- It was the kind of night on which bad novels began. -- 1634: The Galileo Affair by Eric Flint and Andrew Dennis signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 8:15 AM, Jim Fulton wrote: > It appears that MANIFEST.in is ignored if setuptools recognizes your > VCS. Not in setuptools 0.6 it isn't. However, it's really only useful for *adding* files not picked up by revision control; it can't *remove* files found by revision control. Manifest and revision control support are semi-independent, as it's quite possible to have files you don't put in revision control but which nonetheless need to be included in an sdist. (For example, you might want to have Cython/Pyrex-generated C code, or compiled documentation files.) ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 9/20/2013 10:16 AM, Tres Seaver wrote: > On 09/20/2013 09:11 AM, Sebastien Douche wrote: >> On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton >> wrote: >>> Did setuptools recently learn about git? > >> Nop. > > I use setuptools-git regularly without problems. > > >>> I would love an option to ignore VCS and let me specify what I >>> want *explicitly*. > >> Or remove the handling of VCS? > > -1. I haven't yet tried Marius' tool, but manually maintaining > MANIFEST.in is an unaceptable bug magnet. I like having MANIFEST.in correct in case I need to build in a system that's not a checkout. But I agree it's a bug magnet. What I'd really like is a mode that throws an error if MANIFEST.in doesn't agree with the VCS detection. - -- Eric. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.13 (Cygwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJSPF/qAAoJENxauZFcKtNx0KAIAKNRzH3yPw13TIAPBmsIs/Vt sJT0ilbxwY8DV9fEJ5stC5P2QiZVmAxP2UNvYqZPTjC9qwvMS6UdMNznJAZBgRd7 cRESzpF1Sf7PZKKNvrIlbRfYZMZP89lGjD21UT/qbFclSZGkormxLVk3kDnDzpSe eOAM7gfWWBi/69eAuT1G5pJbIt+2KVZd4AYh27NXmu8F5RtB/EDgSiMuPvIwPP6r SrTa+1maqs6Ik5o768ED+GoXkGmGMvaUIr1GWZLasA7Q36oM1m9pX9xZ87RIA/aw n+ZcqlWM6tjnevP3SXAjOKNi+fIY1t2I2ECfn6BtY+JXbrlAUAWnbO/MsIoR+NA= =Lorj -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 4:16 PM, Tres Seaver wrote: > I use setuptools-git regularly without problems. Good to know. >> Or remove the handling of VCS? > > - -1. I haven't yet tried Marius' tool, but manually maintaining > MANIFEST.in is an unaceptable bug magnet. $ cat MANIFEST.in recursive-include docs/source/ * recursive-include src/ * prune docs/build/ Not very complicated. -- Sebastien Douche Twitter: @sdouche / G+: +sdouche ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 09/20/2013 09:11 AM, Sebastien Douche wrote: > On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton wrote: >> Did setuptools recently learn about git? > > Nop. I use setuptools-git regularly without problems. >> I would love an option to ignore VCS and let me specify what I want >> *explicitly*. > > Or remove the handling of VCS? - -1. I haven't yet tried Marius' tool, but manually maintaining MANIFEST.in is an unaceptable bug magnet. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlI8WLwACgkQ+gerLs4ltQ4SIgCgh2dRMexciWxnaJBFrU700PDl dOkAoNUNELNFj9Ee0T+mD6ajC3FvQ8xV =FAe6 -END PGP SIGNATURE- ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 2:15 PM, Jim Fulton wrote: > Did setuptools recently learn about git? Nop. > I would love an option to ignore VCS and let me specify what I > want *explicitly*. Or remove the handling of VCS? -- Sebastien Douche Twitter: @sdouche / G+: +sdouche ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 08:15:43AM -0400, Jim Fulton wrote: > On Fri, Sep 20, 2013 at 6:41 AM, Marius Gedminas wrote: > > On Fri, Sep 20, 2013 at 04:43:23PM +1000, Nick Coghlan wrote: > >> On 20 September 2013 00:02, Benjamin Root wrote: > >> > I will point out that even with setuptools 1.1.6, sdist isn't picking up > >> > all > >> > the files in version control, as I have a few other files under version > >> > control in my package that I didn't list for package_data. So, I still > >> > think > >> > there is an issue with crawling an SVN 1.7 repository. > >> > >> That part I have no idea about :) > > > > I would recommend not relying on setuptools version control magic. > > Instead write a MANIFEST.in. > > It appears that MANIFEST.in is ignored if setuptools recognizes your > VCS. Although there's so much magic in this particular facet of > setuptools that I never have much confidence in my understanding. This is why check-manifest runs 'python setup.py sdist' in a clean exported directory with no VCS metadata for verification. > Did setuptools recently learn about git? For a while, an advantage of > git was that setuptools ignored it. I don't believe the setuptools-git plugin was ever merged into the core. > I would love an option to ignore VCS and let me specify what I > want *explicitly*. Marius Gedminas -- "If it ain't broke, don't fix it." - Bert Lantz signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 6:41 AM, Marius Gedminas wrote: > On Fri, Sep 20, 2013 at 04:43:23PM +1000, Nick Coghlan wrote: >> On 20 September 2013 00:02, Benjamin Root wrote: >> > I will point out that even with setuptools 1.1.6, sdist isn't picking up >> > all >> > the files in version control, as I have a few other files under version >> > control in my package that I didn't list for package_data. So, I still >> > think >> > there is an issue with crawling an SVN 1.7 repository. >> >> That part I have no idea about :) > > I would recommend not relying on setuptools version control magic. > Instead write a MANIFEST.in. It appears that MANIFEST.in is ignored if setuptools recognizes your VCS. Although there's so much magic in this particular facet of setuptools that I never have much confidence in my understanding. Did setuptools recently learn about git? For a while, an advantage of git was that setuptools ignored it. I would love an option to ignore VCS and let me specify what I want *explicitly*. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Fri, Sep 20, 2013 at 04:43:23PM +1000, Nick Coghlan wrote: > On 20 September 2013 00:02, Benjamin Root wrote: > > I will point out that even with setuptools 1.1.6, sdist isn't picking up all > > the files in version control, as I have a few other files under version > > control in my package that I didn't list for package_data. So, I still think > > there is an issue with crawling an SVN 1.7 repository. > > That part I have no idea about :) I would recommend not relying on setuptools version control magic. Instead write a MANIFEST.in. There are tools that make this process somewhat less painful: https://pypi.python.org/pypi/check-manifest Regards, Marius Gedminas -- "Nobody will ever need more than 640k RAM!" -- Bill Gates, 1981 "Windows 95 needs at least 8 MB RAM." -- Bill Gates, 1996 "Nobody will ever need Windows 95." -- logical conclusion signature.asc Description: Digital signature ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On 20 September 2013 00:02, Benjamin Root wrote: > So, on a hunch, I tried making sure I had the latest version of setuptools > installed (mine existing install was a little over a year old). On one > system, easy_install found "setuptools 1.1.6" and installed it. After that, > by setup.py script behaved exactly as expected on my demo package I mocked > up to test out the bug. No need for "include_package_data", or MANIFEST.in > or version control, just package_data. Great! > > To double-check, I tried it on another system (same version of python), and > easy_install found "setuptools 1.1.6" as well and installed it. But the bug > demo I made didn't work. Using "pydoc setuptools", it seems to have been > pointing to distribute 0.6.28 for some reason. I retried "easy_install -U > setuptools", and this time it found "distribute 0.7.3". After installing, > "pydoc setuptools" now says that it is "setuptools 1.1.6". Why it didn't say > that before, I haven't a clue. But the bug example package I made then > worked. Don't know what to make of all that, but in the end, package_data > did work as expected. For a variety of reasons, versions of distribute prior to 0.7 actively interfered with the usage of regular setuptools. With distribute 0.7+ (which was released after the two projects merged back together under the original setuptools name), it's just an empty shell that depends on the underlying setuptools. So that's probably what you're seeing - the distribute version was old, and preventing use of the updated setuptools. Once distribute was updated as well, the new setuptools started to be used as expected. > I will point out that even with setuptools 1.1.6, sdist isn't picking up all > the files in version control, as I have a few other files under version > control in my package that I didn't list for package_data. So, I still think > there is an issue with crawling an SVN 1.7 repository. That part I have no idea about :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Thu, Sep 19, 2013 at 12:13 AM, Benjamin Root wrote: > > On Sep 18, 2013 11:37 PM, "PJ Eby" wrote: > > > > On Wed, Sep 18, 2013 at 5:07 PM, Benjamin Root > wrote: > > > In creating a source distribution, I have found a disparity between the > > > behaviors of distutils and setuptools with respect to package_data. As > of > > > python Issue 2279: http://bugs.python.org/issue2279, entries listed in > > > package_data are used when building an sdist. I have verified that this > > > works in a simple example when setup() is imported from distutils.core. > > > However, if I import setup() from setuptools, it does not pull in the > data > > > as listed in package_data (and presumedly data_files). > > > > You need to use the include_package_data = True flag. > > > > I did and it didn't make a difference. Plus, shouldn't it have picked up > everything that I had version controlled, anyway? Does setuptools recognize > the new svn 1.7 format? Even svn 1.6 clients will refuse to do an 'svn > status' on a 1.7 repo checkout. So, I think there might be multiple bugs in > play. > > I also don't see why I should have to use that keyword argument if > distutils does fine without it. sdist isnt -- from my perspective -- a > setuptools specific feature, so I had no expectation of there being such an > egregious difference. > > > > > > P.S. - on a related note, for a package "foo", foo.egg-info directory > is > > > created with a SOURCES.txt file. I have found that under certain > situations, > > > it seems that a successful install would result in a fully listed > > > SOURCES.txt, and then subsequent calls to sdist seems to use that > > > information to build seemingly correct archives. A co-working > > > double-checking a deployment process I made did an sdist and created a > > > source distribution without the package_data when he did a fresh > checkout, > > > but whenever I did it from my development branch, the source > distribution > > > worked fine. I haven't figured out exactly how this came about, but it > seems > > > to be tied to the SOURCES.txt file. > > > > SOURCES.txt mostly exists so that you can safely build an sdist from > > an sdist, as is required by e.g. bdist_rpm, without having any > > revision control data on hand to guide the process. Setuptools also > > can insert a possibly-modified setup.cfg into an sdist for the same > > reason, so that if you used revision control tags to specify the > > version when building the sdist, any sdists rebuilt from that sdist > > will have the same version tags. > > Yes, I understand what sources.txt is for. The issue is that it seems that > it is possible for different build commands to produce different source.txt > results. Since I haven't figured out how I managed to do that, it isn't the > focus of my bug report. > > Cheers! > Ben Root > Stranger and stranger still... So, on a hunch, I tried making sure I had the latest version of setuptools installed (mine existing install was a little over a year old). On one system, easy_install found "setuptools 1.1.6" and installed it. After that, by setup.py script behaved exactly as expected on my demo package I mocked up to test out the bug. No need for "include_package_data", or MANIFEST.in or version control, just package_data. Great! To double-check, I tried it on another system (same version of python), and easy_install found "setuptools 1.1.6" as well and installed it. But the bug demo I made didn't work. Using "pydoc setuptools", it seems to have been pointing to distribute 0.6.28 for some reason. I retried "easy_install -U setuptools", and this time it found "distribute 0.7.3". After installing, "pydoc setuptools" now says that it is "setuptools 1.1.6". Why it didn't say that before, I haven't a clue. But the bug example package I made then worked. Don't know what to make of all that, but in the end, package_data did work as expected. I will point out that even with setuptools 1.1.6, sdist isn't picking up all the files in version control, as I have a few other files under version control in my package that I didn't list for package_data. So, I still think there is an issue with crawling an SVN 1.7 repository. Cheers! Ben Root ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Sep 18, 2013 11:37 PM, "PJ Eby" wrote: > > On Wed, Sep 18, 2013 at 5:07 PM, Benjamin Root wrote: > > In creating a source distribution, I have found a disparity between the > > behaviors of distutils and setuptools with respect to package_data. As of > > python Issue 2279: http://bugs.python.org/issue2279, entries listed in > > package_data are used when building an sdist. I have verified that this > > works in a simple example when setup() is imported from distutils.core. > > However, if I import setup() from setuptools, it does not pull in the data > > as listed in package_data (and presumedly data_files). > > You need to use the include_package_data = True flag. > I did and it didn't make a difference. Plus, shouldn't it have picked up everything that I had version controlled, anyway? Does setuptools recognize the new svn 1.7 format? Even svn 1.6 clients will refuse to do an 'svn status' on a 1.7 repo checkout. So, I think there might be multiple bugs in play. I also don't see why I should have to use that keyword argument if distutils does fine without it. sdist isnt -- from my perspective -- a setuptools specific feature, so I had no expectation of there being such an egregious difference. > > > P.S. - on a related note, for a package "foo", foo.egg-info directory is > > created with a SOURCES.txt file. I have found that under certain situations, > > it seems that a successful install would result in a fully listed > > SOURCES.txt, and then subsequent calls to sdist seems to use that > > information to build seemingly correct archives. A co-working > > double-checking a deployment process I made did an sdist and created a > > source distribution without the package_data when he did a fresh checkout, > > but whenever I did it from my development branch, the source distribution > > worked fine. I haven't figured out exactly how this came about, but it seems > > to be tied to the SOURCES.txt file. > > SOURCES.txt mostly exists so that you can safely build an sdist from > an sdist, as is required by e.g. bdist_rpm, without having any > revision control data on hand to guide the process. Setuptools also > can insert a possibly-modified setup.cfg into an sdist for the same > reason, so that if you used revision control tags to specify the > version when building the sdist, any sdists rebuilt from that sdist > will have the same version tags. Yes, I understand what sources.txt is for. The issue is that it seems that it is possible for different build commands to produce different source.txt results. Since I haven't figured out how I managed to do that, it isn't the focus of my bug report. Cheers! Ben Root ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] Deviation between distutils and setuptools over package_data
On Wed, Sep 18, 2013 at 5:07 PM, Benjamin Root wrote: > In creating a source distribution, I have found a disparity between the > behaviors of distutils and setuptools with respect to package_data. As of > python Issue 2279: http://bugs.python.org/issue2279, entries listed in > package_data are used when building an sdist. I have verified that this > works in a simple example when setup() is imported from distutils.core. > However, if I import setup() from setuptools, it does not pull in the data > as listed in package_data (and presumedly data_files). You need to use the include_package_data = True flag. > P.S. - on a related note, for a package "foo", foo.egg-info directory is > created with a SOURCES.txt file. I have found that under certain situations, > it seems that a successful install would result in a fully listed > SOURCES.txt, and then subsequent calls to sdist seems to use that > information to build seemingly correct archives. A co-working > double-checking a deployment process I made did an sdist and created a > source distribution without the package_data when he did a fresh checkout, > but whenever I did it from my development branch, the source distribution > worked fine. I haven't figured out exactly how this came about, but it seems > to be tied to the SOURCES.txt file. SOURCES.txt mostly exists so that you can safely build an sdist from an sdist, as is required by e.g. bdist_rpm, without having any revision control data on hand to guide the process. Setuptools also can insert a possibly-modified setup.cfg into an sdist for the same reason, so that if you used revision control tags to specify the version when building the sdist, any sdists rebuilt from that sdist will have the same version tags. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] Deviation between distutils and setuptools over package_data
In creating a source distribution, I have found a disparity between the behaviors of distutils and setuptools with respect to package_data. As of python Issue 2279: http://bugs.python.org/issue2279, entries listed in package_data are used when building an sdist. I have verified that this works in a simple example when setup() is imported from distutils.core. However, if I import setup() from setuptools, it does not pull in the data as listed in package_data (and presumedly data_files). I see that the documented behavior for setuptools is to do some searching to see what files are in source control, but I think that is broken too (maybe an issue with the new SVN 1.7?), as the files I was trying to include are under version control. In any case, this deviation in behavior is very confusing as I am often lead to believe that setuptools builds upon distutils, so I am often using the distutils documentation as they are fairly complete. I would suggest incorporating both distutils' and setuptools' behavior in this regard. Thanks, Ben Root P.S. - on a related note, for a package "foo", foo.egg-info directory is created with a SOURCES.txt file. I have found that under certain situations, it seems that a successful install would result in a fully listed SOURCES.txt, and then subsequent calls to sdist seems to use that information to build seemingly correct archives. A co-working double-checking a deployment process I made did an sdist and created a source distribution without the package_data when he did a fresh checkout, but whenever I did it from my development branch, the source distribution worked fine. I haven't figured out exactly how this came about, but it seems to be tied to the SOURCES.txt file. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
