[issue14171] warnings from valgrind about openssl as used by CPython
Zooko O'Whielacronx zo...@zooko.com added the comment:
Oh, I'm sorry I didn't make that clear at first. First of all, so that others
who encounter these warnings can see how I worked-around them so that they can
do that as well. Second, because Python comes with a valgrind suppressions
file. Here is a patch to that file to suppress these warnings.
--- a/Misc/valgrind-python.supp Wed Feb 22 00:28:46 2012 +0100
+++ b/Misc/valgrind-python.supp Fri Mar 02 20:31:55 2012 -0700
@@ -286,6 +286,38 @@
### fun:MD5_Update
###}
+# Fedora's package openssl-1.0.1-0.1.beta2.fc17.x86_64 on x86_64
+# See http://bugs.python.org/issue14171
+{
+ openssl 1.0.1 prng 1
+ Memcheck:Cond
+ fun:bcmp
+ fun:fips_get_entropy
+ fun:FIPS_drbg_instantiate
+ fun:RAND_init_fips
+ fun:OPENSSL_init_library
+ fun:SSL_library_init
+ fun:init_hashlib
+}
+
+{
+ openssl 1.0.1 prng 2
+ Memcheck:Cond
+ fun:fips_get_entropy
+ fun:FIPS_drbg_instantiate
+ fun:RAND_init_fips
+ fun:OPENSSL_init_library
+ fun:SSL_library_init
+ fun:init_hashlib
+}
+
+{
+ openssl 1.0.1 prng 3
+ Memcheck:Value8
+ fun:_x86_64_AES_encrypt_compact
+ fun:AES_encrypt
+}
+
#
# All of these problems come from using test_socket_ssl
#
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14171
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14171] warnings from valgrind about openssl as used by CPython
New submission from Zooko O'Whielacronx zo...@zooko.com:
The buildbot for the Tahoe-LAFS and pycryptopp projects runs CPython under
valgrind on Fedora, and valgrind emits warnings like this:
==30127== Conditional jump or move depends on uninitialised value(s)
==30127== at 0x4C2AD01: bcmp (mc_replace_strmem.c:889)
==30127== by 0xC1D1646: fips_get_entropy (fips_drbg_lib.c:166)
==30127== by 0xC1D1D6E: FIPS_drbg_instantiate (fips_drbg_lib.c:234)
==30127== by 0xC15F590: RAND_init_fips (rand_lib.c:286)
==30127== by 0xC0F54D3: OPENSSL_init_library (o_init.c:106)
==30127== by 0xBE76AF8: SSL_library_init (ssl_algs.c:68)
==30127== by 0xBC2B39D: init_hashlib (in
/usr/lib64/python2.7/lib-dynload/_hashlib.so)
==30127== by 0x4F1DB00: _PyImport_LoadDynamicModule (in
/usr/lib64/libpython2.7.so.1.0)
You can see the full output from such a buildbot run here:
https://tahoe-lafs.org/buildbot-pycryptopp/builders/Ruben%20Fedora%20syslib/builds/58/steps/test%20valgrind/logs/valgrind
Here is information about the versions of software involved:
https://tahoe-lafs.org/buildbot-pycryptopp/builders/Ruben%20Fedora%20syslib/builds/58/steps/show-tool-versions/logs/stdio
The owner of the buildslave machine says that the openssl package was
openssl-1.0.1-0.1.beta2.fc17.x86_64.
Not having looked closer, I assume this is just a case of openssl using
uninitialized memory as part of the initialization of the PRNG. Accordingly, I
wrote suppressions stanzas for our valgrind suppressions file, which made the
warnings go away.
Here are the suppression expressions:
# generated on buildbot.rubenkerkhof.com, which had, according to Ruben
# Fedora's package openssl-1.0.1-0.1.beta2.fc17.x86_64
{
buildbot.rubenkerkhof.com cond fips openssl 1
Memcheck:Cond
fun:bcmp
fun:fips_get_entropy
fun:FIPS_drbg_instantiate
fun:RAND_init_fips
fun:OPENSSL_init_library
fun:SSL_library_init
fun:init_hashlib
}
{
buildbot.rubenkerkhof.com cond fips openssl 2
Memcheck:Cond
fun:fips_get_entropy
fun:FIPS_drbg_instantiate
fun:RAND_init_fips
fun:OPENSSL_init_library
fun:SSL_library_init
fun:init_hashlib
}
{
buildbot.rubenkerkhof.com val _x86_64_AES_encrypt_compact
Memcheck:Value8
fun:_x86_64_AES_encrypt_compact
fun:AES_encrypt
}
I opened this ticket on launchpad.net to track the handling of this
issue in various projects such as openssl, pycryptopp, CPython,
valgrind, and Fedora:
https://bugs.launchpad.net/pycryptopp/+bug/944585
--
components: Library (Lib)
files: cpython-openssl101.supp
messages: 154742
nosy: zooko
priority: normal
severity: normal
status: open
title: warnings from valgrind about openssl as used by CPython
versions: Python 2.7
Added file: http://bugs.python.org/file24701/cpython-openssl101.supp
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14171
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3871] cross and native build of python for mingw32 with packaging
Changes by Zooko O'Whielacronx zo...@zooko.com: -- nosy: -zooko ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue3871 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7546] msvc9compiler.py: add .asm extension
Zooko O'Whielacronx zo...@zooko.com added the comment: I got a bug report from a user that they encountered this error: http://tahoe-lafs.org/pipermail/tahoe-dev/2011-April/006312.html Then a follow-up in which they say they applied the patch from http://bugs.python.org/issue8597 (this ticket is the superceder of that one) and got a different error: http://tahoe-lafs.org/pipermail/tahoe-dev/2011-April/006316.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7546 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7546] msvc9compiler.py: add .asm extension
Zooko O'Whielacronx zo...@zooko.com added the comment: Thanks, Stefan Krah. I posted your comment to the tahoe-dev mailing list: http://tahoe-lafs.org/pipermail/tahoe-dev/2011-May/006336.html Also Samuel Neves has posted on that thread. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7546 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1322] platform.dist() has unpredictable result under Linux
Zooko O'Whielacronx zo...@zooko.com added the comment: There seems to be some mistake, re #msg134219 and #msg134255. The current version of may patch *does* avoid the cost of a subprocess in the common case. I described this new strategy in #msg73744 and as far as I know it satisfies all of MAL's earlier objection about that. To recap, this code here: http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/src/allmydata/__init__.py?annotate=blamerev=5033#L36 does the following strategy: 1. Parse the /etc/lsb-release file. /etc/lsb-release is not part of the de jure standard, but it is a de facto standard that is available on many distributions. Parsing it is fast and gives the right answer on many distributions. 2. If that didn't work (which can happen on some distributions, including common ones when a certain optional lsb base package isn't installed), then invoke the current platform.dist() code. This is a lot of code, its code has to be changed before it can recognize any new distribution or a change in a distribution, and it gives answers on Ubuntu and Arch Linux which users say are the wrong answer, but it is fast and it gives the answer users want in most cases. 3. If that didn't work (which presumably only happens on distributions that the authors of platform.dist() didn't know about or didn't bother to support), then invoke the de jure standard executable lsb_release. This works on any LSB-compliant system, but it costs a subprocess. 4. If that didn't work, check for /etc/arch-release to signal Arch Linux. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1322 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1322] platform.dist() has unpredictable result under Linux
Zooko O'Whielacronx zo...@zooko.com added the comment: For what it is worth, here is the current version of this code that we are using in Tahoe-LAFS: http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/src/allmydata/__init__.py?annotate=blamerev=5033#L36 You can see the results on our buildbot: http://tahoe-lafs.org/buildbot/waterfall?show_events=false In the tahoe-version build step, e.g.: http://tahoe-lafs.org/buildbot/builders/Eugen%20lenny-amd64/builds/820/steps/tahoe-version/logs/stdio Which says: platform: Linux-debian_5.0.6-x86_64-64bit -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1322 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
announcing pyutil-1.8.0
Folks: pyutil is a collection of modules and functions that we've found useful over the years. Peter Westlake and Ravi Pinjala (who found pyutil through the Tahoe-LAFS project) contributed some documentation about what each module in pyutil does, and SimpleGeo (my employer) is using one small part of pyutil (the jsonutil [1] module) in an upcoming new product, so I packaged up version 1.8.0. The main differences from previous versions are better docs, a very simple declaration of permissive open source licensing [2], and several of the modules have been marked as deprecated. read me: http://tahoe-lafs.org/trac/pyutil/browser/trunk/README.rst download: http://pypi.python.org/pypi/pyutil Regards, Zooko [1] http://tahoe-lafs.org/trac/pyutil/browser/trunk/pyutil/jsonutil.py [2] http://tahoe-lafs.org/trac/pyutil/browser/trunk/COPYING.SPL.txt -- http://mail.python.org/mailman/listinfo/python-list
ANNOUNCING Tahoe, the Least-Authority File System, v1.8.0
Hello, people of python-list. This storage project uses Python for almost everything, except we use C/C++ for the CPU-intensive computations (cryptography and erasure coding) and we use JavaScript for some user interface bits. We're even looking at the possibility of replacing the C/C++ crypto code with pure-Python code, relying on JIT compilers such as PyPy to make it efficient enough. :-) http://twitter.com/#!/fijall/status/25314330015 Regards, Zooko ANNOUNCING Tahoe, the Least-Authority File System, v1.8.0 The Tahoe-LAFS team is pleased to announce the immediate availability of version 1.8.0 of Tahoe-LAFS, an extremely reliable distributed storage system. Get it here: http://tahoe-lafs.org/source/tahoe/trunk/docs/quickstart.html Tahoe-LAFS is the first distributed storage system to offer provider-independent security — meaning that not even the operators of your storage servers can read or alter your data without your consent. Here is the one-page explanation of its unique security and fault-tolerance properties: http://tahoe-lafs.org/source/tahoe/trunk/docs/about.html The previous stable release of Tahoe-LAFS was v1.7.1, which was released July 18, 2010 [1]. v1.8.0 offers greatly improved performance and fault-tolerance of downloads and improved Windows support. See the NEWS file [2] for details. WHAT IS IT GOOD FOR? With Tahoe-LAFS, you distribute your filesystem across multiple servers, and even if some of the servers fail or are taken over by an attacker, the entire filesystem continues to work correctly, and continues to preserve your privacy and security. You can easily share specific files and directories with other people. In addition to the core storage system itself, volunteers have built other projects on top of Tahoe-LAFS and have integrated Tahoe-LAFS with existing systems, including Windows, JavaScript, iPhone, Android, Hadoop, Flume, Django, Puppet, bzr, mercurial, perforce, duplicity, TiddlyWiki, and more. See the Related Projects page on the wiki [3]. We believe that strong cryptography, Free and Open Source Software, erasure coding, and principled engineering practices make Tahoe-LAFS safer than RAID, removable drive, tape, on-line backup or cloud storage. This software is developed under test-driven development, and there are no known bugs or security flaws which would compromise confidentiality or data integrity under recommended use. (For all important issues that we are currently aware of please see the known_issues.txt file [4].) COMPATIBILITY This release is compatible with the version 1 series of Tahoe-LAFS. Clients from this release can write files and directories in the format used by clients of all versions back to v1.0 (which was released March 25, 2008). Clients from this release can read files and directories produced by clients of all versions since v1.0. Servers from this release can serve clients of all versions back to v1.0 and clients from this release can use servers of all versions back to v1.0. This is the eleventh release in the version 1 series. This series of Tahoe-LAFS will be actively supported and maintained for the forseeable future, and future versions of Tahoe-LAFS will retain the ability to read and write files compatible with this series. LICENCE You may use this package under the GNU General Public License, version 2 or, at your option, any later version. See the file COPYING.GPL [5] for the terms of the GNU General Public License, version 2. You may use this package under the Transitive Grace Period Public Licence, version 1 or, at your option, any later version. (The Transitive Grace Period Public Licence has requirements similar to the GPL except that it allows you to delay for up to twelve months after you redistribute a derived work before releasing the source code of your derived work.) See the file COPYING.TGPPL.html [6] for the terms of the Transitive Grace Period Public Licence, version 1. (You may choose to use this package under the terms of either licence, at your option.) INSTALLATION Tahoe-LAFS works on Linux, Mac OS X, Windows, Cygwin, Solaris, *BSD, and probably most other systems. Start with docs/quickstart.html [7]. HACKING AND COMMUNITY Please join us on the mailing list [8]. Patches are gratefully accepted -- the RoadMap page [9] shows the next improvements that we plan to make and CREDITS [10] lists the names of people who've contributed to the project. The Dev page [11] contains resources for hackers. SPONSORSHIP Tahoe-LAFS was originally developed by Allmydata, Inc., a provider of commercial backup services. After discontinuing funding of Tahoe-LAFS RD in early 2009, they continued to provide servers, bandwidth, small personal gifts as tokens of appreciation, and bug reports. Google, Inc. sponsored Tahoe-LAFS development as part of the Google Summer of Code 2010. They awarded four sponsorships to students from around the world to hack on Tahoe-LAFS that summer. Thank you to Allmydata and
[issue1589] New SSL module doesn't seem to verify hostname against commonName in certificate
Zooko O'Whielacronx zo...@zooko.com added the comment: This appears to be a concern for some people. Maybe the builtin ssl module should be deprecated if there isn't a lot of manpower to maintain it and instead the well-maintained pyOpenSSL package should become the recommended tool? Here is a letter that I just received, in my role as a developer of Tahoe-LAFS, from a concerned coder who doesn't know much about Python: An FYI on Python. I'm not sure how businesses handle this (I've always worked in Windows shops), but I imagine some might consider pulling Python until it is properly secured. Pulling Python might affect Tahoe, which I would like to see do well. Here is my reply to him: Thanks for the note warning me about this issue! I appreciate it. The Tahoe-LAFS project doesn't use the builtin ssl module that comes with the Python Standard Library and instead uses the separate pyOpenSSL package (and uses the separate Twisted package for HTTP and other networking protocols). Therefore this isn't an issue for Tahoe-LAFS. I agree that it is potentially a marketing issue in that people might mistakenly think that Tahoe-LAFS is vulnerable or might, as you suggest, blacklist Python as such and thus hit Tahoe-LAFS as collateral damage. There's not much I can do about that from the perspective of a Tahoe-LAFS developer. From the perspective of contributor to Python, I'm also not sure what to do, except perhaps to complain. :-) I guess I'll try to stir the waters a bit by suggesting that Python should deprecate the builtin ssl module and recommend the pyOpenSSL package instead. -- nosy: +zooko ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1589 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
announcing pycryptopp-0.5.20
Folks: pycryptopp is a Python crypto library, so it could be considered an alternative to PyCrypto. However, pycryptopp offers very few algorithms. In fact, it only offers RSA, AES, and SHA-256. So if that's all you need, then great. I just released a new version of pycryptopp. http://tahoe-lafs.org/trac/pycryptopp 2010-09-18 -- pycryptopp v0.5.20 The following things are new in this release: * fix bugs in assembly implementation of SHA-256 from Crypto++ * fix it to compile on *BSD (#39) * improve doc strings * add a quick start-up-self-test of SHA256 (#43) * execute the quick start-up-self-tests of AES and SHA256 on module import Regards, Zooko -- http://mail.python.org/mailman/listinfo/python-list
[issue3937] platform.dist(): detect Linux distribution version in a robust, standard way
Zooko O'Whielacronx zo...@zooko.com added the comment:
Please check what platform.linux_distribution() returns on your platform
using Python 2.6rc2.
Here are the results of that. Summary: looks fine to me.
http://tahoe-lafs.org/buildbot/waterfall
Here are the scripts that are generating these results:
http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/misc/build_helpers/show-tool-versions.py?rev=4643#L8
http://tahoe-lafs.org/trac/tahoe-lafs/browser/trunk/src/allmydata/__init__.py?rev=4687#L80
The relevant parts of the results are summarized below:
http://tahoe-lafs.org/buildbot/builders/Eugen lenny-amd64
platform: Linux-2.6.26-1-vserver-amd64-x86_64-with-debian-5.0.5
python: 2.5.2 (r252:60911, Jan 24 2010, 17:44:40) [GCC 4.3.2]
maxunicode: 1114111
platform: Linux-debian_5.0.5-x86_64-64bit
http://tahoe-lafs.org/buildbot/builders/hardy-amd64
platform: Linux-2.6.32-24-server-x86_64-with-Ubuntu-10.04-lucid
linux_distribution: ('Ubuntu', '10.04', 'lucid')
python: 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) [GCC 4.4.3]
maxunicode: 1114111
platform: Linux-Ubuntu_10.04-x86_64-64bit_ELF
http://tahoe-lafs.org/buildbot/builders/Shawn jaunty amd64
platform: Linux-2.6.32-24-generic-x86_64-with-Ubuntu-10.04-lucid
linux_distribution: ('Ubuntu', '10.04', 'lucid')
python: 2.6.5 (r265:79063, Apr 16 2010, 13:57:41) [GCC 4.4.3]
maxunicode: 1114111
platform: Linux-Ubuntu_10.04-x86_64-64bit_ELF
http://tahoe-lafs.org/buildbot/builders/Zooko ubuntu-amd64 yukyuk
platform: Linux-2.6.32.8yukyuk36-x86_64-with-Ubuntu-10.04-lucid
python: 2.6.4+ (r264:75706, Feb 16 2010, 02:54:39) [GCC 4.4.3], maxunicode:
1114111
platform: Linux-Ubuntu_10.04-x86_64-64bit_ELF
http://tahoe-lafs.org/buildbot/builders/Ruben Fedora
platform: Linux-2.6.33.5-rscloud-x86_64-with-fedora-15-Rawhide
linux_distribution: ('Fedora', '15', 'Rawhide')
python: 2.7 (r27:82500, Jul 26 2010, 18:19:48) [GCC 4.5.0 20100716 (Red Hat
4.5.0-3)]
maxunicode: 1114111
platform: Linux-fedora_15-x86_64-64bit_ELF
http://tahoe-lafs.org/buildbot/builders/Arthur lenny c7 32bit
platform: Linux-2.6.24dedibox-r8-c7-i686-with-debian-5.0.5
python: 2.5.2 (r252:60911, Jan 24 2010, 14:53:14) [GCC 4.3.2]
maxunicode: 1114111
platform: Linux-debian_5.0.5-i686-32bit
http://tahoe-lafs.org/buildbot/builders/David A. OpenSolaris i386
platform: SunOS-5.11-i86pc-i386-32bit
python: 2.5.2 (r252:60911, Mar 11 2009, 04:50:47) [C]
maxunicode: 65535
platform: SunOS-5.11-i86pc-i386-32bit
http://tahoe-lafs.org/buildbot/builders/Randy FreeBSD-amd64
platform: FreeBSD-8.1-STABLE-amd64-64bit-ELF
python: 2.5.5 (r255:77872, Mar 12 2010, 08:40:28) [GCC 4.2.1 20070719
[FreeBSD]]
maxunicode: 1114111
http://tahoe-lafs.org/buildbot/builders/Kyle OpenBSD-4.6 amd64
platform: openbsd-4.6-amd64-genuine_intel-r-_cpu_0...@_2.93ghz-64bit-elf
linux_distribution: ('', '', '')
python: 2.6.2 (r262:71600, Jul 4 2009, 23:29:57) [GCC 3.3.5 (propolice)]
maxunicode: 65535
platform: openbsd-4.6-amd64-genuine_intel-r-_cpu_0...@_2.93ghz-64bit-elf
http://tahoe-lafs.org/buildbot/builders/Ludo NixOS-amd64
platform: Linux-2.6.32.13-x86_64-with-glibc2.2.5
python: 2.6.5 (r265:79063, Mar 23 2010, 09:42:45) [GCC 4.4.3] , maxunicode:
65535 , stdout.encoding: None , stdin.encoding: None , filesystem.encoding:
UTF-8
http://tahoe-lafs.org/buildbot/builders/MM netbsd5 i386 warp
platform: NetBSD-5.0.2-i386-32bit-ELF
linux_distribution: ('', '', '')
python: 2.6.5 (r265:79063, Jun 23 2010, 11:07:30) [GCC 4.1.3 20080704
prerelease (NetBSD nb2 20081120)] , maxunicode: 65535
platform: NetBSD-5.0.2-i386-32bit-ELF
http://tahoe-lafs.org/buildbot/builders/FranXois lenny-armv5tel
platform: Linux-2.6.32-trunk-iop32x-armv5tel-with-debian-squeeze-sid
linux_distribution: ('debian', 'squeeze/sid', '')
python: 2.6.5+ (release26-maint, Jul 7 2010, 00:21:05) [GCC 4.4.4]
maxunicode: 1114111
platform: Linux-debian_squeeze/sid-armv5tel-32bit_ELF
http://tahoe-lafs.org/buildbot/builders/FreeStorm CentOS5-i386
platform: Linux-2.6.18-194.8.1.el5-i686-athlon-with-redhat-5.5-Final
python: 2.4.3 (#1, Sep 3 2009, 15:37:12) [GCC 4.1.2 20080704 (Red Hat
4.1.2-46)]
maxunicode: 1114111
platform: Linux-redhat_5.5-i686-32bit_ELF
http://tahoe-lafs.org/buildbot/builders/zooko ootles Mac-amd64 10.4
platform: Darwin-8.11.1-i386-32bit
python: 2.5.4 (release25-maint:72153M, Apr 30 2009, 12:28:20) [GCC 4.0.1
(Apple Computer, Inc. build 5367)]
maxunicode: 65535
platform: Darwin-8.11.1-i386-32bit
http://tahoe-lafs.org/buildbot/builders/Zooko zomp Mac-amd64 10.6 py2.6
platform: Darwin-10.4.0-i386-64bit
linux_distribution: ('', '', '')
python: 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) [GCC 4.2.1 (Apple Inc. build
5646)]
maxunicode: 65535
platform: Darwin-10.4.0-i386-64bit
http://tahoe-lafs.org/buildbot/builders/FreeStorm WinXP-x86 py2.6
platform: Windows-XP-5.1.2600-SP3
linux_distribution: ('', '', '')
python: 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit
Re: Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP
On Wed, Jul 7, 2010 at 3:32 AM, Jonathan Hartley tart...@tartley.com wrote: I presume this problem would go away if future versions of Python itself were compiled on Windows with something like MinGW gcc. You might want to track issue3871. Roumen Petrov has done a lot of work to make CPython compilable with mingw on Windows, as well as to make it possible to compile CPython on a different operating system and produce a CPython executable for Windows (cross-compile). And by the way, I've usually had success building my native extension modules with mingw. I understand (vaguely) that if a native extension module needs to pass FILE*'s or C++ iostreams back and forth to different extension modules or the the core CPython interpreter then this could lead to segfaults, but none of my extension modules need to do that. I would suggest that people try to build their native extension modules with mingw, and if it doesn't work report a bug (to mingw project and to the Python project) so that we can track more precisely what the issues are. Regards, Zooko http://bugs.python.org/issue3871# cross and native build of python for mingw32 with distutils -- http://mail.python.org/mailman/listinfo/python-list
ANNOUNCING Tahoe, the Least-Authority File System, v1.7.1
Folks: This innovative distributed filesystem is written entirely in Python. Well, actually we rely on some C/C++ extension code in Python packages like zfec and pycryptopp for some mathematical heavy lifting, but all of the code in the tahoe-lafs package is actually pure Python. Regards, Zooko ANNOUNCING Tahoe, the Least-Authority File System, v1.7.1 The Tahoe-LAFS team is pleased to announce the immediate availability of version 1.7.1 of Tahoe-LAFS, an extremely reliable distributed storage system. Tahoe-LAFS is the first distributed storage system which offers provider-independent security—meaning that not even the operators of your storage servers can read or alter your data without your consent. Here is the one-page explanation of its unique security and fault-tolerance properties: http://tahoe-lafs.org/source/tahoe/trunk/docs/about.html Tahoe-LAFS v1.7.1 is the successor to v1.7.0, which was released June 18, 2010 [1]. v1.7.1 is a stable minor release which adds several bugfixes and improvements in security, forward-compatibility, packaging, usability, and portability. See the NEWS file [2] for details. WHAT IS IT GOOD FOR? With Tahoe-LAFS, you distribute your filesystem across multiple servers, and even if some of the servers are compromised by by an attacker, the entire filesystem continues to work correctly, and continues to preserve your privacy and security. You can easily share specific files and directories with other people. In addition to the core storage system itself, volunteers have built other projects on top of Tahoe-LAFS and have integrated Tahoe-LAFS with existing systems. These include frontends for Windows, Macintosh, JavaScript, iPhone, and Android, and plugins for Hadoop, bzr, mercurial, duplicity, TiddlyWiki, and more. See the Related Projects page on the wiki [3]. We believe that strong cryptography, Free and Open Source Software, erasure coding, and principled engineering practices make Tahoe-LAFS safer than RAID, removable drive, tape, on-line backup or cloud storage systems. This software is developed under test-driven development, and there are no known bugs or security flaws which would compromise confidentiality or data integrity under recommended use. (For all currently known issues please see the known_issues.txt file [4].) COMPATIBILITY This release is fully compatible with the version 1 series of Tahoe-LAFS. Clients from this release can write files and directories in the format used by clients of all versions back to v1.0 (which was released March 25, 2008). Clients from this release can read files and directories produced by clients of all versions since v1.0. Servers from this release can serve clients of all versions back to v1.0 and clients from this release can use servers of all versions back to v1.0. This is the tenth release in the version 1 series. This series of Tahoe-LAFS will be actively supported and maintained for the forseeable future, and future versions of Tahoe-LAFS will retain the ability to read and write files compatible with this series. LICENCE You may use this package under the GNU General Public License, version 2 or, at your option, any later version. See the file COPYING.GPL [5] for the terms of the GNU General Public License, version 2. You may use this package under the Transitive Grace Period Public Licence, version 1 or, at your option, any later version. (The Transitive Grace Period Public Licence has requirements similar to the GPL except that it allows you to delay for up to twelve months after you redistribute a derived work before releasing the source code of your derived work.) See the file COPYING.TGPPL.html [6] for the terms of the Transitive Grace Period Public Licence, version 1. (You may choose to use this package under the terms of either licence, at your option.) INSTALLATION Tahoe-LAFS works on Linux, Mac OS X, Windows, Cygwin, Solaris, *BSD, and probably most other systems. Start with docs/quickstart.html [7]. HACKING AND COMMUNITY Please join us on the mailing list [8]. Patches are gratefully accepted -- the RoadMap page [9] shows the next improvements that we plan to make and CREDITS [10] lists the names of people who've contributed to the project. The Dev page [11] contains resources for hackers. SPONSORSHIP Tahoe-LAFS was originally developed by Allmydata, Inc., a provider of commercial backup services. After discontinuing funding of Tahoe-LAFS RD in early 2009, they have continued to provide servers, bandwidth, small personal gifts as tokens of appreciation, and bug reports. Thank you to Allmydata, Inc. for their generous and public-spirited support. Google, Inc. is sponsoring Tahoe-LAFS development as part of the Google Summer of Code 2010. Google suggested that we should apply for the Summer of Code program, and when we did they generously awarded four sponsorships to students from around the world to hack on Tahoe-LAFS this summer. Thank you to Google, Inc. for their generous
I wish for a tool named 2to6.
Folks: I have been (I admit it) a Python 3 skeptic. I even speculated that the Python 3 backward-incompatibility would lead to the obsolescence of Python: http://pubgrid.tahoe-lafs.org/uri/URI:DIR2-RO:ixqhc4kdbjxc7o65xjnveoewym:5x6lwoxghrd5rxhwunzavft2qygfkt27oj3fbxlq4c6p45z5uneq/blog.html However, things are really looking up now because it turns out that it is eminently practical to support both Python 2 and Python 3 with a single codebase. There have been some recent discussions about that on this list. A few references: http://mail.python.org/pipermail/python-list/2010-July/1249312.html http://www.voidspace.org.uk/python/weblog/arch_d7_2010_03_20.shtml#e1167 http://nedbatchelder.com/blog/200910/running_the_same_code_on_python_2x_and_3x.html http://www.mail-archive.com/numpy-discuss...@scipy.org/msg26524.html Benjamin Peterson has even written a library intended to help programmers who want to do that: http://packages.python.org/six/ This note to the list is to express my wish for an automated tool named 2to6 which converts my Python 2.6 codebase to being both py2- and py2- compatible using Benjamin Peterson's six library. Regards, Zooko -- http://mail.python.org/mailman/listinfo/python-list
Re: Python -- floating point arithmetic
On Thu, Jul 8, 2010 at 4:58 AM, Adam Skutt ask...@gmail.com wrote: I can't think of any program I've ever written where the inputs are actually intended to be decimal. Consider a simple video editing program, and the user specifies a frame rate 23.976 fps. Is that what they really wanted? No, they wanted 24000/1001 but didn't feel like typing that. Okay, so there was a lossy conversion from the user's intention (24000/1001) to what they typed in (23.976). instr = '23.976' Now as a programmer you have two choices: 1. accept what they typed in and losslessly store it in a decimal: from decimal import Decimal as D x = D(instr) print x 23.976 2. accept what they typed in and lossily convert it to a float: x = float(instr) print %.60f % (x,) 23.97509050529822707176208496093750 option 2 introduces further error between what you have stored in your program and what the user originally wanted and offers no advantages except for speed, right? I'm sorry, what will never be true? Are you saying that decimals have a disadvantage compared to floats? If so, what is their disadvantage? He's saying that once you get past elementary operations, you quickly run into irrational numbers, which you will not be representing accurately. Moreover, in general, it's impossible to even round operations involving transcendental functions to an arbitrary fixed- precision, you may need effectively infinite precision in order to the computation. In practice, this means the error induced by a lossy input conversion (assuming you hadn't already lost information) is entirely negligible compared to inherent inability to do the necessary calculations. But this is not a disadvantage of decimal compared to float is it? These problems affect both representations. Although perhaps they affect them differently, I'm not sure. I think sometimes people conflate the fact that decimals can easily have higher and more variable precision than floats with the fact that decimals are capable of losslessly storing decimal values but floats aren't. Regards, Zooko -- http://mail.python.org/mailman/listinfo/python-list
Re: Python -- floating point arithmetic
On Thu, Jul 8, 2010 at 11:22 AM, Adam Skutt ask...@gmail.com wrote: On Jul 8, 12:38 pm, Zooko O'Whielacronx zo...@zooko.com wrote: Now as a programmer you have two choices: … 1. accept what they typed in and losslessly store it in a decimal: … 2. accept what they typed in and lossily convert it to a float: No, you have a third choice, and it's the only right choice: 3. Convert the input to user's desired behavior and behave accordingly. Anything else, here, will result in A/V sync issues. Okay, please tell me about how this is done. I've never dealt with this sort of issue directly. As far as I can imagine, any way to implement option 3 will involve accepting the user's input and storing it in memory somehow and then computing on it. As far as I can tell, doing so with a decimal instead of a float will never be worse for your outcome and will often be better. But, please explain in more detail how this works. Thanks! Regards, Zooko -- http://mail.python.org/mailman/listinfo/python-list
Re: Python -- floating point arithmetic
I'm starting to think that one should use Decimals by default and reserve floats for special cases. This is somewhat analogous to the way that Python provides arbitrarily-big integers by default and Python programmers only use old-fashioned fixed-size integers for special cases, such as interoperation with external systems or highly optimized pieces (in numpy or in native extension modules, for example). Floats are confusing. I've studied them more than once over the years but I still can't really predict confidently what floats will do in various situations. And most of the time (in my experience) the inputs and outputs to your system and the literals in your code are actually decimal, so converting them to float immediately introduces a lossy data conversion before you've even done any computation. Decimal doesn't have that problem. From now on I'll probably try to use Decimals exclusively in all my new Python code and switch to floats only if I need to interoperate with an external system that requires floats or I have some tight inner loop that needs to be highly optimized. Regards, Zooko -- http://mail.python.org/mailman/listinfo/python-list
Re: The real problem with Python 3 - no business case for conversion (was I strongly dislike Python 3)
Dear Paul McGuire: Thank you very much for these notes! See also a few other notes: Michael Foord: http://www.voidspace.org.uk/python/weblog/arch_d7_2010_03_20.shtml#e1167 Ned Batchelder: http://nedbatchelder.com/blog/200910/running_the_same_code_on_python_2x_and_3x.html I was wondering if it would be useful to have a library that implements these hacks so that people like me who prefer to maintain a single codebase instead of using a tool like 2to3 could easily adopt them. Oh look! Here is one: http://pybites.blogspot.com/2010/06/six-python-23-compatibility-helpers.html :-) Regards, Zooko -- http://mail.python.org/mailman/listinfo/python-list
Re: Python -- floating point arithmetic
On Wed, Jul 7, 2010 at 10:04 PM, David Cournapeau courn...@gmail.com wrote: Decimal vs float is a different matter altogether: decimal has downsides compared to float. First, there is this irreconcilable fact that no matter how small your range is, it is impossible to represent exactly all (even most) numbers exactly with finite memory - float and decimal are two different solutions to this issue, with different tradeoffs. Decimal are more intuitive than float for numbers that can be represented as decimal - but most numbers cannot be represented as (finite) decimal. This is not a downside of decimal as compared to float, since most numbers also cannot be represented as float. And most of the time (in my experience) the inputs and outputs to your system and the literals in your code are actually decimal, so converting them to float immediately introduces a lossy data conversion before you've even done any computation. Decimal doesn't have that problem. That's not true anymore once you start doing any computation, if by decimal you mean finite decimal. I don't understand. I described two different problems: problem one is that the inputs, outputs and literals of your program might be in a different encoding (in my experience they have most commonly been in decimal). Problem two is that your computations may be lossy. If the inputs, outputs and literals of your program are decimal (as they have been for most of my programs) then using decimal is better than using float because of problem one. Neither has a strict advantage over the other in terms of problem two. (There is also problem zero, which is that floats more confusing, which is how this thread got started. Problem zero is probably the most important problem for many cases.) And that will be never true once you start using non trivial computation (i.e. transcendental functions like log, exp, etc...). I'm sorry, what will never be true? Are you saying that decimals have a disadvantage compared to floats? If so, what is their disadvantage? (And do math libraries like http://code.google.com/p/dmath/ help ?) Regards, Zooko -- http://mail.python.org/mailman/listinfo/python-list
[issue9123] insecure os.urandom on VMS
New submission from Zooko O'Whielacronx zo...@zooko.com: os.urandom() on VMS invokes OpenSSL's RAND_pseudo_bytes(). That is documented on: http://www.openssl.org/docs/crypto/RAND_bytes.html as being predictable and therefore unsuitable for many cryptographic purposes. This is inconsistent with the documentation of os.urandom(): urandom(n) - str\n\n\ Return a string of n random bytes suitable for cryptographic use. This probably means that users of Python on VMS are vulnerable to attack based on the predictability of the results they get from os.urandom(). Honestly, I would have guessed that there *were* no users of Python on VMS when I started this bug report, but look--apparently there are: http://www.vmspython.org To fix this, change the call from RAND_pseudo_bytes() to RAND_bytes(). It has the same type signature and actually does what os.urandom() needs. -- messages: 108963 nosy: zooko priority: normal severity: normal status: open title: insecure os.urandom on VMS ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9123] insecure os.urandom on VMS
Zooko O'Whielacronx zo...@zooko.com added the comment:
HACK Zooko-Ofsimplegeos-MacBook-Pro:~/playground/python/release27-trunk$ svn
diff
Index: Modules/posixmodule.c
===
--- Modules/posixmodule.c (revision 82382)
+++ Modules/posixmodule.c (working copy)
@@ -8481,7 +8481,7 @@
result = PyString_FromStringAndSize(NULL, howMany);
if (result != NULL) {
/* Get random data */
-if (RAND_pseudo_bytes((unsigned char*)
+if (RAND_bytes((unsigned char*)
PyString_AS_STRING(result),
howMany) 0) {
Py_DECREF(result);
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9123
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9123] insecure os.urandom on VMS
Zooko O'Whielacronx zo...@zooko.com added the comment: This issue is a security vulnerability. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue9123 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
ANNOUNCING Tahoe, the Least-Authority File System, v1.7.0
Dear people of python-list: We just released Tahoe-LAFS v1.7, the secure distributed filesystem written entirely [*] in Python. The major new feature is an SFTP server. This means that (with enough installing software and tinkering with your operating system configuration) you can have a normal-looking mount point backed by a Tahoe-LAFS grid. Google is sponsoring us through Google Summer of Code. The next release after this one will hopefully include the resulting improvements. Regards, Zooko [*] That's a lie. The parts that require maximum CPU efficiency—secure hash functions, ciphers, erasure coding—are written in C or C++. But we wrote as much of it as we could in Python. ANNOUNCING Tahoe, the Least-Authority File System, v1.7.0 The Tahoe-LAFS team is pleased to announce the immediate availability of version 1.7.0 of Tahoe-LAFS, an extremely reliable distributed storage system. Tahoe-LAFS is the first distributed storage system which offers provider-independent security—meaning that not even the operator of your storage server can read or alter your data without your consent. Here is the one-page explanation of its unique security and fault-tolerance properties: http://tahoe-lafs.org/source/tahoe/trunk/docs/about.html Tahoe-LAFS v1.7.0 is the successor to v1.6.1, which was released February 27, 2010 [1]. v1.7.0 is a major new release with new features and bugfixes. It adds a fully functional SFTP interface, support for non-ASCII character encodings, and a new upload algorithm which guarantees that each file is spread over multiple servers for fault-tolerance. See the NEWS file [2] for details. WHAT IS IT GOOD FOR? With Tahoe-LAFS, you distribute your filesystem across multiple servers, and even if some of the servers are compromised by by an attacker, the entire filesystem continues to work correctly, and continues to preserve your privacy and security. You can easily share specific files and directories with other people. In addition to the core storage system itself, volunteers have built other projects on top of Tahoe-LAFS and have integrated Tahoe-LAFS with existing systems. These include frontends for Windows, Macintosh, JavaScript, iPhone, and Android, and plugins for Hadoop, bzr, mercurial, duplicity, TiddlyWiki, and more. See the Related Projects page on the wiki [3]. We believe that strong cryptography, Free and Open Source Software, erasure coding, and principled engineering practices make Tahoe-LAFS safer than RAID, removable drive, tape, on-line backup or cloud storage systems. This software is developed under test-driven development, and there are no known bugs or security flaws which would compromise confidentiality or data integrity under recommended use. (For all currently known issues please see the known_issues.txt file [4].) COMPATIBILITY This release is fully compatible with the version 1 series of Tahoe-LAFS. Clients from this release can write files and directories in the format used by clients of all versions back to v1.0 (which was released March 25, 2008). Clients from this release can read files and directories produced by clients of all versions since v1.0. Servers from this release can serve clients of all versions back to v1.0 and clients from this release can use servers of all versions back to v1.0. This is the ninth release in the version 1 series. This series of Tahoe-LAFS will be actively supported and maintained for the forseeable future, and future versions of Tahoe-LAFS will retain the ability to read and write files compatible with Tahoe-LAFS v1. LICENCE You may use this package under the GNU General Public License, version 2 or, at your option, any later version. See the file COPYING.GPL [5] for the terms of the GNU General Public License, version 2. You may use this package under the Transitive Grace Period Public Licence, version 1 or, at your option, any later version. (The Transitive Grace Period Public Licence has requirements similar to the GPL except that it allows you to wait for up to twelve months after you redistribute a derived work before releasing the source code of your derived work.) See the file COPYING.TGPPL.html [6] for the terms of the Transitive Grace Period Public Licence, version 1. (You may choose to use this package under the terms of either licence, at your option.) INSTALLATION Tahoe-LAFS works on Linux, Mac OS X, Windows, Cygwin, Solaris, *BSD, and probably most other systems. Start with docs/quickstart.html [7]. HACKING AND COMMUNITY Please join us on the mailing list [8]. Patches are gratefully accepted -- the RoadMap page [9] shows the next improvements that we plan to make and CREDITS [10] lists the names of people who've contributed to the project. The Dev page [11] contains resources for hackers. SPONSORSHIP Tahoe-LAFS was originally developed by Allmydata, Inc., a provider of commercial backup services. After discontinuing funding of Tahoe-LAFS RD in early 2009, they have continued to provide
[issue3439] create a numbits() method for int and long types
Zooko O'Whielacronx zo...@zooko.com added the comment:
There is a small mistake in the docs:
def bit_length(x):
'Number of bits necessary to represent self in binary.'
s = bin(x) # binary representation: bin(-37) -- '-0b100101'
s = s.lstrip('-0b') # remove leading zeros and minus sign
return len(s) # len('100101') -- 6
is probably supposed to be:
def bit_length(x):
'Number of bits necessary to represent x in binary.'
s = bin(x) # binary representation: bin(-37) -- '-0b100101'
s = s.lstrip('-0b') # remove leading zeros and minus sign
return len(s) # len('100101') -- 6
--
nosy: +zooko
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue3439
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
announcing jsonutil package
Folks: I've uploaded a small package to PyPI which is a small wrapper around simplejson that sets the default behavior so that JSON decimal values are mapped to type Decimal instead of type float: http://pypi.python.org/pypi/jsonutil It has pretty thorough unit tests, including a copy of all the simplejson unit tests, which I copied into the jsonutil package, as well as a few added tests to test the new functionality. I also did some benchmarks to satisfy myself that using jsonutil isn't noticeably slower than using simplejson. You can find the benchmarking code in the package, but I'm not sure if it comes with documentation sufficient to let you easily run the benchmarks yourself. I did this originally on behalf of my employer, SimpleGeo, who now use this wrapper in our production system. Based on my reasoning and also SimpleGeo's experience, this introduced absolutely no compatibility problems between our service (an HTTP API) and users of our service, who themselves parse and construct JSON however they want (typically from other languages like Ruby and PHP). This is because no user of our API sends a string like 0.03 and then expects to get back 0.02 and breaks in some way if we instead send back 0.03. If there were clients like that, then they would have broken when we made this change, because formerly, using simplejson, we would have accepted 0.03 and then sent back 0.02, but now, using jsonutil, we accept 0.03 and send back 0.03. I would expect clients that breaks in such a case to be rare, but of course computer systems are always surprising me with their creative ways to be fragile and failure-prone, so I wouldn't bet against a few such systems existing. If anyone spots one in the wild, please let me know. Converting from simplejson (default float) to jsonutil (default Decimal) did, as expected, cause type mismatches internally inside our server, where some calling code was expecting float and the called code was returning Decimal, or vice versa. So far these problems were all shallow in that they could be fixed by adding a cast or by changing one side or the other, as desired. In short, I advise people not to fear using Python Decimals to manage their JSON decimal strings. It will probably introduce zero incompatibilities across the wire, and any type mismatches it introduces within your own code base will probably be easy to fix. Regards, Zooko -- http://mail.python.org/mailman/listinfo/python-list
[issue8654] Improve ABI compatibility between UCS2 and UCS4 builds
Zooko O'Whielacronx zo...@zooko.com added the comment: Option 1: Make Unicode-agnosticism the default and force anyone who cares about the Unicode setting to include a separate header. If they don't include that header, they can only call safe functions and can't poke at PyUnicodeObject's internals. If they include the header, their module will always generate a link failure if the Unicode settings are mismatched. (Guido proposed this solution in the python-ideas thread) +1 The packaging and compatibility problems are pressing concerns for many people. Poking at PyUnicodeObject's internals seems like a rare pretty rare need for an extension module to do. If I understand correctly, this option provides the best solution to the packaging and compatibility issues (over the long term, as Python and the authors of extension modules upgrade). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8654 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8597] build out-of-line asm on Windows
New submission from Zooko O'Whielacronx zo...@zooko.com: I maintain a Python package which comes with assembly optimizations -- http://tahoe-lafs.org/trac/pycryptopp . Someone was porting this package to Win64 today and discovered that distutils couldn't build it because the Microsoft compiler on Win64 doesn't allow in-line assembly and distutils doesn't know how to build out-of-line assembly (see also http://stackoverflow.com/questions/1664812/can-pythons-distutils-compile-s-assembly which says that there isn't a known way to do this). So the fellow who is porting pycryptopp to Win64 for me, Samuel Neves, found this hack in someone else's setup.py file: http://bitbucket.org/ambroff/greenlet/src/3ad4830aa109/setup.py We're probably going to adapt that hack to the pycryptopp setup.py file in order to work around this problem. But then another fellow named Josip Djolonga who is a GSoC student working on Distutils2 suggested that we could patch the compiler class to use assembly files and pass them to cl.exe. Samuel worked up a patch that does that and then he was able to build pycryptopp on Win64. Here is his patch attached as a file. Note that we will still need to do some work-around in the pycryptopp setup.py file in order to make pycryptopp buildable on Win64 with older versions of Python that do not have this fix. Any suggestions on the best way to do that would be welcome. Oops, I see that Samuel gave me the patch in traditional diff form instead of unified diff form. I hope you can accept it anyway since he has gone to bed. -- assignee: tarek components: Distutils files: patch.diff keywords: patch messages: 104770 nosy: tarek, zooko priority: normal severity: normal status: open title: build out-of-line asm on Windows type: behavior Added file: http://bugs.python.org/file17182/patch.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue8597 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: StringChain -- a data structure for managing large sequences of chunks of bytes
My apologies; I left out the heading on the last of the four structures in the benchmark results. Here are those results again with the missing heading (Stringy) inserted: Regards, Zooko - Hide quoted text - On Sun, Mar 21, 2010 at 11:09 PM, Zooko O'Whielacronx zoo...@gmail.com wrote: impl: StringChain task: _accumulate_then_one_gulp 1 best: 2.694e+00 5 best: 2.742e+00 10 best: 2.310e+00 50 best: 2.040e+00 100 best: 1.988e+00 500 best: 2.193e+00 task: _alternate_str 1 best: 6.509e+00 5 best: 4.559e+00 10 best: 4.308e+00 50 best: 4.070e+00 100 best: 3.991e+00 500 best: 4.000e+00 impl: SimplerStringChain task: _accumulate_then_one_gulp 1 best: 1.407e+00 5 best: 2.317e+00 10 best: 2.012e+00 50 best: 1.902e+00 100 best: 1.897e+00 500 best: 2.104e+00 task: _alternate_str 1 best: 4.888e+00 5 best: 5.198e+00 10 best: 1.750e+01 50 best: 6.233e+01 100 best: 1.134e+02 500 best: 7.599e+02 impl: StringIOy task: _accumulate_then_one_gulp 1 best: 4.196e+00 5 best: 5.522e+00 10 best: 4.499e+00 50 best: 3.756e+00 100 best: 4.176e+00 500 best: 5.414e+00 task: _alternate_str 1 best: 5.484e+00 5 best: 7.863e+00 10 best: 2.126e+01 50 best: 6.972e+01 100 best: 1.219e+02 500 best: 9.463e+02 impl: Stringy - Hide quoted text - task: _accumulate_then_one_gulp 1 best: 1.502e+00 5 best: 1.420e+01 10 best: 2.245e+01 50 best: 8.577e+01 100 best: 2.295e+02 500 best: 1.326e+03 task: _alternate_str 1 best: 3.290e+00 5 best: 4.220e+00 10 best: 1.665e+01 50 best: 6.281e+01 100 best: 1.127e+02 500 best: 7.626e+02 -- http://mail.python.org/mailman/listinfo/python-list
Re: StringChain -- a data structure for managing large sequences of chunks of bytes
On Mon, Mar 22, 2010 at 2:07 AM, Steven D'Aprano ste...@remove.this.cybersource.com.au wrote: Perhaps you should have said that it was a wrapper around deque giving richer functionality, rather than giving the impression that it was a brand new data structure invented by you. People are naturally going to be more skeptical about a newly invented data structure than one based on a reliable, component like deque. The fact that StringChain uses deque to hold the queue of strings isn't that important. I just benchmarked it by swapping in the deque for a list and using the list costs about one third of a nanosecond per byte at the scales that the benchmark exercises (namely 10,000,000 bytes in about 10,000 strings). A third of a nanosecond per byte is about 4% of the runtime. I also implemented and benchmarked a simpler deque-based scheme which puts all the actual bytes from the strings into a deque with self.d.extend(newstr). As you would expect, this shows good asymptotic performance but the constants are relatively bad so that at all of the actual loads measured it was three orders of magnitude worse than StringChain and than String-Chain-with-a-list-instead-of-a-deque. Moral: the constants matter! Those benchmarks are appended. You can run the benchmarks yourself per the README.txt. But anyway, I take your point and I updated the StringChain README to explain that it is a pretty simple data structure that holds a list (actually a deque) of strings and isn't anything too clever or novel. By the way, we could further micro-optimize this kind of thing if ''.join() would accept either strings or buffers instead of requiring strings: ''.join([buffer(abc), def]) Traceback (most recent call last): File stdin, line 1, in module TypeError: sequence item 0: expected string, buffer found Then whenever StringChain needs to slice a string into leading and trailing parts, it could construct a buffer() viewing each part instead of making a copy of each part. it. Maybe you should consider linking to it on PyPI and seeing if there is any interest from others? http://pypi.python.org/pypi/stringchain Regards, Zooko impl: StringChain task: _accumulate_then_one_gulp 1 best: 5.698e+00, 3th-best: 7.486e+00, mean: 7.758e+00, 10 best: 4.640e+00, 3th-best: 4.690e+00, mean: 7.674e+00, 100 best: 3.789e+00, 3th-best: 3.806e+00, mean: 3.995e+00, 1000 best: 4.095e+00, 1th-best: 4.095e+00, mean: 4.095e+00, task: _alternate_str 1 best: 1.378e+01, 3th-best: 1.390e+01, mean: 1.500e+01, 10 best: 9.198e+00, 3th-best: 9.248e+00, mean: 9.385e+00, 100 best: 8.715e+00, 3th-best: 8.755e+00, mean: 8.808e+00, 1000 best: 8.738e+00, 1th-best: 8.738e+00, mean: 8.738e+00, impl: StringChainWithList task: _accumulate_then_one_gulp 1 best: 3.600e+00, 3th-best: 3.695e+00, mean: 4.129e+00, 10 best: 4.070e+00, 3th-best: 4.079e+00, mean: 4.162e+00, 100 best: 3.662e+00, 3th-best: 3.688e+00, mean: 3.721e+00, 1000 best: 3.902e+00, 1th-best: 3.902e+00, mean: 3.902e+00, 1th-worst: 3.902e+00, worst: 3.902e+00 (of 1) task: _alternate_str 1 best: 1.369e+01, 3th-best: 1.380e+01, mean: 1.442e+01, 10 best: 9.251e+00, 3th-best: 9.289e+00, mean: 9.416e+00, 100 best: 8.809e+00, 3th-best: 8.876e+00, mean: 8.943e+00, 1000 best: 9.095e+00, 1th-best: 9.095e+00, mean: 9.095e+00, impl: Dequey task: _accumulate_then_one_gulp 1 best: 2.772e+02, 3th-best: 2.785e+02, mean: 2.911e+02, 10 best: 2.314e+02, 3th-best: 2.334e+02, mean: 2.422e+02, 100 best: 2.282e+02, 3th-best: 2.288e+02, mean: 2.370e+02, 1000 best: 2.587e+02, 1th-best: 2.587e+02, mean: 2.587e+02, task: _alternate_str 1 best: 1.576e+03, 3th-best: 1.580e+03, mean: 1.608e+03, 10 best: 1.301e+03, 3th-best: 1.303e+03, mean: 1.306e+03, 100 best: 1.275e+03, 3th-best: 1.276e+03, mean: 1.278e+03, 1000 best: 1.280e+03, 1th-best: 1.280e+03, mean: 1.280e+03, -- http://mail.python.org/mailman/listinfo/python-list
Re: EURO GIRLS MISS EUROPE MISS FRENCH FRENCH PRETTY GIRLS SEXY FRENCH GIRLS on www.sexyandpretty-girls.blogspot.com SEXY RUSSIAN GIRLS SEXY GREEK GIRLS SEXY DUTCH GIRLS SEXY UK G
On Sat, Mar 20, 2010 at 1:27 PM, Brian J Mingus brian.min...@colorado.edu wrote: Moderating this stuff requires moderating all messages. Not quite. GNU Mailman comes with nice features to ease this task. You can configure it so that everyone who is currently subscribed can post freely, but new subscribers get a moderated bit set on them. The first time this new subscriber attempts to post to the list, a human moderator has to inspect their message and decide whether to approve it or deny it. If they human moderator approves it, they can also on the same web form remove the moderated bit from that poster. Therefore, the volunteer work required would be inspecting the *first* post from each *new* subscriber to see if that post is spam. Regards, Zooko -- http://mail.python.org/mailman/listinfo/python-list
Re: StringChain -- a data structure for managing large sequences of chunks of bytes
Folks: I failed to make something sufficiently clear in my original message about StringChain. The use case that I am talking about is not simply that you need to accumulate a sequence of incoming chunks of data, concatenate them together, and then process the entire result. If that is all you need to do then indeed you can accumulate the incoming strings in a list and then run ''.join(thelist) when you have received all of them and are ready to process them. But the use case that I am talking about is where you need to accumulate new incoming strings into your buffer while alternately processing leading prefixes of the buffer. The typical example is that sequences of bytes are arriving on your TCP socket and you are processing the stream incrementally. You need to process the leading prefixes of the stream as soon as you can (e.g. as soon as a line followed by a newline has accumulated, or as soon as another complete element in your data format has accumulated, etc.); you can't simply wait until the bytes stop coming and then concatenate the entire collection together at once. This is exactly the current situation in foolscap [1] which is causing a performance problem in Tahoe-LAFS [2]. To illustrate what I mean I cooked up some benchmarks showing the task of accumulate a bunch of things then consume them in a single gulp versus the task of alternate between accumulating and consuming (with accumulation going faster than consumption until the input stream runs dry). I implemented a few data structures for benchmarking: StringChain, the naive accumulatorstr += newstr approach (named Stringy in the benchmarks), one based on cStringIO (named StringIOy), one based on accumulating the new strings into a list and then doing ''.join(accumulatorlist) whenever you need to consume a leading prefix (called SimplerStringChain). Below are the abbreviated results of the benchmark. You can easily run this benchmark yourself by following the README.txt in the StringChain source package [3]. These results show that for the load imposed by this benchmark StringChain is the only one of the four that scales and that it is also nice and fast. The left-hand column is the total number of bytes that were run through it. The results are shown in nanoseconds per byte in exponential notation. (e+01 means times 10, e+02 means times 100, and e+03 means times 1000, etc.) Each result is the best of 10 tries, or of 5 tries for the tasks which were taking too long to run it 10 times. Regards, Zooko [1] http://foolscap.lothar.com/trac/ticket/149 [2] http://allmydata.org/pipermail/tahoe-dev/2010-March/004181.html [3] http://tahoe-lafs.org/trac/stringchain impl: StringChain task: _accumulate_then_one_gulp 1 best: 2.694e+00 5 best: 2.742e+00 10 best: 2.310e+00 50 best: 2.040e+00 100 best: 1.988e+00 500 best: 2.193e+00 task: _alternate_str 1 best: 6.509e+00 5 best: 4.559e+00 10 best: 4.308e+00 50 best: 4.070e+00 100 best: 3.991e+00 500 best: 4.000e+00 impl: SimplerStringChain task: _accumulate_then_one_gulp 1 best: 1.407e+00 5 best: 2.317e+00 10 best: 2.012e+00 50 best: 1.902e+00 100 best: 1.897e+00 500 best: 2.104e+00 task: _alternate_str 1 best: 4.888e+00 5 best: 5.198e+00 10 best: 1.750e+01 50 best: 6.233e+01 100 best: 1.134e+02 500 best: 7.599e+02 impl: StringIOy task: _accumulate_then_one_gulp 1 best: 4.196e+00 5 best: 5.522e+00 10 best: 4.499e+00 50 best: 3.756e+00 100 best: 4.176e+00 500 best: 5.414e+00 task: _alternate_str 1 best: 5.484e+00 5 best: 7.863e+00 10 best: 2.126e+01 50 best: 6.972e+01 100 best: 1.219e+02 500 best: 9.463e+02 task: _accumulate_then_one_gulp 1 best: 1.502e+00 5 best: 1.420e+01 10 best: 2.245e+01 50 best: 8.577e+01 100 best: 2.295e+02 500 best: 1.326e+03 task: _alternate_str 1 best: 3.290e+00 5 best: 4.220e+00 10 best: 1.665e+01 50 best: 6.281e+01 100 best: 1.127e+02 500 best: 7.626e+02 -- http://mail.python.org/mailman/listinfo/python-list
StringChain -- a data structure for managing large sequences of chunks of bytes
Folks: Every couple of years I run into a problem where some Python code that worked well at small scales starts burning up my CPU at larger scales, and the underlying issue turns out to be the idiom of accumulating data by string concatenation. It just happened again (http://foolscap.lothar.com/trac/ticket/149 ), and as usual it is hard to make the data accumulator efficient without introducing a bunch of bugs into the surrounding code. So this time around I decided to try encapsulating my preferred more efficient idiom into a reusable class. So I present to you StringChain, which is an efficient way to accumulate and process data in many chunks: http://tahoe-lafs.org/trac/stringchain Here are some benchmarks generated by running python -OOu -c 'from stringchain.bench import bench; bench.quick_bench()' as instructed by the README.txt file. The N: in the left-hand column is how many bytes were in the test dataset. The ave rate: number in the right-hand column is how many bytes per second were processed. naive means the string-based idiom sketched above and strch means using the StringChain class. _buildup init_naive N: 65536, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s:890, ave rate: 58350579 N: 131072, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s:265, ave rate: 34800398 N: 262144, time: best:0.01, 2th-best:0.01, ave:0.01, 2th-worst:0.01, worst:0.01 (of 5), reps/s: 79, ave rate: 20745346 N: 524288, time: best:0.05, 2th-best:0.05, ave:0.05, 2th-worst:0.05, worst:0.05 (of 5), reps/s: 20, ave rate: 10823850 _buildup init_strch N: 65536, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s: 25543, ave rate: 1674043282 N: 131072, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s: 14179, ave rate: 1858538925 N: 262144, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s: 8016, ave rate: 2101513050 N: 524288, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s: 4108, ave rate: 2154215572 _consume init_naive_loaded N: 65536, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s:931, ave rate: 61037862 N: 131072, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s:270, ave rate: 35454393 N: 262144, time: best:0.01, 2th-best:0.01, ave:0.01, 2th-worst:0.01, worst:0.01 (of 5), reps/s: 74, ave rate: 19471963 N: 524288, time: best:0.05, 2th-best:0.05, ave:0.05, 2th-worst:0.05, worst:0.06 (of 5), reps/s: 19, ave rate: 10146747 _consume init_strch_loaded N: 65536, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s: 4309, ave rate: 282447500 N: 131072, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s: 2313, ave rate: 303263357 N: 262144, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s: 1186, ave rate: 311159052 N: 524288, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s:606, ave rate: 317814669 _randomy init_naive N: 65536, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s:479, ave rate: 31450561 N: 131072, time: best:0.01, 2th-best:0.01, ave:0.01, 2th-worst:0.01, worst:0.01 (of 5), reps/s:140, ave rate: 18461191 N: 262144, time: best:0.02, 2th-best:0.02, ave:0.02, 2th-worst:0.03, worst:0.03 (of 5), reps/s: 42, ave rate: 11127714 N: 524288, time: best:0.06, 2th-best:0.07, ave:0.08, 2th-worst:0.08, worst:0.09 (of 5), reps/s: 13, ave rate: 6906341 _randomy init_strch N: 65536, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s:973, ave rate: 63827127 N: 131072, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s:495, ave rate: 64970669 N: 262144, time: best:0.00, 2th-best:0.00, ave:0.00, 2th-worst:0.00, worst:0.00 (of 5), reps/s:239, ave rate: 62913360 N: 524288, time: best:0.01, 2th-best:0.01, ave:0.01, 2th-worst:0.01, worst:0.01 (of 5), reps/s:121, ave rate: 63811569 The naive approach is slower than the StringChain class, and the bigger the dataset the slower it goes. The
[issue7406] int arithmetic relies on C signed overflow behaviour
Zooko O'Whielacronx zo...@zooko.com added the comment:
Here is a way to test for overflow which is correct for any C implementation:
static PyObject *
int_add(PyIntObject *v, PyIntObject *w)
{
register long a, b;
CONVERT_TO_LONG(v, a);
CONVERT_TO_LONG(w, b);
if (((a0)(b0)((LONG_MAX-a)b))
||((a0)(b0)((LONG_MIN-a)b))) {
/* would overflow the long type */
return PyLong_Type.tp_as_number-nb_add((PyObject *)v, (PyObject
*)w);
}
return PyInt_FromLong(a+b);
}
--
nosy: +zooko
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue7406
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7406] int arithmetic relies on C signed overflow behaviour
Zooko O'Whielacronx zo...@zooko.com added the comment: Here is a set of macros that I use to test for overflow: http://allmydata.org/trac/libzutil/browser/libzutil/zutilimp.h -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7406 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7202] python setup.py MYCOMMAND --verbose does not yield an unrecognized option error but also does not set the verbosity
New submission from Zooko O'Whielacronx zo...@zooko.com: This command: python setup.py --verbose darcsver works as expected -- the presence of '--verbose' increases the verbosity of logging. This command: python setup.py darcsver --verbose does not increase the verbosity, nor does it tell me that the --verbose option is an option unknown to the darcsver command. This command: python setup.py darcsver --quux gives this helpful output: usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help error: option --quux not recognized I think that the middle command (python setup.py darcsver --verbose) ought to behave either like the first command by increasing the verbosity, or like the last command by telling the user that --verbose is not a known option in that position. This is also http://bitbucket.org/tarek/distribute/issue/78/ -- assignee: tarek components: Distutils messages: 94453 nosy: tarek, zooko severity: normal status: open title: python setup.py MYCOMMAND --verbose does not yield an unrecognized option error but also does not set the verbosity ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7202 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7183] did 2.6.3 regress for some uses of the __doc__ property?
New submission from Zooko O'Whielacronx zo...@zooko.com: According to https://bugs.edge.launchpad.net/ubuntu/+source/boost1.38/+bug/457688 , Python 2.6.3 stopped working for something that Python 2.6.2 worked for, involving Boost. Andrew Mitchell looked at the Python 2.6.3 release notes, saw http://bugs.python.org/issue5890 which was fixed by http://svn.python.org/view/python/branches/release26- maint/Objects/descrobject.c?r1=71756r2=72302pathrev=72302 . Could this patch be causing this problem? -- components: Interpreter Core messages: 94329 nosy: zooko severity: normal status: open title: did 2.6.3 regress for some uses of the __doc__ property? versions: Python 2.6 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7183 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7146] [PATCH] platform.uname()[4] returns 'amd64' on Windows and 'x86-64' on Linux
New submission from Zooko O'Whielacronx zo...@zooko.com: Looking at http://bugs.python.org/setuptools/issue1 and reading the source of Lib/platform.py, it appears to me that uname() returns different strings identifying the amd64 architecture depending on what operating system is running. It would seem to me that it would be better to report the same arch with the same string on all platforms. Could someone with win64 report the output of: python -c 'import platform;print platform.uname()[4]' Here is a patch against trunk that just replaces any 'x86_64' with 'amd64': HACK yukyuk:~/playground/python/trunk$ svn diff Index: Lib/platform.py === --- Lib/platform.py (revision 75443) +++ Lib/platform.py (working copy) @@ -1225,6 +1225,9 @@ system = 'Windows' release = 'Vista' +# normalize 'amd64' arch +if machine == 'x86_64': +machine = 'amd64' _uname_cache = system,node,release,version,machine,processor return _uname_cache -- components: Library (Lib) messages: 94112 nosy: zooko severity: normal status: open title: [PATCH] platform.uname()[4] returns 'amd64' on Windows and 'x86-64' on Linux ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7146 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1759169] clean up Solaris port and allow C99 extension modules
Zooko O'Whielacronx zo...@zooko.com added the comment: I just tried to port a library of mine -- zfec -- to Nexenta (the Ubuntu variant built on top of OpenSolaris). I hit this bug because the Nexenta folks don't use all the patches that are applied to Python by the Solaris folks. My port of my library is blocked until the Nexenta folks apply this patch to their package of Python: https://bugs.launchpad.net/python/+bug/452667 # can't compile Python modules that use C99 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1759169 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6555] distutils config file should have the same name on both platforms and all scopes
New submission from Zooko O'Whielacronx zo...@zooko.com: Currently we have this: http://docs.python.org/install/#location-and-names-of-config-files The distutils config file can have one of four different names depending on which platform and which location. This makes it harder for people to remember the file's name and location, and more complicated for programs that want to detect it. Why not give it the same name on both platforms and all locations? -- assignee: tarek components: Distutils messages: 90860 nosy: tarek, zooko severity: normal status: open title: distutils config file should have the same name on both platforms and all scopes ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6556] HOME is not a standard environment variable on Windows
New submission from Zooko O'Whielacronx zo...@zooko.com: The distutils looks in an environment variable named HOME on Windows: http://docs.python.org/install/#location-and-names-of-config-files Windows does not by default create such a variable, so only if a user has manually configured one will it work. The standard variable for this purpose on Windows appears to be spelled USERPROFILE: http://en.wikipedia.org/wiki/Environment_variable -- assignee: tarek components: Distutils messages: 90861 nosy: tarek, zooko severity: normal status: open title: HOME is not a standard environment variable on Windows ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6556 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6555] distutils config file should have the same name on both platforms and all scopes
Zooko O'Whielacronx zo...@zooko.com added the comment: I was spurred to write this ticket today because of a conversation with J.P. Calderone: exarkun but, I have no clue where distutils.cfg goes on Windows -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6555 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6296] Native (and default) tarfile support for setup.py sdist in distutils on Windows
Zooko O'Whielacronx zo...@zooko.com added the comment: Antoine, when you say zip has better support everywhere, what do you mean? I don't want to put words in your mouth, but what I think of is that users maybe want to pack or unpack distributions with separate tools instead of with the Python tools. Is that it? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6296] Native (and default) tarfile support for setup.py sdist in distutils on Windows
Zooko O'Whielacronx zo...@zooko.com added the comment: I strongly favor a common approach instead of doing it differently on different platforms. (Aside: check out the gratuitous differences for names and locations of distutils config files: http://docs.python.org/install/index.html#location-and-names-of-config-files .) I weakly favor tar over zip. The current release of tar features LZMA compression, for example: $ ls -lS -rw-rw-r-- 1 zooko zooko 8355840 2009-06-17 09:48 allmydata-tahoe-1.4.1-r3916.tar -rw-rw-r-- 1 zooko zooko 2562835 2009-06-17 09:51 allmydata-tahoe-1.4.1-r3916.zip -rw-rw-r-- 1 zooko zooko 2383653 2009-06-17 09:48 allmydata-tahoe-1.4.1-r3916.tar.gz -rw-rw-r-- 1 zooko zooko 2250149 2009-06-17 09:49 allmydata-tahoe-1.4.1-r3916.tar.bz2 -rw-rw-r-- 1 zooko zooko 2223425 2009-06-17 09:49 allmydata-tahoe-1.4.1-r3916.tar.rz -rw-rw-r-- 1 zooko zooko 1818466 2009-06-17 09:52 allmydata-tahoe-1.4.1-r3916.7z -rw-rw-r-- 1 zooko zooko 1811698 2009-06-17 09:50 allmydata-tahoe-1.4.1-r3916.tar.xz Presumably the current tar module in Python doesn't yet support LZMA, but it is a good possibility for the future. On the other hand, ZIP is more widely used, e.g. by setuptools and especially by tools which were born in Windows world. -- nosy: +zooko ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6296 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6298] http://python.org/download says Python 2.4.5, but I think it means Python 2.4.6.
Changes by Zooko O'Whielacronx zo...@zooko.com: -- assignee: georg.brandl components: Documentation nosy: georg.brandl, zooko severity: normal status: open title: http://python.org/download says Python 2.4.5, but I think it means Python 2.4.6. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6298 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6280] calendar.timegm() belongs in time module, next to time.gmtime()
New submission from Zooko O'Whielacronx zo...@zooko.com: I've been struggling to write a function that takes UTC timestamps in ISO-8601 strings and returns UTC timestamps in unix-seconds-since-epoch. The first implementation used time.mktime() minus time.timezone (http://allmydata.org/trac/tahoe/browser/src/allmydata/util/time_format.py?rev=2424 ), but that doesn't work in London if the argument date is in a certain period during the 1970's. Then there was force-the-tz-to-UTC (http://allmydata.org/trac/tahoe/changeset/3911/src/allmydata/util/time_format.py ), but that doesn't work on Windows. Then there was force-the-tz-to-UTC-except-on-Windows (http://allmydata.org/trac/tahoe/changeset/3913/src/allmydata/util/time_format.py ), but that still doesn't work on Windows. Then there was this horrible hack of converting from string-iso-8601 to localseconds with time.mktime(), then converting from the resulting localseconds to an iso-8601 string, then converting the resulting string back to seconds with time.mktime(), then subtracting the two seconds values to find out the real offset between UTC-seconds and local-seconds for the current tz and for the time indicated by the argument (http://allmydata.org/trac/tahoe/changeset/3914/src/allmydata/util/time_format.py ). This actually works everywhere, but it is horrible. Finally, yesterday, someone pointed out to me that the inverse of time.gmtime() is located in the calendar module and is named calendar.timegm(). Now the implementation of our function is simple (http://allmydata.org/trac/tahoe/changeset/3915/src/allmydata/util/time_format.py ) I suggest that timegm() be moved to the time module next to its sibling gmtime(). -- components: Library (Lib) messages: 89334 nosy: zooko severity: normal status: open title: calendar.timegm() belongs in time module, next to time.gmtime() versions: Python 2.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6280 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6280] calendar.timegm() belongs in time module, next to time.gmtime()
Zooko O'Whielacronx zo...@zooko.com added the comment: Here is the ticket that tracked this issue within the Tahoe-LAFS project: http://allmydata.org/trac/tahoe/ticket/733 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6280 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5720] ctime: I don't think that word means what you think it means.
Zooko O'Whielacronx zo...@zooko.com added the comment: Okay, I posted to python-dev: http://mail.python.org/pipermail/python-dev/2009-June/090021.html -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5720 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: [Python-Dev] how GNU stow is complementary rather than alternative to distutils
following-up to my own post to mention one very important reason why anyone cares: On Sun, May 10, 2009 at 12:04 PM, Zooko Wilcox-O'Hearn zo...@zooko.com wrote: It is a beautiful, elegant hack because it is sooo dumb. It is also very nice to use the same tool to manage packages written in any programming language, provided only that they can build a directory tree of the right shape and content. And, you are not relying on the author of the package that you are installing to avoid accidentally or maliciously screwing up your system. You're not even relying on the authors of the *build system* (e.g. the authors of distutils or easy_install). You are relying *only* on GNU stow to avoid accidentally or maliciously screwing up your system, and GNU stow is very dumb, so it is easy to understand what it is going to do and why that isn't going to irreversibly screw up your system. That is: you don't run the build yourself and install into $prefix step as root. This is an important consideration for a lot of people, who absolutely refuse on principle to ever run sudo python ./setup.py on a system that they care about unless they wrote the setup.py script themselves. (Likewise they refuse to run sudo make install on packages written in C.) Regards, Zooko -- http://mail.python.org/mailman/listinfo/python-list
Re: [Python-Dev] .pth files are evil
.pth files are why I can't easily use GNU stow with easy_install. If installing a Python package involved writing new files into the filesystem, but did not require reading, updating, and re-writing any extant files such as .pth files, then GNU stow would Just Work with easy_install the way it Just Works with most things. Regards, Zooko -- http://mail.python.org/mailman/listinfo/python-list
Re: pyflakes, pylint, pychecker - and other tools
I like pyflakes. I haven't tried the others. I made a setuptools plugin named setuptools_pyflakes. If you install that package, then python ./setup.py flakes runs pyflakes on your package. Regards, Zooko -- http://mail.python.org/mailman/listinfo/python-list
Re: PEP 383: Non-decodable Bytes in System Character Interfaces
Thanks for writing this PEP 383, MvL. I recently ran into this problem in Python 2.x in the Tahoe project [1]. The Tahoe project should be considered a good use case showing what some people need. For example, the assumption that a file will later be written back into the same local filesystem (and thus luckily use the same encoding) from which it originally came doesn't hold for us, because Tahoe is used for file-sharing as well as for backup-and-restore. One of my first conclusions in pursuing this issue is that we can never use the Python 2.x unicode APIs on Linux, just as we can never use the Python 2.x str APIs on Windows [2]. (You mentioned this ugliness in your PEP.) My next conclusion was that the Linux way of doing encoding of filenames really sucks compared to, for example, the Mac OS X way. I'm heartened to see what David Wheeler is trying to persuade the maintainers of Linux filesystems to improve some of this: [3]. My final conclusion was that we needed to have two kinds of workaround for the Linux suckage: first, if decoding using the suggested filesystem encoding fails, then we fall back to mojibake [4] by decoding with iso-8859-1 (or else with windows-1252 -- I'm not sure if it matters and I haven't yet understood if utf-8b offers another alternative for this case). Second, if decoding succeeds using the suggested filesystem encoding on Linux, then write down the encoding that we used and include that with the filename. This expands the size of our filenames significantly, but it is the only way to allow some future programmer to undo the damage of a falsely- successful decoding. Here's our whole plan: [5]. Regards, Zooko [1] http://allmydata.org [2] http://allmydata.org/pipermail/tahoe-dev/2009-March/001379.html # see the footnote of this message [3] http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html [4] http://en.wikipedia.org/wiki/Mojibake [5] http://allmydata.org/trac/tahoe/ticket/534#comment:47 -- http://mail.python.org/mailman/listinfo/python-list
[issue5755] -Wstrict-prototypes is valid for Ada/C/ObjC but not for C++
New submission from Zooko O'Whielacronx zo...@zooko.com: A user of the Tahoe-LAFS project submitted a bug report to us, saying: I get lots of cc1plus: warning: command line option -Wstrict- prototypes is valid for Ada/C/ObjC but not for C++ when compiling A little googling shows that this gets reported frequently to other Python projects that have C++ code: http://trac.sagemath.org/sage_trac/ticket/425 http://www.mail-archive.com/matplotlib- us...@lists.sourceforge.net/msg03947.html Those other projects seem to think that this is distutils's problem, but I don't see evidence that any of them opened a ticket for distutils yet. -- assignee: tarek components: Distutils messages: 85972 nosy: tarek, zooko severity: normal status: open title: -Wstrict-prototypes is valid for Ada/C/ObjC but not for C++ ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5755 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5720] ctime: I don't think that word means what you think it means.
New submission from Zooko O'Whielacronx zo...@zooko.com: The stat module currently uses the st_ctime slot to hold two kinds values which are semantically different but which are frequently confused with one another. It chooses which kind of value to put in there based on platform -- Windows gets the file creation time and all other platforms get the ctime. The only sane way to use this API is then to switch on platform: if platform.system() == Windows: metadata[creation time] = s.st_ctime else: metadata[unix ctime] = s.st_ctime (That is an actual code snippet from the Allmydata-Tahoe project.) Many or even most programmers incorrectly think that unix ctime is file creation time, so instead of using the sane idiom above, they write the following: metadata[ctime] = s.st_ctime thus passing on the confusion to the users of their metadata, who may not know on which platform this metadata was created. This is the situation we have found ourselves in for the Allmydata-Tahoe project -- we now have a bunch of ctime values stored in our filesystem and no way to tell which kind they were. More and more filesystems such as ZFS and Macintosh apparently offer creation time nowadays. I propose the following changes: 1. Add a st_crtime field which gets populated on filesystems (Windows, ZFS, Mac) which can do so. That is hopefully not too controversial and we could proceed to do so even if the next proposal gets bogged down: 2. Add a st_unixctime field which gets populated *only* by the unix ctime and never by any other value (even on Windows, where the unix ctime *is* available even though nobody cares about it), and deprecate the hopelessly ambiguous st_ctime field. You may be interested in http://allmydata.org/trac/tahoe/ticket/628 (mtime and ctime: I don't think that word means what you think it means.) where the Allmydata-Tahoe project is carefully unpicking the mess we made for ourselves by confusing ctime with file-creation time. -- components: Library (Lib) messages: 85750 nosy: zooko severity: normal status: open title: ctime: I don't think that word means what you think it means. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5720 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1322] platform.dist() has unpredictable result under Linux
Zooko O'Whielacronx zo...@zooko.com added the comment: I just read back through this ticket, but I didn't understand exactly what MAL wanted to have different from what this Python function currently does: http://allmydata.org/trac/tahoe/browser/src/allmydata/__init__.py?rev=20081125155118-92b7f-f74fc964ebd9d3c59afde68b6688c56ce20cca39#L31 MAL, could you please restate the changes you want? By the way I think there is some confusion about what is standardized by LSB. As far as I know this one page is the complete spec: http://refspecs.freestandards.org/LSB_3.2.0/LSB-Core-generic/LSB-Core-generic/lsbrelease.html and it doesn't specify the existence of any file that we can parse. So the quote you mention: Linux System Base-compliant systems should have a file called /etc/lsb_release, which may be in addition to a distribution-specific file. is just wrong. More's the pity -- most implementations use a file named /etc/lsb-release, and we can parse that, and if we do it is much faster than executing the lsb_release executable in a subprocess. The slowness of invoking subprocess is why I was forced to back off from my original patch of merely using only what the LSB offers. A second problem with relying on LSB is, as I've mentioned, that some Linux distributions don't come (by default) with the de-facto-standard lsb_release executable although they do come with an /etc/lsb-release file. That's why my current strategy is: 1. Parse the de-facto-nearly-standard /etc/lsb-release file. 2. Ad-hoc techniques encoded into the Python Standard Library's platform.dist(). 3. Execute the de-jure-standard lsb_release in a subprocess. 4. Arch Linux -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1322 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1322] platform.dist() has unpredictable result under Linux
Zooko O'Whielacronx zo...@zooko.com added the comment: doko: thanks for your interest encouraging more formal and generic solutions to this. For what it is worth, the current version of my patch (used in Tahoe) is here: http://allmydata.org/trac/tahoe/browser/src/allmydata/__init__.py?rev=20081125155118-92b7f-f74fc964ebd9d3c59afde68b6688c56ce20cca39#L31 I had to add a special case for Arch Linux, which gets triggered after the three main cases. The cases currently are, in order: 1. Parse /etc/lsb-release (fast, semi-de-facto-standard, generic, hopefully a future de-jure-standard). 2. Invoke the Python Standard Library's platform.dist() (pros: fast, has lots of customized special cases for different linux distros, cons: has lots of customized special cases for different linux distros, gives bogus answers for Ubuntu and Arch Linux) 3. Subprocess execute lsb_release (pros: a real de-jure-standard! cons: slow, and is not actually a de-facto-standard since many important Linux installations don't come by default with the package that provides the de-jure-standard lsb_release executable, even though they do come by default with the de-facto-semi-standard /etc/lsb-release file). 4. Arch Linux -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1322 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5480] .egg-info = .pkg-info
New submission from Zooko O'Whielacronx zo...@zooko.com: The .egg-info files which are produced by distutils in Python = 2.5 are the only standard, cross-platform way for a Python package (distribution) to declare its name and version number in a machine-parseable way. Unfortunately, these files are named .egg-info even when the Python package in question was produced by distutils without setuptools, was never packaged as an egg, and is not installed as an egg. It has nothing to do with egg. The fact that the file has egg in its name causes most people to think that it has something to do with eggs, which leads to all sorts of problems including people removing the file and thus deleting the machine-parseable metadata about the Python package's name and version number. This ticket is a request that distutils start calling this file .pkg-info instead. Obviously we can't just stop including them under the name .egg-info since that would break existing usage, but we could start producing files under the name .pkg-info, and make .egg-info be a symlink or a copy of .pkg-info for backwards compatibility. Also of course we can update the documentation (if there is any) of what the .pkg-info file is. Since the current problems are mostly problems of communication and documentation, this simple change to the name of the file might go a long way to improving things. -- assignee: tarek components: Distutils messages: 83492 nosy: tarek, zooko severity: normal status: open title: .egg-info = .pkg-info type: feature request versions: Python 2.7 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5480 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5480] .egg-info = .pkg-info
Zooko O'Whielacronx zo...@zooko.com added the comment: Thank you. I've now read PEP 376. It is good. However, this same issue remains in PEP 376 like it does in today's distutils. If the new work in PEP 376 is going to continue to use the word egg in its filenames, then we need to send out an announcement of some sort to say HEY! Pay attention! This is actually a supported part of the core of Python even though it has the word 'egg' in it.. I suppose including a sentence to that effect in the distutils release announcements and in the What's new in Python document will suffice. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5480 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5480] .egg-info = .pkg-info
Zooko O'Whielacronx zo...@zooko.com added the comment: By the way, here is the ticket on allmydata.org Tahoe where this issue was bugging me which is why I opened this ticket: http://allmydata.org/trac/tahoe/ticket/149 # unable to use pre-installed non-setuptools^H^H^H^H^H^H^H^H^H^Hdistutils-aware nevow -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5480 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1759169] clean up Solaris port and allow C99 extension modules
Zooko O'Whielacronx zo...@zooko.com added the comment: Martin (sometimes called MvL) mentioned some specific issues about e.g. #pragma redefine_extname sigwait __posix_sigwait I didn't understand exactly what MvL's concern was, and I don't know off the top of my head how Solaris sigwait behavior differs with _XOPEN_SOURCE, and I haven't taken the time to investigate yet, so I haven't replied. (Of course, last time I was in that state and I dropped the ball, this ticket stayed quiet for a full year...) Perhaps John Levon could inquire what semantics MvL is concerned about and let him know what the Solaris behavior is. I think it would be cool if the python executable that people use on Solaris were closer to upstream python. Currently, as I've mentioned, the Solaris folks maintain some kind of tool (I think it is a perl script) which searches and replaces #define _XOPEN_SOURCE to empty as well as applying a few patches. It would be great if the Solaris and Python folks would cooperate to close that gap. Speaking of which, if John Levon or other Solaris folk are reading -- where are those patches? This file mentions a bunch of patches, starting with: http://src.opensolaris.org/source/xref/jds/spec-files/trunk/SUNWPython.spec 28 Patch1: Python-01-solaris-lib-dirs.diff But I can't seem to find Python-01-solaris-lib-dirs.diff. Thanks! --Zooko ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue1759169 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4706] try to build a C module, but don't worry if it doesn't work
Zooko O'Whielacronx zo...@zooko.com added the comment: Hey check it out -- simplejson can try to build its extension module, and when it fails to compile (in this case because there is no Python.h), then it prints out a warning message and finishes a successful build: Added file: http://bugs.python.org/file12910/build-with-no-Python.h.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4706 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4706] try to build a C module, but don't worry if it doesn't work
Zooko O'Whielacronx zo...@zooko.com added the comment: Oh! And simplejson is able to cleanly fall back to pure Python when gcc is not found, as well. Perhaps Twisted http://twistedmatrix.com/trac/ticket/3586 could use simplejson's approach. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4706 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4706] try to build a C module, but don't worry if it doesn't work
Changes by Zooko O'Whielacronx zo...@zooko.com: Added file: http://bugs.python.org/file12911/build-with-no-gcc.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4706 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4706] try to build a C module, but don't worry if it doesn't work
Zooko O'Whielacronx zo...@zooko.com added the comment: Here's Twisted failing to build when gcc is not installed: Added file: http://bugs.python.org/file12912/Twisted-build-with-no-gcc.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4706 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4706] try to build a C module, but don't worry if it doesn't work
Zooko O'Whielacronx zo...@zooko.com added the comment: Here's Twisted failing to build because Python.h isn't found: Added file: http://bugs.python.org/file12913/Twisted-build-no-Python.h.txt ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4706 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4706] try to build a C module, but don't worry if it doesn't work
Zooko O'Whielacronx zo...@zooko.com added the comment: Tarek: Yes, failure to build an extension module for any reason should definitely cause build to fail by default. However, many Python packages seem to come with optional extension modules, typically for a performance speed-up. simplejson, zope.interface, and twisted all do this (these are three packages that allmydata-tahoe depends on). My own pyutil package used to do this, but I eventually just removed the optional C extension module in order to avoid this problem. So, for those it would be good if the package setup.py could specify that an extension module is optional and that failure to build it for any reason does not mean failure of the build. Note that twisted's setup.py already attempts to do this for certain of its extension modules, and if I understand correctly it is able to detect and handle failure to build if gcc fails, but not if gcc is absent. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4706 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4706] try to build a C module, but don't worry if it doesn't work
New submission from Zooko O'Whielacronx zo...@zooko.com: I was trying to install Twisted on my son's OLPC. Twisted comes with a handful of C extensions which are all optional and most of which are not expected to compile on this platform anyway (they are platform-specific for Mac or Windows). If my son's OLPC had a C compiler installed, then it would attempt to build those modules, fail, and proceed to install Twisted. However, since it doesn't have a C compiler, then instead it stops with an error. It would be nice if Twisted could tell distutils to try to build a module, and report whether the build succeeded, but don't stop the entire setup(). http://twistedmatrix.com/trac/ticket/3586 -- components: Distutils messages: 78106 nosy: zooko severity: normal status: open title: try to build a C module, but don't worry if it doesn't work ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4706 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3871] cross building python for mingw32 with distutils
Zooko O'Whielacronx [EMAIL PROTECTED] added the comment: I'm interested in this patch so I'm adding myself to the Nosy list. -- nosy: +zooko ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3871 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4295] closing stdout in a child process on cygwin means that process doesn't receive bytes from stdin anymore. I think.
Zooko O'Whielacronx [EMAIL PROTECTED] added the comment: Corinna Vinschen of cygwin requests a smaller test case: http://www.cygwin.com/ml/cygwin/2008-11/msg00166.html ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4295 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1675] Race condition in os.makedirs
Zooko O'Whielacronx [EMAIL PROTECTED] added the comment: Here's the version of this that I've been using for almost a decade now: http://allmydata.org/trac/pyutil/browser/pyutil/pyutil/fileutil.py?rev=127#L241 Actually I used to have a bigger version that could optionally require certain things of the mode of the directory, but it turned out that I wasn't going to need it. def make_dirs(dirname, mode=0777): An idempotent version of os.makedirs(). If the dir already exists, do nothing and return without raising an exception. If this call creates the dir, return without raising an exception. If there is an error that prevents creation or if the directory gets deleted after make_dirs() creates it and before make_dirs() checks that it exists, raise an exception. tx = None try: os.makedirs(dirname, mode) except OSError, x: tx = x if not os.path.isdir(dirname): if tx: raise tx raise exceptions.IOError, unknown error prevented creation of directory, or deleted the directory immediately after creation: %s % dirname # careful not to construct an IOError with a 2-tuple, as that has a special meaning... -- nosy: +zooko ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1675 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4295] closing stdout in a child process on cygwin means that process doesn't receive bytes from stdin anymore. I think.
Zooko O'Whielacronx [EMAIL PROTECTED] added the comment: I opened a ticket on the cygwin issue tracker: http://sourceware.org/bugzilla/show_bug.cgi?id=7017 # closing stdout in a child python process means that process doesn't receive bytes from stdin anymore. I think. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4295 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4172] distutils too picky about cygwin ld's version number
New submission from Zooko O'Whielacronx [EMAIL PROTECTED]:
When I build an extension module with cygwin g++ and compiler=mingw32
in my distutils config file, the build fails with:
File
c:\python25\lib\site-packages\setuptools-0.6c9.egg\setuptools\command\build_ext.py,
line 46, in run
_build_ext.run(self)
File c:\Python25\lib\distutils\command\build_ext.py, line 264, in run
force=self.force)
File c:\Python25\lib\distutils\ccompiler.py, line 1175, in new_compiler
return klass (None, dry_run, force)
File c:\Python25\lib\distutils\cygwinccompiler.py, line 292, in __init__
CygwinCCompiler.__init__ (self, verbose, dry_run, force)
File c:\Python25\lib\distutils\cygwinccompiler.py, line 84, in __init__
get_versions()
File c:\Python25\lib\distutils\cygwinccompiler.py, line 424, in
get_versions
ld_version = StrictVersion(result.group(1))
File c:\Python25\lib\distutils\version.py, line 40, in __init__
self.parse(vstring)
File c:\Python25\lib\distutils\version.py, line 107, in parse
raise ValueError, invalid version number '%s' % vstring
ValueError: invalid version number '2.18.50.20080625'
If I change StrictVersion to LooseVersion in cygwinccompiler.py,
then the build succeeds:
C:\playground\allmydata\tahoe\installtahoe\allmydata-tahoe-1.2.0-r3092-SUMOdiff
-u C:\Python25\Lib\distutils\cygwinccompiler.py
C:\Python25\Lib\distutils\cyg
winccompiler.py.new
--- C:\Python25\Lib\distutils\cygwinccompiler.py2008-10-22
07:09:26.765625000 -0600
+++ C:\Python25\Lib\distutils\cygwinccompiler.py.new2008-10-22
07:09:06.234375000 -0600
@@ -398,7 +398,7 @@
Try to find out the versions of gcc, ld and dllwrap.
If not possible it returns None for it.
-from distutils.version import StrictVersion
+from distutils.version import LooseVersion, StrictVersion
from distutils.spawn import find_executable
import re
@@ -421,7 +421,7 @@
out.close()
result = re.search('(\d+\.\d+(\.\d+)*)',out_string)
if result:
-ld_version = StrictVersion(result.group(1))
+ld_version = LooseVersion(result.group(1))
else:
ld_version = None
else:
@@ -433,7 +433,7 @@
out.close()
result = re.search(' (\d+\.\d+(\.\d+)*)',out_string)
if result:
-dllwrap_version = StrictVersion(result.group(1))
+dllwrap_version = LooseVersion(result.group(1))
else:
dllwrap_version = None
--
components: Distutils
messages: 75079
nosy: zooko
severity: normal
status: open
title: distutils too picky about cygwin ld's version number
type: behavior
versions: Python 2.5
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4172
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1759169] clean up Solaris port and allow C99 extension modules
Zooko O'Whielacronx [EMAIL PROTECTED] added the comment: Sorry I didn't get back to this ticket, MvL. Recently someone trying to build the Tahoe distributed filesystem on Solaris had a problem: http://allmydata.org/pipermail/tahoe-dev/2008-September/000789.html They had built Python 2.5 themselves from source. It turned out to be this issue: http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6395191 Which is that g++ on Solaris fails when _XOPEN_SOURCE_EXTENDED is defined. This reminded me of this ticket, so here I am again. So, here is the port of Python to Solaris that is maintained by the Solaris engineers: http://src.opensolaris.org/source/xref/jds/spec-files/trunk/SUNWPython.spec It contains the following stanza: 146 # These #define's break the build with gcc and cause problems when 147 # building c99 compliant python modules 148 perl -pi -e 's/^(#define _POSIX_C_SOURCE.*)/\/* $1 *\//' pyconfig.h 149 perl -pi -e 's/^(#define _XOPEN_SOURCE.*)/\/* $1 *\//' pyconfig.h 150 perl -pi -e 's/^(#define _XOPEN_SOURCE_EXTENDED.*)/\/* $1 *\//' pyconfig.h This shows that the Python that comes with Solaris has, for a long time now, had these #define's stripped out of the pyconfig.h. Why? Well, I asked the Solaris engineers about it on IRC, and they were very nice and helpful, and they said, if I am remembering correctly, that Solaris doesn't use these #define's in the way that we Python people think. #defining _XOPEN_SOURCE_EXTENDED on Solaris is *not* understood by the Solaris runtime as a request for XPG4v2 (UNIX98) features to be supported, but more as a declaration that the code we are compiling was written for XPG4v2. So, for example, this disables C99, because after all C99 wasn't available in XPG4v2. It also apparently breaks g++, but that is by accident rather than design. In any case, the Solaris engineers assured me that rather than figuring out which unixy features are turned on by which #defines, we should instead #define __EXTENSIONS__ and then assume all the features we can eat. Whether this is true in general remains to be seen, but apparently _POSIX_C_SOURCE, _XOPEN_SOURCE, and _XOPEN_SOURCE_EXTENDED, at least, are proven to be unnecessary on Solaris by the fact that the version of Python that has been shipping with Solaris for who knows how long has them stripped out. Therefore, I propose to apply my patch so that we do not turn on _XOPEN_SOURCE but we do turn on __EXTENSIONS__, when building for Solaris. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1759169 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1322] platform.dist() has unpredictable result under Linux
Zooko O'Whielacronx [EMAIL PROTECTED] added the comment: Well, for what it is worth I've updated the custom detect linux distribution code in Tahoe yet again. The current version first tries to parse /etc/lsb-version (fast, gives a good answer on Ubuntu, and hopefully at least semi-standardized), then invokes platform.dist() (fast, customized, but gives a bad answer on Ubuntu), and then if platform.dist() returns an empty distribution or empty release (which I'm not sure if that is even possible), it invokes lsb_release -a in a subprocess (slow, totally standardized). Here's the patch: http://allmydata.org/trac/tahoe/changeset/2981 Running it on the Tahoe buildbot shows that we currently have the followings kinds of buildslaves: http://allmydata.org/buildbot/waterfall?show_events=true Linux-Ubuntu_6.06-i686-32bit, Linux-Ubuntu_6.10-i686-32bit, Linux-Ubuntu_7.10-i686-32bit, SunOS-5.11-i86pc-i386-32bit, Linux-Ubuntu_7.04-i686-32bit, Linux-debian_4.0-i686-32bit, Linux-Ubuntu_8.04-i686-32bit, Darwin-9.4.0-i386-32bit, Linux-Ubuntu_8.04-x86_64-64bit, Windows-XP-5.1.2600-SP2, CYGWIN_NT-5.1-1.5.25-0.156-4-2-i686-32bit-WindowsPE ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1322 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3937] platform.dist(): detect Linux distribution version in a robust, standard way
Zooko O'Whielacronx [EMAIL PROTECTED] added the comment: Here is an updated version of my patch which tries parsing /etc/lsb-release first and only if that fails tries executing lsb_release. The reason is that executing lsb_release in a subprocess takes half-a-second on my high-performance Athlon64 Ubuntu workstation, and also that some installations have /etc/lsb-release but not lsb_release. See the docstring for more details about why to do it this way and exactly what it does. Added file: http://bugs.python.org/file11573/dist.patch.txt ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3937 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1322] platform.dist() has unpredictable result under Linux
Zooko O'Whielacronx [EMAIL PROTECTED] added the comment: Please see also #3937 for a patch which first tries to parse /etc/lsb-release, then tries to execute lsb_release, then falls back to the old behavior of platform.dist(). (Note that parsing the file named /etc/lsb-release is not guaranteed to work by the Linux Standard Base spec, but that executing lsb_release is. On the other hand, the former currently seems to work on more Debian/Ubuntu installations than the latter does, and invoking lsb_release in a subprocess takes significantly more time.) Please see the links to the Linux Standard Base specifications which I posted in #3937. (The specification was originally published in 2001, so most of the Linux systems that future versions of Python will be deployed on were developed long after that specification was written.) My patch, in #3937, is against the python-release25-maint branch. I tested the patch locally and it worked, and then I wrote a patch for my project -- allmydata.org Tahoe -- which does something similar: http://allmydata.org/trac/tahoe/browser/src/allmydata/__init__.py?rev=2976 I committed that change to Tahoe, and it was automatically tested on eleven different systems by our buildbot: http://allmydata.org/buildbot/waterfall The resulting distribution-identification can be seen in the logs of those tests. For any machine on the waterfall, click on the test.log hyperlink and search in text for tahoe versions:. For example, here is the one for our Debian etch machine -- debian, 4.0: http://allmydata.org/buildbot/builders/etch/builds/1205/steps/test/logs/test.log here is the one for the Ubuntu dapper machine -- Ubuntu, 6.06: http://allmydata.org/buildbot/builders/dapper/builds/1773/steps/test/logs/test.log and here is the result for my Ubuntu hardy workstation -- Ubuntu, 8.04: http://allmydata.org/buildbot/builders/zooko%20yukyuk%20hardy/builds/216/steps/test/logs/test.log -- nosy: +zooko ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1322 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1322] platform.dist() has unpredictable result under Linux
Zooko O'Whielacronx [EMAIL PROTECTED] added the comment:
Okay, per MAL's request over on #3937, I tried
platform.get_linux_distribution() on the current svn trunk (which I
assume is the version that is about to become python 2.6). It gave the
same not-so-great answer as platform.dist() used to: ('debian',
'lenny/sid', ''). I then ported my patch to trunk (attached), and it
gave the answer I wanted: ('Ubuntu', '8.04', 'hardy').
Note that technique of parsing /etc/lsb-release or invoking lsb_release
is more standard and generic and easier to maintain than the current
technique of trying a series of specific file parses for specific
supported distributions. For example, if some new Linux distribution
is invented, or if this is tried on a distribution that isn't in the
supported list (has anyone tried this on Foresight Linux yet?), then my
technique has a good chance of working on that distribution, where the
current technique has a higher chance of accidentally mis-identifying it
as some other distribution, for example the way that the current trunk
mis-identifies Ubuntu 8.04 as Debian lenny/sid.
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1322
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1322] platform.dist() has unpredictable result under Linux
Changes by Zooko O'Whielacronx [EMAIL PROTECTED]: Added file: http://bugs.python.org/file11574/dist.patch.txt ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1322 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1322] platform.dist() has unpredictable result under Linux
Zooko O'Whielacronx [EMAIL PROTECTED] added the comment: MAL: why do you say it is better to look for /etc/$supportedplatform-release files first instead of looking for /etc/lsb-release first? I do not know if /etc/lsb-release is suitably generic -- I've tried it only on a few platforms. I do know that executing lsb_release is suitably generic since it is standard, but I prefer not to try it first since it imposes about half-a-second delay. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1322 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1322] platform.dist() has unpredictable result under Linux
Zooko O'Whielacronx [EMAIL PROTECTED] added the comment: Because that's exactly what lsb_release does as well. You must know something about common lsb_release implementations that I don't. As far as I saw in the LSB documentation, it is required to print out information in a certain format, but how it is implemented is totally up to the distribution in question. You give examples of SuSE and Fedora as not having /etc/lsb-release files, and I'm sure you are right, but I happen to know that both of them have compliant lsb_release executables (and that they have had for many releases). So, the patch that I've submitted will definitely work correctly for those two distributions, although it will pay the price of having to spawn a subprocess and then wait for the lsb_release executable to do its work (however it does it). However, presumably your SuSE- and Fedora- specific techniques will give correct answers on those platforms faster than the generic lsb_release would. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue1322 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3937] platform.dist(): detect Linux distribution version in a robust, standard way
New submission from Zooko O'Whielacronx [EMAIL PROTECTED]:
platform.dist() returns ('debian', 'lenny/sid', '') on my Ubuntu 8.04
Hardy system. Investigating shows that there are a few techniques in
platform.py to parse the version-number-files of different Linux
distributions. This patch adds a command to try executing lsb_release
first of all. lsb_release is the standard way to do this, originally
published in 2001:
http://refspecs.freestandards.org/LSB_1.0.0/gLSB/lsbrelease.html
and currently standardized here:
http://refspecs.freestandards.org/LSB_3.2.0/LSB-Core-generic/LSB-Core-generic/lsbrelease.html
If invoking lsb_release results in exit code 0 and some non-empty,
non-all-whitespace string on stdout, then dist() returns that. Else,
dist falls back to the old (current) hacks.
There is a drawback to this: invoking three successive subprocesses
takes a bit of time. Hopefully nobody needs to invoke platform.dist()
in a time-critical moment...
With this patch, platform.dist() return:
('Ubuntu', '8.04', 'hardy')
Oh, this patch also updates the docstring of dist() to explain what is
meant by distribution, version, and id.
--
components: Library (Lib)
files: dist.patch.txt
messages: 73601
nosy: zooko
severity: normal
status: open
title: platform.dist(): detect Linux distribution version in a robust, standard
way
type: behavior
versions: Python 2.4, Python 2.5, Python 2.6, Python 3.0
Added file: http://bugs.python.org/file11562/dist.patch.txt
___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue3937
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3937] platform.dist(): detect Linux distribution version in a robust, standard way
Zooko O'Whielacronx [EMAIL PROTECTED] added the comment: Here's a new version of this patch which differs only in having slightly more correct documentation. Added file: http://bugs.python.org/file11563/dist.patch.txt ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3937 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3016] tarfile.py incurs exception from self.chmod() when tarball has g+s
New submission from Zooko O'Whielacronx [EMAIL PROTECTED]: As reported at https://bugs.launchpad.net/pyopenssl/+bug/236190 , tarfile.py incurs an Operation not permitted exception (on Mac OS 10.4) when it tries to untar the pyOpenSSL-0.6.tar.gz tarball, because that tarball has directories in it marked as having the g+s bit. (Why this leads to an Operation not permitted exception, I don't know.) -- messages: 67563 nosy: zooko severity: normal status: open title: tarfile.py incurs exception from self.chmod() when tarball has g+s type: behavior ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue3016 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1759997] poll() on cygwin sometimes fails [PATCH]
Zooko O'Whielacronx added the comment: This bug was fixed in cygwin 1.5.25-7, released 2007-12-17. _ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1759997 _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1104021] wishlist: os.feed_urandom(input)
Zooko O'Whielacronx added the comment: No, I don't want to work on a patch for this at this time. In fact, my current strategy with regard to random bits doesn't require this functionality, in general. _ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1104021 _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1074462] Irregular behavior of datetime.__str__()
Zooko O'Whielacronx added the comment: Here is a note for the next person who comes to this ticket wondering why isoformat() exhibits this slightly un-Pythonic behavior. If you want to use isoformat() to produce, for example, timestamps for your logfiles, you'll need to do something like the following. (I do hope you noticed the documentation and didn't use isoformat() in the naive way, or your log files will very rarely have a different format than you expected.) d = datetime.datetime.utcfromtimestamp(when) if d.microsecond: return d.isoformat( )[:-3]+Z else: return d.isoformat( ) + .000Z http://allmydata.org/trac/tahoe/browser/src/allmydata/node.py#L21 -- nosy: +zooko _ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1074462 _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1074462] Irregular behavior of datetime.__str__()
Zooko O'Whielacronx added the comment: I meant that it special-cases .microseconds == 0. If I want to produce a custom output format using Python Standard Library, I expect to have to slice, add my own fields and so forth, but I don't expect to need an if to handle a special-case that is there for improving the appearance to human readers. That's something I had to do a lot more often when I worked in Perl. Even if the cost of changing the definition of isoformat() is too high at this point, I still wanted to post my code from http://allmydata.org as an example to other users so that they can use isoformat() safely. _ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1074462 _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1074462] Irregular behavior of datetime.__str__()
Zooko O'Whielacronx added the comment: Thank you for the one-liner. I was about to use it in the allmydata.org project, but I remembered that my programming partner would probably prefer the larger but more explicit if:else: over the clever one-liner. Perhaps it will be useful to someone else. I'll have a look at issue1158. _ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1074462 _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue766910] fix one or two bugs in trace.py
Zooko O'Whielacronx added the comment:
Hi! Sorry it took me so long to look at this. I just checked the
source in current trunk, and the relevant code is the same so this
patch is still useful. (See the initial post for details.)
Here is an updated version of the patch which simply removes some
dead code and updates a URL:
regards,
Zooko
diff -rN -u old-up/setuptools-0.6c7/ez_setup.py new-up/
setuptools-0.6c7/ez_setup.py
--- old-up/setuptools-0.6c7/ez_setup.py 2007-09-28 16:41:24.0
-0600
+++ new-up/setuptools-0.6c7/ez_setup.py 2007-09-28 16:41:25.0
-0600
@@ -1,4 +1,4 @@
-#!python
+#!/usr/bin/env python
Bootstrap setuptools installation
If you want to use setuptools in your package's setup.py, just
include this
@@ -13,7 +13,7 @@
This file can also be run as a script to install or upgrade setuptools.
-import sys
+import os, re, subprocess, sys
DEFAULT_VERSION = 0.6c7
DEFAULT_URL = http://pypi.python.org/packages/%s/s/setuptools/;
% sys.version[:3]
@@ -44,8 +44,6 @@
'setuptools-0.6c6-py2.5.egg': 'b2f8a7520709a5b34f80946de5f02f53',
}
-import sys, os
-
def _validate_md5(egg_name, data):
if egg_name in md5_data:
from md5 import md5
@@ -58,6 +56,42 @@
sys.exit(2)
return data
+# The following code to parse versions is copied from
pkg_resources.py so that
+# we can parse versions without importing that module.
+component_re = re.compile(r'(\d+ | [a-z]+ | \.| -)', re.VERBOSE)
+replace = {'pre':'c',
'preview':'c','-':'final-','rc':'c','dev':'@'}.get
+
+def _parse_version_parts(s):
+for part in component_re.split(s):
+part = replace(part,part)
+if not part or part=='.':
+continue
+if part[:1] in '0123456789':
+yield part.zfill(8)# pad for numeric comparison
+else:
+yield '*'+part
+
+yield '*final' # ensure that alpha/beta/candidate are before final
+
+def parse_version(s):
+parts = []
+for part in _parse_version_parts(s.lower()):
+if part.startswith('*'):
+if part'*final': # remove '-' before a prerelease tag
+while parts and parts[-1]=='*final-': parts.pop()
+# remove trailing zeros from each series of numeric parts
+while parts and parts[-1]=='':
+parts.pop()
+parts.append(part)
+return tuple(parts)
+
+def setuptools_is_new_enough(required_version):
+Return True if setuptools is already installed and has a version
+number = required_version.
+sub = subprocess.Popen([sys.executable, -c, import
setuptools;print setuptools.__version__], stdout=subprocess.PIPE)
+verstr = sub.stdout.read().strip()
+ver = parse_version(verstr)
+return ver and ver = parse_version(required_version)
def use_setuptools(
version=DEFAULT_VERSION, download_base=DEFAULT_URL,
to_dir=os.curdir,
@@ -74,32 +108,11 @@
this routine will print a message to ``sys.stderr`` and raise
SystemExit in
an attempt to abort the calling script.
-try:
-import setuptools
-if setuptools.__version__ == '0.0.1':
-print sys.stderr, (
-You have an obsolete version of setuptools installed.
Please\n
-remove it from your system entirely before rerunning
this script.
-)
-sys.exit(2)
-except ImportError:
+if not setuptools_is_new_enough(version):
egg = download_setuptools(version, download_base, to_dir,
download_delay)
sys.path.insert(0, egg)
import setuptools; setuptools.bootstrap_install_from = egg
-import pkg_resources
-try:
-pkg_resources.require(setuptools=+version)
-
-except pkg_resources.VersionConflict, e:
-# XXX could we install in a subprocess here?
-print sys.stderr, (
-The required version of setuptools (=%s) is not
available, and\n
-can't be installed while this script is running. Please
install\n
- a more recent version first.\n\n(Currently using %r)
-) % (version, e.args[0])
-sys.exit(2)
-
def download_setuptools(
version=DEFAULT_VERSION, download_base=DEFAULT_URL,
to_dir=os.curdir,
delay = 15
@@ -150,9 +163,14 @@
def main(argv, version=DEFAULT_VERSION):
Install or upgrade setuptools and EasyInstall
-try:
-import setuptools
-except ImportError:
+if setuptools_is_new_enough(version):
+if argv:
+from setuptools.command.easy_install import main
+main(argv)
+else:
+print Setuptools version,version,or greater has been
installed.
+print '(Run ez_setup.py -U setuptools to reinstall or
upgrade.)'
+else:
egg = None
try:
egg = download_setuptools(version, delay=0)
@@ -162,31 +180,6 @@
finally:
if egg and os.path.exists(egg):
os.unlink(egg
[issue766910] fix one or two bugs in trace.py
Zooko O'Whielacronx added the comment: Here is an updated version of the patch which simply removes some dead code and updates a URL: regards, Zooko diff -rN -u old-up/setuptools-0.6c7/ez_setup.py new-up/ setuptools-0.6c7/ez_setup.py --- old-up/setuptools-0.6c7/ez_setup.py 2007-09-28 16:41:24.0 -0600 +++ new-up/setuptools-0.6c7/ez_setup.py 2007-09-28 16:41:25.0 -0600 Oops, the in-lined patch contents were a different patch entirely, but the attached patch file was correct. Just for completeness, here is the correct in-lined patch contents: Index: Lib/trace.py === --- Lib/trace.py(revision 58282) +++ Lib/trace.py(working copy) @@ -85,7 +85,12 @@ -r, --report Generate a report from a counts file; do not execute any code. `--file' must specify the results file to read, which must have been created in a previous run - with `--count --file=FILE'. + with `--count --file=FILE'. If --coverdir is not + specified, the .cover files will be written into the + directory that the modules were in when the report was + generated. Whether or not --coverdir is specified, + --report will always create the cover file directory if + necessary. Modifiers: -f, --file=file File to accumulate counts over several runs. @@ -197,6 +202,33 @@ filename, ext = os.path.splitext(base) return filename +# The following function is copied from the fileutil module from the pyutil +# project: +# http://pypi.python.org/pypi/pyutil +# We use this function instead of os.makedirs() so that we don't get a +# spurious exception when someone else creates the directory at the same +# moment we do. (For example, another thread or process that is also running +# trace.) +def make_dirs(dirname, mode=0777): + +A threadsafe and idempotent version of os.makedirs(). If the dir already +exists, do nothing and return without raising an exception. If this call +creates the dir, return without raising an exception. If there is an +error that prevents creation or if the directory gets deleted after +make_dirs() creates it and before make_dirs() checks that it exists, raise +an exception. + +tx = None +try: +os.makedirs(dirname, mode) +except OSError, x: +tx = x + +if not os.path.isdir(dirname): +if tx: +raise tx +raise exceptions.IOError, unknown error prevented creation of directory, or deleted the directory immediately after creation: % s % dirname # careful not to construct an IOError with a 2-tuple, as that has a special meaning... + class CoverageResults: def __init__(self, counts=None, calledfuncs=None, infile=None, callers=None, outfile=None): @@ -290,15 +322,15 @@ if filename.endswith((.pyc, .pyo)): filename = filename[:-1] -if coverdir is None: +if coverdir is not None: +dir = coverdir +modulename = fullmodname(filename) +else: dir = os.path.dirname(os.path.abspath(filename)) modulename = modname(filename) -else: -dir = coverdir -if not os.path.exists(dir): -os.makedirs(dir) -modulename = fullmodname(filename) +make_dirs(dir) + # If desired, get a list of the line numbers which represent # executable content (returned as a dict for better lookup speed) if show_missing: Tracker [EMAIL PROTECTED] http://bugs.python.org/issue766910 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
