Re: [Python-Dev] 2.4.4: backport classobject.c HAVE_WEAKREFS?
Fredrik Lundh wrote:
a dynamic registration approach would be even better, with a single entry point
used to register all methods and hooks your C extension has implemented, and
code on the other side that builds a properly initialized type descriptor from that
set, using fallback functions and error stubs where needed.
I knocked out a prototype of this last week, emailed Mr. Lundh about
it, then forgot about it. Would anyone be interested in taking a peek
at it?
I only changed one file to use this new-style initialization,
sha256module.c. The resulting init_sha256() looks like this:
PyMODINIT_FUNC
init_sha256(void)
{
PyObject *m;
SHA224type = PyType_New("_sha256.sha224", sizeof(SHAobject), NULL);
if (SHA224type == NULL)
return;
PyType_SetPointer(SHA224type, pte_dealloc, SHA_dealloc);
PyType_SetPointer(SHA224type, pte_methods, SHA_methods);
PyType_SetPointer(SHA224type, pte_members, SHA_members);
PyType_SetPointer(SHA224type, pte_getset, SHA_getseters);
if (PyType_Ready(SHA224type) 0)
return;
SHA256type = PyType_New("_sha256.sha256", sizeof(SHAobject), NULL);
if (SHA256type == NULL)
return;
PyType_SetPointer(SHA256type, pte_dealloc, SHA_dealloc);
PyType_SetPointer(SHA256type, pte_methods, SHA_methods);
PyType_SetPointer(SHA256type, pte_members, SHA_members);
PyType_SetPointer(SHA256type, pte_getset, SHA_getseters);
if (PyType_Ready(SHA256type) 0)
return;
m = Py_InitModule("_sha256", SHA_functions);
if (m == NULL)
return;
}
In a way this wasn't really a good showpiece for my code. The
"methods", "members", and "getseters" structs still need to be passed
in. However, I did change all four "as_" structures so you can set
those directly. For instance, the "concat" as_sequence method for a
PyString object would be set using
PyType_SetPointer(PyString_Type, pte_sequence_concat,
string_concat);
(I actually converted the PyString object to my new code, but had
chicken-and-egg initialization problems as a result and backed out of
it. The code is still in the branch, just commented out.)
Patch available for interested parties,
larry
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes and win64
Thomas Heller schrieb (this was before Python 2.5 had been released): The _ctypes extension module does currently not even build on Win64. I'm (slowly) working on this (for AMD64, not for itanium), but it may take a good while before it is stable - It is not even fully implemented currently. The win64 msi installer installs the ctypes package anyway, but it cannot be imported. I suggest that it should be removed from the 2.5 win64 msi installers, so that at least, when it is ready, can be installed as separate package. Then, Martin changed the win64 msi installer to exclude the ctypes package when the _ctypes.pyd extension does not exist because it was not built. In the meantime I have integrated patches (in the trunk) so that _ctypes can be built for win64/AMD64, and does even work. Can these changes be merged into release25-maint? IMO this is low-risk because they contain only small changes to the files in Modules/_ctypes/libffi_msvc, plus *some* changes to support the Windows LP64 model. I would prefer to merge these changes into release25-maint, because I want to also release the standalone ctypes packages from this branch (using it with svn:externals from somewhere else). The official Python 2.5.x win64/AMD64 windows installers should still *not* contain the ctypes package, but they could install it separately. Thanks, Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes and win64
[Resent after subscribing to python-dev with this new email address, sorry if it appears twice] Thomas Heller schrieb (this was before Python 2.5 had been released): The _ctypes extension module does currently not even build on Win64. I'm (slowly) working on this (for AMD64, not for itanium), but it may take a good while before it is stable - It is not even fully implemented currently. The win64 msi installer installs the ctypes package anyway, but it cannot be imported. I suggest that it should be removed from the 2.5 win64 msi installers, so that at least, when it is ready, can be installed as separate package. Then, Martin changed the win64 msi installer to exclude the ctypes package when the _ctypes.pyd extension does not exist because it was not built. In the meantime I have integrated patches (in the trunk) so that _ctypes can be built for win64/AMD64, and does even work. Can these changes be merged into release25-maint? IMO this is low-risk because they contain only small changes to the files in Modules/_ctypes/libffi_msvc, plus *some* changes to support the Windows LP64 model. I would prefer to merge these changes into release25-maint, because I want to also release the standalone ctypes packages from this branch (using it with svn:externals from somewhere else). The official Python 2.5.x win64/AMD64 windows installers should still *not* contain the ctypes package, but they could install it separately. Thanks, Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python unit tests failing on Pybots farm
On 10/19/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Grig Gheorghiu schrieb: OK, I deleted the checkout directory on one of my buidslaves and re-ran the build steps. The tests passed. So my conclusion is that a full rebuild is needed for the tests to pass after the last checkins (which included files such as configure and configure.in). Indeed, you had to re-run configure. There was a bug where -Werror was added to the build flags, causing several configure tests to fail (most notably, it would determine that there's no memmove on Linux). Maybe the makefiles should be modified so that a full rebuild is triggered when the configure and configure.in files are changed? The makefiles already do that: if configure changes, a plain make will first re-run configure. Well, that didn't trigger a full rebuild on the Pybots buildslaves though. At this point, I'll have to tell all the Pybots owners to delete their checkout directories and start a new build. Not necessarily. You can also ask, at the buildbot GUI, that a non-existing branch is build. This should cause the checkouts to be deleted (and then the build to fail); the next regular build will check out from scratch. OK, I'll try that next time. Or I can add an extra 'clean checkout dir' step to the buildmaster -- but that would trigger a full rebuild every time, which is not what I want, since some of the buildslaves take a long time to do that. Grig ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python unit tests failing on Pybots farm
Grig Gheorghiu schrieb: Maybe the makefiles should be modified so that a full rebuild is triggered when the configure and configure.in files are changed? The makefiles already do that: if configure changes, a plain make will first re-run configure. Well, that didn't trigger a full rebuild on the Pybots buildslaves though. Can you provide more details? Did it not run configure again, or did that not cause a rebuild? There is an issue with setup.py/distutils not doing the rebuilding properly if header files change; contributions to fix this are welcome (quick-hacked work-arounds are not). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes and win64
Thomas Heller schrieb: I would prefer to merge these changes into release25-maint, because I want to also release the standalone ctypes packages from this branch (using it with svn:externals from somewhere else). That's not a good reason for back-porting. If you want a maintenance branch for ctypes, feel free to create one in the subversion, likewise for tags. OTOH, I can't comment on whether those changes would be acceptable for a backport to the 2.5 maintenance branch - if they don't introduce actual new features, it might be ok. The official Python 2.5.x win64/AMD64 windows installers should still *not* contain the ctypes package, but they could install it separately. I don't really understand. Are you planning to back-port PCbuild changes also? If so, how should including those extensions be suppressed? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python unit tests failing on Pybots farm
On 10/20/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Grig Gheorghiu schrieb: Maybe the makefiles should be modified so that a full rebuild is triggered when the configure and configure.in files are changed? The makefiles already do that: if configure changes, a plain make will first re-run configure. Well, that didn't trigger a full rebuild on the Pybots buildslaves though. Can you provide more details? Did it not run configure again, or did that not cause a rebuild? There is an issue with setup.py/distutils not doing the rebuilding properly if header files change; contributions to fix this are welcome (quick-hacked work-arounds are not). Here are the steps that led to the unit test failures, after your checkin of configure and configure.in. svn update: http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-svn/0 configure: http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-configure/0 compile: http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-compile/0 test: http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-test/0 HTH, Grig ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python unit tests failing on Pybots farm
Grig Gheorghiu schrieb: Here are the steps that led to the unit test failures, after your checkin of configure and configure.in. svn update: http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-svn/0 configure: http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-configure/0 compile: http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-compile/0 test: http://www.python.org/dev/buildbot/community/all/x86%20Ubuntu%20Breezy%20trunk/builds/55/step-test/0 As you can see, it indeed re-ran configure, and it also rebuilt the interpreter. It then did not rebuild any of the extensions except for pyexpat and elementtree. As I said, contributions to fix that are welcome. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ctypes and win64
Martin v. Löwis schrieb: Thomas Heller schrieb: [I was talking about patches to make ctypes work on 64-bit windows] I would prefer to merge these changes into release25-maint, because I want to also release the standalone ctypes packages from this branch (using it with svn:externals from somewhere else). That's not a good reason for back-porting. If you want a maintenance branch for ctypes, feel free to create one in the subversion, likewise for tags. OTOH, I can't comment on whether those changes would be acceptable for a backport to the 2.5 maintenance branch - if they don't introduce actual new features, it might be ok. The official Python 2.5.x win64/AMD64 windows installers should still *not* contain the ctypes package, but they could install it separately. I don't really understand. Are you planning to back-port PCbuild changes also? If so, how should including those extensions be suppressed? Let me try to put it in different words. The official Python-2.5.amd64.msi does *not* contain ctypes, so the official Python-2.5.x.amd64.msi should also not contain ctypes (I assume). Not many people (I assume again) are running 64-bit windows, and use the 64-bit Python version - but that will probably change soon. I would like to merge the 64-bit windows related ctypes changes in trunk, as soon as I'm sure that they work, back into the release25-maint branch. And also make separate ctypes releases from the release25-maint source code. I will only backport these changes if I'm convinced that they do not change the functionality of tehe current code. This way win64 Python users could install ctypes from the separate release. Also this way the source code for ctypes in the separate and the Python bundled releases are exactly the same, without creating too much work because of the different repositories. Hope that makes the plan clear, Thomas ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] OT: fdopen on Windows question
Sorry for the off-topic post. I figured someone here would know the answer and I don't have access to Windows to check experimentally. The ocrad program opens its input like so: if ( std::strcmp( infile_name, - ) == 0 ) infile = stdin; else infile = std::fopen( infile_name, r ); (SpamBayes is starting to use ocrad and PIL to extract text from image spam). Ocrad fails on Windows because the input file is opened in text mode. That r should be rb. What's not clear to me is whether we can do anything about stdin. Will this work: if ( std::strcmp( infile_name, - ) == 0 ) infile = std::fdopen( std::fileno(stdin), rb ); else infile = std::fopen( infile_name, rb ); That is, can I change stdin from text to binary this way or is it destined to always be in text mode? Thx, Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Massive test_sqlite failure on Mac OSX ... sometimes
I'm setting up a buildbot slave for sqlalchemy on one of my Macs at home. When it builds and tests Python's test suite the sqlite test fails. When I ran it alone like this: ./python.exe Lib/test/test_sqlite.py and ./python.exe Lib/test/regrtest.py test_sqlite it succeeded. When I ran the full test suite it failed. I then tried adding -v as the error message suggested. It hung in test_pty waiting for a child process to complete. (Is this a known problem?) I finally redirected stdout and stderr like so: ./python.exe Lib/test/regrtest.py -l -v test.out 21 and it completed. It failed 146 out of 167 tests. Here is a sample of the failure messages: ... CheckClose (sqlite3.test.dbapi.ConnectionTests) ... ERROR CheckCommit (sqlite3.test.dbapi.ConnectionTests) ... ERROR CheckCommitAfterNoChanges (sqlite3.test.dbapi.ConnectionTests) ... ERROR CheckCursor (sqlite3.test.dbapi.ConnectionTests) ... ERROR CheckExceptions (sqlite3.test.dbapi.ConnectionTests) ... ERROR CheckFailedOpen (sqlite3.test.dbapi.ConnectionTests) ... ERROR CheckRollback (sqlite3.test.dbapi.ConnectionTests) ... ERROR CheckRollbackAfterNoChanges (sqlite3.test.dbapi.ConnectionTests) ... ERROR CheckArraySize (sqlite3.test.dbapi.CursorTests) ... ERROR CheckClose (sqlite3.test.dbapi.CursorTests) ... ERROR CheckCursorConnection (sqlite3.test.dbapi.CursorTests) ... ERROR CheckCursorWrongClass (sqlite3.test.dbapi.CursorTests) ... ERROR CheckExecuteArgFloat (sqlite3.test.dbapi.CursorTests) ... ERROR CheckExecuteArgInt (sqlite3.test.dbapi.CursorTests) ... ERROR CheckExecuteArgString (sqlite3.test.dbapi.CursorTests) ... ERROR CheckExecuteDictMapping (sqlite3.test.dbapi.CursorTests) ... ERROR CheckExecuteDictMappingNoArgs (sqlite3.test.dbapi.CursorTests) ... ERROR CheckExecuteDictMappingTooLittleArgs (sqlite3.test.dbapi.CursorTests) ... ERROR CheckExecuteDictMappingUnnamed (sqlite3.test.dbapi.CursorTests) ... ERROR CheckExecuteIllegalSql (sqlite3.test.dbapi.CursorTests) ... ERROR CheckExecuteManyGenerator (sqlite3.test.dbapi.CursorTests) ... ERROR CheckExecuteManyIterator (sqlite3.test.dbapi.CursorTests) ... ERROR CheckExecuteManyNotIterable (sqlite3.test.dbapi.CursorTests) ... ERROR ... A quick check of the tracebacks shows all the errors are of this form (CheckClose is the first failure): == ERROR: CheckClose (sqlite3.test.dbapi.ConnectionTests) -- Traceback (most recent call last): File /Library/Buildbot/pybot/trunk.montanaro-g5/build/Lib/sqlite3/test/dbapi.py, line 85, in setUp self.cx = sqlite.connect(:memory:) ProgrammingError: library routine called out of sequence That is, they all raise the same exception and all exceptions are raised on sqlite.connect(:memory:) calls. Sometimes there is a second parameter to the call. Anybody seen this before? Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] svn.python.org down
Grig Gheorghiu schrieb: FYI -- can't do svn checkouts/updates from the trunk at this point. starting svn operation svn update --revision HEAD in dir /home/twistbot/pybot/trunk.gheorghiu-x86/build (timeout 1200 secs) svn: PROPFIND request failed on '/projects/python/trunk' svn: PROPFIND of '/projects/python/trunk': could not connect to server (http://svn.python.org) It turns out that there was a power surge at the colocation site where the machines are, and, due to an unfortunate series of wreck, power went out for about one second. When power came back, the machine rebooted, but, for some reason, the svn apache server did not. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] The lazy strings patch [was: PATCH submitted: Speed up + for string concatenation, now as fast as .join(x) idiom]
I've significantly enhanced my string-concatenation patch, to the point
where that name is no longer accurate. So I've redubbed it the "lazy
strings" patch.
The major new feature is that string *slices* are also represented with
a lazy-evaluation placeholder for the actual string, just as
concatenated strings were in my original patch. The lazy slice object
stores a reference to the original PyStringObject * it is sliced from,
and the desired start and stop slice markers. (It only supports step =
1.) Its ob_sval is NULL until the string is rendered--but that rarely
happens! Not only does this mean string slices are faster, but I bet
this generally reduces overall memory usage for slices too.
Now, one rule of the Python programming API is that "all strings are
zero-terminated". That part of makes the life of a Python extension
author sane--they don't have to deal with some exotic Python string
class, they can just assume C-style strings everywhere. Ordinarily,
this means a string slice couldn't simply point into the original
string; if it did, and you executed
x = "abcde"
y = x[1:4]
internally y-ob_sval[3] would not be 0, it would be 'e', breaking
the API's rule about strings.
However! When a PyStringObject lives out its life purely within the
Python VM, the only code that strenuously examines its internals is
stringobject.c. And that code almost never needs the trailing zero*.
So I've added a new static method in stringobject.c:
char * PyString_AsUnterminatedString(PyStringObject *)
If you call it on a lazy-evaluation slice object, it gives you back a
pointer into the original string's ob_sval. The s-ob_size'th
element of this *might not* be zero, but if you call this function
you're saying that's a-okay, you promise not to look at it. (If the
PyStringObject * is any other variety, it calls into PyString_AsString,
which renders string concatenation objects then returns ob_sval.)
Again: this behavior is *never* observed by anyone outside of
stringobject.c. External users of PyStringObjects call
PyString_AS_STRING(), which renders all lazy concatenation and lazy
slices so they look just like normal zero-terminated PyStringObjects.
With my patch applied, trunk still passes all expected tests.
Of course, lazy slice objects aren't just for literal slices created
with [x:y].
There are lots of string methods that return what are effectively
string slices, like lstrip() and split().
With this code in place, string slices that aren't examined by modules
are very rarely rendered. I ran "pybench -n 2" (two rounds, warp 10
(whatever that means)) while collecting some statistics. When it
finished, the interpreter had created a total of 640,041 lazy slices,
of which only *19* were ever rendered.
Apart from lazy slices, there's only one more enhancement when compared
with v1: string prepending now reuses lazy concatenation objects much
more often. There was an optimization in string_concatenate
(Python/ceval.c) that said: "if the left-side string has two
references, and we're about to overwrite the second reference by
storing this concatenation to an object, tell that object to drop its
reference". That often meant the reference on the string dropped to 1,
which meant PyString_Resize could just resize the left-side string in
place and append the right-side. I modified it so it drops the
reference to the right-hand operand too. With this change, even with a
reduction in the allowable stack depth for right-hand recursion (so
it's less likely to blow the stack), I was able to prepend over 86k
strings before it forced a render. (Oh, for the record: I ensure depth
limits are enforced when combining lazy slices and lazy concatenations,
so you still won't blow your stack when you mix them together.)
Here are the highlights of a single apples-to-apples pybench run, 2.6
trunk revision 52413 ("this") versus that same revision with my patch
applied ("other"):
Test minimum run-time average
run-time
this other diff this
other diff
---
ConcatStrings: 204ms 76ms +168.4% 213ms 77ms
+177.7%
CreateStringsWithConcat: 159ms 138ms +15.7% 163ms
142ms +15.1%
StringSlicing: 142ms 86ms +65.5% 145ms
88ms +64.6%
---
Totals: 7976ms 7713ms +3.4% 8257ms
7975ms +3.5%
I also ran this totally unfair benchmark:
x = "abcde" * (2) # 100k characters
for i in xrange(1000):
y = x[1:-1]
and found my patched version to be 9759% faster. (You heard that
right, 98x faster.)
I'm ready to post the patch. However, as a result of this work, the
description on the original patch page is really no longer accurate:
http://sourceforge.net/tracker/index.php?func=detailaid=1569040group_id=5470atid=305470
Shall I close/delete that patch and submit a new patch with a more
modern description? After all, there's not a lot of activity on the
old patch page...
Cheers,
larry
* As I
Re: [Python-Dev] Massive test_sqlite failure on Mac OSX ... sometimes
Following up on my earlier post... I svn up'd both my g5 and my g4 powerbook (both running OSX 10.4.8, gcc 4.0.0 apple build 5026), built and tested both. The test suite completed fine on my powerbook, failed on the g5. I tried running regrtest.py twice more on the g5 with the -r flag. It failed the first time, succeeded the second. I then made a series of run with the -f flag (thank you once again for that Señor Peters). I whittled it down to the following reliably failing pair: $ ./python.exe Lib/test/regrtest.py -l -f tests test_ctypes test_sqlite test test_sqlite failed -- errors occurred; run in verbose mode for details 1 test OK. 1 test failed: test_sqlite For confirmation, this pair works fine on my g4 powerbook. I've gone no further so far. It's bedtime. Maybe someone else can at least try to reproduce what I've come up with so far on other platforms or on another Mac g5. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
