Re: [Python-Dev] FWD: FTP URLs for Python source
Martin v. Löwis wrote: * if your target system doesn't have wget, download it locally, then scp/rcp/ftp it to the target system. All of [Free|Net|Open|Dragonfly]BSD have ftp clients that can also retrieve HTTP URLs, though I guess many wouldn't think of that... -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [email protected] (pref) | Snail: PO Box 370 [email protected] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Clean up Python/thread_*.h ?
Martin v. Löwis wrote: Making a decision and removing the files considered unnecessary would clarify what platforms still are/should be supported. I think any such removal should go through the PEP 11 process. Put a #error into the files for 3.2, and a removal notice into the PEP, then remove them in 3.3. As for OS/2: this should probably stay. Yes please (to keeping the OS/2 thread support as-is). Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [email protected] (pref) | Snail: PO Box 370 [email protected] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
Brett Cannon wrote: It's up to Andrew to get the support in. While I have faith he will, this is why we have been scaling back the support for alternative OSs for a while and will continue to do so. I suspect the day Andrew stops keeping up will be the day we push to have OS/2 be externally maintained. Notwithstanding my desire to keep OS/2 supported in the Python tree, keeping up has been more difficult of late: - OS/2 is unquestionably a "legacy" environment, with system APIs different in flavour and semantics from the current mainstream (though surprisingly capable in many ways despite its age). - The EMX runtime my OS/2 port currently relies on to abstract the system API to a Posix-ish API is itself a legacy package, essentially unmaintained for some years :-( This has been a source of increasing pain as Python has moved with the mainstream... with regard to Unicode support and threads in conjunction with multi-processing, in particular. Real Life hasn't been favourably disposed either... I have refrained from applying the extensive patches required to make the port feature complete for 2.6 and later while I investigate an alternate Posix emulating runtime (derived from FreeBSD's C library, and which is used by Mozilla on OS/2), which would allow me to dispense with most of these patches. But it has an issue or two of its own... The cost in effort has been compounded by effectively having to try and maintain two ports - 2.x and 3.x. And the 3.x port has suffered more as its demands are higher. So while I asked to keep the OS/2 thread support alive, if a decision were to be taken to remove OS/2 support from the Python 3.x sources I could live with that. A completed migration to Mercurial might well make future port maintenance easier for me. Regards, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [email protected] (pref) | Snail: PO Box 370 [email protected] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C++
Jeffrey Yasskin wrote: On Fri, Mar 12, 2010 at 7:54 PM, wrote: OS/2: http://www.andymac.org/ I can't find a modern c++ compiler for OS/2. Then again, your link only provides python 2.4. While the gcc used as part of the EMX toolchain is 2.8.1, there are ports of gcc 3.3.5 and 4.4.x using a somewhat compatible runtime (known as klibc). I have an unreleased 2.6 binary package which I'm holding up pending 2.6.5. OpenWatcom is also supported on OS/2, though I don't know whether the C++ support is advanced enough... while I like this compiler, the runtime support (or lack thereof) for Posixish features makes platform support a lot of work. Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [email protected] (pref) | Snail: PO Box 370 [email protected] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Drop OS/2 support?
Victor Stinner wrote: Python contains code specific to OS/2 (eg. see Modules/posixmodule.c). I read in Wikipedia that IBM has discontinued OS/2 support in 2005. Do we still support OS/2 or not? As was recently discussed, what constitutes "support" varies in perception. Python's source contains support for it to be built on OS/2, but python.org's normal build process doesn't deliver binaries. OS/2 lives on as eCS. It is a second tier platform, akin to Cygwin (which also has some source level support but is not part of the normal python.org release process). I'm asking because I'm working on a patch modifying OS2 specific code, but I'm unable to compile nor test my changes. And on IRC we told me that nobody knows/uses OS/2. > If we support OS/2, we need a buildbot. Build-bots for first tier platforms (Windows, Linux & OS/X) are useful for the python.org developer community. IMO build-bots for secondary platforms are the more the responsibility of that platform's community on the basis of best management of python.org resources. I don't think anyone reasonable would expect patch suppliers to deal with second tier platforms - I certainly don't. It is nice to get heads-up messages about issues that might involve such support though, and it shouldn't take much searching to find me to enquire. My patch: http://bugs.python.org/issue8391 (os.execvpe() doesn't support surrogates in env). I'll look at it when I get a chance, but I'm not expecting that my input should affect your patch and I'm not expecting you to try and deal with the issue on OS/2. The 3.x branch needs quite a bit of work on OS/2 to deal with Unicode, as OS/2 was one of the earlier OSes with full multiple language support and IBM developed a unique API. I'm still struggling to come to terms with this, partly because I myself don't "need" it. Regards, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [email protected] (pref) | Snail: PO Box 370 [email protected] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Community buildbots (was Re: User's complaints)
Anthony Baxter wrote: > On Friday 14 July 2006 22:45, Jeremy Hylton wrote: >> Maybe the basic question is right, but the emphasis needs to be >> changed. If we had a rule that said the final release was 90 days >> after the last submission that wasn't to fix a regression, we'd ask >> "Is this feature important enough to warrant delaying the release >> until three months from now?" I'm not sure what I think, but it >> doesn't seem like an implausible policy. > > I really really doubt that this would work. There's always pressure > for "just one more feature" - and as the release drags on, this would > increase, not decrease. Note that dragging the release process out > has it's own costs, as mentioned previously - either the trunk is in > some sort of near-frozen state for an extended period, or else we end > up in the double-applying-bugfixes state by forking much earlier. > > This approach would also make it extremely difficult to plan for > releases. I know that my free time varies across the course of the > year, and I need _some_ sort of plan for when the release will happen > so I can make sure I have the time free to spend on it. On the face of it, it seems to me that branching a new major release at the 1st beta would be one way of managing this. The trunk is not frozen for an extended period, and any "features" and bug fixes could probably be backported from the trunk on clearance from the RE with no more pain than is currently endured. One down side would be the possibility of a loss in emphasis in finishing the job on the release to be. I'm sure there are others... - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] test_mailbox on Cygwin
I just had a look at the Cygwin buildbot's test log, and noticed that test_mailbox has failures that look very similar to those I addressed in rev 50789 for the OS/2 EMX platform. Hopefully this might give someone with access to Cygwin a starting point in getting this test working on that platform. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible platforms to drop in 2.6
[EMAIL PROTECTED] wrote: > Brett> So, here are the platforms I figured we should drop: > > ... > Brett> * OS/2 > > I'm pretty sure Andrew MacIntyre is still maintaining the OS/2+EMX port: > > http://members.pcug.org.au/~andymac/python.html I am, although I haven't managed a binary release of 2.5 yet (should be in a few days). If its just the os2vacpp subdirectory in PC/ that is being discussed, that could go for 2.6. It will be a fair amount of work to disentangle the defines used for the 2 ports - a fair bit of the platform specific code is shared, but with compiler related differences. I would suggest more trouble than its worth for 2.6, but I could work on it. Of course, if the project management decide that even the EMX support should be removed from the official tree - so be it; I will just have to maintain the port outside the official tree. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythreads and BSD descendants
Martin v. Löwis wrote: > Cameron Laird schrieb: >> Folklore that I remember so unreliably I avoid trying to repeat it here >> held that Python threading had problems on BSD and allied Unixes. What's >> the status of this? > > The problem that people run into again and again is the stack size. The > BSDs allow for so little stack so that even the quite conservative > estimates of Python as to how many recursions you can have are > incorrect, and you get an interpreter crash rather than a RuntimeError > (as you should). There are 2 aspects to the thread stack size issue: - the stack size for the primary thread; - the stack size for created threads. I haven't done any investigating for FreeBSD 6.x and later, but I know that FreeBSD 4.x had a hard coded stack size of 1MB for the primary thread in a thread enabled application, which is what Martin's comment above particularly applies to. This affects code that doesn't use threads at all, and was particularly painful with REs prior to SRE being made non-recursive. If you build the interpreter without thread support, stack space is instead controlled by session limits which are usually generous (typically 64MB). I don't recall exactly FreeBSD's default stack size for threads created via pthread_create() but it is fairly small (32kB or 64kB comes to mind).Zope is one application known to be affected by this lack of stack size in created threads. At least the stack size for new threads can be adjusted at runtime, and a mechanism for doing this was added to Python 2.5. > Furthermore, every time we decrease the that number, the next system > release somehow manages to make the limit even smaller. This was > never properly analyzed; I suspect that the stack usage of Python > increases, either due to compiler changes or due to change to Python > itself. I have seen examples of stack consumption increasing with increasing gcc version number, sometimes depending on optimisation choices. Regards, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fate of Windows build directories
Christian Heimes wrote:
{...}
> PC/os2*/
> OS2 build dirs. The files are no longer maintained.
While I haven't been visible much lately, especially in the 3.0 space,
I do have plans to continue supporting the OS/2 EMX environment (PC/os2emx).
For 3.x, the PC/os2vacpp subdirectory can go, as I don't expect to be
able to support the VAC++ toolchain.
--
-
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A word of warning against using sqlite3 from MacPorts
Brett Cannon wrote:
{...}
> Noticing that sqlite 3.5.5 was recently available I had MacPorts
> update. Unfortunately this didn't fix things. I narrowed things down
> to running test_ctypes before test_sqlite as the trigger. In order to
> debug I wanted to use a version of sqlite that I had compiled.
{...}
> So I suspect that sqlite3 from MacPorts is built in such a way as to
> cause issues. This on Leopard which might also somehow influence
> things.
In a separate context, I've seen a reference to there being build related
issues with sqlite 3.5.5 on several platforms (I don't know which) which
have resulted in 3.5.6 being scheduled for release in fairly short order
(probably in the next few days).
--
-
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] int/float freelists vs pymalloc
I note that in r60567 Christian checked in support for compacting the
int and float freelists.
I have no problems with the implementation, but would like to note
that I have been experimenting with an alternate approach which doesn't
require the addition of a knob to initiate the compacting.
Probably in response to the same stimulus as Christian it occurred to me
that the freelist approach had been adopted long before PyMalloc was
enabled as standard (in 2.3), and that much of the performance gains
between 2.2 and 2.3 were in fact due to PyMalloc.
So I've been testing with the freelists ripped out and ints and floats
reverted to fairly standard PyObject_New allocation (retaining the small
int interning) and thus relying on PyMalloc to do any compaction.
The results have been somewhat surprising...
The short version is that:
- for ints, the freelist is ahead of PyMalloc by a very small margin
(<4%)
- for floats, the freelist is a long way behind PyMalloc (>35% slower)
This without invoking freelist compaction by the way (though PyMalloc
is releasing empty arenas).
I don't know what's pessimising the float freelist, but the results are
similar on 2 boxes:
- AMD Athlon XP-1600+, 512MB RAM, FreeBSD 6.1, gcc 3.4.4
(~27k pystones)
- AMD Athlon XP-2500+, 512MB RAM, OS/2 v4 with EMX, gcc 2.8.1
(~38k pystones)
If there's interest in following this up, I can put my patches to
intobject.c and floatobject.c into the tracker.
Test scripts:
a) int
# test routine - integer
import time
def b(time_now=time.clock):
limit_val = 200
start_time = time_now()
for j in xrange(25):
d = {}
for i in xrange(limit_val):
d[i] = i + limit_val
return time_now() - start_time
if __name__ == '__main__':
print 'elapsed: %s s' % b()
b) float
# test routine - float
import time
def b(time_now=time.clock):
limit_val = 100
vals = range(limit_val)
offset = limit_val * 1.0
start_time = time_now()
for j in xrange(25):
d = {}
for i in [float(x) for x in vals]:
d[i] = i + offset
return time_now() - start_time
if __name__ == '__main__':
print 'elapsed: %s s' % b()
The times I've obtained were:
case XP1600/Fbsd XP2500/OS2 (*)
1) int freelist 38.1s 24.6s
2) int pymalloc 39.0s 25.3s
3) float freelist
(with int freelist) 75s 46.1s
4) float pymalloc
-with int freelist55s 29.0s
-with int pymalloc55.5s 29.5s
(*) OS/2 tests based on patched 2.5.1 sources rather than svn trunk.
FreeBSD tests based on svn trunk checked out at 1050UTC 7Feb08.
--
-
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
M.-A. Lemburg wrote: > On 2008-02-07 14:09, Andrew MacIntyre wrote: >> Probably in response to the same stimulus as Christian it occurred to me >> that the freelist approach had been adopted long before PyMalloc was >> enabled as standard (in 2.3), and that much of the performance gains >> between 2.2 and 2.3 were in fact due to PyMalloc. > > One of the hopes of having a custom allocator for Python was to be > able to get rid off all free lists. For some reason that never happened. > Not sure why. People were probably too busy with adding new > features to the language at the time ;-) Very probably ;-) > Something you could try to make PyMalloc perform better for the builtin > types is to check the actual size of the allocated PyObjects and then > make sure that PyMalloc uses arenas large enough to hold a good quantity > of them, e.g. it's possible that the float types fall into the same > arena as some other type and thus don't have enough "room" to use > as free list. Like MvL, I doubt it. Uncle Timmy did a pretty thorough nose-clean on PyMalloc. However, my tests do show that something is funny with the current freelist implementation for floats on at least 2 platforms, and that doing without that sort of optimisation for float objects would likely not be a hardship with PyMalloc. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
M.-A. Lemburg wrote: > As you can see, Integers and floats fall into the same pymalloc size > class. What's strange in Andrew's result is that both integers > and floats use the same free list technique and fall into the same > pymalloc size class, yet the results are different. My take is not that PyMalloc is so much better in the float case, but that the float freelist implementation is suboptimal (though exactly why is subject for conjecture, given it's almost identical to the int implementation). > The only difference that's apparent is that small integers are > shared, so depending on the data set used for the test, fewer > calls to pymalloc or the free list are made. My testing was done with millions of unique integers, so the small int handling should be deep in the measurement noise. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Tim Peters wrote: > pymalloc ensures 8-byte alignment. This is one plausible reason to > keep the current int free list: an int object struct holds 3 4-byte > members on most boxes (type pointer, refcount, and the int's value), > and the int freelist code uses exactly 12 bytes for each on most > boxes. To keep 8-byte alignment, pymalloc would have to hand out a > 16-byte chunk per int object, wasting a fourth of the space (pymalloc > always rounds up a requested size to a multiple of 8, and ensures the > address returned is 8-byte aligned). Hmmm... the funny thing is that the freelist approach is showing higher memory consumption than the PyMalloc approach for the int case... -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Christian Heimes wrote: > Andrew MacIntyre wrote: >> For the int case, that patch is slower than no free list at all. For the >> float case its very slightly faster (55.3s vs 55.5s) than no free list at >> all. Slower than the trunk code for both cases. Did you run the test >> scripts on your own machine? > > I've used a simple timeit call to bench mark the memory allocation. The > first statement saturates the free lists and the second one benchmarks > memory allocation and float creation. > > ./python -m timeit -n10 -s "[float(x) for x in list(range(1000))]" "for > i in range(100): [float(x) for x in list(range(1000))]" Try testing with 100 rather than 1000. My testing was deliberately looking at the handling of large numbers of ints/floats, because small numbers (eg 1000) are not enough to make wasted memory recovery worthwhile. > I've done some experiments with a fixed length *free_list[] like dict > and listobject.c. A fixed length list is faster than a single linked > list. I'm going to upload the patch later. I tried a LIFO stack implementation (though I won't claim to have done it well), and found it slightly slower than no freelist at all. The advantage of such an approach is that the known size of the stack makes deallocating excess objects easy (and thus no need for sys.compact_free_list() ). I have been guilty of not paying attention to the performance of the small cases. >> In addition to the pure performance aspect, there is the issue of memory >> utilisation. The current trunk code running the int test case in my >> original post peaks at 151MB according to top on my FreeBSD box, dropping >> back to about 62MB after the dict is destroyed (without a compaction). >> The same script running on the no-freelist build of the interpreter peaks >> at 119MB, with a minima of around 57MB. > > I wonder why the free list has such a huge impact in memory usage. Int > objects are small (4 byte pointer to type, 4 byte Py_ssize_t and 4 byte > value). A thousand int object should consume less than 20kB including > overhead and padding. My test script allocates 400 ints - ~48MB. Much of the memory "pulse" is the creation/deletion of a dict with 200 items (25MB?). Its interesting to watch the freelist case, as first time through the loop the process size peaks at 109MB, then at the end of the 2nd pass it peaks at 151MB which is maintained on subsequent passes through the loop. (BTW, on this machine just starting the python interpreter leaves the process size at a bit over 3MB). The no-freelist build peaks at its maximum of 119MB on the first pass through the loop. >> On the 2 platforms I have, the current trunk code for ints would appear >> to be as good as it gets speed-wise. If the memory utilisation issue >> were to be considered significant, dropping the int freelist would have >> its attractions... >> >> As far as floats go, all the indications I have suggest that the current >> freelist code has a performance problem. Ditching the freelist and >> reverting to inlined PyObject_New()/PyObject_Del() semantics would be an >> attractive course as it appears to perform to within a whisker of the >> best alternatives while simplifying the object's code and taking >> advantage of pymalloc, however there are a number of other >> platforms/compilers that need testing before it would be prudent to do >> so. > > I can test the code on Linux and Windows XP. My tests have shown that a > linked free list doesn't give a considerable performance boost - even > for code that allocates and frees a lot of ints and floats at once. But > a fixed length pointer array gives a measurable performance gain. I've added my research patches to issue 2039, so I'd be interested to know how they rate in comparison. In addition to the small case testing mentioned above, I would suggest you do some large case testing (such as with the scripts I included in my original posting) as this is the justification for the effort on wasted memory recovery. Cheers, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [Fwd: Re: int/float freelists vs pymalloc]
[fwd as reply was intended for python-dev] Christian Heimes wrote: > Andrew MacIntyre wrote: >> I tried a LIFO stack implementation (though I won't claim to have done it >> well), and found it slightly slower than no freelist at all. The >> advantage of such an approach is that the known size of the stack makes >> deallocating excess objects easy (and thus no need for >> sys.compact_free_list() ). > > I've tried a single linked free list myself. I used the ob_type field to > daisy chain the int and float objects. Although the code was fairly > short it was slightly slower than an attempt without a free list at all. > pymalloc is fast. It's very hard to beat it though. I'm speculating that CPU cache effects can make these differences. The performance of the current trunk float freelist is depressing, given that the same strategy works so well for ints. I seem to recall Tim Peters paying a lot of attention to cache effects when he went over the PyMalloc code before the 2.3 release, which would contribute to its performance. > A fixed size LIFO array like PyFloatObject > *free_list[PyFloat_MAXFREELIST] increased the speed slightly. IMHO a > value of about 80-200 floats and ints is realistic for most apps. More > objects in the free lists could keep too many pymalloced areas occupied. I tested the updated patch you added to issue 2039. With the int freelist set to 500 and the float freelist set to 100, its about the same as the no-freelist version for my tests, but PyBench shows the simple float arithmetic to be about 10% better. I'm inclined to set the int LIFO a bit larger than you suggest, simply as ints are so commonly used - hence the value of 500 I used. Floats are much less common by comparison. Even an int LIFO of 500 is only going to tie up ~8kB on a 32bit box (~16kB on 64bit), which is insignificant enough that I can't see a need for a compaction routine. A 200 entry float LIFO would only account for ~4kB on 32bit (~8kB on 64bit). I'd like to think that these changes could be considered fixes for performance bugs so that they could be applied for 2.5.2, but I doubt that will fly. Cheers, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Christian Heimes wrote: > Andrew MacIntyre wrote: >> I tested the updated patch you added to issue 2039. With the int >> freelist set to 500 and the float freelist set to 100, its about the same >> as the no-freelist version for my tests, but PyBench shows the simple >> float arithmetic to be about 10% better. >> >> I'm inclined to set the int LIFO a bit larger than you suggest, simply as >> ints are so commonly used - hence the value of 500 I used. Floats are >> much less common by comparison. Even an int LIFO of 500 is only going to >> tie up ~8kB on a 32bit box (~16kB on 64bit), which is insignificant >> enough that I can't see a need for a compaction routine. >> >> A 200 entry float LIFO would only account for ~4kB on 32bit (~8kB on >> 64bit). > > I added some profiling code to lists and dicts. Increasing the free list > size didn't make a difference for lists and dicts. I used 200 instead of > 80 for the free list size and the ratio of reuse / allocation increased > minimal. > > Every entry in the free list costs 4 bytes for the pointer and 16 bytes > for the int or float object on 32bit including padding on 8byte address > boundaries. For 500 objects that's about 9kb which isn't a big deal > nowadays. But - andere there is always a but - every object may keep a > 4kb arena alive. The 4kB entities are "pools" (for allocation size classes); arenas are the 256kB chunks of memory PyMalloc gets from the platform malloc. One active allocation in any size class from a pool in an arena will prevent release of an arena. > Unless a program is creating and destroying lots of ints at once a > larger number doesn't make a performance difference. I'd stick to 80, > maybe 120 for ints for now. My thinking was that the price (& I was forgetting that the 12 byte int object still gets a 16 byte allocation) was small enough that being generous minimises surprises in production code; after all, we've been doing this research with synthetic microbenchmarks. When you get right down to it, the object freelist offers such a small gain that its real value is in the noise in many cases despite being measurable. After all, PyMalloc is nothing but a highly optimised package of freelists with the malloc()/realloc()/free() API... It just does a bit more book-keeping than the simple freelists used for ints and floats (which predate PyMalloc being added to the source tree, let alone being enabled by default). I'm not that interested in debating the detail of exactly how big the prospective LIFO freelists are - I just want to see the situation resolved with maximum utilisation of memory for minimum performance penalty. To that end, +1 from me for accepting your revised patch against issue 2039. In addition, unless there are other reasons to retain it, I would be suggesting that the freelist compaction infrastructure you introduced in r60567 be removed for lack of practical utility (assuming acceptance of your patch). Cheers, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
M.-A. Lemburg wrote: > On 2008-02-13 12:56, Andrew MacIntyre wrote: >> I'm not that interested in debating the detail of exactly how big the >> prospective LIFO freelists are - I just want to see the situation >> resolved with maximum utilisation of memory for minimum performance >> penalty. To that end, +1 from me for accepting your revised patch >> against issue 2039. In addition, unless there are other reasons to >> retain it, I would be suggesting that the freelist compaction >> infrastructure you introduced in r60567 be removed for lack of practical >> utility (assuming acceptance of your patch). > > If we're down to voting, here's my vote: > > +1 on removing the freelists from ints and floats, but not the >small int sharing optimization > > +1 on focusing on improving pymalloc to handle int and float >object allocations even better > > -1 on changing the freelist implementations to use pymalloc for >allocation of the freelist members instead of malloc, since >this would potentially lead to pools (and arenas) being held alive >by just a few objects - in the worst case a whole arena (256kB) >for just one int object (14 bytes on 32bit platforms). > > Eventually, all freelists should be removed, unless there's a > significant performance loss. > Sigh... things are never as simple as they seem... Prompted by your comment about the small int sharing, which I was aware of and was not addressing because I was assuming that its value was unimpeachable, I've been doing some more testing with this and the freelists. I don't have time to write it up tonight, but I've concluded that my original test scripts and other tests weren't showing the real performance of the various approaches. Platform (architecture, compiler etc) oddities & differences complicate life too... Cheers, Andrew. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Christian Heimes wrote: > By the way objects are always aligned at 8 byte address boundaries. A 12 > or 4 bytes object occupies 16 bytes. No, with PyMalloc a 4 byte object occupies 8 bytes (see the comments at the top of Objects/obmalloc.c). Cheers, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Andrew MacIntyre wrote: > M.-A. Lemburg wrote: >> If we're down to voting, here's my vote: >> >> +1 on removing the freelists from ints and floats, but not the >>small int sharing optimization >> >> +1 on focusing on improving pymalloc to handle int and float >>object allocations even better >> >> -1 on changing the freelist implementations to use pymalloc for >>allocation of the freelist members instead of malloc, since >>this would potentially lead to pools (and arenas) being held alive >>by just a few objects - in the worst case a whole arena (256kB) >>for just one int object (14 bytes on 32bit platforms). >> >> Eventually, all freelists should be removed, unless there's a >> significant performance loss. > > Sigh... things are never as simple as they seem... > > Prompted by your comment about the small int sharing, which I was aware > of and was not addressing because I was assuming that its value was > unimpeachable, I've been doing some more testing with this and the > freelists. > > I don't have time to write it up tonight, but I've concluded that my > original test scripts and other tests weren't showing the real > performance of the various approaches. Platform (architecture, compiler > etc) oddities & differences complicate life too... I've now written up my testing and attached the write-up to issue 2039 (http://bugs.python.org/issue2039). -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Vladimir Marangozov wrote: > May I chime in? :-) Please do! > Gents, the current implementation of Python's memory management > is really fine and most problems it used to have in the past > have been fixed in recent years (at least the ones I know of). > > Probably the only one left, indeed, is the potential unbounded > growth of int/float objects' freelists (beyond the shared cache > of small ints). Yet, this seems to be a questionable problem > because any change in the freelists' management will invariably > slow down their allocation. By how much, I don't know, but > whether you fallback to pymalloc above a certain threshold or > use something else, the change will have a generic perf hit. The performance hit is there, but my testing indicates its mostly not dramatic (& when its dramatic it only shows in microbenchmarks...). For me, "dramatic" is more than 10% gain/loss in non-trivial benchmarks. > The explanation is simple: you can't beat the free list scheme > performance when you have frequent short bursts of allocations > and deallocations, which is the typical Python pattern observed > on New() & DECREF() calls. I never expected that it could be beaten. I set out to find out how close simple alloc/dealloc via PyMalloc was to the freelists, and my testing suggests its pretty close, to the point that only the int & tuple freelists have IMO a clear-cut claim to retention. > BTW if you have 2 AMD combos showing speedups noone can explain > in an obvious way, then it's a cache effect. Figured as much. No-one with an intel CPU (or a different compiler) has as yet offered any equivalent test results - I would very much like to see them, but I'm not prepared ATM to buy another box to find out... > Optimizing pymalloc for non-standard byte-sizes is a no go, and > although I've seen suggestions for further optimizations tailored > to ints and floats, I haven't seen anyone spelling out what that > optimization of pymalloc would consist of. I figured Tim Peters would have done pretty much anything that could have been done when he polished PyMalloc up for default use in 2.3. > MAL's suggestion to preallocate a pymalloc pool cache for certain > object sizes is something I actually implemented in the earliest > versions of pymalloc, but eventually dropped in favor of the > current dynamic, on-request allocation because pre-allocating pools > (and initializing the free lists) costs time and in general leads > to degraded performance in the average case. I can't perf this > now to prove it, but that was very clear at the time when I wrote > the original stuff and applied the regression test on it. Good to know that alley has been checked and found a dead end. > So I would kindly suggest to get down to the only problem (if any) > which is the uncontrolled growth of the specialized freelists, the > shared int cache notwithstanding. If you can limit a degenerative > growth without a noticeable generic perf sacrifice, that would > sound like an improvement to me, but so far I haven't seen any > convincing arguments. Christian's compaction routine will get the job done, but IMO it should be (as he himself has proposed) called from gc.collect() rather than callable as a function in sys. > And, of course, if the int/float freelist scheme was a real issue > we would have probably heard of it by now in a very sound way. Not much noise has been made here, but I've come across 2 separate complaints in different mailing lists in the last month (one of the complainants had some hope that gc.collect() might help...) Is it a deafening roar? No, but I doubt the volume will do anything but increase... Regards, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] FreeBSD test suite failure -> curses
Jeroen Ruigrok van der Werven wrote: > -On [20080309 23:59], "Martin v. Löwis" ([EMAIL PROTECTED]) wrote: >> So this now *is* a FreeBSD/ncurses expert's question. >> I don't think this is supposed to happen; newscr should >> become non-NULL when initscr is called, and should remain >> that way throughout. > > Looking at the other FreeBSD build it seems to pass the tests. It's > 6.2-RELEASE box. So right now I am getting my VMWare 6.2-R up and running > and seeing if I also get the same results. At least that narrows my search > to the point of my 6.3-STABLE from the 6.2-RELEASE date. test_curses on its own passes on my FreeBSD 6.3 box. It segfaults when run in the context of a full regression test though. I've tried libthr instead of libpthread (via libmap.conf), and it also fails only in the full regression test. The backtrace from the coredump is different from the libpthread case though... libthr: === #0 0x2849513c in wecho_wchar () from /lib/libncursesw.so.6 #1 0x28496a89 in wecho_wchar () from /lib/libncursesw.so.6 #2 0x28497e62 in wecho_wchar () from /lib/libncursesw.so.6 #3 0x2849b134 in doupdate () from /lib/libncursesw.so.6 #4 0x287cf247 in PyCurses_doupdate (self=0x0) at /home/andymac/build/python-svn/trunk/Modules/_cursesmodule.c:182 #5 0x080c602e in PyEval_EvalFrameEx (f=0x93cbc0c, throwflag=0) at Python/ceval.c:3620 #6 0x080c6269 in PyEval_EvalFrameEx (f=0x93cba0c, throwflag=0) at Python/ceval.c:3722 #7 0x080c6269 in PyEval_EvalFrameEx (f=0x935b40c, throwflag=0) at Python/ceval.c:3722 #8 0x080c6eda in PyEval_EvalCodeEx (co=0x9b5bb60, globals=0x284c117c, locals=0x6e, args=0x817002c, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2908 #9 0x080c702e in PyEval_EvalCode (co=0x9b5bb60, globals=0x93fb4f4, locals=0x93fb4f4) at Python/ceval.c:495 #10 0x080d938a in PyImport_ExecCodeModuleEx ( name=0xbfbfdc60 "test.test_curses", co=0x9b5bb60, pathname=0xbfbfd2e0 "/home/andymac/build/python-svn/trunk/Lib/test/test_curses.pyc") at Python/import.c:680 #11 0x080d97f3 in load_source_module (name=0xbfbfdc60 "test.test_curses", pathname=0xbfbfd2e0 "/home/andymac/build/python-svn/trunk/Lib/test/test_curses.pyc", fp=0x282801d8) at Python/import.c:968 ---Type to continue, or q to quit--- libpthread: === #0 0x281a655b in pthread_testcancel () from /lib/libpthread.so.2 #1 0x2819eeec in pthread_mutexattr_init () from /lib/libpthread.so.2 #2 0x28166450 in ?? () This is with a checkout updated to r61352. Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyCon: please review miy pending patches
Christian Heimes wrote: > Memory management of ints, floats and longs > http://bugs.python.org/issue2039 > http://bugs.python.org/issue2013 wrt 2039 - I would like to see the free list compaction called from gc.collect() rather than a function in sys... something you suggested. As I noted in the comments to 2039, in the presence of pymalloc there is almost no value to floats having a freelist as far as I can test - other than in microbenchmarks. I see from a comment in 2013 that you were testing that patch with a debug build, which skews the timings. If your performance evaluation of 2039 was also done with a debug build, I suggest you try it with a non-debug build (which is what I used for all my testing). -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] rename of ConfigParser module?
Martin v. Lo"wis wrote: (Hmm, is changing Modules/Setup enough to sort the Windows build out as well? Or does that need a separate change to some of the Visual Studio files?) The latter. Whenever you add, remove, or rename an extension module, you need to adjust the PCbuild files as well. (technically, you also have to adjust PC/os2emx/Makefile, PC/os2vacpp/makefile.omk, and PC/VC6, but these changes are waived). OS/2 fixes are something I'll address during the betas. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] wrt the beta deadline and freelist management
There are 2 disparate approaches to clearing/compacting free lists for basic types: - APIs of the form Py_ClearFreeList() called from gc.collect() [class, frame, method, tuple & unicode types]; - APIs of the form Py_CompactFreeList() called from sys._compact_freelists() [int & float types]; Both approaches are new for 2.6 & 3.0. The patch at http://bugs.python.org/issue2862 is geared towards bring the int/float free list management into line with the others. The patch at http://bugs.python.org/issue3029 adds free list management to the dict, list & set types. The value of this is probably minor, and this patch is thus not significant in its own right other than for consistency. However Raymond's comment to issue 3029 that the management routines shouldn't be public APIs is IMO important (& I happen to agree). It would be nice to reduce the 2 approaches to one. I would rather the gc.collect() approach, as this seems to be more obvious to many users who have gone looking for free list management in prior versions, however my opinion isn't particularly valuable on this. Can this be resolved for 2.6/3.0? Regards, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] FreeBSD 7 amd64 and large memory tests
Martin v. Löwis wrote: I haven't yet tried posting a query to a FreeBSD list, as it could simply be a bug on amd64, but I was wondering whether there was anything (other than deactivating tests and documenting use of ulimit -v on this platform) that could be done to work around this behaviour. I think it should be possible to debug this (for people with access to such a system, and appropriate skills). I find it hard to believe that a large malloc will simply crash the process, rather than returning NULL. More likely, there is a NULL returned somewhere, and Python (or libc) fails to check for it. A simple C program doing a repetitive malloc(), much as pymalloc would with continuous demand, does indeed not see any NULL from malloc() when swap is exhausted but the process gets KILLed (the allocated memory does have to be written to to force the situation...) I'll take this up with FreeBSD folk, but I'm open to ideas as to how best to deal with the problem in the context of the test suite pending resolution by FreeBSD. Regards, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] FreeBSD 7 amd64 and large memory tests
Andrew MacIntyre wrote: I'll take this up with FreeBSD folk, but I'm open to ideas as to how best to deal with the problem in the context of the test suite pending resolution by FreeBSD. The response I got from Jason Evans (author of the new malloc() implementation), along with that of another respondent, indicates that the behaviour on FreeBSD 7.1 and later will (mostly) be restored to that similar to 6.x and earlier through the default use of sbrk() and consequent obedience to the data segment size limit (ulimit -d) - which defaults to 512MB in a standard FreeBSD install in recent times. The residual problem (as of 7.1) is that malloc() defaults to falling back to the mmap() strategy when it can't get more address space via sbrk(). As noted in the tracker item for issue 3862, the only way to control this is the virtual memory size limit (ulimit -v), which unfortunately defaults to "unlimited"... FreeBSD's malloc() can be tuned in several ways, so it is possible to force use of the sbrk() only strategy (as of 7.1) which would exactly match behaviour of the old malloc(). It seems to me that the most practical way forward is to just institute a policy that tests that want to try and test out of memory behaviour must ensure that appropriate resource limits are in place; if they can't (such as because the platform running the tests doesn't support getrlimit()/setrlimit()) the test should be skipped. As Mark Dickinson has suggested a patch for issue 3862 which should worm around the issue with test_array on 64 bit platforms, I think we can move forward for the time being. Cheers, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
Mike Coleman wrote: I have a program that creates a huge (45GB) defaultdict. (The keys are short strings, the values are short lists of pairs (string, int).) Nothing but possibly the strings and ints is shared. The program takes around 10 minutes to run, but longer than 20 minutes to exit (I gave up at that point). That is, after executing the final statement (a print), it is apparently spending a huge amount of time cleaning up before exiting. I haven't installed any exit handlers or anything like that, all files are already closed and stdout/stderr flushed, and there's nothing special going on. I have done 'gc.disable()' for performance (which is hideous without it)--I have no reason to think there are any loops. Currently I am working around this by doing an os._exit(), which is immediate, but this seems like a bit of hack. Is this something that needs fixing, or that has already been fixed? You don't mention the platform, but... This behaviour was not unknown in the distant past, with much smaller datasets. Most of the problems then related to the platform malloc() doing funny things as stuff was free()ed, like coalescing free space. [I once sat and watched a Python script run in something like 30 seconds and then take nearly 10 minutes to terminate, as you describe (Python 2.1/Solaris 2.5/Ultrasparc E3500)... and that was only a couple of hundred MB of memory - the Solaris 2.5 malloc() had some undesirable properties from Python's point of view] PyMalloc effectively removed this as an issue for most cases and platform malloc()s have also become considerably more sophisticated since then, but I wonder whether the sheer size of your dataset is unmasking related issues. Note that in Python 2.5 PyMalloc does free() unused arenas as a surplus accumulates (2.3 & 2.4 never free()ed arenas). Your platform malloc() might have odd behaviour with 45GB of arenas returned to it piecemeal. This is something that could be checked with a small C program. Calling os._exit() circumvents the free()ing of the arenas. Also consider that, with the exception of small integers (-1..256), no interning of integers is done. If your data contains large quantities of integers with non-unique values (that aren't in the small integer range) you may find it useful to do your own interning. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [email protected] (pref) | Snail: PO Box 370 [email protected] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] extremely slow exit for program having huge (45G) dict (python 2.5.2)
Mike Coleman wrote: Andrew, this is on an (intel) x86_64 box with 64GB of RAM. I don't recall the maker or details of the architecture off the top of my head, but it would be something "off the rack" from Dell or maybe HP. There were other users on the box at the time, but nothing heavy or that gave me any reason to think was affecting my program. It's running CentOS 5 I think, so that might make glibc several years old. Your malloc idea sounds plausible to me. If it is a libc problem, it would be nice if there was some way we could tell malloc to "live for today because there is no tomorrow" in the terminal phase of the program. I'm not sure exactly how to attack this. Callgrind is cool, but no way will work on something this size. Timed ltrace output might be interesting. Or maybe a gprof'ed Python, though that's more work. Some malloc()s (notably FreeBSD's) can be externally tuned at runtime via options in environment variables or other mechanisms - the malloc man page on your system might be helpful if your platform has something like this. It is likely that PyMalloc would be better with a way to disable the free()ing of empty arenas, or move to an arrangement where (like the various type free-lists in 2.6+) explicit action can force pruning of empty arenas - there are other usage patterns than yours which would benefit (performance wise) from not freeing arenas automatically. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [email protected] (pref) | Snail: PO Box 370 [email protected] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] RE: [Python-checkins] python/dist/src/Modules posixmodule.c, 2.300.8.10, 2.300.8.11
On Fri, 17 Dec 2004, Raymond Hettinger wrote:
> The Dec 12th check-ins break tests on WinME:
>
>
> test_glob.py
>
> Traceback (most recent call last):
> File "test_glob.py", line 78, in test_glob_one_directory
> eq(self.glob('a*'), map(self.norm, ['a', 'aab', 'aaa']))
> File "test_glob.py", line 67, in assertSequencesEqual_noorder
> self.assertEqual(set(l1), set(l2))
> AssertionError: set(['/[EMAIL PROTECTED]', '/[EMAIL PROTECTED]',
> '/[EMAIL PROTECTED]
> t_dir\\a']) != set(['[EMAIL PROTECTED]', '[EMAIL PROTECTED]',
> '[EMAIL PROTECTED]
> est_dir\\a'])
>
>
> test_urllib.py
> --
> FAIL: test_basic (__main__.urlretrieve_FileTests)
> --
> Traceback (most recent call last):
> File "test_urllib.py", line 142, in test_basic
> self.assertEqual(result[0], test_support.TESTFN)
> AssertionError: '[EMAIL PROTECTED]' != '/[EMAIL PROTECTED]'
>
>
>
> Please fix,
I don't see any possible way for those checkins to affect any platform
other than OS/2.
2 of the files are platform specific files (PC/os2emx/getpath.c,
PC/os2vacpp/getpath.c), and the checkin to Modules/posixmodule.c is
contained within a platform specific #if/#endif:
...
#ifdef HAVE_POPEN<<<-- line 3241
...
#if defined(PYOS_OS2)
#if defined(PYCC_VACPP)
... <<<-- changes here
#elif defined(PYCC_GCC)
...
#endif /* PYCC_??? */
#elif defined(MS_WINDOWS)
...
#else /* which OS? */
...
#endif /* PYOS_??? */
#endif /* HAVE_POPEN */
...
Note that the posixmodule change affects popen().
The matching PC/getpath.c changes that apply to Windows were checked in by
Tim Peters back in August.
I don't have any Windows development environment, so can't cross check
your report :-(
If you have verified the cause of failure by backing out v2.300.8.11, then
there's a deeper problem about the Windows build picking up explicitly
non-Windows components - which I find hard to believe.
In the absence of more evidence, not guilty your honour.
-
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
___
Python-Dev mailing list
[EMAIL PROTECTED]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
RE: [Python-Dev] re: 2.4 news reaches interesting places
On Fri, 17 Dec 2004, Raymond Hettinger wrote: > > So how about a slogan like "Code it Fast, with Python", or "Python: > Code > > Fast" -- one which emphasizes the (easily defended) claim that > development > > time is shorter with Python, but which at the same time manages to > > associate the word "fast" with "Python". > > I always liked: "Python, the language that wraps tightly around a > problem and swallows it whole." The only problem here is the association with the reptilian python, which is not perceived as being rapid - though it is highly effective as you describe. Whereas cobras... I somewhat suspect a subliminal association linking the reptile and the language, colouring perceptions without factual basis... :-( - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RE: [Python-checkins] python/dist/src/Modules posixmodule.c, 2.300.8.10, 2.300.8.11
On Sat, 18 Dec 2004, Tim Peters wrote: > [Raymond, says test_glob and test_urllib fail on WinME now] > > [Andrew MacIntyre] > >> I don't see any possible way for those checkins to affect any platform > >> other than OS/2. > >> > >> 2 of the files are platform specific files (PC/os2emx/getpath.c, > >> PC/os2vacpp/getpath.c), and the checkin to Modules/posixmodule.c is > >> contained within a platform specific #if/#endif: > > [Brett] > > Perhaps, but if you look at line 3299 you will notice that a comment is > > started > > but not closed off before the next comment on line 3304. How is that > > comment > > supposed to be closed off? > > That's obviously a bug, and was introduced by the patch. Andrew wanted > > /* setup the pipe */ > > not > > /* setup the pipe Fair cop guv. Long exposure to a language with comments of the latter form has been a PITA at times. :-( > But we don't get there unless PYOS_OS2 and PYCC_VACPP are defined, so > that can't explain Raymond's problem. > > FWIW, the tests at issue pass on WinXP for me today w/ current CVS. Good, though Raymond's report appeared to be focussed on the 2.3 branch. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: Prospective Peephole Transformation
Fredrik Lundh wrote: it could be worth expanding them to "if x == 1 or x == 2 or x == 3:" though... C:\>timeit -s "a = 1" "if a in (1, 2, 3): pass" 1000 loops, best of 3: 0.11 usec per loop C:\>timeit -s "a = 1" "if a == 1 or a == 2 or a == 3: pass" 1000 loops, best of 3: 0.0691 usec per loop C:\>timeit -s "a = 2" "if a == 1 or a == 2 or a == 3: pass" 1000 loops, best of 3: 0.123 usec per loop C:\>timeit -s "a = 2" "if a in (1, 2, 3): pass" 1000 loops, best of 3: 0.143 usec per loop C:\>timeit -s "a = 3" "if a == 1 or a == 2 or a == 3: pass" 1000 loops, best of 3: 0.187 usec per loop C:\>timeit -s "a = 3" "if a in (1, 2, 3): pass" 100 loops, best of 3: 0.197 usec per loop C:\>timeit -s "a = 4" "if a in (1, 2, 3): pass" 100 loops, best of 3: 0.225 usec per loop C:\>timeit -s "a = 4" "if a == 1 or a == 2 or a == 3: pass" 1000 loops, best of 3: 0.161 usec per loop Out of curiousity I ran /F's tests on my FreeBSD 4.8 box with a recent checkout: $ ./python Lib/timeit.py -s "a = 1" "if a in (1, 2, 3): pass" 100 loops, best of 3: 0.247 usec per loop $ ./python Lib/timeit.py -s "a = 1" "if a == 1 or a == 2 or a == 3: pass" 100 loops, best of 3: 0.225 usec per loop $ ./python Lib/timeit.py -s "a = 2" "if a in (1, 2, 3): pass" 100 loops, best of 3: 0.343 usec per loop $ ./python Lib/timeit.py -s "a = 2" "if a == 1 or a == 2 or a == 3: pass" 100 loops, best of 3: 0.353 usec per loop $ ./python Lib/timeit.py -s "a = 3" "if a in (1, 2, 3): pass" 100 loops, best of 3: 0.415 usec per loop $ ./python Lib/timeit.py -s "a = 3" "if a == 1 or a == 2 or a == 3: pass" 100 loops, best of 3: 0.457 usec per loop $ ./python Lib/timeit.py -s "a = 4" "if a in (1, 2, 3): pass" 100 loops, best of 3: 0.467 usec per loop $ ./python Lib/timeit.py -s "a = 4" "if a == 1 or a == 2 or a == 3: pass" 100 loops, best of 3: 0.488 usec per loop I then applied this patch: --- Objects/tupleobject.c.orig Fri Jun 11 05:28:08 2004 +++ Objects/tupleobject.c Tue Feb 22 22:10:18 2005 @@ -298,6 +298,11 @@ int i, cmp; for (i = 0, cmp = 0 ; cmp == 0 && i < a->ob_size; ++i) + cmp = (PyTuple_GET_ITEM(a, i) == el); + if (cmp) + return cmp; + + for (i = 0, cmp = 0 ; cmp == 0 && i < a->ob_size; ++i) cmp = PyObject_RichCompareBool(el, PyTuple_GET_ITEM(a, i), Py_EQ); return cmp; Re-running the tests yielded: $ ./python Lib/timeit.py -s "a = 1" "if a in (1, 2, 3): pass" 100 loops, best of 3: 0.234 usec per loop $ ./python Lib/timeit.py -s "a = 1" "if a == 1 or a == 2 or a == 3: pass" 100 loops, best of 3: 0.228 usec per loop $ ./python Lib/timeit.py -s "a = 2" "if a in (1, 2, 3): pass" 100 loops, best of 3: 0.239 usec per loop $ ./python Lib/timeit.py -s "a = 2" "if a == 1 or a == 2 or a == 3: pass" 100 loops, best of 3: 0.36 usec per loop $ ./python Lib/timeit.py -s "a = 3" "if a in (1, 2, 3): pass" 100 loops, best of 3: 0.241 usec per loop $ ./python Lib/timeit.py -s "a = 3" "if a == 1 or a == 2 or a == 3: pass" 100 loops, best of 3: 0.469 usec per loop $ ./python Lib/timeit.py -s "a = 4" "if a in (1, 2, 3): pass" 100 loops, best of 3: 0.475 usec per loop $ ./python Lib/timeit.py -s "a = 4" "if a == 1 or a == 2 or a == 3: pass" 100 loops, best of 3: 0.489 usec per loop - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: [Python-checkins] python/dist/src/Python thread_pthread.h, 2.53, 2.53.4.1
[EMAIL PROTECTED] wrote: Update of /cvsroot/python/python/dist/src/Python In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv25872 Modified Files: Tag: release24-maint thread_pthread.h Log Message: Patch #1163249 - Correctly handle _POSIX_SEMAPHORES == -1 to mean no support for posix semaphores. Index: thread_pthread.h === RCS file: /cvsroot/python/python/dist/src/Python/thread_pthread.h,v retrieving revision 2.53 retrieving revision 2.53.4.1 diff -u -d -r2.53 -r2.53.4.1 --- thread_pthread.h 7 Jul 2004 17:44:12 - 2.53 +++ thread_pthread.h 16 Mar 2005 04:13:29 - 2.53.4.1 @@ -16,9 +16,13 @@ family of functions must indicate this by defining _POSIX_SEMAPHORES. */ #ifdef _POSIX_SEMAPHORES +#if _POSIX_SEMAPHORES == -1 +#define HAVE_BROKEN_POSIX_SEMAPHORES +#else #include #include #endif +#endif #if !defined(pthread_attr_default) # define pthread_attr_default ((pthread_attr_t *)NULL) ___ Python-checkins mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-checkins This change has broken the build on FreeBSD 4.x for me: gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes - I. -I./Include -DPy_BUILD_CORE -o Python/thread.o Python/thread.c In file included from Python/thread.c:101: Python/thread_pthread.h:19: syntax error *** Error code 1 Backing it out allows the build to proceed & there are no unexpected test failures. Compiler is gcc 2.95.4. Regards, Andrew. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: [Python-checkins] python/dist/src/Python thread_pthread.h, 2.53, 2.53.4.1
Martin v. Löwis wrote: Andrew MacIntyre wrote: This change has broken the build on FreeBSD 4.x for me: gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes - I. -I./Include -DPy_BUILD_CORE -o Python/thread.o Python/thread.c In file included from Python/thread.c:101: Python/thread_pthread.h:19: syntax error *** Error code 1 This should be fixed now, please try again and report whether it works. It does. It would be really nice if you could try to analyse such problems deeper in the future. I would have if I'd had the time; sorry that I didn't state that. All I really intended was to alert Anthony to a problem on a widely used but non-primary target platform, so that he could make a decision about whether to worry about it or not for the 2.4.1 release. I had intended (but again didn't state) to follow this up at the earliest opportunity (which probably would have been this evening). In this case, it would have helped if you had reported that _POSIX_SEMAPHORES is defined as #define _POSIX_SEMAPHORES so that the #if line expands to #if == -1 The standard solution in this case is to write #if (_POSIX_SEMAPHORES+0) == -1 First time I can recall seeing that sort of situation - which I guess shows my lack of serious cross-platform C programming experience. Thanks for the fix and taking the time to explain. Regards, Andrew. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python continually calling sigprocmask() on FreeBSD 5
Michael Hudson wrote: > Suleiman Souhlal <[EMAIL PROTECTED]> writes: > >>While investigating why the script used in http://docs.freebsd.org/ >>cgi/getmsg.cgi?fetch=148191+0+current/freebsd-stable used so much >>system time on FreeBSD 5, I noticed that python is continually >>calling sigprocmask(), as can be seen from the following ktrace(1) dump: >> >>673 python 0.07 CALL sigprocmask(0x3,0,0x811d11c) >>673 python 0.05 RET sigprocmask 0 >>673 python 0.09 CALL sigprocmask(0x1,0,0x8113d1c) >>673 python 0.05 RET sigprocmask 0 >>etc.. >> >>This is using Python 2.4.1. >>Any clue about why this is happening? > > > In a word, no. > > As far as I am aware, there are no calls whatsoever to sigprocmask in > Python 2.4.1 (there were in 2.3.X, but these were connected to threads > and the mentioned script doesn't use them). So you need to do some > digging of your own. As I noted in a followup to the FreeBSD list where this came up, this appears to be a consequence of FreeBSD's Python port being configure'ed with the "-with-fpectl" option. This option uses setjmp()/longjump() around floating point ops for FP exception management, and it is the setjmp()/longjmp() in the FreeBSD threaded C libs that calls sigprocmask() as above. This option was certainly required for FreeBSD 4.x and earlier, where SIGFPE wasn't masked by default (at least that's my understanding of why the option was required...). I have the understanding that starting with 5.0 or 5.1, FreeBSD now does mask SIGFPE by default and so this option may not be necessary in this environment. A FreeBSD list member with a 5.3 system tested the microbenchmark that exposed the issue with a Python interpreter built without "-with-fpectl" and the performance issue disappeared. It is not yet clear whether it is appropriate that the port be changed to drop this configure option. Regards, Andrew. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] supported platforms OS2?
Jim Jewett wrote: > Is OS2 still actively supported? If you mean, is Python still actively supported on OS/2, then the answer is yes. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] supported platforms OS2?
Andrew MacIntyre wrote: > Jim Jewett wrote: > >> Is OS2 still actively supported? > > If you mean, is Python still actively supported on OS/2, then the answer > is yes. Just to clarify, I haven't been actively monitoring other changes for impact on the OS/2 port for the last few months (too much happening in the rest of my life), but am getting back into the swing of things now that I have a SVN checkout to work from. Barring disaster, there will be a 2.5 release of the OS/2 port. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Renaming sqlite3
Martin v. Löwis wrote: > I see three options: > 1. rename sqlite3 again > 2. link sqlite3 statically into _sqlite3.pyd > 3. stop treating .DLL files as extension modules > > I'm actually leaning towards option 3: what is the rationale > for allowing Python extension modules to be named .DLL? A datapoint specific to OS/2 which probably has little relevance to Windows or to the specific case at hand: In order to get the curses_panel module to work, I have to forward the necessary curses entry points from the _curses module DLL. On OS/2, this only works for DLLs with the extension .DLL, so I ship _curses.pyd as _curses.dll. As a consequence, I can't implement option 3 for the OS/2 port but I can live with the nasty side-effects given the modest userbase and by documenting the issue in the port README. If you can make option 3 work for Windows, then I would do it now during the alpha to see whether it flushes any problems out. I must admit to being uncomfortable with including version numbers in module names, especially when they reflect a version outside the scope of Python. Ending up with a module name that can match a 3rd party dynamically linkable file would seem problematic no matter which way you look at it. FWIW, Andrew. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] windows buildbot failures
Martin v. Löwis wrote: > It can't be that simple. Python's stdout should indeed be inherited > from cmd.exe, but that, in turn, should have obtained it from > buildbot. So even though cmd.exe closes its handle, Python's handle > should still be fine. If buildbot then closes the other end of the > pipe, Python should get ERROR_BROKEN_PIPE. The only deadlock I > can see here is when buildbot does *not* close the pipe, but stops > reading from it. In that case, Python's WriteFile would block. > > If that happens, it would be useful to attach with a debugger to > find out where Python got stuck. I doubt it has anything to do with this issue, but I just thought I'd mention something strange I've encountered on Windows XP Pro (SP2) at work. If Python terminates due to an uncaught exception, with stdout & stderr redirected externally (ie within the batch file that started Python), the files that were redirected to cannot be deleted/renamed until the system is rebooted. If a bare except is used to trap any such exceptions (and the traceback printed explicitly) so that Python terminates normally, there is no problem (ie the redirected files can be deleted/renamed etc). I've never reported this as a Python bug because I've considered the antivirus SW likely to be the culprit. I don't recall seeing this with Windows 2000, but much was changed in the transition from the Win2k SOE to the WinXP SOE. A wild shot at best... Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] patch #1454481 - runtime tunable thread stack size
http://www.python.org/sf/1454481 I would like to see this make it in to 2.5. To that end I was hoping to elicit any review interest beyond Martin and Hye-Shik, both of whom I thank for their feedback. As I can't readily test on Windows (in particular) or Linux I would appreciate some kind soul(s) actually testing to make sure that the patch doesn't break builds on those platforms. Thanks, Andrew. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] patch #1454481 - runtime tunable thread stack size
Terry Reedy wrote: > If you checked it in (after tests pass on your ?mac?, and while being ready > to revert), wouldn't the next buildbot cycle do the testing you need? > Isn't testing on 'other' platforms what buildbot is for? Only up to a point... In this case, I was after code review as much as the testing. Regards, Andrew. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] windows buildbot failures
Tim Peters wrote:
> [Andrew MacIntyre]
Hmm... I don't appear to have explained what I meant very well :-|
{...}
> This really needs an executable example. Here's my best guess about
> what you mean, but I don't see any of what you're describing on WinXP
> Pro SP2:
And a pretty good guess it was too!
Although I hadn't expected/intended any research to be applied to my note
(except if the original problem involved undeletable files on the
buildbots), I'm relieved your experiment turned up no problem; it makes
the case stronger against something in the local setup.
FWIW, the particular cases I remember seeing involved applications built
with ctypes and Venster (Win32 GUI wrapper over ctypes) with a lot of
COM calls, with Python 2.2.3.
>> I've never reported this as a Python bug
>
> If you do, I'll probably add a comment like the above ;-)
;-)
>> because I've considered the antivirus SW likely to be the culprit.
>
> Could be. FWIW, Norton AV was running during the above.
The anti-virus on my work XP Pro SP2 system has been a source of
considerable annoyance in many situations, so its the first thing
I blame...
>> I don't recall seeing this with Windows 2000, but much was changed
>> in the transition from the Win2k SOE to the WinXP SOE.
>
> What's that? Shitty Out-of-box Experience ;-)?
That too! Our outsourced IT supplier seems to think it means Standard
Operating Environment, but I haven't figured out what's "standard" about
it...
Cheers,
Andrew.
-
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.5 schedule
Neal Norwitz wrote: > I moved up the 2.5 release date by a bit. Ideally, I wanted to have > the final on July 27 which corresponds to OSCON. This seems too > aggressive since I haven't updated the schedule publicly. So the new > schedule has rc1 slated for July 27. > > http://www.python.org/dev/peps/pep-0356/ > > So far we've only had very few 2.5 bug reports for the first 2 alphas. > Hopefully people really are testing. :-) > > I'll continue to bug people that have action items. I'm sorry, but I find such advancing schedules with little warning quite objectionable. Particularly the cutoff for new functionality implicit in the last of the alphas. I have been working on patch 1454481 to try and make the original alpha 3 deadline in the context of my other commitments. I still hope to get it done early next week, but having that deadline advance without apparent warning just made things harder. Andrew. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] patch #1454481 vs buildbot
In reviewing the buildbot logs after committing this patch, I see 2 issues arising that I need advice about... 1. The Solaris build failure in thread.c has me mystified as I can't find any "_sysconf" symbol - is this in a system header? 2. I don't know what to make of the failure of test_threading on Linux, as test_thread succeeds as far as I could see. These tests succeed on my FreeBSD box and also appear to be succeeding on the Windows buildbots. Unfortunately I don't have access to either a Solaris box or a Linux box so could use some hints about resolving these. Thanks, Andrew. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] patch #1454481 vs buildbot
Tim Peters wrote:
> #if THREAD_STACK_MIN < PTHREAD_STACK_MIN
>
> assumes that the expansion of PTHREAD_STACK_MIN acts like a
> compile-time constant expression, but there's no such guarantee.
>
>http://cvs.opensolaris.org/source/xref/on/usr/src/head/limits.h
>
> shows that, on one version of Solaris, it's actually defined via
>
> #definePTHREAD_STACK_MIN ((size_t)_sysconf(_SC_THREAD_STACK_MIN))
>
> That has a runtime value, but not a useful compile-time value. The
> only useful thing you can do with it in an #if expression is
> defined(PTHREAD_STACK_MIN).
Ok.
>> 2. I don't know what to make of the failure of test_threading on Linux,
>> as test_thread succeeds as far as I could see. These tests succeed on my
>> FreeBSD box and also appear to be succeeding on the Windows buildbots.
>
> Not all pthreads-using builds fail, and not all failing pthreads-using
> builds fail in the same way. Welcome to pthreads on Linux ;-)
>
> BTW, this sucks:
>
> test_thread
> /home/buildbot/Buildbot/trunk.baxter-ubuntu/build/Lib/test/test_thread.py:138:
>
>
> RuntimeWarning: thread stack size of 0x1000 bytes not supported
> thread.stack_size(tss)
>
> That's from a successful run. RuntimeWarning really doesn't make
> sense for a failing operation. This should raise an exception
> (xyzError, not xyzWarning), or a failing stack_size() should return an
> error value after ensuring the original stack size is still in effect.
Fair enough.
{...}
> If PyThread_start_new_thread() fails in any way
> (like,pthread_attr_setstacksize() failing), ""can't start new thread"
> is the error we see.
>
> The difference between test_thread and test_threading here is that
> only test_threading asks for a 16MB stack; test_thread doesn't ask for
> a stack larger than 4MB.
Thanks for the analysis!
> Until all this gets resolved, I strongly suggest reverting this patch
> (if you don't, someone else will ...) and hammering out the problems
> on a new branch instead. See python-dev email from yesterday for how
> to force a buildbot slave to build a branch.
I see that you've already reverted this - Thanks & sorry I couldn't get
to it quickly.
Regards,
Andrew.
-
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] patch #1454481 vs buildbot
[EMAIL PROTECTED] wrote: > Andrew, look in your mail for a patch file. Received, thanks. Andrew. - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] beta1 coming real soon
Neal Norwitz wrote: > It's June 9 in most parts of the world. The schedule calls for beta 1 > on June 14. That means there's less than 4 days until the expected > code freeze. Please don't rush to get everything in at the last > minute. The buildbots should remain green to keep Anthony happy and > me sane (or is it the other way around). > > If you plan to make a checkin adding a feature (even a simple one), > you oughta let people know by responding to this message. Please get > the bug fixes in ASAP. Remember to add tests! I still harbour hopes of sorting SF1454481 out in time; if I have it sorted out by 1200UTC (2200AEST) on June 13, I'll merge it from the branch I created for it. "sorted out" includes addressing the issues Tim, Skip & yourself raised. Regards, Andrew. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
