Re: Simple TCP proxy
On 29/07/2022 8:08 am, Chris Angelico wrote: It takes a bit of time to start ten thousand threads, but after that, the system is completely idle again until I notify them all and they shut down. (Interestingly, it takes four times as long to start 20,000 threads, suggesting that something in thread spawning has O(n²) cost. Still, even that leaves the system completely idle once it's done spawning them.) Another cost of threads can be memory allocated as thread stack space, the default size of which varies by OS (see e.g. https://ariadne.space/2021/06/25/understanding-thread-stack-sizes-and-how-alpine-is-different/). threading.stack_size() can be used to check and perhaps adjust the allocation size. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andy...@pcug.org.au(pref) | Snail: PO Box 370 andy...@bullseye.apana.org.au (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- https://mail.python.org/mailman/listinfo/python-list
Re: Python bug in ArcGIS - Urban Network analysis tool
On 30/07/2018 4:02 PM, Станимира Николова wrote: I run Urban network analysis but It shows similar mistake several times. The UNA tool is free plugin that i downloaded, it's not from the default intalled in ArcGIS packed. It ask for integer data. I checked the type of the attributes, it's all integer. PLus it's all in geo data based file. Unfortunately I don't understand Python, I'm one of those who use ArcGIS as sample customer. This is the mistake: Start Time: Fri Jul 27 14:48:32 2018 Running script Centrality... [started] Copying input buildings [finished] [1 started] Computing adjacency list [1 failed] Not successful Completed script Centrality... Succeeded at Fri Jul 27 14:48:36 2018 (Elapsed Time: 4,56 seconds) Any suggestions? How it's calling these adjaency list? What could be wrong? I even don't know how to get debugger, so it could give me more information. If it's a third party plugin, contact the author or source. This mailing list sees very little traffic about ArcGIS usage as it is a highly specialised commercial product. I add in a project the .py file for the adjacency list. That's the main from the debuger: pydev debugger: process 8904 is connecting Connected to pydev debugger (build 182.3684.100) Traceback (most recent call last): File "C:\Program Files\JetBrains\PyCharm 2018.2\helpers\pydev\pydevd.py", line 1664, in main() File "C:\Program Files\JetBrains\PyCharm 2018.2\helpers\pydev\pydevd.py", line 1658, in main globals = debugger.run(setup['file'], None, None, is_module) File "C:\Program Files\JetBrains\PyCharm 2018.2\helpers\pydev\pydevd.py", line 1068, in run pydev_imports.execfile(file, globals, locals) # execute the script File "C:\Program Files\JetBrains\PyCharm 2018.2\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "D:/INSTALL/Urban Network Analysis Toolbox 1.01/src/Centrality/Adjacency_List_Computation.py", line 14, in from arcpy import AddField_management ModuleNotFoundError: No module named 'arcpy' May I found the following lines from the code? Is it documented that the file you've attempted to run via PyCharm actually supports being run from outside ArcGIS? Most plugins don't seem to be... This failure is because PyCharm isn't starting the script with the necessary environment (including the arcpy ArcGIS Python module) accessible. The fact that it starts running from within ArcGIS and produces some status messages before bailing out with the failure message strongly suggests to me that the plugin is working properly but the input given is not sufficient to produce the expected output. In the absence of sufficient documentation for you to figure out the required input, my advice above stands: contact the author or the download source. If you or your organisation has a current ArcGIS maintenance agreement, you might also be able to access the community forums that ESRI run to ask for more info about this plugin. -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370 andy...@pcug.org.au (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- https://mail.python.org/mailman/listinfo/python-list
Re: ctypes for AIX
Waddle, Jim wrote: Is there a policy concerning getting functions like ctypes working on AIX. If you can get it working, post a patch on the bug tracker. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370 andy...@pcug.org.au (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: questions regarding stack size use for multi-threaded python programs
Gabriel Genellina wrote:
En Mon, 09 Nov 2009 16:05:31 -0300, Eyal Gordon eyal.gor...@gmail.com
escribió:
background:
we are using python 2.4.3 on CentOS 5.3 with many threads - and our
shell's
default stack size limit is set to 10240KB (i.e. ~10MB).
we noticed that python's Threading module appears to create threads with
this value as their stack size (we ran a sample program that creates 10
threads and measured its virtual memory size, then reduced the stack size
limit of the shell to 5120KB - and saw that the program's virtual memory
size was reduced by ~50MBs).
the problem:
our program uses numerous threads, and thus the virtual memory size
gets to
be very large. we would like to reduce the size of the stack to reduce
this
size. we were looking for information about recommendation for the stack
size to use, but found none.
You can set the thread stack size (for threads that are going to be
created, not existing threads) using threading.stack_size(SIZE)
http://docs.python.org/library/threading.html#threading.stack_size
Sadly for the OP, that capability was introduced in Python 2.5. Prior
to that, the only way to adjust thread stack size is via a compile time
option (THREAD_STACK_SIZE). In the absence of that compile time option,
the platform default is used - on pthread systems (incl Linux) refer to
the manpage for pthread_attr_setstacksize() for more info, but I believe
what the OP reports could reasonably be expected for his platform.
The threading.stack_size() support would not be hard to backport to
Python 2.4.
questions:
1. is there some rule-of-thumb for the recommended stack size for python
programs of various sorts?
There is no rule of thumb because there are too many variables amongst
the supported platforms. Extensive testing would be required to
validate any selected size.
I would suggest starting with 1MB and working down. On a 32bit platform
I would suggest that 64kb might be adequate for child threads but most
likely not for the primary thread (the thread that the Python
interpreter starts in). If your app uses regexes, your stack space
requirements are likely to be larger.
FWIW the default thread stack size on Win32 is 1MB which anecdotally
seems sufficient for a wide variety of applications on that platform.
{...}
4. would the size of the stacks (which are probably not really
allocated by
the linux virtual memory sub-system, unless used) have a noticeable
performance effect on a python program? same question regarding the
use of a
large number of threads?
I think it doesn't matter, unless you create so many threads as to exhaust
the available addressing space (in 32bits, 4GB address space and 10MB per
thread means 400 threads maximum).
The performance of course will degrade once the committed memory in the
active working set exceeds the available real memory, as the OS will
start swapping. How the OS treats unused stack space will affect that
threshold.
--
-
Andrew I MacIntyre These thoughts are mine alone...
E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370
andy...@pcug.org.au (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python resident memory retention Evan Jones' improvements
Matt Ernst wrote:
{...}
I thought Evan Jones altered Python to deal with this very problem,
and the change went into the release of 2.5.
Here is Tim Peters announcing the change:
http://mail.python.org/pipermail/python-dev/2006-March/061991.html
He included this simple test program to show the improvement:
x = []
for i in xrange(100):
x.append([])
raw_input(full )
del x[:]
raw_input(empty )
If you look at resident size in the full stage, the interpreter has
grown to tens of megabytes. If you look at it in the empty stage, it
goes back down to less than 10 megabytes. But if you run this trivial
variation on the same program, memory use goes up and stays up:
x = []
for i in xrange(100):
x.append([])
raw_input(full )
del x[:]
for i in xrange(100):
x.append([])
del x[:]
raw_input(empty )
At the empty prompt resident memory size has not decreased. I see
this pattern of behavior in CPython 3.1.1, 2.6.3, 2.5.2, and Jython
2.5.1. I have tested under 32 and 64 bit Intel Linux.
At this point I suspect that I am not going to be able to force my
long-running processes to shrink their resident size, since I can't
force it in much simpler tests. I am curious about why it happens
though. That the second program should retain a larger resident memory
footprint than the first is (to me) quite surprising.
There are two things you need to be aware of in this situation:
- not all Python's memory is allocated through Python's specialised
malloc() - int and float objects in particular (in 2.x at least) are
allocated directly via a privately managed free list, and any
allocation requiring more than 256 bytes is directed to the platform
malloc(). Any memory not allocated via Python's malloc() is not
subject to the memory release facility referred to above.
Python 2.6 does improve the management of memory consumed by int and
float objects via the garbage collector.
- while Python attempts to maximally utilise memory arenas to improve
the chances of being able to free them, Python's malloc() does not do
any compaction of memory (ie moving allocations between arenas) within
existing arenas. Nor does garbage collection do this. So fragmented
allocations can cause the retention of nearly empty arenas.
I suspect that what you see with the second test script above is caused
by some sort of fragmentation.
My recollection of Evan's objective with his work was to deal with the
case where long running processes created lots of objects on startup,
but having initialised no longer need most of the created objects.
Without his modification, the memory would have been retained unused for
the remaining life of the process. It also helps with cyclic bursts of
object creation/deletion.
But there are circumstances where it doesn't kick in.
To get a deeper understanding of your issue will require deeper
debugging. I have done this at times by building Python with a wrapper
around malloc() ( friends) to log memory allocation activity.
--
-
Andrew I MacIntyre These thoughts are mine alone...
E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370
andy...@pcug.org.au (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
--
http://mail.python.org/mailman/listinfo/python-list
Re: Mapping in python? Transforming shapefile so that basemap can read them?
C Barr Leigh wrote: I'm trying to get started with plotting maps in python. I need to read shape files (.shp) and make maps. There seem to be many efforts but none is complete? I'm looking for suggestions and troubleshooting. The basemap package is obviously at an impressive stage and comes with some data: http://www.scipy.org/Cookbook/Matplotlib/Maps but it cannot read shapefiles when their coordinates are not in geographic projection. min are in a lambert, so the readshapefile fails. Apparently there is a utility that can convert a .shp file to lat/lon coordinates, but it fails for me. “You can convert the shapefile to geographic - coordinates using the shpproj utility from the shapelib tools - (http://shapelib.maptools.org/shapelib-tools.html) For me, this gives: “unable to process projection, exiting...” Someone else has already mentioned GDAL which is actually the raster part of the GDAL/OGR duo; OGR is the vector toolset which understands shapefiles. There are a couple of other packages possibly worth exploring: - Thuban (http://thuban.intevation.org) - GRASS(http://grass.itc.it/) - Shapelib (http://shapelib.maptools.org/) - FWTools (http://fwtools.maptools.org/) Shapefiles, except for the coordinate system support included in ESRI's more recent products, are documented in a publicly accessible PDF which googling for ESRI shapefile should find. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370 andy...@pcug.org.au (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Building Python with icc on 64-bit Linux
Konrad Hinsen wrote: /home/shr/khinsen/tmp/Python-2.6.2/Modules/_ctypes/libffi/src/x86/ffi64.c(43): \ error: identifier __int128_t is undefined __int128_t sse[MAX_SSE_REGS]; ^ compilation aborted for /home/shr/khinsen/tmp/Python-2.6.2/Modules/_ctypes/libf\ fi/src/x86/ffi64.c (code 2) That seems like a libffi configure failure (not properly handling 128bit integers on this compiler/platform). and failed test: test test_cmath failed -- Traceback (most recent call last): File /home/shr/khinsen/tmp/Python-2.6.2/Lib/test/test_cmath.py, line 366, i\ n test_specific_values self.fail(error_message) AssertionError: acos: acos(complex(0.0, 0.0)) Expected: complex(1.5707963267948966, -0.0) Received: complex(1.5707963267948966, 0.0) Received value insufficiently close to expected value. I've seen this on other compilers/platforms. The floating point support in 2.6 is being more rigorously tested. This compiler/platform/math lib doesn't seem to be explicitly handling -0.0 as different from 0.0. test_math has other similar failures. Adding optimization yields even more failed tests. My configuration options are: configure --prefix=$HOME CC=icc CXX=icc OPT=-O0 Did anyone encounter these problems before? Any solutions? For the FP issues I wonder whether there's a compiler option that affects how -0.0 is handled, or possibly a different math library? The libffi issue would require delving into the libffi source and adjusting its configure script to properly handle the problematic case. Unless you need ctypes, this may be something you can skip over... -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370 andy...@pcug.org.au (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Statically linked extension and relative import
David Cournapeau wrote: Hi, I am trying to build a 3rd party extension and link it statically to python. I managed to get things working by customizing Setup.local in python source tree, but I have a problem for imports of the 'foo.bar' form. For example, let's say the 3rd party module is laid out as follows: foo/__init__.py foo/bar.so and __init__.py contains: import foo.bar I can build the bar extension and link it statically to python such as from python, import bar works. But how can I make it recognize from foo import bar ? I tried to understand how this works at the C level in import.c, but I am a bit lost in the code. The ultimate goal is to manage to get code coverage of the C code of the extension through gcov, which does not support coverage of shared libraries ATM. If you link the bar extension statically, it is not in the foo package namespace (which is why import bar works), so should be imported as bar in foo's __init__.py. from foo import bar should then work from other code by virtue of the package namespace (instantiated by __init__.py) then having a bar symbol. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370 andy...@pcug.org.au (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: geospatial python and how to convert multilinestrings to kml
Danny Shevitz wrote: Howdy, I need to do some geospatial work and am a complete newbie at this. I have access to a PostGIS database and there are lots of MultiLineString objects. I want to run a python algorithm that determines a group of these MultiLineString objects and creates a KML file of the results. Is there a pythonic way (some existing module) to convert PostGIS MultiLineStrings to a KML file format? OGR - http://www.gdal.org/ogr/index.html Note that that page says the Python bindings aren't well documented. You might also find FWTools (http://fwtools.maptools.org/) useful. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370 andy...@pcug.org.au (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Static Map
KMCB wrote: Hello, I'm interested in creating a static map of a region in the US. This map would be set into a picture format, so I can add it to a document. I would like it to contain some town names and road information. Then I would like to add points, based on LAT and LONG, that can be labeled with information I provide. I have done some searching, Google Maps probably will not work because of the License. WuGeo does not seam to have the details. I was considering tw.openlayers, but do not know if that is the best starting point. If I could not create the picture from Python, would use tw and then create a screen shot to get started. Does anyone have thoughts on a better approach? While web-centric, you might find http://www.alistapart.com/articles/takecontrolofyourmaps an interesting read. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370 andy...@pcug.org.au (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: ulimit stack size and python threads
Greg Lindahl wrote: I figure this is a FAQ, but I can't find it in any FAQs. I want to limit the stacksize on my server. If I set it to 8 megs, or unlimited, python is happy. If I set it to 4 gigabytes, things like yum (which is a python program) crash creating a thread. This is on an x86_64 linux kernel, RHEL5, etc etc. Why is Python overloading the meaning of the ulimit -s like this? There are plenty of real non-python programs with huge stack usage, and I'd like my system default stack limit to be less than unlimited but much larger than Python will allow. The Python interpreter itself (absent a call to resource.setrlimit()) does nothing with resource limits - it just uses the environment it is loaded into. In the absence of effective alternative solutions, it may be possible to achieve the effect you desire by calling resource.setrlimit() in your Python installation's site.py, effectively over-riding the system default. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: andy...@bullseye.apana.org.au (pref) | Snail: PO Box 370 andy...@pcug.org.au (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Retrieving BSTR * from a DLL
mzdude wrote: I need to interface with a windows DLL that has the following signature extern C void Foo( BSTR in, BSTR *out ) Code so far from ctypes import * import comtypes LPBSTR = POINTER(comtypes.BSTR) hdl = windll.MyDll.Foo hdl.rettype = None hdl.argtypes = [comtypes.BSTR, LPBSTR] inStr = comtypes.BSTR(u'Some Silly String') out = comtypes.BSTR out = comtypes.BSTR() hdl(inStr,byref(out)) Traceback (most recent call last): File pyshell#14, line 1, in module hdl(inStr,byref(out)) TypeError: byref() argument must be a ctypes instance, not '_ctypes.SimpleType' comtypes.BSTR is a type; the type error makes clear you need an instance, as above. Also tried the following out = comtypes.BSTR(u'') p = pointer(out) hdl(inStr,p) Traceback (most recent call last): File pyshell#19, line 1, in module hdl(inStr,p) ValueError: Procedure probably called with too many arguments (8 bytes in excess) This likely indicates that the DLL is using the C calling convention and not the stdcall calling convention. Use CDLL rather than WinDLL to load the DLL. You might like to join the ctypes-users mailing list at sourceforge. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: ImportError: DLL load failed
Tony May wrote: I'm having trouble importing when I run in Python. The hello world program passes the test during the bjam build but gives an error about loading the dll when I import from a python script. first the test from running bjam. ...patience... ...found 1915 targets... ...using 1 temp target... ...updating 2 targets... ...using pbin\msvc-8.0express\debug \threading-multihello_ext.pyd... capture-output bin\hello.test\msvc-8.0express\debug\threading-multi\hello 1 file(s) copied. **passed** bin\hello.test\msvc-8.0express\debug\threading-multi\hello.test ...updated 2 targets... then trying to run the script from python I copied the pyd file and the rest of the output dir to c:\python25\dlls C:\Program Files\boost\boost_1_35_0\libs\python\example\tutorialpython hello.py Traceback (most recent call last): File hello.py, line 6, in module import hello_ext ImportError: DLL load failed: This application has failed to start because the a pplication configuration is incorrect. Reinstalling the application may fix this problem. The DLL that's actually being imported by your script is not a Python extension module, as the modules initialisation function can't be found. Use the -v option on the Python command line to identify which DLL is actually being imported. You can then decide to move/rename.delete it or your own module as best fits your circumstances. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Help needed - I don't understand how Python manages memory
Hank @ITGroup wrote: In order to deal with 400 thousands texts consisting of 80 million words, and huge sets of corpora , I have to be care about the memory things. I need to track every word's behavior, so there needs to be as many word-objects as words. I am really suffering from the memory problem, even 4G memory space can not survive... Only 10,000 texts can kill it in 2 minutes. By the way, my program has been optimized to ``del`` the objects after traversing, in order not to store the information in memory all the time. In addition to all the other advice you've been given, I've found it can pay dividends in memory consumption when each instance of a value (such as a string) references only 1 object. This is often referred to as interning. Automatic interning is only performed for a small subset of possibilities. For example: z1 = 10 z2 = 10 z1 is z2 True z1 = 1000 z2 = 1000 z1 is z2 False z1 = 'test' z2 = 'test' z1 is z2 True z1 = 'this is a test string pattern' z2 = 'this is a test string pattern' z1 is z2 False Careful use of interning can get a double boost: cutting memory consumption and allowing comparisons to short circuit on identity. It does cost in maintaining the dictionary that interns the objects though, and tracking reference counts can be much harder. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: problem with Python 2.5.2 and gcc 4.3
David P. Riedel wrote: I tried building Python 2.5.2 using gcc 4.3.0. The build completes with no problems but when I run 'make test', I get a segfault part way through the test run. here is the last part of the output from make test test_softspace test_sort test_sqlite test_sqlite skipped -- no sqlite available test_startfile test_startfile skipped -- cannot import name startfile test_str make: *** [test] Segmentation fault You don't identify the platform or O/S, though I'd guess some Linux distro on i386 or x86-64... If you have gdb available, a backtrace might give a clue. However, as this is a new major release of gcc I'm automatically going to assume an optimisation issue. To test this I'd suggest doctoring the makefile generated by configure to reduce the optimisation level - I'd suggest trying -O instead of -O3. If that works, try -O2 or -Os. If -O2 or -Os works, I'd be taking the matter up with the gcc team. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: bus error/segfault from PyArg_ParseTuple in initproc with incorrect arg number
Miles Lubin wrote: I am using PyArg_ParseTuple to parse the arguments (ignoring the keyword arguments) to my initproc for a type I define. It seems that something goes wrong inside PyArg_ParseTuple when it gets the wrong number of arguments (my format string is OO); if the function isn't given exactly two arguments, I get a bus error on OS X and a segfault on Linux. If two arguments are given, the code runs as expected. This does not occur when using PyArg_ParseTuple in a normal method. Am I not using PyArg_ParseTuple correctly? Here's the relevant code: PyObject *a, *b; if (!PyArg_ParseTuple(args, OO, a, b)) return -1; The segfault occurs on this line, not on any line after. I have seen bus errors on FreeBSD when python runs out of stack space. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Memory Leaks and Heapy
Yaakov Nemoy wrote: A couple of developers have mentioned that python might be fragmenting its memory space, and is unable to free up those pages. How can I go about testing for this, and are there any known problems like this? If not, what else can I do to look for leaks? Marc-Andre brought up pymalloc, but it is worth clarifying a couple of issues related to its use: - pymalloc only manages allocations up to (and including) 256 bytes; allocations larger than this are passed to the platform malloc to allocate. - the work that was put in to allow return of empty arenas (in Python 2.5) was geared to handling the general case of applications that created huge volumes of objects (usually at start up) and then destroy most of them. There is no support that I'm aware of for any form of arena rationalisation in the case of sparsely occupied arenas. - it has been my experience that pymalloc is a significant benefit over the platform malloc for the Python interpreter, both in terms of performance and gross memory consumption. Prior to defaulting to using pymalloc (as of 2.3) CPython had run into issues with the platform malloc of just about every platform it had been ported to, heap fragmentation being particularly notable on Windows (though other platforms have also been subject to this). While pymalloc is highly tuned for the general case behaviour of the Python interpreter, just as platform malloc implementations have corner cases so does pymalloc. Be aware that ints and floats are managed via free lists with memory allocation directly by the platform malloc() - these objects are never seen by pymalloc, and neither type has support for relinquishing surplus memory. Be also aware that many C extensions don't use pymalloc even when they could. In addition to Marc-Andre's suggestions, I would suggest paying particular attention to the creation and retention of objects in your code - if something's no longer required, explicitly delete it. It is all too easy to lose sight of references to objects that hang around in ways that defeat the gc support. Watch out for things that might be sensitive to thread-ids for example. Careful algorithm planning can also be useful, leveraging object references to minimise duplicated data (and possibly get better performance). -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: convert pdf to png
Carl K wrote: Andrew MacIntyre wrote: Grant Edwards wrote: On 2007-12-24, Carl K [EMAIL PROTECTED] wrote: If it is a multi page pdf Imagemagick will do: convert file.pdf page-%03d.png I need python code to do this. It is going to be run on a someone else's shared host web server, security and performance is an issue. So I would rather not run stuff via popen. Use subprocess. Trying to eliminate popen because of the overhead when running ghostscript to render PDF (I assume convert uses gs?) is about like trimming an elephants toenails to save weight. Using ctypes to call Ghostscript's API also works well. I've only done this on Windows, but it should also work on other systems with ctypes support. sounds good, but I have 0.0 clue what that actually means. Can you give me what you did with windows in hopes that I can figure out how to do it in Linux? I am guessing it shouldn't be to different. (well, hoping...) ctypes is a foreign function interface (FFI) extension that became part of the standard library with Python 2.5 ( is available for 2.3 2.4). It is supported on Linux, *BSD Solaris (I think) in addition to Windows. Ghostscript for quite some time has had support for being used as a library (DLL on Windows). There are only a small number of API functions exported, and there is information about the net for calling these API functions from Visual Basic. I wrote a wrapper module using ctypes for the API based on the C header and the VB information. To get the best rendering, some understanding of Ghostscript options is required particularly for image format outputs (eg for anti-aliasing text). -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: CPython and a C extension using Boehm GC
malkarouri wrote: Is it possible to write a Python extension that uses the Boehm garbage collector? I have a C library written that makes use of boehm-gc for memory management. To use that, I have to call GC_INIT() at the start of the program that uses the library. Now I want to encapsulate the library as a CPython extension. The question is really is that possible? And will there be conflicts between the boehm-gc and Python memory management? And when should I call GC_INIT? It probably should be possible with some caveats: - memory allocated by Python is never passed into the library such that it also ends up being subject to boehm-gc; - memory allocated by the library is never used by Python objects. So memcpy()ing between library allocated and Python allocated memory would seem to be a way to achieve this. I would call GC_INIT in the extension's import routine (initmodule_name()) for a C extension, and immediately after loading the library if using ctypes. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: convert pdf to png
Grant Edwards wrote: On 2007-12-24, Carl K [EMAIL PROTECTED] wrote: If it is a multi page pdf Imagemagick will do: convert file.pdf page-%03d.png I need python code to do this. It is going to be run on a someone else's shared host web server, security and performance is an issue. So I would rather not run stuff via popen. Use subprocess. Trying to eliminate popen because of the overhead when running ghostscript to render PDF (I assume convert uses gs?) is about like trimming an elephants toenails to save weight. Using ctypes to call Ghostscript's API also works well. I've only done this on Windows, but it should also work on other systems with ctypes support. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Pure Python GUI lib?
oyster wrote: For the word Pure, I mean it is not a C/C++/Z++.. extension, so that we can use it under pythons of different version. Is it possible? I don't like to update the module for different python and the module Currently, I am writing the interface to iup(http://www.tecgraf.puc-rio.br/iup) via ctypes, but find 2 too strange things which have let me feel blue for some days, and I don't know whether it can be successful or not. Can anyone give me some lights? Thank you. :) Venster is a Windows API GUI wrapper on top of ctypes. Fairly low level, but quite viable. I have an application at work using an old version (0.21) on an old version of ctypes (0.6.3) that works fine, though it doesn't do MDI. Sadly Venster appears not to have recently been maintained in line with changes in ctypes, but the code (available from SourceForge) could still prove a useful reference. Cheers, Andrew. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Python memory handling
[EMAIL PROTECTED] wrote: Using the same file with cElementTree took me 217 Mb, with no unreachable object. For me it's not a good behavior, it's not a good way to let the system swap this unused memory instead of freeing it. I think it's a really good idea to have a memory pool for performance reason, but why is there no 'free block' limit ? Python is a really really good language that can do many things in a clear, easier and performance way I think. It has always feet all my needs. But I can't imagine there is no good solution for that problem, by limiting the free block pool size or best, letting the user specify this limit and even better, letting the user completely freeing it (with also the limit manual specification) Like: import pool pool.free() pool.limit(size in megabytes) Why not letting the user choosing that, why not giving the user more flexibility ? Because its not easy, and its an unusual edge case that hasn't attracted developer effort (the PyMalloc change for 2.5 was contributed by someone who desperately needed it, not a core Python developer; it was also a non-trivial effort to get right). You should also appreciate something about PyMalloc: it only handles allocation requests of 256 bytes or smaller, and this limitation is part of PyMalloc's design. If most of your allocations are 256 bytes, you're at the mercy of the platform malloc and heap fragmentation can be a killer. This is probably why the getlines() approach mentioned would appear to relinquish (most of) the memory: the list was probably comprised mostly of PyMalloc allocations. I haven't checked, but cElementTree may internally not be using PyMalloc anyway, as the package is stated to be usable back to Python 1.5 - long before the current allocation management came into effect. In which case, you're at the mercy of the platform malloc... The pure Python ElementTree might play more your way, at a performance cost. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Compiling python extension on amd64 for 32 bit
Mathias Waack wrote:
Andrew MacIntyre wrote:
Mathias Waack wrote:
After switching my development environment to 64 bit I've got a
problem with a python extension for a 32 bit application.
{...}
Ok, thats fine. So why is python complaining? Or even more
interesting, what do I have to do to compile the code?
Is the Python your toolchain is referencing 32 or 64 bit? Based on
what I can see in pyport.h, I'd guess that you're finding a 64 bit
Python (ie SIZEOF_LONG == 8).
There is no such thing as the Python. A biarch environment offers
both the 32 bit and the 64 bit versions of each library. And unique
headers, because its assumed that the headers are independent of the
architecture. Because of the -m32 Flag the pyport.h is used within a
32 bit env.
Every Python installation has an architecture dependent header
(pyconfig.h), unless a vendor specifically creates a replacement which
neutralises this; that header contains the definition (SIZEOF_LONG) that
triggers your problem.
--
-
Andrew I MacIntyre These thoughts are mine alone...
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
--
http://mail.python.org/mailman/listinfo/python-list
Re: Compiling python extension on amd64 for 32 bit
Mathias Waack wrote:
After switching my development environment to 64 bit I've got a problem with
a python extension for a 32 bit application.
{...}
Ok, thats fine. So why is python complaining? Or even more interesting, what
do I have to do to compile the code?
Is the Python your toolchain is referencing 32 or 64 bit? Based on what
I can see in pyport.h, I'd guess that you're finding a 64 bit Python (ie
SIZEOF_LONG == 8).
--
-
Andrew I MacIntyre These thoughts are mine alone...
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
--
http://mail.python.org/mailman/listinfo/python-list
Re: Possible to set cpython heap size?
Chris Mellon wrote: On 22 Feb 2007 11:28:52 -0800, Andy Watson [EMAIL PROTECTED] wrote: On Feb 22, 10:53 am, a bunch of folks wrote: Memory is basically free. This is true if you are simply scanning a file into memory. However, I'm storing the contents in some in-memory data structures and doing some data manipulation. This is my speculation: Several small objects per scanned line get allocated, and then unreferenced. If the heap is relatively small, GC has to do some work in order to make space for subsequent scan results. At some point, it realises it cannot keep up and has to extend the heap. At this point, VM and physical memory is committed, since it needs to be used. And this keeps going on. At some point, GC will take a good deal of time to compact the heap, since I and loading in so much data and creating a lot of smaller objects. If I could have a heap that is larger and does not need to be dynamically extended, then the Python GC could work more efficiently. I haven't even looked at Python memory management internals since 2.3, and not in detail then, so I'm sure someone will correct me in the case that I am wrong. However, I believe that this is almost exactly how CPython GC does not work. CPython is refcounted with a generational GC for cycle detection. There's a memory pool that is used for object allocation (more than one, I think, for different types of objects) and those can be extended but they are not, to my knowledge, compacted. If you're creating the same small objects for each scanned lines, and especially if they are tuples or new-style objects with __slots__, then the memory use for those objects should be more or less constant. Your memory growth is probably related to the information you're saving, not to your scanned objects, and since those are long-lived objects I simple don't see how heap pre-allocation could be helpful there. Python's internal memory management is split: - allocations up to 256 bytes (the majority of objects) are handled by a custom allocator, which uses 256kB arenas malloc()ed from the OS on demand. With 2.5 some additional work was done to allow returning completely empty arenas to the OS; 2.3 and 2.4 don't return arenas at all. - all allocations over 256 bytes, including container objects that are extended beyond 256 bytes, are made by malloc(). I can't recall off-hand whether the free-list structures for ints (and floats?) use the Python allocator or direct malloc(); as the free-lists don't release any entries, I suspect not. The maximum allocation size and arena size used by the Python allocator are hard-coded for algorithmic and performance reasons, and cannot be practically be changed, especially at runtime. No active compaction takes place in arenas, even with GC. The only time object data is relocated between arenas is when an object is resized. If Andy Watson is creating loads of objects that aren't being managed by Python's allocator (by being larger than 256 bytes, or in a type free-list), then the platform malloc() behaviour applies. Some platform allocators can be tuned via environment variables and the like, in which case review of the platform documentation is indicated. Some platform allocators are notorious for poor behaviour in certain circumstances, and coalescing blocks while deallocating is one particularly nasty problem for code that creates and destroys lots of small variably sized objects. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Threads and C Semaphores
Dejan Rodiger wrote: Jeremy said the following on 16.1.2007 8:27: I have a fat C++ extension to a Python 2.3.4 program. In all, I count five threads. Of these, two are started in Python using thread.start_new_thread(), and both of these wait on semaphores in the C++ extension using sem_wait(). There also are two other Python threads and one thread running wholly in the extension. I notice that when one of the Python threads calls the extension and waits on a semaphore, all but the C++ thread halt even when not waiting on any semaphore. How do we get this working right? Check the Global Interpreter Lock More specifically, make sure that any extension code that does not call Python API functions releases the GIL for the duration. Look into the Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS macros and the PyGILState*() API functions (these API functions appeared in Python 2.3). -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Exploiting Dual Core's with Py_NewInterpreter's separated GIL ?
Robin Becker wrote:
Robin Becker wrote:
Andrew MacIntyre wrote:
Robin Becker wrote:
I think it uses sysv semaphores and although freeBSD 6 has them
perhaps there's something I need to do to allow them to work.
IIRC, you need to explicitly configure loading the kernel module, or
compile the kernel with the necessary option in the config file.
I tried loading the module, but it seems already to be in the kernel.
Probably some sysctl stuff to do.
In fact the semaphore modules are built in now. I did some debugging and the
error happens in the code at SemSet.c line 25
for (i = 0; i sets; i++) {
int id = semget(IPC_PRIVATE, MAX_SYSV_SET_SIZE, IPC_CREAT
| S_IREAD | S_IWRITE); //SEM_R | SEM_A);
if (id == -1) {
the first semget(i=0) succeeds and the second fails with an errno ENOSPC
which
from the man page implies a resource issue of some kind.
[ENOSPC] Insufficiently many semaphores are available.
[ENOSPC] The kernel could not allocate a struct semid_ds.
I guess the freebsd limits must be different to the original development
environment.
The number of semaphores is certainly tunable - the SYSV IPC KERNEL
PARAMETERS section in the file /usr/src/sys/conf/NOTES lists the SYSV
semaphore parameters that can be set in the kernel config.
--
-
Andrew I MacIntyre These thoughts are mine alone...
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
--
http://mail.python.org/mailman/listinfo/python-list
Re: Exploiting Dual Core's with Py_NewInterpreter's separated GIL ?
Robin Becker wrote: I think it uses sysv semaphores and although freeBSD 6 has them perhaps there's something I need to do to allow them to work. IIRC, you need to explicitly configure loading the kernel module, or compile the kernel with the necessary option in the config file. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: multythreading app memory consumption
Bryan Olson wrote: In Python 2.5, each thread will be allocated thread.stack_size() bytes of stack address space. Note that address space is not physical memory, nor even virtual memory. On modern operating systems, the memory gets allocated as needed, and 150 threads is not be a problem. Just a note that [thread|threading].stack_size() returns 0 to indicate the platform default, and that value will always be returned unless an explicit value has previously been set. The Posix thread platforms (those that support programmatic setting of this parameter) have the best support for sanity checking the requested size - the value gets checked when actually set, rather than when the thread creation is attempted. The platform default thread stack sizes I can recall are: Windows: 1MB (though this may be affected by linker options) Linux:1MB or 8MB depending on threading library and/or distro FreeBSD: 64kB -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Ghostscript interface
defcon8 wrote: Does a ghostscript interface for python exist? I have searched google quite a bit and all I have been able to find are command line hacks from within python. Thanks in advance for any useful help. I'm not aware of a specific interface to the Ghostscript API, but it is trivial to implement one with ctypes (included with 2.5). -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: _mssql on Python 2.5 Beta 2
[EMAIL PROTECTED] wrote: I'm trying to use the _mssql module from http://pymssql.sourceforge.net/. It works fine on Python 2.4. I've just installed Python 2.5 Beta 2 on my Linux box and, whenever I try and run the mssql.close() function, or close the program, I get the following message: *** glibc detected *** python2.5: free(): invalid pointer: 0xb7f08320 I'm assuming that you recompiled the extension, rather than installed a binary built with a Python 2.4 installation. Python 2.5 tightens the requirements that memory allocated via one memory management API family must be released via calls through the same API family - eg malloc() - free(); PyMem_Malloc() - PyMem_Free(); PyObject_Malloc() - PyObject_Free(). I think I recall something about this in one of the whats new type docs. A fair bit of code erroneously assumes that free() can be used to release any block of memory, regardless of which API family was used to allocate the memory. This code needs to be fixed. -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Problem with odbc and Sql Server
Frank Millman wrote: Assume a table 't1' with a column 'c1' of type varchar(10). From Python, set c1 to an empty string - cur.execute(UPDATE t1 SET c1 = ?,['']) The result is that c1 is actually set to a string of 10 spaces. If I execute the command without using parameters - cur.execute(UPDATE t1 SET c1 = '') it works - c1 is set to an empty string. I am using Windows Server 2003 (SP1), Sql Server 2000 (SP4), and ODBC Driver 2000.86.1830.00. You might want to test with mxODBC, to see whether the problem is with the odbc module or something further down the stack. Given that it works with the MS-Jet (Access) ODBC provider (odbc interface), but not with the SQL Server provider, I'd say that the problem is more probably outside the control of the odbc module. I've not had much to do with SQL Server, but I wonder whether there is some configuration setting that might be affecting this behaviour. - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Slow network reading?
Ivan Voras wrote: Andrew MacIntyre wrote: Comparative CPU memory utilisation statistics, not to mention platform and version of Python, would be useful hints... During benchmarking, all versions cause all CPU to be used, but Python version has ~1.5x more CPU time allocated to it than PHP. Python is 2.4.1 A pretty fair indication of the Python interpreter doing a lot more work... Note that the file-like object returned by makefile() has significant portions of heavy lifting code in Python rather than C which can be a drag on ultimate performance... If on a Unix platform, it may be worth experimenting with os.fdopen() on the socket's fileno() to see whether the core Python file object (implemented in C) can be used in place of the lookalike returned from the makefile method. That's only because I need the .readline() function. In C, I'm using fgets() (with the expectation that iostream will buffer data). The readline method of the file object lookalike returned by makefile implements all of the line splitting logic in Python code, which is very likely where the extra process CPU time is going. Note that this code is in Python for portability reasons, as Windows socket handles cannot be used as file handles the way socket handles on Unix systems can be. If you are running on Windows, a fair bit of work will be required to improve performance as the line splitting logic needs to be moved to native code - I wonder whether psyco could do anything with this?. Even without that, you are specifying a buffer size smaller than the default (8k - see Lib/socket.py). 16k might be even better. The benchmark is such that all of data is 200 bytes. I estimate that in production almost all protocol data will be 4KB. A matter of taste perhaps, but that seems to me like another reason not to bother with a non-default buffer size. Although they're only micro-optimisations, I'd be interested in the relative performance of the query method re-written as: The change (for the better) is minor (3-5%). Given your comments above about how much data is actually involved, I'm a bit surprised that the tweaked version actually produced a measurable gain. - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: releasing interpreter lock in custom code?
Bram Stolk wrote: I've implemented, in C, a function that does a lot of I/O, and thus can block for a long time. If I execute this function in my Python script, it does not relinquish the global interpreter lock, like Python's native blocking functions do, like I/O funcs, and time.sleep() func. How can I have my func release the lock? Search the Python documentation (Python/C API section) for information about the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros. Perusing Python's source would also be educational (eg Modules/posixmodule.c). -- - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Slow network reading?
Ivan Voras wrote:
def query(self, sql):
self.sf.write(SQL %s\r\n % sql)
self.sf.flush()
resp = self.sf.readline().rstrip()
m = SQLCacheD.re_rec.match(resp)
if m != None: # only if some rows are returned (SELECT)
n_rows = int(m.group(1))
n_cols = int(m.group(2))
cols = []
for c in xrange(n_cols):
cols.append(self.sf.readline().rstrip())
rs = []
for r in xrange(n_rows):
row = {}
for c in cols:
row[c] = self.sf.readline().rstrip()
rs.append(row)
return rs
m = SQLCacheD.re_ok.match(resp)
if m != None: # no rows returned (e.g. INSERT/UPDATE/DELETE)
return True
raise SQLCacheD_Exception(resp)
Comparative CPU memory utilisation statistics, not to mention platform
and version of Python, would be useful hints...
Note that the file-like object returned by makefile() has significant
portions of heavy lifting code in Python rather than C which can be a
drag on ultimate performance... If on a Unix platform, it may be worth
experimenting with os.fdopen() on the socket's fileno() to see whether
the core Python file object (implemented in C) can be used in place of
the lookalike returned from the makefile method.
Even without that, you are specifying a buffer size smaller than the
default (8k - see Lib/socket.py). 16k might be even better.
Although they're only micro-optimisations, I'd be interested in the
relative performance of the query method re-written as:
def query(self, sql):
self.sf.write(SQL %s\r\n % sql)
self.sf.flush()
sf_readline = self.sf.readline
resp = sf_readline().rstrip()
m = self.re_rec.match(resp)
if m is not None:
# some rows are returned (SELECT)
rows = range(int(m.group(1)))
cols = range(int(m.group(2)))
for c in cols:
cols[c] = sf_readline().rstrip()
for r in rows:
row = {}
for c in cols:
row[c] = sf_readline().rstrip()
rows[r] = row
return rows
elif self.re_ok.match(resp) is not None:
# no rows returned (e.g. INSERT/UPDATE/DELETE)
return True
raise SQLCacheD_Exception(resp)
This implementation is based on 2 strategies for better performance:
- minimise name lookups by hoisting references from outside the method
to local references;
- pre-allocate lists when the required sizes are known, to avoid the
costs associated with growing them.
Both strategies can pay fair dividends when the repetition counts are
large enough; whether this is the case for your tests I can't say.
--
-
Andrew I MacIntyre These thoughts are mine alone...
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
--
http://mail.python.org/mailman/listinfo/python-list
Re: Building a Dynamic Library (libpython.so) for Python 2.4.3 Final
Dean wrote: I've been trying to make python a dynamic library. I downloaded Python 2.4.3 Final from the Python web site and I cannot get it to create the library. I've tried using the directive: --enable-shared and --enable-shared=yes and both of them had the same effect of creating a bunch of parts of the python interpreter in .so format but not in creating a single libpython2.4.so.X.Y file. I could probably hack something together using ar but I would prefer to do it correctly by setting some options. I'm compiling under OpenBSD 3.5. Well, I just confirmed that ./configure --enable-shared results in a libpython2.4.so.1 (and a libpython2.4.so) on FreeBSD 4.x. The result runs the regression test suite without significant problems (the test box didn't have a 2.4 installation). A quick look at the configure script suggests that --enable-shared should work on OpenBSD. While I believe that there's actually an OpenBSD based Python buildbot (though OpenBSD 3.[89] I think) running, I doubt that this configuration is tested there. If you don't get any other responses, send me a copy of the full make log (ie make make.log 21) and I'll compare it to the log from my test build. - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Python vs. Java gzip performance
Bill wrote:
I've written a small program that, in part, reads in a file and parses
it. Sometimes, the file is gzipped. The code that I use to get the
file object is like so:
if filename.endswith(.gz):
file = GzipFile(filename)
else:
file = open(filename)
Then I parse the contents of the file in the usual way (for line in
file:...)
The equivalent Java code goes like this:
if (isZipped(aFile)) {
input = new BufferedReader(new InputStreamReader(new
GZIPInputStream(new FileInputStream(aFile)));
} else {
input = new BufferedReader(new FileReader(aFile));
}
Then I parse the contents similarly to the Python version (while
nextLine = input.readLine...)
The Java version of this code is roughly 2x-3x faster than the Python
version. I can get around this problem by replacing the Python
GzipFile object with a os.popen call to gzcat, but then I sacrifice
portability. Is there something that can be improved in the Python
version?
The gzip module is implemented in Python on top of the zlib module. If
you peruse its source (particularly the readline() method of the GzipFile
class) you might get an idea of what's going on.
popen()ing a gzcat source achieves better performance by shifting the
decompression to an asynchronous execution stream (separate process)
while allowing the standard Python file object's optimised readline()
implementation (in C) to do the line splitting (which is done in Python
code in GzipFile).
I suspect that Java approach probably implements a similar approach
under the covers using threads.
Short of rewriting the gzip module in C, you may get some better
throughput by using a slightly lower level approach to parsing the file:
while 1:
line = z.readline(size=4096)
if not line:
break
... # process line here
This is probably only likely to be of use for files (such as log files)
with lines longer that the 100 character default in the readline()
method. More intricate approaches using z.readlines(sizehint=size)
might also work.
If you can afford the memory, approaches that read large chunks from the
gzipped stream then line split in one low level operation (so that the
line splitting is mostly done in C code) are the only way to lift
performance.
To me, if the performance matters, using popen() (or better: the
subprocess module) isn't so bad; it is actually quite portable
except for the dependency on gzip (probably better to use gzip -dc
rather than gzcat to maximise portability though). gzip is available
for most systems, and the approach is easily modified to use bzip2 as
well (though Python's bz2 module is implemented totally in C, and so
probably doesn't have the performance issues that gzip has).
-
Andrew I MacIntyre These thoughts are mine alone...
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
--
http://mail.python.org/mailman/listinfo/python-list
Re: Problem with dbi, odbc module and Oracle 9.2 - suffixed L with number data type
[posted mailed] [EMAIL PROTECTED] wrote: The only issue I've had so far is retrieving data from Oracle when an integer has been defined like: number(p)[same thing as number(p,0) evidently] This is from a database I didn't design and can't change. Evidently there are new ways to declare integer data types in Oracle. The problem is that the ODBC module suffixes an L to any integer returned that was defined as data type number(p). For example, an integer stored as: 56 will be returned as 56L. Actually, it now seems to be doing the same thing, at least in some cases, for number data types declared as number(p,s). What gives? Anyone know why this would happen? The 'L' suffix indicates a Python long, which is an arbitrary precision integer. If you're confident that the number in question is in the normal integer range, you can coerce it to a normal int with int(p) so the formatting on display doesn't include the suffix - if the number is too large to coerce an OverflowError will be raised (for 2.2 and earlier at least; 2.4 and later unify ints and longs). Can't use mxODBC because it's a commercial product and can't use cx_oracle at the moment because I am stuck with Python 2.1 (for ESRI geoprocessing), and there is no cx_oracle for Python 2.1 (starts with Python 2.2 and refuses to install for 2.1). I could install a later version of Python independently, but I need to be able to do the geoprocessing that 2.1 allows as well as ODBC calls to Oracle all in the same script. This means dbi,odbc seems to be my only choice. If you have access to a compiler, you may be able to build cx_Oracle for Python 2.1, but you would have to check that the code doesn't require Python 2.2 or later features. The MingW gcc package should work with Python 2.1 (which was built with VC6 as I recall). I've been bugging ESRI about upgrading, and I'm sure others have too. - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: threading IOError
Gabriel Genellina wrote: Hi I'm using Python 2.4.2 on Windows 98 SE. In a program with several threads, sometimes (I cant determine exactly when or why) one thread dies with the following traceback: 12/13/05 02:17:47 (WatchDog ) Unhandled thread exception Traceback (most recent call last): File E:\prog\pop3\TaskScheduler.py, line 60, in run self.finished.wait(self.interval) File C:\Apps\Python\Lib\threading.py, line 348, in wait self.__cond.wait(timeout) File C:\Apps\Python\Lib\threading.py, line 218, in wait remaining = endtime - _time() IOError: [Errno 2] No such file or directory The error appears to be inside the time module, and I can't explain the IOError there. Maybe this crash is related too: sometimes, the full program crashes with an Invalid Page Fault with the following info: PYTHONW provocó un error de página no válida en el módulo PYTHON24.DLL de 016f:1e0ab51f. Registros: EAX=1e19d1af CS=016f EIP=1e0ab51f EFLGS=00010206 EBX=00841f80 SS=0177 ESP=0071e6ec EBP= ECX=73962000 DS=0177 ESI=1e06a1b0 FS=1a07 EDX=1e19d1b0 ES=0177 EDI= GS= Any ideas? Quite some time ago I saw the same sort of issue - inexplicable exceptions from apparently benign code. Tim Peters prognosticated that there was a bug in an extension module, and indeed that proved to be the case (a 3rd party extension, which fortunately I had source for and was able to build). I doubt that any of Python's standard extension modules will be involved (as they are generally mature and widely used and tested) but you should look at the source for any others looking for unhandled error returns. Typically, errno is set but the failure return from the routine setting errno is ignored or not properly handled. What then happens is the next time an exception gets propagated through Python's internals, the errno value gets picked up and is used to identify the exception (incorrectly). The invalid page fault may well be because a garbage pointer is handed to a routine, possibly as a consequence of the mishandled error return. - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Using Python with COM to communicate with proprietary Windows software
On Fri, 09 Sep 2005 08:36:00 +0200, Thomas Heller [EMAIL PROTECTED]
wrote:
{...}
(I have released and announced this 3 weeks ago, but haven't got a
single feedback. So it seems the need to access custom interfaces is
very low.)
I have downloaded it and am trying to find the time to play with it
(unsuccessfully so far).
As someone working with a large, complex, COM library with minimal
IDispatch support, I'm really looking forward to this.
However, at the moment I'm limited to Python 2.2 and ctypes 0.6.3 (which
is allowing me to get the job done!!).
Regardless, I thank you for what you have released!
Cheers,
Andrew.
-
Andrew I MacIntyre These thoughts are mine alone...
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
--
http://mail.python.org/mailman/listinfo/python-list
Re: python and ARCView GIS desktop
GISDude wrote: hi all. I am a newbie, so be kind. I am using ARCView GIS 9.1 and python win. I am trying to develop a module using the GZIP module in my ARCView map session. What I am attempting to do (I think) is use the zip mod to zip up all the files in a .mxd document into one neat little zipped file, ready to copy to cd or email(if it's small enough(. I got this idea while using autocad. There is a function in autocad (E-TRANSMIT) that allows you to press the button on the toolbar and it will zip up all files(and dependent files too) in the current drawing and have it in one zip file ready for email. I think that ESRI intended that the Map Publisher (extra cost; outputs .pmf files as I recall) be used for this purpose. This is at least partly because map documents can contain all sorts of layers that don't exist as readily accessible files on your local machine. I was wondering if anyone had any ideas. Maybe I could use VBA or something else? If you are only using shapefiles, what you are after should be doable, and using VBA is probably the least painful way to do it. One trap that comes to mind is that dataset filenames are stored in .mxds in absolute form by default - you will need to change the setting in the normal.mxt which enables relative paths. There are examples of creating commands in the ArcObjects developers kit (I can't remember whether this is installed by default). You also might be able to use the InfoZip DLL directly from VBA to package everything (after you've figured out what needs to be packaged). - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: FTP over SSL (explicit encryption)
David Isaac wrote: I am looking for a pure Python secure ftp solution. Does it exist? I recall coming across an extension package (pretty sure it wasn't pure Python anyway, certainly not for the SSL bits) with SFTP - I think the name was Paramiko or something like that. - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: PPC floating equality vs. byte compilation
Donn Cave wrote: I ran into a phenomenon that seemed odd to me, while testing a build of Python 2.4.1 on BeOS 5.04, on PowerPC 603e. test_builtin.py, for example, fails a couple of tests with errors claiming that apparently identical floating point values aren't equal. But it only does that when imported, and only when the .pyc file already exists. Not if I execute it directly (python test_builtin.py), or if I delete the .pyc file before importing it and running test_main(). For now, I'm going to just write this off as a flaky build. I would be surprised if 5 people in the world care, and I'm certainly not one of them. I just thought someone might find it interesting. I have a faint recollection of seeing other references to this on other platforms. That faint recollection also seems to point to it being something to do with the marshalling of floats (.pyc files contain constants in a marshalled form). Don't think I've ever seen it myself... - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: IRIX MipsPro compiler chokes on Python.h
Bram Stolk wrote:
Hi there,
I just built and installed Python-2.4.1 on my Irix machine.
My compiler, the MipsPro compiler, chokes on the Python.h include file,
as demonstrated here:
$ CC -v
MIPSpro Compilers: Version 7.41
$ python -V
Python 2.4.1
$ cat l.cxx
#include Python.h
int main()
{
return 0;
}
$ CC -I $HOME/include/python2.4 l.cxx
cc-1311 CC: ERROR File = /usr/include/sys/time.h, Line = 186
The indicated linkage specification is incompatible with previous select
(declared at line 479 of /usr/include/unistd.h).
static int select(
^
1 error detected in the compilation of l.cxx.
Has anyone experienced the same?
How do I solve this?
My suspicion would be directed to a #define that is incorrect, as the
snippet above suggests that select() appears in two system headers
(/usr/include/sys/time.h /usr/include/unistd.h).
That is ./configure got something wrong and may need to be modified for
your platform to sort this out properly, but modifying pyconfig.h to
correct the incorrect definitions should get you out of trouble.
-
Andrew I MacIntyre These thoughts are mine alone...
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
--
http://mail.python.org/mailman/listinfo/python-list
Re: Build error Python 2.4.1 - stat problem?
Brandt, Servatius wrote: The stat values do not make any sense. It seems that the value used for the mode is really the numbers of links (I created the two empty /usr/local... directories to prevent the os.error exception): That sort of suggests that the definition of the stat struct is not what your build expects. If you used the configure script to generate the makefile, you should check whether any #define's modify the definition of this struct (usually in /usr/include/sys/stat.h), assuming configure found your system's stat.h, and see whether any such defines are set in your ./configure'd pyconfig.h. You may have to dig deeper still. - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: bsddb support for berkeley db 4.3?
[EMAIL PROTECTED] wrote: It doesn't seem like the python 2.4(and the recent 2.4.1) support berkeley db 4.3. (4.3 fixes some deadlock bugs I occasionally encounter using 4.2.) bsddb3(at pybsddb.sf.net) already supports 4.3 since last December(but doesn't explicitly support win32 -- see the assert statement in setup.py). I thought the bsddb3 project were merged into the python project. Hasn't it? Yes it has, but which version of BSD DB gets used when the release team builds the Win32 installer depends on the version installed on the installer builder's machine. If you can, I'd suggest posting a bug report on SF against 2.4 to see whether you can encourage the installer builder to upgrade BSD DB - though do be certain to check what's in 2.4.1c1 first. - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Problems with python and threads under Freebsd
snacktime wrote: After debugging this some more I narrowed it down to an encryption function in pycrypto that is triggering the segfault. I posted a bug report. Don't really know enough about threading to know whether it's a python bug or a pycrypto bug, but I'll let someone else sort that out now.. If you get bus errors, there's a possibility that you are running out of stack. AFAIR, the default thread stack size for Posix builds is dependant on the threading implementation. This can be overridden when building Python. With each gcc version since 2.9, the size of stack frames for generated calls has increased. If lowering the optimisation to -O or -Os when building Python improves the situation, I'd suggest pursuing this possibility further. - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Python evolution: Unease
On Wed, 5 Jan 2005, John Roth wrote: I would like to contribute some documentation to Python. I've got the time, I write quite a bit, etc. I've got fairly strong opinions about some things that need to be documented, (such as all the new style class descriptor stuff from 2.2) and I have relatively little difficulty matching the existing style. However, I don't know TEX, Latex, CVS or Sourceforge. (The latter two are on my learn sometime soon so I can put PyFIT where it belongs list.) I have no desire to install Perl to run the documentation toolchain. I also have no particular desire to write up a bunch of final format stuff and drop it on someone else to put into the latex format so it can be included. While being able to make doc changes at the Latex level directly into CVS is the ultimate, Fred Drake and others are quite happy to take straight text (ReST markup would probably help them a bit) as bugs/patches on sourceforge. - Andrew I MacIntyre These thoughts are mine alone... E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
