Re: file seek is slow
I almost wrote a long reply to all this. In the end it boils down to being concerned about how much overhead there is to calling a 'C' function. I assumed that file.seek() simply delegated to fseek() and thus was one way to test this overhead. However, I now think that it must be doing more and may not be a reasonable comparison. I have used the profiler about as much as I can to find where my program is slow, and it appears to me that the overhead to calling 'C' functions is now my biggest problem. I have been using Ctypes, which has been a great tool so far. I just discovered Cython and this looks like it may help me. I had not heard of pythoid, so I will check it out. I did not mean to offend anybody in Cython community. It just seemed funny to me that 21 lines of Python became 1478 lines of 'C'. I wasn't really expecting any response to this. I don't know enough about this to really assume anything. Stephan, I just tested 1e7 loops. 'C': 8.133 seconds Cython: 10.812 seconds I can't figure out what Cython is using for CFLAGS, so this could be important. I used While instead of xrange, because I thought it would be faster in Cython. They had roughly the same execution speed. Thanks all for the suggestions. I think I will just consider this thread closed. -- http://mail.python.org/mailman/listinfo/python-list
Re: file seek is slow
I am assuming that Python delegates the f.seek call to the seek call in the MS C runtime library msvcrt.dll. Does anybody know a nice link to the Python source like was posted above for the BSD 'C' library? Ok, I ran some more tests. C, seek: 0.812 seconds // test from original post Python, f.seek : 1.458 seconds. // test from original post C, time(tm) : 0.671 seconds Python, time.time(): 0.513 seconds. Python, ctypes.msvcrt.time(ctypes.byref(tm)): 0.971 seconds. # factored the overhead to be outside the loop, so really this was func_ptr(ptr). Perhaps I am just comparing apples to oranges. I never tested the overhead of ctypes like this before. Most of my problem timings involve calls through ctypes. -- http://mail.python.org/mailman/listinfo/python-list
Re: file seek is slow
I just tried the seek test with Cython.
Cython fseek() : 1.059 seconds. 30% slower than 'C'
Python f.seek : 1.458 secondds. 80% slower than 'C'.
It is amazing to me that Cython generates a 'C' file that is 1478
lines.
#Cython code
import time
cdef int SEEK_SET = 0
cdef extern from stdio.h:
void* fopen(char* filename, char* mode)
int fseek(void*, long, int)
def main():
cdef void* f1 = fopen('video.txt', 'rb')
cdef int i=100
t0 = time.clock()
while i 0:
fseek(f1, 0, SEEK_SET)
i -= 1
delta = time.clock() - t0
print %.3f % delta
if __name__ == '__main__':
main()
--
http://mail.python.org/mailman/listinfo/python-list
Re: file seek is slow
f1_seek = f1.seek did not change the performance at all. As it turns out each call is only 646 nanoseconds slower than 'C'. However, that is still 80% of the time to perform a file seek, which I would think is a relatively slow operation compared to just making a system call. -- http://mail.python.org/mailman/listinfo/python-list
Re: file seek is slow
Thanks, Tim. Good to know. -- http://mail.python.org/mailman/listinfo/python-list
file seek is slow
I ran a comparison that timed 1e6 file seeks.
The tests were run with Python 2.6.4 and Microsoft Visual C 6.0 on
Windows XP with an Intel 3GHz single processor with hyperthreading.
Results:
C: 0.812 seconds
Python: 1.458 seconds.
difference = 0.646 seconds.
If the file.seek is removed the Python loop takes 2ms so the loop
overhead is minimal.
Without pysco the loop overhead is 4.6ms and Python takes 1.866.
Any ideas what is causing the slow down over the 'C' version.
In general I have been trying to take a video application written in C+
+ and make it work in Python.
There seem to be delays in the handoff to Windows System code that
make the Python version just a touch slower in some areas, but these
slowdowns are critically effecting the work. File seek is not a deal
breaker here, it is just the latest thing I have noticed and the
simplest to demonstrate.
Python version:
import time
def main():
# write temp file
SIZE = 1000
f1 = file('video.txt', 'wb')
f1.write('+' * SIZE)
f1.close()
f1 = file('video.txt', 'rb')
t0 = time.clock()
for i in xrange(100):
f1.seek(0)
delta = time.clock() - t0
print %.3f % delta
f1.close()
if __name__ == '__main__':
import psyco
psyco.full()
main()
// 'C' version
#include stdio.h
#include time.h
#define SIZE 1000
static void main(int argc, char *argv[])
{
FILE *f1;
int i;
int t0;
float delta;
char buffer[SIZE];
// write temp file
memset(buffer, (int)'+', SIZE);
f1 = fopen(video.txt, wb);
fwrite(buffer, SIZE, 1, f1);
fclose(f1);
f1 = fopen(video.txt, rb);
t0 = clock();
for (i=0; i 100; i++)
{
fseek(f1, 0, SEEK_SET);
}
delta = (float)(clock() - t0) / CLOCKS_PER_SEC;
printf(%.3f\n, delta);
fclose(f1);
}
--
http://mail.python.org/mailman/listinfo/python-list
Multiprocessing.Pipe is not concurrently bi-directional (design flaw?)
I wanted to use Multiprocessing.Pipe and Multiprocessing.Process to peform Actor style communication. However, I just can't get what I need done, and I think it is due to inherit problems with the design of Pipe. If I have a process like this def process1(conn): while True: msg = conn.recv() if msg == 'go': res = go() # this is a long running function conn.send(res) # send result back to caller elif msg == 'stop': stop() # this is intended to stop the go() function, for example using deciding to abort operation. In this code the 'stop' message cannot be received, because the code is in the long running go() function. So lets change it. def process1(conn): while True: msg = conn.recv() if msg == 'go': t = threading.Thread(target=go, args=(conn,)) t.start() elif msg == 'stop': stop() # this is intended to stop the go() function. Now the thread-1 is waiting on the blocking recv(). If the go() function completes on thread-2, it wants to send its result via the connection object, but it can't because the connection is in use. Trying to work around this gets complicated. For example, using a second process and a second connection object to read the 'stop' message, and then changing a shared memory flag to stop the go() function. Pipes not being concurrently bi-directional appears to be a design flaw to me. pyCSP appears to be an interesting approach, but it does not support communication across O.S. processes on Win32. -- http://mail.python.org/mailman/listinfo/python-list
Re: Multiprocessing.Pipe is not concurrently bi-directional (design flaw?)
Well, I just realized that I could use a separate pipe to send the result back. This might work ok. -- http://mail.python.org/mailman/listinfo/python-list
Need help with multiprocessing.manager and passing the manager a multiprocessing.Connection
The following code snippet is taken from the Python 2.6
multiprocessing documentation with a simple change and this change
does not work. I would like to know how to make it work or something
similar.
I want to pass a Connection object to the MathsClass.
I get the following error on Windows:
Traceback (most recent call last):
File c:\apps\python26\lib\multiprocessing\managers.py, line 214,
in serve_cl
ient
request = recv()
TypeError: Required argument 'handle' (pos 1) not found
from multiprocessing.managers import BaseManager
from multiprocessing import Pipe# I added this
class MathsClass(object):
def set(self, conn): # I added the set() function
self.conn = conn
def add(self, x, y):
return x + y
def mul(self, x, y):
return x * y
class MyManager(BaseManager):
pass
MyManager.register('Maths', MathsClass)
if __name__ == '__main__':
parent_conn, child_conn = Pipe()# I added this
manager = MyManager()
manager.start()
maths = manager.Maths()
print maths.add(4, 3) # prints 7
print maths.mul(7, 8) # prints 56
maths.set(child_conn) # I added this, error is thrown here
--
http://mail.python.org/mailman/listinfo/python-list
Problem with multiprocessing managers
From the documentation for Using a remote manager there is the
following example code:
from multiprocessing.managers import BaseManager
import Queue
queue = Queue.Queue()
class QueueManager(BaseManager): pass
QueueManager.register('get_queue', callable=lambda:queue)
m = QueueManager(address=('', 5), authkey='abracadabra')
s = m.get_server()
s.serve_forever()
I don't know how to stop the server.
The documentation states that to call shutdown(), start() must be used
instead of server_forever().
If I use m.start() instead of m.get_server().server_forever() I
receive a PicklingError.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Is there a way to tell if a script has been run by Pythonw.exe instead of Python.exe?
Thanks to all, I learned something in each post. When using py2exe to build an executable sys.executable does not provide the name of the python interpreter but the name of the executable generated by py2exe. -- http://mail.python.org/mailman/listinfo/python-list
Is there a way to tell if a script has been run by Pythonw.exe instead of Python.exe?
In particular I want to know how to tell if reading and writing to the console can occur. Something like sys.isConsolePresent() -- http://mail.python.org/mailman/listinfo/python-list
Re: Python for Visual Basic or C# programmers
A.M wrote: Hi, I am trying to find the equivalent functions such as vb's str or asc in Python. Is there any resource that help me to find these kinds of functions in Python faster? Thank you, Alan Alan Python has a str() function that is close to vb's str. The Python version does not produce a leading space. str(123) -- '123' If you want to control the formatting to mimick vb or create different formatting you can use. %d % 123 -- ' 123' The format specifier is just like 'C's printf format strings. A tuple should follow the % sign if formatting more than 1 number. %d %d % (123, 456) Python has a similar function to vb's asc. It is ord(). ord() accepts a single character, whereas asc operates on the first character of a string. To mimick vb you could do s = 'hello' ord(s[0]) -- 104 I am unaware of any reference associating vb functionality with Python functionality. Then again, I never looked for one. -- http://mail.python.org/mailman/listinfo/python-list
Re: ctypes pointers and SendMessage
Ok, I see what I did wrong. I forgot to define the _fields_ variable. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python for Visual Basic or C# programmers
Slight correction. %d % 123 is not quite equivalent to str(123) as it does not handle negative numbers the same way. I am not sure there is a simple direct equivalent. %+d % 123 -- +123 always gives a sign. %4d % 123 -- 123 %4d % -123 -- -123 so this works if you you know how wide the number is. This might be a little too tricky. [ %d, %d][n 0] % n -- selects list[0] or list[1] based upon sign of number -- http://mail.python.org/mailman/listinfo/python-list
ctypes pointers and SendMessage
I would like to call windll.user32.SendMessageA(hwnd, win32con.WM_COMMAND, wParam, lParam) where lParam represents a pointer to an object. And also convert this pointer back to an object reference inside of wnd_proc def wnd_proc(hwnd, msg, wParam, lParam): So something like this: class X(Structure): def __init__(self, x): self.v = x x = X(1) windll.user32.SendMessageA(hwnd, WM_COMMAND, 4000, addressof(x)) def wnd_proc(hwnd, msg, wParam, lParam): if msg == WM_COMMAND and wParam == 4000): x = X.from_address(lParam) print x.v Unfortunately, this does not work. Also to my surprise the following does not work: x1 = X(1) x2 = X.from_address(addressof(x1)) addressof(x1) == addressof(x2) -- True but x2.v -- 'object has no attribute v' x1.v -- 1 Also this does not work as I expect. p = pointer(x) p[0].v -- 'object has no attribute v'. What am I doing wrong? -- http://mail.python.org/mailman/listinfo/python-list
Re: interactive shell -- reload definitions?
Thanks everybody. -- http://mail.python.org/mailman/listinfo/python-list
interactive shell -- reload definitions?
I have a question about the interactive Python shell. Is it possible to reload a file and get the new definitions. For example, if I do import xyz Then I find a bug in some function in xyz. So, I edit xyz.py I would like to reload the definitions in xyz.py without leaving my current shell session. Is this possible? Also, is there any way to save definitions created in the interactive shell to a file? Sometimes I write a simple function in the shell, and this wish I had that function in an external file. Thanks. -- http://mail.python.org/mailman/listinfo/python-list
Re: Queue limitations?
This example is missing a few initialization details although they can possibly be inferred. For example is iq = imageQueue() but imageQueue does not have a put() method. Is SetQueue() called? Is iq.start() called? I like to see small, fully complete and tested examples. The following works using strings as images. It might prove interesting to modify this test case to use your image objects instead of strings. import Queue import threading import unittest class imageQueue(threading.Thread): def __init__(self): threading.Thread.__init__(self) self.filenum = 0 self.theQueue = Queue.Queue(-1) self.done = False self.results = [] def run(self): while not self.done: # for testing, do something simple with images try: img = self.theQueue.get(True, 1) except: pass else: self.results.append(img) self.filenum += 1 def SetQueue(self, q_): self.theQueue = q_ def stop(self): self.done = True class Tester(unittest.TestCase): def setUp(self): pass def tearDown(self): pass def test_1(self): # initialize q = Queue.Queue() iq = imageQueue() iq.SetQueue(q) iq.start() # produce images images = [123, 456, 789] for x in images: q.put(x) # wait till all images consumed while iq.filenum != 3: pass # stop the thread iq.stop() # assert that the consumer consumed what was produced msg = %s != %s % (images, iq.results) self.assert_(images == iq.results, msg) if __name__ == '__main__': unittest.main() -- http://mail.python.org/mailman/listinfo/python-list
