Re: desperately looking for a howto on running my wxPython app on Vista
Mike Driscoll wrote: > > > Hmmm...I'm not familiar with that DLL, but a little googling seems to > indicate that you may be able to get it off your installation CD: it actually is there on my system. So this may be the red herring the Dependency Walker FAQ is warning for maybe I should leave this mess and revert to python 2.5 > > http://www.techimo.com/forum/applications-operating-systems/76550-winxp-msgina-dll-shlwapi-dll-problem.html > > The link above gives the steps for XP, but Vista will probably be > similar. I don't know for sure, but you may have to register the dll > once you've got it copied back onto your machine. > > Mike -- http://mail.python.org/mailman/listinfo/python-list
Re: desperately looking for a howto on running my wxPython app on Vista
Mike Driscoll wrote: > On Apr 29, 4:17 am, Paul Sijben wrote: >> Is there any way to check which is the offending pyd/dll? (normally >> Vista does not give out much data on what went wrong) >> >> Paul > > You might be able to find it using the Dependency Walker utility: > > http://www.dependencywalker.com/ Mike, thanks! That spotted indeed some issues with one specific file: SHLWAPI.DLL, now let's see what is the issue. The report from Dependency Walker:"Warning: At least one module has an unresolved import due to a missing export function in a delay-load dependent module." is not telling me much at this moment. The FAQ from Dependency Walker sais this is not a major issue, but apparently on Vista it is ?!? Does anyone have a clue what I am to do about this? Interestingly On WinXP it flags DWMAPI.DLL as missing. I did not spot that on the Vista box. Paul > > - Mike -- http://mail.python.org/mailman/listinfo/python-list
Re: desperately looking for a howto on running my wxPython app on Vista
Gabriel Genellina wrote: >> I am currently stuck on the infamous R6034 error but I understand that >> after that there may be another issue with certain wxPython functions. > > That should be fixed in Python 2.6.2, I think. > Are you compiling all your dependencies, including Python itself? R6034 > is likely a DLL mismatch between parts of your project. I am using 2.6.2 and am compiling only my own changed modules. For all the other support modules I have taken the most recent ones (win32, wxpython) Is there any way to check which is the offending pyd/dll? (normally Vista does not give out much data on what went wrong) Paul -- http://mail.python.org/mailman/listinfo/python-list
desperately looking for a howto on running my wxPython app on Vista
python 2.6, py2exe and Vista do not make a happy set. Unfortunately I am in dire need to launch my app not only on WinXP but also on Vista. I need 2.6 because of a number of support packages I am using and some of which I am compiling myself (and python 2.5 needs a version of visual studio that is no longer available.) I can find all kinds of advice and discussions with google but I have yet to find a clear explanation on what a poor developer like me is supposed to do to get my python 2.6.2 , stackless, wxpython and all my (self-built) pyds to work happily on Vista. I am currently stuck on the infamous R6034 error but I understand that after that there may be another issue with certain wxPython functions. Can someone please point me to a howto on crafting the right Setup.py and manifests etc. to make this work? Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: default shelve on linux corrupts, does different DB system help?
Thanks very much for a clear and concise explanation of the problem and
the solution!
I am implementing it now in my system. Luckily we caught this one during
testing so no important data has been lost.
Unfortunately windows does not seem to support gdbm. But in our case,
everything that is on the windows client is also available on the linux
server, so we can recreate the DB at the expense of some bandwidth in
case of failures.
Paul
s...@pobox.com wrote:
> Paul> I have the problem that my shelve(s) sometimes corrupt (looks like
> Paul> it has after python has run out of threads).
>
> Paul> I am using the default shelve so on linux I get the dbhash
> Paul> version. Is there a different DB type I can choose that is known
> Paul> to be more resilient? And if so, what is the elegant way of doing
> Paul> that?
>
> You don't say what version of Python you're using or what version of the
> Berkeley DB library underpins your installation, but I am going to guess it
> is 1.85. This has been known to have serious bugs for over a decade. (Just
> in the hash file implementation. The btree and recnum formats are ok.
> Unfortunately, the hash file implementation is what everybody has always
> gravitated to. Sort of like moths to a flame...)
>
> If that's the case, simply pick some other dbm file format for your shelves,
> e.g.:
>
> >>> import gdbm
> >>> import shelve
> >>> f = gdbm.open("/tmp/trash.db", "c")
> >>> f.close()
> >>> db = shelve.open("/tmp/trash.db")
> >>> db["mike"] = "sharon"
> >>> db["4"] = 5
> >>> db.keys()
> ['4', 'mike']
> >>> db.close()
> >>> f = gdbm.open("/tmp/trash.db", "c")
> >>> f.keys()
> ['4', 'mike']
> >>> f['4']
> 'I5\n.'
> >>> f['mike']
> "S'sharon'\np1\n."
>
> As for "uncorrupting" your existing database, see if your Linux distribution
> has a db_recover program. If it does, you might be able to retrieve your
> data, though in the case of BerkDB 1.85's hash file I'm skeptical that can
> be done. I hope you weren't storing something valuable in it like your bank
> account passwords.
>
>
--
http://mail.python.org/mailman/listinfo/python-list
default shelve on linux corrupts, does different DB system help?
I have the problem that my shelve(s) sometimes corrupt (looks like it has after python has run out of threads). I am using the default shelve so on linux I get the dbhash version. Is there a different DB type I can choose that is known to be more resilient? And if so, what is the elegant way of doing that? Paul -- http://mail.python.org/mailman/listinfo/python-list
import * and py2exe
I am trying to turn my application into a WinXP exe. Py2exe has packaged all my files up into one humongous executable. When trying to run the app, it complains that it can not find modules I just saw it include. These invariably are modules that have been imported using from import * Apparently this confuses py2exe. Well I tried unconfusing it by giving those modules as imports in the first place! How can I convince the py2exe-generated app to look for those modules in its own .exe file??? Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: Pyrex bitten by the exception bug
Stefan Behnel wrote: > Paul Sijben wrote: >> I am running into the (apparently) well-known issue with pyrex that >> trying to raise an exception using python2.5 and pyrex will lead to a >> TypeError, as "TypeError: exceptions must be strings, classes, or >> instances, not exceptions.ImportError" > > You should use a recent Pyrex version. I have :-( the most recent from their site >But using Cython is a better idea anyway. > I will look into that. Thanks, Paul > Stefan -- http://mail.python.org/mailman/listinfo/python-list
Pyrex bitten by the exception bug
I am running into the (apparently) well-known issue with pyrex that trying to raise an exception using python2.5 and pyrex will lead to a TypeError, as "TypeError: exceptions must be strings, classes, or instances, not exceptions.ImportError" Is there a fixc for this issue? best regards, Paul Sijben -- http://mail.python.org/mailman/listinfo/python-list
giving imp.load_module not a real file, HOW?
I am running into a problem with the python interpreter's internals. For some reason imp.load_module insists on getting a real open file as the second parameter. I have not able to fool it with stringIO or overloaded file objects. So now I have two questions: 1) why does load_module insist on a real file 2) is there a way around it that does not involve (say) tempfile? Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: encrypting python modules
Robert Latest wrote: > Paul Sijben wrote: > >> The problem: I have a client-server app written in python. I want to >> make sure that the client is not: >> 1) destabilized by users accidentally or on purpose dropping python >> files in the path (after which calling the helpdesk will not be useful) >> 2) extended with "new features" without me knowing about it (again >> resulting in calls to my helpdesk...) > > You could check the MD5 hashes of your files. > > robert indeed but I still need to hook into import to do that reliably, right? -- http://mail.python.org/mailman/listinfo/python-list
Re: encrypting python modules
Mike, thanks for the constructive feedback.Indeed i probably need to patch import in some way. Looks like there is no standard way to get this done. So I guess I have do it myself... In the famous last words department: how hard can that be? ;-) Paul Mike Meyer wrote: > On Sat, 12 Jan 2008 09:47:26 +1100 Ben Finney <[EMAIL PROTECTED]> wrote: > >> Paul Sijben <[EMAIL PROTECTED]> writes: >>> I know that I can not stop a dedicated hacker deconstructing my code. >> A direct consequence of this is that you can not stop *anyone* from >> deconstructing your code if it's in their possession. It takes only >> one dedicated, skilled person to crack your obfuscation system and >> distribute an automated process for doing so to anyone interested. > > Except that's not what he's trying to do. > >>> However I can not imagine that I would be the first one planning to >>> do this. So is there a solution like this available somewhere? >> Trying to make bits uncopyable and unmodifiable is like trying to make >> water not wet. > > And again, that's not what he's trying to do. He wants to arrange > things so that he doesn't have to support unmodified versions of his > code, by making it impossible to import modified modules. While that's > still impossible, once you decide how difficult you want to make it > for people to do that, you can *probably* make it that difficult - but > the process gets progressively more difficult and expensive as you > make it harder. > > I think he's contemplating only the simplest, least expensive step: > adding an import hook that only allows imports of digitally signed > modules. If planning to deploy on Windows, where he has to bundle a > python with his application, he may well implement the hook in the > interpreter instead of in python, so it's harder to find. > > If you wanted to go to the expense, you could probably arrange things > so that the digital signatures are the more vulnerable attack vectors, > but I'd expect to spend millions of dollars doing so. > >http://mail.python.org/mailman/listinfo/python-list
Re: encrypting python modules
> How often do these things *actually* happen? > > Of those that actually do it, how many are clueless enough that when they > run into problems they blame you for it? (And remember that you won't > even find out about the non-clueless ones.) > > This is a rethorical question, right? -- http://mail.python.org/mailman/listinfo/python-list
encrypting python modules
Hello, this question has come by repeatedly in several guises over the past years but has never been solved in this forum as far as I have been able to Google. However since so many people are asking the question, I hope someone has made a solution and is willing to share it. The problem: I have a client-server app written in python. I want to make sure that the client is not: 1) destabilized by users accidentally or on purpose dropping python files in the path (after which calling the helpdesk will not be useful) 2) extended with "new features" without me knowing about it (again resulting in calls to my helpdesk...) 3) trivially decompiled. Added issue, I want the client to be able to update itself when I have fixed bugs or created new functionality. I know that I can not stop a dedicated hacker deconstructing my code. Since all clients need to go through the server I am not afraid of "freeloaders". I am now considering overriding import with some code that will only import modules signed and crypted by me. However I can not imagine that I would be the first one planning to do this. So is there a solution like this available somewhere? Paul Sijben -- http://mail.python.org/mailman/listinfo/python-list
Re: finding bluetooth serial port
thanks very much! Grant Edwards wrote: > On 2007-11-07, Paul Sijben <[EMAIL PROTECTED]> wrote: > >> To automate/ease configuration in my app I am trying to find >> out to which serial port a certain bluetooth device is >> connected. With pybluez I can find out which bluetooth devices >> I have, but it will not tell me the serial port they are >> mapped to. >> >> Is there a way to figure this out from python? (I am >> insterested in the platforms WinXP and linux primarily) > > Under linux, the "right" thing to do is to write a udev rule so > that the device has a predictiable name (or symlink). > > http://reactivated.net/writing_udev_rules.html > > If you don't want to write a udev rule, you'll need to bascally > re-implement udev by parsing the sysfs directory tree until you > find the device you're looking for. Here's how to do it for USB > (I assume BT works in a similar fashion). > > Let's say I know the device has vendor ID 0403, product ID > 6001, and serial number 123456. > > I search through the directories under /sys/devices until I > find a directory containing three files named > > idProduct > idVendor > serial > > Which contain the three strings I'm looking for. > > In this case: > > # cat /sys/devices/pci:00/:00:10.3/usb5/5-1/idVendor > 0403 > # cat /sys/devices/pci:00/:00:10.3/usb5/5-1/idProduct > 6001 > # cat /sys/devices/pci:00/:00:10.3/usb5/5-1/serial > 12345678 > > Once you've found that directory, you can look at the other > entries to find out whatever you want to know about the device: > > /sys/devices/pci:00/:00:10.3/usb5/5-1/ > |-- 5-1:1.0 > | |-- bAlternateSetting > | |-- bInterfaceClass > | |-- bInterfaceNumber > | |-- bInterfaceProtocol > | |-- bInterfaceSubClass > | |-- bNumEndpoints > | |-- bus -> ../../../../../../bus/usb > | |-- driver -> ../../../../../../bus/usb/drivers/ftdi_sio > | |-- ep_02 -> > ../../../../../../devices/pci:00/:00:10.3/usb5/5-1/5-1:1.0/usbdev5.42_ep02 > | |-- ep_81 -> > ../../../../../../devices/pci:00/:00:10.3/usb5/5-1/5-1:1.0/usbdev5.42_ep81 > | |-- interface > | |-- modalias > | |-- power > | | |-- state > | | `-- wakeup > | |-- subsystem -> ../../../../../../bus/usb > | |-- ttyUSB0 > | | |-- bus -> ../../../../../../../bus/usb-serial > | | |-- driver -> ../../../../../../../bus/usb-serial/drivers/ftdi_sio > | | |-- power > | | | |-- state > | | | `-- wakeup > | | |-- subsystem -> ../../../../../../../bus/usb-serial > | | |-- tty:ttyUSB0 -> ../../../../../../../class/tty/ttyUSB0 > | | `-- uevent > | |-- uevent > | |-- usb_endpoint:usbdev5.42_ep02 -> > ../../../../../../devices/pci:00/:00:10.3/usb5/5-1/5-1:1.0/usbdev5.42_ep02 > | |-- usb_endpoint:usbdev5.42_ep81 -> > ../../../../../../devices/pci:00/:00:10.3/usb5/5-1/5-1:1.0/usbdev5.42_ep81 > | |-- usbdev5.42_ep02 > | | |-- bEndpointAddress > | | |-- bInterval > | | |-- bLength > | | |-- bmAttributes > | | |-- dev > | | |-- device -> > ../../../../../../../devices/pci:00/:00:10.3/usb5/5-1/5-1:1.0 > | | |-- direction > | | |-- interval > | | |-- power > | | | |-- state > | | | `-- wakeup > | | |-- subsystem -> ../../../../../../../class/usb_endpoint > | | |-- type > | | |-- uevent > | | `-- wMaxPacketSize > | `-- usbdev5.42_ep81 > | |-- bEndpointAddress > | |-- bInterval > | |-- bLength > | |-- bmAttributes > | |-- dev > | |-- device -> > ../../../../../../../devices/pci:00/:00:10.3/usb5/5-1/5-1:1.0 > | |-- direction > | |-- interval > | |-- power > | | |-- state > | | `-- wakeup > | |-- subsystem -> ../../../../../../../class/usb_endpoint > | |-- type > | |-- uevent > | `-- wMaxPacketSize > |-- bConfigurationValue > |-- bDeviceClass > |-- bDeviceProtocol > |-- bDeviceSubClass > |-- bMaxPacketSize0 > |-- bMaxPower > |-- bNumConfigurations > |-- bNumInterfaces > |-- bcdDevice > |-- bmAttributes > |-- bus -> ../../../../../bus/usb > |-- configuration > |-- devnum > |-- driver -> ../../../../../bus/usb/drivers/usb > |-- ep_00 -> > ../../../../../devices/pci:00/:00:10.3/usb5/5-1/usbdev5.42_ep00 > |-- event_char > |-- idProduct > |-- idVendor > |-- manufacturer > |-- maxchild > |-- power > | |-- state > | `-- wakeu
finding bluetooth serial port
To automate/ease configuration in my app I am trying to find out to which serial port a certain bluetooth device is connected. With pybluez I can find out which bluetooth devices I have, but it will not tell me the serial port they are mapped to. Is there a way to figure this out from python? (I am insterested in the platforms WinXP and linux primarily) Paul Sijben -- http://mail.python.org/mailman/listinfo/python-list
Re: python 2.5.1 segfault, multithreading & dual core issue?
thanks very much! I am currently compiling python with the patch and will test it over the coming days. Paul Hrvoje Niksic wrote: > Paul Sijben <[EMAIL PROTECTED]> writes: > >> I am running a multi-threaded python application in a dual core >> intel running Ubuntu. > [...] > > Judging from the stack trace, this patch has a good chance of fixing > your problem: > > http://mail.python.org/pipermail/python-dev/2007-August/074232.html -- http://mail.python.org/mailman/listinfo/python-list
python 2.5.1 segfault, multithreading & dual core issue?
I am running a multi-threaded python application in a dual core intel running Ubuntu. I am using python 2.5.1 that I compiled myself. At random points I am getting segmentation faults (sometimes indicating a duplicate free). Below is the backtrace of the latest segfault. I am thinking this might be an issue related to the dual core CPU so I am now running the app with affinity to one CPU to test this hypothesis. Where can I put such a bugreport? Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1486967920 (LWP 4579)] 0x009196ed in fclose@@GLIBC_2.1 () from /lib/libc.so.6 (gdb) bt #0 0x009196ed in fclose@@GLIBC_2.1 () from /lib/libc.so.6 #1 0x0806af9b in file_close (f=0xa259e30) at Objects/fileobject.c:446 #2 0x080c51b0 in PyEval_EvalFrameEx (f=0xa3c790c, throwflag=0) at Python/ceval.c:3548 #3 0x080c5795 in PyEval_EvalFrameEx (f=0xa3c77a4, throwflag=0) at Python/ceval.c:3650 #4 0x080c5795 in PyEval_EvalFrameEx (f=0xa3afccc, throwflag=0) at Python/ceval.c:3650 #5 0x080c5795 in PyEval_EvalFrameEx (f=0xa3c8804, throwflag=0) at Python/ceval.c:3650 #6 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f315c0, globals=0xb7f29824, locals=0x0, args=0xa39b9a8, argcount=3, kws=0xa39b9b4, kwcount=0, defs=0xb7c8fb98, defcount=1, closure=0x0) at Python/ceval.c:2831 #7 0x080c4a59 in PyEval_EvalFrameEx (f=0xa39b864, throwflag=0) at Python/ceval.c:3660 #8 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f2ec38, globals=0xb7f29824, locals=0x0, args=0xa399920, argcount=3, kws=0xa39992c, kwcount=0, defs=0xb7c8fad8, defcount=1, closure=0x0) at Python/ceval.c:2831 #9 0x080c4a59 in PyEval_EvalFrameEx (f=0xa3997c4, throwflag=0) at Python/ceval.c:3660 #10 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f2eda0, globals=0xb7f29824, locals=0x0, args=0xa340e30, argcount=3, kws=0xa340e3c, kwcount=0, defs=0xb7cea330, defcount=4, closure=0x0) at Python/ceval.c:2831 #11 0x080c4a59 in PyEval_EvalFrameEx (f=0xa340cb4, throwflag=0) at Python/ceval.c:3660 #12 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f317b8, globals=0xb7f29824, locals=0x0, args=0xa39cedc, argcount=2, kws=0xa39cee4, kwcount=2, defs=0xb7c905b0, defcount=3, closure=0x0) at Python/ceval.c:2831 #13 0x080c4a59 in PyEval_EvalFrameEx (f=0xa39cd7c, throwflag=0) at Python/ceval.c:3660 #14 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f316e0, globals=0xb7f29824, locals=0x0, args=0xa3aff8c, argcount=4, kws=0xa3aff9c, kwcount=1, defs=0xb7c8fbb8, defcount=2, closure=0x0) at Python/ceval.c:2831 #15 0x080c4a59 in PyEval_EvalFrameEx (f=0xa3afe34, throwflag=0) at Python/ceval.c:3660 #16 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f07728, globals=0xb7f59acc, locals=0x0, args=0xa32b9ac, argcount=3, kws=0xa32b9b8, kwcount=0, defs=0xb7c0a1f8, defcount=2, closure=0x0) at Python/ceval.c:2831 #17 0x080c4a59 in PyEval_EvalFrameEx (f=0xa32b86c, throwflag=0) at Python/ceval.c:3660 #18 0x080c5795 in PyEval_EvalFrameEx (f=0xa3c84f4, throwflag=0) at Python/ceval.c:3650 #19 0x080c5795 in PyEval_EvalFrameEx (f=0xa3a4ef4, throwflag=0) at Python/ceval.c:3650 #20 0x080c65a5 in PyEval_EvalCodeEx (co=0xb7f1d1d0, globals=0xb7f11dfc, locals=0x0, args=0xa2d8cd8, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2831 #21 0x0810d6a1 in function_call (func=0xb7f26e2c, arg=0xa2d8ccc, kw=0x0) at Objects/funcobject.c:517 #22 0x0805a257 in PyObject_Call (func=0xa48ff4, arg=0xa2d8ccc, kw=0x0) at Objects/abstract.c:1860 #23 0x08060387 in instancemethod_call (func=0xa23b2fc, arg=0xa2d8ccc, kw=0x0) at Objects/classobject.c:2497 #24 0x0805a257 in PyObject_Call (func=0xa48ff4, arg=0xb7f4102c, kw=0x0) at Objects/abstract.c:1860 #25 0x080be79c in PyEval_CallObjectWithKeywords (func=0xa23b2fc, arg=0xb7f4102c, kw=0x0) at Python/ceval.c:3433 #26 0x080f01b8 in t_bootstrap (boot_raw=0xa3c8048) at ./Modules/threadmodule.c:424 #27 0x00a3a45b in start_thread () from /lib/libpthread.so.0 #28 0x0099223e in clone () from /lib/libc.so.6 -- http://mail.python.org/mailman/listinfo/python-list
Re: Windows XP timezone language issue
MRAB wrote:
> On Jun 13, 7:31 am, Paul Sijben <[EMAIL PROTECTED]> wrote:
>> I ran into an internationalization issue. I need a consistent idea about
>> the timezone my application is running on. However when I run the following:
>> >>> import time
>> >>> time.tzname
>>
>> I get back ('West-Europa (standaardtijd)', 'West-Europa (zomertijd)')
>> which is in dutch (the language of the host machine) and verbose.
>> I wanted to get ('CEST','CET') or something international so I can work
>> with itin the same way on all platforms.
>>
>> That is the right way to find out the timezone in a consistent way
>> across platforms (windows/linux/mac) and languages?
>>
> Well, time.timezone will return the timezone as an integer.
>
true, I am wondering if that is enough for my app. (with some of the odd
timezones in the world and DTS issues and all
--
http://mail.python.org/mailman/listinfo/python-list
Windows XP timezone language issue
I ran into an internationalization issue. I need a consistent idea about
the timezone my application is running on. However when I run the following:
>>> import time
>>> time.tzname
I get back ('West-Europa (standaardtijd)', 'West-Europa (zomertijd)')
which is in dutch (the language of the host machine) and verbose.
I wanted to get ('CEST','CET') or something international so I can work
with itin the same way on all platforms.
That is the right way to find out the timezone in a consistent way
across platforms (windows/linux/mac) and languages?
Paul
--
http://mail.python.org/mailman/listinfo/python-list
Re: audio video streaming communications
Hi Ken, I am looking for something similar. I can do the communications myself but need to be able to select a video feed, capture it and also need to display it through wxPython. Trawled the web and even tried to hire coders to create it for me. So far I have been having no luck. I did learn that fastaudio is a good way to bluescreen a WinXP box ;-) Have you found something? Paul Ken Seehart wrote: > Hello, > > I am looking for a good audio/video conferencing library. Ideally it > should work with wxPython (or have some means of making it work there). > > So far my main difficulty in my attempt at searching for such a package > is that there is so much stuff out there on downloading music and videos. > > I am not interested in download torrents, etc. I'm just looking video > conferencing, and live video broadcasts, etc. > > Any recommendations? > > Thanks, > - Ken > > -- http://mail.python.org/mailman/listinfo/python-list
Re: worker thread catching exceptions and putting them in queue
Stargaming & Diez, thanks very much! Stargaming wrote: > Paul Sijben schrieb: >> All, >> >> in a worker thread setup that communicates via queues is it possible to >> catch exceptions raised by the worker executed, put them in an object >> and send them over the queue to another thread where the exception is >> raised in that scope? >> >> considering that an exception is an object I feel it ought to be >> possible, however I do not see how to go about it. >> >> does anyone have a pointer towards the solution? >> >> Paul > > You're right, even exceptions are objects in Python. For further > studies, read http://docs.python.org/lib/module-exceptions.html > > You can catch an exception like this: > try: > worker.do_some_work_that_may_raise_an_exception() > except Exception, e: > # the first argument is the type of error you want to handle > # it is Exception here, the baseclass of all computation exceptions > # the second argument is the variable (name) where you want to save > # the specific exception raised to > # it's 'e' here, a common shortcut for exception > exception_handler.handle(e) > # notice that you can pass e around as you like > > For further information on exceptions and how to handle them, read > chapter 8 of the tutorial, especially starting from 8.3: > http://docs.python.org/tut/node10.html#SECTION001030 > > P.S. I don't know if what I told still applies to Python 3.0 -- a lot of > changes are upcoming related to exception raising and handling. -- http://mail.python.org/mailman/listinfo/python-list
worker thread catching exceptions and putting them in queue
All, in a worker thread setup that communicates via queues is it possible to catch exceptions raised by the worker executed, put them in an object and send them over the queue to another thread where the exception is raised in that scope? considering that an exception is an object I feel it ought to be possible, however I do not see how to go about it. does anyone have a pointer towards the solution? Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: maximum number of threads
All thanks for all the input! This was very informative. Looks like I indeed need stackless as my code benefits from being concurrently designed. Paul Jean-Paul Calderone wrote: > On Wed, 10 Jan 2007 12:11:59 -0200, Felipe Almeida Lessa > <[EMAIL PROTECTED]> wrote: >> On 1/10/07, Laurent Pointal <[EMAIL PROTECTED]> wrote: >>> This is a system configurable limit (up to a maximum). >>> >>> See ulimit man pages. >>> >>> test >>> >>> ulimit -a >>> >>> to see what are the current limits, and try with >>> >>> ulimit -u 2000 >>> >>> to modify the maximum number of user process (AFAIK each thread use a >>> process entry on Linux) >> >> I don't think it's only this. > > Indeed you are correct. The actual limit you are hitting is the size > of your address space. Each thread is allocated 8MB of stack. 382 > threads consumes about 3GB of address space. Even though most of this > memory isn't actually allocated, the address space is still used up. So, > when you try to create the 383rd thread, the kernel can't find anyplace > to put its stack. So you can't create it. > > Try reducing your stack size or reducing the number of threads you create. > There's really actually almost no good reason to have this many threads, > even though it's possible. > >[EMAIL PROTECTED]:~$ python Desktop/test.py >50 >100 >150 >200 >250 >300 >350 >Exception raised: can't start new thread > Biggest number of threads: 382 >[EMAIL PROTECTED]:~$ ulimit -Ss 4096 >[EMAIL PROTECTED]:~$ python Desktop/test.py >50 >100 >150 >200 >250 >300 >350 >400 >450 >500 >550 >600 >650 >700 >750 >Exception raised: can't start new thread > Biggest number of threads: 764 >[EMAIL PROTECTED]:~$ >Jean-Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: convert binary data to int
in some cases struct.unpack would also help [EMAIL PROTECTED] wrote: > Hello, > > > I need to convert a 3 byte binary string like > "\x41\x00\x00" to 3 int values ( (65,0,0) in this case). > The string might contain characters not escaped with a \x, like > "A\x00\x00" > > > Any ideas? > > > Daniel > -- http://mail.python.org/mailman/listinfo/python-list
Re: maximum number of threads
Felipe Almeida Lessa wrote: > Maybe Stackless could help the OP? > http://www.stackless.com/ > thanks I will look into it! -- http://mail.python.org/mailman/listinfo/python-list
Re: maximum number of threads
Gabriel Genellina wrote: > > Simply you can't, as you can't have 1 open files at once. Computer > resources are not infinite. sure but *how* fast they run out is the issue here > Do you really need so many threads? I might be able to do with a few less but I still need many. I have done a quick test. on WinXP I can create 1030 threads on Fedora Core 6 I can only create 301 (both python2.4 and 2.5) now the 301 is rather low I'd say. Paul -- http://mail.python.org/mailman/listinfo/python-list
maximum number of threads
I have a server in Python 2.5 that generates a lot of threads. It is running on a linux server (Fedora Core 6). The server quickly runs out of threads. I am seeing the following error. File "/home/sijben/ORCA/src/libmercury_mt.py", line 565, in __init__ MercuryObject.__init__(self,mylink) File "/home/sijben/ORCA/src/libmercury_mt.py", line 223, in __init__ self.start() File "/usr/local/lib/python2.5/threading.py", line 434, in start _start_new_thread(self.__bootstrap, ()) error: can't start new thread Does anyone know what it going on here and how I can ensure that I have all the threads I need? Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: Python UPnP on Linux?
You are right of course but I was hoping to avoid that. Twisted is very large and has all kinds of internal dependencies. Sybren Stuvel wrote: > Paul Sijben enlightened us with: >> Googling on this I have found win32 implementations and Twisted >> implementations yet I am looking for a way to do it on Linux WITHOUT >> Twisted. > > Twisted is Open Source, so you could browse the source and see how > they do it. > > Sybren -- http://mail.python.org/mailman/listinfo/python-list
Python UPnP on Linux?
I am writing some client software that needs to accept connections from the Internet. So I am looking for a UPnP implementation for a client of an Internet gateway so I can request the NAT binding. Googling on this I have found win32 implementations and Twisted implementations yet I am looking for a way to do it on Linux WITHOUT Twisted. Who knows a pointer? Paul -- http://mail.python.org/mailman/listinfo/python-list
Re: FOUNDIT (was Re: massive threading performance)
Lawrence D'Oliveiro wrote: > In article <[EMAIL PROTECTED]>, > Paul Sijben <[EMAIL PROTECTED]> wrote: > >> I found that the problem was caused by the sending thread not giving >> control back quickly enough to the receiving thread. >> >> Also in going through the code I found an old self.s.setblocking(0)call >> that was no longer relevant. Removing that solved my problem. >> >> Something that took 20 seconds now takes just 1. > > You might also find that it goes still faster if you forego threading > and use a select.select loop. thanks for that. I will have to try it. -- http://mail.python.org/mailman/listinfo/python-list
massive threading performance (was:Re: UDP performance)
OK the problem I posted about earlier is NOT a UDP/socket problem, it is
a threading problem. Albeit one that only happens when you have many
thrreads
I have made two little scripts using the code I copied below, one client
and one server (attached).
If I run them concurrently on the same machine I see the following times:
>>1145526192.825508
<<1145526192.825848
>>1145526193.829325
<<1145526193.834927
a transfer time in the milliseconds. Much less than the times I see in
the full application.
OK so I put them both in a multithreaded script (also attached)
>>1145526971.619558
<<1145526971.619909
>>1145526972.619241
<<1145526972.619647
again transfer time in milliseconds.
Not like this that I get from the profile of my code:
<< 1145517554.363850 send
>> 1145517554.647485 recv
which uses the same communication and threading but has 20+ threads?
Now I am completely baffled!
again I really appreciate if anyone can shed light on this!
Paul
Paul Sijben wrote:
> I am stumped by the following problem. I have a large multi-threaded
> server accepting communications on one UDP port (chosen for its supposed
> speed).
>
> I have been profiling the code and found that the UDP communication is
> my biggest drain on performance! Communication where the client and the
> server are on the same machine still takes 300ms or sometimes much more
> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).
>
> I must be doing something wrong and would really appreciate feedback on
> my code below:
>
> I open the server port with
>
> self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> self.s.bind((myaddress, myport))
>
> I then open a client port with
>
> self.s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> self.s.connect((host, port))
>
> the client sends data with
>
> self.s.sendall(data)
>
> and the server with
>
> self.s.sendto(data,link.remoteaddress)
>
> both receive with
>
> buf, address = socket.recvfrom(8192)
>
> The sender and receiver are in separate threads (threading.Thread).
>
> Does anyone know what is going wrong here, the socket communication, the
> thread scheduling?
>
> Paul Sijben
>
import socket
import time
profile=[]
s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(('127.0.0.1', 42420))
print ">>%f"%time.time()
s.sendall('testerdetest')
time.sleep(1)
print ">>%f"%time.time()
s.sendall('testerdetest')
print ">>%f"%time.time()
import socket
import time
profile=[]
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', 42420))
buf, address =s.recvfrom(8192)
print "<<%f"%time.time()
print buf, address
buf, address =s.recvfrom(8192)
print "<<%f"%time.time()
print buf, addressimport socket
import time
import threading
profile=[]
class server(threading.Thread):
def __init__(self):
self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.s.bind(('', 42420))
threading.Thread.__init__(self)
self.start()
def run(self):
buf, address =self.s.recvfrom(8192)
print "<<%f"%time.time()
print buf, address
buf, address =self.s.recvfrom(8192)
print "<<%f"%time.time()
print buf, address
class client(threading.Thread):
def __init__(self):
self.s =socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.connect(('127.0.0.1', 42420))
threading.Thread.__init__(self)
self.start()
def run(self):
print ">>%f"%time.time()
self.s.sendall('testerdetest')
time.sleep(1)
print ">>%f"%time.time()
self.s.sendall('testerdetest')
print ">>%f"%time.time()
serv=server()
cl=client()--
http://mail.python.org/mailman/listinfo/python-list
FOUNDIT (was Re: massive threading performance)
I found that the problem was caused by the sending thread not giving
control back quickly enough to the receiving thread.
Also in going through the code I found an old self.s.setblocking(0)call
that was no longer relevant. Removing that solved my problem.
Something that took 20 seconds now takes just 1.
Thanks to those who took the time to respond to my earlier messages.
Paul
Paul Sijben wrote:
> OK the problem I posted about earlier is NOT a UDP/socket problem, it is
> a threading problem. Albeit one that only happens when you have many
> thrreads
>
> I have made two little scripts using the code I copied below, one client
> and one server (attached).
>
> If I run them concurrently on the same machine I see the following times:
>>> 1145526192.825508
> <<1145526192.825848
>
>>> 1145526193.829325
> <<1145526193.834927
>
> a transfer time in the milliseconds. Much less than the times I see in
> the full application.
>
> OK so I put them both in a multithreaded script (also attached)
>
>
>>> 1145526971.619558
> <<1145526971.619909
>
>>> 1145526972.619241
> <<1145526972.619647
>
> again transfer time in milliseconds.
>
> Not like this that I get from the profile of my code:
> << 1145517554.363850 send
>>> 1145517554.647485 recv
>
> which uses the same communication and threading but has 20+ threads?
>
> Now I am completely baffled!
>
> again I really appreciate if anyone can shed light on this!
>
> Paul
>
>
> Paul Sijben wrote:
>> I am stumped by the following problem. I have a large multi-threaded
>> server accepting communications on one UDP port (chosen for its supposed
>> speed).
>>
>> I have been profiling the code and found that the UDP communication is
>> my biggest drain on performance! Communication where the client and the
>> server are on the same machine still takes 300ms or sometimes much more
>> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).
>>
>> I must be doing something wrong and would really appreciate feedback on
>> my code below:
>>
>> I open the server port with
>>
>> self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>> self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
>> self.s.bind((myaddress, myport))
>>
>> I then open a client port with
>>
>> self.s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
>> self.s.connect((host, port))
>>
>> the client sends data with
>>
>> self.s.sendall(data)
>>
>> and the server with
>>
>> self.s.sendto(data,link.remoteaddress)
>>
>> both receive with
>>
>> buf, address = socket.recvfrom(8192)
>>
>> The sender and receiver are in separate threads (threading.Thread).
>>
>> Does anyone know what is going wrong here, the socket communication, the
>> thread scheduling?
>>
>> Paul Sijben
>>
>
>
>
>
>
> import socket
> import time
>
> profile=[]
>
> s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> s.connect(('127.0.0.1', 42420))
> print ">>%f"%time.time()
> s.sendall('testerdetest')
> time.sleep(1)
> print ">>%f"%time.time()
> s.sendall('testerdetest')
> print ">>%f"%time.time()
>
>
>
>
>
> import socket
> import time
>
> profile=[]
>
>
> s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> s.bind(('', 42420))
>
>
> buf, address =s.recvfrom(8192)
> print "<<%f"%time.time()
> print buf, address
> buf, address =s.recvfrom(8192)
> print "<<%f"%time.time()
> print buf, address
>
>
>
>
> import socket
> import time
> import threading
> profile=[]
>
> class server(threading.Thread):
> def __init__(self):
> self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> self.s.bind(('', 42420))
> threading.Thread.__init__(self)
> self.start()
>
> def run(self):
> buf, address =self.s.recvfrom(8192)
> print "<<%f"%time.time()
>
massive threading performance (was:Re: UDP performance)
OK the problem I posted about earlier is NOT a UDP/socket problem, it is
a threading problem. Albeit one that only happens when you have many
thrreads
I have made two little scripts using the code I copied below, one client
and one server (attached).
If I run them concurrently on the same machine I see the following times:
>>1145526192.825508
<<1145526192.825848
>>1145526193.829325
<<1145526193.834927
a transfer time in the milliseconds. Much less than the times I see in
the full application.
OK so I put them both in a multithreaded script (also attached)
>>1145526971.619558
<<1145526971.619909
>>1145526972.619241
<<1145526972.619647
again transfer time in milliseconds.
Not like this that I get from the profile of my code:
<< 1145517554.363850 send
>> 1145517554.647485 recv
which uses the same communication and threading but has 20+ threads?
Now I am completely baffled!
again I really appreciate if anyone can shed light on this!
Paul
Paul Sijben wrote:
> I am stumped by the following problem. I have a large multi-threaded
> server accepting communications on one UDP port (chosen for its supposed
> speed).
>
> I have been profiling the code and found that the UDP communication is
> my biggest drain on performance! Communication where the client and the
> server are on the same machine still takes 300ms or sometimes much more
> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).
>
> I must be doing something wrong and would really appreciate feedback on
> my code below:
>
> I open the server port with
>
> self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
> self.s.bind((myaddress, myport))
>
> I then open a client port with
>
> self.s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
> self.s.connect((host, port))
>
> the client sends data with
>
> self.s.sendall(data)
>
> and the server with
>
> self.s.sendto(data,link.remoteaddress)
>
> both receive with
>
> buf, address = socket.recvfrom(8192)
>
> The sender and receiver are in separate threads (threading.Thread).
>
> Does anyone know what is going wrong here, the socket communication, the
> thread scheduling?
>
> Paul Sijben
>
import socket
import time
profile=[]
s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(('127.0.0.1', 42420))
print ">>%f"%time.time()
s.sendall('testerdetest')
time.sleep(1)
print ">>%f"%time.time()
s.sendall('testerdetest')
print ">>%f"%time.time()
import socket
import time
profile=[]
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', 42420))
buf, address =s.recvfrom(8192)
print "<<%f"%time.time()
print buf, address
buf, address =s.recvfrom(8192)
print "<<%f"%time.time()
print buf, address
import socket
import time
import threading
profile=[]
class server(threading.Thread):
def __init__(self):
self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.s.bind(('', 42420))
threading.Thread.__init__(self)
self.start()
def run(self):
buf, address =self.s.recvfrom(8192)
print "<<%f"%time.time()
print buf, address
buf, address =self.s.recvfrom(8192)
print "<<%f"%time.time()
print buf, address
class client(threading.Thread):
def __init__(self):
self.s =socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.s.connect(('127.0.0.1', 42420))
threading.Thread.__init__(self)
self.start()
def run(self):
print ">>%f"%time.time()
self.s.sendall('testerdetest')
time.sleep(1)
print ">>%f"%time.time()
self.s.sendall('testerdetest')
print ">>%f"%time.time()
serv=server()
cl=client()
--
http://mail.python.org/mailman/listinfo/python-list
Re: UDP performance
Martin P. Hellwig wrote: [snip] > Is the connection 1:1 i.e. the receiving end receives data only from one > sender at the time? And how do you handle lost packages in your > application? > no the server is receiving from multiple clients. and at the moment I do not handle lost packets. -- http://mail.python.org/mailman/listinfo/python-list
Re: UDP performance
Serge Orlov wrote: > Paul Sijben wrote: >> Serge Orlov wrote: >>> Paul Sijben wrote: >>>> I am stumped by the following problem. I have a large multi-threaded >>>> server accepting communications on one UDP port (chosen for its supposed >>>> speed). >>>> >>>> I have been profiling the code and found that the UDP communication is >>>> my biggest drain on performance! Communication where the client and the >>>> server are on the same machine still takes 300ms or sometimes much more >>>> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64). >>> [snip] >>> >>>> buf, address = socket.recvfrom(8192) >>> I'm not an expert here, but I AFAIK UDP packet size should be kept >>> below 512 bytes otherwise it can cause fragmentation and hence all the >>> slow stuff like acknoledgements, timeouts, etc... >>> >> good point in general but in practice they are in this case. but this is >> less of a problem on the loopback interface anyway. > > Isn't it still controlled by MTU even on the loopback? What is your MTU > on the loopback? > loLink encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 eth0 Link encap:Ethernet HWaddr 00:14:85:35:A4:5D inet addr:192.168.0.124 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::214:85ff:fe35:a45d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 -- http://mail.python.org/mailman/listinfo/python-list
Re: UDP performance
Serge Orlov wrote: > Paul Sijben wrote: >> I am stumped by the following problem. I have a large multi-threaded >> server accepting communications on one UDP port (chosen for its supposed >> speed). >> >> I have been profiling the code and found that the UDP communication is >> my biggest drain on performance! Communication where the client and the >> server are on the same machine still takes 300ms or sometimes much more >> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64). > > [snip] > >> buf, address = socket.recvfrom(8192) > > I'm not an expert here, but I AFAIK UDP packet size should be kept > below 512 bytes otherwise it can cause fragmentation and hence all the > slow stuff like acknoledgements, timeouts, etc... > good point in general but in practice they are in this case. but this is less of a problem on the loopback interface anyway. -- http://mail.python.org/mailman/listinfo/python-list
UDP performance
I am stumped by the following problem. I have a large multi-threaded server accepting communications on one UDP port (chosen for its supposed speed). I have been profiling the code and found that the UDP communication is my biggest drain on performance! Communication where the client and the server are on the same machine still takes 300ms or sometimes much more per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64). I must be doing something wrong and would really appreciate feedback on my code below: I open the server port with self.s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.s.bind((myaddress, myport)) I then open a client port with self.s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM) self.s.connect((host, port)) the client sends data with self.s.sendall(data) and the server with self.s.sendto(data,link.remoteaddress) both receive with buf, address = socket.recvfrom(8192) The sender and receiver are in separate threads (threading.Thread). Does anyone know what is going wrong here, the socket communication, the thread scheduling? Paul Sijben -- http://mail.python.org/mailman/listinfo/python-list
