Re: Python threads and memory usage
On May 30, 9:42 am, Mike <[EMAIL PROTECTED]> wrote: > On May 30, 9:16 am, Dennis Lee Bieber <[EMAIL PROTECTED]> wrote: > > > > > On Thu, 29 May 2008 12:01:30 -0700 (PDT), Mike <[EMAIL PROTECTED]> > > declaimed the following in comp.lang.python: > > > > I observed, that every thread reserved some memory, and after exit > > > thread doesn't freed it. When i leaved my server working for 3 days, > > > then it takes 15% of 512MB memory (during that time about 15000 > > > threads were created and stopped). When server starts it only takes > > > about 1% of memory. > > > Do you have any outstanding references to the threads? If so, have > > you performed a .join() with the thread? Until you join it, the thread > > state (thread local objects/"variables") are probably being held for > > access from outside the thread. > > -- > > Wulfraed Dennis Lee Bieber KD6MOG > > [EMAIL PROTECTED] [EMAIL PROTECTED] > > HTTP://wlfraed.home.netcom.com/ > > (Bestiaria Support Staff: [EMAIL PROTECTED]) > > HTTP://www.bestiaria.com/ > > I'm joining threads only during my program exit. I'll try to do what > You suggest. > > THX It helped. Now all threads are added to thread list and every some period of time I'm checking which threads are alive (enumerate), and joining all which aren't. Now [memory usage is still on 1% :D:D:D Thanks -- http://mail.python.org/mailman/listinfo/python-list
Re: Python threads and memory usage
On May 30, 9:16 am, Dennis Lee Bieber <[EMAIL PROTECTED]> wrote: > On Thu, 29 May 2008 12:01:30 -0700 (PDT), Mike <[EMAIL PROTECTED]> > declaimed the following in comp.lang.python: > > > I observed, that every thread reserved some memory, and after exit > > thread doesn't freed it. When i leaved my server working for 3 days, > > then it takes 15% of 512MB memory (during that time about 15000 > > threads were created and stopped). When server starts it only takes > > about 1% of memory. > > Do you have any outstanding references to the threads? If so, have > you performed a .join() with the thread? Until you join it, the thread > state (thread local objects/"variables") are probably being held for > access from outside the thread. > -- > Wulfraed Dennis Lee Bieber KD6MOG > [EMAIL PROTECTED] [EMAIL PROTECTED] > HTTP://wlfraed.home.netcom.com/ > (Bestiaria Support Staff: [EMAIL PROTECTED]) > HTTP://www.bestiaria.com/ I'm joining threads only during my program exit. I'll try to do what You suggest. THX -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Threads - stopped vs. gone vs. Alive
On Wed, 28 May 2008 11:38:53 -0700, RossGK wrote: > > I've answered my own question about the "None" state - an event was > setting the thread to None where I didn't expect it. > > However, my question slightly repositioned is if a Thread is "Stopped" > it still seems to exist. Is there someway to make it go away, send it > to garbage collection etc? > You have to call the join() method of the thread object from another thread. This will terminate the thread and free its resources. This is usually the task of the parent thread which usually does something like: t = MyTread(...) t.start() # do your own stuff, then before quitting t.join() # Note that this waits until t is terminated > Other part of the original Q - I assume a Stopped thread gets a false > from "isAlive" and a Running thread gets a true? Yes. You can try yourself: >>> import threading >>> class MyThread(threading.Thread): def __init__(self): threading.Thread.__init__(self) def stop(self): self._stop = True def run(self): self._stop= False while self._stop == False: import time time.sleep(0.1) >>> t = MyThread() >>> t.start() >>> t.isAlive() True >>> t.isAlive() False >>> t.join() >>> t.isAlive() False Ciao FB -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Threads - stopped vs. gone vs. Alive
I've answered my own question about the "None" state - an event was setting the thread to None where I didn't expect it. However, my question slightly repositioned is if a Thread is "Stopped" it still seems to exist. Is there someway to make it go away, send it to garbage collection etc? Other part of the original Q - I assume a Stopped thread gets a false from "isAlive" and a Running thread gets a true? -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Threads - stopped vs. gone vs. Alive
On May 28, 12:01 pm, RossGK <[EMAIL PROTECTED]> wrote: > I'm a newbie to python threads, and playing with some simple client > server stuff and lots of print statements. > > My server thread launched with > self.worker = WorkerThread(self) > completes an interaction and then if I check on it's status with > print "Status:", self.workerI getStatus none > > A client thread behaves differently. Launched as > self.clientWorker( = ClientThreadself) > when it finishes it's work, I instead get: > Status: > > If I check the isAlive status on each of those, self.worker.isAlive > throws an exception, 'cause there is nothing there anymore to check > isAlive on. But self.clientWorker comes back as False, since it is a > stopped thread and hasn't gone away (which I'd like it to after it > finishes its work). > > So my question is when a thread finishes its work how do I more > predictably control whether it is just stopped, or goes away all > together? I don't want to do a double nested 'if' statement to check > if it exists before I check if it's alive. Pls ignore the obvious typos, this isn't a syntax question and google groups seems to be messing with my typing and spacing(!) e.g. > self.clientWorker( = ClientThreadself) should have read self.clientWorker = ClientThread(self) As I swear I had typed it... -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Threads -
> Can you please suggest a technique in Python where we can spawn few number > of worker threads and later map them to a function/s to execute individual > Jobs. Have a look at the threading module: http://docs.python.org/lib/module-threading.html HTH, Daniel -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Threads -
S.Mohideen <[EMAIL PROTECTED]> wrote: > Hi All, > > Can you please suggest a technique in Python where we can spawn few number > of worker threads and later map them to a function/s to execute individual > Jobs. > > Any references would be helpful.. I believe I give some examples in the Nutshell, but the basic idea is very simple: start the N worker threads you want in your pool -- they all use .get on the same Queue.Queue instance qI which contains "work requests" (WRs). A WR could e.g. be a tuple with a Queue qO (or None to indicate that the result is not interesting), a callable, and 0+ arguments (or an empty tuple to request thread termination); the worker thread just calls the callable on the arguments and puts the result to qO (if qO is not None). Many other similar arrangements are possible, depending on your application's exact needs, but I hope the general idea is clear. Alex -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Threads -
In article <[EMAIL PROTECTED]>, S.Mohideen <[EMAIL PROTECTED]> wrote: > >Can you please suggest a technique in Python where we can spawn few number >of worker threads and later map them to a function/s to execute individual >Jobs. You can see an example for a web spider on my Python website. -- Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ Help a hearing-impaired person: http://rule6.info/hearing.html -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Threads
I think that I found a solution to my thread issues, however I know it is
not the most efficient method possible.
Just to give you a little information on what this project is all about
I have 3 lists of email addresses.
(1) "host email address" = contains a list of all of my emails
address (around 150,000 users)
(2) "email addresses" = contains a list of email addresses that I
have to match with the file "host email address". If there are any matches,
then I have to print them out to a file. (this could be up to 8 million
users)
(3) "domain addresses" = contains a list of email domains that I
have to match with the "host email address" file. If there are any matched,
then I have to print them out to a file. (could be 2000 or more domains)
When running the application, you will have the "host email address" and
can have either one or both of the other files running at the same time.
My problem was that when the application ran, it appeared to stall. I
decided to use threads for (1) the processing of data and (2) the progress
bar. The solution I found that enabled the two threads to communicate was
the use of global variables.
I know this is this is not the most efficient method but, using this
solution, I do not see the stalling issue that I found before (which is a
good thing). I am still not happy with it, because I know it is not
efficient, but I found this to be the best solution for my needs.
Thoughts?
The code is below. Before you see the code, I must thank everyone who
helped me with this project (including the open source coders).
===
#! /usr/bin/env python
import difflib, sys, thread, re, os, time import Tkinter from Tkinter import
* from sets import Set import tkFileDialog, tkMessageBox from tkFileDialog
import *
listName = ['','','']
threadStat = 0
mailsrch = re.compile(r'[EMAIL PROTECTED],4}')
domsrch = re.compile(r"@(\S+)")
statusVar = 0.0 # for the progress bar
startProgress = 0
class Meter(Tkinter.Frame):
def __init__(self, master, width=300, height=20, bg='black',
fillcolor='cyan',\
value=0.0, text=None, font=None, textcolor='white', *args,
**kw):
Tkinter.Frame.__init__(self, master, bg=bg, width=width,
height=height, *args, **kw)
self._value = value
self._canv = Tkinter.Canvas(self, bg=self['bg'],
width=self['width'], height=self['height'],\
highlightthickness=0, relief='flat',
bd=0)
self._canv.pack(fill='both', expand=1)
self._rect = self._canv.create_rectangle(0, 0, 0,
self._canv.winfo_reqheight(), fill=fillcolor,\
width=0)
self._text = self._canv.create_text(self._canv.winfo_reqwidth()/2,
self._canv.winfo_reqheight()/2,\
text='', fill=textcolor)
if font:
self._canv.itemconfigure(self._text, font=font)
self.set(value, text)
self.bind('', self._update_coords)
def _update_coords(self, event):
'''Updates the position of the text and rectangle inside the canvas
when the size of
the widget gets changed.'''
self._canv.update_idletasks()
self._canv.coords(self._text, self._canv.winfo_width()/2,
self._canv.winfo_height()/2)
self._canv.coords(self._rect, 0, 0,
self._canv.winfo_width()*self._value, self._canv.winfo_height())
self._canv.update_idletasks()
def get(self):
return self._value, self._canv.itemcget(self._text, 'text')
def set(self, value=0.0, text=None):
#make the value failsafe:
if value < 0.0:
value = 0.0
elif value > 1.0:
value = 1.0
self._value = value
if text == None:
#if no text is specified use the default percentage string:
text = str(int(round(100 * value))) + ' %'
self._canv.coords(self._rect, 0, 0, self._canv.winfo_width()*value,
self._canv.winfo_height())
self._canv.itemconfigure(self._text, text=text)
self._canv.update_idletasks()
##
def fail(msg):
out = sys.stderr.write
out(msg + "\n\n")
out(__doc__)
return 0
def fopen(fname):
try:
return open(fname, 'U')
except IOError, detail:
return fail("couldn't open " + fname + ": " + str(detail))
def fetchFiles(file1,file2,file3): #file1: host list file2 = email list;
file3=domain; method=method = ''
print file1
print file2
print file3
f1 = fopen(file1)
a = f1.readlines(); f1.close()
d1 = {}
for c in a:
for m in mailsrch.findall(c):
d1[m.lower()] = None
print "starting list 2"
thread.start_new_thread(showProcessing, ())
#DOMAIN COMPARISONif file2 == '':
domain(d1,file3)
#EM
Re: Python Threads
Great, thanks for the tip Gabriel! On 2/18/07, Gabriel Genellina <[EMAIL PROTECTED]> wrote: En Sun, 18 Feb 2007 23:37:02 -0300, Sick Monkey <[EMAIL PROTECTED]> escribió: > Well if this cannot be done, can a thread call a function in the main > method? > I have been trying and have not been successive. Perhaps I am using > thread > incorrectly. The safe way to pass information between threads is to use Queue. From inside the working thread, you put() an item with enough state information. On the main (GUI) thread, you use after() to check for any data in the queue, and then update the interfase accordingly. I think there is a recipe in the Python Cookbook http://aspn.activestate.com/ASPN/Cookbook/Python -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Threads
En Sun, 18 Feb 2007 23:37:02 -0300, Sick Monkey <[EMAIL PROTECTED]> escribió: > Well if this cannot be done, can a thread call a function in the main > method? > I have been trying and have not been successive. Perhaps I am using > thread > incorrectly. The safe way to pass information between threads is to use Queue. From inside the working thread, you put() an item with enough state information. On the main (GUI) thread, you use after() to check for any data in the queue, and then update the interfase accordingly. I think there is a recipe in the Python Cookbook http://aspn.activestate.com/ASPN/Cookbook/Python -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Threads
Well if this cannot be done, can a thread call a function in the main
method?
I have been trying and have not been successive. Perhaps I am using thread
incorrectly.
On 2/18/07, Sick Monkey <[EMAIL PROTECTED]> wrote:
Is there anyway to get 2 python threads to talk to one another?
I have a GUI which is spawning a thread to make a large calculation (that
way the GUI does not appear to be non responsive). I am trying to attach a
progress bar to the threaded action. As the thread is calculating data, I
would like it to communicate with the other (progress) thread to update it.
On windows, when I spawn the progress bar without using threads, the
progress bar stalls <> So I thought by
spawning the progress bar by using a thread, it would not stall... If
anyone can think of another technique I am all ears.
~~START SAMPLE
def progressBar(status):
mroot = Tkinter.Tk(className='Worker Bee')
metric = Meter(mroot, relief='ridge', bd=3)
metric.pack(fill='x')
metric.set(status, 'Starting ...')
def fetchFiles(file1,file2,file3):
method = ''
print file1
print file2
print file3
f1 = fopen(file1)
a = f1.readlines(); f1.close()
d1 = {}
for c in a:
for m in mailsrch.findall(c):
d1[m.lower()] = None
I Would like to Update the progress bar running in the other
thread here.
## set status = .33 and update progress bar.
if file2 == '':
domain(d1,file3)
#...
def startProc():
status = 0
thread.start_new_thread(fetchFiles, (f1name,f2name,f3name,))
thread.start_new_thread(progressBar, (status,))
~~END SAMPLE
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python Threads and C Semaphores
Dejan Rodiger wrote: > Jeremy said the following on 16.1.2007 8:27: > >> I have a fat C++ extension to a Python 2.3.4 program. In all, I count >> five threads. Of these, two are started in Python using >> thread.start_new_thread(), and both of these wait on semaphores in the C++ >> extension using sem_wait(). There also are two other Python threads and one >> thread running wholly in >> the extension. >> >> I notice that when one of the Python threads calls the extension and waits >> on a semaphore, all but the C++ thread halt even when not waiting on any >> semaphore. How do we get this working right? > > Check the Global Interpreter Lock More specifically, make sure that any extension code that does not call Python API functions releases the GIL for the duration. Look into the Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS macros and the PyGILState*() API functions (these API functions appeared in Python 2.3). -- - Andrew I MacIntyre "These thoughts are mine alone..." E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370 [EMAIL PROTECTED] (alt) |Belconnen ACT 2616 Web:http://www.andymac.org/ |Australia -- http://mail.python.org/mailman/listinfo/python-list
Re: Python Threads and C Semaphores
Jeremy said the following on 16.1.2007 8:27: > Hello, > > I have a fat C++ extension to a Python 2.3.4 program. In all, I count > five threads. Of these, two are started in Python using > thread.start_new_thread(), and both of these wait on semaphores in the C++ > extension using sem_wait(). There also are two other Python threads and one > thread running wholly in > the extension. > > I notice that when one of the Python threads calls the extension and waits > on a semaphore, all but the C++ thread halt even when not waiting on any > semaphore. How do we get this working right? Check the Global Interpreter Lock -- Dejan Rodiger - PGP ID 0xAC8722DC Delete wirus from e-mail address -- http://mail.python.org/mailman/listinfo/python-list
Re: Python threads and Numeric/SciPy exploit Dual Core ?
robert wrote: > Fredrik Lundh wrote: > >> "robert" wrote: >> >>> Simple Python code obviously cannot use the dual core by Python threads. >>> Yet, a program drawing CPU mainly for matrix computations - preferably >>> with Numeric/SciPy - will this profit from a dual core when using 2 (or >>> more) Python threads? >> >> as long as the binding releases the GIL, sure. > > Thus - does Numeric/SciPy release it? Depends. Some of the linear algebra functions in scipy do. Most operations do not. People are looking at adding more. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: Python threads and Numeric/SciPy exploit Dual Core ?
Fredrik Lundh wrote: > "robert" wrote: > > >>Simple Python code obviously cannot use the dual core by Python threads. >>Yet, a program drawing CPU mainly for matrix computations - preferably >>with Numeric/SciPy - will this profit from a dual core when using 2 (or >>more) Python threads? > > > as long as the binding releases the GIL, sure. Thus - does Numeric/SciPy release it? The same q about the orange (data mining) lib. Robert -- http://mail.python.org/mailman/listinfo/python-list
Re: Python threads and Numeric/SciPy exploit Dual Core ?
Oeyvind Brandtsegg wrote: > I've been trying to make my music app use dual core, > and would very much like some more detailed information on this. > > Excuse my lack of knowledge, > but how do I explicitly release the GIL ? http://docs.python.org/api/threads.html > I haven't learned this, but have found through experimentation that I > can release a thread by using time.sleep(0) inside a thread's "run > while true" loop. This seems to create an interrupt, and give other > threads a chance to do their thing. that (momentarily) blocks the current thread, and forces a rescheduling. -- http://mail.python.org/mailman/listinfo/python-list
Re: Python threads and Numeric/SciPy exploit Dual Core ?
I've been trying to make my music app use dual core, and would very much like some more detailed information on this. Excuse my lack of knowledge, but how do I explicitly release the GIL ? I haven't learned this, but have found through experimentation that I can release a thread by using time.sleep(0) inside a thread's "run while true" loop. This seems to create an interrupt, and give other threads a chance to do their thing. If this is terribly wrong (it works, but I dont' know how stable it is), please do point me in the direction of a proper way to implement it. As stated in an earlier post, my threads do not work on shared data, so I have not implemented any sort of lock or mutex. best Oeyvind On 10/2/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote: > "robert" wrote: > > > Simple Python code obviously cannot use the dual core by Python threads. > > Yet, a program drawing CPU mainly for matrix computations - preferably > > with Numeric/SciPy - will this profit from a dual core when using 2 (or > > more) Python threads? > > as long as the binding releases the GIL, sure. > > > > > > -- > http://mail.python.org/mailman/listinfo/python-list > -- http://mail.python.org/mailman/listinfo/python-list
Re: Python threads and Numeric/SciPy exploit Dual Core ?
"robert" wrote: > Simple Python code obviously cannot use the dual core by Python threads. > Yet, a program drawing CPU mainly for matrix computations - preferably > with Numeric/SciPy - will this profit from a dual core when using 2 (or > more) Python threads? as long as the binding releases the GIL, sure. -- http://mail.python.org/mailman/listinfo/python-list
