Re: non-blocking PIPE read on Windows
Hi !Sorry, but I want to share my experiences. I hope this help to you.I think that specialized MSWindows based services too complicated. They have to many bug possibilites.So I trying with normal, "in python accessable" pipes. I see that with flush(), and some of the bintotext tricks I can use the subprocess/masterprocess communication.
Ok, this is not asynchronous. I can send some job to sp(s), and I can receive the report from sp(s).But with threading I can create non-blocking communication.You can see in the example: the PipeBPPThr define a pipe based process-pool thread.
This can communicate with a subprocess, can send/receive jobs, etc.If you collect these threads, and write a process pool object, you can handle all of the communications with one object.I hope to these examples can help to you.
If not, you can try with wm_copydata messages in Windows.http://msdn.microsoft.com/library/default.asp?url=""
This way of data exchanging is based on the message handling/sending.dd
import os, sys, threading, Queue, time
pp_SPTHR_READY = 1
pp_SPTHR_SENDING= 2
pp_SPTHR_RECEIVING = 3
class CustomProcessThread(threading.Thread):
def __init__(self,UniqID,ThreadID,Params):
threading.Thread.__init__(self)
self.UniqID=UniqID
self.ThreadID=ThreadID
self.Params=Params
self.IPCObj=self._CreateIPCObj(UniqID,ThreadID)
self.ProcessObj=self._CreateProcess(UniqID,ThreadID,self.IPCObj)
self.State=pp_SPTHR_READY
self._Input=Queue.Queue()
self._Output=Queue.Queue()
def _CreateIPCObj(self,UniqID,ThreadID):
pass
def _CreateProcess(self,UniqID,ThreadID,IPCObj):
pass
def SendJob(self,JobID,Data):
self._Input.put([JobID,Data])
def HaveFinishedJob(self):
return not self._Output.empty()
def ReceiveJob(self):
return self._Output.get()
def Abort(self):
self._Input.put(None)
def _SendToSP(self,JobID,Data):
pass
def _ReceiveFromSP(self):
pass
def run(self):
while 1:
inp=self._Input.get()
if inp==None:
break
jobid,data=inp
self._SendToSP(jobid,data)
rdata=self._ReceiveFromSP()
self._Output.put([jobid,rdata])
def Wait(self):
while self.isAlive():
time.sleep(0.001)
import PipeBPPThr, sys
while 1:
jobid,data=PipeBPPThr.ReadBinPacket(sys.stdin)
if jobid==-1:
PipeBPPThr.WriteBinPacket(sys.stdout,None)
break
PipeBPPThr.WriteBinPacket(sys.stdout,[jobid,[1,data]])
import os, sys, threading, Queue, subprocess, CustPPThread
from cPickle import dumps, loads
from binascii import hexlify, unhexlify
def ReadTextPacket(SourceStream):
packet=SourceStream.read(6)
psize=int(packet)
packet=SourceStream.read(psize)
return packet
def WriteTextPacket(DestStream,Packet):
Packet=str(Packet)
DestStream.write('%06d'%len(Packet))
DestStream.write(Packet)
DestStream.flush()
def ReadBinPacket(SourceStream):
txtpacket=ReadTextPacket(SourceStream)
obj=loads(unhexlify(txtpacket))
return obj
def WriteBinPacket(DestStream,Obj):
pckpacket=hexlify(dumps(Obj,1))
WriteTextPacket(DestStream,pckpacket)
class PIPEBasedProcessThread(CustPPThread.CustomProcessThread):
def _CreateIPCObj(self,UniqID,ThreadID):
return None
def _CreateProcess(self,UniqID,ThreadID,IPCObj):
spfn = self.Params['scriptfilename']
cmd = [r'c:\python24\python.exe', spfn]
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
return p
def _SendToSP(self,JobID,Data):
WriteBinPacket(self.ProcessObj.stdin,[JobID,Data])
def _ReceiveFromSP(self):
return ReadBinPacket(self.ProcessObj.stdout)
if __name__=='__main__':
print "Start"
'''
spfn='ppbpt_sub.py'
cmd = [r'c:\python24\python.exe', spfn]
p = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
for i in range(10):
WriteBinPacket(p.stdin,[1,'AAA'])
print ReadBinPacket(p.stdout)
WriteBinPacket(p.stdin,[-1,None])
print ReadBinPacket(p.stdout)
sys.exit()
'''
import time
st=time.time()
thr=PIPEBasedProcessThread(1,1,{'scriptfilename':'ppbpt_sub.py'})
thr.start()
for i in range(100):
thr.SendJob(i,'AAABBB')
print thr.ReceiveJob()
thr.SendJob(-1,None)
print thr.ReceiveJob()
thr.Abort()
thr.Wait()
print "End"
st=time.time()-st
print st
--
http://mail.python.org/mailman/listinfo/python-list
Re: non-blocking PIPE read on Windows
placid wrote:
> What i need to do is, create a process using subprocess.Popen, where
> the subprocess outputs information on one line (but the info
> continuesly changes and its always on the same line) and read this
> information without blocking, so i can retrieve other data from the
> line i read in then put this in a GUI interface.
> readline() blocks until the newline character is read, but when i use
> read(X) where X is a number of bytes then it doesnt block(expected
> functionality) but i dont know how many bytes the line will be and its
> not constant so i cant use this too.
I wrote something for this the other day. The class has a getline()
method, which either returns a line or raises an exception instead of
blocking. It also raises an exception instead of returning EOF.
My use case had me reading from multiple processes at once; since
select() doesn't work on files in win32, I had to get a little cheesy.
I've appended the function that implements that use case, for
reference.
The central idea is to use PeekNamedPipe to figure out what's in the
pipe. You can then read that data without fear of blocking. I wrote it
quickly, therefore the code is a little embarassing, but... hopefully
useful all the same.
. class NoLineError(Exception): pass
. class NoMoreLineError(Exception): pass
. class liner(object):
. """Helper class for multi_readlines."""
. def __init__(self, f):
. self.fd = f.fileno()
. self.osf = msvcrt.get_osfhandle(self.fd)
. self.buf = ''
.
. def getline(self):
. """Returns a line of text, or raises NoLineError, or
NoMoreLineError."""
. try:
. data, avail, _ = win32pipe.PeekNamedPipe(self.osf, 0)
. except pywintypes.error:
. # Pipe closed: give up what we have, then that's it
. if self.buf:
. ret, self.buf = self.buf, None
. return ret
. else:
. raise NoMoreLineError
. if avail:
. self.buf += os.read(self.fd, avail)
.
. idx = self.buf.find('\n')
. if idx >= 0:
. ret, self.buf = self.buf[:idx+1], self.buf[idx+1:]
. return ret
. else:
. raise NoLineError
.
.
. def multi_readlines(fs):
. """Read lines from |fs|, a list of file objects.
. The lines come out in arbitrary order, depending on which files
. have output available first."""
. if type(fs) not in (list, tuple):
. raise Exception("argument must be a list.")
. objs = [liner(f) for f in fs]
. for i,obj in enumerate(objs): obj._index = i
. while objs:
. for i,obj in enumerate(objs):
. try:
. yield (obj._index, obj.getline())
. except NoLineError:
. pass
. except NoMoreLineError:
. del objs[i]
. break # Because we mutated the array
--
http://mail.python.org/mailman/listinfo/python-list
Re: non-blocking PIPE read on Windows
Hi !A new version with binary data handling. 103 seconds with 1000 data exchange.import os, sys, time, binascii, cPicklebpath,bname=os.path.split(sys.argv[0])def Log(Msg,IsMaster,First=False):
fn=sys.argv[0]+'.'+['c','m'][int(IsMaster)]+'.log' mode='aw'[int(First)] f=open(fn,mode) f.write('\n%s:\n'%time.time()) f.write('%s\n'%Msg) f.flush() f.close()def ReadTextPacket(SourceStream):
packet=SourceStream.read(6) psize=int(packet) packet=SourceStream.read(psize) return packetdef WriteTextPacket(DestStream,Packet): Packet=str(Packet) DestStream.write('%06d'%len(Packet))
DestStream.write(Packet) DestStream.flush() import base64def PackObj(Obj): pckpacket=cPickle.dumps(Obj) enstr=base64.encodestring(pckpacket) return enstrdef UnpackObj(Packet):
pckpacket=base64.decodestring(Packet) obj=cPickle.loads(pckpacket) return obj#s=PackObj([1,None,'A']*10)#print s#print UnpackObj(s)#sys.exit()def ReadBinPacket(SourceStream):
txtpacket=ReadTextPacket(SourceStream) obj=UnpackObj(txtpacket) return objdef WriteBinPacket(DestStream,Obj): txtpacket=PackObj(Obj) WriteTextPacket(DestStream,txtpacket)
if 'C' in sys.argv: Log('Client started',0,1) try: while 1: #Log('Waiting for packet',0,0) data=""> #Log('Packet received',0,0)
#Log('The packet is: %s'%([data]),0,0) #Log('Print the result',0,0) WriteBinPacket(sys.stdout,"Master wrote: %s"%([data])) if str(data).strip()=='quit':
Log('Quit packet received',0,0) break except Exception,E: Log(str(E),0,0) Log('Client finished',0,0)else: Log('Master started',1,1) try:
Log('Start subprocess',1,0) import time st=time.time() child_stdin,child_stdout=os.popen2(r'c:\python24\python.exe %s C'%(bname)) for i in range(1000): #Log('Send packet',1,0)
WriteBinPacket(child_stdin,['Alma'*100,i]) #Log('Waiting for packet',1,0) s=ReadBinPacket(child_stdout) #Log('Packet is: %s'%([s]),1,0) #Log('Print packet',1,0)
#print "Client's answer",[s] import time time.sleep(0.1) #Log('Send packet',1,0) WriteBinPacket(child_stdin,'quit') #Log('Waiting for packet',1,0)
s=ReadBinPacket(child_stdout) #Log('Packet is: %s'%([s]),1,0) #Log('Print packet',1,0) #print "Client's answer",[s] Log('Master finished',1,0) except Exception,E:
Log(str(E),1,0) print time.time()-stdd
--
http://mail.python.org/mailman/listinfo/python-list
Re: non-blocking PIPE read on Windows
Hi !If you don't want to use MS-specific things, you can use the normal pipes.See this code. If you want to use non-blocking version, you need to create a thread that handle the reads/writes.
import os, sys, time, binascii, cPicklebpath,bname=os.path.split(sys.argv[0])def Log(Msg,IsMaster,First=False): fn=sys.argv[0]+'.'+['c','m'][int(IsMaster)]+'.log' mode='aw'[int(First)] f=open(fn,mode)
f.write('\n%s:\n'%time.time()) f.write('%s\n'%Msg) f.flush() f.close()def ReadTextPacket(SourceStream): packet=SourceStream.read(6) psize=int(packet) packet=SourceStream.read
(psize) return packetdef WriteTextPacket(DestStream,Packet): Packet=str(Packet) DestStream.write('%06d'%len(Packet)) DestStream.write(Packet) DestStream.flush()'''def ReadBinPacket(SourceStream):
txtpacket=ReadTextPacket(SourceStream) pckpacket=binascii.unhexlify(txtpacket) obj=cPickle.loads(pckpacket) return objdef WriteBinPacket(DestStream,Obj): pckpacket=cPickle.dumps
(Obj) txtpacket=binascii.hexlify(pckpacket) WriteTextPacket(DestStream,txtpacket)'''if 'C' in sys.argv: #Log('Client started',0,1) while 1: #Log('Waiting for packet',0,0)
data=""> #Log('Packet received',0,0) #Log('The packet is: %s'%([data]),0,0) #Log('Print the result',0,0) WriteTextPacket(sys.stdout,"Master wrote: %s"%([data]))
if data.strip()=='quit': #Log('Quit packet received',0,0) break #Log('Client finished',0,0)else: #Log('Master started',1,1) #Log('Start subprocess',1,0)
import time st=time.time() child_stdin,child_stdout=os.popen2(r'c:\python24\python.exe %s C'%(bname)) for i in range(1000): #Log('Send packet',1,0) WriteTextPacket(child_stdin,['Alma'*100,i])
#Log('Waiting for packet',1,0) s=ReadTextPacket(child_stdout) #Log('Packet is: %s'%([s]),1,0) #Log('Print packet',1,0) #print "Client's answer",[s] import time
time.sleep(0.1) #Log('Send packet',1,0) WriteTextPacket(child_stdin,'quit') #Log('Waiting for packet',1,0) s=ReadTextPacket(child_stdout) #Log('Packet is: %s'%([s]),1,0) #Log('Print packet',1,0)
#print "Client's answer",[s] #Log('Master finished',1,0) print time.time()-stdd2006/7/28, Dennis Lee Bieber <
[EMAIL PROTECTED]>:On 27 Jul 2006 22:26:25 -0700, "placid" <
[EMAIL PROTECTED]> declaimed thefollowing in comp.lang.python:>> readline() blocks until the newline character is read, but when i use> read(X) where X is a number of bytes then it doesnt block(expected
> functionality) but i dont know how many bytes the line will be and its> not constant so i cant use this too.>> Any ideas of solving this problem?>Use a thread that reads one character at a time; when it sees
whatever signals "end of line" (it sounds like you're reading a progressbar implemented via overwrite). Combine the characters into astring, return the string to the main program via a queue.
If there is no such "end of line" character, but there IS anoticeable delay between "writes", a more complex method might suffice-- in which one thread does the byte reads, setting a time value on each
read; a related thread then does a sleep() loop, checking the "last readtime" against the pause length -- if close enough to the pause duration,combine and return...Alternatively, take a good old style terminal keyboard (a VT100
Tempest-rated model should be ideal), and use it to beat Bill Gates overthe head until he agrees to push a high-priority upgrade to the commandline I/O system... or makes files work with select() (so you can combine
the time-out with the byte read)--WulfraedDennis Lee Bieber KD6MOG[EMAIL PROTECTED]
[EMAIL PROTECTED]HTTP://wlfraed.home.netcom.com/(Bestiaria Support Staff: [EMAIL PROTECTED]
)HTTP://www.bestiaria.com/--http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
Re: non-blocking PIPE read on Windows
Dennis Lee Bieber wrote: > On 30 Jul 2006 16:22:34 -0700, "placid" <[EMAIL PROTECTED]> declaimed the > following in comp.lang.python: > > > > > ;) Tsk Tsk > > Have you ever seen a Tempest VT-100? Lead shielding on the monitor > FACE... Turn the brightness all the way up and it still looked dim. > Fiber optic to the computer. And a shield keyboard about three inches > thick and weighing 5 lbs. > > All to keep "them" from using radio equipment to pick up key strokes > or the display scan line radiation (use an external vertical/horizontal > sweep generator to an oscilloscope, and a high-gain amplifier to detect > the brightness variation of the CRT). Nope, i have not seen one. -- http://mail.python.org/mailman/listinfo/python-list
Re: non-blocking PIPE read on Windows
Dennis Lee Bieber wrote: > On 27 Jul 2006 22:26:25 -0700, "placid" <[EMAIL PROTECTED]> declaimed the > following in comp.lang.python: > > > > > readline() blocks until the newline character is read, but when i use > > read(X) where X is a number of bytes then it doesnt block(expected > > functionality) but i dont know how many bytes the line will be and its > > not constant so i cant use this too. > > > > Any ideas of solving this problem? > > > Use a thread that reads one character at a time; when it sees > whatever signals "end of line" (it sounds like you're reading a progress > bar implemented via overwrite). Combine the characters into a > string, return the string to the main program via a queue. > Yes it is a progress bar implemented via overwrite. I will try this. > If there is no such "end of line" character, but there IS a > noticeable delay between "writes", a more complex method might suffice > -- in which one thread does the byte reads, setting a time value on each > read; a related thread then does a sleep() loop, checking the "last read > time" against the pause length -- if close enough to the pause duration, > combine and return... i dont think there is a noticeable delay between "writes". > Alternatively, take a good old style terminal keyboard (a VT100 > Tempest-rated model should be ideal), and use it to beat Bill Gates over > the head until he agrees to push a high-priority upgrade to the command > line I/O system... or makes files work with select() (so you can combine > the time-out with the byte read) ;) Tsk Tsk -- http://mail.python.org/mailman/listinfo/python-list
Re: non-blocking PIPE read on Windows
"placid" <[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > Hi all, > > I have been looking into non-blocking read (readline) operations on > PIPES on windows XP and there seems to be no way of doing this. Ive > read that you could use a Thread to read from the pipe, but if you > still use readline() wouldnt the Thread block too? http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554 > What i need to do is, create a process using subprocess.Popen, where > the subprocess outputs information on one line (but the info > continuesly changes and its always on the same line) and read this > information without blocking, so i can retrieve other data from the > line i read in then put this in a GUI interface. > > > readline() blocks until the newline character is read, but when i use > read(X) where X is a number of bytes then it doesnt block(expected > functionality) but i dont know how many bytes the line will be and its > not constant so i cant use this too. > > Any ideas of solving this problem? > > > Cheers I realized something very similar to what you described in http://sourceforge.net/projects/bestgui - the subprocess2.py module realizes the non blocking I/O - the outputparser.py module processes the output from the controlled process and updates the progress-bar, the status-bar and the log messages in the GUI. Incomplete lines are stored in a buffer and processed at the next read. ciao -- Antonio Valentino -- Posted via Mailgate.ORG Server - http://www.Mailgate.ORG -- http://mail.python.org/mailman/listinfo/python-list
Re: non-blocking PIPE read on Windows
placid wrote: > Hi all, > > I have been looking into non-blocking read (readline) operations on > PIPES on windows XP and there seems to be no way of doing this. Ive > read that you could use a Thread to read from the pipe, but if you > still use readline() wouldnt the Thread block too? Yes it will, but that's ok. In this case that's what it's for. While the thread waits for the readline(), the rest of your program continues to carry on. > > What i need to do is, create a process using subprocess.Popen, where > the subprocess outputs information on one line (but the info > continuesly changes and its always on the same line) and read this > information without blocking, so i can retrieve other data from the > line i read in then put this in a GUI interface. > > > readline() blocks until the newline character is read, but when i use > read(X) where X is a number of bytes then it doesnt block(expected > functionality) but i dont know how many bytes the line will be and its > not constant so i cant use this too. > > Any ideas of solving this problem? > > > Cheers -- http://mail.python.org/mailman/listinfo/python-list
non-blocking PIPE read on Windows
Hi all, I have been looking into non-blocking read (readline) operations on PIPES on windows XP and there seems to be no way of doing this. Ive read that you could use a Thread to read from the pipe, but if you still use readline() wouldnt the Thread block too? What i need to do is, create a process using subprocess.Popen, where the subprocess outputs information on one line (but the info continuesly changes and its always on the same line) and read this information without blocking, so i can retrieve other data from the line i read in then put this in a GUI interface. readline() blocks until the newline character is read, but when i use read(X) where X is a number of bytes then it doesnt block(expected functionality) but i dont know how many bytes the line will be and its not constant so i cant use this too. Any ideas of solving this problem? Cheers -- http://mail.python.org/mailman/listinfo/python-list
