On Thu, Jul 21, 2005 at 10:26:21PM +1000, James Bunton <[EMAIL PROTECTED]> wrote: > Sorry, I still can't apply them, or open them with a text editor (with > proper font display). > > Could you make sure they're saved as UTF8?
They are. [EMAIL PROTECTED]:~$ file lang.py-avatars.diff lang.py-avatars.diff: UTF-8 Unicode C++ program text Try downloading them with wget, it worked for me. -- | Lucas Nussbaum | [EMAIL PROTECTED] http://www.lucas-nussbaum.net/ | | jabber: [EMAIL PROTECTED] GPG: 1024D/023B3F4F | From [EMAIL PROTECTED] Thu Jul 21 12:45:43 2005 From: [EMAIL PROTECTED] (Magnus Henoch) Date: Thu Jul 21 12:44:32 2005 Subject: [py-transports] More about PyICQt In-Reply-To: <[EMAIL PROTECTED]> (James Bunton's message of "Tue, 19 Jul 2005 09:42:46 +1000") References: <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> Message-ID: <[EMAIL PROTECTED]> James Bunton <[EMAIL PROTECTED]> writes: > Yeah, roster-subsync doesn't look like it'll become a JEP any time soon. > JEP-0144, the proposed replacement will work ok, even if it doesn't > support a few of the situations that will come up. > I'll maintain support for roster-subsync though because clients still > always need a way to get these contacts. A mass amount of subscription > requests they have to handle by hand is better than not seeing any > contacts at all :) A thought: it would be nice if roster-subsync supported an iq request for getting all contacts with associated info. Something like: <iq type='get' from='[EMAIL PROTECTED]/bar' to='msn.example.com' id='1'> <contacts xmlns='http://jabber.org/protocol/roster-subsync'/> </iq> And the service would reply: <iq type='get' from='[EMAIL PROTECTED]/bar' to='msn.example.com' id='1'> <contacts xmlns='http://jabber.org/protocol/roster-subsync'> <item jid='[EMAIL PROTECTED]' name="Some contact" subscription="both"> <group>Friends</group> <group>Colleagues</group> </item> <item jid='[EMAIL PROTECTED]' name='Other contact' subscription='both'/> [ etc ] </contacts> </iq> This way, a client which supports the protocol can fix up the roster even if a non-compliant client has added all the contacts without nicknames etc. > This is documentation of "Roster Item Exchange". No implementation for > this exists anywhere as far as I know. > http://www.jabber.org/jeps/jep-0144.html That depends on how you define "implementation" ;-) The club component is an implementation of the server side parts: http://www.cd.chalmers.se/projekten/jabber/clubs.html No clients have followed yet, though... Magnus From [EMAIL PROTECTED] Thu Jul 21 13:17:43 2005 From: [EMAIL PROTECTED] (Chris Carlin) Date: Thu Jul 21 13:16:32 2005 Subject: [py-transports] Re: pyaim-t rate management patch In-Reply-To: <[EMAIL PROTECTED]> References: <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> Message-ID: <[EMAIL PROTECTED]> I did a "svn diff" from the trunk/ directory. Attached is the output. I believe this to be working, though I never tested it heavily. I also separately tested a patch emailed to the list a while back that claimed to fix, among other things, the case where AOL disconnects and PyAIM-t doesn't notice. That patched version also seems to be working without a hitch. ~Chris On Mon, 18 Jul 2005, Daniel Chandran wrote: > Chris, > > Any idea when these changes might be available? > > Thanks, > daniel > > On 7/10/05, Chris Carlin <[EMAIL PROTECTED]> wrote: >> Because a couple of people on this list have a real interest in changes to >> the >> rate management stuff in pyaim-t (as per discussions offlist), I just wanted >> to >> give a heads up that I'm now testing a rewrite of the rate management code. >> >> Before, pyaim-t would spawn a new thread for every message, and that thread >> would sleep until it was cleared to send. It turned out that this was a Bad >> Thing on heavily loaded systems. A couple of sleeping threads per session >> really add up. >> >> The new code, besides being all cute and object oriented, creates queues for >> each class of packet, and spawns one thread for each queue. Additionally, the >> threads for empty queues die off. This seems to leave between zero and two >> queueing threads per running session. >> >> Of course because of the OO approach it should be pretty easy to snap in a >> single-threaded implementation, but that Queue.Queue class just called to me. >> Not that it did me much good in the end, but... >> >> Anyway, it seems to be working, so unless I run across any problems I'll be >> submitting a patch in the next couple of days. >> >> Thanks and gig'em, >> Chris >> >> _______________________________________________ >> py-transports mailing list >> [email protected] >> http://www.modevia.com/cgi-bin/mailman/listinfo/py-transports >> > -------------- next part -------------- Index: src/tlib/scheduler.py =================================================================== --- src/tlib/scheduler.py (revision 0) +++ src/tlib/scheduler.py (revision 21) @@ -0,0 +1,151 @@ +#This scheduler works by creating a separate thread for each class of packets, and letting these threads +#manage the execution of their respective messages. If a thread finds that it has no more messages in its +#queue even after waiting a short while the thread will terminate, saving its state. This is also the exit +#condition for the Scheduler; there is no way to shut down all threads when a user logs out. If another +#message shows up destined for a terminated thread, the thread is recreated from the data it saved when +#it exited. +# +#I believe this will lead to two or three threads per online user after the startup period is over. + +import threading +import time +import Queue +import sys + +class Scheduler: + + def __init__(self,handler): + self.freezer={} + self.handler=handler + self.bigLock=threading.Lock() + # Create default class, just in case + self.threads={'default' : self.QueueThread('default',handler,self.freezer)} + self.snacs={'default':'default'} + + def enqueue(self,fam,sub,snac): + snacid=str(fam)+str(sub) + if (not self.snacs.has_key(snacid)): + # We don't have a class, so assume class "default" + snacid='default' + classid=self.snacs[snacid] + self.bigLock.acquire() + if (not self.threads.has_key(classid) or not self.threads[classid].isAlive()): + self.threads[classid]=self.QueueThread(classid,self.handler,self.freezer) + self.threads[classid].enqueue(snac) + self.bigLock.release() + + def bindIntoClass(self,fam,sub,classid): + """ + Messages come in marked with fam and sub; we need to bind them into classes. + AOL tells us what fam,sub combination goes to which class. + """ + snacid=str(fam)+str(sub) + classid=str(classid) + self.snacs[snacid]=classid + + def setStat(self,classid,window=-1,clear=-1,alert=-1,limit=-1,disconnect=-1,rate=-1,lasttime=-1,maxrate=-1): + """ + AOL also tells us what our limits are and what our current rate is. + """ + classid=str(classid) + target=clear + self.bigLock.acquire() + if (not self.threads.has_key(classid) or not self.threads[classid].isAlive()): + self.threads[classid]=self.QueueThread(classid,self.handler,self.freezer) + self.threads[classid].setStat(window=window,rate=rate,target=target,lasttime=lasttime,max=maxrate) + self.bigLock.release() + + class QueueThread(threading.Thread): + + def __init__(self,name,handler,freezer): + threading.Thread.__init__(self) + self.name=name + self.handler=handler + self.freezer=freezer + if (freezer.has_key(name)): + self.rm=freezer[name] + else: + self.rm=Scheduler.RateManager() + self.freezer[name]=self.rm + self.q=Queue.Queue() + self.setDaemon(True) + self.start() + + def run(self): + while True: + try: + snac=self.q.get(True,self.rm.waithint) + delay=self.rm.getDelay() + time.sleep(delay) + self.__process(snac) + except Queue.Empty: + break + + def setStat(self,window=-1,rate=-1,target=-1,lasttime=-1,max=-1): + self.rm.setStat(window=window,rate=rate,target=target,lasttime=lasttime,max=max) + + def enqueue(self,snac): + self.q.put(snac) + + def __process(self,snac): + self.handler(snac) + self.rm.updateRate() + + class RateManager: + #This class calculates the current rate and delay needed not to overrun a target rate. + #Remember, it's not "rate" so much as "average delay". It goes down as traffic increases! + # + #This class should be general enough to use with any scheduler. + + def __init__(self): + self.lock=threading.RLock() + self.waithint=60 + self.rate=-1 + self.target=-1 + self.window=-1 + self.lasttime=-1 + self.max=-1 + + def setStat(self,window=-1,rate=-1,target=-1,lasttime=-1,max=-1): + self.lock.acquire() + if (window != -1): + self.ratehint=window + self.window=window + if (rate != -1): + self.rate=rate + if (target != -1): + self.target=target + if (lasttime != -1): + self.lasttime=lasttime + if (max != -1): + self.max=max + self.lock.release() + + def getDelay(self): + """ + Get the delay needed not to overrun target rate. + """ + self.lock.acquire() + nexttime=(self.window*self.target-(self.window-1)*self.rate)/1000.+self.lasttime + now=time.time() + self.lock.release() + if (nexttime < now or self.rate == -1): + return 0 + else: + return (nexttime-now) + + def updateRate(self): + """ + Record that a message has been sent and update data. + """ + self.lock.acquire() + if (self.window == -1): + return + now=time.time() + newrate=(self.window-1.)/self.window * self.rate + 1./self.window * (now-self.lasttime)*1000 + if (newrate > self.max): + self.rate=self.max + else: + self.rate=newrate + self.lasttime=now + self.lock.release() Index: src/tlib/oscar.py =================================================================== --- src/tlib/oscar.py (revision 11) +++ src/tlib/oscar.py (revision 21) @@ -14,6 +14,8 @@ from twisted.internet import reactor, main, defer, protocol from twisted.python import log +from scheduler import Scheduler + import struct import md5 import string @@ -26,6 +28,7 @@ import threading import socks5, sockserror + def logPacketData(data): lines = len(data)/16 if lines*16 != len(data): lines=lines+1 @@ -341,8 +344,8 @@ def f(reason): pass self.connectionLost = f self.transport.loseConnection() + - class SNACBased(OscarConnection): snacFamilies = { # family : (version, toolID, toolVersion) @@ -352,9 +355,7 @@ self.lastID=0 self.supportedFamilies = () self.requestCallbacks={} # request id:Deferred - self.outRateTable={} - self.outRateInfo={} - self.outRateLock=threading.Lock() + self.scheduler=Scheduler(self.sendFLAP) def sendSNAC(self,fam,sub,data,flags=[0,0]): """ @@ -368,15 +369,8 @@ #d.addErrback(self._ebDeferredError,fam,sub,data) # XXX for testing self.requestCallbacks[reqid] = d - #self.outRateLock.acquire() - #delay=self.calcDelay(fam,sub) - #time.sleep(delay) - #self.sendFLAP(SNAC(fam,sub,reqid,data)) - #self.updateRate(fam,sub) - #self.outRateLock.release() - delay=self.reserveDelay(fam,sub) snac=SNAC(fam,sub,reqid,data) - threading.Timer(delay,self.sendFLAP,[snac]).start() + self.scheduler.enqueue(fam,sub,snac) return d def _ebDeferredError(self, error, fam, sub, data): @@ -388,58 +382,9 @@ """ send a snac, but don't bother adding a deferred, we don't care. """ - #self.outRateLock.acquire() - #delay=self.calcDelay(fam,sub) - #time.sleep(delay) - #self.sendFLAP(SNAC(fam,sub,0x10000*fam+sub,data)) - #self.updateRate(fam,sub) - #self.outRateLock.release() - delay=self.reserveDelay(fam,sub) snac=SNAC(fam,sub,0x10000*fam+sub,data) - threading.Timer(delay,self.sendFLAP,[snac]).start() + self.scheduler.enqueue(fam,sub,snac) - def reserveDelay(self,fam,sub): - self.outRateLock.acquire() - delay=self.calcDelay(fam,sub) - self.updateRate(fam,sub,delay) - self.outRateLock.release() - return delay - - def calcDelay(self,fam,sub): - if (not self.outRateTable.has_key(str(fam)+str(sub))): - #print "no rateclass for "+ str(fam)+","+str(sub) - return 0 - rateclass=self.outRateTable[str(fam)+str(sub)] - target=self.outRateInfo[rateclass]['clear'] - window=self.outRateInfo[rateclass]['window'] - lasttime=self.outRateInfo[rateclass]['lasttime'] - currentrate=self.outRateInfo[rateclass]['currentrate'] - nextTime=(window*target-(window-1)*currentrate)/1000.+lasttime - now=time.time() - if (nextTime < now): - return 0 - else: - #print "delay "+ str(nextTime-now) - return (nextTime-now) - - def updateRate(self,fam,sub,delay=0): - if (not self.outRateTable.has_key(str(fam)+str(sub))): - return - rateclass=self.outRateTable[str(fam)+str(sub)] - window=self.outRateInfo[rateclass]['window'] - lasttime=self.outRateInfo[rateclass]['lasttime'] - currentrate=self.outRateInfo[rateclass]['currentrate'] - maxrate=self.outRateInfo[rateclass]['maxrate'] - now=time.time()+delay - - newrate=(window-1.)/window * currentrate + 1./window * (now-lasttime)*1000 - if (newrate > maxrate): - newrate=maxrate - #print "rateclass "+str(rateclass)+" newrate "+str(newrate) - - self.outRateInfo[rateclass]['lasttime']=now - self.outRateInfo[rateclass]['currentrate']=newrate - def oscar_(self,data): self.sendFLAP("\000\000\000\001"+TLV(6,self.cookie), 0x01) return "Data" @@ -489,17 +434,10 @@ limit=info[5] disconnect=info[6] current=info[7] - maximum=info[8] + maxrate=info[8] - self.outRateLock.acquire() - self.outRateInfo[rateclass]['window']=window - self.outRateInfo[rateclass]['clear']=clear - self.outRateInfo[rateclass]['currentrate']=current - self.outRateInfo[rateclass]['maxrate']=maximum - self.outRateLock.release() - - #print "CHRIS: " - #print info + self.scheduler.setStat(rateclass,window=window,clear=clear,alert=alert,limit=limit,disconnect=disconnect,rate=current,maxrate=maxrate) + if (code==3): import sys sys.exit() @@ -521,6 +459,7 @@ d = d + struct.pack('!4H',fam,version,toolID,toolVersion) self.sendSNACnr(0x01,0x02,d) + class BOSConnection(SNACBased): snacFamilies = { 0x01:(3, 0x0110, 0x059b), @@ -593,25 +532,26 @@ self.outRateInfo={} self.outRateTable={} count=struct.unpack('!H',snac[3][0:2])[0] - #print 'count: '+str(count) snac[3]=snac[3][2:] for i in range(count): info=struct.unpack('!HLLLLLLL',snac[3][:30]) - #print "class ID: "+str(info[0]) - self.outRateInfo[info[0]]={'window':info[1],'clear':info[2],'currentrate':info[6],'lasttime':time.time(),'maxrate':info[7]} + classid=info[0] + window=info[1] + clear=info[2] + currentrate=info[6] + lasttime=time.time() + maxrate=info[7] + self.scheduler.setStat(classid,window=window,clear=clear,rate=currentrate,lasttime=lasttime,maxrate=maxrate) snac[3]=snac[3][35:] - #print len(snac[3]) - #print self.outRateInfo + while (len(snac[3]) > 0): info=struct.unpack('!HH',snac[3][:4]) - #print "class ID: "+str(info[0])+" count: "+str(info[1]) - #print "reqiring "+str(info[1]*4)+" bytes out of "+str(len(snac[3][4:])) - classID=info[0] + classid=info[0] count=info[1] info=struct.unpack('!'+str(2*count)+'H',snac[3][4:4+count*4]) while (len(info)>0): - snacid=str(info[0])+str(info[1]) - self.outRateTable[snacid]=classID + fam,sub=str(info[0]),str(info[1]) + self.scheduler.bindIntoClass(fam,sub,classid) info=info[2:] snac[3]=snac[3][4+count*4:] From [EMAIL PROTECTED] Thu Jul 21 13:38:39 2005 From: [EMAIL PROTECTED] (Daniel Henninger) Date: Thu Jul 21 13:38:35 2005 Subject: [py-transports] Re: pyaim-t rate management patch In-Reply-To: <[EMAIL PROTECTED]> References: <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> Message-ID: <[EMAIL PROTECTED]> Excellent! Thank you Chris! I plan on putting together a lot of patches today and merging them into my tree. I also hope to have things stabilized enough to commit them to SVN. Daniel -- "The most addictive drug in the world is music." - The Lost Boyz > I did a "svn diff" from the trunk/ directory. Attached is the output. > > I believe this to be working, though I never tested it heavily. I also > separately tested a patch emailed to the list a while back that claimed to > fix, among other things, the case where AOL disconnects and PyAIM-t > doesn't notice. That patched version also seems to be working without a > hitch. > > ~Chris > > On Mon, 18 Jul 2005, Daniel Chandran wrote: > >> Chris, >> >> Any idea when these changes might be available? >> >> Thanks, >> daniel >> >> On 7/10/05, Chris Carlin <[EMAIL PROTECTED]> wrote: >>> Because a couple of people on this list have a real interest in changes >>> to the >>> rate management stuff in pyaim-t (as per discussions offlist), I just >>> wanted to >>> give a heads up that I'm now testing a rewrite of the rate management >>> code. >>> >>> Before, pyaim-t would spawn a new thread for every message, and that >>> thread >>> would sleep until it was cleared to send. It turned out that this was a >>> Bad >>> Thing on heavily loaded systems. A couple of sleeping threads per >>> session >>> really add up. >>> >>> The new code, besides being all cute and object oriented, creates >>> queues for >>> each class of packet, and spawns one thread for each queue. >>> Additionally, the >>> threads for empty queues die off. This seems to leave between zero and >>> two >>> queueing threads per running session. >>> >>> Of course because of the OO approach it should be pretty easy to snap >>> in a >>> single-threaded implementation, but that Queue.Queue class just called >>> to me. >>> Not that it did me much good in the end, but... >>> >>> Anyway, it seems to be working, so unless I run across any problems >>> I'll be >>> submitting a patch in the next couple of days. >>> >>> Thanks and gig'em, >>> Chris >>> >>> _______________________________________________ >>> py-transports mailing list >>> [email protected] >>> http://www.modevia.com/cgi-bin/mailman/listinfo/py-transports >>> >>_______________________________________________ > py-transports mailing list > [email protected] > http://www.modevia.com/cgi-bin/mailman/listinfo/py-transports >
