Hi Guido, Cool tool! Is this something I would be able to run just in the evenings/weekends etc? Will it pick up that fact that I don't have boehm automatically?
Cheers, Ben [EMAIL PROTECTED] wrote on 26/07/2006 12:18:51: > Author: guido > Date: Wed Jul 26 13:18:25 2006 > New Revision: 30565 > > Added: > pypy/dist/pypy/tool/build/ (props changed) > pypy/dist/pypy/tool/build/README.txt (contents, props changed) > pypy/dist/pypy/tool/build/__init__.py (contents, props changed) > pypy/dist/pypy/tool/build/bin/ (props changed) > pypy/dist/pypy/tool/build/bin/client (contents, props changed) > pypy/dist/pypy/tool/build/bin/path.py (contents, props changed) > pypy/dist/pypy/tool/build/bin/server (contents, props changed) > pypy/dist/pypy/tool/build/bin/startcompile (contents, props changed) > pypy/dist/pypy/tool/build/builds/ > pypy/dist/pypy/tool/build/client.py (contents, props changed) > pypy/dist/pypy/tool/build/config.py (contents, props changed) > pypy/dist/pypy/tool/build/conftest.py (contents, props changed) > pypy/dist/pypy/tool/build/execnetconference.py (contents, props changed) > pypy/dist/pypy/tool/build/server.py (contents, props changed) > pypy/dist/pypy/tool/build/test/ (props changed) > pypy/dist/pypy/tool/build/test/fake.py (contents, props changed) > pypy/dist/pypy/tool/build/test/path.py (contents, props changed) > pypy/dist/pypy/tool/build/test/test.zip (contents, props changed) > pypy/dist/pypy/tool/build/test/test_client.py (contents, props changed) > pypy/dist/pypy/tool/build/test/test_pypybuilder.py (contents, > props changed) > pypy/dist/pypy/tool/build/test/test_request_storage.py > (contents, props changed) > pypy/dist/pypy/tool/build/test/test_server.py (contents, props changed) > Log: > Added 'pypybuilder', a tool to build a 'build farm' from > participating clients, > the clients register to a server with information about what they can build, > then the server waits for build requests and dispatches to clients. Clients > send back a build when they're done, on which the server sends out emails to > whoever is waiting for the build. When a build is already available, the > requestor is provided with a path (will be URL in the future) to the build, > if no client is available for a certain request the request is queued until > there is one. > Worked on this from https://merlinux.de/svn/user/guido/pypybuilder before > checking in here. > > > Added: pypy/dist/pypy/tool/build/README.txt > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/README.txt Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,59 @@ > +============ > +PyPyBuilder > +============ > + > +What is this? > +============= > + > +PyPyBuilder is an application that allows people to build PyPy instances on > +demand. If you have a nice idle machine connected to the Internet, and don't > +mind us 'borrowing' it every once in a while, you can start up the client > +script (in bin/client) and have the server send compile jobs to your machine. > +If someone requests a build of PyPy that is not already available on the PyPy > +website, and your machine is capable of making such a build, the > server may ask > +your machine to create it. If enough people participate, with diverse enough > +machines, an ad-hoc 'build farm' is created this way. > + > +Components > +========== > + > +The application consists of 3 main components: a server component, > a client and > +a small component to start compile jobs, which we'll call 'startcompile' for > +now. > + > +The server waits for clients to register, and for compile job requests. When > +clients register, they pass the server information about what > compilations they > +can handle (system info). Then when the 'startcompile' component requests a > +compilation job, the server first checks whether a binary is > already available, > +and if so returns that. > + > +If there isn't one, the server walks through a list of connected > clients to see > +if there is one that can handle the job, and if so tells it to perform it. If > +there's no client to handle the job, it gets queued until there is. > + > +Once a build is available, the server will send an email to all > email addresses > +(it could be more than one person asked for the same build at the same time!) > +passed to it by 'startcompile'. > + > +Installation > +============ > + > +Client > +------ > + > +Installing the system should not be required: just run '. > /bin/client' to start > +the client. Note that it depends on the `py lib`_. > + > +Server > +------ > + > +Also for the server there's no real setup required, and again there's a > +dependency on the `py lib`_. > + > +.. _`py lib`: http://codespeak.net/py > + > +More info > +========= > + > +For more information, bug reports, patches, etc., please send an email to > [EMAIL PROTECTED] > > Added: pypy/dist/pypy/tool/build/__init__.py > ============================================================================== > > Added: pypy/dist/pypy/tool/build/bin/client > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/bin/client Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,58 @@ > +#!/usr/bin/python > + > +BUFSIZE = 1024 > + > +import path > +import sys > +import random > +from pypy.tool.build import config > + > +# XXX using random values for testing > +modules = ['_stackless', '_socket'] > + > +""" > +random.shuffle(modules) > +sysinfo = { > + 'maxint': random.choice((sys.maxint, (2 ** 63 - 1))), > + 'use_modules': modules[:random.randrange(len(modules) + 1)], > + 'byteorder': random.choice(('little', 'big')), > +} > +""" > + > +sysinfo = { > + 'maxint': sys.maxint, > + 'use_modules': ['_stackless', '_socket'], > + 'byteorder': sys.byteorder, > +} > + > +if __name__ == '__main__': > + from py.execnet import SshGateway > + from pypy.tool.build.client import init > + gw = SshGateway(config.server) > + channel = init(gw, sysinfo, path=config.path, port=config.port) > + print channel.receive() # welcome message > + try: > + while 1: > + data = channel.receive() > + if not isinstance(data, dict): # needs more checks here > + raise ValueError( > + 'received wrong unexpected data of type %s' % > (type(data),) > + ) > + info = data > + # XXX we should compile here, using data dict for info > + print 'compilation requested for info %r, now faking > that' % (info,) > + import time; time.sleep(10) > + > + # write the zip to the server in chunks to server > + # XXX we're still faking this > + zipfp = (path.packagedir / 'test/test.zip').open() > + while True: > + chunk = zipfp.read(BUFSIZE) > + if not chunk: > + break > + channel.send(chunk) > + channel.send(None) # tell the server we're done > + print 'done with compilation, waiting for next' > + finally: > + channel.close() > + gw.exit() > > Added: pypy/dist/pypy/tool/build/bin/path.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/bin/path.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,5 @@ > +import py > + > +packagedir = py.magic.autopath().dirpath().dirpath() > +rootpath = packagedir.dirpath().dirpath().dirpath() > +py.std.sys.path.append(str(rootpath)) > > Added: pypy/dist/pypy/tool/build/bin/server > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/bin/server Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,27 @@ > +#!/usr/bin/python > + > +import path > +from pypy.tool.build import config > + > +from py.execnet import SshGateway > + > +if __name__ == '__main__': > + from py.execnet import SshGateway > + from pypy.tool.build.server import init > + > + gw = SshGateway(config.server) > + channel = init(gw, port=config.port, path=config.path, > + projectname=config.projectname, > + buildpath=str(config.buildpath), > + mailhost=config.mailhost, > + mailport=config.mailport, > + mailfrom=config.mailfrom) > + > + try: > + while 1: > + data = channel.receive() > + assert isinstance(data, str) > + print data > + finally: > + channel.close() > + gw.exit() > > Added: pypy/dist/pypy/tool/build/bin/startcompile > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/bin/startcompile Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,57 @@ > +#!/usr/bin/python > + > +import path > +import sys > +import random > +from pypy.tool.build import config > + > +initcode = """ > + import sys > + sys.path += %r > + > + from pypy.tool.build import ppbserver > + channel.send(ppbserver.compile(%r, %r)) > + channel.close() > +""" > +def init(gw, sysinfo, email, port=12321): > + from pypy.tool.build import execnetconference > + > + conference = execnetconference.conference(gw, port, False) > + channel = conference.remote_exec(initcode % (config.path, > email, sysinfo)) > + return channel > + > +if __name__ == '__main__': > + from py.execnet import SshGateway > + > + from optparse import OptionParser > + optparser = OptionParser('%prog [options] email') > + for args, kwargs in config.options: > + optparser.add_option(*args, **kwargs) > + optparser.add_option('-r', '--revision', dest='revision', > default='trunk', > + help='SVN revision (defaults to "trunk")') > + > + (options, args) = optparser.parse_args() > + > + if not args or len(args) != 1: > + optparser.error('please provide an email address') > + > + sysinfo = dict([(attr, getattr(options, attr)) for attr in > dir(options) if > + not attr.startswith('_') and > + not callable(getattr(options, attr))]) > + > + print 'going to start compile job with info:' > + for k, v in sysinfo.items(): > + print '%s: %r' % (k, v) > + print > + > + gw = SshGateway(config.server) > + channel = init(gw, sysinfo, args[0], port=config.port) > + ispath, data = channel.receive() > + if ispath: > + print ('a suitable result is already available, you can find it ' > + 'at "%s"' % (data,)) > + else: > + print data > + print 'you will be mailed once it\'s ready' > + channel.close() > + gw.exit() > > Added: pypy/dist/pypy/tool/build/client.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/client.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,71 @@ > +import time > +import thread > + > +class PPBClient(object): > + def __init__(self, channel, sysinfo, testing=False): > + self.channel = channel > + self.sysinfo = sysinfo > + self.busy_on = None > + self.testing = testing > + > + from pypy.tool.build import ppbserver > + self.server = ppbserver > + self.server.register(self) > + > + def sit_and_wait(self): > + """connect to the host and wait for commands""" > + self.channel.waitclose() > + self.channel.close() > + > + def compile(self, info): > + """send a compile job to the client side > + > + this waits until the client is done, and assumes the client sends > + back the whole binary as a single string (XXX this > should change ;) > + """ > + self.busy_on = info > + self.channel.send(info) > + thread.start_new_thread(self.wait_until_done, (info,)) > + > + def wait_until_done(self, info): > + buildpath = self.server.get_new_buildpath(info) > + > + fp = buildpath.zipfile.open('w') > + if not self.testing: > + try: > + while True: > + try: > + chunk = self.channel.receive() > + except EOFError: > + # stop compilation, client has disconnected > + return > + if chunk is None: > + break > + fp.write(chunk) > + finally: > + fp.close() > + > + self.server.compilation_done(info, buildpath) > + self.busy_on = None > + > +initcode = """ > + import sys > + sys.path += %r > + > + from pypy.tool.build.client import PPBClient > + > + try: > + client = PPBClient(channel, %r, %r) > + client.sit_and_wait() > + finally: > + channel.close() > +""" > +def init(gw, sysinfo, path=None, port=12321, testing=False): > + from pypy.tool.build import execnetconference > + > + if path is None: > + path = [] > + > + conference = execnetconference.conference(gw, port, False) > + channel = conference.remote_exec(initcode % (path, sysinfo, testing)) > + return channel > > Added: pypy/dist/pypy/tool/build/config.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/config.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,45 @@ > +import py > + > +# general settings, used by both server and client > +server = 'johnnydebris.net' > +port = 12321 > +path = ['/home/johnny/temp/pypy-dist'] > + > +# option definitions for the startcompile script > +# for now we have them here, we should probably use pypy's config instead > +# though... > +import sys > +def _use_modules_callback(option, opt_str, value, parser): > + parser.values.use_modules = [m.strip() for m in value.split(',') > + if m.strip()] > + > +def _maxint_callback(option, opt_str, value, parser): > + parser.values.maxint = 2 ** (int(value) - 1) - 1 > + > +options = [ > + (('-m', '--use-modules'), {'action': 'callback', 'type': 'string', > + 'callback': _use_modules_callback, > + 'dest': 'use_modules', 'default': [], > + 'help': 'select the modules you > want to use'}), > + (('-i', '--maxint'), {'action': 'callback', 'callback': _maxint_callback, > + 'default': sys.maxint, 'dest': 'maxint', > + 'type': 'string', > + 'help': ('size of an int in bits (32/64, ' > + 'defaults to sys.maxint)')}), > + (('-b', '--byteorder'), {'action': 'store', > + 'dest': 'byteorder', 'default': > sys.byteorder, > + 'nargs': 1, > + 'help': ('byte order (little/big, defaults ' > + 'to sys.byteorder)')}), > +] > + > +# settings for the server > +projectname = 'pypy' > +buildpath = '/home/johnny/temp/pypy-dist/pypy/tool/build/builds' > +mailhost = '127.0.0.1' > +mailport = 25 > +mailfrom = '[EMAIL PROTECTED]' > + > +# settings for the tests > +testpath = [str(py.magic.autopath().dirpath().dirpath())] > + > > Added: pypy/dist/pypy/tool/build/conftest.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/conftest.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,20 @@ > +import py > +from py.__.documentation.conftest import Directory as Dir, DoctestText, \ > + ReSTChecker > +mypath = py.magic.autopath().dirpath() > + > +Option = py.test.Config.Option > +option = py.test.Config.addoptions("pypybuilder test options", > + Option('', '--functional', > + action="store_true", dest="functional", default=False, > + help="run pypybuilder functional tests" > + ), > +) > + > +py.test.pypybuilder_option = option > + > +class Directory(Dir): > + def run(self): > + if self.fspath == mypath: > + return ['README.txt', 'test'] > + return super(Directory, self).run() > > Added: pypy/dist/pypy/tool/build/execnetconference.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/execnetconference.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,126 @@ > +""" > +An extension to py.execnet to allow multiple programs to exchange information > +via a common server. The idea is that all programs first open a gateway to > +the same server (e.g. an SshGateway), and then call the conference() function > +with a local TCP port number. The first program must pass is_server=True and > +the next ones is_server=False: the first program's remote gateway is used as > +shared server for the next ones. > + > +For all programs, the conference() call returns a new gateway that is > +connected to the Python process of this shared server. Information can > +be exchanged by passing data around within this Python process. > +""" > +import py > +from py.__.execnet.register import InstallableGateway > + > + > +def conference(gateway, port, is_server='auto'): > + if is_server: # True or 'auto' > + channel = gateway.remote_exec(r""" > + import thread > + from socket import * > + s = socket(AF_INET, SOCK_STREAM) > + port = channel.receive() > + try: > + s.bind(('', port)) > + s.listen(5) > + except error: > + channel.send(0) > + else: > + channel.send(1) > + > + def readall(s, n): > + result = '' > + while len(result) < n: > + t = s.read(n-len(result)) > + if not t: > + raise EOFError > + result += t > + return result > + > + def handle_connexion(clientsock, address): > + clientfile = clientsock.makefile('r+b',0) > + source = clientfile.readline().rstrip() > + clientfile.close() > + g = {'clientsock' : clientsock, 'address' : address} > + source = eval(source) > + if source: > + g = {'clientsock' : clientsock, 'address' : address} > + co = compile(source+'\n', source, 'exec') > + exec co in g > + > + while True: > + conn, addr = s.accept() > + if addr[0] == '127.0.0.1': # else connexion refused > + thread.start_new_thread(handle_connexion, > (conn, addr)) > + del conn > + """) > + channel.send(port) > + ok = channel.receive() > + if ok: > + return gateway > + if is_server == 'auto': > + pass # fall-through and try as a client > + else: > + raise IOError("cannot listen on port %d (already in > use?)" % port) > + > + if 1: # client > + channel = gateway.remote_exec(r""" > + import thread > + from socket import * > + s = socket(AF_INET, SOCK_STREAM) > + port = channel.receive() > + s.connect(('', port)) > + channel.send(1) > + def receiver(s, channel): > + while True: > + data = s.recv(4096) > + #print >> open('LOG','a'), 'backward', repr(data) > + channel.send(data) > + if not data: break > + thread.start_new_thread(receiver, (s, channel)) > + try: > + for data in channel: > + #print >> open('LOG','a'), 'forward', repr(data) > + s.sendall(data) > + finally: > + s.shutdown(1) > + """) > + channel.send(port) > + ok = channel.receive() > + assert ok > + return InstallableGateway(ConferenceChannelIO(channel)) > + > + > +class ConferenceChannelIO: > + server_stmt = """ > +io = SocketIO(clientsock) > +""" > + > + error = (EOFError,) > + > + def __init__(self, channel): > + self.channel = channel > + self.buffer = '' > + > + def read(self, numbytes): > + #print >> open('LOG', 'a'), 'read %d bytes' % numbytes > + while len(self.buffer) < numbytes: > + t = self.channel.receive() > + if not t: > + #print >> open('LOG', 'a'), 'EOFError' > + raise EOFError > + self.buffer += t > + buf, self.buffer = self.buffer[:numbytes], self.buffer[numbytes:] > + #print >> open('LOG', 'a'), '--->', repr(buf) > + return buf > + > + def write(self, data): > + #print >> open('LOG', 'a'), 'write(%r)' % (data,) > + self.channel.send(data) > + > + def close_read(self): > + pass > + > + def close_write(self): > + self.channel.close() > > Added: pypy/dist/pypy/tool/build/server.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/server.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,332 @@ > +import random > +import time > +import thread > +import smtplib > +import py > + > +def issubdict(d1, d2): > + """sees whether a dict is a 'subset' of another dict > + > + dictvalues can be immutable data types and list and dicts of > + immutable data types and lists and ... (recursive) > + """ > + for k, v in d1.iteritems(): > + if not k in d2: > + return False > + d2v = d2[k] > + if isinstance(v, dict): > + if not issubdict(v, d2v): > + return False > + elif isinstance(v, list): > + if not set(v).issubset(set(d2v)): > + return False > + elif v != d2v: > + return False > + return True > + > +# XXX note that all this should be made thread-safe at some point (meaning it > +# currently isn't)!! > + > +class RequestStorage(object): > + """simple registry that manages information""" > + def __init__(self, info_to_path=[]): > + self._id_to_info = {} # id -> info dict > + self._id_to_emails = {} # id -> requestor email address > + self._id_to_path = {} # id -> filepath > + > + self._last_id = 0 > + self._id_lock = thread.allocate_lock() > + > + self._build_initial(info_to_path) > + > + def request(self, email, info): > + """place a request > + > + this either returns a path to the binary (if it's available > + already) or an id for the info > + """ > + self._normalize(info) > + infoid = self.get_info_id(info) > + path = self._id_to_path.get(infoid) > + if path is not None: > + return path > + self._id_to_emails.setdefault(infoid, []).append(email) > + > + def get_info_id(self, info): > + """retrieve or create an id for an info dict""" > + self._id_lock.acquire() > + try: > + self._normalize(info) > + for k, v in self._id_to_info.iteritems(): > + if v == info: > + return k > + self._last_id += 1 > + id = self._last_id > + self._id_to_info[id] = info > + return id > + finally: > + self._id_lock.release() > + > + def add_build(self, info, path): > + """store the data for a build and make it available > + > + returns a list of email addresses for the people that should be > + warned > + """ > + self._normalize(info) > + infoid = self.get_info_id(info) > + emails = self._id_to_emails.pop(infoid) > + self._id_to_path[infoid] = path > + return emails > + > + def _build_initial(self, info_to_path): > + """fill the dicts with info about files that are already built""" > + for info, path in info_to_path: > + id = self.get_info_id(info) > + self._id_to_path[id] = path > + > + def _normalize(self, info): > + for k, v in info.iteritems(): > + if isinstance(v, list): > + v.sort() > + > +from py.__.path.local.local import LocalPath > +class BuildPath(LocalPath): > + def _info(self): > + info = getattr(self, '_info_value', {}) > + if info: > + return info > + infopath = self / 'info.txt' > + if not infopath.check(): > + return {} > + for line in infopath.readlines(): > + line = line.strip() > + if not line: > + continue > + chunks = line.split(':') > + key = chunks.pop(0) > + value = ':'.join(chunks) > + info[key] = eval(value) > + self._info_value = info > + return info > + > + def _set_info(self, info): > + self._info_value = info > + infopath = self / 'info.txt' > + infopath.ensure() > + fp = infopath.open('w') > + try: > + for key, value in info.iteritems(): > + fp.write('%s: %r\n' % (key, value)) > + finally: > + fp.close() > + > + info = property(_info, _set_info) > + > + def _zipfile(self): > + return py.path.local(self / 'data.zip') > + > + def _set_zipfile(self, iterable): > + # XXX not in use right now... > + fp = self._zipfile().open('w') > + try: > + for chunk in iterable: > + fp.write(chunk) > + finally: > + fp.close() > + > + zipfile = property(_zipfile, _set_zipfile) > + > +class PPBServer(object): > + retry_interval = 10 > + > + def __init__(self, projname, channel, builddir, mailhost=None, > + mailport=None, mailfrom=None): > + self._projname = projname > + self._channel = channel > + self._builddir = builddir > + self._mailhost = mailhost > + self._mailport = mailport > + self._mailfrom = mailfrom > + > + self._buildpath = py.path.local(builddir) > + self._clients = [] > + info_to_path = [(p.info, str(p)) for p in > + self._get_buildpaths(builddir)] > + self._requeststorage = RequestStorage(info_to_path) > + self._queued = [] > + > + self._queuelock = thread.allocate_lock() > + self._namelock = thread.allocate_lock() > + > + def register(self, client): > + self._clients.append(client) > + self._channel.send('registered %s with info %r' % ( > + client, client.sysinfo)) > + client.channel.send('welcome') > + > + def compile(self, requester_email, info): > + """start a compilation > + > + returns a tuple (ispath, data) > + > + if there's already a build available for info, this will return > + a tuple (True, path), if not, this will return (False, message), > + where message describes what is happening with the request (is > + a build made rightaway, or is there no client available?) > + > + in any case, if the first item of the tuple returned is False, > + an email will be sent once the build is available > + """ > + path = self._requeststorage.request(requester_email, info) > + if path is not None: > + self._channel.send('already a build for this info available') > + return (True, path) > + for client in self._clients: > + if client.busy_on == info: > + self._channel.send('build for %r currently in progress' % > + (info,)) > + return (False, 'this build is already in progress') > + # we don't have a build for this yet, find a client to compile it > + if self.run(info): > + return (False, 'found a suitable client, going to build') > + else: > + self._queuelock.acquire() > + try: > + self._queued.append(info) > + finally: > + self._queuelock.release() > + return (False, 'no suitable client found; your request > is queued') > + > + def run(self, info): > + """find a suitable client and run the job if possible""" > + # XXX shuffle should be replaced by something smarter obviously ;) > + clients = self._clients[:] > + random.shuffle(clients) > + rev = info.pop('revision', 'trunk') > + for client in clients: > + # popping out revision here, going to add later... the client > + # should be able to retrieve source code for any revision (so > + # it doesn't need to match a revision field in client.sysinfo) > + if client.busy_on or not issubdict(info, client.sysinfo): > + continue > + else: > + info['revision'] = rev > + self._channel.send( > + 'going to send compile job with info %r to %s' % ( > + info, client > + ) > + ) > + client.compile(info) > + return True > + info['revision'] = rev > + self._channel.send( > + 'no suitable client available for compilation with info %r' % ( > + info, > + ) > + ) > + > + def serve_forever(self): > + """this keeps the script from dying, and re-tries jobs""" > + self._channel.send('going to serve') > + while 1: > + time.sleep(self.retry_interval) > + self._cleanup_clients() > + self._try_queued() > + > + def get_new_buildpath(self, info): > + path = BuildPath(str(self._buildpath / self._create_filename())) > + path.info = info > + return path > + > + def compilation_done(self, info, path): > + """client is done with compiling and sends data""" > + self._channel.send('compilation done for %r, written to %s' % ( > + info, path)) > + emails = self._requeststorage.add_build(info, path) > + for emailaddr in emails: > + self._send_email(emailaddr, info, path) > + > + def _cleanup_clients(self): > + self._queuelock.acquire() > + try: > + clients = self._clients[:] > + for client in clients: > + if client.channel.isclosed(): > + if client.busy_on: > + self._queued.append(client.busy_on) > + self._clients.remove(client) > + finally: > + self._queuelock.release() > + > + def _try_queued(self): > + self._queuelock.acquire() > + try: > + toremove = [] > + for info in self._queued: > + if self.run(info): > + toremove.append(info) > + for info in toremove: > + self._queued.remove(info) > + finally: > + self._queuelock.release() > + > + def _get_buildpaths(self, dirpath): > + for p in py.path.local(dirpath).listdir(): > + yield BuildPath(str(p)) > + > + _i = 0 > + def _create_filename(self): > + self._namelock.acquire() > + try: > + today = time.strftime('%Y%m%d') > + buildnames = [p.basename for p in > + py.path.local(self._buildpath).listdir()] > + while True: > + name = '%s-%s-%s' % (self._projname, today, self._i) > + self._i += 1 > + if name not in buildnames: > + return name > + finally: > + self._namelock.release() > + > + def _send_email(self, addr, info, path): > + self._channel.send('going to send email to %s' % (addr,)) > + if self._mailhost is not None: > + msg = '\r\n'.join([ > + 'From: %s' % (self._mailfrom,), > + 'To: %s' % (addr,), > + 'Subject: %s compilation done' % (self._projname,), > + '', > + 'The compilation you requested is done. You can find it at', > + str(path), > + ]) > + server = smtplib.SMTP(self._mailhost, self._mailport) > + server.set_debuglevel(0) > + server.sendmail(self._mailfrom, addr, msg) > + server.quit() > + > +initcode = """ > + import sys > + sys.path += %r > + > + try: > + from pypy.tool.build.server import PPBServer > + server = PPBServer(%r, channel, %r, %r, %r, %r) > + > + # make the server available to clients as pypy.tool.build.ppbserver > + from pypy.tool import build > + build.ppbserver = server > + > + server.serve_forever() > + finally: > + channel.close() > +""" > +def init(gw, port=12321, path=[], projectname='pypy', buildpath=None, > + mailhost=None, mailport=25, mailfrom=None): > + from pypy.tool.build import execnetconference > + conference = execnetconference.conference(gw, port, True) > + channel = conference.remote_exec(initcode % (path, projectname,buildpath, > + mailhost, mailport, > + mailfrom)) > + return channel > > Added: pypy/dist/pypy/tool/build/test/fake.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/test/fake.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,54 @@ > +from pypy.tool.build.server import BuildPath > + > +class FakeChannel(object): > + def __init__(self): > + self._buffer = [] > + > + def send(self, item): > + self._buffer.append(item) > + > + def receive(self): > + return self._buffer.pop(0) > + > + def close(self): > + pass > + > + def waitclose(self): > + pass > + > +class FakeClient(object): > + def __init__(self, info): > + self.channel = FakeChannel() > + self.sysinfo = info > + self.busy_on = None > + > + def compile(self, info): > + info.pop('revision') > + for k, v in info.items(): > + self.channel.send('%s: %r' % (k, v)) > + self.channel.send(None) > + self.busy_on = info > + > +class FakeServer(object): > + def __init__(self, builddirpath): > + builddirpath.ensure(dir=True) > + self._channel = FakeChannel() > + self._builddirpath = builddirpath > + self._clients = [] > + self._done = [] > + > + def register(self, client): > + self._clients.append(client) > + > + def compilation_done(self, info, data): > + self._done.append((info, data)) > + > + i = 0 > + def get_new_buildpath(self, info): > + name = 'build-%s' % (self.i,) > + self.i += 1 > + bp = BuildPath(str(self._builddirpath / name)) > + bp.info = info > + bp.ensure(dir=1) > + return bp > + > > Added: pypy/dist/pypy/tool/build/test/path.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/test/path.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,6 @@ > +import py > + > +testpath = py.magic.autopath().dirpath() > +packagepath = testpath.dirpath() > +rootpath = packagepath.dirpath().dirpath().dirpath() > +py.std.sys.path.append(str(rootpath)) > > Added: pypy/dist/pypy/tool/build/test/test.zip > ============================================================================== > Binary file. No diff available. > > Added: pypy/dist/pypy/tool/build/test/test_client.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/test/test_client.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,43 @@ > +import path > +from pypy.tool.build import client > +import py > +import time > +from fake import FakeChannel, FakeServer > + > +class ClientForTests(client.PPBClient): > + def __init__(self, *args, **kwargs): > + super(ClientForTests, self).__init__(*args, **kwargs) > + self._done = [] > + > +def setup_module(mod): > + mod.temp = temp = py.test.ensuretemp('pypybuilder-client') > + mod.svr = svr = FakeServer(temp) > + > + import pypy.tool.build > + pypy.tool.build.ppbserver = svr > + > + mod.c1c = c1c = FakeChannel() > + mod.c1 = c1 = ClientForTests(c1c, {'foo': 1, 'bar': [1,2]}) > + svr.register(c1) > + > + mod.c2c = c2c = FakeChannel() > + mod.c2 = c2 = ClientForTests(c2c, {'foo': 2, 'bar': [2,3]}) > + svr.register(c2) > + > +def test_compile(): > + info = {'foo': 1} > + c1.compile(info) > + c1.channel.receive() > + c1.channel.send('foo bar') > + c1.channel.send(None) > + > + # meanwhile the client starts a thread that waits until there's data > + # available on its own channel, with our FakeChannel it has > data rightaway, > + # though (the channel out and in are the same, and we just sent 'info' > + # over the out one) > + time.sleep(1) > + > + done = svr._done.pop() > + > + assert done[0] == info > + assert done[1] == (temp / 'build-0') > > Added: pypy/dist/pypy/tool/build/test/test_pypybuilder.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/test/test_pypybuilder.py Wed Jul 26 > 13:18:25 2006 > @@ -0,0 +1,137 @@ > +import path > +from pypy.tool.build import client, server, execnetconference > +from pypy.tool.build import config > +import py > + > +# some functional tests (although some of the rest aren't strictly > +# unit tests either), to run use --functional as an arg to py.test > +def test_functional_1(): > + if not py.test.pypybuilder_option.functional: > + py.test.skip('skipping functional test, use --functional to run it') > + > + # XXX this one is a bit messy, it's a quick functional test forthe whole > + # system, but for instance contains time.sleep()s to make sure > all threads > + # get the time to perform tasks and such... > + > + sleep_interval = 0.3 > + > + # first initialize a server > + sgw = py.execnet.PopenGateway() > + temppath = py.test.ensuretemp('pypybuilder-functional') > + sc = server.init(sgw, port=config.port, path=config.testpath, > + buildpath=str(temppath)) > + > + # give the server some time to wake up > + py.std.time.sleep(sleep_interval) > + > + # then two clients, both with different system info > + sysinfo1 = { > + 'foo': 1, > + 'bar': [1,2], > + } > + cgw1 = py.execnet.PopenGateway() > + cc1 = client.init(cgw1, sysinfo1, port=config.port, testing=True) > + > + sysinfo2 = { > + 'foo': 2, > + 'bar': [1], > + } > + cgw2 = py.execnet.PopenGateway() > + cc2 = client.init(cgw2, sysinfo2, port=config.port, testing=True) > + > + # give the clients some time to register themselves > + py.std.time.sleep(sleep_interval) > + > + # now we're going to send some compile jobs > + code = """ > + import sys > + sys.path += %r > + > + from pypy.tool.build import ppbserver > + channel.send(ppbserver.compile(%r, %r)) > + channel.close() > + """ > + compgw = py.execnet.PopenGateway() > + compconf = execnetconference.conference(compgw, config.port) > + > + # this one should fail because there's no client found for foo = 3 > + compc = compconf.remote_exec(code % (config.testpath, '[EMAIL PROTECTED]', > + {'foo': 3})) > + > + # sorry... > + py.std.time.sleep(sleep_interval) > + > + ret = compc.receive() > + assert not ret[0] > + assert ret[1].find('no suitable client found') > -1 > + > + # this one should be handled by client 1 > + compc = compconf.remote_exec(code % (config.testpath, '[EMAIL PROTECTED]', > + {'foo': 1, 'bar': [1]})) > + > + # and another one > + py.std.time.sleep(sleep_interval) > + > + ret = compc.receive() > + assert not ret[0] > + assert ret[1].find('found a suitable client') > -1 > + > + # the messages may take a bit to arrive, too > + py.std.time.sleep(sleep_interval) > + > + # client 1 should by now have received the info to build for > + cc1.receive() # 'welcome' > + ret = cc1.receive() > + assert ret == {'foo': 1, 'bar': [1], 'revision': 'trunk'} > + > + # this should have created a package in the temp dir > + assert len(temppath.listdir()) == 1 > + > + # now we're going to satisfy the first request by adding a new client > + sysinfo3 = {'foo': 3} > + cgw3 = py.execnet.PopenGateway() > + cc3 = client.init(cgw3, sysinfo3, port=config.port, testing=True) > + > + # again a bit of waiting may be desired > + py.std.time.sleep(sleep_interval) > + > + # _try_queued() should check whether there are new clients available for > + # queued jobs > + code = """ > + import sys, time > + sys.path += %r > + > + from pypy.tool.build import ppbserver > + ppbserver._try_queued() > + # give the server some time, the clients 'compile' in threads > + time.sleep(%s) > + channel.send(ppbserver._requeststorage._id_to_emails) > + channel.close() > + """ > + compgw2 = py.execnet.PopenGateway() > + compconf2 = execnetconference.conference(compgw2, config.port) > + > + compc2 = compconf2.remote_exec(code % (config.testpath, sleep_interval)) > + > + > + # we check whether all emails are now sent, since after adding the third > + # client, and calling _try_queued(), both jobs should have beenprocessed > + ret = compc2.receive() > + assert ret.values() == [] > + > + # this should also have created another package in the temp dir > + assert len(temppath.listdir()) == 2 > + > + # some cleanup (this should all be in nested try/finallys, blegh) > + cc1.close() > + cc2.close() > + cc3.close() > + compc.close() > + compc2.close() > + sc.close() > + > + cgw1.exit() > + cgw2.exit() > + compgw.exit() > + compgw2.exit() > + sgw.exit() > > Added: pypy/dist/pypy/tool/build/test/test_request_storage.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/test/test_request_storage.py Wed Jul > 26 13:18:25 2006 > @@ -0,0 +1,64 @@ > +import path > +import py > +from pypy.tool.build.server import RequestStorage > + > +def test_request_storage(): > + s = RequestStorage() > + > + assert s._id_to_info == {} > + assert s._id_to_emails == {} > + assert s._id_to_path == {} > + > + info = {'foo': 1} > + infoid = s.get_info_id(info) > + > + path = s.request('[EMAIL PROTECTED]', info) > + assert path is None > + assert s._id_to_info == {infoid: info} > + assert s._id_to_emails == {infoid: ['[EMAIL PROTECTED]']} > + assert s._id_to_path == {} > + > + path = s.request('[EMAIL PROTECTED]', info) > + assert path is None > + assert s._id_to_info == {infoid: info} > + assert s._id_to_emails == {infoid: ['[EMAIL PROTECTED]', '[EMAIL > PROTECTED]']} > + assert s._id_to_path == {} > + > + emails = s.add_build(info, 'foobar') > + assert emails == ['[EMAIL PROTECTED]', '[EMAIL PROTECTED]'] > + assert s._id_to_info == {infoid: info} > + assert s._id_to_emails == {} > + assert s._id_to_path == {infoid: 'foobar'} > + > + info2 = {'foo': 2, 'bar': [1,2]} > + infoid2 = s.get_info_id(info2) > + > + path = s.request('[EMAIL PROTECTED]', info2) > + assert path is None > + assert s._id_to_info == {infoid: info, infoid2: info2} > + assert s._id_to_emails == {infoid2: ['[EMAIL PROTECTED]']} > + assert s._id_to_path == {infoid: 'foobar'} > + > + emails = s.add_build(info2, 'foobaz') > + assert emails == ['[EMAIL PROTECTED]'] > + assert s._id_to_info == {infoid: info, infoid2: info2} > + assert s._id_to_emails == {} > + assert s._id_to_path == {infoid: 'foobar', infoid2: 'foobaz'} > + > + path = s.request('[EMAIL PROTECTED]', info) > + assert path == 'foobar' > + > +def test__build_initial(): > + s = RequestStorage([({'foo': 1}, 'foo'), ({'foo': 2}, 'bar'),]) > + > + id1 = s.get_info_id({'foo': 1}) > + id2 = s.get_info_id({'foo': 2}) > + > + assert s._id_to_info == {id1: {'foo': 1}, id2: {'foo': 2}} > + assert s._id_to_emails == {} > + assert s._id_to_path == {id1: 'foo', id2: 'bar'} > + > +def test__normalize(): > + s = RequestStorage() > + assert (s._normalize({'foo': ['bar', 'baz']}) == > + s._normalize({'foo': ['baz', 'bar']})) > > Added: pypy/dist/pypy/tool/build/test/test_server.py > ============================================================================== > --- (empty file) > +++ pypy/dist/pypy/tool/build/test/test_server.py Wed Jul 26 13:18:25 2006 > @@ -0,0 +1,132 @@ > +import path > +from pypy.tool.build import server > +import py > +from fake import FakeChannel, FakeClient > +from pypy.tool.build.server import RequestStorage > +from pypy.tool.build.server import BuildPath > +import time > + > +def setup_module(mod): > + mod.temppath = temppath = py.test.ensuretemp('pypybuilder-server') > + mod.svr = server.PPBServer('pypytest', FakeChannel(), str(temppath)) > + > + mod.c1 = FakeClient({'foo': 1, 'bar': [1,2]}) > + mod.svr.register(mod.c1) > + > + mod.c2 = FakeClient({'foo': 2, 'bar': [2,3]}) > + mod.svr.register(mod.c2) > + > +def test_server_issubdict(): > + from pypy.tool.build.server import issubdict > + assert issubdict({'foo': 1, 'bar': 2}, {'foo': 1, 'bar': 2, 'baz': 3}) > + assert not issubdict({'foo': 1, 'bar': 2}, {'foo': 1, 'baz': 3}) > + assert not issubdict({'foo': 1, 'bar': 3}, {'foo': 1, 'bar': 2,'baz': 3}) > + assert issubdict({'foo': [1,2]}, {'foo': [1,2,3]}) > + assert not issubdict({'foo': [1,2,3]}, {'foo': [1,2]}) > + assert issubdict({'foo': 1L}, {'foo': 1}) > + assert issubdict({}, {'foo': 1}) > + assert issubdict({'foo': [1,2]}, {'foo': [1,2,3,4], 'bar': [1,2]}) > + > +# XXX: note that the order of the tests matters! the first test reads the > +# information from the channels that was set by the setup_module() function, > +# the rest assumes this information is already read... > + > +def test_register(): > + assert len(svr._clients) == 2 > + assert svr._clients[0] == c1 > + assert svr._clients[1] == c2 > + > + assert c1.channel.receive() == 'welcome' > + assert c2.channel.receive() == 'welcome' > + py.test.raises(IndexError, "c1.channel.receive()") > + > + assert svr._channel.receive().find('registered') > -1 > + assert svr._channel.receive().find('registered') > -1 > + py.test.raises(IndexError, 'svr._channel.receive()') > + > +def test_compile(): > + # XXX this relies on the output not changing... quite scary > + info = {'foo': 1} > + ret = svr.compile('[EMAIL PROTECTED]', info) > + assert not ret[0] > + assert ret[1].find('found a suitable client') > -1 > + assert svr._channel.receive().find('going to send compile job') > -1 > + assert c1.channel.receive() == 'foo: 1' > + assert c1.channel.receive() is None > + py.test.raises(IndexError, "c2.channel.receive()") > + > + svr.compile('[EMAIL PROTECTED]', {'foo': 3}) > + assert svr._channel.receive().find('no suitable client available') > -1 > + > + info = {'bar': [3]} > + ret = svr.compile('[EMAIL PROTECTED]', info) > + assert svr._channel.receive().find('going to send') > -1 > + assert c2.channel.receive() == 'bar: [3]' > + assert c2.channel.receive() is None > + py.test.raises(IndexError, "c1.channel.receive()") > + > + info = {'foo': 1} > + ret = svr.compile('[EMAIL PROTECTED]', info) > + assert not ret[0] > + assert ret[1].find('this build is already') > -1 > + assert svr._channel.receive().find('currently in progress') > -1 > + > + c1.busy_on = None > + bp = BuildPath(str(temppath / 'foo')) > + svr.compilation_done(info, bp) > + ret = svr.compile('[EMAIL PROTECTED]', info) > + assert ret[0] > + assert isinstance(ret[1], BuildPath) > + assert ret[1] == bp > + assert svr._channel.receive().find('compilation done for') > -1 > + for i in range(2): > + assert svr._channel.receive().find('going to send email to') > -1 > + assert svr._channel.receive().find('already a build for this info') > -1 > + > +def test_buildpath(): > + tempdir = py.test.ensuretemp('pypybuilder-buildpath') > + # grmbl... local.__new__ checks for class equality :( > + bp = BuildPath(str(tempdir / 'test1')) > + assert not bp.check() > + assert bp.info == {} > + > + bp.info = {'foo': 1, 'bar': [1,2]} > + assert bp.info == {'foo': 1, 'bar': [1,2]} > + assert (sorted((bp / 'info.txt').readlines()) == > + ['bar: [1, 2]\n', 'foo: 1\n']) > + > + assert isinstance(bp.zipfile, py.path.local) > + bp.zipfile = ['foo', 'bar', 'baz'] > + assert bp.zipfile.read() == 'foobarbaz' > + > +def test__create_filename(): > + svr._i = 0 # reset counter > + today = time.strftime('%Y%m%d') > + name1 = svr._create_filename() > + assert name1 == 'pypytest-%s-0' % (today,) > + assert svr._create_filename() == ('pypytest-%s-1' % (today,)) > + bp = BuildPath(str(temppath / ('pypytest-%s-2' % (today,)))) > + try: > + bp.ensure() > + assert svr._create_filename() == 'pypytest-%s-3'% (today,) > + finally: > + bp.remove() > + > +def test_get_new_buildpath(): > + svr._i = 0 > + today = time.strftime('%Y%m%d') > + > + path1 = svr.get_new_buildpath({'foo': 'bar'}) > + try: > + assert isinstance(path1, BuildPath) > + assert path1.info == {'foo': 'bar'} > + assert path1.basename == 'pypytest-%s-0' % (today,) > + > + try: > + path2 = svr.get_new_buildpath({'foo': 'baz'}) > + assert path2.info == {'foo': 'baz'} > + assert path2.basename == 'pypytest-%s-1' % (today,) > + finally: > + path2.remove() > + finally: > + path1.remove() > _______________________________________________ > pypy-svn mailing list > [email protected] > http://codespeak.net/mailman/listinfo/pypy-svn > _______________________________________________ [email protected] http://codespeak.net/mailman/listinfo/pypy-dev
