-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 It has been on the radar for a while that the time to connect to the codehosting service is a bit long. Specifically, the time it takes to do a full ssh handshake, and then get a live bzr smart server to talk to can take as long as 9 seconds, though seems to average about 3-4s.
https://lpstats.canonical.com/graphs/CodehostingPerformance/ [1] The idea is that essentially, we are paying the python startup overhead every time we get a connection, while we could instead preload all of that into a process, and just fork() it when a request comes in. (And arguably we could eventually prefork, and just use this one we already have over here.) My basic proof of concept seems to be working, but I'm a bit stalled as to how to get from PoC to having it landed and running on Codebrowse. As for numbers: 900ms 'time (echo hello | bzr serve --inet)' 2400ms 'time (echo hello | bzr lp-serve --inet)' ~80ms 'time (overly complex stuff to spawn and do the same work)' The last is a bit of an approximation, because my test front end hack uses select.poll() over file descriptors, and doesn't seem to be getting POLLHUP on any of the fifos. Anyway, it certainly looks like the concept has the potential to a couple of seconds from the connection time. The code is available here: lp:~jameinel/launchpad/lp-service There isn't any unit testing yet, but part of that was because I wasn't 100% sure of the specific design. So I'd like to solicit some feedback. I'll try to bring up my specific doubts, but any feedback on the work is welcome. 1) Plain socket.socket() service versus a 'twisted' implementation. I'm currently just creating a simple TCP/IP socket, and listening on it for a request to spawn a child (or quit), then forking, creating fifos on disk, reporting the path to the requester. The forked child then hangs on those handles, and remaps stdin/out/err and running the lp-serve code. a) Eventually I would hook up the existing lib/lp/codehosting/sshserver code to make the request to the daemon, and hook up a fake ProcessTransport(?) instead of the self._transport = self.reactor.spawnProcess() b) Current service is single threaded, though I thought it would probably be enough because it only blocks long enough to fork(). c) If most of the launchpad internals are based around twisted services, it probably would be best for consistency to also have a twisted service. I personally know very very little about twisted. 2) Robert mentioned in the past 'why not just have sshserver fork itself'. This would be possible, though I'm already seeing some issues wrt isolation. (logging and bzrlib.ui.ui_factory have some state wrt sys.std* that I have to manually poke at.) 3) Prefork. I think the existing code would lend itself pretty easily to preforking. The forking code would probably be reworked a \ little, and timing would need to be sorted out. (At the moment the spawned child doesn't return the path to the caller until the files have been created, but w/ prefork we would need a way to inform the daemon that the files exist now, though the daemon itself could create them and inform the child what path to use.) 4) Launchpad code review, testing, landing, rollout, etc. I really don't have much knowledge beyond "submit a merge proposal". I've read dev.launchpad.net/Getting /Help and /Build. And certainly have things building locally. /Help basically says to ask here :) I guess I'm basically asking for some mentoring, and not really sure who to talk to about it. (stuff like ec2-land and running the lp test suite still scare me a bit) Thanks for whatever help you can provide, John =:-> [1]: That graph is a local network connection. The primary overhead seems to be the time for 'bzr lp-serve' to start up a new bzr instance, and load all the related libraries. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkxpmRIACgkQJdeBCYSNAAO6YgCfUXud72PaxFiCr8uqEPGHJLGH fMwAn0vDHrLe6knYVjkcKQSiuu+mC+nJ =0a4y -----END PGP SIGNATURE----- _______________________________________________ Mailing list: https://launchpad.net/~launchpad-dev Post to : [email protected] Unsubscribe : https://launchpad.net/~launchpad-dev More help : https://help.launchpad.net/ListHelp
