On Tue, May 18, 2010 at 5:21 PM, Guido Trotter <[email protected]> wrote: > On Tue, May 18, 2010 at 5:19 PM, Michael Hanselmann <[email protected]> wrote: >> 2010/5/18 Guido Trotter <[email protected]>: >>> Basically each job would have a subscribers list, and it would be the >>> job of the thread which updates the job to also update the subscribers >>> on the change (by queueing data to be sent to them, which will be done >>> by asyncore in the main thread at its convenience). >> >> I think there should be more separation between jobs (or mcpu/jqueue) >> here. Jobs should just notify another thread which will take care of >> all wait-for-changes requests (comparing previous to current >> information, etc.). This way you don't delay the job with composing >> replies to LUXI requests. >> > > Makes sense. I will add this to the design. >
Interdiff: diff --git a/doc/design-2.2.rst b/doc/design-2.2.rst index c18e7a7..bf2f078 100644 --- a/doc/design-2.2.rst +++ b/doc/design-2.2.rst @@ -50,7 +50,9 @@ Currently the Ganeti master daemon is based on four sets of threads: - The rpc worker pool (10) interacts with the nodes via http-based-rpc This means that every masterd currently runs 52 threads to do its job. -Being able to reduce this number would make the master a lot simpler. +Being able to reduce the number of thread sets would make the master's +architecture a lot simpler. Moreover even having less threads can help +decreasing lock contention and log pollution. Also, even with this big number of threads masterd suffers from quite a few scalability issues: @@ -60,8 +62,8 @@ few scalability issues: number of threads won't help with lock contention. - Some luxi operations (in particular REQ_WAIT_FOR_JOB_CHANGE) make the relevant client thread block on its job for a relatively long time. - This makes it easier to finish the 16 client threads. -- The luxi lock is quite heavily contended, and certain easily + This makes it easier to exhaust the 16 client threads. +- The job queue lock is quite heavily contended, and certain easily reproducible worklogs show that's it's very easy to put masterd in trouble: for example running ~15 background instance reinstall jobs, results in a master daemon that, even without having finished the @@ -76,10 +78,17 @@ following core changes: - The main thread of masterd is moved to asyncore (so it can share the mainloop code with all other ganeti daemons) and handles all client - connections. + connections. Data is read from the client sockets as it becomes + available, and pushed into a buffer. Complete messages are then passed + to the worker pool for parsing and processing. A method on the message + object can then be used to provide replies. - The REQ_WAIT_FOR_JOB_CHANGE luxi request is changed to be subscription-based, so that the executing thread doesn't have to be - hogged while changes arrive. + waiting for the changes to arrive. Threads producing messages (job + queue executors) will signal the client work queue threads when there + is a change, and those will notify the subscribed clients. This should + be implemented without requiring the work queue threads to reacquire + the job queue lock, to decrease its contention. - The job queue lock is reviewed to decrease its contention, making the job queue more interactive. Thanks, Guido
