Is it possible to use shared filesystem to store/manage semaphore locks? On Jul 22, 2013, at 17:20 , Josh Thompson <[email protected]> wrote:
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > The problem with load balancing the front end is that there is a semaphore > lock (php's built in implementation) around the scheduling code. Without it, > 2 users coming in at the same time can get assigned the same computer. So, > having multiple frontends would allow the same thing to happen. > > I've considered just locking a table in the database as a semaphore. > However, > in my testing of this, if the web server locks a table and then goes offline > before unlocking it, the table stays locked. So, done this way, if there > were > multiple frontends and one went offline while in the middle of the lock, none > of them would be able to create reservations. I don't remember what had to > be > done to release the lock. > > The option I'd like to use that I've not gotten around to implementing (I > believe there is a really old JIRA issue for it) is to add an XMLRPC API > function for calling the scheduling code. Then, have one frontend assigned > as > the one that would handle all of the scheduling. The others would make calls > to that one for just the scheduling part, but would do everything else > normally. Optionally, there could be an election process so that a new > frontend could be selected to do the scheduling if the one that had been > doing > it went down. > > Aaron C. listed some good ideas, but I think the above would be more > straightforward since it would not involve bringing in other technologies. > > Josh > > On Thursday, July 18, 2013 9:49:39 AM Aaron Peeler wrote: >> I like both of these approaches, especially 1, but I'm not sure of the >> effort it would take to convert. >> -AP >> >> On Wed, Jul 17, 2013 at 2:59 PM, Aaron Coburn <[email protected]> wrote: >>> In my opinion, this issue has largely been solved by the NoSQL and >>> application messaging communities. But as long as the VCL web server(s) >>> use local, file-based semaphores to control resource allocation, this >>> will be a hard nut to crack. >>> >>> If I were going to tackle this problem, I would take one of two general >>> approaches: >>> >>> 1) Use an asynchronous messaging queue (i.e. Apache ActiveMQ or something >>> like Redis) >>> >>> In this way, when a reservation is made, the request is pushed onto a >>> centralized queue, and some intermediate (i.e. vcld) process will shift >>> messages off the queue and assign the reservations. If we used ActiveMQ, >>> the php and perl code would communicate with the message broker over the >>> STOMP protocol [1]. Redis would also work because it runs in a single >>> thread and all operations are atomic -- requests can simply be pushed >>> onto a FIFO-type list structure [2]. >>> >>> 2. Use a NoSQL database such as CouchDB or Riak that uses a type of >>> optimistic locking model for all writes. >>> >>> That is, if reservations are stored in such a way that the resource id >>> (i.e. computer id) forms the database key, the assignment of a user to a >>> particular compute resource requires sending the correct revision_id of >>> that resource in order for the write to be successful. If successful, it >>> returns an HTTP 200 status code and the client proceeds normally; >>> otherwise, it sends a 409 header and it is up to the client to try again >>> [3]. It would also be possible to use the existing MySQL database to >>> emulate something like #2, but under a heavy load, I imagine that the row >>> locking could really slow things down. Using Couch would really be much >>> more scalable than trying to do everything in MySQL. >>> >>> Either of these approaches would then allow you to distribute the web >>> front end across multiple machines. >>> >>> Cheers, >>> Aaron Coburn >>> >>> [1] http://activemq.apache.org/stomp.html >>> [2] http://redis.io/commands/lpush >>> [3] http://wiki.apache.org/couchdb/HTTP_Document_API#PUT >>> >>> On Jul 17, 2013, at 1:06 PM, Aaron Peeler <[email protected]> wrote: >>>> This is definitely desired. >>>> >>>> An additional challenge with multiple web servers is to figure how to >>>> sanely lock/unlock the resource to prevent it from getting assigned >>>> two separate users that are making requests at the same instant. >>>> >>>> AP >>>> >>>> On Wed, Jul 17, 2013 at 12:57 PM, James O'Dell <[email protected]> > wrote: >>>>> I also tried load balancing the web. I didn't have any success. >>>>> >>>>> I also tried an SSL offload using the F5 >>>>> >>>>> The SSLoffload didn't work because >>>>> an internal VCL check noticed the header wasn't https >>>>> and redirected to https. Basically it kept looping >>>>> >>>>> On 7/17/2013 9:36 AM, Dmitri Chebotarov wrote: >>>>> >>>>> Hi >>>>> >>>>> I would like to load balance multiple front-end VCL servers. >>>>> It is F5 load balancer. The LB configuration allows to enable session >>>>> persistency. >>>>> Is this will be OK with VCL's front-end? I remember someone mentioned >>>>> some >>>>> issues with having multiple front-end servers, but don't remember >>>>> details. >>>>> >>>>> Thanks. >>>> >>>> -- >>>> Aaron Peeler >>>> Program Manager >>>> Virtual Computing Lab >>>> NC State University >>>> >>>> All electronic mail messages in connection with State business which >>>> are sent to or received by this account are subject to the NC Public >>>> Records Law and may be disclosed to third parties. > - -- > - ------------------------------- > Josh Thompson > VCL Developer > North Carolina State University > > my GPG/PGP key can be found at pgp.mit.edu > > All electronic mail messages in connection with State business which > are sent to or received by this account are subject to the NC Public > Records Law and may be disclosed to third parties. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.19 (GNU/Linux) > > iEYEARECAAYFAlHtojYACgkQV/LQcNdtPQNNjwCfQ+PRt/yZXI4tf2YDiH2IZu2m > coYAn2QbjICSa5MUR0cR9DIxniZVQFtK > =/YZs > -----END PGP SIGNATURE----- > > -- Thank you, Dmitri Chebotarov VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging 223 Aquia Building, Ffx, MSN: 1B5 Phone: (703) 993-6175 | Fax: (703) 993-3404
