On Tue, Jun 30, 2009 at 2:42 PM, Tomas Doran<bobtf...@bobtfish.net> wrote: > > You're doing it wrong. > > Don't block app server threads on a remote service if you have a slow remote > service, the only thing that lies down that route is doom and fail. >
I don't see the problem. In fact, this was the _main and central_ point of our application: to handle as many concurrent requests acting like a dam to the slower service. At first, it seems like a perfect application for EDA, but any attempt in this sense failed miserably... > Use a job queue or something so that your application servers don't sit > waiting for slow remote services. > I tried POE, AnyEvent and several other EDA patterns, tests, etc. making an external "Job Manager", and our customer even resorted to hire someone to hack the Catalyst Engine making it even-based and it got really ugly in the actual tests. Threads with mod_worker worked fine and proved to be far more stable/scalable than any other option we tried. Incredible as it sounds, multi-threaded was way more stable than the other paths we tried. The way I see it LWP was exactly the solution we needed for this. > If you really actually need to block architecturally, then a heavy weight > application server is just the wrong solution, full stop. > In this case, practical proved better than theory. We heavily tested in FreeBSD 6 and 7 and LInux 2.6.20+. The only thing we found was that Linux's VM still needs a lot of work once you start heavily using swap, it usually doesn't recover; it's usually faster switching than FreeBSD 6, and even 7 but a lot less stable. FreeBSD recovered in every case. > I'm sure that the worker mpm will give you more headroom if you have loads > of mod_perl processes blocking than prefork, but I very much consider this > to be an optimisation, not a solution. > Ok. What would you have done? - not meant as a defensive question but really, we would like to hear options for this application! believe me, we tried almost everything. The event-based Catalyst engine was a great idea, but the actual tests proved that it really had no real advantages over process/thread with mod_worker. Besides, it proved very problematic, unstable and hard to maintain in the long run. I remembered my previous experiences while working on Enterprise Service Bus technology and our results were very similar to Welsh's final conclusions on SEDA: "It is also worth noting that a number of recent research papers have demonstrated that the SEDA prototype (in Java) performs poorly compared to threaded or event-based systems implemented in C. This would seem to contradict the findings in my work. (For more information I invite you to read recent papers by Vivek Pai's group at Princeton and the Capriccio work from UC Berkeley...." Actually, no further work after SEDA has ever proved that Event-Based is actually better than process/thread. I confirmed this with people at Priceton (Vivek Pai himself) about a year ago. As stated, we tried AnyEvent and POE and fiddled with the idea of implementing some kind of Async mechanism to/from the client. It's a B2B server so the syncing problems could have gotten very messy. We needed to block anywhere from 1 to 7 seconds and we expected several thousand hits per app server to make it cost effective. This is what we came up with Catalyst: 50 Apache processes with 64 threads each for a total of 3200 threads using about 10GB of RAM (12GB is actually on each server, for the OS, the mutex and to avoid swapping at all costs). The 3200 blocking load can be handled with 2 dual core AMD 64 Bit processors (4 are actually used). not a bad deal cost-wise and it works. Each 64 thread process grows to about 200MB RSS (> 750MB VSZ), plus a base Apache root process of about 115MB. So we have 3200 actual blocking requests + the Apache mutex, so the overall capacity of each app server is quite decent for the price. At the moment we reset each process every 10000 requests with no observable or serious leaks or otherwise any sort of problems. We take about 65ms per request in our whole processing but sadly we have to block for the amount stated above. The app does have some complexity because it does considerable XML processing with LibXML 2 (DTD validations, XSL transformatiosn, XPath queries and such), database interaction with MySQL and talks to another server via LWP::UserAgent (this is where each thread blocks). We chose Catalyst precisely because of the flexibility to adapt to different scenarios (mod_perl, FastCGI, etc.) with little or no code change. > That said, I'm not trying to be disparaging, and I'm happy this works for > you, and is a viable option :) > It did and it works fine. This is why I wouldn't discourage people to use mod_worker, although as we have both agreed it all depends on the actual application. > The one thing that worries me about this is that it uses threads, and > threads and perl don't get on.. For example, Moose's string style type I've done many things with Perl and threads and they all have worked fine, albeit not trivial, as long as you get around the many gotchas. Even multi-threaded GTK2 stuff works with no serious limitations (long-lived threads though). In the web environment, with multi-threaded mod_perl you have the max requests per child param so you can work around and live with leaky/problematic modules or code. > constraints are a bad world (because the regex used makes various versions > of at least perl 5.8 core dump). I don't think that this is an issue for > anything in the Catalyst stack, as we're either type-constraint free, or use > exclusively MooseX::Types stuff (which doesn't use those regular > expressions, and therefore is safe, I think) - but may be a problem for > other code.. > Yeah. This app was built before the Moose port, and I honestly don't know much about Moose other than the fact that it was a pain in the butt to upgrade my apps ;-) I still have to get up to speed with Moose and the Catalyst port. We did most of the same tests to our apps after the port and we haven't seen any major differences so we just upgraded without much thought. > I can certainly remember an un-fun world of apache puking it's guts on a > coworker's machine as he was using the worker mpm.. > A lot of people stay away from threads like the plague, but it proved to be a better solution to this particular application. From that experience, we have used mod_worker in other apps with similar results, so I don't agree that it's a bad choice by any means. > So YMMV, depending on what portion of CPAN you use, and/or what your > codebase looks like... :-/ > Of course. But many things are in fact thread safe or can be used in a thread safe manner. > Indeed - it all very much depends on the application & load profile etc - so > all of this is just painting the bike shed, unless we're discussing a > specific application on a real (i.e. something like production) set of > hardware, and have actual performance metrics we want to ensure it fulfill. > :) > Ok. I gave you some metrics. From a TCO standpoint, don't you think that Catalyst, mod_perl, mpm_worker was a good choice? Homegrown solution may have been a better solution but this was a good compromise for us for the blocking problem. >> BTW, Ashley suggested I write a how-to on the WIki or something like >> that. Could some suggest exactly where, and I may have time to that >> this week. > > http://dev.catalyst.perl.org/wiki/deployment > > Making a section linked from 'Apache' in there, briefly outlining your > config and what benefits you get from doing mod_perl this way would be great > :) > Ok. Will do. Thanks for you ample insight into this matter. For about a year I had yet to find someone I could discuss this with! Best, Alejandro BTW are you (or any others here) by any chance going to OSCON 09? If any are, we could maybe meet there and have a Catalyst get-together there! > Cheers > t0m > > > _______________________________________________ > List: Catalyst@lists.scsys.co.uk > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > Dev site: http://dev.catalyst.perl.org/ > _______________________________________________ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/