Re: mod_perl vs. C for high performance Apache modules
... work on Mars. The investor claims to have evaluated Perl vs. C years ago, to have witnessed that every single hit on the webserver under mod_perl causes a CPU usage spike that isn't seen with C, and that under heavy load ]- this seems to me like non-compiled script, module or eventually running under Apache::PerlRun :))
RE: mod_perl vs. C for high performance Apache modules
On Fri, 2001-12-14 at 14:27, Thomas Moore wrote: I spoke to the technical lead at Yahoo who said mod_perl will not scale as well as c++ when you get to their level of traffic, but for a large Isn't that coming from a company using Python? I see that most of their URLs include a tell-tale .py. Of course, whether that's really Python or not Matt
Re: mod_perl vs. C for high performance Apache modules
On Mon, Dec 17, 2001 at 10:32:58AM -0600, Matthew Kennedy wrote: On Fri, 2001-12-14 at 14:27, Thomas Moore wrote: I spoke to the technical lead at Yahoo who said mod_perl will not scale as well as c++ when you get to their level of traffic, but for a large Isn't that coming from a company using Python? I see that most of their URLs include a tell-tale .py. Of course, whether that's really Python or not searching join.yahoo.com I notice 12 job descriptions contain the word perl, and only 2 that contain the word python.. FWIW, the only place I notice the .py extension is maps.yahoo.com. -- Paul Lindner [EMAIL PROTECTED]| | | | | | | | | | mod_perl Developer's Cookbook http://www.modperlcookbook.org Human Rights Declaration http://www.unhchr.ch/udhr/index.htm
Re: mod_perl vs. C for high performance Apache modules
On Fri, 14 Dec 2001, Jeff Yoak wrote: All, I wasn't sure what volume of response to expect when I originally wrote. Thank you all for the comments that you all are making. They are helping. Given that the response is fairly high, I'm waiting for stuff to roll in rather than replying to each of you. Don't think it is falling on unappreciating ears. :-) To respond to a few recurring comments / questions: Me? I've spent most of the last four years working on mod_perl-based stuff and most of the last eight working with Perl. Actually I've worked with folks who were involved with some of the projects you've mentioned, having been at idealab!, a parent of eToys and CitySearch. One of the original (THE original?) developer at CitySearch was probably the most helpful mentor / teacher I've ever worked with. I programmed in C a lot early in my career, but at this point I couldn't write anything substantial without brushing up, and frankly wouldn't care to. It just isn't as fun to work with C. But then, the argument, But if you used C, you wouldn't get to work with ME! may not convince some of these people with their values all screwed up... ;-) Actually that would be my argument. When you're getting investors in, the primary thing they should be looking to buy into is the quality of the people there, not necessarily the code, because only one out of those two can be fixed easily (even in our current times, totally replacing a programming team is a difficult thing to do). I write C. I write Perl. And I combine them both to good effect. But, I wouldn't even consider writing anything but time critical routines in C - I do as much as possible in Perl for the following reasons: - It's fast enough. - It's safer. - It has a built in test harness (Test::Harness). - It's more fun. - It's faster to develop in. - It's OO, and that saves me time and effort. - It has an infinitely better community than C. The last point is probably my favourite, though probably means bugger all to an investor. -- !-- Matt -- :-Get a smart net/:-
Re: mod_perl vs. C for high performance Apache modules
While this advocacy thread is hot, please remember my request to send me your success stories so we have more material others can use to prove their point to their investors, bosses, girlfriends, moms :) I've received only three new stories since my request (I didn't put them online yet, they will go into the new site). If you do send them to me (or list), please use a plain text and follow this format: URL: Title: Contact Person: Traffic: (hits/day/month/whatever) Success Story: and anything else you wish to tell. Check if your story is already online and send an update if there is something to change. See http://perl.apache.org/stories/ and http://perl.apache.org/sites.html. Thanks! _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://ticketmaster.com http://apacheweek.com http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: mod_perl vs. C for high performance Apache modules
I know this sounds kind of simple minded but why not bench test the site, set everything up in the office get a good switch plug the site into 1 port and 5-10 client boxes with some load testing software and plug it in to the same switch and beat the crap out of it. After you do this for a while and find all the hot spots show it to the customer and go here it works. marc - Original Message - From: Matt Sergeant [EMAIL PROTECTED] To: Jeff Yoak [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Saturday, December 15, 2001 3:24 AM Subject: Re: mod_perl vs. C for high performance Apache modules On Fri, 14 Dec 2001, Jeff Yoak wrote: All, I wasn't sure what volume of response to expect when I originally wrote. Thank you all for the comments that you all are making. They are helping. Given that the response is fairly high, I'm waiting for stuff to roll in rather than replying to each of you. Don't think it is falling on unappreciating ears. :-) To respond to a few recurring comments / questions: Me? I've spent most of the last four years working on mod_perl-based stuff and most of the last eight working with Perl. Actually I've worked with folks who were involved with some of the projects you've mentioned, having been at idealab!, a parent of eToys and CitySearch. One of the original (THE original?) developer at CitySearch was probably the most helpful mentor / teacher I've ever worked with. I programmed in C a lot early in my career, but at this point I couldn't write anything substantial without brushing up, and frankly wouldn't care to. It just isn't as fun to work with C. But then, the argument, But if you used C, you wouldn't get to work with ME! may not convince some of these people with their values all screwed up... ;-) Actually that would be my argument. When you're getting investors in, the primary thing they should be looking to buy into is the quality of the people there, not necessarily the code, because only one out of those two can be fixed easily (even in our current times, totally replacing a programming team is a difficult thing to do). I write C. I write Perl. And I combine them both to good effect. But, I wouldn't even consider writing anything but time critical routines in C - I do as much as possible in Perl for the following reasons: - It's fast enough. - It's safer. - It has a built in test harness (Test::Harness). - It's more fun. - It's faster to develop in. - It's OO, and that saves me time and effort. - It has an infinitely better community than C. The last point is probably my favourite, though probably means bugger all to an investor. -- !-- Matt -- :-Get a smart net/:-
RE: mod_perl vs. C for high performance Apache modules
http://www.perl.com/pub/a/2001/10/17/etoys.html Yea, mod_perl really sucks ; ) I have even worked on poorly architectured and coded sites which still performed fairly well. Recently I did a substantial project for a client in using mod_perl. That client is happy with the work, but an investor with their company is very angry because of what a horrible choice mod_perl is for high-load web applications compared with Apache modules and even CGI programs, written in C. If anyone on this list could forward any resources that do comparisons along these lines, or even analysis of mod_perl's handling of high-load web traffic, I would be very grateful.
Re: mod_perl vs. C for high performance Apache modules
On Fri, Dec 14, 2001 at 12:12:09PM -0800, Jeff Yoak wrote: Hi All, Recently I did a substantial project for a client in using mod_perl. That client is happy with the work, but an investor with their company is very angry because of what a horrible choice mod_perl is for high-load web applications compared with Apache modules and even CGI programs, written in C. If anyone on this list could forward any resources that do comparisons along these lines, or even analysis of mod_perl's handling of high-load web traffic, I would be very grateful. CGI programs (even written in C) are very likely to be a lot slower than mod_perl handlers so I can't see where he gets that argument from. Does he really know what mod_perl is? The key to mod_perl development is speed, there are numerous testimonials from users implementing a lot of work in a very short time with mod_perl. Ask the clients investor wheter he wants to pay for having everything you did rewritten as an Apache module in C. That is very likely going to take a lot of time. Take a look at Joshua Chamas benchmarks (Although they're only hello world style apps) it shows that mod_perl is pretty fast. my $cent = 2; -- Thomas Eibner http://thomas.eibner.dk/ DnsZone http://dnszone.org/ mod_pointer http://stderr.net/mod_pointer
Re: mod_perl vs. C for high performance Apache modules
Investors suck like that. I have had to fight many of these battles. The first thing to do is find out specifically _why_ the investor thinks that so you can counter their claims. Trying to counter vague notions of 'terrible' is impossible. The opponent has to commit to an opinion before you can fight it. Then just point to many of the well-known sites on the success pages that are high volume and do quite well with mod_perl Explain that to do this project in Java or C or CGI would have costed more, taken longer and given negligible benefits. Thats what I do. I usually also show how my architecture can interact with other architectures via XML or directly at teh data store level. Good luck J On Friday, December 14, 2001, at 12:12 PM, Jeff Yoak wrote: Hi All, Recently I did a substantial project for a client in using mod_perl. That client is happy with the work, but an investor with their company is very angry because of what a horrible choice mod_perl is for high-load web applications compared with Apache modules and even CGI programs, written in C. If anyone on this list could forward any resources that do comparisons along these lines, or even analysis of mod_perl's handling of high-load web traffic, I would be very grateful. Cheers, Jeff -- Jeff Yoak 626-705-6996
Re: mod_perl vs. C for high performance Apache modules
Jeff Yoak [EMAIL PROTECTED] writes: Hi All, Recently I did a substantial project for a client in using mod_perl. That client is happy with the work, but an investor with their company is very angry because of what a horrible choice mod_perl is for high-load web applications compared with Apache modules and even CGI programs, written in C. If anyone on this list could forward any resources that do comparisons along these lines, or even analysis of mod_perl's handling of high-load web traffic, I would be very grateful. Kill the investors. Really. -- David Hodgkinson, Wizard for Hirehttp://www.davehodgkinson.com Editor-in-chief, The Highway Star http://www.deep-purple.com Deep Purple Family Tree news http://www.slashrock.com Interim Technical Director, Web Architecture Consultant for hire
RE: mod_perl vs. C for high performance Apache modules
I spoke to the technical lead at Yahoo who said mod_perl will not scale as well as c++ when you get to their level of traffic, but for a large ecommerce site mod_perl is fine. I think people just stick to what they started with. We wrote our site racesearch.com using mod_perl and it is super fast. (faster than Amazon.com) -tom -Original Message- From: Jeff Yoak [mailto:[EMAIL PROTECTED]] Sent: Friday, December 14, 2001 12:12 PM To: [EMAIL PROTECTED] Subject: mod_perl vs. C for high performance Apache modules Hi All, Recently I did a substantial project for a client in using mod_perl. That client is happy with the work, but an investor with their company is very angry because of what a horrible choice mod_perl is for high-load web applications compared with Apache modules and even CGI programs, written in C. If anyone on this list could forward any resources that do comparisons along these lines, or even analysis of mod_perl's handling of high-load web traffic, I would be very grateful. Cheers, Jeff -- Jeff Yoak 626-705-6996
RE: mod_perl vs. C for high performance Apache modules
I spoke to the technical lead at Yahoo who said mod_perl will not scale as well as c++ when you get to their level of traffic, but for a large ecommerce site mod_perl is fine. Scalability has less to do with language/execution environment than which database you are using. Path length is affected by language, but that's usually not the major factor in scalability. You want short path lengths to get more efficiency out of your machines. Rob
Re: mod_perl vs. C for high performance Apache modules
I spoke to the technical lead at Yahoo who said mod_perl will not scale as well as c++ when you get to their level of traffic, but for a large ecommerce site mod_perl is fine. According to something I once read by David Filo, Yahoo also had to tweak the FreeBSD code because they had trouble scaling *TCP/IP*! I would say their experience is not typical. - Perrin
Re: mod_perl vs. C for high performance Apache modules
Perrin Harkins [EMAIL PROTECTED] writes: I spoke to the technical lead at Yahoo who said mod_perl will not scale as well as c++ when you get to their level of traffic, but for a large ecommerce site mod_perl is fine. According to something I once read by David Filo, Yahoo also had to tweak the FreeBSD code because they had trouble scaling *TCP/IP*! I would say their experience is not typical. Increasing the number of file handles, I'd wager. That was an issue on 2.x linux kernels too. -- David Hodgkinson, Wizard for Hirehttp://www.davehodgkinson.com Editor-in-chief, The Highway Star http://www.deep-purple.com Deep Purple Family Tree news http://www.slashrock.com Interim Technical Director, Web Architecture Consultant for hire
Re: mod_perl vs. C for high performance Apache modules
At 09:15 PM 12/14/2001 +0100, Thomas Eibner wrote: The key to mod_perl development is speed, there are numerous testimonials from users implementing a lot of work in a very short time with mod_perl. Ask the clients investor wheter he wants to pay for having everything you did rewritten as an Apache module in C. That is very likely going to take a lot of time. Thank you for your reply. I realized in reading it that my tone leads one to the common image of a buzzword driven doody-head who wants this because of what he read in Byte. That's certainly common enough, and I've never had a problem dealing with such types. (Well... not an unsolvable problem... :-) This is something different. The investor is in a related business, and has developed substantially similar software for years. And it is really good. What's worse is that my normal, biggest argument isn't compelling in this case, that by the time this would be done in C, I'd be doing contract work on Mars. The investor claims to have evaluated Perl vs. C years ago, to have witnessed that every single hit on the webserver under mod_perl causes a CPU usage spike that isn't seen with C, and that under heavy load mod_perl completely falls apart where C doesn't. (This code is, of course, LONG gone so I can't evaluate it for whether the C was good and the Perl was screwy.) At any rate, because of this, he's spent years having good stuff written in C. Unbeknownst to either me or my client, both this software and its developer were available to us, so in this case it would have been faster, cheaper and honestly even better, by which I mean more fully-featured. So he's upset. Everyone acknowledges that given our particular circumstances, it would have been better to build upon what we already had, but because of his previous experience he feels that mod_perl wasn't even a responsible choice even within the limits of our lack of knowledge of his software and its availability. So I'm trying to show that mod_perl doesn't suck, and that it is, in fact, a reasonable choice. Though within these limits it is still reasonable to point out the development cycle, emotionally it is the least compelling form of argument, because the investor has a hard time removing from consideration that given our particular situation, there was a very fast solution in using his C-based routines. Take a look at Joshua Chamas benchmarks (Although they're only hello world style apps) it shows that mod_perl is pretty fast. I will look for this particularly. Thanks. Cheers, Jeff -- Jeff Yoak 626-705-6996
Re: mod_perl vs. C for high performance Apache modules
At 03:12 PM 12/14/01, Jeff Yoak wrote: Recently I did a substantial project for a client in using mod_perl. That client is happy with the work, but an investor with their company is very angry because of what a horrible choice mod_perl is for high-load web applications compared with Apache modules and even CGI programs, written in C. If anyone on this list could forward any resources that do comparisons along these lines, or even analysis of mod_perl's handling of high-load web traffic, I would be very grateful. How about the myths page? http://perl.apache.org/perl_myth.html Todd
RE: mod_perl vs. C for high performance Apache modules
On Fri, 14 Dec 2001, Thomas Moore wrote: I spoke to the technical lead at Yahoo who said mod_perl will not scale as well as c++ when you get to their level of traffic, but for a large ecommerce site mod_perl is fine. Well, Yahoo is _extremely_ atypical. And they do a lot of stuff that involves custom coded Apache modules (in C, I think, not C++, though maybe both). Amazon reportedly uses a similar approach. Jeremy Zawodny is a frequent contributor on the MySQL list and a Perl guy so if you're really curious you could probably email him a polite question for more details. He's a nice guy. -dave /*== www.urth.org We await the New Sun ==*/
Re: mod_perl vs. C for high performance Apache modules
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, Dec 14, 2001 at 12:12:09PM -0800, Jeff Yoak wrote: Recently I did a substantial project for a client in using mod_perl. That client is happy with the work, but an investor with their company is very angry because of what a horrible choice mod_perl is for high-load web applications compared with Apache modules and even CGI programs, written in C. Word of the day: ultracrepidarianism Of course, financial investors are experts in the field of backend systems. One in this situation wishes your client would pick up on that. The fact that he's comparing mod_perl (an Apache module) to an Apache module should be a glaring sign of said investor's utter cluelessness. The fact that he's even mentioning CGI versus a server API-level module -- well, I'll stop now before I embarass someone. If anyone on this list could forward any resources that do comparisons along these lines, or even analysis of mod_perl's handling of high-load web traffic, I would be very grateful. Whether C is more efficient than a RAD language like Perl (or Python or PHP) is an incredibly stupid question, because one with sufficient knowledge should know already that the answer is yes. The real question is whether said efficiency is worth the arduous hassle of building the app in C. Considering the speed at which modern servers execute interpreted or JIT-compiled languages, that answer is almost always no. There is fast, but then there's fast enough (and finished developing in 25% of the time). They'll have to decide what's more important. Now, as for case studies, here's a quick list: http://perl.apache.org/sites.html And then this article, outlining how mod_perl was used to build the eToys site: http://www.perl.com/pub/a/2001/10/17/etoys.html And now for a completely shameless plug: http://www.iqcoordinator.com/ Which is a processing-intensive work order management system, built entirely on mod_perl. (Some day I will write an article myself :) Our systems don't even break a sweat. Actually, mod_perl saved us from having to buy more hardware. It's plenty fast. - -- Stephen Clouse [EMAIL PROTECTED] Senior Programmer, IQ Coordinator Project Lead The IQ Group, Inc. http://www.theiqgroup.com/ -BEGIN PGP SIGNATURE- Version: PGP 6.5.8 iQA/AwUBPBpphQOGqGs0PadnEQLW2gCdGonCCAU0aCauPTTsIZMvzcWU1mIAnj3J BBNeCw2RdSN90qTCYDFLUYPP =Exa7 -END PGP SIGNATURE-
Re: mod_perl vs. C for high performance Apache modules
I dont think its your responsibility anymore. If the investor had a preference he should have stated it BEFORE work began. If your client did not keep him informed then your client has that burden to bear. You did your job, the client likes what you did, it works. Let them fight the political battle. Comparing mod_perl 'years ago' is totally irrelevant. Modperl has grown up so much since 'years ago' that its not even funny. J- On Friday, December 14, 2001, at 12:58 PM, Jeff Yoak wrote: At 09:15 PM 12/14/2001 +0100, Thomas Eibner wrote: The key to mod_perl development is speed, there are numerous testimonials from users implementing a lot of work in a very short time with mod_perl. Ask the clients investor wheter he wants to pay for having everything you did rewritten as an Apache module in C. That is very likely going to take a lot of time. Thank you for your reply. I realized in reading it that my tone leads one to the common image of a buzzword driven doody-head who wants this because of what he read in Byte. That's certainly common enough, and I've never had a problem dealing with such types. (Well... not an unsolvable problem... :-) This is something different. The investor is in a related business, and has developed substantially similar software for years. And it is really good. What's worse is that my normal, biggest argument isn't compelling in this case, that by the time this would be done in C, I'd be doing contract work on Mars. The investor claims to have evaluated Perl vs. C years ago, to have witnessed that every single hit on the webserver under mod_perl causes a CPU usage spike that isn't seen with C, and that under heavy load mod_perl completely falls apart where C doesn't. (This code is, of course, LONG gone so I can't evaluate it for whether the C was good and the Perl was screwy.) At any rate, because of this, he's spent years having good stuff written in C. Unbeknownst to either me or my client, both this software and its developer were available to us, so in this case it would have been faster, cheaper and honestly even better, by which I mean more fully-featured. So he's upset. Everyone acknowledges that given our particular circumstances, it would have been better to build upon what we already had, but because of his previous experience he feels that mod_perl wasn't even a responsible choice even within the limits of our lack of knowledge of his software and its availability. So I'm trying to show that mod_perl doesn't suck, and that it is, in fact, a reasonable choice. Though within these limits it is still reasonable to point out the development cycle, emotionally it is the least compelling form of argument, because the investor has a hard time removing from consideration that given our particular situation, there was a very fast solution in using his C-based routines. Take a look at Joshua Chamas benchmarks (Although they're only hello world style apps) it shows that mod_perl is pretty fast. I will look for this particularly. Thanks. Cheers, Jeff -- Jeff Yoak 626-705-6996
RE: mod_perl vs. C for high performance Apache modules
Mod_perl doesn't suck, and it certainly doesn't have a huge hit on the CPU. (of course it all depends what you're doing, but for the most part it's small) Having used many high level web development environments, from C to Java to TCL and perl, I find mod_perl at the top end of the scalability scale. Plenty of fast and heavily loaded sites use mod_perl without throwing tonnes of hardware at it, AND as a side bonus, chances are the development time was faster than writing in almost anyything else. I run a wee little web site, it runs on 4 servers (1 static server, 2 mysql servers, and one mod_perl server) all told, this hardware cost about $10k usd. Yesterday, the site served 2,000,000 page loads (ads, impressions, how every you count it), with about 6 Million requests to the mod_perl server, which in turn passed on about 200 Million requests to the DB server. Yes, I struggle with performance, but under most other development environments I'd have been up against the wall a long time ago. Mod_perl is NOT CPU intensive. It IS memory intensive. Due to the architecture of apache, and the memory overhead some care must be taken when planning a large capacity site, but fortunately all of this is nicely documented in the guide. There are other solutions. Yes, I suppose I could rewrite large parts of the site in C. It certainly wouldn't help with development (which is always patching this, adding feature Z), it might, maybe reduce some of the memory overhead. My largest challenge has actually been to get mysql (written in C) to properly handle it's share of the workload. Take a perusal at the Stories section on perl.apache.org http://perl.apache.org/stories/ ValueClick serves an incredible number of impressions via their mod_perl systems. It scales, it's fast, and it doesn't take a lot of hardware. --A -Original Message- From: Jeff Yoak [mailto:[EMAIL PROTECTED]] This is something different. The investor is in a related business, and has developed substantially similar software for years. And it is really good. What's worse is that my normal, biggest argument isn't compelling in this case, that by the time this would be done in C, I'd be doing contract work on Mars. The investor claims to have evaluated Perl vs. C years ago, to have witnessed that every single hit on the webserver under mod_perl causes a CPU usage spike that isn't seen with C, and that under heavy load mod_perl completely falls apart where C doesn't. (This code is, of course, LONG gone so I can't evaluate it for whether the C was good and the Perl was screwy.) At any rate, because of this, he's spent years having good stuff written in C. Unbeknownst to either me or my client, both this software and its developer were available to us, so in this case it would have been faster, cheaper and honestly even better, by which I mean more fully-featured. So he's upset. Everyone acknowledges that given our particular circumstances, it would have been better to build upon what we already had, but because of his previous experience he feels that mod_perl wasn't even a responsible choice even within the limits of our lack of knowledge of his software and its availability. So I'm trying to show that mod_perl doesn't suck, and that it is, in fact, a reasonable choice. Though within these limits it is still reasonable to point out the development cycle, emotionally it is the least compelling form of argument, because the investor has a hard time removing from consideration that given our particular situation, there was a very fast solution in using his C-based routines.
Re: mod_perl vs. C for high performance Apache modules
On Fri, Dec 14, 2001 at 12:58:51PM -0800, Jeff Yoak wrote: At 09:15 PM 12/14/2001 +0100, Thomas Eibner wrote: The key to mod_perl development is speed, there are numerous testimonials from users implementing a lot of work in a very short time with mod_perl. Ask the clients investor wheter he wants to pay for having everything you did rewritten as an Apache module in C. That is very likely going to take a lot of time. Thank you for your reply. I realized in reading it that my tone leads one to the common image of a buzzword driven doody-head who wants this because of what he read in Byte. That's certainly common enough, and I've never had a problem dealing with such types. (Well... not an unsolvable problem... :-) True, it sounded a bit like that. This is something different. The investor is in a related business, and has developed substantially similar software for years. And it is really good. What's worse is that my normal, biggest argument isn't compelling in this case, that by the time this would be done in C, I'd be doing contract work on Mars. The investor claims to have evaluated Perl vs. C years ago, to have witnessed that every single hit on the webserver under mod_perl causes a CPU usage spike that isn't seen with C, and that under heavy load mod_perl completely falls apart where C doesn't. (This code is, of course, LONG gone so I can't evaluate it for whether the C was good and the Perl was screwy.) At any rate, because of this, he's spent years having good stuff written in C. Unbeknownst to either me or my client, both this software and its developer were available to us, so in this case it would have been faster, cheaper and honestly even better, by which I mean more fully-featured. I'm wondering where the investor have been hiding if he isn't telling you this until now. It's hard to pull on resources that you have no idea are available to you. If he did the comparision years ago he should be doing his tests over again, since a lot of things have happened over the last few years. From what you write it seems to me that he can't just be using plain CGI in C, since it still has the fork overhead that mod_perl doesn't and even a small hello world test would prove that mod_perl is faster there (I didn't try it, but I'm sure we could make some tests). Although you haven't told us much about what the actual application you wrote for your client does, I must say that I've found it more error prone and time-consuming doing any database stuff in C. I'm not a C programmer of heart, but I have done a fair amount of programming with it and it certainly hasn't been as fast to develop as applications relying on DBI. You haven't told us what you normally program in either, so we don't know if you do mod_perl most of the time or you just write your stuff in C regulary, this could also have an impact on the length of the project in the end. So he's upset. Everyone acknowledges that given our particular circumstances, it would have been better to build upon what we already had, but because of his previous experience he feels that mod_perl wasn't even a responsible choice even within the limits of our lack of knowledge of his software and its availability. To me it seems like he should be upset with himself. If he knows he has some good tools he should have told your client way before this project even started anyway. So I'm trying to show that mod_perl doesn't suck, and that it is, in fact, a reasonable choice. Though within these limits it is still reasonable to point out the development cycle, emotionally it is the least compelling form of argument, because the investor has a hard time removing from consideration that given our particular situation, there was a very fast solution in using his C-based routines. I certainly hope that you find something that will show him that mod_perl isn't a bad choice. -- Thomas Eibner http://thomas.eibner.dk/ DnsZone http://dnszone.org/ mod_pointer http://stderr.net/mod_pointer
Re: mod_perl vs. C for high performance Apache modules
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, Dec 14, 2001 at 12:58:51PM -0800, Jeff Yoak wrote: This is something different. The investor is in a related business, and has developed substantially similar software for years. And it is really good. What's worse is that my normal, biggest argument isn't compelling in this case, that by the time this would be done in C, I'd be doing contract work on Mars. The investor claims to have evaluated Perl vs. C years ago, to have witnessed that every single hit on the webserver under mod_perl causes a CPU usage spike that isn't seen with C, and that under heavy load mod_perl completely falls apart where C doesn't. (This code is, of course, LONG gone so I can't evaluate it for whether the C was good and the Perl was screwy.) At any rate, because of this, he's spent years having good stuff written in C. Unbeknownst to either me or my client, both this software and its developer were available to us, so in this case it would have been faster, cheaper and honestly even better, by which I mean more fully-featured. Ah, the good old my timeless knowledge from the Nixon administration still applies argument. Remember that the speed of both languages has increased geometrically as a function of hardware speed. 5 years ago you may have been talking a difference of a second or two between Perl and C. Nowadays you'll be measuring the differences in milliseconds, if that much. And you're right, one still can't verify the veracity of the ancient benchmarked code. On another list someone was dealing with Perl and DBI against PHP for a database-driven site, with nothing to go on but some ApacheBench numbers showing the PHP page about six orders of magnitude faster. Then one minor difference of note: the Perl program had output about 70MB of data; the PHP program, about 940 bytes. I assume that the PHP program was doing a null loop or something where the query should have been. Get him to do a real unbiased comparison on modern hardware, and then defy him to claim that the milliseconds he saves are worth the effort. - -- Stephen Clouse [EMAIL PROTECTED] Senior Programmer, IQ Coordinator Project Lead The IQ Group, Inc. http://www.theiqgroup.com/ -BEGIN PGP SIGNATURE- Version: PGP 6.5.8 iQA/AwUBPBpsFQOGqGs0PadnEQJ9TgCgtZnIhFLq/L/DeZA5CS4yAjzBRiEAn3ED lTRl2/yaG9eH7BK785GKajp3 =jLX5 -END PGP SIGNATURE-
Re: mod_perl vs. C for high performance Apache modules
Hi there, On Fri, 14 Dec 2001, Jeff Yoak wrote: This is something different. [big snip] Indeed it is. It's a refreshingly honest appraisal of what might, in hindsight, have been easily avoided mistakes. And nobody ever did anything without making a few. Thanks. 73, Ged. PS: Are any of those C routines available Open Source? :)
Re: mod_perl vs. C for high performance Apache modules
At 03:58 PM 12/14/2001, Jeff Yoak wrote: At 09:15 PM 12/14/2001 +0100, Thomas Eibner wrote: The key to mod_perl development is speed, there are numerous testimonials from users implementing a lot of work in a very short time with mod_perl. Ask the clients investor wheter he wants to pay for having everything you did rewritten as an Apache module in C. That is very likely going to take a lot of time. Thank you for your reply. I realized in reading it that my tone leads one to the common image of a buzzword driven doody-head who wants this because of what he read in Byte. That's certainly common enough, and I've never had a problem dealing with such types. (Well... not an unsolvable problem... :-) This is something different. The investor is in a related business, and has developed substantially similar software for years. And it is really good. What's worse is that my normal, biggest argument isn't compelling in this case, that by the time this would be done in C, I'd be doing contract work on Mars. The investor claims to have evaluated Perl vs. C years ago, to have witnessed that every single hit on the webserver under mod_perl causes a CPU usage spike that isn't seen with C, and that under heavy load mod_perl completely falls apart where C doesn't. (This code is, of course, LONG gone so I can't evaluate it for whether the C was good and the Perl was screwy.) At any rate, because of this, he's spent years having good stuff written in C. Unbeknownst to either me or my client, both this software and its developer were available to us, so in this case it would have been faster, cheaper and honestly even better, by which I mean more fully-featured. CPU usage is certainly one factor... but CPUs are cheap compared to development man-hours. Since you haven't provided any details on the application, this may not be relevant, but most of the web apps that we write (and I read about here) spend much of their time waiting for responses from other back-end servers - databases, NFS mounted file systems, or whatever. It's probably undeniable that a well written C application will run faster than almost anything in an interpreted language, but that may not make much of a difference to the total response time. -Simon Simon Rosenthal ([EMAIL PROTECTED]) Web Systems Architect Northern Light Technology One Athenaeum Street. Suite 1700, Cambridge, MA 02142 Phone: (617)621-5296: URL: http://www.northernlight.com Northern Light - Just what you've been searching for
Re: mod_perl vs. C for high performance Apache modules
So I'm trying to show that mod_perl doesn't suck, and that it is, in fact, a reasonable choice. Though within these limits it is still reasonable to point out the development cycle, emotionally it is the least compelling form of argument, because the investor has a hard time removing from consideration that given our particular situation, there was a very fast solution in using his C-based routines. Well, that is the primary reason for using Perl over C, and you really have to count maintenance and the relative likelihood of C-ish bugs like buffer overflows as part of it. Well-coded C should be faster than Perl, but Perl is fast enough for nearly any web-based application. If this guy saw CPU spikes, he probably had something else wrong, like running out of memory. You might find this article aboout C and Perl performance useful: http://www.perl.com/pub/a/2001/06/27/ctoperl.html - Perrin
Re: mod_perl vs. C for high performance Apache modules
... years ago ... Are you even sure he evaluated mod_perl and not Perl CGI scripts?? Launching the interpreter and compiling every time might spike the CPU. Like others have said, you would really have to benchmark the mod_perl and Apache that you're using now; both have improved considerably over the years. You can point to the success stories others have mentioned to show that it really is one of the standard industry solutions that works well in many high-traffic situations. He's clearly arguing from outdated data and emotion. Also, if you delivered what you said you would deliver, that ought to be the end of it. If the site is performing according to expectations and promises, then he doesn't have any real basis to complain. If it would truly deliver tangible advantages to the client to rewrite it in C, or incorporate some of his C code in your work (Perl and C can get along sometimes), then you might consider giving him a quote to rewrite it. Ask him to put up or shut up. A gentler approach might be, Given we did what we did (and it works), what do you really want us to do now? What should happen from this point forward? If there isn't a constructive answer to that, then all you're dealing with is a temper tantrum. ---Wes Sheldahl ( father of four young'uns ) Jeff Yoak [EMAIL PROTECTED] on 12/14/2001 03:58:51 PM To: Thomas Eibner [EMAIL PROTECTED] cc: [EMAIL PROTECTED] (bcc: Wesley Sheldahl/Lex/Lexmark) Subject: Re: mod_perl vs. C for high performance Apache modules [snip] So he's upset. Everyone acknowledges that given our particular circumstances, it would have been better to build upon what we already had, but because of his previous experience he feels that mod_perl wasn't even a responsible choice even within the limits of our lack of knowledge of his software and its availability. [snip] Cheers, Jeff
Re: mod_perl vs. C for high performance Apache modules
All, I wasn't sure what volume of response to expect when I originally wrote. Thank you all for the comments that you all are making. They are helping. Given that the response is fairly high, I'm waiting for stuff to roll in rather than replying to each of you. Don't think it is falling on unappreciating ears. :-) To respond to a few recurring comments / questions: Me? I've spent most of the last four years working on mod_perl-based stuff and most of the last eight working with Perl. Actually I've worked with folks who were involved with some of the projects you've mentioned, having been at idealab!, a parent of eToys and CitySearch. One of the original (THE original?) developer at CitySearch was probably the most helpful mentor / teacher I've ever worked with. I programmed in C a lot early in my career, but at this point I couldn't write anything substantial without brushing up, and frankly wouldn't care to. It just isn't as fun to work with C. But then, the argument, But if you used C, you wouldn't get to work with ME! may not convince some of these people with their values all screwed up... ;-) The project? What I consider to be standard web junk. About 30% ecommerce combined with lots of database-driven, interactive content, some authentication foo and things like that. The thing is that it is in the adult industry and the investor in question turns on the hose... well... there will be lots of traffic. Mistakes and misunderstandings? Sure. And yes, as some of you have pointed out publicly or privately, not my fault. I was kept very insulated from the people who were familiar with the alternatives. My involvement at this point is to try to smooth things over and keep the project functional. These immediate problems aside, the people involved are great to work with and if everyone feels better about the situation and things move forward in the best possible way from here, there will be a lot of valuable work for me. So I'm trying to help. More than you cared to hear about and terribly off-topic for the list? Sure. But you did sorta ask, and somehow it seemed rude to not reply. Forgive me and thanks for providing all of your commentary. Cheers, Jeff -- Jeff Yoak 626-705-6996
Re: mod_perl vs. C for high performance Apache modules
Dave Hodgkinson wrote on Fri, Dec 14 2001 (20:54:22 +): Perrin Harkins [EMAIL PROTECTED] writes: According to something I once read by David Filo, Yahoo also had to tweak the FreeBSD code because they had trouble scaling *TCP/IP*! I would say their experience is not typical. Increasing the number of file handles, I'd wager. That was an issue on you really don't have to tweak any FreeBSD code for that, just do sysctl -w kern.maxfiles=10 and the file table will grow up to the new limit. 2.x linux kernels too. that was an issue with 2.0.x, since 2.2.x you can do it with echo 10 /proc/sys/fs/file-max cheers, -- Toni Andjelkovic [EMAIL PROTECTED]
Re: mod_perl vs. C for high performance Apache modules
Toni Andjelkovic [EMAIL PROTECTED] writes: 2.x linux kernels too. that was an issue with 2.0.x, since 2.2.x you can do it with That was what I meant...decimal point in the wrong place... :-) -- David Hodgkinson, Wizard for Hirehttp://www.davehodgkinson.com Editor-in-chief, The Highway Star http://www.deep-purple.com Deep Purple Family Tree news http://www.slashrock.com Interim Technical Director, Web Architecture Consultant for hire
RE: mod_perl vs. C for high performance Apache modules
On Fri, 14 Dec 2001, Thomas Moore wrote: I spoke to the technical lead at Yahoo who said mod_perl will not scale as well as c++ when you get to their level of traffic, but for a large ecommerce site mod_perl is fine. the old memory is cheap rationalization doesn't go over very well at that scale either, eh.
Re: mod_perl vs. C for high performance Apache modules
-- Jeff Yoak [EMAIL PROTECTED] on 12/14/01 12:58:51 -0800 This is something different. The investor is in a related business, and has developed substantially similar software for years. And it is really good. What's worse is that my normal, biggest argument isn't compelling in this case, that by the time this would be done in C, I'd be doing contract work on Mars. The investor claims to have evaluated Perl vs. C years ago, to have witnessed that every single hit on the webserver under mod_perl causes a CPU usage spike that isn't seen with C, and that under heavy load mod_perl completely falls apart where C doesn't. (This code is, of course, LONG gone so I can't evaluate it for whether the C was good and the Perl was screwy.) At any rate, because of this, he's spent years having good stuff written in C. Unbeknownst to either me or my client, both this software and its developer were available to us, so in this case it would have been faster, cheaper and honestly even better, by which I mean more fully-featured. Constructing the $r object in perl-space is an overhead that mod_perl causes. This overhead has been managed more effectively in recent versions of perl/mod_perl. A study done a few years ago probably involved machines with significantly less core and CPU horsepower than the average kiddie-games PC does today. Net result is that any overhead caused by mod_perl in the previous study may well have been either mitigated with better code or obviated by faster hardware [how's that for a sentence?]. Net result is that the objection is probably based on once- valid but now out of date analysis. -- Steven Lembark 2930 W. Palmer Workhorse Computing Chicago, IL 60647 +1 800 762 1582
Re: mod_perl vs. C for high performance Apache modules
So I'm trying to show that mod_perl doesn't suck, and that it is, in fact, a reasonable choice. I think one of the selling points for mod_perl is its extensibility: modules can be written in C. Depending on the C code you have access to, a good solution might be to try to wrap it into mod_perl modules. Tim
Re: mod_perl vs. C for high performance Apache modules
The original poster talked about C++ CGI programs. I have been using mod_perl since 0.7x days and I can tell you there is no way a fork+exec CGI program no matter what language its written in will come anywhere close to a perl handler written against the mod_perl Apache API in execution speed (when they are doing equivalnet types of work). Using C++ to build web applications is something developers who grew up in the heyday of client server would think is a good idea. In the internet web applications business by the time you get a C++ program debugged and ready to roll the market has evolved and your software is out of date. C++ is a good language for systems programming, databases, etc., but web apps need shorter life cycles. I had an investor question similar to the one we are talking about 3 years ago. I was questioned as to why we used Apache, mod_perl, and mysql instead of C++ and Oracle's DB and Web Devel kit. Needless to say our mod_perl systems have thrived while most of the investor's other investments have had their expensive hardware auctioned off on Ebay recently. The essence of mod_perl is that it allows to to take an idea and build a working prototype very quickly. When you prove that the prototype works you don't need to rewrite - mod_perl scales up better than any other web application technology available - period. -jon On Fri, 14 Dec 2001 [EMAIL PROTECTED] wrote: -- Jeff Yoak [EMAIL PROTECTED] on 12/14/01 12:58:51 -0800 This is something different. The investor is in a related business, and has developed substantially similar software for years. And it is really good. What's worse is that my normal, biggest argument isn't compelling in this case, that by the time this would be done in C, I'd be doing contract work on Mars. The investor claims to have evaluated Perl vs. C years ago, to have witnessed that every single hit on the webserver under mod_perl causes a CPU usage spike that isn't seen with C, and that under heavy load mod_perl completely falls apart where C doesn't. (This code is, of course, LONG gone so I can't evaluate it for whether the C was good and the Perl was screwy.) At any rate, because of this, he's spent years having good stuff written in C. Unbeknownst to either me or my client, both this software and its developer were available to us, so in this case it would have been faster, cheaper and honestly even better, by which I mean more fully-featured. Constructing the $r object in perl-space is an overhead that mod_perl causes. This overhead has been managed more effectively in recent versions of perl/mod_perl. A study done a few years ago probably involved machines with significantly less core and CPU horsepower than the average kiddie-games PC does today. Net result is that any overhead caused by mod_perl in the previous study may well have been either mitigated with better code or obviated by faster hardware [how's that for a sentence?]. Net result is that the objection is probably based on once- valid but now out of date analysis. -- Steven Lembark 2930 W. Palmer Workhorse Computing Chicago, IL 60647 +1 800 762 1582 -- C. Jon Larsen Chief Technology Officer, Richweb.com (804.307.6939) SMTP: [EMAIL PROTECTED] (http://richweb.com/cjl_pgp_pub_key.txt) Richweb.com: Designing Open Source Internet Business Solutions since 1995 Building Safe, Secure, Reliable Cisco-Powered Networks since 1995
Re: mod_perl vs. C for high performance Apache modules
Flamebait For some really high performance sites, compiled C is the way to go. It's faster and as long as you remove all compilers from the machines in question, it's also more secure. /flamebait Having said that, I will also add that the downside is that in order to keep pace with your competitors who are writing code in Perl, PHP, and Python, it is going to cost you some serious dollars. You have to have deep pockets and in house coders to make it worth looking at, which normally limits this kind of thing to the Fortune 100. For the entire rest of the planet, perl is a perfectly good solution. IMHO, Jimi - Original Message - From: Perrin Harkins [EMAIL PROTECTED] To: Thomas Eibner [EMAIL PROTECTED]; Jeff Yoak [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Friday, December 14, 2001 3:32 PM Subject: Re: mod_perl vs. C for high performance Apache modules So I'm trying to show that mod_perl doesn't suck, and that it is, in fact, a reasonable choice. Though within these limits it is still reasonable to point out the development cycle, emotionally it is the least compelling form of argument, because the investor has a hard time removing from consideration that given our particular situation, there was a very fast solution in using his C-based routines. Well, that is the primary reason for using Perl over C, and you really have to count maintenance and the relative likelihood of C-ish bugs like buffer overflows as part of it. Well-coded C should be faster than Perl, but Perl is fast enough for nearly any web-based application. If this guy saw CPU spikes, he probably had something else wrong, like running out of memory. You might find this article aboout C and Perl performance useful: http://www.perl.com/pub/a/2001/06/27/ctoperl.html - Perrin
