Re: mod_perl vs. C for high performance Apache modules

[raptor]

...
> work on Mars.  The investor claims to have evaluated Perl vs. C years ago, 
> to have witnessed that every single hit on the webserver under mod_perl 
> causes a CPU usage spike that isn't seen with C, and that under heavy load 
]- this seems to me like non-compiled script, module or eventually
running under Apache::PerlRun :"))

Re: mod_perl vs. C for high performance Apache modules

[Paul Lindner]

On Mon, Dec 17, 2001 at 10:32:58AM -0600, Matthew Kennedy wrote:
> On Fri, 2001-12-14 at 14:27, Thomas Moore wrote:
> > I spoke to the technical lead at Yahoo who said mod_perl will not scale as
> > well as c++ when you get to their level of traffic, but for a large
> 
> Isn't that coming from a company using Python?
> 
> I see that most of their URLs include a tell-tale ".py". Of course,
> whether that's really Python or not

searching join.yahoo.com I notice 12 job descriptions contain the word
perl, and only 2 that contain the word python..

FWIW, the only place I notice the .py extension is maps.yahoo.com.

-- 
Paul Lindner   [EMAIL PROTECTED]| | | | |  |  |  |   |   |

mod_perl Developer's Cookbook   http://www.modperlcookbook.org
 Human Rights Declaration   http://www.unhchr.ch/udhr/index.htm

RE: mod_perl vs. C for high performance Apache modules

[Matthew Kennedy]

On Fri, 2001-12-14 at 14:27, Thomas Moore wrote:
> I spoke to the technical lead at Yahoo who said mod_perl will not scale as
> well as c++ when you get to their level of traffic, but for a large

Isn't that coming from a company using Python?

I see that most of their URLs include a tell-tale ".py". Of course,
whether that's really Python or not

Matt

Re: mod_perl vs. C for high performance Apache modules

[Marc Spitzer]

I know this sounds kind of simple minded but why not bench test the site,
set everything up in the office get a good switch plug the site into 1 port
and 5-10 client boxes with some load testing software and plug it in to the
same switch and beat the crap out of it.  After you do this for a while and
find all the hot spots show it to the customer and go here it works.

marc

- Original Message -
From: "Matt Sergeant" <[EMAIL PROTECTED]>
To: "Jeff Yoak" <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>
Sent: Saturday, December 15, 2001 3:24 AM
Subject: Re: mod_perl vs. C for high performance Apache modules


> On Fri, 14 Dec 2001, Jeff Yoak wrote:
>
> > All,
> >
> >  I wasn't sure what volume of response to expect when I
originally
> > wrote.  Thank you all for the comments that you all are making.  They
are
> > helping.  Given that the response is fairly high, I'm waiting for stuff
to
> > roll in rather than replying to each of you.  Don't think it is falling
on
> > unappreciating ears.  :-)
> >  To respond to a few recurring comments / questions:
> >
> > Me?  I've spent most of the last four years working on mod_perl-based
stuff
> > and most of the last eight working with Perl.  Actually I've worked with
> > folks who were involved with some of the projects you've mentioned,
having
> > been at idealab!, a parent of eToys and CitySearch.  One of the original
> > (THE original?) developer at CitySearch was probably the most helpful
> > mentor / teacher I've ever worked with.  I programmed in C a lot early
in
> > my career, but at this point I couldn't write anything substantial
without
> > brushing up, and frankly wouldn't care to.  It just isn't as fun to work
> > with C.  But then, the argument, "But if you used C, you wouldn't get to
> > work with ME!" may not convince some of these people with their values
all
> > screwed up...  ;-)
>
> Actually that would be my argument. When you're getting investors in, the
> primary thing they should be looking to buy into is the quality of the
> people there, not necessarily the code, because only one out of those two
> can be fixed easily (even in our current times, totally replacing a
> programming team is a difficult thing to do).
>
> I write C. I write Perl. And I combine them both to good effect. But, I
> wouldn't even consider writing anything but time critical routines in C -
> I do as much as possible in Perl for the following reasons:
>
>  - It's fast enough.
>  - It's safer.
>  - It has a built in test harness (Test::Harness).
>  - It's more fun.
>  - It's faster to develop in.
>  - It's OO, and that saves me time and effort.
>  - It has an infinitely better community than C.
>
> The last point is probably my favourite, though probably means bugger all
> to an investor.
>
> --
> 
> <:->Get a smart net
>

Re: mod_perl vs. C for high performance Apache modules

[Stas Bekman]

While this advocacy thread is hot, please remember my request to send me 
your success stories so we have more material others can use to prove 
their point to their investors, bosses, girlfriends, moms :)

I've received only three new stories since my request (I didn't put them 
online yet, they will go into the new site).

If you do send them to me (or list), please use a plain text and follow 
this format:

URL:
Title:
Contact Person:
Traffic: (hits/day/month/whatever)
Success Story:

and anything else you wish to tell.


Check if your story is already online and send an update if there is 
something to change. See http://perl.apache.org/stories/ and 
http://perl.apache.org/sites.html.

Thanks!

_
Stas Bekman JAm_pH  --   Just Another mod_perl Hacker
http://stason.org/  mod_perl Guide   http://perl.apache.org/guide
mailto:[EMAIL PROTECTED]  http://ticketmaster.com http://apacheweek.com
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/

Re: mod_perl vs. C for high performance Apache modules

[Matt Sergeant]

On Fri, 14 Dec 2001, Jeff Yoak wrote:

> All,
>
>  I wasn't sure what volume of response to expect when I originally
> wrote.  Thank you all for the comments that you all are making.  They are
> helping.  Given that the response is fairly high, I'm waiting for stuff to
> roll in rather than replying to each of you.  Don't think it is falling on
> unappreciating ears.  :-)
>  To respond to a few recurring comments / questions:
>
> Me?  I've spent most of the last four years working on mod_perl-based stuff
> and most of the last eight working with Perl.  Actually I've worked with
> folks who were involved with some of the projects you've mentioned, having
> been at idealab!, a parent of eToys and CitySearch.  One of the original
> (THE original?) developer at CitySearch was probably the most helpful
> mentor / teacher I've ever worked with.  I programmed in C a lot early in
> my career, but at this point I couldn't write anything substantial without
> brushing up, and frankly wouldn't care to.  It just isn't as fun to work
> with C.  But then, the argument, "But if you used C, you wouldn't get to
> work with ME!" may not convince some of these people with their values all
> screwed up...  ;-)

Actually that would be my argument. When you're getting investors in, the
primary thing they should be looking to buy into is the quality of the
people there, not necessarily the code, because only one out of those two
can be fixed easily (even in our current times, totally replacing a
programming team is a difficult thing to do).

I write C. I write Perl. And I combine them both to good effect. But, I
wouldn't even consider writing anything but time critical routines in C -
I do as much as possible in Perl for the following reasons:

 - It's fast enough.
 - It's safer.
 - It has a built in test harness (Test::Harness).
 - It's more fun.
 - It's faster to develop in.
 - It's OO, and that saves me time and effort.
 - It has an infinitely better community than C.

The last point is probably my favourite, though probably means bugger all
to an investor.

-- 

<:->Get a smart net

Re: mod_perl vs. C for high performance Apache modules

[Jimi Thompson]

For some really high performance sites, compiled C is the way to go.  It's
faster and as long as you remove all compilers from the machines in
question, it's also more secure.

Having said that, I will also add that the downside is that in order to keep
pace with your competitors who are writing code in Perl, PHP, and Python, it
is going to cost you some serious dollars.  You have to have deep pockets
and in house coders to make it worth looking at, which normally limits this
kind of thing to the Fortune 100.

For the entire rest of the planet, perl is a perfectly good solution.

IMHO,

Jimi
- Original Message -
From: "Perrin Harkins" <[EMAIL PROTECTED]>
To: "Thomas Eibner" <[EMAIL PROTECTED]>; "Jeff Yoak" <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>
Sent: Friday, December 14, 2001 3:32 PM
Subject: Re: mod_perl vs. C for high performance Apache modules


> > So I'm trying to show that mod_perl doesn't suck, and that it is, in
fact,
> > a reasonable choice.  Though within these limits it is still reasonable
to
> > point out the development cycle, emotionally it is the least compelling
> > form of argument, because the investor has a hard time removing from
> > consideration that given our particular situation, there was a very fast
> > solution in using his C-based routines.
>
> Well, that is the primary reason for using Perl over C, and you really
have
> to count maintenance and the relative likelihood of C-ish bugs like buffer
> overflows as part of it.
>
> Well-coded C should be faster than Perl, but Perl is fast enough for
nearly
> any web-based application.  If this guy saw CPU spikes, he probably had
> something else wrong, like running out of memory.
>
> You might find this article aboout C and Perl performance useful:
> http://www.perl.com/pub/a/2001/06/27/ctoperl.html
>
> - Perrin
>

Re: mod_perl vs. C for high performance Apache modules

[C. Jon Larsen]

The original poster talked about C++ CGI programs. I have been using
mod_perl since 0.7x days and I can tell you there is no way a fork+exec
CGI program no matter what language its written in will come anywhere
close to a perl handler written against the mod_perl Apache API in
execution speed (when they are doing equivalnet types of work). Using C++
to build web applications is something developers who grew up in the
heyday of client server would think is a good idea. In the internet web
applications business by the time you get a C++ program debugged and ready
to roll the market has evolved and your software is out of date.  C++ is a
good language for systems programming, databases, etc., but web apps need
shorter life cycles.

I had an investor question similar to the one we are talking about 3 years
ago. I was questioned as to why we used Apache, mod_perl, and mysql
instead of C++ and Oracle's DB and Web Devel kit. Needless to say our
mod_perl systems have thrived while most of the investor's other
investments have had their expensive hardware auctioned off on Ebay
recently.

The essence of mod_perl is that it allows to to take an idea and build a
working prototype very quickly. When you prove that the prototype works
you don't need to rewrite - mod_perl scales up better than any other web
application technology available - period.

-jon

On Fri, 14 Dec 2001 [EMAIL PROTECTED] wrote:

>
>
> -- Jeff Yoak <[EMAIL PROTECTED]> on 12/14/01 12:58:51 -0800
>
> > This is something different.  The investor is in a related business, and has
> > developed substantially similar software for years.  And it is really good.
> > What's worse is that my normal, biggest argument isn't compelling in this
> > case, that by the time this would be done in C, I'd be doing contract work on
> > Mars.  The investor claims to have evaluated Perl vs. C years ago, to have
> > witnessed that every single hit on the webserver under mod_perl causes a CPU
> > usage spike that isn't seen with C, and that under heavy load mod_perl
> > completely falls apart where C doesn't.  (This code is, of course, LONG gone
> > so I can't evaluate it for whether the C was good and the Perl was screwy.)
> > At any rate, because of this, he's spent years having good stuff written in
> > C.  Unbeknownst to either me or my client, both this software and its
> > developer were available to us, so in this case it would have been faster,
> > cheaper and honestly even better, by which I mean more fully-featured.
>
> Constructing the $r object in perl-space is an overhead
> that mod_perl causes. This overhead has been managed more
> effectively in recent versions of perl/mod_perl. A study
> done "a few years ago" probably involved machines with
> significantly less core and CPU horsepower than the average
> kiddie-games PC does today. Net result is that any overhead
> caused by mod_perl in the previous study may well have been
> either mitigated with better code or obviated by faster
> hardware [how's that for a sentence?].
>
> Net result is that the objection is probably based on once-
> valid but now out of date analysis.
>
> --
> Steven Lembark  2930 W. Palmer
> Workhorse Computing  Chicago, IL 60647
>+1 800 762 1582
>

-- 

C. Jon Larsen Chief Technology Officer, Richweb.com (804.307.6939)
SMTP: [EMAIL PROTECTED] (http://richweb.com/cjl_pgp_pub_key.txt)

Richweb.com:
Designing Open Source Internet Business Solutions since 1995
Building Safe, Secure, Reliable Cisco-Powered Networks since 1995

Re: mod_perl vs. C for high performance Apache modules

[Tim Gardner]

>So I'm trying to show that mod_perl doesn't suck, and that it is, in
>fact, a reasonable choice.

I think one of the selling points for mod_perl is its extensibility: 
modules can be written in C.  Depending on the C code you have access 
to, a good solution might be to try to wrap it into mod_perl modules.

Tim

Re: mod_perl vs. C for high performance Apache modules

[lembark]

-- Jeff Yoak <[EMAIL PROTECTED]> on 12/14/01 12:58:51 -0800

> This is something different.  The investor is in a related business, and has
> developed substantially similar software for years.  And it is really good.
> What's worse is that my normal, biggest argument isn't compelling in this
> case, that by the time this would be done in C, I'd be doing contract work on
> Mars.  The investor claims to have evaluated Perl vs. C years ago, to have
> witnessed that every single hit on the webserver under mod_perl causes a CPU
> usage spike that isn't seen with C, and that under heavy load mod_perl
> completely falls apart where C doesn't.  (This code is, of course, LONG gone
> so I can't evaluate it for whether the C was good and the Perl was screwy.)
> At any rate, because of this, he's spent years having good stuff written in
> C.  Unbeknownst to either me or my client, both this software and its
> developer were available to us, so in this case it would have been faster,
> cheaper and honestly even better, by which I mean more fully-featured.

Constructing the $r object in perl-space is an overhead
that mod_perl causes. This overhead has been managed more
effectively in recent versions of perl/mod_perl. A study
done "a few years ago" probably involved machines with 
significantly less core and CPU horsepower than the average
kiddie-games PC does today. Net result is that any overhead
caused by mod_perl in the previous study may well have been
either mitigated with better code or obviated by faster
hardware [how's that for a sentence?].

Net result is that the objection is probably based on once-
valid but now out of date analysis. 

--
Steven Lembark  2930 W. Palmer
Workhorse Computing  Chicago, IL 60647
   +1 800 762 1582

RE: mod_perl vs. C for high performance Apache modules

[brian moseley]

On Fri, 14 Dec 2001, Thomas Moore wrote:

> I spoke to the technical lead at Yahoo who said mod_perl
> will not scale as well as c++ when you get to their
> level of traffic, but for a large ecommerce site
> mod_perl is fine.

the old "memory is cheap" rationalization doesn't go over
very well at that scale either, eh.

Re: mod_perl vs. C for high performance Apache modules

[Dave Hodgkinson]

Toni Andjelkovic <[EMAIL PROTECTED]> writes:

> > 2.x linux kernels too.
> 
> that was an issue with 2.0.x, since 2.2.x
> you can do it with

That was what I meant...decimal point in the wrong place... :-)

-- 
David Hodgkinson, Wizard for Hirehttp://www.davehodgkinson.com
Editor-in-chief, The Highway Star   http://www.deep-purple.com
Deep Purple Family Tree news  http://www.slashrock.com
   Interim Technical Director, Web Architecture Consultant for hire

Re: mod_perl vs. C for high performance Apache modules

[Toni Andjelkovic]

Dave Hodgkinson wrote on Fri, Dec 14 2001 (20:54:22 +):
> "Perrin Harkins" <[EMAIL PROTECTED]> writes:
> > According to something I once read by David Filo, Yahoo also had to tweak
> > the FreeBSD code because they had trouble scaling *TCP/IP*!  I would say
> > their experience is not typical.
> 
> Increasing the number of file handles, I'd wager. That was an issue on

you really don't have to tweak any FreeBSD code
for that, just do

sysctl -w kern.maxfiles=10

and the file table will grow up to the new limit.

> 2.x linux kernels too.

that was an issue with 2.0.x, since 2.2.x
you can do it with

echo 10 > /proc/sys/fs/file-max

cheers,
-- 
Toni Andjelkovic
<[EMAIL PROTECTED]>

Re: mod_perl vs. C for high performance Apache modules

[Jeff Yoak]

All,

 I wasn't sure what volume of response to expect when I originally 
wrote.  Thank you all for the comments that you all are making.  They are 
helping.  Given that the response is fairly high, I'm waiting for stuff to 
roll in rather than replying to each of you.  Don't think it is falling on 
unappreciating ears.  :-)
 To respond to a few recurring comments / questions:

Me?  I've spent most of the last four years working on mod_perl-based stuff 
and most of the last eight working with Perl.  Actually I've worked with 
folks who were involved with some of the projects you've mentioned, having 
been at idealab!, a parent of eToys and CitySearch.  One of the original 
(THE original?) developer at CitySearch was probably the most helpful 
mentor / teacher I've ever worked with.  I programmed in C a lot early in 
my career, but at this point I couldn't write anything substantial without 
brushing up, and frankly wouldn't care to.  It just isn't as fun to work 
with C.  But then, the argument, "But if you used C, you wouldn't get to 
work with ME!" may not convince some of these people with their values all 
screwed up...  ;-)

The project?  What I consider to be standard web junk.  About 30% ecommerce 
combined with lots of database-driven, interactive content, some 
authentication foo and things like that.  The thing is that it is in the 
adult industry and the investor in question turns on the hose... well... 
there will be lots of traffic.

Mistakes and misunderstandings?  Sure.  And yes, as some of you have 
pointed out publicly or privately, not my fault.  I was kept very insulated 
from the people who were familiar with the alternatives.  My involvement at 
this point is to try to smooth things over and keep the project 
functional.  These immediate problems aside, the people involved are great 
to work with and if everyone feels better about the situation and things 
move forward in the best possible way from here, there will be a lot of 
valuable work for me.  So I'm trying to help.

More than you cared to hear about and terribly off-topic for the 
list?  Sure.  But you did sorta ask, and somehow it seemed rude to not 
reply.  Forgive me and thanks for providing all of your commentary.

Cheers,
Jeff




-- 
Jeff Yoak   626-705-6996

Re: mod_perl vs. C for high performance Apache modules

[wsheldah]

"... years ago ..."   Are you even sure he evaluated mod_perl and not Perl CGI
scripts?? Launching the interpreter and compiling every time might spike the
CPU.  Like others have said, you would really have to benchmark the mod_perl and
Apache that you're using now; both have improved considerably over the years.
You can point to the success stories others have mentioned to show that it
really is one of the standard industry solutions that works well in many
high-traffic situations. He's clearly arguing from outdated data and emotion.

Also, if you delivered what you said you would deliver, that ought to be the end
of it. If the site is performing according to expectations and promises, then he
doesn't have any real basis to complain. If it would truly deliver tangible
advantages to the client to rewrite it in C, or incorporate some of his C code
in your work (Perl and C can get along sometimes), then you might consider
giving him a quote to rewrite it. Ask him to put up or shut up. A gentler
approach might be, "Given we did what we did (and it works), what do you really
want us to do now? What should happen from this point forward?" If there isn't a
constructive answer to that, then all you're dealing with is a temper tantrum.

---Wes Sheldahl
( father of four young'uns )



Jeff Yoak <[EMAIL PROTECTED]> on 12/14/2001 03:58:51 PM

To:   Thomas Eibner <[EMAIL PROTECTED]>
cc:   [EMAIL PROTECTED] (bcc: Wesley
      Sheldahl/Lex/Lexmark)
Subject:  Re: mod_perl vs. C for high performance Apache modules


[snip]


So he's upset.  Everyone acknowledges that given our particular
circumstances, it would have been better to build upon what we already had,
but because of his previous experience he feels that mod_perl wasn't even a
responsible choice even within the limits of our lack of knowledge of his
software and its availability.

[snip]

Cheers,
Jeff

Re: mod_perl vs. C for high performance Apache modules

[Perrin Harkins]

> So I'm trying to show that mod_perl doesn't suck, and that it is, in fact,
> a reasonable choice.  Though within these limits it is still reasonable to
> point out the development cycle, emotionally it is the least compelling
> form of argument, because the investor has a hard time removing from
> consideration that given our particular situation, there was a very fast
> solution in using his C-based routines.

Well, that is the primary reason for using Perl over C, and you really have
to count maintenance and the relative likelihood of C-ish bugs like buffer
overflows as part of it.

Well-coded C should be faster than Perl, but Perl is fast enough for nearly
any web-based application.  If this guy saw CPU spikes, he probably had
something else wrong, like running out of memory.

You might find this article aboout C and Perl performance useful:
http://www.perl.com/pub/a/2001/06/27/ctoperl.html

- Perrin

Re: mod_perl vs. C for high performance Apache modules

[Simon Rosenthal]

At 03:58 PM 12/14/2001, Jeff Yoak wrote:
>At 09:15 PM 12/14/2001 +0100, Thomas Eibner wrote:
>>The key to mod_perl development is speed, there are numerous testimonials
>>from users implementing a lot of work in a very short time with mod_perl.
>>Ask the clients investor wheter he wants to pay for having everything you
>>did rewritten as an Apache module in C. That is very likely going to take
>>a lot of time.
>
>Thank you for your reply.  I realized in reading it that my tone leads one 
>to the common image of a buzzword driven doody-head who wants this because 
>of what he read in Byte.  That's certainly common enough, and I've never 
>had a problem dealing with such types.  (Well... not an unsolvable 
>problem... :-)
>
>This is something different.  The investor is in a related business, and 
>has developed substantially similar software for years.  And it is really 
>good.  What's worse is that my normal, biggest argument isn't compelling 
>in this case, that by the time this would be done in C, I'd be doing 
>contract work on Mars.  The investor claims to have evaluated Perl vs. C 
>years ago, to have witnessed that every single hit on the webserver under 
>mod_perl causes a CPU usage spike that isn't seen with C, and that under 
>heavy load mod_perl completely falls apart where C doesn't.  (This code 
>is, of course, LONG gone so I can't evaluate it for whether the C was good 
>and the Perl was screwy.)  At any rate, because of this, he's spent years 
>having good stuff written in C.  Unbeknownst to either me or my client, 
>both this software and its developer were available to us, so in this case 
>it would have been faster, cheaper and honestly even better, by which I 
>mean more fully-featured.

CPU usage is certainly one factor... but CPUs are cheap compared to 
development man-hours.

Since you haven't provided any details on the application, this may not be 
relevant, but most of the web apps that we write (and I read about 
here)  spend much of their time waiting for responses from other back-end 
servers - databases, NFS mounted file systems, or whatever. It's probably 
undeniable that a well written C application will run faster than almost 
anything in an interpreted language, but that may not make much of a 
difference to the total response time.

-Simon





Simon Rosenthal ([EMAIL PROTECTED])
Web Systems Architect
Northern Light Technology
One Athenaeum Street. Suite 1700, Cambridge, MA  02142
Phone:  (617)621-5296: URL:  http://www.northernlight.com
"Northern Light - Just what you've been searching for"

Re: mod_perl vs. C for high performance Apache modules

[Ged Haywood]

Hi there,

On Fri, 14 Dec 2001, Jeff Yoak wrote:

> This is something different. [big snip]

Indeed it is.  It's a refreshingly honest appraisal of what might,
in hindsight, have been easily avoided mistakes.

And nobody ever did anything without making a few.

Thanks.

73,
Ged.

PS: Are any of those C routines available Open Source? :)

Re: mod_perl vs. C for high performance Apache modules

[Stephen Clouse]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Fri, Dec 14, 2001 at 12:58:51PM -0800, Jeff Yoak wrote:
> This is something different.  The investor is in a related business, and 
> has developed substantially similar software for years.  And it is really 
> good.  What's worse is that my normal, biggest argument isn't compelling in 
> this case, that by the time this would be done in C, I'd be doing contract 
> work on Mars.  The investor claims to have evaluated Perl vs. C years ago, 
> to have witnessed that every single hit on the webserver under mod_perl 
> causes a CPU usage spike that isn't seen with C, and that under heavy load 
> mod_perl completely falls apart where C doesn't.  (This code is, of course, 
> LONG gone so I can't evaluate it for whether the C was good and the Perl 
> was screwy.)  At any rate, because of this, he's spent years having good 
> stuff written in C.  Unbeknownst to either me or my client, both this 
> software and its developer were available to us, so in this case it would 
> have been faster, cheaper and honestly even better, by which I mean more 
> fully-featured.

Ah, the good old "my timeless knowledge from the Nixon administration still 
applies" argument.  Remember that the speed of both languages has increased 
geometrically as a function of hardware speed.  5 years ago you may have been 
talking a difference of a second or two between Perl and C.  Nowadays you'll be 
measuring the differences in milliseconds, if that much.

And you're right, one still can't verify the veracity of the ancient benchmarked 
code.  On another list someone was dealing with Perl and DBI against PHP for a 
database-driven site, with nothing to go on but some ApacheBench numbers showing 
the PHP page about six orders of magnitude faster.  Then one "minor" difference 
of note: the Perl program had output about 70MB of data; the PHP program, about 
940 bytes.  I assume that the PHP program was doing a null loop or something 
where the query should have been.

Get him to do a real unbiased comparison on modern hardware, and then defy him 
to claim that the milliseconds he saves are worth the effort.

- -- 
Stephen Clouse <[EMAIL PROTECTED]>
Senior Programmer, IQ Coordinator Project Lead
The IQ Group, Inc. 

-BEGIN PGP SIGNATURE-
Version: PGP 6.5.8

iQA/AwUBPBpsFQOGqGs0PadnEQJ9TgCgtZnIhFLq/L/DeZA5CS4yAjzBRiEAn3ED
lTRl2/yaG9eH7BK785GKajp3
=jLX5
-END PGP SIGNATURE-

Re: mod_perl vs. C for high performance Apache modules

[Thomas Eibner]

On Fri, Dec 14, 2001 at 12:58:51PM -0800, Jeff Yoak wrote:
> At 09:15 PM 12/14/2001 +0100, Thomas Eibner wrote:
> >The key to mod_perl development is speed, there are numerous testimonials
> >from users implementing a lot of work in a very short time with mod_perl.
> >Ask the clients investor wheter he wants to pay for having everything you
> >did rewritten as an Apache module in C. That is very likely going to take
> >a lot of time.
> 
> Thank you for your reply.  I realized in reading it that my tone leads one 
> to the common image of a buzzword driven doody-head who wants this because 
> of what he read in Byte.  That's certainly common enough, and I've never 
> had a problem dealing with such types.  (Well... not an unsolvable 
> problem... :-)

True, it sounded a bit like that.

> This is something different.  The investor is in a related business, and 
> has developed substantially similar software for years.  And it is really 
> good.  What's worse is that my normal, biggest argument isn't compelling in 
> this case, that by the time this would be done in C, I'd be doing contract 
> work on Mars.  The investor claims to have evaluated Perl vs. C years ago, 
> to have witnessed that every single hit on the webserver under mod_perl 
> causes a CPU usage spike that isn't seen with C, and that under heavy load 
> mod_perl completely falls apart where C doesn't.  (This code is, of course, 
> LONG gone so I can't evaluate it for whether the C was good and the Perl 
> was screwy.)  At any rate, because of this, he's spent years having good 
> stuff written in C.  Unbeknownst to either me or my client, both this 
> software and its developer were available to us, so in this case it would 
> have been faster, cheaper and honestly even better, by which I mean more 
> fully-featured.

I'm wondering where the investor have been hiding if he isn't telling you
this until now. It's hard to pull on resources that you have no idea are
available to you. If he did the comparision years ago he should be doing
his tests over again, since a lot of things have happened over the last
few years. From what you write it seems to me that he can't just be using
plain CGI in C, since it still has the fork overhead that mod_perl doesn't
and even a small hello world test would prove that mod_perl is faster
there (I didn't try it, but I'm sure we could make some tests). 
Although you haven't told us much about what the actual application you
wrote for your client does, I must say that I've found it more error
prone and time-consuming doing any database stuff in C. I'm not a C
programmer of heart, but I have done a fair amount of programming with
it and it certainly hasn't been as fast to develop as applications
relying on DBI.
You haven't told us what you normally program in either, so we don't
know if you do mod_perl most of the time or you just write your stuff
in C regulary, this could also have an impact on the length of the
project in the end.

> So he's upset.  Everyone acknowledges that given our particular 
> circumstances, it would have been better to build upon what we already had, 
> but because of his previous experience he feels that mod_perl wasn't even a 
> responsible choice even within the limits of our lack of knowledge of his 
> software and its availability.

To me it seems like he should be upset with himself. If he knows he has
some good tools he should have told your client way before this project
even started anyway.

> So I'm trying to show that mod_perl doesn't suck, and that it is, in fact, 
> a reasonable choice.  Though within these limits it is still reasonable to 
> point out the development cycle, emotionally it is the least compelling 
> form of argument, because the investor has a hard time removing from 
> consideration that given our particular situation, there was a very fast 
> solution in using his C-based routines.

I certainly hope that you find something that will show him that mod_perl
isn't a bad choice. 

-- 
  Thomas Eibner  DnsZone 
  mod_pointer  

RE: mod_perl vs. C for high performance Apache modules

[Andy Sharp]

Mod_perl doesn't suck, and it certainly doesn't have a huge hit on the
CPU.  (of course it all depends what you're doing, but for the most part
it's small)

Having used many high level web development environments,  from C to
Java to TCL and perl, I find mod_perl at the top end of the scalability
scale.  Plenty of fast and heavily loaded sites use mod_perl without
throwing tonnes of hardware at it, AND as a side bonus, chances are the
development time was faster than writing in almost anyything else.

I run a wee little web site,  it runs on 4 servers (1 static server, 2
mysql servers, and one mod_perl server)  all told, this hardware cost
about $10k usd.

Yesterday,  the site served 2,000,000 page loads (ads, impressions, how
every you count it), with about 6 Million requests to the mod_perl
server, which in turn passed on about 200 Million requests to the DB
server.  Yes, I struggle with performance, but under most other
development environments I'd have been up against the wall a long time
ago.  

Mod_perl is NOT CPU intensive.  It IS memory intensive.  Due to the
architecture of apache, and the memory overhead some care must be taken
when planning a large capacity site, but fortunately all of this is
nicely documented in the guide.

There are other solutions.  Yes, I suppose I could rewrite large parts
of the site in C.  It certainly wouldn't help with development (which is
always patching this, adding feature Z),  it might, maybe reduce some of
the memory overhead.  My largest challenge has actually been to get
mysql (written in C) to properly handle it's share of the workload.

Take a perusal at the Stories section on perl.apache.org
http://perl.apache.org/stories/

ValueClick serves an incredible number of impressions via their mod_perl
systems.  

It scales,  it's fast,  and it doesn't take a lot of hardware.

--A




> -Original Message-
> From: Jeff Yoak [mailto:[EMAIL PROTECTED]] 

> This is something different.  The investor is in a related 
> business, and 
> has developed substantially similar software for years.  And 
> it is really 
> good.  What's worse is that my normal, biggest argument isn't 
> compelling in 
> this case, that by the time this would be done in C, I'd be 
> doing contract 
> work on Mars.  The investor claims to have evaluated Perl vs. 
> C years ago, 
> to have witnessed that every single hit on the webserver 
> under mod_perl 
> causes a CPU usage spike that isn't seen with C, and that 
> under heavy load 
> mod_perl completely falls apart where C doesn't.  (This code 
> is, of course, 
> LONG gone so I can't evaluate it for whether the C was good 
> and the Perl 
> was screwy.)  At any rate, because of this, he's spent years 
> having good 
> stuff written in C.  Unbeknownst to either me or my client, both this 
> software and its developer were available to us, so in this 
> case it would 
> have been faster, cheaper and honestly even better, by which 
> I mean more 
> fully-featured.
> 
> So he's upset.  Everyone acknowledges that given our particular 
> circumstances, it would have been better to build upon what 
> we already had, 
> but because of his previous experience he feels that mod_perl 
> wasn't even a 
> responsible choice even within the limits of our lack of 
> knowledge of his 
> software and its availability.
> 
> So I'm trying to show that mod_perl doesn't suck, and that it 
> is, in fact, 
> a reasonable choice.  Though within these limits it is still 
> reasonable to 
> point out the development cycle, emotionally it is the least 
> compelling 
> form of argument, because the investor has a hard time removing from 
> consideration that given our particular situation, there was 
> a very fast 
> solution in using his C-based routines.

Re: mod_perl vs. C for high performance Apache modules

[John Armstrong]

I dont think its your responsibility anymore. If the investor had a 
preference he should have stated it BEFORE work began. If your client 
did not keep him informed then your client has that burden to bear.

You did your job, the client likes what you did, it works. Let them 
fight the political battle.

Comparing mod_perl 'years ago' is totally irrelevant. Modperl has grown 
up so much since 'years ago' that its not even funny.

J-

On Friday, December 14, 2001, at 12:58 PM, Jeff Yoak wrote:

> At 09:15 PM 12/14/2001 +0100, Thomas Eibner wrote:
>> The key to mod_perl development is speed, there are numerous 
>> testimonials
>> from users implementing a lot of work in a very short time with 
>> mod_perl.
>> Ask the clients investor wheter he wants to pay for having everything 
>> you
>> did rewritten as an Apache module in C. That is very likely going to 
>> take
>> a lot of time.
>
> Thank you for your reply.  I realized in reading it that my tone leads 
> one to the common image of a buzzword driven doody-head who wants this 
> because of what he read in Byte.  That's certainly common enough, and 
> I've never had a problem dealing with such types.  (Well... not an 
> unsolvable problem... :-)
>
> This is something different.  The investor is in a related business, 
> and has developed substantially similar software for years.  And it is 
> really good.  What's worse is that my normal, biggest argument isn't 
> compelling in this case, that by the time this would be done in C, I'd 
> be doing contract work on Mars.  The investor claims to have evaluated 
> Perl vs. C years ago, to have witnessed that every single hit on the 
> webserver under mod_perl causes a CPU usage spike that isn't seen with 
> C, and that under heavy load mod_perl completely falls apart where C 
> doesn't.  (This code is, of course, LONG gone so I can't evaluate it 
> for whether the C was good and the Perl was screwy.)  At any rate, 
> because of this, he's spent years having good stuff written in C.  
> Unbeknownst to either me or my client, both this software and its 
> developer were available to us, so in this case it would have been 
> faster, cheaper and honestly even better, by which I mean more 
> fully-featured.
>
> So he's upset.  Everyone acknowledges that given our particular 
> circumstances, it would have been better to build upon what we already 
> had, but because of his previous experience he feels that mod_perl 
> wasn't even a responsible choice even within the limits of our lack of 
> knowledge of his software and its availability.
>
> So I'm trying to show that mod_perl doesn't suck, and that it is, in 
> fact, a reasonable choice.  Though within these limits it is still 
> reasonable to point out the development cycle, emotionally it is the 
> least compelling form of argument, because the investor has a hard time 
> removing from consideration that given our particular situation, there 
> was a very fast solution in using his C-based routines.
>
>> Take a look at Joshua Chamas benchmarks (Although they're only hello
>> world style apps) it shows that mod_perl is pretty fast.
>
> I will look for this particularly.  Thanks.
>
> Cheers,
> Jeff
>
>
>>
>
> -- Jeff Yoak   626-705-6996
>

Re: mod_perl vs. C for high performance Apache modules

[Stephen Clouse]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Fri, Dec 14, 2001 at 12:12:09PM -0800, Jeff Yoak wrote:
>  Recently I did a substantial project for a client in using 
> mod_perl.  That client is happy with the work, but an investor with their 
> company is very angry because of what a horrible choice mod_perl is for 
> high-load web applications compared with Apache modules and even CGI 
> programs, written in C.

Word of the day: ultracrepidarianism

Of course, financial investors are experts in the field of backend systems.  One 
in this situation wishes your client would pick up on that.  The fact that he's 
comparing mod_perl (an Apache module) to an Apache module should be a glaring 
sign of said investor's utter cluelessness.  The fact that he's even mentioning 
CGI versus a server API-level module -- well, I'll stop now before I embarass 
someone.

>  If anyone on this list could forward any resources 
> that do comparisons along these lines, or even analysis of mod_perl's 
> handling of high-load web traffic, I would be very grateful.

Whether C is more efficient than a RAD language like Perl (or Python or PHP) is 
an incredibly stupid question, because one with sufficient knowledge should know 
already that the answer is yes.  The real question is whether said efficiency is 
worth the arduous hassle of building the app in C.  Considering the speed at 
which modern servers execute interpreted or JIT-compiled languages, that answer 
is almost always no.  There is "fast", but then there's "fast enough (and 
finished developing in 25% of the time)".  They'll have to decide what's more 
important.

Now, as for case studies, here's a quick list:

http://perl.apache.org/sites.html

And then this article, outlining how mod_perl was used to build the eToys site:

http://www.perl.com/pub/a/2001/10/17/etoys.html

And now for a completely shameless plug:

http://www.iqcoordinator.com/

Which is a processing-intensive work order management system, built entirely on 
mod_perl.  (Some day I will write an article myself :)  Our systems don't even 
break a sweat.  Actually, mod_perl saved us from having to buy more hardware.  
It's plenty fast.

- -- 
Stephen Clouse <[EMAIL PROTECTED]>
Senior Programmer, IQ Coordinator Project Lead
The IQ Group, Inc. 

-BEGIN PGP SIGNATURE-
Version: PGP 6.5.8

iQA/AwUBPBpphQOGqGs0PadnEQLW2gCdGonCCAU0aCauPTTsIZMvzcWU1mIAnj3J
BBNeCw2RdSN90qTCYDFLUYPP
=Exa7
-END PGP SIGNATURE-

RE: mod_perl vs. C for high performance Apache modules

[Dave Rolsky]

On Fri, 14 Dec 2001, Thomas Moore wrote:

> I spoke to the technical lead at Yahoo who said mod_perl will not scale as
> well as c++ when you get to their level of traffic, but for a large
> ecommerce site mod_perl is fine.

Well, Yahoo is _extremely_ atypical.  And they do a lot of stuff that
involves custom coded Apache modules (in C, I think, not C++, though maybe
both).  Amazon reportedly uses a similar approach.  Jeremy Zawodny is a
frequent contributor on the MySQL list and a Perl guy so if you're really
curious you could probably email him a polite question for more details.
He's a nice guy.


-dave

/*==
www.urth.org
We await the New Sun
==*/

Re: mod_perl vs. C for high performance Apache modules

[Todd Finney]

At 03:12 PM 12/14/01, Jeff Yoak wrote:
>Recently I did a substantial project for a client in using 
> mod_perl.  That client is happy with the work, but an investor with 
> their company is very angry because of what a horrible choice 
> mod_perl is for high-load web applications compared with Apache 
> modules and even CGI programs, written in C.  If anyone on this list 
> could forward any resources that do comparisons along these lines, or 
> even analysis of mod_perl's handling of high-load web traffic, I 
> would be very grateful.

How about the myths page?

http://perl.apache.org/perl_myth.html

Todd

Re: mod_perl vs. C for high performance Apache modules

[Jeff Yoak]

At 09:15 PM 12/14/2001 +0100, Thomas Eibner wrote:
>The key to mod_perl development is speed, there are numerous testimonials
>from users implementing a lot of work in a very short time with mod_perl.
>Ask the clients investor wheter he wants to pay for having everything you
>did rewritten as an Apache module in C. That is very likely going to take
>a lot of time.

Thank you for your reply.  I realized in reading it that my tone leads one 
to the common image of a buzzword driven doody-head who wants this because 
of what he read in Byte.  That's certainly common enough, and I've never 
had a problem dealing with such types.  (Well... not an unsolvable 
problem... :-)

This is something different.  The investor is in a related business, and 
has developed substantially similar software for years.  And it is really 
good.  What's worse is that my normal, biggest argument isn't compelling in 
this case, that by the time this would be done in C, I'd be doing contract 
work on Mars.  The investor claims to have evaluated Perl vs. C years ago, 
to have witnessed that every single hit on the webserver under mod_perl 
causes a CPU usage spike that isn't seen with C, and that under heavy load 
mod_perl completely falls apart where C doesn't.  (This code is, of course, 
LONG gone so I can't evaluate it for whether the C was good and the Perl 
was screwy.)  At any rate, because of this, he's spent years having good 
stuff written in C.  Unbeknownst to either me or my client, both this 
software and its developer were available to us, so in this case it would 
have been faster, cheaper and honestly even better, by which I mean more 
fully-featured.

So he's upset.  Everyone acknowledges that given our particular 
circumstances, it would have been better to build upon what we already had, 
but because of his previous experience he feels that mod_perl wasn't even a 
responsible choice even within the limits of our lack of knowledge of his 
software and its availability.

So I'm trying to show that mod_perl doesn't suck, and that it is, in fact, 
a reasonable choice.  Though within these limits it is still reasonable to 
point out the development cycle, emotionally it is the least compelling 
form of argument, because the investor has a hard time removing from 
consideration that given our particular situation, there was a very fast 
solution in using his C-based routines.

>Take a look at Joshua Chamas benchmarks (Although they're only hello
>world style apps) it shows that mod_perl is pretty fast.

I will look for this particularly.  Thanks.

Cheers,
Jeff


>

-- 
Jeff Yoak   626-705-6996

Re: mod_perl vs. C for high performance Apache modules

[Dave Hodgkinson]

"Perrin Harkins" <[EMAIL PROTECTED]> writes:

> > I spoke to the technical lead at Yahoo who said mod_perl will not scale as
> > well as c++ when you get to their level of traffic, but for a large
> > ecommerce site mod_perl is fine.
> 
> According to something I once read by David Filo, Yahoo also had to tweak
> the FreeBSD code because they had trouble scaling *TCP/IP*!  I would say
> their experience is not typical.

Increasing the number of file handles, I'd wager. That was an issue on
2.x linux kernels too.

-- 
David Hodgkinson, Wizard for Hirehttp://www.davehodgkinson.com
Editor-in-chief, The Highway Star   http://www.deep-purple.com
Deep Purple Family Tree news  http://www.slashrock.com
   Interim Technical Director, Web Architecture Consultant for hire

Re: mod_perl vs. C for high performance Apache modules

[Perrin Harkins]

> I spoke to the technical lead at Yahoo who said mod_perl will not scale as
> well as c++ when you get to their level of traffic, but for a large
> ecommerce site mod_perl is fine.

According to something I once read by David Filo, Yahoo also had to tweak
the FreeBSD code because they had trouble scaling *TCP/IP*!  I would say
their experience is not typical.
- Perrin

RE: mod_perl vs. C for high performance Apache modules

[Rob Nagler]

> I spoke to the technical lead at Yahoo who said mod_perl will not scale as
> well as c++ when you get to their level of traffic, but for a large
> ecommerce site mod_perl is fine.

Scalability has less to do with language/execution environment than
which database you are using.  Path length is affected by language,
but that's usually not the major factor in scalability.  You want
short path lengths to get more efficiency out of your machines.

Rob

RE: mod_perl vs. C for high performance Apache modules

[Thomas Moore]

I spoke to the technical lead at Yahoo who said mod_perl will not scale as
well as c++ when you get to their level of traffic, but for a large
ecommerce site mod_perl is fine.

I think people just stick to what they started with. We wrote our site
racesearch.com using mod_perl and it is super fast. (faster than Amazon.com)

-tom


-Original Message-
From: Jeff Yoak [mailto:[EMAIL PROTECTED]]
Sent: Friday, December 14, 2001 12:12 PM
To: [EMAIL PROTECTED]
Subject: mod_perl vs. C for high performance Apache modules



Hi All,

 Recently I did a substantial project for a client in using
mod_perl.  That client is happy with the work, but an investor with their
company is very angry because of what a horrible choice mod_perl is for
high-load web applications compared with Apache modules and even CGI
programs, written in C.  If anyone on this list could forward any resources
that do comparisons along these lines, or even analysis of mod_perl's
handling of high-load web traffic, I would be very grateful.

Cheers,
Jeff



--
Jeff Yoak   626-705-6996

Re: mod_perl vs. C for high performance Apache modules

[Dave Hodgkinson]

Jeff Yoak <[EMAIL PROTECTED]> writes:

> Hi All,
> 
>  Recently I did a substantial project for a client in using
>  mod_perl.  That client is happy with the work, but an
>  investor with their company is very angry because of what a
>  horrible choice mod_perl is for high-load web applications
>  compared with Apache modules and even CGI programs, written
>  in C.  If anyone on this list could forward any resources
>  that do comparisons along these lines, or even analysis of
>  mod_perl's handling of high-load web traffic, I would be very
>  grateful.

Kill the investors. Really.

-- 
David Hodgkinson, Wizard for Hirehttp://www.davehodgkinson.com
Editor-in-chief, The Highway Star   http://www.deep-purple.com
Deep Purple Family Tree news  http://www.slashrock.com
   Interim Technical Director, Web Architecture Consultant for hire

Re: mod_perl vs. C for high performance Apache modules

[John Armstrong]

Investors suck like that. I have had to fight many of these battles.

The first thing to do is find out specifically _why_ the investor thinks 
that so you can counter their claims. Trying to counter vague notions of 
'terrible' is impossible. The opponent has to commit to an opinion 
before you can fight it.

Then just point to many of the well-known sites on the success pages 
that are high volume and do quite well with mod_perl

Explain that to do this project in Java or C or CGI would have costed 
more, taken longer and given negligible benefits.

Thats what I do. I usually also show how my architecture can interact 
with other architectures via XML or directly at teh data store level.

Good luck
J

On Friday, December 14, 2001, at 12:12 PM, Jeff Yoak wrote:

>
> Hi All,
>
> Recently I did a substantial project for a client in using 
> mod_perl.  That client is happy with the work, but an investor with 
> their company is very angry because of what a horrible choice mod_perl 
> is for high-load web applications compared with Apache modules and even 
> CGI programs, written in C.  If anyone on this list could forward any 
> resources that do comparisons along these lines, or even analysis of 
> mod_perl's handling of high-load web traffic, I would be very grateful.
>
> Cheers,
> Jeff
>
>
>
> -- Jeff Yoak   626-705-6996
>

Re: mod_perl vs. C for high performance Apache modules

[Thomas Eibner]

On Fri, Dec 14, 2001 at 12:12:09PM -0800, Jeff Yoak wrote:
> 
> Hi All,
> 
>  Recently I did a substantial project for a client in using 
> mod_perl.  That client is happy with the work, but an investor with their 
> company is very angry because of what a horrible choice mod_perl is for 
> high-load web applications compared with Apache modules and even CGI 
> programs, written in C.  If anyone on this list could forward any resources 
> that do comparisons along these lines, or even analysis of mod_perl's 
> handling of high-load web traffic, I would be very grateful.

CGI programs (even written in C) are very likely to be a lot slower than
mod_perl handlers so I can't see where he gets that argument from. Does
he really know what mod_perl is?
The key to mod_perl development is speed, there are numerous testimonials
from users implementing a lot of work in a very short time with mod_perl.
Ask the clients investor wheter he wants to pay for having everything you
did rewritten as an Apache module in C. That is very likely going to take
a lot of time.
Take a look at Joshua Chamas benchmarks (Although they're only hello
world style apps) it shows that mod_perl is pretty fast.

my $cent = 2;

-- 
  Thomas Eibner  DnsZone 
  mod_pointer  

RE: mod_perl vs. C for high performance Apache modules

[Stathy Touloumis]

http://www.perl.com/pub/a/2001/10/17/etoys.html

Yea, mod_perl really sucks ; )

I have even worked on poorly architectured and coded sites which still
performed fairly well.

> Recently I did a substantial project for a client in using
> mod_perl.  That client is happy with the work, but an investor with their
> company is very angry because of what a horrible choice mod_perl is for
> high-load web applications compared with Apache modules and even CGI
> programs, written in C.  If anyone on this list could forward any
> resources
> that do comparisons along these lines, or even analysis of mod_perl's
> handling of high-load web traffic, I would be very grateful.