Hi, I've just emerged from about 5 months of fairly continuous late
night development, what started as a few Berkeley db files and a tiny
cgi perl script has now grown to (for me) a monster size, kept largely
at bay by modperl..
I hope this story will be interesting to those who are taking the first
steps, and forgive any idiocy or simple wrong-headedness, because
i was too dumb to RTFM (or rather, RTFMLArchive), so I learned some
things the hard way, and am still learning.
The site is a not a commercial one, which is why I am free to
ramble on about it.. I am sure there are far more impressive modperl
sites now under NDAs or corporate lawyer mufflers, but without those
to read, you get mine ;)
As it stands, the site is entirely dynamically generated, and visited
by 4 to 10 thousand people a day. I wish that I could say that I read
the modperl FAQ and wrote it to handle that load from day one, but
that was not the case:
My understanding of why modperl was cool came tangentially from
Philip Greenspun on photo.net rather single-mindedly promoting, albeit
with persuasion, the AOLserver... the story that made most impression
was the bill gates personal wealth clock.. getting into netscape home page
for a day he had to cope with 2 hits per second that needed code run.
Thereby proving the point neatly that a fork/exec/cgi isn't going to cut
it. At the time, 2 hits a second seemed pretty impressive.. my god, thats
two people per SECOND.. i mean, oh what, 7600 per hour? 76,000 during
daylight ? well anyway, I digress.
Using AOLserver wasn't an option for me, personally because I am too tired
to get back into TCL and stub my toe 10 times a week once the programs
got so big that side-effects ruled.. I already thought Apache was the
swiss army knife of webservers, perl5 and cpan was already my home, and
the fact that slashdot used it (and modperl, as I slowly discovered),
just cemented the choice.
But just when you've climbed a peak you discover whole new and steeper
slopes ahead...
First quandry
I had no idea what traffic for my website was likely to be, and even
less idea what it that translated to in terms of bandwidth.. these things
they don't teach you in cnet! I wince to remember at one point,
misunderstanding a Malda comment about an ISDN line to their house,
I even asked in wonderment if they hosted slashdot on 128kbps!
Nevertheless, the relationship between visitors, pages, and bandwidth
is not well understood because asking at some high powered places got
me no good answers, including at Philips site as well.
Well, I figured, bandwidth is at least something you can get more of if
you need it and want to pay, so I decided to watch it, and let that
one answer itself. (more on that later).
The first attempts.
Initially I setup squid in accelerator mode, and looked at the result,
but during development, hits were few, and bugs were many, so dealing
with an extra layer of config files and logs was too much so I put squid
behind 'in case of emergency break glass', and switched back to
straight apache/modperl.
Due to the excellent online Apache modperl book, which I think back in
June was still being put together, and particularly the performance
tuning guide, I discovered how to setup a startup.pl that gave nice
backtraces for errors (the Carp module and code fragment), plus I managed
to navigate the global variable problem, and always used strict..
to this day though, I find having a small set of global variables
that needs to be shared amongst some modules to be somewhat of a pain..
my current solution of putting them all in a hash array stored in
another module is ok, but kind of annoys me for some reason.
Newbie mistakes
Watching my machine choke when a syntax error caused apache to pile up
until swap was exhausted was not fun, especially on a live server, so I
became paranoid about syntax checking .. a vimrc from dejanews
made a great perl IDE, allowing me to '!make' and get taken to each
error, as I put them in, rather than in a batch at the end from the
command line, this hack I think, was my personal favourite of 1999
(so sue me, a vi bigot.. I cant/wont sit in emacs all day).
DB files are no way to run a war
Starting with DB files was probably the silliest decision I made. They
work fine in testing, but when you pressure them, (ie, "go live"), they
start to fall apart. Elaborate locking unlocking code fragments exist
but I don't trust them, and have the corrupted DBs to prove it.
MySQL goes with modperl like sushi and rice
I cant say enough good things about MySQL. Of literally all the lego
I've used in this web site, this has been rock solid... Despite doing
at times ridiculously silly things with tables rows and fields, it
has never ONCE crashed or corrupted the data. At my place of employ
you cannot say that about any commercial database they use, and they
are actually under less stress, plus have DBAs running around keeping
them on course... Oracle may not corrupt, but I'm damn
